[ANNOUNCE] Progress of Apache Flink 1.10 #1
Locked
 
|
Hi community,
Since we are now more than one month into the Flink 1.10 release cycle, we thought it would be adequate to give a progress update. Below we have included a list of the ongoing efforts that we are aware of, together with a brief summary of their state. As always, the list is not meant to be exhaustive. If you are working on something that is not included here, feel free to use this thread to share your progress. Note that because we are still relatively at the beginning of the release cycle, most of the progress is limited to FLIPs that are accepted or being voted on. - Improving Flink’s build system & CI - Repository Split [1] - Discussed on the ML but consensus to split the repository was not reached. - Reduce Build Time [2] - Discussion is ongoing. Currently, using Azure Pipelines and Gradle are being evaluated. - Support Java 11 [3] - Implementation is in progress (18/21 subtasks resolved) - Table API improvements - FLIP-54 Evolve ConfigOption and Configuration [4] - Under discussion. - FLIP-59 Enable Execution Configuration from Configuration Object [5] - Under discussion. - Full Data Type Support in Planner [6] - Implementation in progress. - FLIP-66 Support Time Attribute in SQL DDL [7] - FLIP voting. - FLIP-70 Support Computed Column [8] - Under discussion. - FLIP-63 Rework Table Partition Support [9] - FLIP voting - FLIP-51 Rework of Expression Design [10] - FLIP accepted, implementation in progress. - FLIP-55 Introduction of a TableAPI Java Expression DSL [11] - Under discussion. - FLIP-64 Support for Temporary Objects in Table Module [12] - Under discussion. - FLIP-65 New Type Inference for Table API UDFs - Under discussion. - Hive compatibility completion (DDL/UDF) to support full Hive integration - FLIP-57 Rework FunctionCatalog [13] - FLIP voting - FLIP-68 Extend Core Table System with Modular Plugins [14] - FLIP voting was initiated [15] but temporarily withdrawn due to lack of community bandwidth. - Finer grained resource management - FLIP-49: Unified Memory Configuration for TaskExecutors [16] - FLIP accepted. Implementation is in progress. - FLIP-53: Fine Grained Operator Resource Management [17] - FLIP accepted. Implementation details are under discussion. - FLIP-56: Dynamic Slot Allocation [18] - FLIP accepted. Implementation not started yet. - Finish scheduler re-architecture [19] - Implementation is in progress. - FLIP-27: Refactor Source Interface [20] - FLIP accepted. Implementation is in progress. - Executor/Client refactoring [21] - Discussion already reached consensus - FLIP is coming. A PoC implementation is also ready. - FLIP-36 Support Interactive Programming [22] - Reviewing FLIP-67, which changes the intermediate result management in runtime, which is what FLIP-36 will be built on top of. - FLIP-58: Flink Python User-Defined Stateless Function for Table [23] - Implementation is in progress (3/15 subtask resolved). - Python environment and dependency management under discussion - FLIP-50: Spill-able Heap Keyed State Backend [24] - FLIP was accepted. Implementation is in progress. - RocksDB Backend Memory Control [25] - Verified capping memory usage through Write Buffer Manager [26] works in production. - New RocksDB version TBD, 5.18.3/6.2.2 has performance regression [27] compared to the currently used version 5.17.2. - FLIP of MemoryManager interface for reserving memory to be opened. - Unaligned Checkpoints [28] - Design under discussion. - FLIP document is under development and will be released shortly - Separate framework and user class loader in per-job mode [29] - Pull request is being reviewed. - Active Kubernetes Integration [30] - PoC completed. More details need to be discussed before updating the PRs. - FLIP-39 Flink ML pipeline and ML libs [31] - ML pipeline API PRs (FLINK-13339) have been opened and are being reviewed. - Algorithms are waiting for the new ML pipeline API to be merged. - Add vertex subtask log url on WebUI [32] - This makes it easier for users of the WebUI to access the logs of the TaskManager that executes a specific subtask. - A pull request is opened and currently being reviewed. As a reminder, the feature freeze is targeted to be at the end of November. This leaves us with approximately another 2 months of development time. We will send another announcement later in the release cycle to make the date of the feature freeze official. Best, Gary & Yu [1] https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E [2] https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E [3] https://issues.apache.org/jira/browse/FLINK-10725 [4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration [5] https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object [6] https://issues.apache.org/jira/browse/FLINK-14079 [7] https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL [8] https://docs.google.com/document/d/110TseRtTCphxETPY7uhiHpu-dph3NEesh3mYKtJ7QOY/edit?usp=sharing [9] https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support [10] https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design [11] https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL [12] https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module [13] https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog [14] https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Modular+Plugins [15] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-68-Extend-Core-Table-System-with-Modular-Plugins-td33372.html [16] https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors [17] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management [18] https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation [19] https://issues.apache.org/jira/browse/FLINK-10429 [20] https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface [21] https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E [22] https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink [23] https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table [24] https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend [25] https://issues.apache.org/jira/browse/FLINK-7289 [26] https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager [27] https://github.com/facebook/rocksdb/issues/5774 [28] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html [29] https://issues.apache.org/jira/browse/FLINK-13993 [30] https://issues.apache.org/jira/browse/FLINK-9953 [31] https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs [32] https://issues.apache.org/jira/browse/FLINK-13894 |
|
|
Thanks Yu and Gary for the detailed summary and update!
On Fri, Sep 27, 2019 at 6:54 AM Yu Li <[hidden email]> wrote: > Hi community, > > Since we are now more than one month into the Flink 1.10 release cycle, we > thought it would be adequate to give a progress update. Below we have > included a list of the ongoing efforts that we are aware of, together with > a brief summary of their state. As always, the list is not meant to be > exhaustive. If you are working on something that is not included here, feel > free to use this thread to share your progress. > > Note that because we are still relatively at the beginning of the release > cycle, most of the progress is limited to FLIPs that are accepted or being > voted on. > > - Improving Flink’s build system & CI > - Repository Split [1] > - Discussed on the ML but consensus to split the repository was not > reached. > - Reduce Build Time [2] > - Discussion is ongoing. Currently, using Azure Pipelines and > Gradle are being evaluated. > > - Support Java 11 [3] > - Implementation is in progress (18/21 subtasks resolved) > > - Table API improvements > - FLIP-54 Evolve ConfigOption and Configuration [4] > - Under discussion. > - FLIP-59 Enable Execution Configuration from Configuration Object [5] > - Under discussion. > - Full Data Type Support in Planner [6] > - Implementation in progress. > - FLIP-66 Support Time Attribute in SQL DDL [7] > - FLIP voting. > - FLIP-70 Support Computed Column [8] > - Under discussion. > - FLIP-63 Rework Table Partition Support [9] > - FLIP voting > - FLIP-51 Rework of Expression Design [10] > - FLIP accepted, implementation in progress. > - FLIP-55 Introduction of a TableAPI Java Expression DSL [11] > - Under discussion. > - FLIP-64 Support for Temporary Objects in Table Module [12] > - Under discussion. > - FLIP-65 New Type Inference for Table API UDFs > - Under discussion. > > - Hive compatibility completion (DDL/UDF) to support full Hive integration > - FLIP-57 Rework FunctionCatalog [13] > - FLIP voting > - FLIP-68 Extend Core Table System with Modular Plugins [14] > - FLIP voting was initiated [15] but temporarily withdrawn due to > lack of community bandwidth. > > - Finer grained resource management > - FLIP-49: Unified Memory Configuration for TaskExecutors [16] > - FLIP accepted. Implementation is in progress. > - FLIP-53: Fine Grained Operator Resource Management [17] > - FLIP accepted. Implementation details are under discussion. > - FLIP-56: Dynamic Slot Allocation [18] > - FLIP accepted. Implementation not started yet. > > - Finish scheduler re-architecture [19] > - Implementation is in progress. > > - FLIP-27: Refactor Source Interface [20] > - FLIP accepted. Implementation is in progress. > > - Executor/Client refactoring [21] > - Discussion already reached consensus > - FLIP is coming. A PoC implementation is also ready. > > - FLIP-36 Support Interactive Programming [22] > - Reviewing FLIP-67, which changes the intermediate result management > in runtime, which is what FLIP-36 will be built on top of. > > - FLIP-58: Flink Python User-Defined Stateless Function for Table [23] > - Implementation is in progress (3/15 subtask resolved). > - Python environment and dependency management under discussion > > - FLIP-50: Spill-able Heap Keyed State Backend [24] > - FLIP was accepted. Implementation is in progress. > > - RocksDB Backend Memory Control [25] > - Verified capping memory usage through Write Buffer Manager [26] works > in production. > - New RocksDB version TBD, 5.18.3/6.2.2 has performance regression [27] > compared to the currently used version 5.17.2. > - FLIP of MemoryManager interface for reserving memory to be opened. > > - Unaligned Checkpoints [28] > - Design under discussion. > - FLIP document is under development and will be released shortly > > - Separate framework and user class loader in per-job mode [29] > - Pull request is being reviewed. > > - Active Kubernetes Integration [30] > - PoC completed. More details need to be discussed before updating the > PRs. > > - FLIP-39 Flink ML pipeline and ML libs [31] > - ML pipeline API PRs (FLINK-13339) have been opened and are being > reviewed. > - Algorithms are waiting for the new ML pipeline API to be merged. > > - Add vertex subtask log url on WebUI [32] > - This makes it easier for users of the WebUI to access the logs of the > TaskManager that executes a specific subtask. > - A pull request is opened and currently being reviewed. > > As a reminder, the feature freeze is targeted to be at the end of November. > This leaves us with approximately another 2 months of development time. We > will send another announcement later in the release cycle to make the date > of the feature freeze official. > > Best, > Gary & Yu > > [1] > > https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E > [2] > > https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E > [3] https://issues.apache.org/jira/browse/FLINK-10725 > [4] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration > [5] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object > [6] https://issues.apache.org/jira/browse/FLINK-14079 > [7] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL > [8] > > https://docs.google.com/document/d/110TseRtTCphxETPY7uhiHpu-dph3NEesh3mYKtJ7QOY/edit?usp=sharing > [9] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > [10] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > [11] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL > [12] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module > [13] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog > [14] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Modular+Plugins > [15] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-68-Extend-Core-Table-System-with-Modular-Plugins-td33372.html > [16] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > [17] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > [18] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > [19] https://issues.apache.org/jira/browse/FLINK-10429 > [20] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > [21] > > https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E > [22] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > [23] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > [24] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend > [25] https://issues.apache.org/jira/browse/FLINK-7289 > [26] https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager > [27] https://github.com/facebook/rocksdb/issues/5774 > [28] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html > [29] https://issues.apache.org/jira/browse/FLINK-13993 > [30] https://issues.apache.org/jira/browse/FLINK-9953 > [31] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs > [32] https://issues.apache.org/jira/browse/FLINK-13894 > |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |