Hi Flink community,
I serve as chair on the Kubernetes Big Data SIG, and we've had some interest from the SIG in learning more about what the Flink community is working on in the kubernetes space. I'm putting out feelers to see if anybody from the community would be interested in giving a brief talk and/or demo of Flink integrations with k8s at one of our regular meetings. The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next meeting is Oct 31). Feel free to attend our meetings, or visit our slack channel: https://kubernetes.slack.com/messages/C0ELB338T/ More SIG info is available on our regular meeting minutes doc: https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/ Thanks! Erik Erlandson |
Hi Erik,
thanks for reaching out to the Flink community. The Flink community is actually actively working on improving our K8s support because we see a lot of adoption and our users would like to run Flink on K8s. Apart from the existing Flink docker images there are currently two development threads: 1) Active K8s integration [1] The idea of the active K8s integration is to enable Flink to talk to the K8s master in order to actively start or terminate new Pods which run Flink's TaskManager. That way, Flink would be able to adjust the set of available resources dynamically based on the actual needs (coming from a Flink job). I know that the community is actively working on drafting a design document for this mode which will be published very soon. Moreover, I have a dev branch where I prototyped the K8s integration [2]. 2) Reactive container mode [3] The reactive container mode works actually independent of the underlying container framework but K8s will be the natural deployment target for this. The idea is that not Flink but an external system/user is responsible for allocating/releasing resources. E.g. some auto-scaling groups could decide that Flink is currently not keeping up wrt some metric and start some new Pods (running Flink's TaskManager). If this happens, Flink should react to such an event and use the newly available resources. Usually this means that Flink scales up the parallelism of its job to deploy new tasks to the newly started TaskManager. There is actually a design document linked under the respective JIRA issue for more details. I would be happy to join your meeting to give an overview what the Flink community is currently developing, answering questions and gathering feedback for what we intend to do. How would I join this meeting? [1] https://issues.apache.org/jira/browse/FLINK-9953 [2] https://github.com/tillrohrmann/flink/tree/nativeKubernetes [3] https://issues.apache.org/jira/browse/FLINK-10407 Cheers, Till On Mon, Oct 22, 2018 at 6:55 PM Erik Erlandson <[hidden email]> wrote: > Hi Flink community, > > I serve as chair on the Kubernetes Big Data SIG, and we've had some > interest from the SIG in learning more about what the Flink community is > working on in the kubernetes space. > > I'm putting out feelers to see if anybody from the community would be > interested in giving a brief talk and/or demo of Flink integrations with > k8s at one of our regular meetings. > > The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next > meeting is Oct 31). Feel free to attend our meetings, or visit our slack > channel: > https://kubernetes.slack.com/messages/C0ELB338T/ > > More SIG info is available on our regular meeting minutes doc: > > https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/ > > Thanks! > Erik Erlandson > |
In reply to this post by Erik Erlandson
Hi Erik,
We @Flyte team within Lyft have been working on SparkK8Operator for quite some time, and now we have built a FlinkK8Operator to run Flink applications on Kubernetes. We would be happy to demo the Flink Operator during the meeting. Also I am in the process of fleshing out all the details about the Operator in a doc, that I would share with the Flink community in a week or so. Thanks, Anand On Mon, Oct 22, 2018 at 9:55 AM Erik Erlandson <[hidden email]> wrote: > Hi Flink community, > > I serve as chair on the Kubernetes Big Data SIG, and we've had some > interest from the SIG in learning more about what the Flink community is > working on in the kubernetes space. > > I'm putting out feelers to see if anybody from the community would be > interested in giving a brief talk and/or demo of Flink integrations with > k8s at one of our regular meetings. > > The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next > meeting is Oct 31). Feel free to attend our meetings, or visit our slack > channel: > https://kubernetes.slack.com/messages/C0ELB338T/ > > More SIG info is available on our regular meeting minutes doc: > > https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/ > > Thanks! > Erik Erlandson > |
@Erik, I’m working with Till regarding together for the Flink K8s native integration, we have a draft of design document, and plan to post the document next week and collect feedbacks.
@Anand, operator is a interesting topic, look forward to see your design document. I will keep you updated. > On Oct 23, 2018, at 3:32 PM, Anand Swaminathan <[hidden email]> wrote: > > Hi Erik, > > We @Flyte team within Lyft have been working on SparkK8Operator for quite > some time, and now we have built a FlinkK8Operator to run Flink > applications on Kubernetes. > We would be happy to demo the Flink Operator during the meeting. > > Also I am in the process of fleshing out all the details about the Operator > in a doc, that I would share with the Flink community in a week or so. > > Thanks, > Anand > > On Mon, Oct 22, 2018 at 9:55 AM Erik Erlandson <[hidden email]> wrote: > >> Hi Flink community, >> >> I serve as chair on the Kubernetes Big Data SIG, and we've had some >> interest from the SIG in learning more about what the Flink community is >> working on in the kubernetes space. >> >> I'm putting out feelers to see if anybody from the community would be >> interested in giving a brief talk and/or demo of Flink integrations with >> k8s at one of our regular meetings. >> >> The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next >> meeting is Oct 31). Feel free to attend our meetings, or visit our slack >> channel: >> https://kubernetes.slack.com/messages/C0ELB338T/ >> >> More SIG info is available on our regular meeting minutes doc: >> >> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/ >> >> Thanks! >> Erik Erlandson >> |
Free forum by Nabble | Edit this page |