Dripping the Flink-on-Tez code for Flink 1.0

[Stephan Ewen]

Hi all!

Currently, Flink has a module to run batch program code on Tez rather than
Flink's own distributed execution engine.

I would suggest that we drop this code for the next release (1.0) as part
of a code consolidation:

  - There seems little in both the Flink and the Tez community to use and
expand this functionality.

  - The original motivation (better exploit resource elasticity in YARN)
will no longer be valid in the near future. I am re-working the YARN
integration currently to make it more elastic and make it possible to run
Flink on Mesos.

  - The Flink-on-Tez code is rather POC status. Large scale testing it and
making adding all missing features will take more effort than making
Flink's own YARN integration resource elastic.


Please let me know what you think!
Especially @Kostas, since you wrote the initial POC, I'd be interested in
your opinion.

Greetings,
Stephan

[Kostas Tzoumas-2]

+1 from my side

Flink on Tez never got a lot of user traction. It served well as a
prototype of "this is possible", but since the core functionality will be
subsumed by making Flink on YARN resource elastic, I don't see any reason
we should have it as part of the Flink codebase.

Best,
Kostas

On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:

> Hi all!
>
> Currently, Flink has a module to run batch program code on Tez rather than
> Flink's own distributed execution engine.
>
> I would suggest that we drop this code for the next release (1.0) as part
> of a code consolidation:
>
>   - There seems little in both the Flink and the Tez community to use and
> expand this functionality.
>
>   - The original motivation (better exploit resource elasticity in YARN)
> will no longer be valid in the near future. I am re-working the YARN
> integration currently to make it more elastic and make it possible to run
> Flink on Mesos.
>
>   - The Flink-on-Tez code is rather POC status. Large scale testing it and
> making adding all missing features will take more effort than making
> Flink's own YARN integration resource elastic.
>
>
> Please let me know what you think!
> Especially @Kostas, since you wrote the initial POC, I'd be interested in
> your opinion.
>
> Greetings,
> Stephan
>

[Kostas Tzoumas-2]

for clarification, I was talking about dropping the code, I am unsure about
the consequences of dripping code :-)

On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <[hidden email]> wrote:

> +1 from my side
>
> Flink on Tez never got a lot of user traction. It served well as a
> prototype of "this is possible", but since the core functionality will be
> subsumed by making Flink on YARN resource elastic, I don't see any reason
> we should have it as part of the Flink codebase.
>
> Best,
> Kostas
>
> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:
>
>> Hi all!
>>
>> Currently, Flink has a module to run batch program code on Tez rather than
>> Flink's own distributed execution engine.
>>
>> I would suggest that we drop this code for the next release (1.0) as part
>> of a code consolidation:
>>
>>   - There seems little in both the Flink and the Tez community to use and
>> expand this functionality.
>>
>>   - The original motivation (better exploit resource elasticity in YARN)
>> will no longer be valid in the near future. I am re-working the YARN
>> integration currently to make it more elastic and make it possible to run
>> Flink on Mesos.
>>
>>   - The Flink-on-Tez code is rather POC status. Large scale testing it and
>> making adding all missing features will take more effort than making
>> Flink's own YARN integration resource elastic.
>>
>>
>> Please let me know what you think!
>> Especially @Kostas, since you wrote the initial POC, I'd be interested in
>> your opinion.
>>
>> Greetings,
>> Stephan
>>
>
>

[Ufuk Celebi-2]

+1

I wanted to make a similar proposal.

– Ufuk

> On 08 Jan 2016, at 17:03, Kostas Tzoumas <[hidden email]> wrote:
>
> for clarification, I was talking about dropping the code, I am unsure about
> the consequences of dripping code :-)
>
> On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <[hidden email]> wrote:
>
>> +1 from my side
>>
>> Flink on Tez never got a lot of user traction. It served well as a
>> prototype of "this is possible", but since the core functionality will be
>> subsumed by making Flink on YARN resource elastic, I don't see any reason
>> we should have it as part of the Flink codebase.
>>
>> Best,
>> Kostas
>>
>> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:
>>
>>> Hi all!
>>>
>>> Currently, Flink has a module to run batch program code on Tez rather than
>>> Flink's own distributed execution engine.
>>>
>>> I would suggest that we drop this code for the next release (1.0) as part
>>> of a code consolidation:
>>>
>>>  - There seems little in both the Flink and the Tez community to use and
>>> expand this functionality.
>>>
>>>  - The original motivation (better exploit resource elasticity in YARN)
>>> will no longer be valid in the near future. I am re-working the YARN
>>> integration currently to make it more elastic and make it possible to run
>>> Flink on Mesos.
>>>
>>>  - The Flink-on-Tez code is rather POC status. Large scale testing it and
>>> making adding all missing features will take more effort than making
>>> Flink's own YARN integration resource elastic.
>>>
>>>
>>> Please let me know what you think!
>>> Especially @Kostas, since you wrote the initial POC, I'd be interested in
>>> your opinion.
>>>
>>> Greetings,
>>> Stephan
>>>
>>
>>

[Till Rohrmann]

+1 since it increase maintainability of the code base if it is not really
used and thus removed.

On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <[hidden email]> wrote:

> +1
>
> I wanted to make a similar proposal.
>
> – Ufuk
>
> > On 08 Jan 2016, at 17:03, Kostas Tzoumas <[hidden email]> wrote:
> >
> > for clarification, I was talking about dropping the code, I am unsure
> about
> > the consequences of dripping code :-)
> >
> > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <[hidden email]>
> wrote:
> >
> >> +1 from my side
> >>
> >> Flink on Tez never got a lot of user traction. It served well as a
> >> prototype of "this is possible", but since the core functionality will
> be
> >> subsumed by making Flink on YARN resource elastic, I don't see any
> reason
> >> we should have it as part of the Flink codebase.
> >>
> >> Best,
> >> Kostas
> >>
> >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]> wrote:
> >>
> >>> Hi all!
> >>>
> >>> Currently, Flink has a module to run batch program code on Tez rather
> than
> >>> Flink's own distributed execution engine.
> >>>
> >>> I would suggest that we drop this code for the next release (1.0) as
> part
> >>> of a code consolidation:
> >>>
> >>>  - There seems little in both the Flink and the Tez community to use
> and
> >>> expand this functionality.
> >>>
> >>>  - The original motivation (better exploit resource elasticity in YARN)
> >>> will no longer be valid in the near future. I am re-working the YARN
> >>> integration currently to make it more elastic and make it possible to
> run
> >>> Flink on Mesos.
> >>>
> >>>  - The Flink-on-Tez code is rather POC status. Large scale testing it
> and
> >>> making adding all missing features will take more effort than making
> >>> Flink's own YARN integration resource elastic.
> >>>
> >>>
> >>> Please let me know what you think!
> >>> Especially @Kostas, since you wrote the initial POC, I'd be interested
> in
> >>> your opinion.
> >>>
> >>> Greetings,
> >>> Stephan
> >>>
> >>
> >>
>
>

[Fabian Hueske-2]

+1

2016-01-08 17:35 GMT+01:00 Till Rohrmann <[hidden email]>:

> +1 since it increase maintainability of the code base if it is not really
> used and thus removed.
>
> On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <[hidden email]> wrote:
>
> > +1
> >
> > I wanted to make a similar proposal.
> >
> > – Ufuk
> >
> > > On 08 Jan 2016, at 17:03, Kostas Tzoumas <[hidden email]> wrote:
> > >
> > > for clarification, I was talking about dropping the code, I am unsure
> > about
> > > the consequences of dripping code :-)
> > >
> > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <[hidden email]>
> > wrote:
> > >
> > >> +1 from my side
> > >>
> > >> Flink on Tez never got a lot of user traction. It served well as a
> > >> prototype of "this is possible", but since the core functionality will
> > be
> > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > reason
> > >> we should have it as part of the Flink codebase.
> > >>
> > >> Best,
> > >> Kostas
> > >>
> > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]>
> wrote:
> > >>
> > >>> Hi all!
> > >>>
> > >>> Currently, Flink has a module to run batch program code on Tez rather
> > than
> > >>> Flink's own distributed execution engine.
> > >>>
> > >>> I would suggest that we drop this code for the next release (1.0) as
> > part
> > >>> of a code consolidation:
> > >>>
> > >>>  - There seems little in both the Flink and the Tez community to use
> > and
> > >>> expand this functionality.
> > >>>
> > >>>  - The original motivation (better exploit resource elasticity in
> YARN)
> > >>> will no longer be valid in the near future. I am re-working the YARN
> > >>> integration currently to make it more elastic and make it possible to
> > run
> > >>> Flink on Mesos.
> > >>>
> > >>>  - The Flink-on-Tez code is rather POC status. Large scale testing it
> > and
> > >>> making adding all missing features will take more effort than making
> > >>> Flink's own YARN integration resource elastic.
> > >>>
> > >>>
> > >>> Please let me know what you think!
> > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> interested
> > in
> > >>> your opinion.
> > >>>
> > >>> Greetings,
> > >>> Stephan
> > >>>
> > >>
> > >>
> >
> >
>

[Stephan Ewen]

I was typing with fat fingers again. Meant "dropping code" of course, not
"dripping" ;-)

On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske <[hidden email]> wrote:

> +1
>
> 2016-01-08 17:35 GMT+01:00 Till Rohrmann <[hidden email]>:
>
> > +1 since it increase maintainability of the code base if it is not really
> > used and thus removed.
> >
> > On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <[hidden email]> wrote:
> >
> > > +1
> > >
> > > I wanted to make a similar proposal.
> > >
> > > – Ufuk
> > >
> > > > On 08 Jan 2016, at 17:03, Kostas Tzoumas <[hidden email]>
> wrote:
> > > >
> > > > for clarification, I was talking about dropping the code, I am unsure
> > > about
> > > > the consequences of dripping code :-)
> > > >
> > > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <[hidden email]>
> > > wrote:
> > > >
> > > >> +1 from my side
> > > >>
> > > >> Flink on Tez never got a lot of user traction. It served well as a
> > > >> prototype of "this is possible", but since the core functionality
> will
> > > be
> > > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > > reason
> > > >> we should have it as part of the Flink codebase.
> > > >>
> > > >> Best,
> > > >> Kostas
> > > >>
> > > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]>
> > wrote:
> > > >>
> > > >>> Hi all!
> > > >>>
> > > >>> Currently, Flink has a module to run batch program code on Tez
> rather
> > > than
> > > >>> Flink's own distributed execution engine.
> > > >>>
> > > >>> I would suggest that we drop this code for the next release (1.0)
> as
> > > part
> > > >>> of a code consolidation:
> > > >>>
> > > >>>  - There seems little in both the Flink and the Tez community to
> use
> > > and
> > > >>> expand this functionality.
> > > >>>
> > > >>>  - The original motivation (better exploit resource elasticity in
> > YARN)
> > > >>> will no longer be valid in the near future. I am re-working the
> YARN
> > > >>> integration currently to make it more elastic and make it possible
> to
> > > run
> > > >>> Flink on Mesos.
> > > >>>
> > > >>>  - The Flink-on-Tez code is rather POC status. Large scale testing
> it
> > > and
> > > >>> making adding all missing features will take more effort than
> making
> > > >>> Flink's own YARN integration resource elastic.
> > > >>>
> > > >>>
> > > >>> Please let me know what you think!
> > > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> > interested
> > > in
> > > >>> your opinion.
> > > >>>
> > > >>> Greetings,
> > > >>> Stephan
> > > >>>
> > > >>
> > > >>
> > >
> > >
> >
>

[Henry Saputra]

+1

I am always for simplifying our code base when possible.



On Sunday, January 10, 2016, Stephan Ewen <[hidden email]> wrote:

> I was typing with fat fingers again. Meant "dropping code" of course, not
> "dripping" ;-)
>
> On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske <[hidden email]
> <javascript:;>> wrote:
>
> > +1
> >
> > 2016-01-08 17:35 GMT+01:00 Till Rohrmann <[hidden email]
> <javascript:;>>:
> >
> > > +1 since it increase maintainability of the code base if it is not
> really
> > > used and thus removed.
> > >
> > > On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <[hidden email]
> <javascript:;>> wrote:
> > >
> > > > +1
> > > >
> > > > I wanted to make a similar proposal.
> > > >
> > > > – Ufuk
> > > >
> > > > > On 08 Jan 2016, at 17:03, Kostas Tzoumas <[hidden email]
> <javascript:;>>
> > wrote:
> > > > >
> > > > > for clarification, I was talking about dropping the code, I am
> unsure
> > > > about
> > > > > the consequences of dripping code :-)
> > > > >
> > > > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <
> [hidden email] <javascript:;>>
> > > > wrote:
> > > > >
> > > > >> +1 from my side
> > > > >>
> > > > >> Flink on Tez never got a lot of user traction. It served well as a
> > > > >> prototype of "this is possible", but since the core functionality
> > will
> > > > be
> > > > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > > > reason
> > > > >> we should have it as part of the Flink codebase.
> > > > >>
> > > > >> Best,
> > > > >> Kostas
> > > > >>
> > > > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]
> <javascript:;>>
> > > wrote:
> > > > >>
> > > > >>> Hi all!
> > > > >>>
> > > > >>> Currently, Flink has a module to run batch program code on Tez
> > rather
> > > > than
> > > > >>> Flink's own distributed execution engine.
> > > > >>>
> > > > >>> I would suggest that we drop this code for the next release (1.0)
> > as
> > > > part
> > > > >>> of a code consolidation:
> > > > >>>
> > > > >>>  - There seems little in both the Flink and the Tez community to
> > use
> > > > and
> > > > >>> expand this functionality.
> > > > >>>
> > > > >>>  - The original motivation (better exploit resource elasticity in
> > > YARN)
> > > > >>> will no longer be valid in the near future. I am re-working the
> > YARN
> > > > >>> integration currently to make it more elastic and make it
> possible
> > to
> > > > run
> > > > >>> Flink on Mesos.
> > > > >>>
> > > > >>>  - The Flink-on-Tez code is rather POC status. Large scale
> testing
> > it
> > > > and
> > > > >>> making adding all missing features will take more effort than
> > making
> > > > >>> Flink's own YARN integration resource elastic.
> > > > >>>
> > > > >>>
> > > > >>> Please let me know what you think!
> > > > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> > > interested
> > > > in
> > > > >>> your opinion.
> > > > >>>
> > > > >>> Greetings,
> > > > >>> Stephan
> > > > >>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

[Ufuk Celebi-2]

I’ve created an issue [1] and closed all other Tez issues.

[1] https://issues.apache.org/jira/browse/FLINK-3235

> On 11 Jan 2016, at 02:52, Henry Saputra <[hidden email]> wrote:
>
> +1
>
> I am always for simplifying our code base when possible.
>
>
>
> On Sunday, January 10, 2016, Stephan Ewen <[hidden email]> wrote:
>
>> I was typing with fat fingers again. Meant "dropping code" of course, not
>> "dripping" ;-)
>>
>> On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske <[hidden email]
>> <javascript:;>> wrote:
>>
>>> +1
>>>
>>> 2016-01-08 17:35 GMT+01:00 Till Rohrmann <[hidden email]
>> <javascript:;>>:
>>>
>>>> +1 since it increase maintainability of the code base if it is not
>> really
>>>> used and thus removed.
>>>>
>>>> On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <[hidden email]
>> <javascript:;>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> I wanted to make a similar proposal.
>>>>>
>>>>> – Ufuk
>>>>>
>>>>>> On 08 Jan 2016, at 17:03, Kostas Tzoumas <[hidden email]
>> <javascript:;>>
>>> wrote:
>>>>>>
>>>>>> for clarification, I was talking about dropping the code, I am
>> unsure
>>>>> about
>>>>>> the consequences of dripping code :-)
>>>>>>
>>>>>> On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <
>> [hidden email] <javascript:;>>
>>>>> wrote:
>>>>>>
>>>>>>> +1 from my side
>>>>>>>
>>>>>>> Flink on Tez never got a lot of user traction. It served well as a
>>>>>>> prototype of "this is possible", but since the core functionality
>>> will
>>>>> be
>>>>>>> subsumed by making Flink on YARN resource elastic, I don't see any
>>>>> reason
>>>>>>> we should have it as part of the Flink codebase.
>>>>>>>
>>>>>>> Best,
>>>>>>> Kostas
>>>>>>>
>>>>>>> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <[hidden email]
>> <javascript:;>>
>>>> wrote:
>>>>>>>
>>>>>>>> Hi all!
>>>>>>>>
>>>>>>>> Currently, Flink has a module to run batch program code on Tez
>>> rather
>>>>> than
>>>>>>>> Flink's own distributed execution engine.
>>>>>>>>
>>>>>>>> I would suggest that we drop this code for the next release (1.0)
>>> as
>>>>> part
>>>>>>>> of a code consolidation:
>>>>>>>>
>>>>>>>> - There seems little in both the Flink and the Tez community to
>>> use
>>>>> and
>>>>>>>> expand this functionality.
>>>>>>>>
>>>>>>>> - The original motivation (better exploit resource elasticity in
>>>> YARN)
>>>>>>>> will no longer be valid in the near future. I am re-working the
>>> YARN
>>>>>>>> integration currently to make it more elastic and make it
>> possible
>>> to
>>>>> run
>>>>>>>> Flink on Mesos.
>>>>>>>>
>>>>>>>> - The Flink-on-Tez code is rather POC status. Large scale
>> testing
>>> it
>>>>> and
>>>>>>>> making adding all missing features will take more effort than
>>> making
>>>>>>>> Flink's own YARN integration resource elastic.
>>>>>>>>
>>>>>>>>
>>>>>>>> Please let me know what you think!
>>>>>>>> Especially @Kostas, since you wrote the initial POC, I'd be
>>>> interested
>>>>> in
>>>>>>>> your opinion.
>>>>>>>>
>>>>>>>> Greetings,
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>