Drop support for CDH4 / Hadoop 2.0.0-alpha

[Robert Metzger]

Hi,

I'm currently working on https://issues.apache.org/jira/browse/FLINK-1605
and its a hell of a mess.

I got almost everything working, except for the hadoop 2.0.0-alpha profile.
The profile exists because google protobuf has a different version in that
Hadoop release.
Since maven is setting the version of protobuf for the entire project to
the older version, we have to use an older akka version which is causing
issues.

The logical conclusion from that would be shading Hadoop's protobuf version
into the Hadoop jars. That by itself is working, however its not working
for the "flink-yarn-tests".

I think I can also solve the issue with the flink-yarn-tests, but it would
be a very dirty hack (either injecting shaded code into the failsafe
tests-classpath or putting test code into src/main).

But the general question remains: Are we willing to continue spending a lot
of time on maintaining the profile?
Till has spend a lot of time recently to fix failing testcases for that old
akka version, I spend almost two days now on getting the
shading/dependencies right, and I'm sure we'll keep having troubles with
the profile.


Therefore, I was wondering if this is the right time to drop support for
CDH4 / Hadoop 2.0.0-alpha.


Best,
Robert

[Stephan Ewen]

I have no very strong love for the Hadoop 2.0.0-alpha version, and it seems
that most users go through YARN anyways.

Just to understand: The solution would be to not share protobuf in the fat
Hadoop jar at all? Is that not a problem for other situations, like users
with an earlier protobuf version?

On Thu, Feb 26, 2015 at 5:57 PM, Robert Metzger <[hidden email]> wrote:

> Hi,
>
> I'm currently working on https://issues.apache.org/jira/browse/FLINK-1605
> and its a hell of a mess.
>
> I got almost everything working, except for the hadoop 2.0.0-alpha profile.
> The profile exists because google protobuf has a different version in that
> Hadoop release.
> Since maven is setting the version of protobuf for the entire project to
> the older version, we have to use an older akka version which is causing
> issues.
>
> The logical conclusion from that would be shading Hadoop's protobuf version
> into the Hadoop jars. That by itself is working, however its not working
> for the "flink-yarn-tests".
>
> I think I can also solve the issue with the flink-yarn-tests, but it would
> be a very dirty hack (either injecting shaded code into the failsafe
> tests-classpath or putting test code into src/main).
>
> But the general question remains: Are we willing to continue spending a lot
> of time on maintaining the profile?
> Till has spend a lot of time recently to fix failing testcases for that old
> akka version, I spend almost two days now on getting the
> shading/dependencies right, and I'm sure we'll keep having troubles with
> the profile.
>
>
> Therefore, I was wondering if this is the right time to drop support for
> CDH4 / Hadoop 2.0.0-alpha.
>
>
> Best,
> Robert
>

[Henry Saputra]

If we were to drop CDH4 / Hadoop 2.0.0-alpha, would this mean we do
not even to shade the hadoop fat jars, or we do still needed to
support 1.x ?

- Henry

On Thu, Feb 26, 2015 at 8:57 AM, Robert Metzger <[hidden email]> wrote:

> Hi,
>
> I'm currently working on https://issues.apache.org/jira/browse/FLINK-1605
> and its a hell of a mess.
>
> I got almost everything working, except for the hadoop 2.0.0-alpha profile.
> The profile exists because google protobuf has a different version in that
> Hadoop release.
> Since maven is setting the version of protobuf for the entire project to
> the older version, we have to use an older akka version which is causing
> issues.
>
> The logical conclusion from that would be shading Hadoop's protobuf version
> into the Hadoop jars. That by itself is working, however its not working
> for the "flink-yarn-tests".
>
> I think I can also solve the issue with the flink-yarn-tests, but it would
> be a very dirty hack (either injecting shaded code into the failsafe
> tests-classpath or putting test code into src/main).
>
> But the general question remains: Are we willing to continue spending a lot
> of time on maintaining the profile?
> Till has spend a lot of time recently to fix failing testcases for that old
> akka version, I spend almost two days now on getting the
> shading/dependencies right, and I'm sure we'll keep having troubles with
> the profile.
>
>
> Therefore, I was wondering if this is the right time to drop support for
> CDH4 / Hadoop 2.0.0-alpha.
>
>
> Best,
> Robert

[Robert Metzger]

@Henry: We would still shade Hadoop because of its Guava / ASM dependencies
which interfere with our dependencies.
The nice thing of my change is that all the other flink modules don't have
to care about the details of our Hadoop dependencie. Its basically an
abstract hadoop dependency, without guava and asm ;)

@Stephan: The actual issue is not CDH4 itself but shading protobuf into
hadoop. If you think we should shade protobuf out of Hadoop, then we can
keep the CDH4 support and I have to figure out a way to get the
"flink-yarn-tests" running.
I'm not aware of a user who complained about our protobuf dependency.



On Fri, Feb 27, 2015 at 1:23 AM, Henry Saputra <[hidden email]>
wrote:

> If we were to drop CDH4 / Hadoop 2.0.0-alpha, would this mean we do
> not even to shade the hadoop fat jars, or we do still needed to
> support 1.x ?
>
> - Henry
>
> On Thu, Feb 26, 2015 at 8:57 AM, Robert Metzger <[hidden email]>
> wrote:
> > Hi,
> >
> > I'm currently working on
> https://issues.apache.org/jira/browse/FLINK-1605
> > and its a hell of a mess.
> >
> > I got almost everything working, except for the hadoop 2.0.0-alpha
> profile.
> > The profile exists because google protobuf has a different version in
> that
> > Hadoop release.
> > Since maven is setting the version of protobuf for the entire project to
> > the older version, we have to use an older akka version which is causing
> > issues.
> >
> > The logical conclusion from that would be shading Hadoop's protobuf
> version
> > into the Hadoop jars. That by itself is working, however its not working
> > for the "flink-yarn-tests".
> >
> > I think I can also solve the issue with the flink-yarn-tests, but it
> would
> > be a very dirty hack (either injecting shaded code into the failsafe
> > tests-classpath or putting test code into src/main).
> >
> > But the general question remains: Are we willing to continue spending a
> lot
> > of time on maintaining the profile?
> > Till has spend a lot of time recently to fix failing testcases for that
> old
> > akka version, I spend almost two days now on getting the
> > shading/dependencies right, and I'm sure we'll keep having troubles with
> > the profile.
> >
> >
> > Therefore, I was wondering if this is the right time to drop support for
> > CDH4 / Hadoop 2.0.0-alpha.
> >
> >
> > Best,
> > Robert
>