Re: Building with Hadoop 3

[vino yang]

cc @Chesnay Schepler <[hidden email]> to answer this question.

Foster, Craig <[hidden email]> 于2019年12月4日周三 上午1:22写道：

> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
> a document or JIRA that now exists which would point to what needs to be
> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
>
>
> Thanks!
>
> Craig
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>
>
>

[Chesnay Schepler-3]

There's no JIRA and no one actively working on it. I'm not aware of any
investigations on the matter; hence the first step would be to just try
it out.

A flink-shaded artifact isn't a hard requirement; Flink will work with
any 2.X hadoop distribution (provided that there aren't any dependency
clashes).

On 03/12/2019 18:22, Foster, Craig wrote:

>
> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA
> here from a year ago that no one is looking into Hadoop 3 support [1].
> Is there a document or JIRA that now exists which would point to what
> needs to be done to support Hadoop 3? Right now builds with Hadoop 3
> don’t work obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
> Thanks!
>
> Craig
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>

[Márton Balassi]

Wearing my Cloudera hat I can tell you that we have done this exercise for
our distros of the  3.0 and 3.1 Hadoop versions. We have not contributed
these back just yet, but we are open to do so. If the community is
interested we can contribute those changes back to flink-shaded and suggest
the necessay changes to flink too. The task was not overly complex, but it
certainly involved a bit of dependency hell. :-)

Right now we are focused on internal timelines, but we could invest into
contributing this back in the end of January timeframe if the community
deems this a worthwhile effort.

Best,
Marton

On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <[hidden email]> wrote:

> There's no JIRA and no one actively working on it. I'm not aware of any
> investigations on the matter; hence the first step would be to just try it
> out.
>
> A flink-shaded artifact isn't a hard requirement; Flink will work with any
> 2.X hadoop distribution (provided that there aren't any dependency clashes).
>
> On 03/12/2019 18:22, Foster, Craig wrote:
>
> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
> a document or JIRA that now exists which would point to what needs to be
> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
>
>
> Thanks!
>
> Craig
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>
>
>
>
>

[vino yang]

Hi Marton,

Thanks for your explanation. Personally, I look forward to your
contribution!

Best,
Vino

Márton Balassi <[hidden email]> 于2019年12月4日周三 下午5:15写道：

> Wearing my Cloudera hat I can tell you that we have done this exercise for
> our distros of the  3.0 and 3.1 Hadoop versions. We have not contributed
> these back just yet, but we are open to do so. If the community is
> interested we can contribute those changes back to flink-shaded and suggest
> the necessay changes to flink too. The task was not overly complex, but it
> certainly involved a bit of dependency hell. :-)
>
> Right now we are focused on internal timelines, but we could invest into
> contributing this back in the end of January timeframe if the community
> deems this a worthwhile effort.
>
> Best,
> Marton
>
> On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <[hidden email]>
> wrote:
>
>> There's no JIRA and no one actively working on it. I'm not aware of any
>> investigations on the matter; hence the first step would be to just try it
>> out.
>>
>> A flink-shaded artifact isn't a hard requirement; Flink will work with
>> any 2.X hadoop distribution (provided that there aren't any dependency
>> clashes).
>>
>> On 03/12/2019 18:22, Foster, Craig wrote:
>>
>> Hi:
>>
>> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
>> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
>> a document or JIRA that now exists which would point to what needs to be
>> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
>> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>>
>>
>>
>> Thanks!
>>
>> Craig
>>
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-11086
>>
>>
>>
>>
>>

[LINZ, Arnaud]

Hello,
Have you shared it somewhere on the web already?
Best,
Arnaud

De : vino yang <[hidden email]>
Envoyé : mercredi 4 décembre 2019 11:55
À : Márton Balassi <[hidden email]>
Cc : Chesnay Schepler <[hidden email]>; Foster, Craig <[hidden email]>; [hidden email]; [hidden email]
Objet : Re: Building with Hadoop 3

Hi Marton,

Thanks for your explanation. Personally, I look forward to your contribution!

Best,
Vino

Márton Balassi <[hidden email]<mailto:[hidden email]>> 于2019年12月4日周三 下午5:15写道：
Wearing my Cloudera hat I can tell you that we have done this exercise for our distros of the  3.0 and 3.1 Hadoop versions. We have not contributed these back just yet, but we are open to do so. If the community is interested we can contribute those changes back to flink-shaded and suggest the necessay changes to flink too. The task was not overly complex, but it certainly involved a bit of dependency hell. :-)

Right now we are focused on internal timelines, but we could invest into contributing this back in the end of January timeframe if the community deems this a worthwhile effort.

Best,
Marton

On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <[hidden email]<mailto:[hidden email]>> wrote:
There's no JIRA and no one actively working on it. I'm not aware of any investigations on the matter; hence the first step would be to just try it out.

A flink-shaded artifact isn't a hard requirement; Flink will work with any 2.X hadoop distribution (provided that there aren't any dependency clashes).

On 03/12/2019 18:22, Foster, Craig wrote:
Hi:
I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here from a year ago that no one is looking into Hadoop 3 support [1]. Is there a document or JIRA that now exists which would point to what needs to be done to support Hadoop 3? Right now builds with Hadoop 3 don’t work obviously because there’s no flink-shaded-hadoop-3 artifacts.

Thanks!
Craig

[1] https://issues.apache.org/jira/browse/FLINK-11086




________________________________

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[Stephan Ewen]

Have you tried to just export Hadoop 3's classpath to `HADOOP_CLASSPATH`
and see if that works out of the box?

If the main use case is HDFS access, then there is a fair chance it might
just work, because Flink uses only a small subset of the Hadoop FS API
which is stable between 2.x and 3.x, as far as I tried it out a while back.

On Tue, Mar 3, 2020 at 6:03 PM LINZ, Arnaud <[hidden email]>
wrote:

> Hello,
>
> Have you shared it somewhere on the web already?
>
> Best,
>
> Arnaud
>
>
>
> *De :* vino yang <[hidden email]>
> *Envoyé :* mercredi 4 décembre 2019 11:55
> *À :* Márton Balassi <[hidden email]>
> *Cc :* Chesnay Schepler <[hidden email]>; Foster, Craig <
> [hidden email]>; [hidden email]; [hidden email]
> *Objet :* Re: Building with Hadoop 3
>
>
>
> Hi Marton,
>
>
>
> Thanks for your explanation. Personally, I look forward to your
> contribution!
>
>
>
> Best,
>
> Vino
>
>
>
> Márton Balassi <[hidden email]> 于2019年12月4日周三 下午5:15写道：
>
> Wearing my Cloudera hat I can tell you that we have done this exercise for
> our distros of the  3.0 and 3.1 Hadoop versions. We have not contributed
> these back just yet, but we are open to do so. If the community is
> interested we can contribute those changes back to flink-shaded and suggest
> the necessay changes to flink too. The task was not overly complex, but it
> certainly involved a bit of dependency hell. :-)
>
>
>
> Right now we are focused on internal timelines, but we could invest into
> contributing this back in the end of January timeframe if the community
> deems this a worthwhile effort.
>
>
>
> Best,
>
> Marton
>
>
>
> On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <[hidden email]>
> wrote:
>
> There's no JIRA and no one actively working on it. I'm not aware of any
> investigations on the matter; hence the first step would be to just try it
> out.
>
>
>
> A flink-shaded artifact isn't a hard requirement; Flink will work with any
> 2.X hadoop distribution (provided that there aren't any dependency clashes).
>
>
>
> On 03/12/2019 18:22, Foster, Craig wrote:
>
> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
> a document or JIRA that now exists which would point to what needs to be
> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
>
>
> Thanks!
>
> Craig
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>
>
>
>
>
>
> ------------------------------
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>