Source of Cloudera 3 Hadoop Flink shaded uber jars?

[Adam Roberts]

Hey everyone, I realise this could be a long shot (but perhaps not given
the list of committers and perhaps some are from Cloudera!) - I'm wondering
if anyone knows the source location and build instructions for the jars at
https://repository.cloudera.com/artifactory/libs-release-local/org/apache/flink/flink-shaded-hadoop-3-uber/
.

I ask because, AIAU, Hadoop 3 isn't really included/properly supported in
Flink, but we have users who are looking to use this exact library, and
obviously we want to make sure it's security issue free, and easy for us to
maintain and has an awesome license.

My biggest hunch is that it's from this repository
https://github.com/apache/hadoop/tree/rel/release-3.1.1/hadoop-client-modules
but... that tag is over three years old, and the Cloudera builds linked
above date from 2020. Again - apologies if it is such a long shot, perhaps
I've just missed something obvious.


Thanks!

[Chesnay Schepler-3]

I can't tell you where it comes from, just that the Flink project had
nothing to do with it.

FYI; Hadoop 3 has been supported by Flink since 1.11 . We do no longer
publish shaded-hadoop jars though because it is a maintenance nightmare.

On 2/10/2021 6:54 PM, Adam Roberts wrote:

> Hey everyone, I realise this could be a long shot (but perhaps not given
> the list of committers and perhaps some are from Cloudera!) - I'm wondering
> if anyone knows the source location and build instructions for the jars at
> https://repository.cloudera.com/artifactory/libs-release-local/org/apache/flink/flink-shaded-hadoop-3-uber/
> .
>
> I ask because, AIAU, Hadoop 3 isn't really included/properly supported in
> Flink, but we have users who are looking to use this exact library, and
> obviously we want to make sure it's security issue free, and easy for us to
> maintain and has an awesome license.
>
> My biggest hunch is that it's from this repository
> https://github.com/apache/hadoop/tree/rel/release-3.1.1/hadoop-client-modules
> but... that tag is over three years old, and the Cloudera builds linked
> above date from 2020. Again - apologies if it is such a long shot, perhaps
> I've just missed something obvious.
>
>
> Thanks!
>

[Gyula Fóra]

Hi Adam!

Cloudera has published these artifacts before but they are no longer used
similarly to the Flink project.

You should set the HADOOP_CLASSPATH to the hadoop libs instead of using the
shaded jars.

Cheers
Gyula

On Wed, Feb 10, 2021 at 7:27 PM Chesnay Schepler <[hidden email]> wrote:

> I can't tell you where it comes from, just that the Flink project had
> nothing to do with it.
>
> FYI; Hadoop 3 has been supported by Flink since 1.11 . We do no longer
> publish shaded-hadoop jars though because it is a maintenance nightmare.
>
> On 2/10/2021 6:54 PM, Adam Roberts wrote:
> > Hey everyone, I realise this could be a long shot (but perhaps not given
> > the list of committers and perhaps some are from Cloudera!) - I'm
> wondering
> > if anyone knows the source location and build instructions for the jars
> at
> >
> https://repository.cloudera.com/artifactory/libs-release-local/org/apache/flink/flink-shaded-hadoop-3-uber/
> > .
> >
> > I ask because, AIAU, Hadoop 3 isn't really included/properly supported in
> > Flink, but we have users who are looking to use this exact library, and
> > obviously we want to make sure it's security issue free, and easy for us
> to
> > maintain and has an awesome license.
> >
> > My biggest hunch is that it's from this repository
> >
> https://github.com/apache/hadoop/tree/rel/release-3.1.1/hadoop-client-modules
> > but... that tag is over three years old, and the Cloudera builds linked
> > above date from 2020. Again - apologies if it is such a long shot,
> perhaps
> > I've just missed something obvious.
> >
> >
> > Thanks!
> >
>
>

[Adam Roberts]

Awesome, thank you everyone for the prompt replies! That does clear things
up for me. I'll go back to those asking for it to confirm why the
particular library is to be used instead (and if there is a good reason I
shall continue my search).

Cheers!

On Wed, 10 Feb 2021, 19:25 Gyula Fóra, <[hidden email]> wrote:

> Hi Adam!
>
> Cloudera has published these artifacts before but they are no longer used
> similarly to the Flink project.
>
> You should set the HADOOP_CLASSPATH to the hadoop libs instead of using
> the shaded jars.
>
> Cheers
> Gyula
>
> On Wed, Feb 10, 2021 at 7:27 PM Chesnay Schepler <[hidden email]>
> wrote:
>
>> I can't tell you where it comes from, just that the Flink project had
>> nothing to do with it.
>>
>> FYI; Hadoop 3 has been supported by Flink since 1.11 . We do no longer
>> publish shaded-hadoop jars though because it is a maintenance nightmare.
>>
>> On 2/10/2021 6:54 PM, Adam Roberts wrote:
>> > Hey everyone, I realise this could be a long shot (but perhaps not given
>> > the list of committers and perhaps some are from Cloudera!) - I'm
>> wondering
>> > if anyone knows the source location and build instructions for the jars
>> at
>> >
>> https://repository.cloudera.com/artifactory/libs-release-local/org/apache/flink/flink-shaded-hadoop-3-uber/
>> > .
>> >
>> > I ask because, AIAU, Hadoop 3 isn't really included/properly supported
>> in
>> > Flink, but we have users who are looking to use this exact library, and
>> > obviously we want to make sure it's security issue free, and easy for
>> us to
>> > maintain and has an awesome license.
>> >
>> > My biggest hunch is that it's from this repository
>> >
>> https://github.com/apache/hadoop/tree/rel/release-3.1.1/hadoop-client-modules
>> > but... that tag is over three years old, and the Cloudera builds linked
>> > above date from 2020. Again - apologies if it is such a long shot,
>> perhaps
>> > I've just missed something obvious.
>> >
>> >
>> > Thanks!
>> >
>>
>>