Hi,
While developing the out-of-core state backend that will store state directly to hdfs (either TFiles or BloomMapFiles), I realised that some file formats and features I use are hadoop 2.x only. What is the suggested way to handle features that use hadoop 2.x api? Can these be excluded from the travis build for the hadoop 1 profile somehow? Thanks, Gyula |
Hi Gyula,
will the out of core backend be in a separate maven module? If so, you can include the module only in the "hadoop-2" profile. As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests" are also "hadoop2" only modules: <profile> <id>hadoop-2</id> <activation> <property> <!-- Please do not remove the 'hadoop2' comment. See ./tools/generate_specific_pom.sh --> <!--hadoop2--><name>!hadoop.profile</name> </property> </activation> <properties> <hadoop.version>${hadoop-two.version}</hadoop.version> <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name> <shading-artifact-module.name>flink-shaded-hadoop2</shading-artifact-module.name> </properties> <modules> <module>flink-yarn</module> <!-- Include the flink-fs-tests project only for HD2. The HDFS minicluster interfaces changed between the two versions. --> <module>flink-fs-tests</module> </modules> </profile> If the backend is not in a separate maven module, you can use reflection. Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush" only if the method is available ;) On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <[hidden email]> wrote: > Hi, > > While developing the out-of-core state backend that will store state > directly to hdfs (either TFiles or BloomMapFiles), I realised that some > file formats and features I use are hadoop 2.x only. > > What is the suggested way to handle features that use hadoop 2.x api? Can > these be excluded from the travis build for the hadoop 1 profile somehow? > > Thanks, > Gyula > |
Alright, thanks.
My dilemma is that currently all the out-of-core state stuff I put under streaming-contrib. So I either make the whole contrib package hadoop 2 only which is probably kind of awkward, or I move the main interfaces from contrib to somewhere else then I can put the HDFS backend stuff into a module that is hadoop 2 only. Gyula Robert Metzger <[hidden email]> ezt írta (időpont: 2016. jan. 22., P, 22:31): > Hi Gyula, > > will the out of core backend be in a separate maven module? If so, you can > include the module only in the "hadoop-2" profile. > > As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests" are > also "hadoop2" only modules: > > <profile> > <id>hadoop-2</id> > <activation> > <property> > <!-- Please do not remove the 'hadoop2' comment. See > ./tools/generate_specific_pom.sh --> > <!--hadoop2--><name>!hadoop.profile</name> > </property> > </activation> > <properties> > <hadoop.version>${hadoop-two.version}</hadoop.version> > <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name> > <shading-artifact-module.name>flink-shaded-hadoop2</ > shading-artifact-module.name> > </properties> > <modules> > <module>flink-yarn</module> > <!-- Include the flink-fs-tests project only for HD2. > The HDFS minicluster interfaces changed between the two versions. > --> > <module>flink-fs-tests</module> > </modules> > </profile> > > > If the backend is not in a separate maven module, you can use reflection. > Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush" > only if the method is available ;) > > > > On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <[hidden email]> wrote: > > > Hi, > > > > While developing the out-of-core state backend that will store state > > directly to hdfs (either TFiles or BloomMapFiles), I realised that some > > file formats and features I use are hadoop 2.x only. > > > > What is the suggested way to handle features that use hadoop 2.x api? Can > > these be excluded from the travis build for the hadoop 1 profile somehow? > > > > Thanks, > > Gyula > > > |
At some point, the state backends should probably have their own parent-
and subprojects, similar to the batch-connectors and streaming-connectors. Then you have the flexibility to include and exclude them as needed. Stephan On Sat, Jan 23, 2016 at 6:42 PM, Gyula Fóra <[hidden email]> wrote: > Alright, thanks. > > My dilemma is that currently all the out-of-core state stuff I put under > streaming-contrib. So I either make the whole contrib package hadoop 2 only > which is probably kind of awkward, or I move the main interfaces from > contrib to somewhere else then I can put the HDFS backend stuff into a > module that is hadoop 2 only. > > Gyula > > Robert Metzger <[hidden email]> ezt írta (időpont: 2016. jan. 22., P, > 22:31): > > > Hi Gyula, > > > > will the out of core backend be in a separate maven module? If so, you > can > > include the module only in the "hadoop-2" profile. > > > > As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests" > are > > also "hadoop2" only modules: > > > > <profile> > > <id>hadoop-2</id> > > <activation> > > <property> > > <!-- Please do not remove the 'hadoop2' comment. See > > ./tools/generate_specific_pom.sh --> > > <!--hadoop2--><name>!hadoop.profile</name> > > </property> > > </activation> > > <properties> > > <hadoop.version>${hadoop-two.version}</hadoop.version> > > <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name > > > > <shading-artifact-module.name>flink-shaded-hadoop2</ > > shading-artifact-module.name> > > </properties> > > <modules> > > <module>flink-yarn</module> > > <!-- Include the flink-fs-tests project only for HD2. > > The HDFS minicluster interfaces changed between the two > versions. > > --> > > <module>flink-fs-tests</module> > > </modules> > > </profile> > > > > > > If the backend is not in a separate maven module, you can use reflection. > > Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush" > > only if the method is available ;) > > > > > > > > On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <[hidden email]> wrote: > > > > > Hi, > > > > > > While developing the out-of-core state backend that will store state > > > directly to hdfs (either TFiles or BloomMapFiles), I realised that some > > > file formats and features I use are hadoop 2.x only. > > > > > > What is the suggested way to handle features that use hadoop 2.x api? > Can > > > these be excluded from the travis build for the hadoop 1 profile > somehow? > > > > > > Thanks, > > > Gyula > > > > > > |
Free forum by Nabble | Edit this page |