Flink 1.2-SNAPSHOT Bucketing Sink problem

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink 1.2-SNAPSHOT Bucketing Sink problem

Ozan DENİZ
Hi everyone,


We are currently using BucketingSink (Flink 1.2-SNAPSHOT) function for sink operation. The BucketingSink function was working correctly. However today, when I put the job on cluster I got this error;


java.lang.NoSuchMethodError: org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getProcessingTimeService()Lorg/apache/flink/streaming/runtime/tasks/ProcessingTimeService;
        at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.open(BucketingSink.java:327)
        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:372)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:261)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
        at java.lang.Thread.run(Thread.java:745)


Is there any way to solve this problem?


(The rolling sink function is working based on the processing time and we need event time for hdfs. Therefore we have choosen bucketing sink function.)


Thank you for your helps.


Ozan.
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.2-SNAPSHOT Bucketing Sink problem

Fabian Hueske-2
Hi Ozan,

a NoSuchMethodError indicates a version mismatch. Since you are using a
SNAPSHOT build it is likely that dependencies changed when you recompiled
Flink or your job.
You should make sure that you are using the same version (on SNAPSHOT the
same commit) for jobs and cluster.

Best, Fabian

2016-10-26 18:30 GMT+02:00 Ozan DENİZ <[hidden email]>:

> Hi everyone,
>
>
> We are currently using BucketingSink (Flink 1.2-SNAPSHOT) function for
> sink operation. The BucketingSink function was working correctly. However
> today, when I put the job on cluster I got this error;
>
>
> java.lang.NoSuchMethodError: org.apache.flink.streaming.api.operators.
> StreamingRuntimeContext.getProcessingTimeService()
> Lorg/apache/flink/streaming/runtime/tasks/ProcessingTimeService;
>         at org.apache.flink.streaming.connectors.fs.bucketing.
> BucketingSink.open(BucketingSink.java:327)
>         at org.apache.flink.api.common.functions.util.FunctionUtils.
> openFunction(FunctionUtils.java:36)
>         at org.apache.flink.streaming.api.operators.
> AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
>         at org.apache.flink.streaming.runtime.tasks.StreamTask.
> openAllOperators(StreamTask.java:372)
>         at org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:261)
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> Is there any way to solve this problem?
>
>
> (The rolling sink function is working based on the processing time and we
> need event time for hdfs. Therefore we have choosen bucketing sink
> function.)
>
>
> Thank you for your helps.
>
>
> Ozan.
>
Reply | Threaded
Open this post in threaded view
|

Ynt: Flink 1.2-SNAPSHOT Bucketing Sink problem

Ozan DENİZ
Hi Fabian,


Thank you for quick response. We will check the versions of the flink in cluster and maven.


By the way, the BucketingSink function is really good at sinking to hdfs (Especially for event time file naming).  However, currently in version 1.1.3, the RollingSink function is used. Is there any plan to release the BucketingSink function in stable version?

________________________________
Gönderen: Fabian Hueske <[hidden email]>
Gönderildi: 26 Ekim 2016 Çarşamba 20:38:56
Kime: [hidden email]
Konu: Re: Flink 1.2-SNAPSHOT Bucketing Sink problem

Hi Ozan,

a NoSuchMethodError indicates a version mismatch. Since you are using a
SNAPSHOT build it is likely that dependencies changed when you recompiled
Flink or your job.
You should make sure that you are using the same version (on SNAPSHOT the
same commit) for jobs and cluster.

Best, Fabian

2016-10-26 18:30 GMT+02:00 Ozan DENİZ <[hidden email]>:

> Hi everyone,
>
>
> We are currently using BucketingSink (Flink 1.2-SNAPSHOT) function for
> sink operation. The BucketingSink function was working correctly. However
> today, when I put the job on cluster I got this error;
>
>
> java.lang.NoSuchMethodError: org.apache.flink.streaming.api.operators.
> StreamingRuntimeContext.getProcessingTimeService()
> Lorg/apache/flink/streaming/runtime/tasks/ProcessingTimeService;
>         at org.apache.flink.streaming.connectors.fs.bucketing.
> BucketingSink.open(BucketingSink.java:327)
>         at org.apache.flink.api.common.functions.util.FunctionUtils.
> openFunction(FunctionUtils.java:36)
>         at org.apache.flink.streaming.api.operators.
> AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
>         at org.apache.flink.streaming.runtime.tasks.StreamTask.
> openAllOperators(StreamTask.java:372)
>         at org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:261)
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> Is there any way to solve this problem?
>
>
> (The rolling sink function is working based on the processing time and we
> need event time for hdfs. Therefore we have choosen bucketing sink
> function.)
>
>
> Thank you for your helps.
>
>
> Ozan.
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink 1.2-SNAPSHOT Bucketing Sink problem

Fabian Hueske-2
This feature will be included in the upcoming 1.2.0 release.
The discussion about the next minor release has already started [1].

According to the current schedule we are targeting a first release end of
November which means we might have a release by mid December.

Cheers, Fabian

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Schedule-and-Scope-for-Flink-1-2-tp14062.html

2016-10-26 20:08 GMT+02:00 Ozan DENİZ <[hidden email]>:

> Hi Fabian,
>
>
> Thank you for quick response. We will check the versions of the flink in
> cluster and maven.
>
>
> By the way, the BucketingSink function is really good at sinking to hdfs
> (Especially for event time file naming).  However, currently in version
> 1.1.3, the RollingSink function is used. Is there any plan to release the
> BucketingSink function in stable version?
>
> ________________________________
> Gönderen: Fabian Hueske <[hidden email]>
> Gönderildi: 26 Ekim 2016 Çarşamba 20:38:56
> Kime: [hidden email]
> Konu: Re: Flink 1.2-SNAPSHOT Bucketing Sink problem
>
> Hi Ozan,
>
> a NoSuchMethodError indicates a version mismatch. Since you are using a
> SNAPSHOT build it is likely that dependencies changed when you recompiled
> Flink or your job.
> You should make sure that you are using the same version (on SNAPSHOT the
> same commit) for jobs and cluster.
>
> Best, Fabian
>
> 2016-10-26 18:30 GMT+02:00 Ozan DENİZ <[hidden email]>:
>
> > Hi everyone,
> >
> >
> > We are currently using BucketingSink (Flink 1.2-SNAPSHOT) function for
> > sink operation. The BucketingSink function was working correctly. However
> > today, when I put the job on cluster I got this error;
> >
> >
> > java.lang.NoSuchMethodError: org.apache.flink.streaming.api.operators.
> > StreamingRuntimeContext.getProcessingTimeService()
> > Lorg/apache/flink/streaming/runtime/tasks/ProcessingTimeService;
> >         at org.apache.flink.streaming.connectors.fs.bucketing.
> > BucketingSink.open(BucketingSink.java:327)
> >         at org.apache.flink.api.common.functions.util.FunctionUtils.
> > openFunction(FunctionUtils.java:36)
> >         at org.apache.flink.streaming.api.operators.
> > AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> >         at org.apache.flink.streaming.runtime.tasks.StreamTask.
> > openAllOperators(StreamTask.java:372)
> >         at org.apache.flink.streaming.runtime.tasks.StreamTask.
> > invoke(StreamTask.java:261)
> >         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
> >         at java.lang.Thread.run(Thread.java:745)
> >
> >
> > Is there any way to solve this problem?
> >
> >
> > (The rolling sink function is working based on the processing time and we
> > need event time for hdfs. Therefore we have choosen bucketing sink
> > function.)
> >
> >
> > Thank you for your helps.
> >
> >
> > Ozan.
> >
>