Specifying parallelism on join operation

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Specifying parallelism on join operation

Roshan Naik-2
 I cant find any place to specify the parallelism for the join here. 

stream1.join( stream2 )
                     .where( .. )
                    .equalTo( .. )
                    .window( .. )
                    .apply( .. );

How can we specify that ? 

-roshan
Reply | Threaded
Open this post in threaded view
|

Re: Specifying parallelism on join operation

Xiaowei Jiang
You can use with(JoinFunction)
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/datastream/JoinedStreams.WithWindow.html#with-org.apache.flink.api.common.functions.JoinFunction->
to workaround it. See JavaDoc for Flink 1.8:

@PublicEvolving
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/annotation/PublicEvolving.html>
 @Deprecated <http://docs.oracle.com/javase/7/docs/api/java/lang/Deprecated.html?is-external=true>
public <T> SingleOutputStreamOperator
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.html><T>
with(JoinFunction
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/api/common/functions/JoinFunction.html><T1
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/datastream/JoinedStreams.WithWindow.html>,T2
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/datastream/JoinedStreams.WithWindow.html>,T>
function)

Deprecated. This method will be removed once the apply(JoinFunction)
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/datastream/JoinedStreams.WithWindow.html#apply-org.apache.flink.api.common.functions.JoinFunction->
method
is fixed in the next major version of Flink (2.0).
Completes the join operation with the user function that is executed for
each combination of elements with the same key in a window.

*Note:* This is a temporary workaround while the apply(JoinFunction)
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/streaming/api/datastream/JoinedStreams.WithWindow.html#apply-org.apache.flink.api.common.functions.JoinFunction->
method
has the wrong return type and hence does not allow one to set an
operator-specific parallelism

On Sat, Jun 22, 2019 at 9:14 AM Roshan Naik <[hidden email]>
wrote:

>  I cant find any place to specify the parallelism for the join here.
>
> stream1.join( stream2 )
>                      .where( .. )
>                     .equalTo( .. )
>                     .window( .. )
>                     .apply( .. );
>
> How can we specify that ?
>
> -roshan
>