How to parallel the subsequent flow of the split operator

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to parallel the subsequent flow of the split operator

钟旭阳
Hello:
I am a graduate student from ustc. I have a little doubt about flink programming recently. I hope to get your reply, thank you.
I use the split operator to split a stream into two streams. However, I found that the parallelism of the two streams can only be set separately, so that each of the two streams is itself parallel in multiple slots. But my goal is to get the two streams, one in one slot and the other in another.
Specifically, I split the two streams of the split operator, and the degree of parallelism is set to 1, respectively, then they are equivalent to running in the same slot. But my goal is to have them run in two slots separately.
Maybe I should use a keyedStream stream and customize KeySelector?
I hope to hear from you, thank you again.
Best
Xuyangzhong
Reply | Threaded
Open this post in threaded view
|

Re: How to parallel the subsequent flow of the split operator

Till Rohrmann
Hi Xuyangzhong,

the reason why both split consumers share the same slot is that per default
slot sharing is activated. You can define separate slot sharing groups via
`.slotSharingGroup("new_slot_sharing_group")` for the two consumers. This
will tell Flink to deploy both consumers in different slots.

Cheers,
Till

On Mon, Oct 21, 2019 at 10:41 AM 钟旭阳 <[hidden email]> wrote:

> Hello:
> I am a graduate student from ustc. I have a little doubt about flink
> programming recently. I hope to get your reply, thank you.
> I use the split operator to split a stream into two streams. However, I
> found that the parallelism of the two streams can only be set separately,
> so that each of the two streams is itself parallel in multiple slots. But
> my goal is to get the two streams, one in one slot and the other in another.
> Specifically, I split the two streams of the split operator, and the
> degree of parallelism is set to 1, respectively, then they are equivalent
> to running in the same slot. But my goal is to have them run in two slots
> separately.
> Maybe I should use a keyedStream stream and customize KeySelector?
> I hope to hear from you, thank you again.
> Best
> Xuyangzhong
Reply | Threaded
Open this post in threaded view
|

Re: Re: How to parallel the subsequent flow of the split operator

钟旭阳
That's what i really need!Thank you so much!
Yours
Xuyangzhong

在2019-10-21 17:07:21,Till Rohrmann<[hidden email]>写道:

> Hi Xuyangzhong,
>
> the reason why both split consumers share the same slot is that per default
> slot sharing is activated. You can define separate slot sharing groups via
> `.slotSharingGroup("new_slot_sharing_group")` for the two consumers. This
> will tell Flink to deploy both consumers in different slots.
>
> Cheers,
> Till
>
> On Mon, Oct 21, 2019 at 10:41 AM 钟旭阳 <[hidden email]> wrote:
>
> > Hello:
> > I am a graduate student from ustc. I have a little doubt about flink
> > programming recently. I hope to get your reply, thank you.
> > I use the split operator to split a stream into two streams. However, I
> > found that the parallelism of the two streams can only be set separately,
> > so that each of the two streams is itself parallel in multiple slots. But
> > my goal is to get the two streams, one in one slot and the other in another.
> > Specifically, I split the two streams of the split operator, and the
> > degree of parallelism is set to 1, respectively, then they are equivalent
> > to running in the same slot. But my goal is to have them run in two slots
> > separately.
> > Maybe I should use a keyedStream stream and customize KeySelector?
> > I hope to hear from you, thank you again.
> > Best
> > Xuyangzhong