Option to disable chaining?

[Greg Hogan]

Is it possible to force operator chaining to be disabled? Similar to how
object reuse can be enabled or disabled?

Greg

[Nick Dimiduk-2]

https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()

On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <[hidden email]> wrote:

> Is it possible to force operator chaining to be disabled? Similar to how
> object reuse can be enabled or disabled?
>
> Greg
>

[Greg Hogan]

When is this useful in streaming?

On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <[hidden email]> wrote:

>
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
>
> On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <[hidden email]> wrote:
>
> > Is it possible to force operator chaining to be disabled? Similar to how
> > object reuse can be enabled or disabled?
> >
> > Greg
> >
>

[Stephan Ewen]

It may be useful for example when you have two MapFunctions and each does
something CPU intensive, or communicates with an external service.

Without chaining, you will have two threads and an elastic channel between
the functions to buffer some records, which may help in such a case.


On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <[hidden email]> wrote:

> When is this useful in streaming?
>
> On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <[hidden email]> wrote:
>
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> >
> > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <[hidden email]> wrote:
> >
> > > Is it possible to force operator chaining to be disabled? Similar to
> how
> > > object reuse can be enabled or disabled?
> > >
> > > Greg
> > >
> >
>

... [show rest of quote]

[Greg Hogan]

Is there a reason to not also implement this for batch processing? This
would allow object reuse to be truly disabled.

On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <[hidden email]> wrote:

> It may be useful for example when you have two MapFunctions and each does
> something CPU intensive, or communicates with an external service.
>
> Without chaining, you will have two threads and an elastic channel between
> the functions to buffer some records, which may help in such a case.
>
>
> On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <[hidden email]> wrote:
>
> > When is this useful in streaming?
> >
> > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <[hidden email]>
> wrote:
> >
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > >
> > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <[hidden email]>
> wrote:
> > >
> > > > Is it possible to force operator chaining to be disabled? Similar to
> > how
> > > > object reuse can be enabled or disabled?
> > > >
> > > > Greg
> > > >
> > >
> >
>

... [show rest of quote]

[Stephan Ewen]

There is no fundamental reason to not implement this for batch as well.

In Streaming, users seem to want more control about threads and resources
(given that these are often continuous pipelines), while in batch that was
not requested so far.

But I see that a non-chained function is more safe with respect to
accidental object reuse by a user in their functions...

On Mon, Feb 8, 2016 at 8:19 PM, Greg Hogan <[hidden email]> wrote:

> Is there a reason to not also implement this for batch processing? This
> would allow object reuse to be truly disabled.
>
> On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <[hidden email]> wrote:
>
> > It may be useful for example when you have two MapFunctions and each does
> > something CPU intensive, or communicates with an external service.
> >
> > Without chaining, you will have two threads and an elastic channel
> between
> > the functions to buffer some records, which may help in such a case.
> >
> >
> > On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <[hidden email]> wrote:
> >
> > > When is this useful in streaming?
> > >
> > > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <[hidden email]>
> > wrote:
> > >
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > > >
> > > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <[hidden email]>
> > wrote:
> > > >
> > > > > Is it possible to force operator chaining to be disabled? Similar
> to
> > > how
> > > > > object reuse can be enabled or disabled?
> > > > >
> > > > > Greg
> > > > >
> > > >
> > >
> >
>

... [show rest of quote]

[Nick Dimiduk]

Disabling operator chaining is also the only way to see metrics from each
step right now.

On Monday, February 8, 2016, Stephan Ewen <[hidden email]> wrote:

> There is no fundamental reason to not implement this for batch as well.
>
> In Streaming, users seem to want more control about threads and resources
> (given that these are often continuous pipelines), while in batch that was
> not requested so far.
>
> But I see that a non-chained function is more safe with respect to
> accidental object reuse by a user in their functions...
>
> On Mon, Feb 8, 2016 at 8:19 PM, Greg Hogan <[hidden email]
> <javascript:;>> wrote:
>
> > Is there a reason to not also implement this for batch processing? This
> > would allow object reuse to be truly disabled.
> >
> > On Mon, Feb 8, 2016 at 2:02 PM, Stephan Ewen <[hidden email]
> <javascript:;>> wrote:
> >
> > > It may be useful for example when you have two MapFunctions and each
> does
> > > something CPU intensive, or communicates with an external service.
> > >
> > > Without chaining, you will have two threads and an elastic channel
> > between
> > > the functions to buffer some records, which may help in such a case.
> > >
> > >
> > > On Mon, Feb 8, 2016 at 7:57 PM, Greg Hogan <[hidden email]
> <javascript:;>> wrote:
> > >
> > > > When is this useful in streaming?
> > > >
> > > > On Mon, Feb 8, 2016 at 1:46 PM, Nick Dimiduk <[hidden email]
> <javascript:;>>
> > > wrote:
> > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.html#disableOperatorChaining()
> > > > >
> > > > > On Mon, Feb 8, 2016 at 10:34 AM, Greg Hogan <[hidden email]
> <javascript:;>>
> > > wrote:
> > > > >
> > > > > > Is it possible to force operator chaining to be disabled? Similar
> > to
> > > > how
> > > > > > object reuse can be enabled or disabled?
> > > > > >
> > > > > > Greg
> > > > > >
> > > > >
> > > >
> > >
> >
>

... [show rest of quote]