Drafting a roadmap for Flink

[Stephan Ewen]

Hi everyone!

It is time we bring the Flink roadmap up to speed with what has happened in
the last months and what further goals features ideas have come up.

The link below leads to a Google Doc that contains an initial set of
suggestions that some of the committers have come up with. Please share
your opinion on those suggestions and feel free to suggest additional items
to put on the roadmap.

https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing

This is specifically open to everyone, not only committers. The link should
allow everyone to add suggestions and comments to the doc (but not to edit
it directly).

For new suggestions, it would help a lot if you could also mention whether
you would be available to help out with that feature or idea - that helps a
lot with prioritizing and estimate the time line.

For general suggestions to the "road mapping" process, please respond to
this mail.

Greetings and happy drafting!
Stephan

[Aljoscha Krettek-2]

I added some text about my work on the Logical Query feature.

On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]> wrote:

> Hi everyone!
>
> It is time we bring the Flink roadmap up to speed with what has happened in
> the last months and what further goals features ideas have come up.
>
> The link below leads to a Google Doc that contains an initial set of
> suggestions that some of the committers have come up with. Please share
> your opinion on those suggestions and feel free to suggest additional items
> to put on the roadmap.
>
>
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
>
> This is specifically open to everyone, not only committers. The link should
> allow everyone to add suggestions and comments to the doc (but not to edit
> it directly).
>
> For new suggestions, it would help a lot if you could also mention whether
> you would be available to help out with that feature or idea - that helps a
> lot with prioritizing and estimate the time line.
>
> For general suggestions to the "road mapping" process, please respond to
> this mail.
>
> Greetings and happy drafting!
> Stephan
>

[Fabian Hueske]

Hi,

I think the roadmap should show the long-term development goals of Flink,
i.e., show that we are going for a ML library, SQL support, Batch-Streaming
integration, etc.

Right now, it is quite detailed and with very optimistic time estimates,
IMO.
If we would do everything in time, we would be done with the roadmap in  Q3
2015...
I would not even put a time on all issues, esp. on things which depend on
other developments (which might not even have started). Also I would make
the estimates more coarse-grained. For short-term goals we could use
quarters, everything does not need an estimate, IMO. Issues that will be
solved in two months don't even need to be listed.


2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:

> I added some text about my work on the Logical Query feature.
>
> On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]> wrote:
>
> > Hi everyone!
> >
> > It is time we bring the Flink roadmap up to speed with what has happened
> in
> > the last months and what further goals features ideas have come up.
> >
> > The link below leads to a Google Doc that contains an initial set of
> > suggestions that some of the committers have come up with. Please share
> > your opinion on those suggestions and feel free to suggest additional
> items
> > to put on the roadmap.
> >
> >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> >
> > This is specifically open to everyone, not only committers. The link
> should
> > allow everyone to add suggestions and comments to the doc (but not to
> edit
> > it directly).
> >
> > For new suggestions, it would help a lot if you could also mention
> whether
> > you would be available to help out with that feature or idea - that
> helps a
> > lot with prioritizing and estimate the time line.
> >
> > For general suggestions to the "road mapping" process, please respond to
> > this mail.
> >
> > Greetings and happy drafting!
> > Stephan
> >
>

[Robert Metzger]

Hi,

I very much like the "PIG Journal" here:
https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
Its basically a nice view (however outdated in that case) on whats going on
in the PIG community. You can see finished features on the top, current
features being developed in the middle and ideas in the end.

The document posted by Stephan is a good start to create a "Flink Journal".
I agree with Fabian that the estimates are very optimistic. Implementing
all these features including unit tests, documentation and testing a takes
a lot of time.
I would suggest to only add estimates (finish dates) to features which are
currently work in progress.
The remainder ("ideas") can have time estimates in months but should not
have finish dates. Similarly to Pig, we should put a disclaimer on top that
we do not guarantee for any feature being developed.




On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]> wrote:

> Hi,
>
> I think the roadmap should show the long-term development goals of Flink,
> i.e., show that we are going for a ML library, SQL support, Batch-Streaming
> integration, etc.
>
> Right now, it is quite detailed and with very optimistic time estimates,
> IMO.
> If we would do everything in time, we would be done with the roadmap in  Q3
> 2015...
> I would not even put a time on all issues, esp. on things which depend on
> other developments (which might not even have started). Also I would make
> the estimates more coarse-grained. For short-term goals we could use
> quarters, everything does not need an estimate, IMO. Issues that will be
> solved in two months don't even need to be listed.
>
>
> 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:
>
> > I added some text about my work on the Logical Query feature.
> >
> > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Hi everyone!
> > >
> > > It is time we bring the Flink roadmap up to speed with what has
> happened
> > in
> > > the last months and what further goals features ideas have come up.
> > >
> > > The link below leads to a Google Doc that contains an initial set of
> > > suggestions that some of the committers have come up with. Please share
> > > your opinion on those suggestions and feel free to suggest additional
> > items
> > > to put on the roadmap.
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> > >
> > > This is specifically open to everyone, not only committers. The link
> > should
> > > allow everyone to add suggestions and comments to the doc (but not to
> > edit
> > > it directly).
> > >
> > > For new suggestions, it would help a lot if you could also mention
> > whether
> > > you would be available to help out with that feature or idea - that
> > helps a
> > > lot with prioritizing and estimate the time line.
> > >
> > > For general suggestions to the "road mapping" process, please respond
> to
> > > this mail.
> > >
> > > Greetings and happy drafting!
> > > Stephan
> > >
> >
>

[Stephan Ewen]

The Pig Journal is nice, we could have this evolve into something like that.

I think that we need to give some time estimate on the features / issues.
Otherwise, it is of rather little value - all it says is that people
thought about that, no one knows when you can plan with it.

We can coarsen the time estimates, though...

Stephan


On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <[hidden email]> wrote:

> Hi,
>
> I very much like the "PIG Journal" here:
> https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
> Its basically a nice view (however outdated in that case) on whats going on
> in the PIG community. You can see finished features on the top, current
> features being developed in the middle and ideas in the end.
>
> The document posted by Stephan is a good start to create a "Flink Journal".
> I agree with Fabian that the estimates are very optimistic. Implementing
> all these features including unit tests, documentation and testing a takes
> a lot of time.
> I would suggest to only add estimates (finish dates) to features which are
> currently work in progress.
> The remainder ("ideas") can have time estimates in months but should not
> have finish dates. Similarly to Pig, we should put a disclaimer on top that
> we do not guarantee for any feature being developed.
>
>
>
>
> On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi,
> >
> > I think the roadmap should show the long-term development goals of Flink,
> > i.e., show that we are going for a ML library, SQL support,
> Batch-Streaming
> > integration, etc.
> >
> > Right now, it is quite detailed and with very optimistic time estimates,
> > IMO.
> > If we would do everything in time, we would be done with the roadmap in
> Q3
> > 2015...
> > I would not even put a time on all issues, esp. on things which depend on
> > other developments (which might not even have started). Also I would make
> > the estimates more coarse-grained. For short-term goals we could use
> > quarters, everything does not need an estimate, IMO. Issues that will be
> > solved in two months don't even need to be listed.
> >
> >
> > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:
> >
> > > I added some text about my work on the Logical Query feature.
> > >
> > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]>
> wrote:
> > >
> > > > Hi everyone!
> > > >
> > > > It is time we bring the Flink roadmap up to speed with what has
> > happened
> > > in
> > > > the last months and what further goals features ideas have come up.
> > > >
> > > > The link below leads to a Google Doc that contains an initial set of
> > > > suggestions that some of the committers have come up with. Please
> share
> > > > your opinion on those suggestions and feel free to suggest additional
> > > items
> > > > to put on the roadmap.
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> > > >
> > > > This is specifically open to everyone, not only committers. The link
> > > should
> > > > allow everyone to add suggestions and comments to the doc (but not to
> > > edit
> > > > it directly).
> > > >
> > > > For new suggestions, it would help a lot if you could also mention
> > > whether
> > > > you would be available to help out with that feature or idea - that
> > > helps a
> > > > lot with prioritizing and estimate the time line.
> > > >
> > > > For general suggestions to the "road mapping" process, please respond
> > to
> > > > this mail.
> > > >
> > > > Greetings and happy drafting!
> > > > Stephan
> > > >
> > >
> >
>

[Fabian Hueske]

Having an estimate assigned to an issue might give the impression that it
is already assigned to somebody.
This would not help to find external contributors who are interested in
helping with a certain feature.

Issues without estimates are still useful as they show in which direction
the project plans to evolve.

2015-01-08 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:

> The Pig Journal is nice, we could have this evolve into something like
> that.
>
> I think that we need to give some time estimate on the features / issues.
> Otherwise, it is of rather little value - all it says is that people
> thought about that, no one knows when you can plan with it.
>
> We can coarsen the time estimates, though...
>
> Stephan
>
>
> On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <[hidden email]>
> wrote:
>
> > Hi,
> >
> > I very much like the "PIG Journal" here:
> > https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
> > Its basically a nice view (however outdated in that case) on whats going
> on
> > in the PIG community. You can see finished features on the top, current
> > features being developed in the middle and ideas in the end.
> >
> > The document posted by Stephan is a good start to create a "Flink
> Journal".
> > I agree with Fabian that the estimates are very optimistic. Implementing
> > all these features including unit tests, documentation and testing a
> takes
> > a lot of time.
> > I would suggest to only add estimates (finish dates) to features which
> are
> > currently work in progress.
> > The remainder ("ideas") can have time estimates in months but should not
> > have finish dates. Similarly to Pig, we should put a disclaimer on top
> that
> > we do not guarantee for any feature being developed.
> >
> >
> >
> >
> > On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi,
> > >
> > > I think the roadmap should show the long-term development goals of
> Flink,
> > > i.e., show that we are going for a ML library, SQL support,
> > Batch-Streaming
> > > integration, etc.
> > >
> > > Right now, it is quite detailed and with very optimistic time
> estimates,
> > > IMO.
> > > If we would do everything in time, we would be done with the roadmap in
> > Q3
> > > 2015...
> > > I would not even put a time on all issues, esp. on things which depend
> on
> > > other developments (which might not even have started). Also I would
> make
> > > the estimates more coarse-grained. For short-term goals we could use
> > > quarters, everything does not need an estimate, IMO. Issues that will
> be
> > > solved in two months don't even need to be listed.
> > >
> > >
> > > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:
> > >
> > > > I added some text about my work on the Logical Query feature.
> > > >
> > > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]>
> > wrote:
> > > >
> > > > > Hi everyone!
> > > > >
> > > > > It is time we bring the Flink roadmap up to speed with what has
> > > happened
> > > > in
> > > > > the last months and what further goals features ideas have come up.
> > > > >
> > > > > The link below leads to a Google Doc that contains an initial set
> of
> > > > > suggestions that some of the committers have come up with. Please
> > share
> > > > > your opinion on those suggestions and feel free to suggest
> additional
> > > > items
> > > > > to put on the roadmap.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> > > > >
> > > > > This is specifically open to everyone, not only committers. The
> link
> > > > should
> > > > > allow everyone to add suggestions and comments to the doc (but not
> to
> > > > edit
> > > > > it directly).
> > > > >
> > > > > For new suggestions, it would help a lot if you could also mention
> > > > whether
> > > > > you would be available to help out with that feature or idea - that
> > > > helps a
> > > > > lot with prioritizing and estimate the time line.
> > > > >
> > > > > For general suggestions to the "road mapping" process, please
> respond
> > > to
> > > > > this mail.
> > > > >
> > > > > Greetings and happy drafting!
> > > > > Stephan
> > > > >
> > > >
> > >
> >
>

[Stephan Ewen]

Okay, I see what you are going. Some issues have people working on them,
some do not.

How about we add a "responsible person" to the items that have someone in
charge already, and mark others as open?

Associating a responsible person (that need not be the one that does all
the work, but the one that supervises the issue) may be a good idea in
general.

On Thu, Jan 8, 2015 at 11:26 AM, Fabian Hueske <[hidden email]> wrote:

> Having an estimate assigned to an issue might give the impression that it
> is already assigned to somebody.
> This would not help to find external contributors who are interested in
> helping with a certain feature.
>
> Issues without estimates are still useful as they show in which direction
> the project plans to evolve.
>
> 2015-01-08 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:
>
> > The Pig Journal is nice, we could have this evolve into something like
> > that.
> >
> > I think that we need to give some time estimate on the features / issues.
> > Otherwise, it is of rather little value - all it says is that people
> > thought about that, no one knows when you can plan with it.
> >
> > We can coarsen the time estimates, though...
> >
> > Stephan
> >
> >
> > On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > I very much like the "PIG Journal" here:
> > > https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
> > > Its basically a nice view (however outdated in that case) on whats
> going
> > on
> > > in the PIG community. You can see finished features on the top, current
> > > features being developed in the middle and ideas in the end.
> > >
> > > The document posted by Stephan is a good start to create a "Flink
> > Journal".
> > > I agree with Fabian that the estimates are very optimistic.
> Implementing
> > > all these features including unit tests, documentation and testing a
> > takes
> > > a lot of time.
> > > I would suggest to only add estimates (finish dates) to features which
> > are
> > > currently work in progress.
> > > The remainder ("ideas") can have time estimates in months but should
> not
> > > have finish dates. Similarly to Pig, we should put a disclaimer on top
> > that
> > > we do not guarantee for any feature being developed.
> > >
> > >
> > >
> > >
> > > On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I think the roadmap should show the long-term development goals of
> > Flink,
> > > > i.e., show that we are going for a ML library, SQL support,
> > > Batch-Streaming
> > > > integration, etc.
> > > >
> > > > Right now, it is quite detailed and with very optimistic time
> > estimates,
> > > > IMO.
> > > > If we would do everything in time, we would be done with the roadmap
> in
> > > Q3
> > > > 2015...
> > > > I would not even put a time on all issues, esp. on things which
> depend
> > on
> > > > other developments (which might not even have started). Also I would
> > make
> > > > the estimates more coarse-grained. For short-term goals we could use
> > > > quarters, everything does not need an estimate, IMO. Issues that will
> > be
> > > > solved in two months don't even need to be listed.
> > > >
> > > >
> > > > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:
> > > >
> > > > > I added some text about my work on the Logical Query feature.
> > > > >
> > > > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hi everyone!
> > > > > >
> > > > > > It is time we bring the Flink roadmap up to speed with what has
> > > > happened
> > > > > in
> > > > > > the last months and what further goals features ideas have come
> up.
> > > > > >
> > > > > > The link below leads to a Google Doc that contains an initial set
> > of
> > > > > > suggestions that some of the committers have come up with. Please
> > > share
> > > > > > your opinion on those suggestions and feel free to suggest
> > additional
> > > > > items
> > > > > > to put on the roadmap.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> > > > > >
> > > > > > This is specifically open to everyone, not only committers. The
> > link
> > > > > should
> > > > > > allow everyone to add suggestions and comments to the doc (but
> not
> > to
> > > > > edit
> > > > > > it directly).
> > > > > >
> > > > > > For new suggestions, it would help a lot if you could also
> mention
> > > > > whether
> > > > > > you would be available to help out with that feature or idea -
> that
> > > > > helps a
> > > > > > lot with prioritizing and estimate the time line.
> > > > > >
> > > > > > For general suggestions to the "road mapping" process, please
> > respond
> > > > to
> > > > > > this mail.
> > > > > >
> > > > > > Greetings and happy drafting!
> > > > > > Stephan
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Fabian Hueske]

Adding a responsible person sounds good. We should make sure that this role
is clearly communicated though.

How about adding priorities instead of time estimates?
That would help to see how the priorities are set in Flink and which
features to expect next (without having a date assigned to it though).

2015-01-08 11:30 GMT+01:00 Stephan Ewen <[hidden email]>:

> Okay, I see what you are going. Some issues have people working on them,
> some do not.
>
> How about we add a "responsible person" to the items that have someone in
> charge already, and mark others as open?
>
> Associating a responsible person (that need not be the one that does all
> the work, but the one that supervises the issue) may be a good idea in
> general.
>
> On Thu, Jan 8, 2015 at 11:26 AM, Fabian Hueske <[hidden email]> wrote:
>
> > Having an estimate assigned to an issue might give the impression that it
> > is already assigned to somebody.
> > This would not help to find external contributors who are interested in
> > helping with a certain feature.
> >
> > Issues without estimates are still useful as they show in which direction
> > the project plans to evolve.
> >
> > 2015-01-08 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:
> >
> > > The Pig Journal is nice, we could have this evolve into something like
> > > that.
> > >
> > > I think that we need to give some time estimate on the features /
> issues.
> > > Otherwise, it is of rather little value - all it says is that people
> > > thought about that, no one knows when you can plan with it.
> > >
> > > We can coarsen the time estimates, though...
> > >
> > > Stephan
> > >
> > >
> > > On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <[hidden email]>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I very much like the "PIG Journal" here:
> > > > https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
> > > > Its basically a nice view (however outdated in that case) on whats
> > going
> > > on
> > > > in the PIG community. You can see finished features on the top,
> current
> > > > features being developed in the middle and ideas in the end.
> > > >
> > > > The document posted by Stephan is a good start to create a "Flink
> > > Journal".
> > > > I agree with Fabian that the estimates are very optimistic.
> > Implementing
> > > > all these features including unit tests, documentation and testing a
> > > takes
> > > > a lot of time.
> > > > I would suggest to only add estimates (finish dates) to features
> which
> > > are
> > > > currently work in progress.
> > > > The remainder ("ideas") can have time estimates in months but should
> > not
> > > > have finish dates. Similarly to Pig, we should put a disclaimer on
> top
> > > that
> > > > we do not guarantee for any feature being developed.
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I think the roadmap should show the long-term development goals of
> > > Flink,
> > > > > i.e., show that we are going for a ML library, SQL support,
> > > > Batch-Streaming
> > > > > integration, etc.
> > > > >
> > > > > Right now, it is quite detailed and with very optimistic time
> > > estimates,
> > > > > IMO.
> > > > > If we would do everything in time, we would be done with the
> roadmap
> > in
> > > > Q3
> > > > > 2015...
> > > > > I would not even put a time on all issues, esp. on things which
> > depend
> > > on
> > > > > other developments (which might not even have started). Also I
> would
> > > make
> > > > > the estimates more coarse-grained. For short-term goals we could
> use
> > > > > quarters, everything does not need an estimate, IMO. Issues that
> will
> > > be
> > > > > solved in two months don't even need to be listed.
> > > > >
> > > > >
> > > > > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:
> > > > >
> > > > > > I added some text about my work on the Logical Query feature.
> > > > > >
> > > > > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > > > Hi everyone!
> > > > > > >
> > > > > > > It is time we bring the Flink roadmap up to speed with what has
> > > > > happened
> > > > > > in
> > > > > > > the last months and what further goals features ideas have come
> > up.
> > > > > > >
> > > > > > > The link below leads to a Google Doc that contains an initial
> set
> > > of
> > > > > > > suggestions that some of the committers have come up with.
> Please
> > > > share
> > > > > > > your opinion on those suggestions and feel free to suggest
> > > additional
> > > > > > items
> > > > > > > to put on the roadmap.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> > > > > > >
> > > > > > > This is specifically open to everyone, not only committers. The
> > > link
> > > > > > should
> > > > > > > allow everyone to add suggestions and comments to the doc (but
> > not
> > > to
> > > > > > edit
> > > > > > > it directly).
> > > > > > >
> > > > > > > For new suggestions, it would help a lot if you could also
> > mention
> > > > > > whether
> > > > > > > you would be available to help out with that feature or idea -
> > that
> > > > > > helps a
> > > > > > > lot with prioritizing and estimate the time line.
> > > > > > >
> > > > > > > For general suggestions to the "road mapping" process, please
> > > respond
> > > > > to
> > > > > > > this mail.
> > > > > > >
> > > > > > > Greetings and happy drafting!
> > > > > > > Stephan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Kostas Tzoumas-2]

+1 for indicating the person currently working on the issue, we can just
open a JIRA issue for each of these. And we can clearly indicate that other
features are not being currently worked on.

How about indicating rough time goals (quarters) for issues that are
currently being worked on (of course with the concern of the assignee)?

I have a problem with priorities: the only priorities I see right now are
P1 (someone is working on this) and P2 (noone is working on this), and this
information is already conveyed by the JIRAs. We can come up with a more
detailed priority scheme, but would this be easier to implement than
date-to-complete goals?

Kostas



On Thu, Jan 8, 2015 at 11:39 AM, Fabian Hueske <[hidden email]> wrote:

> Adding a responsible person sounds good. We should make sure that this role
> is clearly communicated though.
>
> How about adding priorities instead of time estimates?
> That would help to see how the priorities are set in Flink and which
> features to expect next (without having a date assigned to it though).
>
> 2015-01-08 11:30 GMT+01:00 Stephan Ewen <[hidden email]>:
>
> > Okay, I see what you are going. Some issues have people working on them,
> > some do not.
> >
> > How about we add a "responsible person" to the items that have someone in
> > charge already, and mark others as open?
> >
> > Associating a responsible person (that need not be the one that does all
> > the work, but the one that supervises the issue) may be a good idea in
> > general.
> >
> > On Thu, Jan 8, 2015 at 11:26 AM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Having an estimate assigned to an issue might give the impression that
> it
> > > is already assigned to somebody.
> > > This would not help to find external contributors who are interested in
> > > helping with a certain feature.
> > >
> > > Issues without estimates are still useful as they show in which
> direction
> > > the project plans to evolve.
> > >
> > > 2015-01-08 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:
> > >
> > > > The Pig Journal is nice, we could have this evolve into something
> like
> > > > that.
> > > >
> > > > I think that we need to give some time estimate on the features /
> > issues.
> > > > Otherwise, it is of rather little value - all it says is that people
> > > > thought about that, no one knows when you can plan with it.
> > > >
> > > > We can coarsen the time estimates, though...
> > > >
> > > > Stephan
> > > >
> > > >
> > > > On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I very much like the "PIG Journal" here:
> > > > > https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
> > > > > Its basically a nice view (however outdated in that case) on whats
> > > going
> > > > on
> > > > > in the PIG community. You can see finished features on the top,
> > current
> > > > > features being developed in the middle and ideas in the end.
> > > > >
> > > > > The document posted by Stephan is a good start to create a "Flink
> > > > Journal".
> > > > > I agree with Fabian that the estimates are very optimistic.
> > > Implementing
> > > > > all these features including unit tests, documentation and testing
> a
> > > > takes
> > > > > a lot of time.
> > > > > I would suggest to only add estimates (finish dates) to features
> > which
> > > > are
> > > > > currently work in progress.
> > > > > The remainder ("ideas") can have time estimates in months but
> should
> > > not
> > > > > have finish dates. Similarly to Pig, we should put a disclaimer on
> > top
> > > > that
> > > > > we do not guarantee for any feature being developed.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I think the roadmap should show the long-term development goals
> of
> > > > Flink,
> > > > > > i.e., show that we are going for a ML library, SQL support,
> > > > > Batch-Streaming
> > > > > > integration, etc.
> > > > > >
> > > > > > Right now, it is quite detailed and with very optimistic time
> > > > estimates,
> > > > > > IMO.
> > > > > > If we would do everything in time, we would be done with the
> > roadmap
> > > in
> > > > > Q3
> > > > > > 2015...
> > > > > > I would not even put a time on all issues, esp. on things which
> > > depend
> > > > on
> > > > > > other developments (which might not even have started). Also I
> > would
> > > > make
> > > > > > the estimates more coarse-grained. For short-term goals we could
> > use
> > > > > > quarters, everything does not need an estimate, IMO. Issues that
> > will
> > > > be
> > > > > > solved in two months don't even need to be listed.
> > > > > >
> > > > > >
> > > > > > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]
> >:
> > > > > >
> > > > > > > I added some text about my work on the Logical Query feature.
> > > > > > >
> > > > > > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <
> [hidden email]>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone!
> > > > > > > >
> > > > > > > > It is time we bring the Flink roadmap up to speed with what
> has
> > > > > > happened
> > > > > > > in
> > > > > > > > the last months and what further goals features ideas have
> come
> > > up.
> > > > > > > >
> > > > > > > > The link below leads to a Google Doc that contains an initial
> > set
> > > > of
> > > > > > > > suggestions that some of the committers have come up with.
> > Please
> > > > > share
> > > > > > > > your opinion on those suggestions and feel free to suggest
> > > > additional
> > > > > > > items
> > > > > > > > to put on the roadmap.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
> > > > > > > >
> > > > > > > > This is specifically open to everyone, not only committers.
> The
> > > > link
> > > > > > > should
> > > > > > > > allow everyone to add suggestions and comments to the doc
> (but
> > > not
> > > > to
> > > > > > > edit
> > > > > > > > it directly).
> > > > > > > >
> > > > > > > > For new suggestions, it would help a lot if you could also
> > > mention
> > > > > > > whether
> > > > > > > > you would be available to help out with that feature or idea
> -
> > > that
> > > > > > > helps a
> > > > > > > > lot with prioritizing and estimate the time line.
> > > > > > > >
> > > > > > > > For general suggestions to the "road mapping" process, please
> > > > respond
> > > > > > to
> > > > > > > > this mail.
> > > > > > > >
> > > > > > > > Greetings and happy drafting!
> > > > > > > > Stephan
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Kostas Tzoumas-2]

Hi everyone,

I started a wiki page about this:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Roadmap

If you are working on one of these features, could you insert the
corresponding JIRA ticket and expand the description if you think it's not
informative enough?

I saw that there is a streaming roadmap page as well, I think we should
have only one. The styles are currently a bit different though. Perhaps we
could isolate some JIRAs/specific features from the projects mentioned in
the streaming roadmap and insert them in the general roadmap? What do you
think?

Best,
Kostas

On Thu, Jan 8, 2015 at 9:16 AM, Kostas Tzoumas <[hidden email]> wrote:

> +1 for indicating the person currently working on the issue, we can just
> open a JIRA issue for each of these. And we can clearly indicate that other
> features are not being currently worked on.
>
> How about indicating rough time goals (quarters) for issues that are
> currently being worked on (of course with the concern of the assignee)?
>
> I have a problem with priorities: the only priorities I see right now are
> P1 (someone is working on this) and P2 (noone is working on this), and this
> information is already conveyed by the JIRAs. We can come up with a more
> detailed priority scheme, but would this be easier to implement than
> date-to-complete goals?
>
> Kostas
>
>
>
> On Thu, Jan 8, 2015 at 11:39 AM, Fabian Hueske <[hidden email]> wrote:
>
>> Adding a responsible person sounds good. We should make sure that this
>> role
>> is clearly communicated though.
>>
>> How about adding priorities instead of time estimates?
>> That would help to see how the priorities are set in Flink and which
>> features to expect next (without having a date assigned to it though).
>>
>> 2015-01-08 11:30 GMT+01:00 Stephan Ewen <[hidden email]>:
>>
>> > Okay, I see what you are going. Some issues have people working on them,
>> > some do not.
>> >
>> > How about we add a "responsible person" to the items that have someone
>> in
>> > charge already, and mark others as open?
>> >
>> > Associating a responsible person (that need not be the one that does all
>> > the work, but the one that supervises the issue) may be a good idea in
>> > general.
>> >
>> > On Thu, Jan 8, 2015 at 11:26 AM, Fabian Hueske <[hidden email]>
>> wrote:
>> >
>> > > Having an estimate assigned to an issue might give the impression
>> that it
>> > > is already assigned to somebody.
>> > > This would not help to find external contributors who are interested
>> in
>> > > helping with a certain feature.
>> > >
>> > > Issues without estimates are still useful as they show in which
>> direction
>> > > the project plans to evolve.
>> > >
>> > > 2015-01-08 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:
>> > >
>> > > > The Pig Journal is nice, we could have this evolve into something
>> like
>> > > > that.
>> > > >
>> > > > I think that we need to give some time estimate on the features /
>> > issues.
>> > > > Otherwise, it is of rather little value - all it says is that people
>> > > > thought about that, no one knows when you can plan with it.
>> > > >
>> > > > We can coarsen the time estimates, though...
>> > > >
>> > > > Stephan
>> > > >
>> > > >
>> > > > On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <
>> [hidden email]>
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I very much like the "PIG Journal" here:
>> > > > > https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
>> > > > > Its basically a nice view (however outdated in that case) on whats
>> > > going
>> > > > on
>> > > > > in the PIG community. You can see finished features on the top,
>> > current
>> > > > > features being developed in the middle and ideas in the end.
>> > > > >
>> > > > > The document posted by Stephan is a good start to create a "Flink
>> > > > Journal".
>> > > > > I agree with Fabian that the estimates are very optimistic.
>> > > Implementing
>> > > > > all these features including unit tests, documentation and
>> testing a
>> > > > takes
>> > > > > a lot of time.
>> > > > > I would suggest to only add estimates (finish dates) to features
>> > which
>> > > > are
>> > > > > currently work in progress.
>> > > > > The remainder ("ideas") can have time estimates in months but
>> should
>> > > not
>> > > > > have finish dates. Similarly to Pig, we should put a disclaimer on
>> > top
>> > > > that
>> > > > > we do not guarantee for any feature being developed.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > I think the roadmap should show the long-term development goals
>> of
>> > > > Flink,
>> > > > > > i.e., show that we are going for a ML library, SQL support,
>> > > > > Batch-Streaming
>> > > > > > integration, etc.
>> > > > > >
>> > > > > > Right now, it is quite detailed and with very optimistic time
>> > > > estimates,
>> > > > > > IMO.
>> > > > > > If we would do everything in time, we would be done with the
>> > roadmap
>> > > in
>> > > > > Q3
>> > > > > > 2015...
>> > > > > > I would not even put a time on all issues, esp. on things which
>> > > depend
>> > > > on
>> > > > > > other developments (which might not even have started). Also I
>> > would
>> > > > make
>> > > > > > the estimates more coarse-grained. For short-term goals we could
>> > use
>> > > > > > quarters, everything does not need an estimate, IMO. Issues that
>> > will
>> > > > be
>> > > > > > solved in two months don't even need to be listed.
>> > > > > >
>> > > > > >
>> > > > > > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]
>> >:
>> > > > > >
>> > > > > > > I added some text about my work on the Logical Query feature.
>> > > > > > >
>> > > > > > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <
>> [hidden email]>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi everyone!
>> > > > > > > >
>> > > > > > > > It is time we bring the Flink roadmap up to speed with what
>> has
>> > > > > > happened
>> > > > > > > in
>> > > > > > > > the last months and what further goals features ideas have
>> come
>> > > up.
>> > > > > > > >
>> > > > > > > > The link below leads to a Google Doc that contains an
>> initial
>> > set
>> > > > of
>> > > > > > > > suggestions that some of the committers have come up with.
>> > Please
>> > > > > share
>> > > > > > > > your opinion on those suggestions and feel free to suggest
>> > > > additional
>> > > > > > > items
>> > > > > > > > to put on the roadmap.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
>> > > > > > > >
>> > > > > > > > This is specifically open to everyone, not only committers.
>> The
>> > > > link
>> > > > > > > should
>> > > > > > > > allow everyone to add suggestions and comments to the doc
>> (but
>> > > not
>> > > > to
>> > > > > > > edit
>> > > > > > > > it directly).
>> > > > > > > >
>> > > > > > > > For new suggestions, it would help a lot if you could also
>> > > mention
>> > > > > > > whether
>> > > > > > > > you would be available to help out with that feature or
>> idea -
>> > > that
>> > > > > > > helps a
>> > > > > > > > lot with prioritizing and estimate the time line.
>> > > > > > > >
>> > > > > > > > For general suggestions to the "road mapping" process,
>> please
>> > > > respond
>> > > > > > to
>> > > > > > > > this mail.
>> > > > > > > >
>> > > > > > > > Greetings and happy drafting!
>> > > > > > > > Stephan
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

[Max Michels]

Hi Kostas,

Thanks for putting this into the wiki. I added the JIRA link for the
off-heap memory. Now the wiki displays:

> Error rendering macro 'jira' : com.atlassian.confluence.macro.MacroExecutionException: java.lang.RuntimeException: Not Found

If persistent, we should report this.

Is this the Flink streaming roadmap you were referring to?
https://cwiki.apache.org/confluence/display/FLINK/Flink+Streaming

I agree, there should only be one roadmap. As far as I see, a lot of
points in the streaming roadmap are also contained in the general
roadmap. We should further join them were possible and leave the
remaining parts of the streaming roadmap for future reference.

Best regards,
Max

On Mon, Jan 19, 2015 at 7:57 PM, Kostas Tzoumas <[hidden email]> wrote:

> Hi everyone,
>
> I started a wiki page about this:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Roadmap
>
> If you are working on one of these features, could you insert the
> corresponding JIRA ticket and expand the description if you think it's not
> informative enough?
>
> I saw that there is a streaming roadmap page as well, I think we should
> have only one. The styles are currently a bit different though. Perhaps we
> could isolate some JIRAs/specific features from the projects mentioned in
> the streaming roadmap and insert them in the general roadmap? What do you
> think?
>
> Best,
> Kostas
>
> On Thu, Jan 8, 2015 at 9:16 AM, Kostas Tzoumas <[hidden email]> wrote:
>
>> +1 for indicating the person currently working on the issue, we can just
>> open a JIRA issue for each of these. And we can clearly indicate that other
>> features are not being currently worked on.
>>
>> How about indicating rough time goals (quarters) for issues that are
>> currently being worked on (of course with the concern of the assignee)?
>>
>> I have a problem with priorities: the only priorities I see right now are
>> P1 (someone is working on this) and P2 (noone is working on this), and this
>> information is already conveyed by the JIRAs. We can come up with a more
>> detailed priority scheme, but would this be easier to implement than
>> date-to-complete goals?
>>
>> Kostas
>>
>>
>>
>> On Thu, Jan 8, 2015 at 11:39 AM, Fabian Hueske <[hidden email]> wrote:
>>
>>> Adding a responsible person sounds good. We should make sure that this
>>> role
>>> is clearly communicated though.
>>>
>>> How about adding priorities instead of time estimates?
>>> That would help to see how the priorities are set in Flink and which
>>> features to expect next (without having a date assigned to it though).
>>>
>>> 2015-01-08 11:30 GMT+01:00 Stephan Ewen <[hidden email]>:
>>>
>>> > Okay, I see what you are going. Some issues have people working on them,
>>> > some do not.
>>> >
>>> > How about we add a "responsible person" to the items that have someone
>>> in
>>> > charge already, and mark others as open?
>>> >
>>> > Associating a responsible person (that need not be the one that does all
>>> > the work, but the one that supervises the issue) may be a good idea in
>>> > general.
>>> >
>>> > On Thu, Jan 8, 2015 at 11:26 AM, Fabian Hueske <[hidden email]>
>>> wrote:
>>> >
>>> > > Having an estimate assigned to an issue might give the impression
>>> that it
>>> > > is already assigned to somebody.
>>> > > This would not help to find external contributors who are interested
>>> in
>>> > > helping with a certain feature.
>>> > >
>>> > > Issues without estimates are still useful as they show in which
>>> direction
>>> > > the project plans to evolve.
>>> > >
>>> > > 2015-01-08 11:13 GMT+01:00 Stephan Ewen <[hidden email]>:
>>> > >
>>> > > > The Pig Journal is nice, we could have this evolve into something
>>> like
>>> > > > that.
>>> > > >
>>> > > > I think that we need to give some time estimate on the features /
>>> > issues.
>>> > > > Otherwise, it is of rather little value - all it says is that people
>>> > > > thought about that, no one knows when you can plan with it.
>>> > > >
>>> > > > We can coarsen the time estimates, though...
>>> > > >
>>> > > > Stephan
>>> > > >
>>> > > >
>>> > > > On Thu, Jan 8, 2015 at 10:57 AM, Robert Metzger <
>>> [hidden email]>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > I very much like the "PIG Journal" here:
>>> > > > > https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
>>> > > > > Its basically a nice view (however outdated in that case) on whats
>>> > > going
>>> > > > on
>>> > > > > in the PIG community. You can see finished features on the top,
>>> > current
>>> > > > > features being developed in the middle and ideas in the end.
>>> > > > >
>>> > > > > The document posted by Stephan is a good start to create a "Flink
>>> > > > Journal".
>>> > > > > I agree with Fabian that the estimates are very optimistic.
>>> > > Implementing
>>> > > > > all these features including unit tests, documentation and
>>> testing a
>>> > > > takes
>>> > > > > a lot of time.
>>> > > > > I would suggest to only add estimates (finish dates) to features
>>> > which
>>> > > > are
>>> > > > > currently work in progress.
>>> > > > > The remainder ("ideas") can have time estimates in months but
>>> should
>>> > > not
>>> > > > > have finish dates. Similarly to Pig, we should put a disclaimer on
>>> > top
>>> > > > that
>>> > > > > we do not guarantee for any feature being developed.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]
>>> >
>>> > > > wrote:
>>> > > > >
>>> > > > > > Hi,
>>> > > > > >
>>> > > > > > I think the roadmap should show the long-term development goals
>>> of
>>> > > > Flink,
>>> > > > > > i.e., show that we are going for a ML library, SQL support,
>>> > > > > Batch-Streaming
>>> > > > > > integration, etc.
>>> > > > > >
>>> > > > > > Right now, it is quite detailed and with very optimistic time
>>> > > > estimates,
>>> > > > > > IMO.
>>> > > > > > If we would do everything in time, we would be done with the
>>> > roadmap
>>> > > in
>>> > > > > Q3
>>> > > > > > 2015...
>>> > > > > > I would not even put a time on all issues, esp. on things which
>>> > > depend
>>> > > > on
>>> > > > > > other developments (which might not even have started). Also I
>>> > would
>>> > > > make
>>> > > > > > the estimates more coarse-grained. For short-term goals we could
>>> > use
>>> > > > > > quarters, everything does not need an estimate, IMO. Issues that
>>> > will
>>> > > > be
>>> > > > > > solved in two months don't even need to be listed.
>>> > > > > >
>>> > > > > >
>>> > > > > > 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]
>>> >:
>>> > > > > >
>>> > > > > > > I added some text about my work on the Logical Query feature.
>>> > > > > > >
>>> > > > > > > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <
>>> [hidden email]>
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Hi everyone!
>>> > > > > > > >
>>> > > > > > > > It is time we bring the Flink roadmap up to speed with what
>>> has
>>> > > > > > happened
>>> > > > > > > in
>>> > > > > > > > the last months and what further goals features ideas have
>>> come
>>> > > up.
>>> > > > > > > >
>>> > > > > > > > The link below leads to a Google Doc that contains an
>>> initial
>>> > set
>>> > > > of
>>> > > > > > > > suggestions that some of the committers have come up with.
>>> > Please
>>> > > > > share
>>> > > > > > > > your opinion on those suggestions and feel free to suggest
>>> > > > additional
>>> > > > > > > items
>>> > > > > > > > to put on the roadmap.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
>>> > > > > > > >
>>> > > > > > > > This is specifically open to everyone, not only committers.
>>> The
>>> > > > link
>>> > > > > > > should
>>> > > > > > > > allow everyone to add suggestions and comments to the doc
>>> (but
>>> > > not
>>> > > > to
>>> > > > > > > edit
>>> > > > > > > > it directly).
>>> > > > > > > >
>>> > > > > > > > For new suggestions, it would help a lot if you could also
>>> > > mention
>>> > > > > > > whether
>>> > > > > > > > you would be available to help out with that feature or
>>> idea -
>>> > > that
>>> > > > > > > helps a
>>> > > > > > > > lot with prioritizing and estimate the time line.
>>> > > > > > > >
>>> > > > > > > > For general suggestions to the "road mapping" process,
>>> please
>>> > > > respond
>>> > > > > > to
>>> > > > > > > > this mail.
>>> > > > > > > >
>>> > > > > > > > Greetings and happy drafting!
>>> > > > > > > > Stephan
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>

[Henry Saputra]

Hi All,

I am not sure about "Interactive Scala shell". I just feel like adding
feature like this may look like following Spark.

We could probably focus more on CLI and client libraries toward better
and scale backend execution engine compare to Spark.

- Henry

On Thu, Jan 8, 2015 at 1:57 AM, Robert Metzger <[hidden email]> wrote:

> Hi,
>
> I very much like the "PIG Journal" here:
> https://cwiki.apache.org/confluence/display/PIG/Pig+Journal
> Its basically a nice view (however outdated in that case) on whats going on
> in the PIG community. You can see finished features on the top, current
> features being developed in the middle and ideas in the end.
>
> The document posted by Stephan is a good start to create a "Flink Journal".
> I agree with Fabian that the estimates are very optimistic. Implementing
> all these features including unit tests, documentation and testing a takes
> a lot of time.
> I would suggest to only add estimates (finish dates) to features which are
> currently work in progress.
> The remainder ("ideas") can have time estimates in months but should not
> have finish dates. Similarly to Pig, we should put a disclaimer on top that
> we do not guarantee for any feature being developed.
>
>
>
>
> On Thu, Jan 8, 2015 at 9:59 AM, Fabian Hueske <[hidden email]> wrote:
>
>> Hi,
>>
>> I think the roadmap should show the long-term development goals of Flink,
>> i.e., show that we are going for a ML library, SQL support, Batch-Streaming
>> integration, etc.
>>
>> Right now, it is quite detailed and with very optimistic time estimates,
>> IMO.
>> If we would do everything in time, we would be done with the roadmap in  Q3
>> 2015...
>> I would not even put a time on all issues, esp. on things which depend on
>> other developments (which might not even have started). Also I would make
>> the estimates more coarse-grained. For short-term goals we could use
>> quarters, everything does not need an estimate, IMO. Issues that will be
>> solved in two months don't even need to be listed.
>>
>>
>> 2015-01-08 7:50 GMT+01:00 Aljoscha Krettek <[hidden email]>:
>>
>> > I added some text about my work on the Logical Query feature.
>> >
>> > On Thu, Jan 8, 2015 at 12:42 AM, Stephan Ewen <[hidden email]> wrote:
>> >
>> > > Hi everyone!
>> > >
>> > > It is time we bring the Flink roadmap up to speed with what has
>> happened
>> > in
>> > > the last months and what further goals features ideas have come up.
>> > >
>> > > The link below leads to a Google Doc that contains an initial set of
>> > > suggestions that some of the committers have come up with. Please share
>> > > your opinion on those suggestions and feel free to suggest additional
>> > items
>> > > to put on the roadmap.
>> > >
>> > >
>> > >
>> >
>> https://docs.google.com/document/d/1QZ0NJC03pOBqE6vbK1Ot4bXwoBcszIqzbZ8a6B5vSEo/edit?usp=sharing
>> > >
>> > > This is specifically open to everyone, not only committers. The link
>> > should
>> > > allow everyone to add suggestions and comments to the doc (but not to
>> > edit
>> > > it directly).
>> > >
>> > > For new suggestions, it would help a lot if you could also mention
>> > whether
>> > > you would be available to help out with that feature or idea - that
>> > helps a
>> > > lot with prioritizing and estimate the time line.
>> > >
>> > > For general suggestions to the "road mapping" process, please respond
>> to
>> > > this mail.
>> > >
>> > > Greetings and happy drafting!
>> > > Stephan
>> > >
>> >
>>

[Stephan Ewen]

I agree, Henry. To me, this is a low priority feature and not one in the
main line of what I see as the core Flink direction.

I included it in there, because it is actually a fairly low hanging
fruit. Once the stuff for the Zeppelin Integration is in place, it is a
neat project for someone (external) who likes a bit if Scala hacking.

As such, we may actually move it to the "interesting projects on Flink"
section of the Wiki, rather than have it in the roadmap.

[Kostas Tzoumas-2]

Yeah, this is something that nobody is working on AFAIK and not a major
focus of the project. I moved it to the interesting projects page (and
cleaned up the page a bit removing streaming, tez, and mahout as these are
ongoing efforts).

On Wed, Feb 4, 2015 at 9:34 AM, Stephan Ewen <[hidden email]> wrote:

> I agree, Henry. To me, this is a low priority feature and not one in the
> main line of what I see as the core Flink direction.
>
> I included it in there, because it is actually a fairly low hanging
> fruit. Once the stuff for the Zeppelin Integration is in place, it is a
> neat project for someone (external) who likes a bit if Scala hacking.
>
> As such, we may actually move it to the "interesting projects on Flink"
> section of the Wiki, rather than have it in the roadmap.
>

[Henry Saputra]

Thanks Stephan, Kostas, sounds good to me

On Wed, Feb 4, 2015 at 2:57 AM, Kostas Tzoumas <[hidden email]> wrote:

> Yeah, this is something that nobody is working on AFAIK and not a major
> focus of the project. I moved it to the interesting projects page (and
> cleaned up the page a bit removing streaming, tez, and mahout as these are
> ongoing efforts).
>
> On Wed, Feb 4, 2015 at 9:34 AM, Stephan Ewen <[hidden email]> wrote:
>
>> I agree, Henry. To me, this is a low priority feature and not one in the
>> main line of what I see as the core Flink direction.
>>
>> I included it in there, because it is actually a fairly low hanging
>> fruit. Once the stuff for the Zeppelin Integration is in place, it is a
>> neat project for someone (external) who likes a bit if Scala hacking.
>>
>> As such, we may actually move it to the "interesting projects on Flink"
>> section of the Wiki, rather than have it in the roadmap.
>>

[Dmitriy Lyubimov]

fwiw re: shell, this is just scala being incredibly useful. If anything,
spark is following scala. So is for example BIDMat/BIDMach (and, sigh*
mahout). I don't think differentiation means throwing away common baseline
tools, there's gotta be more than that. (I'm of course advocating using
shell in Mahout. flink having shell would mean online computations on par
with mahout's shell for spark).



On Wed, Feb 4, 2015 at 12:34 AM, Stephan Ewen <[hidden email]> wrote:

> I agree, Henry. To me, this is a low priority feature and not one in the
> main line of what I see as the core Flink direction.
>
> I included it in there, because it is actually a fairly low hanging
> fruit. Once the stuff for the Zeppelin Integration is in place, it is a
> neat project for someone (external) who likes a bit if Scala hacking.
>
> As such, we may actually move it to the "interesting projects on Flink"
> section of the Wiki, rather than have it in the roadmap.
>