[DISCUSS] Extend (or maintain) "shell" script support for Windows

[Aljoscha Krettek-2]

Hi,

the background is this series of Jira Issues and PRs around extending
the .bat scripts for windows:
https://issues.apache.org/jira/browse/FLINK-5333.

I would like to resolve this, by either closing the Jira Issues as
"Won't Do" or finally merging these PRs. The questions I have are:

  - Should we add more full-featured (complicated?) windows scripts that
are essentially re-implementations of our existing "unix" scripts?
  - Would windows users use these or would they, by now, use the linux
subsystem for windows or cygwin?
  - Should we even remove the existing .bat scripts that we have?

Maintaining the windows scripts is hard because we only have one (I
think, Chesnay) developer on windows and no CI for windows.

What do you think?

Best,
Aljoscha

[Chesnay Schepler-3]

Note that with FLINK-15925 we effectively broke the start-cluster.bat
script in 1.10.

Overall, I would be fine with dropping the .bat scripts because they are
a big maintenance problem for us.
We don't have anyone using these scripts regularly on our end (even I
don't).
If there were a way to unify the unix/windows scripts I would be all
ears, but this doesn't seem possible without lots and lots of branches.

However, we should definitely ensure that Flink continues to work in the
IDE on windows for training/demo purposes.

As for CI, with the new Azure setup this may actually be feasible now
from a CI budged perspective. But it would require a fair upfront time
investment to fix failing tests or adding assumptions for failing tests.
That said, this would have some nice benefits for finding resource
leaks, particularly in the filesystem.

On 19/02/2020 16:46, Aljoscha Krettek wrote:

> Hi,
>
> the background is this series of Jira Issues and PRs around extending
> the .bat scripts for windows:
> https://issues.apache.org/jira/browse/FLINK-5333.
>
> I would like to resolve this, by either closing the Jira Issues as
> "Won't Do" or finally merging these PRs. The questions I have are:
>
>  - Should we add more full-featured (complicated?) windows scripts
> that are essentially re-implementations of our existing "unix" scripts?
>  - Would windows users use these or would they, by now, use the linux
> subsystem for windows or cygwin?
>  - Should we even remove the existing .bat scripts that we have?
>
> Maintaining the windows scripts is hard because we only have one (I
> think, Chesnay) developer on windows and no CI for windows.
>
> What do you think?
>
> Best,
> Aljoscha
>

[Yang Wang]

I tend to drop the windows bat script to start/stop the cluster. Since there
are various windows versions(e.g. xp, win7, win10, etc.) and do not have
sensational compatibility. Also we do not have the e2e tests to guarantee
the scripts work well.

On the other hand, i completely agree that local deployment, especially in
the IDE, should work on windows. For standalone deployment, i think using
the docker is a good choice. I think many windows users are also using
docker. For Yarn/K8s deployment, i am not sure whether we need to support
to submit a Flink session/per-job cluster from windows machine.


Best,
Yang

Chesnay Schepler <[hidden email]> 于2020年2月20日周四 上午12:15写道：

> Note that with FLINK-15925 we effectively broke the start-cluster.bat
> script in 1.10.
>
> Overall, I would be fine with dropping the .bat scripts because they are
> a big maintenance problem for us.
> We don't have anyone using these scripts regularly on our end (even I
> don't).
> If there were a way to unify the unix/windows scripts I would be all
> ears, but this doesn't seem possible without lots and lots of branches.
>
> However, we should definitely ensure that Flink continues to work in the
> IDE on windows for training/demo purposes.
>
> As for CI, with the new Azure setup this may actually be feasible now
> from a CI budged perspective. But it would require a fair upfront time
> investment to fix failing tests or adding assumptions for failing tests.
> That said, this would have some nice benefits for finding resource
> leaks, particularly in the filesystem.
>
> On 19/02/2020 16:46, Aljoscha Krettek wrote:
> > Hi,
> >
> > the background is this series of Jira Issues and PRs around extending
> > the .bat scripts for windows:
> > https://issues.apache.org/jira/browse/FLINK-5333.
> >
> > I would like to resolve this, by either closing the Jira Issues as
> > "Won't Do" or finally merging these PRs. The questions I have are:
> >
> >  - Should we add more full-featured (complicated?) windows scripts
> > that are essentially re-implementations of our existing "unix" scripts?
> >  - Would windows users use these or would they, by now, use the linux
> > subsystem for windows or cygwin?
> >  - Should we even remove the existing .bat scripts that we have?
> >
> > Maintaining the windows scripts is hard because we only have one (I
> > think, Chesnay) developer on windows and no CI for windows.
> >
> > What do you think?
> >
> > Best,
> > Aljoscha
> >
>
>

[Xintong Song]

I'm also in favor of dropping the bat scripts. I can hardly image
people using Flink Windows deployment in their production. But maybe
throwing a survey on the user ML to double check on that?

Thank you~

Xintong Song



On Thu, Feb 20, 2020 at 10:28 AM Yang Wang <[hidden email]> wrote:

> I tend to drop the windows bat script to start/stop the cluster. Since
> there
> are various windows versions(e.g. xp, win7, win10, etc.) and do not have
> sensational compatibility. Also we do not have the e2e tests to guarantee
> the scripts work well.
>
> On the other hand, i completely agree that local deployment, especially in
> the IDE, should work on windows. For standalone deployment, i think using
> the docker is a good choice. I think many windows users are also using
> docker. For Yarn/K8s deployment, i am not sure whether we need to support
> to submit a Flink session/per-job cluster from windows machine.
>
>
> Best,
> Yang
>
> Chesnay Schepler <[hidden email]> 于2020年2月20日周四 上午12:15写道：
>
> > Note that with FLINK-15925 we effectively broke the start-cluster.bat
> > script in 1.10.
> >
> > Overall, I would be fine with dropping the .bat scripts because they are
> > a big maintenance problem for us.
> > We don't have anyone using these scripts regularly on our end (even I
> > don't).
> > If there were a way to unify the unix/windows scripts I would be all
> > ears, but this doesn't seem possible without lots and lots of branches.
> >
> > However, we should definitely ensure that Flink continues to work in the
> > IDE on windows for training/demo purposes.
> >
> > As for CI, with the new Azure setup this may actually be feasible now
> > from a CI budged perspective. But it would require a fair upfront time
> > investment to fix failing tests or adding assumptions for failing tests.
> > That said, this would have some nice benefits for finding resource
> > leaks, particularly in the filesystem.
> >
> > On 19/02/2020 16:46, Aljoscha Krettek wrote:
> > > Hi,
> > >
> > > the background is this series of Jira Issues and PRs around extending
> > > the .bat scripts for windows:
> > > https://issues.apache.org/jira/browse/FLINK-5333.
> > >
> > > I would like to resolve this, by either closing the Jira Issues as
> > > "Won't Do" or finally merging these PRs. The questions I have are:
> > >
> > >  - Should we add more full-featured (complicated?) windows scripts
> > > that are essentially re-implementations of our existing "unix" scripts?
> > >  - Would windows users use these or would they, by now, use the linux
> > > subsystem for windows or cygwin?
> > >  - Should we even remove the existing .bat scripts that we have?
> > >
> > > Maintaining the windows scripts is hard because we only have one (I
> > > think, Chesnay) developer on windows and no CI for windows.
> > >
> > > What do you think?
> > >
> > > Best,
> > > Aljoscha
> > >
> >
> >
>

[Chesnay Schepler-3]

Realistically you can't really use the windows scripts in production
since you can only start a standalone cluster consisting of 1 JM and 1 TM.

The .bat scripts are already limited to a significant degree.

On 20/02/2020 03:48, Xintong Song wrote:

> I'm also in favor of dropping the bat scripts. I can hardly image
> people using Flink Windows deployment in their production. But maybe
> throwing a survey on the user ML to double check on that?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Feb 20, 2020 at 10:28 AM Yang Wang <[hidden email]> wrote:
>
>> I tend to drop the windows bat script to start/stop the cluster. Since
>> there
>> are various windows versions(e.g. xp, win7, win10, etc.) and do not have
>> sensational compatibility. Also we do not have the e2e tests to guarantee
>> the scripts work well.
>>
>> On the other hand, i completely agree that local deployment, especially in
>> the IDE, should work on windows. For standalone deployment, i think using
>> the docker is a good choice. I think many windows users are also using
>> docker. For Yarn/K8s deployment, i am not sure whether we need to support
>> to submit a Flink session/per-job cluster from windows machine.
>>
>>
>> Best,
>> Yang
>>
>> Chesnay Schepler <[hidden email]> 于2020年2月20日周四 上午12:15写道：
>>
>>> Note that with FLINK-15925 we effectively broke the start-cluster.bat
>>> script in 1.10.
>>>
>>> Overall, I would be fine with dropping the .bat scripts because they are
>>> a big maintenance problem for us.
>>> We don't have anyone using these scripts regularly on our end (even I
>>> don't).
>>> If there were a way to unify the unix/windows scripts I would be all
>>> ears, but this doesn't seem possible without lots and lots of branches.
>>>
>>> However, we should definitely ensure that Flink continues to work in the
>>> IDE on windows for training/demo purposes.
>>>
>>> As for CI, with the new Azure setup this may actually be feasible now
>>> from a CI budged perspective. But it would require a fair upfront time
>>> investment to fix failing tests or adding assumptions for failing tests.
>>> That said, this would have some nice benefits for finding resource
>>> leaks, particularly in the filesystem.
>>>
>>> On 19/02/2020 16:46, Aljoscha Krettek wrote:
>>>> Hi,
>>>>
>>>> the background is this series of Jira Issues and PRs around extending
>>>> the .bat scripts for windows:
>>>> https://issues.apache.org/jira/browse/FLINK-5333.
>>>>
>>>> I would like to resolve this, by either closing the Jira Issues as
>>>> "Won't Do" or finally merging these PRs. The questions I have are:
>>>>
>>>>   - Should we add more full-featured (complicated?) windows scripts
>>>> that are essentially re-implementations of our existing "unix" scripts?
>>>>   - Would windows users use these or would they, by now, use the linux
>>>> subsystem for windows or cygwin?
>>>>   - Should we even remove the existing .bat scripts that we have?
>>>>
>>>> Maintaining the windows scripts is hard because we only have one (I
>>>> think, Chesnay) developer on windows and no CI for windows.
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>

[Stephan Ewen]

I think this question should go on the user mailing list, not on dev.

You can also use Twitter to collect some data points.

On Thu, Feb 20, 2020 at 11:07 AM Chesnay Schepler <[hidden email]>
wrote:

> Realistically you can't really use the windows scripts in production
> since you can only start a standalone cluster consisting of 1 JM and 1 TM.
>
> The .bat scripts are already limited to a significant degree.
>
> On 20/02/2020 03:48, Xintong Song wrote:
> > I'm also in favor of dropping the bat scripts. I can hardly image
> > people using Flink Windows deployment in their production. But maybe
> > throwing a survey on the user ML to double check on that?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Feb 20, 2020 at 10:28 AM Yang Wang <[hidden email]>
> wrote:
> >
> >> I tend to drop the windows bat script to start/stop the cluster. Since
> >> there
> >> are various windows versions(e.g. xp, win7, win10, etc.) and do not have
> >> sensational compatibility. Also we do not have the e2e tests to
> guarantee
> >> the scripts work well.
> >>
> >> On the other hand, i completely agree that local deployment, especially
> in
> >> the IDE, should work on windows. For standalone deployment, i think
> using
> >> the docker is a good choice. I think many windows users are also using
> >> docker. For Yarn/K8s deployment, i am not sure whether we need to
> support
> >> to submit a Flink session/per-job cluster from windows machine.
> >>
> >>
> >> Best,
> >> Yang
> >>
> >> Chesnay Schepler <[hidden email]> 于2020年2月20日周四 上午12:15写道：
> >>
> >>> Note that with FLINK-15925 we effectively broke the start-cluster.bat
> >>> script in 1.10.
> >>>
> >>> Overall, I would be fine with dropping the .bat scripts because they
> are
> >>> a big maintenance problem for us.
> >>> We don't have anyone using these scripts regularly on our end (even I
> >>> don't).
> >>> If there were a way to unify the unix/windows scripts I would be all
> >>> ears, but this doesn't seem possible without lots and lots of branches.
> >>>
> >>> However, we should definitely ensure that Flink continues to work in
> the
> >>> IDE on windows for training/demo purposes.
> >>>
> >>> As for CI, with the new Azure setup this may actually be feasible now
> >>> from a CI budged perspective. But it would require a fair upfront time
> >>> investment to fix failing tests or adding assumptions for failing
> tests.
> >>> That said, this would have some nice benefits for finding resource
> >>> leaks, particularly in the filesystem.
> >>>
> >>> On 19/02/2020 16:46, Aljoscha Krettek wrote:
> >>>> Hi,
> >>>>
> >>>> the background is this series of Jira Issues and PRs around extending
> >>>> the .bat scripts for windows:
> >>>> https://issues.apache.org/jira/browse/FLINK-5333.
> >>>>
> >>>> I would like to resolve this, by either closing the Jira Issues as
> >>>> "Won't Do" or finally merging these PRs. The questions I have are:
> >>>>
> >>>>   - Should we add more full-featured (complicated?) windows scripts
> >>>> that are essentially re-implementations of our existing "unix"
> scripts?
> >>>>   - Would windows users use these or would they, by now, use the linux
> >>>> subsystem for windows or cygwin?
> >>>>   - Should we even remove the existing .bat scripts that we have?
> >>>>
> >>>> Maintaining the windows scripts is hard because we only have one (I
> >>>> think, Chesnay) developer on windows and no CI for windows.
> >>>>
> >>>> What do you think?
> >>>>
> >>>> Best,
> >>>> Aljoscha
> >>>>
> >>>
>
>

[David Anderson-2]

We stopped using these windows scripts in our training sessions a couple of
years ago, and I can't remember seeing anyone use them at all since then.
While our community includes many developers using Windows for development,
in my experience pretty much everyone moves to some sort of linux-based
environment once they are done testing in the IDE, be that via WSL, Docker,
a local VM, or a staging server.

David

On Thu, Feb 20, 2020 at 11:09 AM Stephan Ewen <[hidden email]> wrote:

> I think this question should go on the user mailing list, not on dev.
>
> You can also use Twitter to collect some data points.
>
> On Thu, Feb 20, 2020 at 11:07 AM Chesnay Schepler <[hidden email]>
> wrote:
>
> > Realistically you can't really use the windows scripts in production
> > since you can only start a standalone cluster consisting of 1 JM and 1
> TM.
> >
> > The .bat scripts are already limited to a significant degree.
> >
> > On 20/02/2020 03:48, Xintong Song wrote:
> > > I'm also in favor of dropping the bat scripts. I can hardly image
> > > people using Flink Windows deployment in their production. But maybe
> > > throwing a survey on the user ML to double check on that?
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Feb 20, 2020 at 10:28 AM Yang Wang <[hidden email]>
> > wrote:
> > >
> > >> I tend to drop the windows bat script to start/stop the cluster. Since
> > >> there
> > >> are various windows versions(e.g. xp, win7, win10, etc.) and do not
> have
> > >> sensational compatibility. Also we do not have the e2e tests to
> > guarantee
> > >> the scripts work well.
> > >>
> > >> On the other hand, i completely agree that local deployment,
> especially
> > in
> > >> the IDE, should work on windows. For standalone deployment, i think
> > using
> > >> the docker is a good choice. I think many windows users are also using
> > >> docker. For Yarn/K8s deployment, i am not sure whether we need to
> > support
> > >> to submit a Flink session/per-job cluster from windows machine.
> > >>
> > >>
> > >> Best,
> > >> Yang
> > >>
> > >> Chesnay Schepler <[hidden email]> 于2020年2月20日周四 上午12:15写道：
> > >>
> > >>> Note that with FLINK-15925 we effectively broke the start-cluster.bat
> > >>> script in 1.10.
> > >>>
> > >>> Overall, I would be fine with dropping the .bat scripts because they
> > are
> > >>> a big maintenance problem for us.
> > >>> We don't have anyone using these scripts regularly on our end (even I
> > >>> don't).
> > >>> If there were a way to unify the unix/windows scripts I would be all
> > >>> ears, but this doesn't seem possible without lots and lots of
> branches.
> > >>>
> > >>> However, we should definitely ensure that Flink continues to work in
> > the
> > >>> IDE on windows for training/demo purposes.
> > >>>
> > >>> As for CI, with the new Azure setup this may actually be feasible now
> > >>> from a CI budged perspective. But it would require a fair upfront
> time
> > >>> investment to fix failing tests or adding assumptions for failing
> > tests.
> > >>> That said, this would have some nice benefits for finding resource
> > >>> leaks, particularly in the filesystem.
> > >>>
> > >>> On 19/02/2020 16:46, Aljoscha Krettek wrote:
> > >>>> Hi,
> > >>>>
> > >>>> the background is this series of Jira Issues and PRs around
> extending
> > >>>> the .bat scripts for windows:
> > >>>> https://issues.apache.org/jira/browse/FLINK-5333.
> > >>>>
> > >>>> I would like to resolve this, by either closing the Jira Issues as
> > >>>> "Won't Do" or finally merging these PRs. The questions I have are:
> > >>>>
> > >>>>   - Should we add more full-featured (complicated?) windows scripts
> > >>>> that are essentially re-implementations of our existing "unix"
> > scripts?
> > >>>>   - Would windows users use these or would they, by now, use the
> linux
> > >>>> subsystem for windows or cygwin?
> > >>>>   - Should we even remove the existing .bat scripts that we have?
> > >>>>
> > >>>> Maintaining the windows scripts is hard because we only have one (I
> > >>>> think, Chesnay) developer on windows and no CI for windows.
> > >>>>
> > >>>> What do you think?
> > >>>>
> > >>>> Best,
> > >>>> Aljoscha
> > >>>>
> > >>>
> >
> >
>

[Aljoscha Krettek-2]

Since there was no-one that said we should keep the windows scripts and
no-one responded on the user ML thread I'll close the Jira issues/PRs
about extending the scripts.

Aljoscha

[Robert Metzger]

I'm wondering whether we should file a ticket to remove the *.bat files in
bin/ ?

On Tue, Mar 10, 2020 at 12:46 PM Aljoscha Krettek <[hidden email]>
wrote:

> Since there was no-one that said we should keep the windows scripts and
> no-one responded on the user ML thread I'll close the Jira issues/PRs
> about extending the scripts.
>
> Aljoscha
>

[Aljoscha Krettek-2]

On 10.03.20 14:35, Robert Metzger wrote:
> I'm wondering whether we should file a ticket to remove the *.bat files in
> bin/ ?

We can leave them there because they're not doing much harm, and
removing them might actively break some existing setup.

Best,
Aljoscha

[Xintong Song]

Thanks for the survey, Aljoscha. I'll also close FLINK-15925.

I second Robert's concern. It sounds a bit weird that we keep something in
our project that does not work and is not maintained. Might also cause
confusions for future new comers who may not be aware of our conclusion
today. Maybe we should remove them? Like what we do for other legacy codes.

Thank you~

Xintong Song



On Tue, Mar 10, 2020 at 10:01 PM Aljoscha Krettek <[hidden email]>
wrote:

> On 10.03.20 14:35, Robert Metzger wrote:
> > I'm wondering whether we should file a ticket to remove the *.bat files
> in
> > bin/ ?
>
> We can leave them there because they're not doing much harm, and
> removing them might actively break some existing setup.
>
> Best,
> Aljoscha
>
>

[Chesnay Schepler-3]

+1 on removing them.

They are so limited in terms of functionality that I doubt anyone would
be significantly impaired by us removing them.

On 11/03/2020 02:13, Xintong Song wrote:

> Thanks for the survey, Aljoscha. I'll also close FLINK-15925.
>
> I second Robert's concern. It sounds a bit weird that we keep something in
> our project that does not work and is not maintained. Might also cause
> confusions for future new comers who may not be aware of our conclusion
> today. Maybe we should remove them? Like what we do for other legacy codes.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Mar 10, 2020 at 10:01 PM Aljoscha Krettek <[hidden email]>
> wrote:
>
>> On 10.03.20 14:35, Robert Metzger wrote:
>>> I'm wondering whether we should file a ticket to remove the *.bat files
>> in
>>> bin/ ?
>> We can leave them there because they're not doing much harm, and
>> removing them might actively break some existing setup.
>>
>> Best,
>> Aljoscha
>>
>>

[Till Rohrmann]

+1 for removing them.

On Wed, Mar 11, 2020 at 10:06 AM Chesnay Schepler <[hidden email]>
wrote:

> +1 on removing them.
>
> They are so limited in terms of functionality that I doubt anyone would
> be significantly impaired by us removing them.
>
> On 11/03/2020 02:13, Xintong Song wrote:
> > Thanks for the survey, Aljoscha. I'll also close FLINK-15925.
> >
> > I second Robert's concern. It sounds a bit weird that we keep something
> in
> > our project that does not work and is not maintained. Might also cause
> > confusions for future new comers who may not be aware of our conclusion
> > today. Maybe we should remove them? Like what we do for other legacy
> codes.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Mar 10, 2020 at 10:01 PM Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> >> On 10.03.20 14:35, Robert Metzger wrote:
> >>> I'm wondering whether we should file a ticket to remove the *.bat files
> >> in
> >>> bin/ ?
> >> We can leave them there because they're not doing much harm, and
> >> removing them might actively break some existing setup.
> >>
> >> Best,
> >> Aljoscha
> >>
> >>
>
>

[Robert Metzger]

Thank you for your opinions. I filed a ticket for this:
https://issues.apache.org/jira/browse/FLINK-16683

On Tue, Mar 17, 2020 at 3:34 PM Till Rohrmann <[hidden email]> wrote:

> +1 for removing them.
>
> On Wed, Mar 11, 2020 at 10:06 AM Chesnay Schepler <[hidden email]>
> wrote:
>
> > +1 on removing them.
> >
> > They are so limited in terms of functionality that I doubt anyone would
> > be significantly impaired by us removing them.
> >
> > On 11/03/2020 02:13, Xintong Song wrote:
> > > Thanks for the survey, Aljoscha. I'll also close FLINK-15925.
> > >
> > > I second Robert's concern. It sounds a bit weird that we keep something
> > in
> > > our project that does not work and is not maintained. Might also cause
> > > confusions for future new comers who may not be aware of our conclusion
> > > today. Maybe we should remove them? Like what we do for other legacy
> > codes.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Tue, Mar 10, 2020 at 10:01 PM Aljoscha Krettek <[hidden email]
> >
> > > wrote:
> > >
> > >> On 10.03.20 14:35, Robert Metzger wrote:
> > >>> I'm wondering whether we should file a ticket to remove the *.bat
> files
> > >> in
> > >>> bin/ ?
> > >> We can leave them there because they're not doing much harm, and
> > >> removing them might actively break some existing setup.
> > >>
> > >> Best,
> > >> Aljoscha
> > >>
> > >>
> >
> >
>