[DISCUSS] Move license check utility to a new repository to share it with flink-statefun

[Robert Metzger]

Hi all,

for the upcoming 1.12 release (and beyond ;) ), I added a utility [1] that
checks for the most common licensing issues. It doesn't find everything,
but the by far most common case of forgetting to add a version upgrade or
changed transitive dependency to the NOTICE file is covered.

Chesnay had the idea of creating a new repository for the tool and sharing
it with flink-statefun to do the license checks automatically there as well.


1. Is this relevant for flink-statefun?

2. For the repository name, what do you think about "flink-project-utils" ?
I'd like to use a generic name so that we can potentially share other
internal utilities.


Let me know what you think.

Robert


[1] https://github.com/apache/flink/blob/master/tools/ci/license_check.sh

[Tzu-Li (Gordon) Tai]

Hi Robert,

I think this could be useful in flink-statefun.

StateFun currently has two modules that bundles dependencies, most
importantly the `flink-statefun-distribution` module which currently
bundles some Flink dependencies as well as Flink connectors (Kafka,
Kinesis).
Upgrading the Flink version in StateFun typically involves a bulk update on
the NOTICE of that module, so some automatic validation in CI could help
with that.
The other module that bundles dependencies is the StateFun examples, which
we've been thinking about stopping to release Maven artifacts for.

On Thu, Nov 5, 2020 at 9:54 PM Robert Metzger <[hidden email]> wrote:

> 1. Is this relevant for flink-statefun?
>

So, really there is only one module that would benefit from this tool
(which could possibly be enough already for sharing to make sense).
To justify the efforts for sharing this nice utility, I'd like to have a
better idea of: how do you intend downstream CIs in flink / flink-statefun
to be using this? Would there be released artifacts from
`flink-project-utils` to expose each tool (e.g. the `LicenseChecker`)?
It almost looks as if it would be easiest to reuse this tool if it was
provided as a Maven plugin, though I'm not sure how possible that is and
probably out-of-scope for this discussion.


>
> 2. For the repository name, what do you think about "flink-project-utils" ?
> I'd like to use a generic name so that we can potentially share other
> internal utilities.
>

I like the idea of sharing internal utilities in general across the two
repos.

To gauge how useful this repo would be in the end, here's some info on what
StateFun has copied already to flink-statefun:

   - About to copy checking for dead links in docs [1]
   - Several release-related scripts for creating source bundles, deploying
   staging jars, updating branch version, etc. [2]. However, these have
   somewhat evolved in StateFun to tailor it for flink-statefun, so I guess it
   would not make sense to share release related tooling.
   - Utility around building the documentation (currently flink and
   flink-statefun share the same Jekyll setup).

Best,
Gordon

[1] https://github.com/apache/flink-statefun/pull/171
[2] https://github.com/apache/flink-statefun/tree/master/tools/releasing

[Robert Metzger]

Thanks a lot for your response, and sorry for my late response.

how do you intend downstream CIs in flink / flink-statefun to be using
> this?
> Would there be released artifacts from `flink-project-utils` to expose
> each tool (e.g. the `LicenseChecker`)?


My idea was that flink and flink-statefun would just clone the
flink-project-utils repository, build the tool locally and run it, without
releasing the artifacts anywhere.
The build takes just a few seconds, and the maintenance overhead is a
little lower with this approach.

I also considered building a maven plugin, but I guess forking the
maven-shade-plugin is the only feasible option. Adding a custom
maven-shade-transformer is not feasible, since we won't have access to the
required data.
All this sounded too complicated, so I discarded the idea.


Thanks a lot for the list of copied utilities. I guess if we are adding
more sister projects to Flink, or splitting the main repo in the future, we
need to look into coming up with a properly written set of shared release
utilities.

I have the feeling that currently the effort for creating a separate
repository is not justified for just checking one module in flink-statefun.
Let's revisit this in the future.


On Fri, Nov 6, 2020 at 4:28 AM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> Hi Robert,
>
> I think this could be useful in flink-statefun.
>
> StateFun currently has two modules that bundles dependencies, most
> importantly the `flink-statefun-distribution` module which currently
> bundles some Flink dependencies as well as Flink connectors (Kafka,
> Kinesis).
> Upgrading the Flink version in StateFun typically involves a bulk update on
> the NOTICE of that module, so some automatic validation in CI could help
> with that.
> The other module that bundles dependencies is the StateFun examples, which
> we've been thinking about stopping to release Maven artifacts for.
>
> On Thu, Nov 5, 2020 at 9:54 PM Robert Metzger <[hidden email]> wrote:
>
> > 1. Is this relevant for flink-statefun?
> >
>
> So, really there is only one module that would benefit from this tool
> (which could possibly be enough already for sharing to make sense).
> To justify the efforts for sharing this nice utility, I'd like to have a
> better idea of: how do you intend downstream CIs in flink / flink-statefun
> to be using this? Would there be released artifacts from
> `flink-project-utils` to expose each tool (e.g. the `LicenseChecker`)?
> It almost looks as if it would be easiest to reuse this tool if it was
> provided as a Maven plugin, though I'm not sure how possible that is and
> probably out-of-scope for this discussion.
>
>
> >
> > 2. For the repository name, what do you think about
> "flink-project-utils" ?
> > I'd like to use a generic name so that we can potentially share other
> > internal utilities.
> >
>
> I like the idea of sharing internal utilities in general across the two
> repos.
>
> To gauge how useful this repo would be in the end, here's some info on what
> StateFun has copied already to flink-statefun:
>
>    - About to copy checking for dead links in docs [1]
>    - Several release-related scripts for creating source bundles, deploying
>    staging jars, updating branch version, etc. [2]. However, these have
>    somewhat evolved in StateFun to tailor it for flink-statefun, so I
> guess it
>    would not make sense to share release related tooling.
>    - Utility around building the documentation (currently flink and
>    flink-statefun share the same Jekyll setup).
>
> Best,
> Gordon
>
> [1] https://github.com/apache/flink-statefun/pull/171
> [2] https://github.com/apache/flink-statefun/tree/master/tools/releasing
>