(DEPRECATED) Apache Flink Mailing List archive.

[VOTE] Release 1.11.0, release candidate #4

Classic

List

Threaded

57 messages Options

123

dwysakowicz

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (binding)

Checks:

- built from sources

-verified signatures & no binaries in the source archive

- run all tests locally (mvn clean install)

    here I had a couple of problems:

        * https://issues.apache.org/jira/browse/FLINK-18476

        * https://issues.apache.org/jira/browse/FLINK-18470

        * UnsignedTypeConversionITCase failed because it requires
libncursed5 installed

None of the issues should be blockers imo as all three fail because
the tests assume certain configuration of the environment.

- started local cluster & run a couple of table examples

    the ChangelogSocketExample did not work for me. I think it would
make sense to only bundle examples that work out of the box in the dist.
Nevertheless as it is a new example in the release and it is only an
example I would not the release because of it.
(https://issues.apache.org/jira/browse/FLINK-18477)

- started sql-client and run a few very simple queries

- verified a couple of license files:

    Here I have more of a question. If we bundle an artifact with a
classifier. Shall we include the classifier as part of the entry in
LICENSE file?

    We bundle org.apache.orc:orc-core:jar:nohive:1.4.3 in
flink-sql-connector-hive-1.2.2, but in the LICENSE file we list it
without the nohive classifier.

Side note. We do bundle some python files as part of the distribution. I
have not seen anyone trying that out in the thread so far. Shall we ask
somebody more familiar with the python module to check that?

Best,

Dawid

On 02/07/2020 20:37, Kostas Kloudas wrote:

> Hi all,
>
> As far as the issue that Chesnay mentioned that leads to a "Caused by:
> org.apache.flink.api.common.InvalidProgramException:" for DataSet
> examples with print() collect() or count() as sink, this was a
> semi-intensional side-effect of the application mode. Before, in these
> cases, the output was simply ignored. Now we have the same behavior as
> in the "detached" mode. I already opened a PR for the release notes
> (sorry for not doing it earlier although this was a known change in
> behavior, as mentioned it in the PR here
> https://github.com/apache/flink/pull/11460 ) and I will merge it
> today.
>
> Cheers,
> Kostas
>
> On Thu, Jul 2, 2020 at 8:07 PM Robert Metzger <[hidden email]> wrote:
>> +1 (binding)
>>
>> Checks:
>> - source archive compiles
>> - checked artifacts in staging repo
>> - flink-azure-fs-hadoop-1.11.0.jar seems to have a correct NOTICE file
>> - versions in pom seem correct
>> - checked some other jars
>> - deployed Flink on YARN on Azure HDInsight (which uses Hadoop 3.1.1)
>> - Reported some tiny log sanity issue:
>> https://issues.apache.org/jira/browse/FLINK-18474
>> - Wordcount against HDFS works
>>
>>
>> On Thu, Jul 2, 2020 at 7:07 PM Thomas Weise <[hidden email]> wrote:
>>
>>> Hi Zhijiang,
>>>
>>> The performance degradation manifests in backpressure which leads to
>>> growing backlog in the source. I switched a few times between 1.10 and 1.11
>>> and the behavior is consistent.
>>>
>>> The DAG is:
>>>
>>> KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
>>> ---------> KinesisProducer
>>>
>>> Parallelism: 160
>>> No shuffle/rebalance.
>>>
>>> Checkpointing config:
>>>
>>> Checkpointing Mode Exactly Once
>>> Interval 10s
>>> Timeout 10m 0s
>>> Minimum Pause Between Checkpoints 10s
>>> Maximum Concurrent Checkpoints 1
>>> Persist Checkpoints Externally Enabled (delete on cancellation)
>>>
>>> State backend: rocksdb (filesystem leads to same symptoms)
>>> Checkpoint size is tiny (500KB)
>>>
>>> An interesting difference to another job that I had upgraded successfully
>>> is the low checkpointing interval.
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
>>> .invalid>
>>> wrote:
>>>
>>>> Hi Thomas,
>>>>
>>>> Thanks for the efficient feedback.
>>>>
>>>> Regarding the suggestion of adding the release notes document, I agree
>>>> with your point. Maybe we should adjust the vote template accordingly in
>>>> the respective wiki to guide the following release processes.
>>>>
>>>> Regarding the performance regression, could you provide some more details
>>>> for our better measurement or reproducing on our sides?
>>>> E.g. I guess the topology only includes two vertexes source and sink?
>>>> What is the parallelism for every vertex?
>>>> The upstream shuffles data to the downstream via rebalance partitioner or
>>>> other?
>>>> The checkpoint mode is exactly-once with rocksDB state backend?
>>>> The backpressure happened in this case?
>>>> How much percentage regression in this case?
>>>>
>>>> Best,
>>>> Zhijiang
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------
>>>> From:Thomas Weise <[hidden email]>
>>>> Send Time:2020年7月2日(星期四) 09:54
>>>> To:dev <[hidden email]>
>>>> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>>>>
>>>> Hi Till,
>>>>
>>>> Yes, we don't have the setting in flink-conf.yaml.
>>>>
>>>> Generally, we carry forward the existing configuration and any change to
>>>> default configuration values would impact the upgrade.
>>>>
>>>> Yes, since it is an incompatible change I would state it in the release
>>>> notes.
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>> BTW I found a performance regression while trying to upgrade another
>>>> pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't able
>>>> to pin it down yet, symptoms include increased checkpoint alignment time.
>>>>
>>>> On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi Thomas,
>>>>>
>>>>> just to confirm: When starting the image in local mode, then you don't
>>>> have
>>>>> any of the JobManager memory configuration settings configured in the
>>>>> effective flink-conf.yaml, right? Does this mean that you have
>>> explicitly
>>>>> removed `jobmanager.heap.size: 1024m` from the default configuration?
>>> If
>>>>> this is the case, then I believe it was more of an unintentional
>>> artifact
>>>>> that it worked before and it has been corrected now so that one needs
>>> to
>>>>> specify the memory of the JM process explicitly. Do you think it would
>>>> help
>>>>> to explicitly state this in the release notes?
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]> wrote:
>>>>>
>>>>>> Thanks for preparing another RC!
>>>>>>
>>>>>> As mentioned in the previous RC thread, it would be super helpful if
>>>> the
>>>>>> release notes that are part of the documentation can be included [1].
>>>>> It's
>>>>>> a significant time-saver to have read those first.
>>>>>>
>>>>>> I found one more non-backward compatible change that would be worth
>>>>>> addressing/mentioning:
>>>>>>
>>>>>> It is now necessary to configure the jobmanager heap size in
>>>>>> flink-conf.yaml (with either jobmanager.heap.size
>>>>>> or jobmanager.memory.heap.size). Why would I not want to do that
>>>> anyways?
>>>>>> Well, we set it dynamically for a cluster deployment via the
>>>>>> flinkk8soperator, but the container image can also be used for
>>> testing
>>>>> with
>>>>>> local mode (./bin/jobmanager.sh start-foreground local). That will
>>> fail
>>>>> if
>>>>>> the heap wasn't configured and that's how I noticed it.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
>>>>>> On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <[hidden email]
>>>>>> .invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Please review and vote on the release candidate #4 for the version
>>>>>> 1.11.0,
>>>>>>> as follows:
>>>>>>> [ ] +1, Approve the release
>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>> comments)
>>>>>>> The complete staging area is available for your review, which
>>>> includes:
>>>>>>> * JIRA release notes [1],
>>>>>>> * the official Apache source release and binary convenience
>>> releases
>>>> to
>>>>>> be
>>>>>>> deployed to dist.apache.org [2], which are signed with the key
>>> with
>>>>>>> fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
>>>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>>>>> * source code tag "release-1.11.0-rc4" [5],
>>>>>>> * website pull request listing the new release and adding
>>>> announcement
>>>>>>> blog post [6].
>>>>>>>
>>>>>>> The vote will be open for at least 72 hours. It is adopted by
>>>> majority
>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Release Manager
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
>>>>>>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
>>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>>>>>>> [4]
>>>>>>>
>>> https://repository.apache.org/content/repositories/orgapacheflink-1377/
>>>>>>> [5]
>>> https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
>>>>>>> [6] https://github.com/apache/flink-web/pull/352
>>>>>>>
>>>>>>>
>>>>

signature.asc (849 bytes) Download Attachment

Zhijiang(wangzhijiang999)

Re: [VOTE] Release 1.11.0, release candidate #4

In reply to this post by Thomas Weise

Hi Thomas,

Thanks for your reply with rich information!

We are trying to reproduce your case in our cluster to further verify it, and @Yingjie Cao is working on it now.
As we have not kinesis consumer and producer internally, so we will construct the common source and sink instead in the case of backpressure.

Firstly, we can dismiss the rockdb factor in this release, since you also mentioned that "filesystem leads to same symptoms".

Secondly, if my understanding is right, you emphasis that the regression only exists for the jobs with low checkpoint interval (10s).
Based on that, I have two suspicions with the network related changes in this release:
- [1]: Limited the maximum backlog value (default 10) in subpartition queue.
- [2]: Delay send the following buffers after checkpoint barrier on upstream side until barrier alignment on downstream side.

These changes are motivated for reducing the in-flight buffers to speedup checkpoint especially in the case of backpressure.
In theory they should have very minor performance effect and actually we also tested in cluster to verify within expectation before merging them,
but maybe there are other corner cases we have not thought of before.

Before the testing result on our side comes out for your respective job case, I have some other questions to confirm for further analysis:
- How much percentage regression you found after switching to 1.11?
- Are there any network bottleneck in your cluster? E.g. the network bandwidth is full caused by other jobs? If so, it might have more effects by above [2]
- Did you adjust the default network buffer setting? E.g. "taskmanager.network.memory.floating-buffers-per-gate" or "taskmanager.network.memory.buffers-per-channel"
- I guess the topology has three vertexes "KinesisConsumer -> Chained FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer -> FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the edge connection is one-to-one, not all-to-all, then the above [1][2] should no effects in theory with default network buffer setting.
- By slot sharing, I guess these three vertex parallelism task would probably be deployed into the same slot, then the data shuffle is by memory queue, not network stack. If so, the above [2] should no effect.
- I also saw some Jira changes for kinesis in this release, could you confirm that these changes would not effect the performance?

Best,
Zhijiang

------------------------------------------------------------------
From:Thomas Weise <[hidden email]>
Send Time:2020年7月3日(星期五) 01:07
To:dev <[hidden email]>; Zhijiang <[hidden email]>
Subject:Re: [VOTE] Release 1.11.0, release candidate #4

Hi Zhijiang,

The performance degradation manifests in backpressure which leads to
growing backlog in the source. I switched a few times between 1.10 and 1.11
and the behavior is consistent.

The DAG is:

KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
---------> KinesisProducer

Parallelism: 160
No shuffle/rebalance.

Checkpointing config:

Checkpointing Mode Exactly Once
Interval 10s
Timeout 10m 0s
Minimum Pause Between Checkpoints 10s
Maximum Concurrent Checkpoints 1
Persist Checkpoints Externally Enabled (delete on cancellation)

State backend: rocksdb (filesystem leads to same symptoms)
Checkpoint size is tiny (500KB)

An interesting difference to another job that I had upgraded successfully
is the low checkpointing interval.

Thanks,
Thomas

On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]>
wrote:

> Hi Thomas,
>
> Thanks for the efficient feedback.
>
> Regarding the suggestion of adding the release notes document, I agree
> with your point. Maybe we should adjust the vote template accordingly in
> the respective wiki to guide the following release processes.
>
> Regarding the performance regression, could you provide some more details
> for our better measurement or reproducing on our sides?
> E.g. I guess the topology only includes two vertexes source and sink?
> What is the parallelism for every vertex?
> The upstream shuffles data to the downstream via rebalance partitioner or
> other?
> The checkpoint mode is exactly-once with rocksDB state backend?
> The backpressure happened in this case?
> How much percentage regression in this case?
>
> Best,
> Zhijiang
>
>
>
> ------------------------------------------------------------------
> From:Thomas Weise <[hidden email]>
> Send Time:2020年7月2日(星期四) 09:54
> To:dev <[hidden email]>
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Till,
>
> Yes, we don't have the setting in flink-conf.yaml.
>
> Generally, we carry forward the existing configuration and any change to
> default configuration values would impact the upgrade.
>
> Yes, since it is an incompatible change I would state it in the release
> notes.
>
> Thanks,
> Thomas
>
> BTW I found a performance regression while trying to upgrade another
> pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't able
> to pin it down yet, symptoms include increased checkpoint alignment time.
>
> On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi Thomas,
> >
> > just to confirm: When starting the image in local mode, then you don't
> have
> > any of the JobManager memory configuration settings configured in the
> > effective flink-conf.yaml, right? Does this mean that you have explicitly
> > removed `jobmanager.heap.size: 1024m` from the default configuration? If
> > this is the case, then I believe it was more of an unintentional artifact
> > that it worked before and it has been corrected now so that one needs to
> > specify the memory of the JM process explicitly. Do you think it would
> help
> > to explicitly state this in the release notes?
> >
> > Cheers,
> > Till
> >
> > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]> wrote:
> >
> > > Thanks for preparing another RC!
> > >
> > > As mentioned in the previous RC thread, it would be super helpful if
> the
> > > release notes that are part of the documentation can be included [1].
> > It's
> > > a significant time-saver to have read those first.
> > >
> > > I found one more non-backward compatible change that would be worth
> > > addressing/mentioning:
> > >
> > > It is now necessary to configure the jobmanager heap size in
> > > flink-conf.yaml (with either jobmanager.heap.size
> > > or jobmanager.memory.heap.size). Why would I not want to do that
> anyways?
> > > Well, we set it dynamically for a cluster deployment via the
> > > flinkk8soperator, but the container image can also be used for testing
> > with
> > > local mode (./bin/jobmanager.sh start-foreground local). That will fail
> > if
> > > the heap wasn't configured and that's how I noticed it.
> > >
> > > Thanks,
> > > Thomas
> > >
> > > [1]
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > >
> > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <[hidden email]
> > > .invalid>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #4 for the version
> > > 1.11.0,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release and binary convenience releases
> to
> > > be
> > > > deployed to dist.apache.org [2], which are signed with the key with
> > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag "release-1.11.0-rc4" [5],
> > > > * website pull request listing the new release and adding
> announcement
> > > > blog post [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > [5] https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > [6] https://github.com/apache/flink-web/pull/352
> > > >
> > > >
> > >
> >
>
>

Till Rohrmann

Re: [VOTE] Release 1.11.0, release candidate #4

@Dawid I think it would be correct to also include the classifier for the
org.apache.orc:orc-core:jar:nohive:1.4.3 dependency because it is different
from the non-classified artifact. I would not block the release on it,
though, because it is a ASL 2.0 dependency which we are not required to
list. Can you open a PR for fixing this problem?

Concerning the Python module I believe that Jincheng could help us with the
verification process.

Cheers,
Till

On Fri, Jul 3, 2020 at 8:46 AM Zhijiang <[hidden email]>
wrote:

> Hi Thomas,
>
> Thanks for your reply with rich information!
>
> We are trying to reproduce your case in our cluster to further verify it,
> and @Yingjie Cao is working on it now.
> As we have not kinesis consumer and producer internally, so we will
> construct the common source and sink instead in the case of backpressure.
>
> Firstly, we can dismiss the rockdb factor in this release, since you also
> mentioned that "filesystem leads to same symptoms".
>
> Secondly, if my understanding is right, you emphasis that the regression
> only exists for the jobs with low checkpoint interval (10s).
> Based on that, I have two suspicions with the network related changes in
> this release:
> - [1]: Limited the maximum backlog value (default 10) in subpartition
> queue.
> - [2]: Delay send the following buffers after checkpoint barrier on
> upstream side until barrier alignment on downstream side.
>
> These changes are motivated for reducing the in-flight buffers to speedup
> checkpoint especially in the case of backpressure.
> In theory they should have very minor performance effect and actually we
> also tested in cluster to verify within expectation before merging them,
> but maybe there are other corner cases we have not thought of before.
>
> Before the testing result on our side comes out for your respective job
> case, I have some other questions to confirm for further analysis:
> - How much percentage regression you found after switching to 1.11?
> - Are there any network bottleneck in your cluster? E.g. the network
> bandwidth is full caused by other jobs? If so, it might have more effects
> by above [2]
> - Did you adjust the default network buffer setting? E.g.
> "taskmanager.network.memory.floating-buffers-per-gate" or
> "taskmanager.network.memory.buffers-per-channel"
> - I guess the topology has three vertexes "KinesisConsumer -> Chained
> FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer ->
> FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the edge
> connection is one-to-one, not all-to-all, then the above [1][2] should no
> effects in theory with default network buffer setting.
> - By slot sharing, I guess these three vertex parallelism task would
> probably be deployed into the same slot, then the data shuffle is by memory
> queue, not network stack. If so, the above [2] should no effect.
> - I also saw some Jira changes for kinesis in this release, could you
> confirm that these changes would not effect the performance?
>
> Best,
> Zhijiang
>
>
> ------------------------------------------------------------------
> From:Thomas Weise <[hidden email]>
> Send Time:2020年7月3日(星期五) 01:07
> To:dev <[hidden email]>; Zhijiang <[hidden email]>
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Zhijiang,
>
> The performance degradation manifests in backpressure which leads to
> growing backlog in the source. I switched a few times between 1.10 and 1.11
> and the behavior is consistent.
>
> The DAG is:
>
> KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> ---------> KinesisProducer
>
> Parallelism: 160
> No shuffle/rebalance.
>
> Checkpointing config:
>
> Checkpointing Mode Exactly Once
> Interval 10s
> Timeout 10m 0s
> Minimum Pause Between Checkpoints 10s
> Maximum Concurrent Checkpoints 1
> Persist Checkpoints Externally Enabled (delete on cancellation)
>
> State backend: rocksdb (filesystem leads to same symptoms)
> Checkpoint size is tiny (500KB)
>
> An interesting difference to another job that I had upgraded successfully
> is the low checkpointing interval.
>
> Thanks,
> Thomas
>
>
> On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the efficient feedback.
> >
> > Regarding the suggestion of adding the release notes document, I agree
> > with your point. Maybe we should adjust the vote template accordingly in
> > the respective wiki to guide the following release processes.
> >
> > Regarding the performance regression, could you provide some more details
> > for our better measurement or reproducing on our sides?
> > E.g. I guess the topology only includes two vertexes source and sink?
> > What is the parallelism for every vertex?
> > The upstream shuffles data to the downstream via rebalance partitioner or
> > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月2日(星期四) 09:54
> > To:dev <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Till,
> >
> > Yes, we don't have the setting in flink-conf.yaml.
> >
> > Generally, we carry forward the existing configuration and any change to
> > default configuration values would impact the upgrade.
> >
> > Yes, since it is an incompatible change I would state it in the release
> > notes.
> >
> > Thanks,
> > Thomas
> >
> > BTW I found a performance regression while trying to upgrade another
> > pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't able
> > to pin it down yet, symptoms include increased checkpoint alignment time.
> >
> > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > just to confirm: When starting the image in local mode, then you don't
> > have
> > > any of the JobManager memory configuration settings configured in the
> > > effective flink-conf.yaml, right? Does this mean that you have
> explicitly
> > > removed `jobmanager.heap.size: 1024m` from the default configuration?
> If
> > > this is the case, then I believe it was more of an unintentional
> artifact
> > > that it worked before and it has been corrected now so that one needs
> to
> > > specify the memory of the JM process explicitly. Do you think it would
> > help
> > > to explicitly state this in the release notes?
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]> wrote:
> > >
> > > > Thanks for preparing another RC!
> > > >
> > > > As mentioned in the previous RC thread, it would be super helpful if
> > the
> > > > release notes that are part of the documentation can be included [1].
> > > It's
> > > > a significant time-saver to have read those first.
> > > >
> > > > I found one more non-backward compatible change that would be worth
> > > > addressing/mentioning:
> > > >
> > > > It is now necessary to configure the jobmanager heap size in
> > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > or jobmanager.memory.heap.size). Why would I not want to do that
> > anyways?
> > > > Well, we set it dynamically for a cluster deployment via the
> > > > flinkk8soperator, but the container image can also be used for
> testing
> > > with
> > > > local mode (./bin/jobmanager.sh start-foreground local). That will
> fail
> > > if
> > > > the heap wasn't configured and that's how I noticed it.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > >
> > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <[hidden email]
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Please review and vote on the release candidate #4 for the version
> > > > 1.11.0,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release and binary convenience
> releases
> > to
> > > > be
> > > > > deployed to dist.apache.org [2], which are signed with the key
> with
> > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > * website pull request listing the new release and adding
> > announcement
> > > > > blog post [6].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Release Manager
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > >
> > > > >
> > > >
> > >
> >
> >
>
>

dwysakowicz

Re: [VOTE] Release 1.11.0, release candidate #4

Thanks Till for the clarification. I opened
https://github.com/apache/flink/pull/12816

On 03/07/2020 10:15, Till Rohrmann wrote:

> @Dawid I think it would be correct to also include the classifier for the
> org.apache.orc:orc-core:jar:nohive:1.4.3 dependency because it is different
> from the non-classified artifact. I would not block the release on it,
> though, because it is a ASL 2.0 dependency which we are not required to
> list. Can you open a PR for fixing this problem?
>
> Concerning the Python module I believe that Jincheng could help us with the
> verification process.
>
> Cheers,
> Till
>
> On Fri, Jul 3, 2020 at 8:46 AM Zhijiang <[hidden email]>
> wrote:
>
>> Hi Thomas,
>>
>> Thanks for your reply with rich information!
>>
>> We are trying to reproduce your case in our cluster to further verify it,
>> and @Yingjie Cao is working on it now.
>> As we have not kinesis consumer and producer internally, so we will
>> construct the common source and sink instead in the case of backpressure.
>>
>> Firstly, we can dismiss the rockdb factor in this release, since you also
>> mentioned that "filesystem leads to same symptoms".
>>
>> Secondly, if my understanding is right, you emphasis that the regression
>> only exists for the jobs with low checkpoint interval (10s).
>> Based on that, I have two suspicions with the network related changes in
>> this release:
>> - [1]: Limited the maximum backlog value (default 10) in subpartition
>> queue.
>> - [2]: Delay send the following buffers after checkpoint barrier on
>> upstream side until barrier alignment on downstream side.
>>
>> These changes are motivated for reducing the in-flight buffers to speedup
>> checkpoint especially in the case of backpressure.
>> In theory they should have very minor performance effect and actually we
>> also tested in cluster to verify within expectation before merging them,
>> but maybe there are other corner cases we have not thought of before.
>>
>> Before the testing result on our side comes out for your respective job
>> case, I have some other questions to confirm for further analysis:
>> - How much percentage regression you found after switching to 1.11?
>> - Are there any network bottleneck in your cluster? E.g. the network
>> bandwidth is full caused by other jobs? If so, it might have more effects
>> by above [2]
>> - Did you adjust the default network buffer setting? E.g.
>> "taskmanager.network.memory.floating-buffers-per-gate" or
>> "taskmanager.network.memory.buffers-per-channel"
>> - I guess the topology has three vertexes "KinesisConsumer -> Chained
>> FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer ->
>> FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the edge
>> connection is one-to-one, not all-to-all, then the above [1][2] should no
>> effects in theory with default network buffer setting.
>> - By slot sharing, I guess these three vertex parallelism task would
>> probably be deployed into the same slot, then the data shuffle is by memory
>> queue, not network stack. If so, the above [2] should no effect.
>> - I also saw some Jira changes for kinesis in this release, could you
>> confirm that these changes would not effect the performance?
>>
>> Best,
>> Zhijiang
>>
>>
>> ------------------------------------------------------------------
>> From:Thomas Weise <[hidden email]>
>> Send Time:2020年7月3日(星期五) 01:07
>> To:dev <[hidden email]>; Zhijiang <[hidden email]>
>> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>>
>> Hi Zhijiang,
>>
>> The performance degradation manifests in backpressure which leads to
>> growing backlog in the source. I switched a few times between 1.10 and 1.11
>> and the behavior is consistent.
>>
>> The DAG is:
>>
>> KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
>> ---------> KinesisProducer
>>
>> Parallelism: 160
>> No shuffle/rebalance.
>>
>> Checkpointing config:
>>
>> Checkpointing Mode Exactly Once
>> Interval 10s
>> Timeout 10m 0s
>> Minimum Pause Between Checkpoints 10s
>> Maximum Concurrent Checkpoints 1
>> Persist Checkpoints Externally Enabled (delete on cancellation)
>>
>> State backend: rocksdb (filesystem leads to same symptoms)
>> Checkpoint size is tiny (500KB)
>>
>> An interesting difference to another job that I had upgraded successfully
>> is the low checkpointing interval.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
>> .invalid>
>> wrote:
>>
>>> Hi Thomas,
>>>
>>> Thanks for the efficient feedback.
>>>
>>> Regarding the suggestion of adding the release notes document, I agree
>>> with your point. Maybe we should adjust the vote template accordingly in
>>> the respective wiki to guide the following release processes.
>>>
>>> Regarding the performance regression, could you provide some more details
>>> for our better measurement or reproducing on our sides?
>>> E.g. I guess the topology only includes two vertexes source and sink?
>>> What is the parallelism for every vertex?
>>> The upstream shuffles data to the downstream via rebalance partitioner or
>>> other?
>>> The checkpoint mode is exactly-once with rocksDB state backend?
>>> The backpressure happened in this case?
>>> How much percentage regression in this case?
>>>
>>> Best,
>>> Zhijiang
>>>
>>>
>>>
>>> ------------------------------------------------------------------
>>> From:Thomas Weise <[hidden email]>
>>> Send Time:2020年7月2日(星期四) 09:54
>>> To:dev <[hidden email]>
>>> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>>>
>>> Hi Till,
>>>
>>> Yes, we don't have the setting in flink-conf.yaml.
>>>
>>> Generally, we carry forward the existing configuration and any change to
>>> default configuration values would impact the upgrade.
>>>
>>> Yes, since it is an incompatible change I would state it in the release
>>> notes.
>>>
>>> Thanks,
>>> Thomas
>>>
>>> BTW I found a performance regression while trying to upgrade another
>>> pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't able
>>> to pin it down yet, symptoms include increased checkpoint alignment time.
>>>
>>> On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
>>> wrote:
>>>
>>>> Hi Thomas,
>>>>
>>>> just to confirm: When starting the image in local mode, then you don't
>>> have
>>>> any of the JobManager memory configuration settings configured in the
>>>> effective flink-conf.yaml, right? Does this mean that you have
>> explicitly
>>>> removed `jobmanager.heap.size: 1024m` from the default configuration?
>> If
>>>> this is the case, then I believe it was more of an unintentional
>> artifact
>>>> that it worked before and it has been corrected now so that one needs
>> to
>>>> specify the memory of the JM process explicitly. Do you think it would
>>> help
>>>> to explicitly state this in the release notes?
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]> wrote:
>>>>
>>>>> Thanks for preparing another RC!
>>>>>
>>>>> As mentioned in the previous RC thread, it would be super helpful if
>>> the
>>>>> release notes that are part of the documentation can be included [1].
>>>> It's
>>>>> a significant time-saver to have read those first.
>>>>>
>>>>> I found one more non-backward compatible change that would be worth
>>>>> addressing/mentioning:
>>>>>
>>>>> It is now necessary to configure the jobmanager heap size in
>>>>> flink-conf.yaml (with either jobmanager.heap.size
>>>>> or jobmanager.memory.heap.size). Why would I not want to do that
>>> anyways?
>>>>> Well, we set it dynamically for a cluster deployment via the
>>>>> flinkk8soperator, but the container image can also be used for
>> testing
>>>> with
>>>>> local mode (./bin/jobmanager.sh start-foreground local). That will
>> fail
>>>> if
>>>>> the heap wasn't configured and that's how I noticed it.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>> [1]
>>>>>
>>>>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
>>>>> On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <[hidden email]
>>>>> .invalid>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Please review and vote on the release candidate #4 for the version
>>>>> 1.11.0,
>>>>>> as follows:
>>>>>> [ ] +1, Approve the release
>>>>>> [ ] -1, Do not approve the release (please provide specific
>> comments)
>>>>>> The complete staging area is available for your review, which
>>> includes:
>>>>>> * JIRA release notes [1],
>>>>>> * the official Apache source release and binary convenience
>> releases
>>> to
>>>>> be
>>>>>> deployed to dist.apache.org [2], which are signed with the key
>> with
>>>>>> fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
>>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>>>> * source code tag "release-1.11.0-rc4" [5],
>>>>>> * website pull request listing the new release and adding
>>> announcement
>>>>>> blog post [6].
>>>>>>
>>>>>> The vote will be open for at least 72 hours. It is adopted by
>>> majority
>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>
>>>>>> Thanks,
>>>>>> Release Manager
>>>>>>
>>>>>> [1]
>>>>>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
>>>>>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>>>>>> [4]
>>>>>>
>> https://repository.apache.org/content/repositories/orgapacheflink-1377/
>>>>>> [5]
>> https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
>>>>>> [6] https://github.com/apache/flink-web/pull/352
>>>>>>
>>>>>>
>>>
>>

signature.asc (849 bytes) Download Attachment

yingjie

Re: [VOTE] Release 1.11.0, release candidate #4

In reply to this post by Thomas Weise

Hi Thomas,

I tried to reproduce the regression by constructing a Job with the same
topology, parallelism and checkpoint interval (Kinesis source and sink are
replaced for we do not have the test environment). But unfortunately, no
regression is observed both for back pressure and no back pressure cases.

Maybe we need more information to further investigate the case. Beside what
Zhijiang have asked, I have one more question: how many records/bytes each
vertex processes per second and is there any imbalance, for example, some
tasks are slower or process more records/bytes than others?

Best,
Yingjie

Thomas Weise <[hidden email]>于2020年7月3日周五上午1:07写道：

> Hi Zhijiang,
>
> The performance degradation manifests in backpressure which leads to
> growing backlog in the source. I switched a few times between 1.10 and 1.11
> and the behavior is consistent.
>
> The DAG is:
>
> KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> ---------> KinesisProducer
>
> Parallelism: 160
> No shuffle/rebalance.
>
> Checkpointing config:
>
> Checkpointing Mode Exactly Once
> Interval 10s
> Timeout 10m 0s
> Minimum Pause Between Checkpoints 10s
> Maximum Concurrent Checkpoints 1
> Persist Checkpoints Externally Enabled (delete on cancellation)
>
> State backend: rocksdb (filesystem leads to same symptoms)
> Checkpoint size is tiny (500KB)
>
> An interesting difference to another job that I had upgraded successfully
> is the low checkpointing interval.
>
> Thanks,
> Thomas
>
>
> On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the efficient feedback.
> >
> > Regarding the suggestion of adding the release notes document, I agree
> > with your point. Maybe we should adjust the vote template accordingly in
> > the respective wiki to guide the following release processes.
> >
> > Regarding the performance regression, could you provide some more details
> > for our better measurement or reproducing on our sides?
> > E.g. I guess the topology only includes two vertexes source and sink?
> > What is the parallelism for every vertex?
> > The upstream shuffles data to the downstream via rebalance partitioner or
> > other?
> > The checkpoint mode is exactly-once with rocksDB state backend?
> > The backpressure happened in this case?
> > How much percentage regression in this case?
> >
> > Best,
> > Zhijiang
> >
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月2日(星期四) 09:54
> > To:dev <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Till,
> >
> > Yes, we don't have the setting in flink-conf.yaml.
> >
> > Generally, we carry forward the existing configuration and any change to
> > default configuration values would impact the upgrade.
> >
> > Yes, since it is an incompatible change I would state it in the release
> > notes.
> >
> > Thanks,
> > Thomas
> >
> > BTW I found a performance regression while trying to upgrade another
> > pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't able
> > to pin it down yet, symptoms include increased checkpoint alignment time.
> >
> > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > just to confirm: When starting the image in local mode, then you don't
> > have
> > > any of the JobManager memory configuration settings configured in the
> > > effective flink-conf.yaml, right? Does this mean that you have
> explicitly
> > > removed `jobmanager.heap.size: 1024m` from the default configuration?
> If
> > > this is the case, then I believe it was more of an unintentional
> artifact
> > > that it worked before and it has been corrected now so that one needs
> to
> > > specify the memory of the JM process explicitly. Do you think it would
> > help
> > > to explicitly state this in the release notes?
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]> wrote:
> > >
> > > > Thanks for preparing another RC!
> > > >
> > > > As mentioned in the previous RC thread, it would be super helpful if
> > the
> > > > release notes that are part of the documentation can be included [1].
> > > It's
> > > > a significant time-saver to have read those first.
> > > >
> > > > I found one more non-backward compatible change that would be worth
> > > > addressing/mentioning:
> > > >
> > > > It is now necessary to configure the jobmanager heap size in
> > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > or jobmanager.memory.heap.size). Why would I not want to do that
> > anyways?
> > > > Well, we set it dynamically for a cluster deployment via the
> > > > flinkk8soperator, but the container image can also be used for
> testing
> > > with
> > > > local mode (./bin/jobmanager.sh start-foreground local). That will
> fail
> > > if
> > > > the heap wasn't configured and that's how I noticed it.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > >
> > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <[hidden email]
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Please review and vote on the release candidate #4 for the version
> > > > 1.11.0,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release and binary convenience
> releases
> > to
> > > > be
> > > > > deployed to dist.apache.org [2], which are signed with the key
> with
> > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > * website pull request listing the new release and adding
> > announcement
> > > > > blog post [6].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Release Manager
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > [5]
> https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > >
> > > > >
> > > >
> > >
> >
> >
>

Leonard Xu

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding)

- checked/verified signatures and hashes
- built from source sing scala 2.11 succeeded
- go through all issues which "fixVersion" property is 1.11.0, there is no blocker.
- checked that there are no missing artifacts
- test SQL connector Elasticsearch7/JDBC/HBase/Kafka (new connector) with some queries in SQL Client, they works well and the result is expected
- started a cluster, WebUI was accessible, submitted a wordcount job and ran succeeded, no suspicious log output
- the web PR looks good

Best,
Leonard Xu

jincheng sun

Re: [VOTE] Release 1.11.0, release candidate #4

+1(binding)

Checks:
- check wheel package consistency
- test the built from the wheel package
- checked the signature and checksum
- pip installed the Python package
`apache_flink-1.11.0-cp37-cp37m-macosx_10_9_x86_64.whl` successfully and
run a simple word count example successfully
- verify the performance for PyFlink UDFs
- test Python UDTF support

Best,
Jincheng

Leonard Xu <[hidden email]> 于2020年7月3日周五下午6:02写道：

> +1 (non-binding)
>
> - checked/verified signatures and hashes
> - built from source sing scala 2.11 succeeded
> - go through all issues which "fixVersion" property is 1.11.0, there is no
> blocker.
> - checked that there are no missing artifacts
> - test SQL connector Elasticsearch7/JDBC/HBase/Kafka (new connector) with
> some queries in SQL Client, they works well and the result is expected
> - started a cluster, WebUI was accessible, submitted a wordcount job and
> ran succeeded, no suspicious log output
> - the web PR looks good
>
> Best,
> Leonard Xu

Dian Fu-2

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding)

- built from source with scala 2.11 successfully
- checked the signature and checksum of the binary packages
- installed PyFlink on MacOS, Windows and Linux successfully
- tested the functionality of Pandas UDF and the conversion between PyFlink Table and Pandas DataFrame
- verified that the Python dependency management functionality works well

Regards,
Dian

> 在 2020年7月3日，下午8:24，jincheng sun <[hidden email]> 写道：
>
> +1(binding)
>
> Checks:
> - check wheel package consistency
> - test the built from the wheel package
> - checked the signature and checksum
> - pip installed the Python package
> `apache_flink-1.11.0-cp37-cp37m-macosx_10_9_x86_64.whl` successfully and
> run a simple word count example successfully
> - verify the performance for PyFlink UDFs
> - test Python UDTF support
>
> Best,
> Jincheng
>
>
> Leonard Xu <[hidden email]> 于2020年7月3日周五下午6:02写道：
>
>> +1 (non-binding)
>>
>> - checked/verified signatures and hashes
>> - built from source sing scala 2.11 succeeded
>> - go through all issues which "fixVersion" property is 1.11.0, there is no
>> blocker.
>> - checked that there are no missing artifacts
>> - test SQL connector Elasticsearch7/JDBC/HBase/Kafka (new connector) with
>> some queries in SQL Client, they works well and the result is expected
>> - started a cluster, WebUI was accessible, submitted a wordcount job and
>> ran succeeded, no suspicious log output
>> - the web PR looks good
>>
>> Best,
>> Leonard Xu

Xingbo Huang

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding)

- check wheel package consistency with the built from the source code
- test the built from the wheel package in mac os with python 3.6
- verify the performance for PyFlink UDFs including Python General UDF and
Pandas UDF
- test Python UDTF

Best,
Xingbo

Dian Fu <[hidden email]> 于2020年7月3日周五下午8:45写道：

> +1 (non-binding)
>
> - built from source with scala 2.11 successfully
> - checked the signature and checksum of the binary packages
> - installed PyFlink on MacOS, Windows and Linux successfully
> - tested the functionality of Pandas UDF and the conversion between
> PyFlink Table and Pandas DataFrame
> - verified that the Python dependency management functionality works well
>
> Regards,
> Dian
>
> > 在 2020年7月3日，下午8:24，jincheng sun <[hidden email]> 写道：
> >
> > +1(binding)
> >
> > Checks:
> > - check wheel package consistency
> > - test the built from the wheel package
> > - checked the signature and checksum
> > - pip installed the Python package
> > `apache_flink-1.11.0-cp37-cp37m-macosx_10_9_x86_64.whl` successfully and
> > run a simple word count example successfully
> > - verify the performance for PyFlink UDFs
> > - test Python UDTF support
> >
> > Best,
> > Jincheng
> >
> >
> > Leonard Xu <[hidden email]> 于2020年7月3日周五下午6:02写道：
> >
> >> +1 (non-binding)
> >>
> >> - checked/verified signatures and hashes
> >> - built from source sing scala 2.11 succeeded
> >> - go through all issues which "fixVersion" property is 1.11.0, there is
> no
> >> blocker.
> >> - checked that there are no missing artifacts
> >> - test SQL connector Elasticsearch7/JDBC/HBase/Kafka (new connector)
> with
> >> some queries in SQL Client, they works well and the result is expected
> >> - started a cluster, WebUI was accessible, submitted a wordcount job and
> >> ran succeeded, no suspicious log output
> >> - the web PR looks good
> >>
> >> Best,
> >> Leonard Xu
>
>

yingjie

Re: [VOTE] Release 1.11.0, release candidate #4

In reply to this post by Thomas Weise

Hi Thomas,

Thanks a lot for offering these information.

We have decided to try to reproduce the regression on AWS. It will be
really appreciated if you can share some demo code with us, and if it is
not convenient, could you give us some more information about the record
type and size, the processing logic of each operator? It will help us to
write a more similar Job and reproduce the regression.

Besides, I have a question to ask, if increasing the checkpoint interval
helps to reduce the regression? I am asking because I wonder if the
regression is really related to the checkpoint interval.

Best,
Yingjie

Thomas Weise <[hidden email]>于2020年7月3日周五上午1:07写道：

Thomas Weise

Re: [VOTE] Release 1.11.0, release candidate #4

In reply to this post by Zhijiang(wangzhijiang999)

Hi Zhijiang,

It will probably be best if we connect next week and discuss the issue
directly since this could be quite difficult to reproduce.

Before the testing result on our side comes out for your respective job
case, I have some other questions to confirm for further analysis:
- How much percentage regression you found after switching to 1.11?

~40% throughput decline

- Are there any network bottleneck in your cluster? E.g. the network
bandwidth is full caused by other jobs? If so, it might have more effects
by above [2]

The test runs on a k8s cluster that is also used for other production jobs.
There is no reason be believe network is the bottleneck.

- Did you adjust the default network buffer setting? E.g.
"taskmanager.network.memory.floating-buffers-per-gate" or
"taskmanager.network.memory.buffers-per-channel"

The job is using the defaults, i.e we don't configure the settings. If you
want me to try specific settings in the hope that it will help to isolate
the issue please let me know.

- I guess the topology has three vertexes "KinesisConsumer -> Chained
FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer ->
FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the edge
connection is one-to-one, not all-to-all, then the above [1][2] should no
effects in theory with default network buffer setting.

There are only 2 vertices and the edge is "forward".

- By slot sharing, I guess these three vertex parallelism task would
probably be deployed into the same slot, then the data shuffle is by memory
queue, not network stack. If so, the above [2] should no effect.

Yes, vertices share slots.

- I also saw some Jira changes for kinesis in this release, could you
confirm that these changes would not effect the performance?

I will need to take a look. 1.10 already had a regression introduced by the
Kinesis producer update.

Thanks,
Thomas

On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]>
wrote:

Zhijiang(wangzhijiang999)

Re: [VOTE] Release 1.11.0, release candidate #4

Hi Thomas,

Thanks for the further update information.

I guess we can dismiss the network stack changes, since in your case the downstream and upstream would probably be deployed in the same slot bypassing the network data shuffle.
Also I guess release-1.11 will not bring general performance regression in runtime engine, as we also did the performance testing for all general cases by [1] in real cluster before and the testing results should fit the expectation. But we indeed did not test the specific source and sink connectors yet as I known.

Regarding your performance regression with 40%, I wonder it is probably related to specific source/sink changes (e.g. kinesis) or environment issues with corner case.
If possible, it would be helpful to further locate whether the regression is caused by kinesis, by replacing the kinesis source & sink and keeping the others same.

As you said, it would be efficient to contact with you directly next week to further discuss this issue. And we are willing/eager to provide any help to resolve this issue soon.

Besides that, I guess this issue should not be the blocker for the release, since it is probably a corner case based on the current analysis.
If we really conclude anything need to be resolved after the final release, then we can also make the next minor release-1.11.1 come soon.

[1] https://issues.apache.org/jira/browse/FLINK-18433

Best,
Zhijiang

------------------------------------------------------------------
From:Thomas Weise <[hidden email]>
Send Time:2020年7月4日(星期六) 12:26
To:dev <[hidden email]>; Zhijiang <[hidden email]>
Cc:Yingjie Cao <[hidden email]>
Subject:Re: [VOTE] Release 1.11.0, release candidate #4

Hi Zhijiang,

It will probably be best if we connect next week and discuss the issue
directly since this could be quite difficult to reproduce.

Before the testing result on our side comes out for your respective job
case, I have some other questions to confirm for further analysis:
- How much percentage regression you found after switching to 1.11?

~40% throughput decline

- Are there any network bottleneck in your cluster? E.g. the network
bandwidth is full caused by other jobs? If so, it might have more effects
by above [2]

The test runs on a k8s cluster that is also used for other production jobs.
There is no reason be believe network is the bottleneck.

- Did you adjust the default network buffer setting? E.g.
"taskmanager.network.memory.floating-buffers-per-gate" or
"taskmanager.network.memory.buffers-per-channel"

The job is using the defaults, i.e we don't configure the settings. If you
want me to try specific settings in the hope that it will help to isolate
the issue please let me know.

- I guess the topology has three vertexes "KinesisConsumer -> Chained
FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer ->
FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the edge
connection is one-to-one, not all-to-all, then the above [1][2] should no
effects in theory with default network buffer setting.

There are only 2 vertices and the edge is "forward".

- By slot sharing, I guess these three vertex parallelism task would
probably be deployed into the same slot, then the data shuffle is by memory
queue, not network stack. If so, the above [2] should no effect.

Yes, vertices share slots.

- I also saw some Jira changes for kinesis in this release, could you
confirm that these changes would not effect the performance?

I will need to take a look. 1.10 already had a regression introduced by the
Kinesis producer update.

Thanks,
Thomas

On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]>
wrote:

Steven Wu

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding)

- rolled out to thousands of router jobs in our test env
- tested with a large-state job. Did simple resilience and
checkpoint/savepoint tests. General performance metrics look on par.
- tested with a high-parallelism stateless transformation job. General
performance metrics look on par.

On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]>
wrote:

> Hi Thomas,
>
> Thanks for the further update information.
>
> I guess we can dismiss the network stack changes, since in your case the
> downstream and upstream would probably be deployed in the same slot
> bypassing the network data shuffle.
> Also I guess release-1.11 will not bring general performance regression in
> runtime engine, as we also did the performance testing for all general
> cases by [1] in real cluster before and the testing results should fit the
> expectation. But we indeed did not test the specific source and sink
> connectors yet as I known.
>
> Regarding your performance regression with 40%, I wonder it is probably
> related to specific source/sink changes (e.g. kinesis) or environment
> issues with corner case.
> If possible, it would be helpful to further locate whether the regression
> is caused by kinesis, by replacing the kinesis source & sink and keeping
> the others same.
>
> As you said, it would be efficient to contact with you directly next week
> to further discuss this issue. And we are willing/eager to provide any help
> to resolve this issue soon.
>
> Besides that, I guess this issue should not be the blocker for the
> release, since it is probably a corner case based on the current analysis.
> If we really conclude anything need to be resolved after the final
> release, then we can also make the next minor release-1.11.1 come soon.
>
> [1] https://issues.apache.org/jira/browse/FLINK-18433
>
> Best,
> Zhijiang
>
>
> ------------------------------------------------------------------
> From:Thomas Weise <[hidden email]>
> Send Time:2020年7月4日(星期六) 12:26
> To:dev <[hidden email]>; Zhijiang <[hidden email]>
> Cc:Yingjie Cao <[hidden email]>
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Zhijiang,
>
> It will probably be best if we connect next week and discuss the issue
> directly since this could be quite difficult to reproduce.
>
> Before the testing result on our side comes out for your respective job
> case, I have some other questions to confirm for further analysis:
> - How much percentage regression you found after switching to 1.11?
>
> ~40% throughput decline
>
> - Are there any network bottleneck in your cluster? E.g. the network
> bandwidth is full caused by other jobs? If so, it might have more effects
> by above [2]
>
> The test runs on a k8s cluster that is also used for other production jobs.
> There is no reason be believe network is the bottleneck.
>
> - Did you adjust the default network buffer setting? E.g.
> "taskmanager.network.memory.floating-buffers-per-gate" or
> "taskmanager.network.memory.buffers-per-channel"
>
> The job is using the defaults, i.e we don't configure the settings. If you
> want me to try specific settings in the hope that it will help to isolate
> the issue please let me know.
>
> - I guess the topology has three vertexes "KinesisConsumer -> Chained
> FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer ->
> FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the edge
> connection is one-to-one, not all-to-all, then the above [1][2] should no
> effects in theory with default network buffer setting.
>
> There are only 2 vertices and the edge is "forward".
>
> - By slot sharing, I guess these three vertex parallelism task would
> probably be deployed into the same slot, then the data shuffle is by memory
> queue, not network stack. If so, the above [2] should no effect.
>
> Yes, vertices share slots.
>
> - I also saw some Jira changes for kinesis in this release, could you
> confirm that these changes would not effect the performance?
>
> I will need to take a look. 1.10 already had a regression introduced by the
> Kinesis producer update.
>
>
> Thanks,
> Thomas
>
>
> On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for your reply with rich information!
> >
> > We are trying to reproduce your case in our cluster to further verify it,
> > and @Yingjie Cao is working on it now.
> > As we have not kinesis consumer and producer internally, so we will
> > construct the common source and sink instead in the case of backpressure.
> >
> > Firstly, we can dismiss the rockdb factor in this release, since you also
> > mentioned that "filesystem leads to same symptoms".
> >
> > Secondly, if my understanding is right, you emphasis that the regression
> > only exists for the jobs with low checkpoint interval (10s).
> > Based on that, I have two suspicions with the network related changes in
> > this release:
> > - [1]: Limited the maximum backlog value (default 10) in subpartition
> > queue.
> > - [2]: Delay send the following buffers after checkpoint barrier on
> > upstream side until barrier alignment on downstream side.
> >
> > These changes are motivated for reducing the in-flight buffers to speedup
> > checkpoint especially in the case of backpressure.
> > In theory they should have very minor performance effect and actually we
> > also tested in cluster to verify within expectation before merging them,
> > but maybe there are other corner cases we have not thought of before.
> >
> > Before the testing result on our side comes out for your respective job
> > case, I have some other questions to confirm for further analysis:
> > - How much percentage regression you found after switching to 1.11?
> > - Are there any network bottleneck in your cluster? E.g. the network
> > bandwidth is full caused by other jobs? If so, it might have more effects
> > by above [2]
> > - Did you adjust the default network buffer setting? E.g.
> > "taskmanager.network.memory.floating-buffers-per-gate" or
> > "taskmanager.network.memory.buffers-per-channel"
> > - I guess the topology has three vertexes "KinesisConsumer ->
> Chained
> > FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer
> ->
> > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> edge
> > connection is one-to-one, not all-to-all, then the above [1][2] should no
> > effects in theory with default network buffer setting.
> > - By slot sharing, I guess these three vertex parallelism task would
> > probably be deployed into the same slot, then the data shuffle is by
> memory
> > queue, not network stack. If so, the above [2] should no effect.
> > - I also saw some Jira changes for kinesis in this release, could you
> > confirm that these changes would not effect the performance?
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月3日(星期五) 01:07
> > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Zhijiang,
> >
> > The performance degradation manifests in backpressure which leads to
> > growing backlog in the source. I switched a few times between 1.10 and
> 1.11
> > and the behavior is consistent.
> >
> > The DAG is:
> >
> > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> > ---------> KinesisProducer
> >
> > Parallelism: 160
> > No shuffle/rebalance.
> >
> > Checkpointing config:
> >
> > Checkpointing Mode Exactly Once
> > Interval 10s
> > Timeout 10m 0s
> > Minimum Pause Between Checkpoints 10s
> > Maximum Concurrent Checkpoints 1
> > Persist Checkpoints Externally Enabled (delete on cancellation)
> >
> > State backend: rocksdb (filesystem leads to same symptoms)
> > Checkpoint size is tiny (500KB)
> >
> > An interesting difference to another job that I had upgraded successfully
> > is the low checkpointing interval.
> >
> > Thanks,
> > Thomas
> >
> >
> > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> > .invalid>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > Thanks for the efficient feedback.
> > >
> > > Regarding the suggestion of adding the release notes document, I agree
> > > with your point. Maybe we should adjust the vote template accordingly
> in
> > > the respective wiki to guide the following release processes.
> > >
> > > Regarding the performance regression, could you provide some more
> details
> > > for our better measurement or reproducing on our sides?
> > > E.g. I guess the topology only includes two vertexes source and sink?
> > > What is the parallelism for every vertex?
> > > The upstream shuffles data to the downstream via rebalance partitioner
> or
> > > other?
> > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > The backpressure happened in this case?
> > > How much percentage regression in this case?
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Thomas Weise <[hidden email]>
> > > Send Time:2020年7月2日(星期四) 09:54
> > > To:dev <[hidden email]>
> > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > >
> > > Hi Till,
> > >
> > > Yes, we don't have the setting in flink-conf.yaml.
> > >
> > > Generally, we carry forward the existing configuration and any change
> to
> > > default configuration values would impact the upgrade.
> > >
> > > Yes, since it is an incompatible change I would state it in the release
> > > notes.
> > >
> > > Thanks,
> > > Thomas
> > >
> > > BTW I found a performance regression while trying to upgrade another
> > > pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't
> able
> > > to pin it down yet, symptoms include increased checkpoint alignment
> time.
> > >
> > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > just to confirm: When starting the image in local mode, then you
> don't
> > > have
> > > > any of the JobManager memory configuration settings configured in the
> > > > effective flink-conf.yaml, right? Does this mean that you have
> > explicitly
> > > > removed `jobmanager.heap.size: 1024m` from the default configuration?
> > If
> > > > this is the case, then I believe it was more of an unintentional
> > artifact
> > > > that it worked before and it has been corrected now so that one needs
> > to
> > > > specify the memory of the JM process explicitly. Do you think it
> would
> > > help
> > > > to explicitly state this in the release notes?
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]> wrote:
> > > >
> > > > > Thanks for preparing another RC!
> > > > >
> > > > > As mentioned in the previous RC thread, it would be super helpful
> if
> > > the
> > > > > release notes that are part of the documentation can be included
> [1].
> > > > It's
> > > > > a significant time-saver to have read those first.
> > > > >
> > > > > I found one more non-backward compatible change that would be worth
> > > > > addressing/mentioning:
> > > > >
> > > > > It is now necessary to configure the jobmanager heap size in
> > > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > > or jobmanager.memory.heap.size). Why would I not want to do that
> > > anyways?
> > > > > Well, we set it dynamically for a cluster deployment via the
> > > > > flinkk8soperator, but the container image can also be used for
> > testing
> > > > with
> > > > > local mode (./bin/jobmanager.sh start-foreground local). That will
> > fail
> > > > if
> > > > > the heap wasn't configured and that's how I noticed it.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > > >
> > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <
> [hidden email]
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Please review and vote on the release candidate #4 for the
> version
> > > > > 1.11.0,
> > > > > > as follows:
> > > > > > [ ] +1, Approve the release
> > > > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > > >
> > > > > > The complete staging area is available for your review, which
> > > includes:
> > > > > > * JIRA release notes [1],
> > > > > > * the official Apache source release and binary convenience
> > releases
> > > to
> > > > > be
> > > > > > deployed to dist.apache.org [2], which are signed with the key
> > with
> > > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > > * website pull request listing the new release and adding
> > > announcement
> > > > > > blog post [6].
> > > > > >
> > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > >
> > > > > > Thanks,
> > > > > > Release Manager
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > [4]
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > > [5]
> > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
>
>

Congxian Qiu

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding) % some NOTICE files missing

I found some projects do not contain NOTICE file (I'm not sure these
projects do not need NOTICE file or the NOTICE file is missing. I report
the findings here.).
The pom.xml change is based on the compare between
release-1.10.0..release-1.11-RC4[1], the dependency for below projects has
uploaded to gist[2]
- flink-connector-elasticsearch6
- flink-connector-elasticsearch7
- flink-connector-hbase
- flink-hcatlog
- flink-orc
- flink-parquet
- flink-sequence-file

checked
- sha512 & gpg check, ok
- build from source, ok
- all pom point to 1.11.0
- run some demos on a real cluster, ok
- manual test savepoint relocatable, ok
- some license check, apart from the NOTICE file, looks good to me.

[1] https://
github.com/apache/flink/compare/release-1.10.0..release-1.11.0-rc4
[2] https://gist.github.com/klion26/026a79897334fdeefec381cf7cdd5d93

Best,
Congxian

Steven Wu <[hidden email]> 于2020年7月5日周日上午1:41写道：

> +1 (non-binding)
>
> - rolled out to thousands of router jobs in our test env
> - tested with a large-state job. Did simple resilience and
> checkpoint/savepoint tests. General performance metrics look on par.
> - tested with a high-parallelism stateless transformation job. General
> performance metrics look on par.
>
> On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the further update information.
> >
> > I guess we can dismiss the network stack changes, since in your case the
> > downstream and upstream would probably be deployed in the same slot
> > bypassing the network data shuffle.
> > Also I guess release-1.11 will not bring general performance regression
> in
> > runtime engine, as we also did the performance testing for all general
> > cases by [1] in real cluster before and the testing results should fit
> the
> > expectation. But we indeed did not test the specific source and sink
> > connectors yet as I known.
> >
> > Regarding your performance regression with 40%, I wonder it is probably
> > related to specific source/sink changes (e.g. kinesis) or environment
> > issues with corner case.
> > If possible, it would be helpful to further locate whether the regression
> > is caused by kinesis, by replacing the kinesis source & sink and keeping
> > the others same.
> >
> > As you said, it would be efficient to contact with you directly next week
> > to further discuss this issue. And we are willing/eager to provide any
> help
> > to resolve this issue soon.
> >
> > Besides that, I guess this issue should not be the blocker for the
> > release, since it is probably a corner case based on the current
> analysis.
> > If we really conclude anything need to be resolved after the final
> > release, then we can also make the next minor release-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月4日(星期六) 12:26
> > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > Cc:Yingjie Cao <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Zhijiang,
> >
> > It will probably be best if we connect next week and discuss the issue
> > directly since this could be quite difficult to reproduce.
> >
> > Before the testing result on our side comes out for your respective job
> > case, I have some other questions to confirm for further analysis:
> > - How much percentage regression you found after switching to 1.11?
> >
> > ~40% throughput decline
> >
> > - Are there any network bottleneck in your cluster? E.g. the network
> > bandwidth is full caused by other jobs? If so, it might have more effects
> > by above [2]
> >
> > The test runs on a k8s cluster that is also used for other production
> jobs.
> > There is no reason be believe network is the bottleneck.
> >
> > - Did you adjust the default network buffer setting? E.g.
> > "taskmanager.network.memory.floating-buffers-per-gate" or
> > "taskmanager.network.memory.buffers-per-channel"
> >
> > The job is using the defaults, i.e we don't configure the settings. If
> you
> > want me to try specific settings in the hope that it will help to isolate
> > the issue please let me know.
> >
> > - I guess the topology has three vertexes "KinesisConsumer ->
> Chained
> > FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer
> ->
> > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> edge
> > connection is one-to-one, not all-to-all, then the above [1][2] should no
> > effects in theory with default network buffer setting.
> >
> > There are only 2 vertices and the edge is "forward".
> >
> > - By slot sharing, I guess these three vertex parallelism task would
> > probably be deployed into the same slot, then the data shuffle is by
> memory
> > queue, not network stack. If so, the above [2] should no effect.
> >
> > Yes, vertices share slots.
> >
> > - I also saw some Jira changes for kinesis in this release, could you
> > confirm that these changes would not effect the performance?
> >
> > I will need to take a look. 1.10 already had a regression introduced by
> the
> > Kinesis producer update.
> >
> >
> > Thanks,
> > Thomas
> >
> >
> > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]
> > .invalid>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > Thanks for your reply with rich information!
> > >
> > > We are trying to reproduce your case in our cluster to further verify
> it,
> > > and @Yingjie Cao is working on it now.
> > > As we have not kinesis consumer and producer internally, so we will
> > > construct the common source and sink instead in the case of
> backpressure.
> > >
> > > Firstly, we can dismiss the rockdb factor in this release, since you
> also
> > > mentioned that "filesystem leads to same symptoms".
> > >
> > > Secondly, if my understanding is right, you emphasis that the
> regression
> > > only exists for the jobs with low checkpoint interval (10s).
> > > Based on that, I have two suspicions with the network related changes
> in
> > > this release:
> > > - [1]: Limited the maximum backlog value (default 10) in
> subpartition
> > > queue.
> > > - [2]: Delay send the following buffers after checkpoint barrier on
> > > upstream side until barrier alignment on downstream side.
> > >
> > > These changes are motivated for reducing the in-flight buffers to
> speedup
> > > checkpoint especially in the case of backpressure.
> > > In theory they should have very minor performance effect and actually
> we
> > > also tested in cluster to verify within expectation before merging
> them,
> > > but maybe there are other corner cases we have not thought of before.
> > >
> > > Before the testing result on our side comes out for your respective job
> > > case, I have some other questions to confirm for further analysis:
> > > - How much percentage regression you found after switching to
> 1.11?
> > > - Are there any network bottleneck in your cluster? E.g. the
> network
> > > bandwidth is full caused by other jobs? If so, it might have more
> effects
> > > by above [2]
> > > - Did you adjust the default network buffer setting? E.g.
> > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > "taskmanager.network.memory.buffers-per-channel"
> > > - I guess the topology has three vertexes "KinesisConsumer ->
> > Chained
> > > FlatMap -> KinesisProducer", and the partition mode for
> "KinesisConsumer
> > ->
> > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> > edge
> > > connection is one-to-one, not all-to-all, then the above [1][2] should
> no
> > > effects in theory with default network buffer setting.
> > > - By slot sharing, I guess these three vertex parallelism task
> would
> > > probably be deployed into the same slot, then the data shuffle is by
> > memory
> > > queue, not network stack. If so, the above [2] should no effect.
> > > - I also saw some Jira changes for kinesis in this release, could
> you
> > > confirm that these changes would not effect the performance?
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Thomas Weise <[hidden email]>
> > > Send Time:2020年7月3日(星期五) 01:07
> > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > >
> > > Hi Zhijiang,
> > >
> > > The performance degradation manifests in backpressure which leads to
> > > growing backlog in the source. I switched a few times between 1.10 and
> > 1.11
> > > and the behavior is consistent.
> > >
> > > The DAG is:
> > >
> > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> > > ---------> KinesisProducer
> > >
> > > Parallelism: 160
> > > No shuffle/rebalance.
> > >
> > > Checkpointing config:
> > >
> > > Checkpointing Mode Exactly Once
> > > Interval 10s
> > > Timeout 10m 0s
> > > Minimum Pause Between Checkpoints 10s
> > > Maximum Concurrent Checkpoints 1
> > > Persist Checkpoints Externally Enabled (delete on cancellation)
> > >
> > > State backend: rocksdb (filesystem leads to same symptoms)
> > > Checkpoint size is tiny (500KB)
> > >
> > > An interesting difference to another job that I had upgraded
> successfully
> > > is the low checkpointing interval.
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> > > .invalid>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > Thanks for the efficient feedback.
> > > >
> > > > Regarding the suggestion of adding the release notes document, I
> agree
> > > > with your point. Maybe we should adjust the vote template accordingly
> > in
> > > > the respective wiki to guide the following release processes.
> > > >
> > > > Regarding the performance regression, could you provide some more
> > details
> > > > for our better measurement or reproducing on our sides?
> > > > E.g. I guess the topology only includes two vertexes source and sink?
> > > > What is the parallelism for every vertex?
> > > > The upstream shuffles data to the downstream via rebalance
> partitioner
> > or
> > > > other?
> > > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > > The backpressure happened in this case?
> > > > How much percentage regression in this case?
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Thomas Weise <[hidden email]>
> > > > Send Time:2020年7月2日(星期四) 09:54
> > > > To:dev <[hidden email]>
> > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > >
> > > > Hi Till,
> > > >
> > > > Yes, we don't have the setting in flink-conf.yaml.
> > > >
> > > > Generally, we carry forward the existing configuration and any change
> > to
> > > > default configuration values would impact the upgrade.
> > > >
> > > > Yes, since it is an incompatible change I would state it in the
> release
> > > > notes.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > BTW I found a performance regression while trying to upgrade another
> > > > pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't
> > able
> > > > to pin it down yet, symptoms include increased checkpoint alignment
> > time.
> > > >
> > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > just to confirm: When starting the image in local mode, then you
> > don't
> > > > have
> > > > > any of the JobManager memory configuration settings configured in
> the
> > > > > effective flink-conf.yaml, right? Does this mean that you have
> > > explicitly
> > > > > removed `jobmanager.heap.size: 1024m` from the default
> configuration?
> > > If
> > > > > this is the case, then I believe it was more of an unintentional
> > > artifact
> > > > > that it worked before and it has been corrected now so that one
> needs
> > > to
> > > > > specify the memory of the JM process explicitly. Do you think it
> > would
> > > > help
> > > > > to explicitly state this in the release notes?
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]>
> wrote:
> > > > >
> > > > > > Thanks for preparing another RC!
> > > > > >
> > > > > > As mentioned in the previous RC thread, it would be super helpful
> > if
> > > > the
> > > > > > release notes that are part of the documentation can be included
> > [1].
> > > > > It's
> > > > > > a significant time-saver to have read those first.
> > > > > >
> > > > > > I found one more non-backward compatible change that would be
> worth
> > > > > > addressing/mentioning:
> > > > > >
> > > > > > It is now necessary to configure the jobmanager heap size in
> > > > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > > > or jobmanager.memory.heap.size). Why would I not want to do that
> > > > anyways?
> > > > > > Well, we set it dynamically for a cluster deployment via the
> > > > > > flinkk8soperator, but the container image can also be used for
> > > testing
> > > > > with
> > > > > > local mode (./bin/jobmanager.sh start-foreground local). That
> will
> > > fail
> > > > > if
> > > > > > the heap wasn't configured and that's how I noticed it.
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > > > >
> > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <
> > [hidden email]
> > > > > > .invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Please review and vote on the release candidate #4 for the
> > version
> > > > > > 1.11.0,
> > > > > > > as follows:
> > > > > > > [ ] +1, Approve the release
> > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > > >
> > > > > > > The complete staging area is available for your review, which
> > > > includes:
> > > > > > > * JIRA release notes [1],
> > > > > > > * the official Apache source release and binary convenience
> > > releases
> > > > to
> > > > > > be
> > > > > > > deployed to dist.apache.org [2], which are signed with the key
> > > with
> > > > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > > > * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > > > * website pull request listing the new release and adding
> > > > announcement
> > > > > > > blog post [6].
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Release Manager
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > > > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > [4]
> > > > > > >
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > > > [5]
> > > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>

Thomas Weise

Re: [VOTE] Release 1.11.0, release candidate #4

In reply to this post by Zhijiang(wangzhijiang999)

Hi Zhijiang,

Could you please point me to more details regarding: "[2]: Delay send the
following buffers after checkpoint barrier on upstream side until barrier
alignment on downstream side."

In this case, the downstream task has a high average checkpoint duration
(~30s, sync part). If there was a change to hold buffers depending on
downstream performance, could this possibly apply to this case (even when
there is no shuffle that would require alignment)?

Thanks,
Thomas

On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]>
wrote:

Jark Wu-2

Re: [VOTE] Release 1.11.0, release candidate #4

+1

- started cluster and ran some examples, verified web ui and log output,
nothing unexpected, except ChangelogSocketExample which has been reported
[2] by Dawid.
- started cluster to run e2e SQL queries with millions of records with
Kafka, MySQL, Elasticsearch as sources/lookup/sinks. Works well and the
results are as expected.
- use SQL CLI to SELECT kafka source with debezium data, the result is as
expected.
- reviewed the release PR, left a comment [1] to add a note for
the FLINK-18461 issue.

Regarding the CDC issue FLINK-18461, the fix has been merged into
release-1.11 branch.
I think we have a conclusion to not block the RC by this issue. We can
quickly launch the next release-1.11.1 to cover it as Robert suggested.

Best,
Jark

[1]: https://github.com/apache/flink-web/pull/352/files#r449830524
[2]: https://issues.apache.org/jira/browse/FLINK-18477

On Sun, 5 Jul 2020 at 12:22, Thomas Weise <[hidden email]> wrote:

> Hi Zhijiang,
>
> Could you please point me to more details regarding: "[2]: Delay send the
> following buffers after checkpoint barrier on upstream side until barrier
> alignment on downstream side."
>
> In this case, the downstream task has a high average checkpoint duration
> (~30s, sync part). If there was a change to hold buffers depending on
> downstream performance, could this possibly apply to this case (even when
> there is no shuffle that would require alignment)?
>
> Thanks,
> Thomas
>
>
> On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the further update information.
> >
> > I guess we can dismiss the network stack changes, since in your case the
> > downstream and upstream would probably be deployed in the same slot
> > bypassing the network data shuffle.
> > Also I guess release-1.11 will not bring general performance regression
> in
> > runtime engine, as we also did the performance testing for all general
> > cases by [1] in real cluster before and the testing results should fit
> the
> > expectation. But we indeed did not test the specific source and sink
> > connectors yet as I known.
> >
> > Regarding your performance regression with 40%, I wonder it is probably
> > related to specific source/sink changes (e.g. kinesis) or environment
> > issues with corner case.
> > If possible, it would be helpful to further locate whether the regression
> > is caused by kinesis, by replacing the kinesis source & sink and keeping
> > the others same.
> >
> > As you said, it would be efficient to contact with you directly next week
> > to further discuss this issue. And we are willing/eager to provide any
> help
> > to resolve this issue soon.
> >
> > Besides that, I guess this issue should not be the blocker for the
> > release, since it is probably a corner case based on the current
> analysis.
> > If we really conclude anything need to be resolved after the final
> > release, then we can also make the next minor release-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月4日(星期六) 12:26
> > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > Cc:Yingjie Cao <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Zhijiang,
> >
> > It will probably be best if we connect next week and discuss the issue
> > directly since this could be quite difficult to reproduce.
> >
> > Before the testing result on our side comes out for your respective job
> > case, I have some other questions to confirm for further analysis:
> > - How much percentage regression you found after switching to 1.11?
> >
> > ~40% throughput decline
> >
> > - Are there any network bottleneck in your cluster? E.g. the network
> > bandwidth is full caused by other jobs? If so, it might have more effects
> > by above [2]
> >
> > The test runs on a k8s cluster that is also used for other production
> jobs.
> > There is no reason be believe network is the bottleneck.
> >
> > - Did you adjust the default network buffer setting? E.g.
> > "taskmanager.network.memory.floating-buffers-per-gate" or
> > "taskmanager.network.memory.buffers-per-channel"
> >
> > The job is using the defaults, i.e we don't configure the settings. If
> you
> > want me to try specific settings in the hope that it will help to isolate
> > the issue please let me know.
> >
> > - I guess the topology has three vertexes "KinesisConsumer ->
> Chained
> > FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer
> ->
> > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> edge
> > connection is one-to-one, not all-to-all, then the above [1][2] should no
> > effects in theory with default network buffer setting.
> >
> > There are only 2 vertices and the edge is "forward".
> >
> > - By slot sharing, I guess these three vertex parallelism task would
> > probably be deployed into the same slot, then the data shuffle is by
> memory
> > queue, not network stack. If so, the above [2] should no effect.
> >
> > Yes, vertices share slots.
> >
> > - I also saw some Jira changes for kinesis in this release, could you
> > confirm that these changes would not effect the performance?
> >
> > I will need to take a look. 1.10 already had a regression introduced by
> the
> > Kinesis producer update.
> >
> >
> > Thanks,
> > Thomas
> >
> >
> > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]
> > .invalid>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > Thanks for your reply with rich information!
> > >
> > > We are trying to reproduce your case in our cluster to further verify
> it,
> > > and @Yingjie Cao is working on it now.
> > > As we have not kinesis consumer and producer internally, so we will
> > > construct the common source and sink instead in the case of
> backpressure.
> > >
> > > Firstly, we can dismiss the rockdb factor in this release, since you
> also
> > > mentioned that "filesystem leads to same symptoms".
> > >
> > > Secondly, if my understanding is right, you emphasis that the
> regression
> > > only exists for the jobs with low checkpoint interval (10s).
> > > Based on that, I have two suspicions with the network related changes
> in
> > > this release:
> > > - [1]: Limited the maximum backlog value (default 10) in
> subpartition
> > > queue.
> > > - [2]: Delay send the following buffers after checkpoint barrier on
> > > upstream side until barrier alignment on downstream side.
> > >
> > > These changes are motivated for reducing the in-flight buffers to
> speedup
> > > checkpoint especially in the case of backpressure.
> > > In theory they should have very minor performance effect and actually
> we
> > > also tested in cluster to verify within expectation before merging
> them,
> > > but maybe there are other corner cases we have not thought of before.
> > >
> > > Before the testing result on our side comes out for your respective job
> > > case, I have some other questions to confirm for further analysis:
> > > - How much percentage regression you found after switching to
> 1.11?
> > > - Are there any network bottleneck in your cluster? E.g. the
> network
> > > bandwidth is full caused by other jobs? If so, it might have more
> effects
> > > by above [2]
> > > - Did you adjust the default network buffer setting? E.g.
> > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > "taskmanager.network.memory.buffers-per-channel"
> > > - I guess the topology has three vertexes "KinesisConsumer ->
> > Chained
> > > FlatMap -> KinesisProducer", and the partition mode for
> "KinesisConsumer
> > ->
> > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> > edge
> > > connection is one-to-one, not all-to-all, then the above [1][2] should
> no
> > > effects in theory with default network buffer setting.
> > > - By slot sharing, I guess these three vertex parallelism task
> would
> > > probably be deployed into the same slot, then the data shuffle is by
> > memory
> > > queue, not network stack. If so, the above [2] should no effect.
> > > - I also saw some Jira changes for kinesis in this release, could
> you
> > > confirm that these changes would not effect the performance?
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Thomas Weise <[hidden email]>
> > > Send Time:2020年7月3日(星期五) 01:07
> > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > >
> > > Hi Zhijiang,
> > >
> > > The performance degradation manifests in backpressure which leads to
> > > growing backlog in the source. I switched a few times between 1.10 and
> > 1.11
> > > and the behavior is consistent.
> > >
> > > The DAG is:
> > >
> > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> > > ---------> KinesisProducer
> > >
> > > Parallelism: 160
> > > No shuffle/rebalance.
> > >
> > > Checkpointing config:
> > >
> > > Checkpointing Mode Exactly Once
> > > Interval 10s
> > > Timeout 10m 0s
> > > Minimum Pause Between Checkpoints 10s
> > > Maximum Concurrent Checkpoints 1
> > > Persist Checkpoints Externally Enabled (delete on cancellation)
> > >
> > > State backend: rocksdb (filesystem leads to same symptoms)
> > > Checkpoint size is tiny (500KB)
> > >
> > > An interesting difference to another job that I had upgraded
> successfully
> > > is the low checkpointing interval.
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> > > .invalid>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > Thanks for the efficient feedback.
> > > >
> > > > Regarding the suggestion of adding the release notes document, I
> agree
> > > > with your point. Maybe we should adjust the vote template accordingly
> > in
> > > > the respective wiki to guide the following release processes.
> > > >
> > > > Regarding the performance regression, could you provide some more
> > details
> > > > for our better measurement or reproducing on our sides?
> > > > E.g. I guess the topology only includes two vertexes source and sink?
> > > > What is the parallelism for every vertex?
> > > > The upstream shuffles data to the downstream via rebalance
> partitioner
> > or
> > > > other?
> > > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > > The backpressure happened in this case?
> > > > How much percentage regression in this case?
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Thomas Weise <[hidden email]>
> > > > Send Time:2020年7月2日(星期四) 09:54
> > > > To:dev <[hidden email]>
> > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > >
> > > > Hi Till,
> > > >
> > > > Yes, we don't have the setting in flink-conf.yaml.
> > > >
> > > > Generally, we carry forward the existing configuration and any change
> > to
> > > > default configuration values would impact the upgrade.
> > > >
> > > > Yes, since it is an incompatible change I would state it in the
> release
> > > > notes.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > BTW I found a performance regression while trying to upgrade another
> > > > pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't
> > able
> > > > to pin it down yet, symptoms include increased checkpoint alignment
> > time.
> > > >
> > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > just to confirm: When starting the image in local mode, then you
> > don't
> > > > have
> > > > > any of the JobManager memory configuration settings configured in
> the
> > > > > effective flink-conf.yaml, right? Does this mean that you have
> > > explicitly
> > > > > removed `jobmanager.heap.size: 1024m` from the default
> configuration?
> > > If
> > > > > this is the case, then I believe it was more of an unintentional
> > > artifact
> > > > > that it worked before and it has been corrected now so that one
> needs
> > > to
> > > > > specify the memory of the JM process explicitly. Do you think it
> > would
> > > > help
> > > > > to explicitly state this in the release notes?
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]>
> wrote:
> > > > >
> > > > > > Thanks for preparing another RC!
> > > > > >
> > > > > > As mentioned in the previous RC thread, it would be super helpful
> > if
> > > > the
> > > > > > release notes that are part of the documentation can be included
> > [1].
> > > > > It's
> > > > > > a significant time-saver to have read those first.
> > > > > >
> > > > > > I found one more non-backward compatible change that would be
> worth
> > > > > > addressing/mentioning:
> > > > > >
> > > > > > It is now necessary to configure the jobmanager heap size in
> > > > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > > > or jobmanager.memory.heap.size). Why would I not want to do that
> > > > anyways?
> > > > > > Well, we set it dynamically for a cluster deployment via the
> > > > > > flinkk8soperator, but the container image can also be used for
> > > testing
> > > > > with
> > > > > > local mode (./bin/jobmanager.sh start-foreground local). That
> will
> > > fail
> > > > > if
> > > > > > the heap wasn't configured and that's how I noticed it.
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > > > >
> > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <
> > [hidden email]
> > > > > > .invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Please review and vote on the release candidate #4 for the
> > version
> > > > > > 1.11.0,
> > > > > > > as follows:
> > > > > > > [ ] +1, Approve the release
> > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > > >
> > > > > > > The complete staging area is available for your review, which
> > > > includes:
> > > > > > > * JIRA release notes [1],
> > > > > > > * the official Apache source release and binary convenience
> > > releases
> > > > to
> > > > > > be
> > > > > > > deployed to dist.apache.org [2], which are signed with the key
> > > with
> > > > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > > > * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > > > * website pull request listing the new release and adding
> > > > announcement
> > > > > > > blog post [6].
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Release Manager
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > > > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > [4]
> > > > > > >
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > > > [5]
> > > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>

Zhijiang(wangzhijiang999)

Re: [VOTE] Release 1.11.0, release candidate #4

In reply to this post by Thomas Weise

Hi Thomas,

Regarding [2], it has more detail infos in the Jira description (https://issues.apache.org/jira/browse/FLINK-16404).

I can also give some basic explanations here to dismiss the concern.
1. In the past, the following buffers after the barrier will be cached on downstream side before alignment.
2. In 1.11, the upstream would not send the buffers after the barrier. When the downstream finishes the alignment, it will notify the downstream of continuing sending following buffers, since it can process them after alignment.
3. The only difference is that the temporary blocked buffers are cached either on downstream side or on upstream side before alignment.
4. The side effect would be the additional notification cost for every barrier alignment. If the downstream and upstream are deployed in separate TaskManager, the cost is network transport delay (the effect can be ignored based on our testing with 1s checkpoint interval). For sharing slot in your case, the cost is only one method call in processor, can be ignored also.

You mentioned "In this case, the downstream task has a high average checkpoint duration(~30s, sync part)." This duration is not reflecting the changes above, and it is only indicating the duration for calling `Operation.snapshotState`.
If this duration is beyond your expectation, you can check or debug whether the source/sink operations might take more time to finish `snapshotState` in practice. E.g. you can
make the implementation of this method as empty to further verify the effect.

Best,
Zhijiang

------------------------------------------------------------------
From:Thomas Weise <[hidden email]>
Send Time:2020年7月5日(星期日) 12:22
To:dev <[hidden email]>; Zhijiang <[hidden email]>
Cc:Yingjie Cao <[hidden email]>
Subject:Re: [VOTE] Release 1.11.0, release candidate #4

Hi Zhijiang,

Could you please point me to more details regarding: "[2]: Delay send the
following buffers after checkpoint barrier on upstream side until barrier
alignment on downstream side."

In this case, the downstream task has a high average checkpoint duration
(~30s, sync part). If there was a change to hold buffers depending on
downstream performance, could this possibly apply to this case (even when
there is no shuffle that would require alignment)?

Thanks,
Thomas

On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]>
wrote:

Benchao Li-2

Re: [VOTE] Release 1.11.0, release candidate #4

+1 （non-binding)

Checks:
- verified signature and shasum of release files [OK]
- build from source [OK]
- started standalone cluster, sql-client [mostly OK except one issue]
- played with sql-client
- played with new features: LIKE / Table Options
- checked Web UI functionality
- canceled job from UI

While I'm playing with the new table factories, I found one issue[1] which
surprises me.
I don't think this should be a blocker, hence I'll still vote my +1.

[1] https://issues.apache.org/jira/browse/FLINK-18487

Zhijiang <[hidden email]> 于2020年7月5日周日下午1:10写道：

> Hi Thomas,
>
> Regarding [2], it has more detail infos in the Jira description (
> https://issues.apache.org/jira/browse/FLINK-16404).
>
> I can also give some basic explanations here to dismiss the concern.
> 1. In the past, the following buffers after the barrier will be cached on
> downstream side before alignment.
> 2. In 1.11, the upstream would not send the buffers after the barrier.
> When the downstream finishes the alignment, it will notify the downstream
> of continuing sending following buffers, since it can process them after
> alignment.
> 3. The only difference is that the temporary blocked buffers are cached
> either on downstream side or on upstream side before alignment.
> 4. The side effect would be the additional notification cost for every
> barrier alignment. If the downstream and upstream are deployed in separate
> TaskManager, the cost is network transport delay (the effect can be ignored
> based on our testing with 1s checkpoint interval). For sharing slot in your
> case, the cost is only one method call in processor, can be ignored also.
>
> You mentioned "In this case, the downstream task has a high average
> checkpoint duration(~30s, sync part)." This duration is not reflecting the
> changes above, and it is only indicating the duration for calling
> `Operation.snapshotState`.
> If this duration is beyond your expectation, you can check or debug
> whether the source/sink operations might take more time to finish
> `snapshotState` in practice. E.g. you can
> make the implementation of this method as empty to further verify the
> effect.
>
> Best,
> Zhijiang
>
>
> ------------------------------------------------------------------
> From:Thomas Weise <[hidden email]>
> Send Time:2020年7月5日(星期日) 12:22
> To:dev <[hidden email]>; Zhijiang <[hidden email]>
> Cc:Yingjie Cao <[hidden email]>
> Subject:Re: [VOTE] Release 1.11.0, release candidate #4
>
> Hi Zhijiang,
>
> Could you please point me to more details regarding: "[2]: Delay send the
> following buffers after checkpoint barrier on upstream side until barrier
> alignment on downstream side."
>
> In this case, the downstream task has a high average checkpoint duration
> (~30s, sync part). If there was a change to hold buffers depending on
> downstream performance, could this possibly apply to this case (even when
> there is no shuffle that would require alignment)?
>
> Thanks,
> Thomas
>
>
> On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the further update information.
> >
> > I guess we can dismiss the network stack changes, since in your case the
> > downstream and upstream would probably be deployed in the same slot
> > bypassing the network data shuffle.
> > Also I guess release-1.11 will not bring general performance regression
> in
> > runtime engine, as we also did the performance testing for all general
> > cases by [1] in real cluster before and the testing results should fit
> the
> > expectation. But we indeed did not test the specific source and sink
> > connectors yet as I known.
> >
> > Regarding your performance regression with 40%, I wonder it is probably
> > related to specific source/sink changes (e.g. kinesis) or environment
> > issues with corner case.
> > If possible, it would be helpful to further locate whether the regression
> > is caused by kinesis, by replacing the kinesis source & sink and keeping
> > the others same.
> >
> > As you said, it would be efficient to contact with you directly next week
> > to further discuss this issue. And we are willing/eager to provide any
> help
> > to resolve this issue soon.
> >
> > Besides that, I guess this issue should not be the blocker for the
> > release, since it is probably a corner case based on the current
> analysis.
> > If we really conclude anything need to be resolved after the final
> > release, then we can also make the next minor release-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月4日(星期六) 12:26
> > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > Cc:Yingjie Cao <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Zhijiang,
> >
> > It will probably be best if we connect next week and discuss the issue
> > directly since this could be quite difficult to reproduce.
> >
> > Before the testing result on our side comes out for your respective job
> > case, I have some other questions to confirm for further analysis:
> > - How much percentage regression you found after switching to 1.11?
> >
> > ~40% throughput decline
> >
> > - Are there any network bottleneck in your cluster? E.g. the network
> > bandwidth is full caused by other jobs? If so, it might have more effects
> > by above [2]
> >
> > The test runs on a k8s cluster that is also used for other production
> jobs.
> > There is no reason be believe network is the bottleneck.
> >
> > - Did you adjust the default network buffer setting? E.g.
> > "taskmanager.network.memory.floating-buffers-per-gate" or
> > "taskmanager.network.memory.buffers-per-channel"
> >
> > The job is using the defaults, i.e we don't configure the settings. If
> you
> > want me to try specific settings in the hope that it will help to isolate
> > the issue please let me know.
> >
> > - I guess the topology has three vertexes "KinesisConsumer ->
> Chained
> > FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer
> ->
> > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> edge
> > connection is one-to-one, not all-to-all, then the above [1][2] should no
> > effects in theory with default network buffer setting.
> >
> > There are only 2 vertices and the edge is "forward".
> >
> > - By slot sharing, I guess these three vertex parallelism task would
> > probably be deployed into the same slot, then the data shuffle is by
> memory
> > queue, not network stack. If so, the above [2] should no effect.
> >
> > Yes, vertices share slots.
> >
> > - I also saw some Jira changes for kinesis in this release, could you
> > confirm that these changes would not effect the performance?
> >
> > I will need to take a look. 1.10 already had a regression introduced by
> the
> > Kinesis producer update.
> >
> >
> > Thanks,
> > Thomas
> >
> >
> > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]
> > .invalid>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > Thanks for your reply with rich information!
> > >
> > > We are trying to reproduce your case in our cluster to further verify
> it,
> > > and @Yingjie Cao is working on it now.
> > > As we have not kinesis consumer and producer internally, so we will
> > > construct the common source and sink instead in the case of
> backpressure.
> > >
> > > Firstly, we can dismiss the rockdb factor in this release, since you
> also
> > > mentioned that "filesystem leads to same symptoms".
> > >
> > > Secondly, if my understanding is right, you emphasis that the
> regression
> > > only exists for the jobs with low checkpoint interval (10s).
> > > Based on that, I have two suspicions with the network related changes
> in
> > > this release:
> > > - [1]: Limited the maximum backlog value (default 10) in
> subpartition
> > > queue.
> > > - [2]: Delay send the following buffers after checkpoint barrier on
> > > upstream side until barrier alignment on downstream side.
> > >
> > > These changes are motivated for reducing the in-flight buffers to
> speedup
> > > checkpoint especially in the case of backpressure.
> > > In theory they should have very minor performance effect and actually
> we
> > > also tested in cluster to verify within expectation before merging
> them,
> > > but maybe there are other corner cases we have not thought of before.
> > >
> > > Before the testing result on our side comes out for your respective job
> > > case, I have some other questions to confirm for further analysis:
> > > - How much percentage regression you found after switching to
> 1.11?
> > > - Are there any network bottleneck in your cluster? E.g. the
> network
> > > bandwidth is full caused by other jobs? If so, it might have more
> effects
> > > by above [2]
> > > - Did you adjust the default network buffer setting? E.g.
> > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > "taskmanager.network.memory.buffers-per-channel"
> > > - I guess the topology has three vertexes "KinesisConsumer ->
> > Chained
> > > FlatMap -> KinesisProducer", and the partition mode for
> "KinesisConsumer
> > ->
> > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> > edge
> > > connection is one-to-one, not all-to-all, then the above [1][2] should
> no
> > > effects in theory with default network buffer setting.
> > > - By slot sharing, I guess these three vertex parallelism task
> would
> > > probably be deployed into the same slot, then the data shuffle is by
> > memory
> > > queue, not network stack. If so, the above [2] should no effect.
> > > - I also saw some Jira changes for kinesis in this release, could
> you
> > > confirm that these changes would not effect the performance?
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Thomas Weise <[hidden email]>
> > > Send Time:2020年7月3日(星期五) 01:07
> > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > >
> > > Hi Zhijiang,
> > >
> > > The performance degradation manifests in backpressure which leads to
> > > growing backlog in the source. I switched a few times between 1.10 and
> > 1.11
> > > and the behavior is consistent.
> > >
> > > The DAG is:
> > >
> > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> > > ---------> KinesisProducer
> > >
> > > Parallelism: 160
> > > No shuffle/rebalance.
> > >
> > > Checkpointing config:
> > >
> > > Checkpointing Mode Exactly Once
> > > Interval 10s
> > > Timeout 10m 0s
> > > Minimum Pause Between Checkpoints 10s
> > > Maximum Concurrent Checkpoints 1
> > > Persist Checkpoints Externally Enabled (delete on cancellation)
> > >
> > > State backend: rocksdb (filesystem leads to same symptoms)
> > > Checkpoint size is tiny (500KB)
> > >
> > > An interesting difference to another job that I had upgraded
> successfully
> > > is the low checkpointing interval.
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> > > .invalid>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > Thanks for the efficient feedback.
> > > >
> > > > Regarding the suggestion of adding the release notes document, I
> agree
> > > > with your point. Maybe we should adjust the vote template accordingly
> > in
> > > > the respective wiki to guide the following release processes.
> > > >
> > > > Regarding the performance regression, could you provide some more
> > details
> > > > for our better measurement or reproducing on our sides?
> > > > E.g. I guess the topology only includes two vertexes source and sink?
> > > > What is the parallelism for every vertex?
> > > > The upstream shuffles data to the downstream via rebalance
> partitioner
> > or
> > > > other?
> > > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > > The backpressure happened in this case?
> > > > How much percentage regression in this case?
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Thomas Weise <[hidden email]>
> > > > Send Time:2020年7月2日(星期四) 09:54
> > > > To:dev <[hidden email]>
> > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > >
> > > > Hi Till,
> > > >
> > > > Yes, we don't have the setting in flink-conf.yaml.
> > > >
> > > > Generally, we carry forward the existing configuration and any change
> > to
> > > > default configuration values would impact the upgrade.
> > > >
> > > > Yes, since it is an incompatible change I would state it in the
> release
> > > > notes.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > BTW I found a performance regression while trying to upgrade another
> > > > pipeline with this RC. It is a simple Kinesis to Kinesis job. Wasn't
> > able
> > > > to pin it down yet, symptoms include increased checkpoint alignment
> > time.
> > > >
> > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <[hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > just to confirm: When starting the image in local mode, then you
> > don't
> > > > have
> > > > > any of the JobManager memory configuration settings configured in
> the
> > > > > effective flink-conf.yaml, right? Does this mean that you have
> > > explicitly
> > > > > removed `jobmanager.heap.size: 1024m` from the default
> configuration?
> > > If
> > > > > this is the case, then I believe it was more of an unintentional
> > > artifact
> > > > > that it worked before and it has been corrected now so that one
> needs
> > > to
> > > > > specify the memory of the JM process explicitly. Do you think it
> > would
> > > > help
> > > > > to explicitly state this in the release notes?
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]>
> wrote:
> > > > >
> > > > > > Thanks for preparing another RC!
> > > > > >
> > > > > > As mentioned in the previous RC thread, it would be super helpful
> > if
> > > > the
> > > > > > release notes that are part of the documentation can be included
> > [1].
> > > > > It's
> > > > > > a significant time-saver to have read those first.
> > > > > >
> > > > > > I found one more non-backward compatible change that would be
> worth
> > > > > > addressing/mentioning:
> > > > > >
> > > > > > It is now necessary to configure the jobmanager heap size in
> > > > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > > > or jobmanager.memory.heap.size). Why would I not want to do that
> > > > anyways?
> > > > > > Well, we set it dynamically for a cluster deployment via the
> > > > > > flinkk8soperator, but the container image can also be used for
> > > testing
> > > > > with
> > > > > > local mode (./bin/jobmanager.sh start-foreground local). That
> will
> > > fail
> > > > > if
> > > > > > the heap wasn't configured and that's how I noticed it.
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > > > >
> > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <
> > [hidden email]
> > > > > > .invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Please review and vote on the release candidate #4 for the
> > version
> > > > > > 1.11.0,
> > > > > > > as follows:
> > > > > > > [ ] +1, Approve the release
> > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > > >
> > > > > > > The complete staging area is available for your review, which
> > > > includes:
> > > > > > > * JIRA release notes [1],
> > > > > > > * the official Apache source release and binary convenience
> > > releases
> > > > to
> > > > > > be
> > > > > > > deployed to dist.apache.org [2], which are signed with the key
> > > with
> > > > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > > > * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > > > * website pull request listing the new release and adding
> > > > announcement
> > > > > > > blog post [6].
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Release Manager
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > > > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > [4]
> > > > > > >
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > > > [5]
> > > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

--

Best,
Benchao Li

Xintong Song

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding)

- verified signature and checksum
- build from source
- checked log sanity
- checked webui
- played with memory configurations
- played with binding addresses/ports

Thank you~

Xintong Song

On Sun, Jul 5, 2020 at 9:41 PM Benchao Li <[hidden email]> wrote:

> +1 （non-binding)
>
> Checks:
> - verified signature and shasum of release files [OK]
> - build from source [OK]
> - started standalone cluster, sql-client [mostly OK except one issue]
> - played with sql-client
> - played with new features: LIKE / Table Options
> - checked Web UI functionality
> - canceled job from UI
>
> While I'm playing with the new table factories, I found one issue[1] which
> surprises me.
> I don't think this should be a blocker, hence I'll still vote my +1.
>
> [1] https://issues.apache.org/jira/browse/FLINK-18487
>
> Zhijiang <[hidden email]> 于2020年7月5日周日下午1:10写道：
>
> > Hi Thomas,
> >
> > Regarding [2], it has more detail infos in the Jira description (
> > https://issues.apache.org/jira/browse/FLINK-16404).
> >
> > I can also give some basic explanations here to dismiss the concern.
> > 1. In the past, the following buffers after the barrier will be cached on
> > downstream side before alignment.
> > 2. In 1.11, the upstream would not send the buffers after the barrier.
> > When the downstream finishes the alignment, it will notify the downstream
> > of continuing sending following buffers, since it can process them after
> > alignment.
> > 3. The only difference is that the temporary blocked buffers are cached
> > either on downstream side or on upstream side before alignment.
> > 4. The side effect would be the additional notification cost for every
> > barrier alignment. If the downstream and upstream are deployed in
> separate
> > TaskManager, the cost is network transport delay (the effect can be
> ignored
> > based on our testing with 1s checkpoint interval). For sharing slot in
> your
> > case, the cost is only one method call in processor, can be ignored also.
> >
> > You mentioned "In this case, the downstream task has a high average
> > checkpoint duration(~30s, sync part)." This duration is not reflecting
> the
> > changes above, and it is only indicating the duration for calling
> > `Operation.snapshotState`.
> > If this duration is beyond your expectation, you can check or debug
> > whether the source/sink operations might take more time to finish
> > `snapshotState` in practice. E.g. you can
> > make the implementation of this method as empty to further verify the
> > effect.
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Thomas Weise <[hidden email]>
> > Send Time:2020年7月5日(星期日) 12:22
> > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > Cc:Yingjie Cao <[hidden email]>
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Zhijiang,
> >
> > Could you please point me to more details regarding: "[2]: Delay send the
> > following buffers after checkpoint barrier on upstream side until barrier
> > alignment on downstream side."
> >
> > In this case, the downstream task has a high average checkpoint duration
> > (~30s, sync part). If there was a change to hold buffers depending on
> > downstream performance, could this possibly apply to this case (even when
> > there is no shuffle that would require alignment)?
> >
> > Thanks,
> > Thomas
> >
> >
> > On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]
> > .invalid>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > Thanks for the further update information.
> > >
> > > I guess we can dismiss the network stack changes, since in your case
> the
> > > downstream and upstream would probably be deployed in the same slot
> > > bypassing the network data shuffle.
> > > Also I guess release-1.11 will not bring general performance regression
> > in
> > > runtime engine, as we also did the performance testing for all general
> > > cases by [1] in real cluster before and the testing results should fit
> > the
> > > expectation. But we indeed did not test the specific source and sink
> > > connectors yet as I known.
> > >
> > > Regarding your performance regression with 40%, I wonder it is probably
> > > related to specific source/sink changes (e.g. kinesis) or environment
> > > issues with corner case.
> > > If possible, it would be helpful to further locate whether the
> regression
> > > is caused by kinesis, by replacing the kinesis source & sink and
> keeping
> > > the others same.
> > >
> > > As you said, it would be efficient to contact with you directly next
> week
> > > to further discuss this issue. And we are willing/eager to provide any
> > help
> > > to resolve this issue soon.
> > >
> > > Besides that, I guess this issue should not be the blocker for the
> > > release, since it is probably a corner case based on the current
> > analysis.
> > > If we really conclude anything need to be resolved after the final
> > > release, then we can also make the next minor release-1.11.1 come soon.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-18433
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Thomas Weise <[hidden email]>
> > > Send Time:2020年7月4日(星期六) 12:26
> > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > Cc:Yingjie Cao <[hidden email]>
> > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > >
> > > Hi Zhijiang,
> > >
> > > It will probably be best if we connect next week and discuss the issue
> > > directly since this could be quite difficult to reproduce.
> > >
> > > Before the testing result on our side comes out for your respective job
> > > case, I have some other questions to confirm for further analysis:
> > > - How much percentage regression you found after switching to
> 1.11?
> > >
> > > ~40% throughput decline
> > >
> > > - Are there any network bottleneck in your cluster? E.g. the
> network
> > > bandwidth is full caused by other jobs? If so, it might have more
> effects
> > > by above [2]
> > >
> > > The test runs on a k8s cluster that is also used for other production
> > jobs.
> > > There is no reason be believe network is the bottleneck.
> > >
> > > - Did you adjust the default network buffer setting? E.g.
> > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > "taskmanager.network.memory.buffers-per-channel"
> > >
> > > The job is using the defaults, i.e we don't configure the settings. If
> > you
> > > want me to try specific settings in the hope that it will help to
> isolate
> > > the issue please let me know.
> > >
> > > - I guess the topology has three vertexes "KinesisConsumer ->
> > Chained
> > > FlatMap -> KinesisProducer", and the partition mode for
> "KinesisConsumer
> > ->
> > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> > edge
> > > connection is one-to-one, not all-to-all, then the above [1][2] should
> no
> > > effects in theory with default network buffer setting.
> > >
> > > There are only 2 vertices and the edge is "forward".
> > >
> > > - By slot sharing, I guess these three vertex parallelism task
> would
> > > probably be deployed into the same slot, then the data shuffle is by
> > memory
> > > queue, not network stack. If so, the above [2] should no effect.
> > >
> > > Yes, vertices share slots.
> > >
> > > - I also saw some Jira changes for kinesis in this release, could
> you
> > > confirm that these changes would not effect the performance?
> > >
> > > I will need to take a look. 1.10 already had a regression introduced by
> > the
> > > Kinesis producer update.
> > >
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]
> > > .invalid>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > Thanks for your reply with rich information!
> > > >
> > > > We are trying to reproduce your case in our cluster to further verify
> > it,
> > > > and @Yingjie Cao is working on it now.
> > > > As we have not kinesis consumer and producer internally, so we will
> > > > construct the common source and sink instead in the case of
> > backpressure.
> > > >
> > > > Firstly, we can dismiss the rockdb factor in this release, since you
> > also
> > > > mentioned that "filesystem leads to same symptoms".
> > > >
> > > > Secondly, if my understanding is right, you emphasis that the
> > regression
> > > > only exists for the jobs with low checkpoint interval (10s).
> > > > Based on that, I have two suspicions with the network related changes
> > in
> > > > this release:
> > > > - [1]: Limited the maximum backlog value (default 10) in
> > subpartition
> > > > queue.
> > > > - [2]: Delay send the following buffers after checkpoint barrier
> on
> > > > upstream side until barrier alignment on downstream side.
> > > >
> > > > These changes are motivated for reducing the in-flight buffers to
> > speedup
> > > > checkpoint especially in the case of backpressure.
> > > > In theory they should have very minor performance effect and actually
> > we
> > > > also tested in cluster to verify within expectation before merging
> > them,
> > > > but maybe there are other corner cases we have not thought of
> before.
> > > >
> > > > Before the testing result on our side comes out for your respective
> job
> > > > case, I have some other questions to confirm for further analysis:
> > > > - How much percentage regression you found after switching to
> > 1.11?
> > > > - Are there any network bottleneck in your cluster? E.g. the
> > network
> > > > bandwidth is full caused by other jobs? If so, it might have more
> > effects
> > > > by above [2]
> > > > - Did you adjust the default network buffer setting? E.g.
> > > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > > "taskmanager.network.memory.buffers-per-channel"
> > > > - I guess the topology has three vertexes "KinesisConsumer ->
> > > Chained
> > > > FlatMap -> KinesisProducer", and the partition mode for
> > "KinesisConsumer
> > > ->
> > > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so,
> the
> > > edge
> > > > connection is one-to-one, not all-to-all, then the above [1][2]
> should
> > no
> > > > effects in theory with default network buffer setting.
> > > > - By slot sharing, I guess these three vertex parallelism task
> > would
> > > > probably be deployed into the same slot, then the data shuffle is by
> > > memory
> > > > queue, not network stack. If so, the above [2] should no effect.
> > > > - I also saw some Jira changes for kinesis in this release, could
> > you
> > > > confirm that these changes would not effect the performance?
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Thomas Weise <[hidden email]>
> > > > Send Time:2020年7月3日(星期五) 01:07
> > > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > >
> > > > Hi Zhijiang,
> > > >
> > > > The performance degradation manifests in backpressure which leads to
> > > > growing backlog in the source. I switched a few times between 1.10
> and
> > > 1.11
> > > > and the behavior is consistent.
> > > >
> > > > The DAG is:
> > > >
> > > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) -------- forward
> > > > ---------> KinesisProducer
> > > >
> > > > Parallelism: 160
> > > > No shuffle/rebalance.
> > > >
> > > > Checkpointing config:
> > > >
> > > > Checkpointing Mode Exactly Once
> > > > Interval 10s
> > > > Timeout 10m 0s
> > > > Minimum Pause Between Checkpoints 10s
> > > > Maximum Concurrent Checkpoints 1
> > > > Persist Checkpoints Externally Enabled (delete on cancellation)
> > > >
> > > > State backend: rocksdb (filesystem leads to same symptoms)
> > > > Checkpoint size is tiny (500KB)
> > > >
> > > > An interesting difference to another job that I had upgraded
> > successfully
> > > > is the low checkpointing interval.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <[hidden email]
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > Thanks for the efficient feedback.
> > > > >
> > > > > Regarding the suggestion of adding the release notes document, I
> > agree
> > > > > with your point. Maybe we should adjust the vote template
> accordingly
> > > in
> > > > > the respective wiki to guide the following release processes.
> > > > >
> > > > > Regarding the performance regression, could you provide some more
> > > details
> > > > > for our better measurement or reproducing on our sides?
> > > > > E.g. I guess the topology only includes two vertexes source and
> sink?
> > > > > What is the parallelism for every vertex?
> > > > > The upstream shuffles data to the downstream via rebalance
> > partitioner
> > > or
> > > > > other?
> > > > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > > > The backpressure happened in this case?
> > > > > How much percentage regression in this case?
> > > > >
> > > > > Best,
> > > > > Zhijiang
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------------------------------------------
> > > > > From:Thomas Weise <[hidden email]>
> > > > > Send Time:2020年7月2日(星期四) 09:54
> > > > > To:dev <[hidden email]>
> > > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > > >
> > > > > Hi Till,
> > > > >
> > > > > Yes, we don't have the setting in flink-conf.yaml.
> > > > >
> > > > > Generally, we carry forward the existing configuration and any
> change
> > > to
> > > > > default configuration values would impact the upgrade.
> > > > >
> > > > > Yes, since it is an incompatible change I would state it in the
> > release
> > > > > notes.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > > BTW I found a performance regression while trying to upgrade
> another
> > > > > pipeline with this RC. It is a simple Kinesis to Kinesis job.
> Wasn't
> > > able
> > > > > to pin it down yet, symptoms include increased checkpoint alignment
> > > time.
> > > > >
> > > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi Thomas,
> > > > > >
> > > > > > just to confirm: When starting the image in local mode, then you
> > > don't
> > > > > have
> > > > > > any of the JobManager memory configuration settings configured in
> > the
> > > > > > effective flink-conf.yaml, right? Does this mean that you have
> > > > explicitly
> > > > > > removed `jobmanager.heap.size: 1024m` from the default
> > configuration?
> > > > If
> > > > > > this is the case, then I believe it was more of an unintentional
> > > > artifact
> > > > > > that it worked before and it has been corrected now so that one
> > needs
> > > > to
> > > > > > specify the memory of the JM process explicitly. Do you think it
> > > would
> > > > > help
> > > > > > to explicitly state this in the release notes?
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]>
> > wrote:
> > > > > >
> > > > > > > Thanks for preparing another RC!
> > > > > > >
> > > > > > > As mentioned in the previous RC thread, it would be super
> helpful
> > > if
> > > > > the
> > > > > > > release notes that are part of the documentation can be
> included
> > > [1].
> > > > > > It's
> > > > > > > a significant time-saver to have read those first.
> > > > > > >
> > > > > > > I found one more non-backward compatible change that would be
> > worth
> > > > > > > addressing/mentioning:
> > > > > > >
> > > > > > > It is now necessary to configure the jobmanager heap size in
> > > > > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > > > > or jobmanager.memory.heap.size). Why would I not want to do
> that
> > > > > anyways?
> > > > > > > Well, we set it dynamically for a cluster deployment via the
> > > > > > > flinkk8soperator, but the container image can also be used for
> > > > testing
> > > > > > with
> > > > > > > local mode (./bin/jobmanager.sh start-foreground local). That
> > will
> > > > fail
> > > > > > if
> > > > > > > the heap wasn't configured and that's how I noticed it.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Thomas
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > > > > >
> > > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <
> > > [hidden email]
> > > > > > > .invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Please review and vote on the release candidate #4 for the
> > > version
> > > > > > > 1.11.0,
> > > > > > > > as follows:
> > > > > > > > [ ] +1, Approve the release
> > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > comments)
> > > > > > > >
> > > > > > > > The complete staging area is available for your review, which
> > > > > includes:
> > > > > > > > * JIRA release notes [1],
> > > > > > > > * the official Apache source release and binary convenience
> > > > releases
> > > > > to
> > > > > > > be
> > > > > > > > deployed to dist.apache.org [2], which are signed with the
> key
> > > > with
> > > > > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > > > > * all artifacts to be deployed to the Maven Central
> Repository
> > > [4],
> > > > > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > > > > * website pull request listing the new release and adding
> > > > > announcement
> > > > > > > > blog post [6].
> > > > > > > >
> > > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > > majority
> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Release Manager
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > > [4]
> > > > > > > >
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > > > > [5]
> > > > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
>
> Best,
> Benchao Li
>

Jingsong Li

Re: [VOTE] Release 1.11.0, release candidate #4

+1 (non-binding)

- verified signature and checksum
- build from source
- checked webui and log sanity
- played with filesystem and new connectors
- played with Hive connector

Best,
Jingsonga

On Mon, Jul 6, 2020 at 9:50 AM Xintong Song <[hidden email]> wrote:

> +1 (non-binding)
>
> - verified signature and checksum
> - build from source
> - checked log sanity
> - checked webui
> - played with memory configurations
> - played with binding addresses/ports
>
> Thank you~
>
> Xintong Song
>
>
>
> On Sun, Jul 5, 2020 at 9:41 PM Benchao Li <[hidden email]> wrote:
>
> > +1 （non-binding)
> >
> > Checks:
> > - verified signature and shasum of release files [OK]
> > - build from source [OK]
> > - started standalone cluster, sql-client [mostly OK except one issue]
> > - played with sql-client
> > - played with new features: LIKE / Table Options
> > - checked Web UI functionality
> > - canceled job from UI
> >
> > While I'm playing with the new table factories, I found one issue[1]
> which
> > surprises me.
> > I don't think this should be a blocker, hence I'll still vote my +1.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18487
> >
> > Zhijiang <[hidden email]> 于2020年7月5日周日下午1:10写道：
> >
> > > Hi Thomas,
> > >
> > > Regarding [2], it has more detail infos in the Jira description (
> > > https://issues.apache.org/jira/browse/FLINK-16404).
> > >
> > > I can also give some basic explanations here to dismiss the concern.
> > > 1. In the past, the following buffers after the barrier will be cached
> on
> > > downstream side before alignment.
> > > 2. In 1.11, the upstream would not send the buffers after the barrier.
> > > When the downstream finishes the alignment, it will notify the
> downstream
> > > of continuing sending following buffers, since it can process them
> after
> > > alignment.
> > > 3. The only difference is that the temporary blocked buffers are cached
> > > either on downstream side or on upstream side before alignment.
> > > 4. The side effect would be the additional notification cost for every
> > > barrier alignment. If the downstream and upstream are deployed in
> > separate
> > > TaskManager, the cost is network transport delay (the effect can be
> > ignored
> > > based on our testing with 1s checkpoint interval). For sharing slot in
> > your
> > > case, the cost is only one method call in processor, can be ignored
> also.
> > >
> > > You mentioned "In this case, the downstream task has a high average
> > > checkpoint duration(~30s, sync part)." This duration is not reflecting
> > the
> > > changes above, and it is only indicating the duration for calling
> > > `Operation.snapshotState`.
> > > If this duration is beyond your expectation, you can check or debug
> > > whether the source/sink operations might take more time to finish
> > > `snapshotState` in practice. E.g. you can
> > > make the implementation of this method as empty to further verify the
> > > effect.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Thomas Weise <[hidden email]>
> > > Send Time:2020年7月5日(星期日) 12:22
> > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > Cc:Yingjie Cao <[hidden email]>
> > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > >
> > > Hi Zhijiang,
> > >
> > > Could you please point me to more details regarding: "[2]: Delay send
> the
> > > following buffers after checkpoint barrier on upstream side until
> barrier
> > > alignment on downstream side."
> > >
> > > In this case, the downstream task has a high average checkpoint
> duration
> > > (~30s, sync part). If there was a change to hold buffers depending on
> > > downstream performance, could this possibly apply to this case (even
> when
> > > there is no shuffle that would require alignment)?
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Sat, Jul 4, 2020 at 7:39 AM Zhijiang <[hidden email]
> > > .invalid>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > Thanks for the further update information.
> > > >
> > > > I guess we can dismiss the network stack changes, since in your case
> > the
> > > > downstream and upstream would probably be deployed in the same slot
> > > > bypassing the network data shuffle.
> > > > Also I guess release-1.11 will not bring general performance
> regression
> > > in
> > > > runtime engine, as we also did the performance testing for all
> general
> > > > cases by [1] in real cluster before and the testing results should
> fit
> > > the
> > > > expectation. But we indeed did not test the specific source and sink
> > > > connectors yet as I known.
> > > >
> > > > Regarding your performance regression with 40%, I wonder it is
> probably
> > > > related to specific source/sink changes (e.g. kinesis) or environment
> > > > issues with corner case.
> > > > If possible, it would be helpful to further locate whether the
> > regression
> > > > is caused by kinesis, by replacing the kinesis source & sink and
> > keeping
> > > > the others same.
> > > >
> > > > As you said, it would be efficient to contact with you directly next
> > week
> > > > to further discuss this issue. And we are willing/eager to provide
> any
> > > help
> > > > to resolve this issue soon.
> > > >
> > > > Besides that, I guess this issue should not be the blocker for the
> > > > release, since it is probably a corner case based on the current
> > > analysis.
> > > > If we really conclude anything need to be resolved after the final
> > > > release, then we can also make the next minor release-1.11.1 come
> soon.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-18433
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Thomas Weise <[hidden email]>
> > > > Send Time:2020年7月4日(星期六) 12:26
> > > > To:dev <[hidden email]>; Zhijiang <[hidden email]>
> > > > Cc:Yingjie Cao <[hidden email]>
> > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > >
> > > > Hi Zhijiang,
> > > >
> > > > It will probably be best if we connect next week and discuss the
> issue
> > > > directly since this could be quite difficult to reproduce.
> > > >
> > > > Before the testing result on our side comes out for your respective
> job
> > > > case, I have some other questions to confirm for further analysis:
> > > > - How much percentage regression you found after switching to
> > 1.11?
> > > >
> > > > ~40% throughput decline
> > > >
> > > > - Are there any network bottleneck in your cluster? E.g. the
> > network
> > > > bandwidth is full caused by other jobs? If so, it might have more
> > effects
> > > > by above [2]
> > > >
> > > > The test runs on a k8s cluster that is also used for other production
> > > jobs.
> > > > There is no reason be believe network is the bottleneck.
> > > >
> > > > - Did you adjust the default network buffer setting? E.g.
> > > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > > "taskmanager.network.memory.buffers-per-channel"
> > > >
> > > > The job is using the defaults, i.e we don't configure the settings.
> If
> > > you
> > > > want me to try specific settings in the hope that it will help to
> > isolate
> > > > the issue please let me know.
> > > >
> > > > - I guess the topology has three vertexes "KinesisConsumer ->
> > > Chained
> > > > FlatMap -> KinesisProducer", and the partition mode for
> > "KinesisConsumer
> > > ->
> > > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so,
> the
> > > edge
> > > > connection is one-to-one, not all-to-all, then the above [1][2]
> should
> > no
> > > > effects in theory with default network buffer setting.
> > > >
> > > > There are only 2 vertices and the edge is "forward".
> > > >
> > > > - By slot sharing, I guess these three vertex parallelism task
> > would
> > > > probably be deployed into the same slot, then the data shuffle is by
> > > memory
> > > > queue, not network stack. If so, the above [2] should no effect.
> > > >
> > > > Yes, vertices share slots.
> > > >
> > > > - I also saw some Jira changes for kinesis in this release, could
> > you
> > > > confirm that these changes would not effect the performance?
> > > >
> > > > I will need to take a look. 1.10 already had a regression introduced
> by
> > > the
> > > > Kinesis producer update.
> > > >
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Thu, Jul 2, 2020 at 11:46 PM Zhijiang <[hidden email]
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > Thanks for your reply with rich information!
> > > > >
> > > > > We are trying to reproduce your case in our cluster to further
> verify
> > > it,
> > > > > and @Yingjie Cao is working on it now.
> > > > > As we have not kinesis consumer and producer internally, so we
> will
> > > > > construct the common source and sink instead in the case of
> > > backpressure.
> > > > >
> > > > > Firstly, we can dismiss the rockdb factor in this release, since
> you
> > > also
> > > > > mentioned that "filesystem leads to same symptoms".
> > > > >
> > > > > Secondly, if my understanding is right, you emphasis that the
> > > regression
> > > > > only exists for the jobs with low checkpoint interval (10s).
> > > > > Based on that, I have two suspicions with the network related
> changes
> > > in
> > > > > this release:
> > > > > - [1]: Limited the maximum backlog value (default 10) in
> > > subpartition
> > > > > queue.
> > > > > - [2]: Delay send the following buffers after checkpoint
> barrier
> > on
> > > > > upstream side until barrier alignment on downstream side.
> > > > >
> > > > > These changes are motivated for reducing the in-flight buffers to
> > > speedup
> > > > > checkpoint especially in the case of backpressure.
> > > > > In theory they should have very minor performance effect and
> actually
> > > we
> > > > > also tested in cluster to verify within expectation before merging
> > > them,
> > > > > but maybe there are other corner cases we have not thought of
> > before.
> > > > >
> > > > > Before the testing result on our side comes out for your respective
> > job
> > > > > case, I have some other questions to confirm for further analysis:
> > > > > - How much percentage regression you found after switching to
> > > 1.11?
> > > > > - Are there any network bottleneck in your cluster? E.g. the
> > > network
> > > > > bandwidth is full caused by other jobs? If so, it might have more
> > > effects
> > > > > by above [2]
> > > > > - Did you adjust the default network buffer setting? E.g.
> > > > > "taskmanager.network.memory.floating-buffers-per-gate" or
> > > > > "taskmanager.network.memory.buffers-per-channel"
> > > > > - I guess the topology has three vertexes "KinesisConsumer ->
> > > > Chained
> > > > > FlatMap -> KinesisProducer", and the partition mode for
> > > "KinesisConsumer
> > > > ->
> > > > > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so,
> > the
> > > > edge
> > > > > connection is one-to-one, not all-to-all, then the above [1][2]
> > should
> > > no
> > > > > effects in theory with default network buffer setting.
> > > > > - By slot sharing, I guess these three vertex parallelism task
> > > would
> > > > > probably be deployed into the same slot, then the data shuffle is
> by
> > > > memory
> > > > > queue, not network stack. If so, the above [2] should no effect.
> > > > > - I also saw some Jira changes for kinesis in this release,
> could
> > > you
> > > > > confirm that these changes would not effect the performance?
> > > > >
> > > > > Best,
> > > > > Zhijiang
> > > > >
> > > > >
> > > > > ------------------------------------------------------------------
> > > > > From:Thomas Weise <[hidden email]>
> > > > > Send Time:2020年7月3日(星期五) 01:07
> > > > > To:dev <[hidden email]>; Zhijiang <
> [hidden email]>
> > > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > > >
> > > > > Hi Zhijiang,
> > > > >
> > > > > The performance degradation manifests in backpressure which leads
> to
> > > > > growing backlog in the source. I switched a few times between 1.10
> > and
> > > > 1.11
> > > > > and the behavior is consistent.
> > > > >
> > > > > The DAG is:
> > > > >
> > > > > KinesisConsumer -> (Flat Map, Flat Map, Flat Map) --------
> forward
> > > > > ---------> KinesisProducer
> > > > >
> > > > > Parallelism: 160
> > > > > No shuffle/rebalance.
> > > > >
> > > > > Checkpointing config:
> > > > >
> > > > > Checkpointing Mode Exactly Once
> > > > > Interval 10s
> > > > > Timeout 10m 0s
> > > > > Minimum Pause Between Checkpoints 10s
> > > > > Maximum Concurrent Checkpoints 1
> > > > > Persist Checkpoints Externally Enabled (delete on cancellation)
> > > > >
> > > > > State backend: rocksdb (filesystem leads to same symptoms)
> > > > > Checkpoint size is tiny (500KB)
> > > > >
> > > > > An interesting difference to another job that I had upgraded
> > > successfully
> > > > > is the low checkpointing interval.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > >
> > > > > On Wed, Jul 1, 2020 at 9:02 PM Zhijiang <
> [hidden email]
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi Thomas,
> > > > > >
> > > > > > Thanks for the efficient feedback.
> > > > > >
> > > > > > Regarding the suggestion of adding the release notes document, I
> > > agree
> > > > > > with your point. Maybe we should adjust the vote template
> > accordingly
> > > > in
> > > > > > the respective wiki to guide the following release processes.
> > > > > >
> > > > > > Regarding the performance regression, could you provide some more
> > > > details
> > > > > > for our better measurement or reproducing on our sides?
> > > > > > E.g. I guess the topology only includes two vertexes source and
> > sink?
> > > > > > What is the parallelism for every vertex?
> > > > > > The upstream shuffles data to the downstream via rebalance
> > > partitioner
> > > > or
> > > > > > other?
> > > > > > The checkpoint mode is exactly-once with rocksDB state backend?
> > > > > > The backpressure happened in this case?
> > > > > > How much percentage regression in this case?
> > > > > >
> > > > > > Best,
> > > > > > Zhijiang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> ------------------------------------------------------------------
> > > > > > From:Thomas Weise <[hidden email]>
> > > > > > Send Time:2020年7月2日(星期四) 09:54
> > > > > > To:dev <[hidden email]>
> > > > > > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> > > > > >
> > > > > > Hi Till,
> > > > > >
> > > > > > Yes, we don't have the setting in flink-conf.yaml.
> > > > > >
> > > > > > Generally, we carry forward the existing configuration and any
> > change
> > > > to
> > > > > > default configuration values would impact the upgrade.
> > > > > >
> > > > > > Yes, since it is an incompatible change I would state it in the
> > > release
> > > > > > notes.
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > > > BTW I found a performance regression while trying to upgrade
> > another
> > > > > > pipeline with this RC. It is a simple Kinesis to Kinesis job.
> > Wasn't
> > > > able
> > > > > > to pin it down yet, symptoms include increased checkpoint
> alignment
> > > > time.
> > > > > >
> > > > > > On Wed, Jul 1, 2020 at 12:04 AM Till Rohrmann <
> > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Thomas,
> > > > > > >
> > > > > > > just to confirm: When starting the image in local mode, then
> you
> > > > don't
> > > > > > have
> > > > > > > any of the JobManager memory configuration settings configured
> in
> > > the
> > > > > > > effective flink-conf.yaml, right? Does this mean that you have
> > > > > explicitly
> > > > > > > removed `jobmanager.heap.size: 1024m` from the default
> > > configuration?
> > > > > If
> > > > > > > this is the case, then I believe it was more of an
> unintentional
> > > > > artifact
> > > > > > > that it worked before and it has been corrected now so that one
> > > needs
> > > > > to
> > > > > > > specify the memory of the JM process explicitly. Do you think
> it
> > > > would
> > > > > > help
> > > > > > > to explicitly state this in the release notes?
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Wed, Jul 1, 2020 at 7:01 AM Thomas Weise <[hidden email]>
> > > wrote:
> > > > > > >
> > > > > > > > Thanks for preparing another RC!
> > > > > > > >
> > > > > > > > As mentioned in the previous RC thread, it would be super
> > helpful
> > > > if
> > > > > > the
> > > > > > > > release notes that are part of the documentation can be
> > included
> > > > [1].
> > > > > > > It's
> > > > > > > > a significant time-saver to have read those first.
> > > > > > > >
> > > > > > > > I found one more non-backward compatible change that would be
> > > worth
> > > > > > > > addressing/mentioning:
> > > > > > > >
> > > > > > > > It is now necessary to configure the jobmanager heap size in
> > > > > > > > flink-conf.yaml (with either jobmanager.heap.size
> > > > > > > > or jobmanager.memory.heap.size). Why would I not want to do
> > that
> > > > > > anyways?
> > > > > > > > Well, we set it dynamically for a cluster deployment via the
> > > > > > > > flinkk8soperator, but the container image can also be used
> for
> > > > > testing
> > > > > > > with
> > > > > > > > local mode (./bin/jobmanager.sh start-foreground local). That
> > > will
> > > > > fail
> > > > > > > if
> > > > > > > > the heap wasn't configured and that's how I noticed it.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Thomas
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html
> > > > > > > >
> > > > > > > > On Tue, Jun 30, 2020 at 3:18 AM Zhijiang <
> > > > [hidden email]
> > > > > > > > .invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > Please review and vote on the release candidate #4 for the
> > > > version
> > > > > > > > 1.11.0,
> > > > > > > > > as follows:
> > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > > comments)
> > > > > > > > >
> > > > > > > > > The complete staging area is available for your review,
> which
> > > > > > includes:
> > > > > > > > > * JIRA release notes [1],
> > > > > > > > > * the official Apache source release and binary convenience
> > > > > releases
> > > > > > to
> > > > > > > > be
> > > > > > > > > deployed to dist.apache.org [2], which are signed with the
> > key
> > > > > with
> > > > > > > > > fingerprint 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [3],
> > > > > > > > > * all artifacts to be deployed to the Maven Central
> > Repository
> > > > [4],
> > > > > > > > > * source code tag "release-1.11.0-rc4" [5],
> > > > > > > > > * website pull request listing the new release and adding
> > > > > > announcement
> > > > > > > > > blog post [6].
> > > > > > > > >
> > > > > > > > > The vote will be open for at least 72 hours. It is adopted
> by
> > > > > > majority
> > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Release Manager
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
> > > > > > > > > [2]
> > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc4/
> > > > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > > > [4]
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1377/
> > > > > > > > > [5]
> > > > > https://github.com/apache/flink/releases/tag/release-1.11.0-rc4
> > > > > > > > > [6] https://github.com/apache/flink-web/pull/352
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>

--
Best, Jingsong Lee

123