[jira] [Created] (FLINK-21591) Flink using OSS Backend on Native K8S found some errors

[Shang Yuanchun (Jira)]

yufan wang created FLINK-21591:
----------------------------------

             Summary: Flink using OSS Backend on Native K8S found some errors
                 Key: FLINK-21591
                 URL: https://issues.apache.org/jira/browse/FLINK-21591
             Project: Flink
          Issue Type: Bug
            Reporter: yufan wang


Flink run on Native Kubernetes Application Mode

Using OSS as the backend. When the job starts, I get some error logs with Jobmanager and Taskmanager.

[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404BFDBD813C3838043521
[HostId]: null


start command:

./bin/flink run-application \
 --target kubernetes-application \
 -Dkubernetes.cluster-id=demo \
 -Dkubernetes.container.image=xx/xx/job:2.0.20 \
 -Dstate.backend=filesystem \
 -Dstate.checkpoints.dir=oss://xx/backend/ \
 -Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
 -Dfs.oss.accessKeyId=xx \
 -Dfs.oss.accessKeySecret=xx \
 -Dhigh-availability.storageDir=oss://xx/flink-ha/ \
 -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
 local:///opt/flink/usrlib/my-flink-job.jar

morelog:

Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file=/opt/flink/log/jobmanager.log -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=201326592b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,721 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2021-03-04 02:54:50,723 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12, Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
2021-03-04 02:54:50,723 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink
2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: <no hadoop dependency found>
2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.275-b01
2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 989 MiBytes
2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-8
2021-03-04 02:54:50,773 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available
2021-03-04 02:54:50,773 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/jobmanager.log
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
2021-03-04 02:54:50,775 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,775 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.off-heap.size=134217728b
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.min=201326592b
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-metaspace.size=268435456b
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.heap.size=1073741824b
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.max=201326592b
2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.12.0.jar:/opt/flink/lib/flink-json-1.12.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.12.0.jar:/opt/flink/lib/flink-table_2.12-1.12.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/mysql-connector-java-5.1.49.jar:/opt/flink/lib/flink-dist_2.12-1.12.0.jar:::
2021-03-04 02:54:50,777 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2021-03-04 02:54:50,777 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT]
2021-03-04 02:54:50,788 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.storageDir, oss://***/flink-ha/
2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:50,790 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, **/**/job:2.0.20
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:50,791 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-application
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.attached, true
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend, filesystem
2021-03-04 02:54:50,794 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:51,100 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint.
2021-03-04 02:54:51,108 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem.
2021-03-04 02:54:51,111 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2021-03-04 02:54:51,188 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context.
2021-03-04 02:54:51,195 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-03-04 02:54:51,198 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-5821322633800809072.conf.
2021-03-04 02:54:51,202 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2021-03-04 02:54:51,203 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services.
2021-03-04 02:54:51,219 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.18.242.149:6123, bind address 0.0.0.0:6123.
2021-03-04 02:54:52,216 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2021-03-04 02:54:52,278 INFO akka.remote.Remoting [] - Starting remoting
2021-03-04 02:54:52,407 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink@172.18.242.149:6123]
2021-03-04 02:54:52,519 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@172.18.242.149:6123
2021-03-04 02:54:53,499 INFO org.apache.flink.runtime.blob.FileSystemBlobStore [] - Creating highly available BLOB storage directory at oss://***/flink-ha/demo/blob
2021-03-04 02:54:53,855 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404BFDBD813C3838043521
[HostId]: null
2021-03-04 02:54:53,866 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/blobStore-6d45b5b0-9271-4c3a-9b42-0cda22da25aa
2021-03-04 02:54:53,868 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2021-03-04 02:54:53,877 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter configured, no metrics will be exposed/reported.
2021-03-04 02:54:53,879 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.18.242.149:0, bind address 0.0.0.0:0.
2021-03-04 02:54:53,895 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2021-03-04 02:54:53,900 INFO akka.remote.Remoting [] - Starting remoting
2021-03-04 02:54:53,907 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.18.242.149:45382]
2021-03-04 02:54:53,979 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@172.18.242.149:45382
2021-03-04 02:54:53,990 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
2021-03-04 02:54:54,082 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload does not exist.
2021-03-04 02:54:54,083 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload for file uploads.
2021-03-04 02:54:54,094 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting rest endpoint.
2021-03-04 02:54:54,412 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/jobmanager.log
2021-03-04 02:54:54,412 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/jobmanager.out
2021-03-04 02:54:54,605 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest endpoint listening at 172.18.242.149:8081
2021-03-04 02:54:54,788 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector demo-restserver-leader with lock identity ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:54,982 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver\{configMapName='demo-restserver-leader'}.
2021-03-04 02:54:54,983 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend listening at http://172.18.242.149:8081.
2021-03-04 02:54:54,993 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.cluster-id, demo
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, demo
2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeySecret, ******
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeyId, ***
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.storageDir, oss://***/flink-ha/
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.savepoint.ignore-unclaimed-state, false
2021-03-04 02:54:55,006 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, **/**/job:2.0.20
2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2021-03-04 02:54:55,073 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-03-04 02:54:55,073 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
2021-03-04 02:54:55,073 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.jobmanager.port, 6123
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-application
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.attached, true
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.shutdown-on-attached-exit, false
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/my-flink-job.jar
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend, filesystem
2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, oss://***/backend/
2021-03-04 02:54:55,081 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at akka://flink/user/rpc/resourcemanager_0 .
2021-03-04 02:54:55,093 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector demo-dispatcher-leader with lock identity ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:55,099 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver\{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,279 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver\{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:55,285 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver\{configMapName='demo-dispatcher-leader'}.
2021-03-04 02:54:55,492 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for demo-restserver-leader.
2021-03-04 02:54:55,493 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://172.18.242.149:8081 was granted leadership with leaderSessionID=77f241fd-89a6-4183-977d-4c4633f400fe
2021-03-04 02:54:55,494 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for demo-dispatcher-leader.
2021-03-04 02:54:55,578 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess.
2021-03-04 02:54:55,581 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs.
2021-03-04 02:54:55,588 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job ids [] from KubernetesStateHandleStore\{configMapName='demo-dispatcher-leader'}
2021-03-04 02:54:55,588 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 0 persisted job graphs.
2021-03-04 02:54:55,595 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_1 .
2021-03-04 02:54:55,684 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: false)
2021-03-04 02:54:56,089 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0 pods from previous attempts, current attempt id is 1.
2021-03-04 02:54:56,090 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Recovered 0 workers from previous attempt.
2021-03-04 02:54:56,090 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector demo-resourcemanager-leader with lock identity ecd54426-0378-4460-9a73-6fb2b239e76f.
2021-03-04 02:54:56,097 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver\{configMapName='demo-resourcemanager-leader'}.
2021-03-04 02:54:56,175 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for demo-resourcemanager-leader.
2021-03-04 02:54:56,274 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - ResourceManager akka.tcp://flink@172.18.242.149:6123/user/rpc/resourcemanager_0 was granted leadership with fencing token aae3a328d005a1fdb189546b6f3c46ba
2021-03-04 02:54:56,278 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Starting the SlotManager.
2021-03-04 02:54:56,512 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class com.**.entity.TransactionBase$Source is not public so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2021-03-04 02:54:56,519 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class com.**.entity.TransactionBase$Source is not public so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2021-03-04 02:54:56,575 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class com.**.entity.TransactionKeyBy is missing a default constructor so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2021-03-04 02:54:56,614 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class com.**.entity.TransactionSum is missing a default constructor so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2021-03-04 02:54:56,806 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 00000000000000000000000000000000 is submitted.
2021-03-04 02:54:56,806 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=00000000000000000000000000000000.
2021-03-04 02:54:57,913 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C38389D6321
[HostId]: null
2021-03-04 02:54:57,915 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C01BD813C3838AA6321
[HostId]: null
2021-03-04 02:54:58,902 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Unhandled exception
java.io.IOException: Connection reset by peer // Exception ignored, caused by cloud provider LoadBalance
...
2021-03-04 02:55:02,535 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838DC9C21
[HostId]: null
2021-03-04 02:55:02,537 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E69C21
[HostId]: null
2021-03-04 02:55:02,634 WARN org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to find 'mime.types' file in classpath
2021-03-04 02:55:02,768 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C38387B9F21
[HostId]: null
2021-03-04 02:55:02,770 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838869F21
[HostId]: null
2021-03-04 02:55:02,888 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838E4A021
[HostId]: null
2021-03-04 02:55:02,890 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 60404C06BD813C3838ECA021
[HostId]: null



--
This message was sent by Atlassian Jira
(v8.3.4#803005)