Flink 1.2 / YARN on Secure MapR Cluster

[dschexna]

I am attempting to run flink / YARN on a secure MapR 5.2 cluster.  The cluster is secured using "MapR Native Security", not kerberos.

I did include the MapR zookeeper when building:

opt/apache-maven-3.3.9/bin/mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.0-mapr-1607 -Dzookeeper.version=3.4.5-mapr-1604

When I try to launch yarn-session.sh, I receive the following error:

[mapr@maprdemo flink]$ ./bin/yarn-session.sh -n 1
2017-03-08 14:47:24,223 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:24,224 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:24,224 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:24,224 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:24,225 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:24,225 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:24,225 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2017-03-08 14:47:24,225 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:24,292 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:24,292 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:24,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:24,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:24,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:24,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:24,293 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2017-03-08 14:47:24,294 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:26,107 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-03-08 14:47:26,390 WARN  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop security is enabled but current login user does not have Kerberos credentials
2017-03-08 14:47:26,390 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to mapr (auth:CUSTOM)
2017-03-08 14:47:26,431 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:26,431 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:26,431 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:26,431 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:26,432 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:26,432 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:26,433 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2017-03-08 14:47:26,433 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:26,440 ERROR org.apache.flink.yarn.YarnClusterDescriptor                   - Hadoop security is enabled but the login user does not have Kerberos credentials
Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:421)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:620)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)
        at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
        at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)
Caused by: java.lang.RuntimeException: Hadoop security is enabled but the login user does not have Kerberos credentials
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:415)

The 'mapr' user does have a ticket:

[mapr@maprdemo flink]$ maprlogin print
Opening keyfile /tmp/maprticket_2000
demo.mapr.com: user = mapr, created = 'Wed Mar 01 07:26:02 PST 2017', expires = 'Wed Mar 15 08:26:02 PDT 2017', RenewalTill = 'Fri Mar 31 08:26:02 PDT 2017', uid = 2000, gids = 2000, 2001, CanImpersonate = false

I can launch other YARN jobs without error.  Most of the MapR environment variables are picked up in /opt/mapr/conf/env.sh and /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hadoop-env.sh.

I have set the following environment variables:

export FLINK_HOME=/opt/flink-1.2
export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.7.0
export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop

[Till Rohrmann]

Hi,

could it be that this issue is related to [1]? If so, then it should soon
be fixed.

[1] https://issues.apache.org/jira/browse/FLINK-5949

Cheers,
Till

On Wed, Mar 8, 2017 at 11:50 PM, dschexna <[hidden email]> wrote:

> I am attempting to run flink / YARN on a secure MapR 5.2 cluster.  The
> cluster is secured using "MapR Native Security", not kerberos.
>
> I did include the MapR zookeeper when building:
>
> opt/apache-maven-3.3.9/bin/mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.7.0-mapr-1607 -Dzookeeper.version=3.4.5-mapr-1604
>
> When I try to launch yarn-session.sh, I receive the following error:
>
> [mapr@maprdemo flink]$ ./bin/yarn-session.sh -n 1
> 2017-03-08 14:47:24,223 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2017-03-08 14:47:24,224 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.rpc.port, 6123
> 2017-03-08 14:47:24,224 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.heap.mb, 256
> 2017-03-08 14:47:24,224 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.heap.mb, 512
> 2017-03-08 14:47:24,225 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2017-03-08 14:47:24,225 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.memory.preallocate, false
> 2017-03-08 14:47:24,225 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: parallelism.default, 1
> 2017-03-08 14:47:24,225 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.web.port, 8081
> 2017-03-08 14:47:24,292 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2017-03-08 14:47:24,292 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.rpc.port, 6123
> 2017-03-08 14:47:24,293 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.heap.mb, 256
> 2017-03-08 14:47:24,293 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.heap.mb, 512
> 2017-03-08 14:47:24,293 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2017-03-08 14:47:24,293 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.memory.preallocate, false
> 2017-03-08 14:47:24,293 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: parallelism.default, 1
> 2017-03-08 14:47:24,294 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.web.port, 8081
> 2017-03-08 14:47:26,107 WARN  org.apache.hadoop.util.NativeCodeLoader
> - Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2017-03-08 14:47:26,390 WARN
> org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop
> security is enabled but current login user does not have Kerberos
> credentials
> 2017-03-08 14:47:26,390 INFO
> org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop
> user
> set to mapr (auth:CUSTOM)
> 2017-03-08 14:47:26,431 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.rpc.address, localhost
> 2017-03-08 14:47:26,431 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.rpc.port, 6123
> 2017-03-08 14:47:26,431 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.heap.mb, 256
> 2017-03-08 14:47:26,431 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.heap.mb, 512
> 2017-03-08 14:47:26,432 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.numberOfTaskSlots, 1
> 2017-03-08 14:47:26,432 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: taskmanager.memory.preallocate, false
> 2017-03-08 14:47:26,433 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: parallelism.default, 1
> 2017-03-08 14:47:26,433 INFO
> org.apache.flink.configuration.GlobalConfiguration            - Loading
> configuration property: jobmanager.web.port, 8081
> 2017-03-08 14:47:26,440 ERROR org.apache.flink.yarn.YarnClusterDescriptor
> - Hadoop security is enabled but the login user does not have Kerberos
> credentials
> Error while deploying YARN cluster: Couldn't deploy Yarn cluster
> java.lang.RuntimeException: Couldn't deploy Yarn cluster
>         at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(
> AbstractYarnClusterDescriptor.java:421)
>         at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(
> FlinkYarnSessionCli.java:620)
>         at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(
> FlinkYarnSessionCli.java:476)
>         at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(
> FlinkYarnSessionCli.java:473)
>         at
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(
> HadoopSecurityContext.java:43)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1595)
>         at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(
> HadoopSecurityContext.java:40)
>         at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(
> FlinkYarnSessionCli.java:473)
> Caused by: java.lang.RuntimeException: Hadoop security is enabled but the
> login user does not have Kerberos credentials
>         at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(
> AbstractYarnClusterDescriptor.java:415)
>
> The 'mapr' user does have a ticket:
>
> [mapr@maprdemo flink]$ maprlogin print
> Opening keyfile /tmp/maprticket_2000
> demo.mapr.com: user = mapr, created = 'Wed Mar 01 07:26:02 PST 2017',
> expires = 'Wed Mar 15 08:26:02 PDT 2017', RenewalTill = 'Fri Mar 31
> 08:26:02
> PDT 2017', uid = 2000, gids = 2000, 2001, CanImpersonate = false
>
> I can launch other YARN jobs without error.  Most of the MapR environment
> variables are picked up in /opt/mapr/conf/env.sh and
> /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hadoop-env.sh.
>
> I have set the following environment variables:
>
> export FLINK_HOME=/opt/flink-1.2
> export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
> export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.7.0
> export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
>
>
>
> --
> View this message in context: http://apache-flink-mailing-
> list-archive.1008284.n3.nabble.com/Flink-1-2-YARN-on-
> Secure-MapR-Cluster-tp16399.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>