Hi Team,
I am trying to create a connector for carbondata. Currently i am working on creating OutputFormat for carbondata. For that i need to create a carboncontext which i initialise in configure method. Internally CarbonContext uses sparkContext. I am able to create carbonContext through my code but when it run statement "executionEnvironment.execute() Exception in thread "main" org.apache.flink.optimizer.CompilerException: Error translating node 'Data Sink "org.apache.flink.CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not write the user code wrapper class org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106) at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:198) at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192) at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188) at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91) at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:926) at CarbonJDBCExample.main(CarbonJDBCExample.java:86) Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could not write the user code wrapper class org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper at org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:281) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSinkVertex(JobGraphGenerator.java:903) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:278) ... 8 more Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:317) at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:254) at org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279) ... 10 more -- Warm Regards, Sangeeta Gulia Software Consultant Knoldus Software LLP |
Hi Team,
Please ignore the previous mail. I am trying to create a connector for carbondata. Currently i am working on creating OutputFormat for carbondata. For that i need to create a carboncontext which i initialise in configure method. Internally CarbonContext uses sparkContext. I am able to create carbonContext through my code but when it run statement "executionEnvironment.execute() it throws the below error: Exception in thread "main" org.apache.flink.optimizer.CompilerException: Error translating node 'Data Sink "org.apache.flink.CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not write the user code wrapper class org.apache.flink.api.common. operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( JobGraphGenerator.java:381) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( JobGraphGenerator.java:106) at org.apache.flink.optimizer.plan.SingleInputPlanNode. accept(SingleInputPlanNode.java:198) at org.apache.flink.optimizer.plan.OptimizedPlan.accept( OptimizedPlan.java:128) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. compileJobGraph(JobGraphGenerator.java:192) at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188) at org.apache.flink.api.java.LocalEnvironment.execute( LocalEnvironment.java:91) at org.apache.flink.api.java.ExecutionEnvironment.execute( ExecutionEnvironment.java:926) at CarbonJDBCExample.main(CarbonJDBCExample.java:86) Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could not write the user code wrapper class org.apache.flink.api.common. operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper at org.apache.flink.runtime.operators.util.TaskConfig. setStubWrapper(TaskConfig.java:281) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. createDataSinkVertex(JobGraphGenerator.java:903) at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( JobGraphGenerator.java:278) ... 8 more Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive. client.ClientWrapper at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.defaultWriteFields( ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject( ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields( ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject( ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields( ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject( ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.flink.util.InstantiationUtil.serializeObject( InstantiationUtil.java:317) at org.apache.flink.util.InstantiationUtil.writeObjectToConfig( InstantiationUtil.java:254) at org.apache.flink.runtime.operators.util.TaskConfig. setStubWrapper(TaskConfig.java:279) ... 10 more Help me with the same. On Mon, Apr 17, 2017 at 12:39 PM, Sangeeta Gulia <[hidden email]> wrote: > Hi Team, > > I am trying to create a connector for carbondata. Currently i am working > on creating OutputFormat for carbondata. For that i need to create a > carboncontext which i initialise in configure method. Internally > CarbonContext uses sparkContext. > > I am able to create carbonContext through my code but when it run statement > "executionEnvironment.execute() > > Exception in thread "main" org.apache.flink.optimizer.CompilerException: > Error translating node 'Data Sink "org.apache.flink. > CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties > [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null, > grouped=null, unique=null] ]]': Could not write the user code wrapper class > org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io > .NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( > JobGraphGenerator.java:381) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( > JobGraphGenerator.java:106) > at org.apache.flink.optimizer.plan.SingleInputPlanNode. > accept(SingleInputPlanNode.java:198) > at org.apache.flink.optimizer.plan.OptimizedPlan.accept( > OptimizedPlan.java:128) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. > compileJobGraph(JobGraphGenerator.java:192) > at org.apache.flink.client.LocalExecutor.executePlan( > LocalExecutor.java:188) > at org.apache.flink.api.java.LocalEnvironment.execute( > LocalEnvironment.java:91) > at org.apache.flink.api.java.ExecutionEnvironment.execute( > ExecutionEnvironment.java:926) > at CarbonJDBCExample.main(CarbonJDBCExample.java:86) > Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: > Could not write the user code wrapper class org.apache.flink.api.common. > operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: > org.apache.spark.sql.hive.client.ClientWrapper > at org.apache.flink.runtime.operators.util.TaskConfig. > setStubWrapper(TaskConfig.java:281) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. > createDataSinkVertex(JobGraphGenerator.java:903) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( > JobGraphGenerator.java:278) > ... 8 more > Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive. > client.ClientWrapper > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > at java.io.ObjectOutputStream.defaultWriteFields( > ObjectOutputStream.java:1548) > at java.io.ObjectOutputStream.writeSerialData( > ObjectOutputStream.java:1509) > at java.io.ObjectOutputStream.writeOrdinaryObject( > ObjectOutputStream.java:1432) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.defaultWriteFields( > ObjectOutputStream.java:1548) > at java.io.ObjectOutputStream.writeSerialData( > ObjectOutputStream.java:1509) > at java.io.ObjectOutputStream.writeOrdinaryObject( > ObjectOutputStream.java:1432) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.defaultWriteFields( > ObjectOutputStream.java:1548) > at java.io.ObjectOutputStream.writeSerialData( > ObjectOutputStream.java:1509) > at java.io.ObjectOutputStream.writeOrdinaryObject( > ObjectOutputStream.java:1432) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at org.apache.flink.util.InstantiationUtil.serializeObject( > InstantiationUtil.java:317) > at org.apache.flink.util.InstantiationUtil.writeObjectToConfig( > InstantiationUtil.java:254) > at org.apache.flink.runtime.operators.util.TaskConfig. > setStubWrapper(TaskConfig.java:279) > ... 10 more > > > > -- > Warm Regards, > > Sangeeta Gulia > Software Consultant > Knoldus Software LLP > -- Warm Regards, Sangeeta Gulia Software Consultant Knoldus Software LLP |
As the exceptions says your outputformat isn't serializable; is the
carboncontext marked as transient? On 17.04.2017 09:12, Sangeeta Gulia wrote: > Hi Team, > > Please ignore the previous mail. > > I am trying to create a connector for carbondata. Currently i am working on > creating OutputFormat for carbondata. For that i need to create a > carboncontext which i initialise in configure method. Internally > CarbonContext uses sparkContext. > > I am able to create carbonContext through my code but when it run statement > "executionEnvironment.execute() > it throws the below error: > > Exception in thread "main" org.apache.flink.optimizer.CompilerException: > Error translating node 'Data Sink > "org.apache.flink.CarbonDataOutputFormat@76225435" > : NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ > LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not > write the user code wrapper class org.apache.flink.api.common. > operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: > org.apache.spark.sql.hive.client.ClientWrapper > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( > JobGraphGenerator.java:381) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( > JobGraphGenerator.java:106) > at org.apache.flink.optimizer.plan.SingleInputPlanNode. > accept(SingleInputPlanNode.java:198) > at org.apache.flink.optimizer.plan.OptimizedPlan.accept( > OptimizedPlan.java:128) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. > compileJobGraph(JobGraphGenerator.java:192) > at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188) > at org.apache.flink.api.java.LocalEnvironment.execute( > LocalEnvironment.java:91) > at org.apache.flink.api.java.ExecutionEnvironment.execute( > ExecutionEnvironment.java:926) > at CarbonJDBCExample.main(CarbonJDBCExample.java:86) > Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: > Could not write the user code wrapper class org.apache.flink.api.common. > operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: > org.apache.spark.sql.hive.client.ClientWrapper > at org.apache.flink.runtime.operators.util.TaskConfig. > setStubWrapper(TaskConfig.java:281) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. > createDataSinkVertex(JobGraphGenerator.java:903) > at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( > JobGraphGenerator.java:278) > ... 8 more > Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive. > client.ClientWrapper > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > at java.io.ObjectOutputStream.defaultWriteFields( > ObjectOutputStream.java:1548) > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > at java.io.ObjectOutputStream.writeOrdinaryObject( > ObjectOutputStream.java:1432) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.defaultWriteFields( > ObjectOutputStream.java:1548) > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > at java.io.ObjectOutputStream.writeOrdinaryObject( > ObjectOutputStream.java:1432) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.defaultWriteFields( > ObjectOutputStream.java:1548) > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > at java.io.ObjectOutputStream.writeOrdinaryObject( > ObjectOutputStream.java:1432) > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at org.apache.flink.util.InstantiationUtil.serializeObject( > InstantiationUtil.java:317) > at org.apache.flink.util.InstantiationUtil.writeObjectToConfig( > InstantiationUtil.java:254) > at org.apache.flink.runtime.operators.util.TaskConfig. > setStubWrapper(TaskConfig.java:279) > ... 10 more > > Help me with the same. > > On Mon, Apr 17, 2017 at 12:39 PM, Sangeeta Gulia <[hidden email]> > wrote: > >> Hi Team, >> >> I am trying to create a connector for carbondata. Currently i am working >> on creating OutputFormat for carbondata. For that i need to create a >> carboncontext which i initialise in configure method. Internally >> CarbonContext uses sparkContext. >> >> I am able to create carbonContext through my code but when it run statement >> "executionEnvironment.execute() >> >> Exception in thread "main" org.apache.flink.optimizer.CompilerException: >> Error translating node 'Data Sink "org.apache.flink. >> CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties >> [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null, >> grouped=null, unique=null] ]]': Could not write the user code wrapper class >> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io >> .NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper >> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( >> JobGraphGenerator.java:381) >> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( >> JobGraphGenerator.java:106) >> at org.apache.flink.optimizer.plan.SingleInputPlanNode. >> accept(SingleInputPlanNode.java:198) >> at org.apache.flink.optimizer.plan.OptimizedPlan.accept( >> OptimizedPlan.java:128) >> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. >> compileJobGraph(JobGraphGenerator.java:192) >> at org.apache.flink.client.LocalExecutor.executePlan( >> LocalExecutor.java:188) >> at org.apache.flink.api.java.LocalEnvironment.execute( >> LocalEnvironment.java:91) >> at org.apache.flink.api.java.ExecutionEnvironment.execute( >> ExecutionEnvironment.java:926) >> at CarbonJDBCExample.main(CarbonJDBCExample.java:86) >> Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: >> Could not write the user code wrapper class org.apache.flink.api.common. >> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException: >> org.apache.spark.sql.hive.client.ClientWrapper >> at org.apache.flink.runtime.operators.util.TaskConfig. >> setStubWrapper(TaskConfig.java:281) >> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator. >> createDataSinkVertex(JobGraphGenerator.java:903) >> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit( >> JobGraphGenerator.java:278) >> ... 8 more >> Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive. >> client.ClientWrapper >> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) >> at java.io.ObjectOutputStream.defaultWriteFields( >> ObjectOutputStream.java:1548) >> at java.io.ObjectOutputStream.writeSerialData( >> ObjectOutputStream.java:1509) >> at java.io.ObjectOutputStream.writeOrdinaryObject( >> ObjectOutputStream.java:1432) >> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) >> at java.io.ObjectOutputStream.defaultWriteFields( >> ObjectOutputStream.java:1548) >> at java.io.ObjectOutputStream.writeSerialData( >> ObjectOutputStream.java:1509) >> at java.io.ObjectOutputStream.writeOrdinaryObject( >> ObjectOutputStream.java:1432) >> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) >> at java.io.ObjectOutputStream.defaultWriteFields( >> ObjectOutputStream.java:1548) >> at java.io.ObjectOutputStream.writeSerialData( >> ObjectOutputStream.java:1509) >> at java.io.ObjectOutputStream.writeOrdinaryObject( >> ObjectOutputStream.java:1432) >> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) >> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) >> at org.apache.flink.util.InstantiationUtil.serializeObject( >> InstantiationUtil.java:317) >> at org.apache.flink.util.InstantiationUtil.writeObjectToConfig( >> InstantiationUtil.java:254) >> at org.apache.flink.runtime.operators.util.TaskConfig. >> setStubWrapper(TaskConfig.java:279) >> ... 10 more >> >> >> >> -- >> Warm Regards, >> >> Sangeeta Gulia >> Software Consultant >> Knoldus Software LLP >> > > |
Free forum by Nabble | Edit this page |