java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance

java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance

Problem Description:

I am totally new to Spark, My spark master is on GCP Kubernetes Standard cluster.
I am using a gremlin client to trigger the spark job, on my local machine configuration it works totally fine and the job gets completed, But on Kubernetes, I am seeing the below error.

Gremlin Query and error

==>tinkerpop.hadoop activated
gremlin> :plugin use tinkerpop.spark
==>tinkerpop.spark activated
gremlin> graph = GraphFactory.open('/opt/janusgraph-0.6.2/conf/hadoop-graph/read-hbase-standalone-cluster-stg-bigtable-spark.properties')
==>hadoopgraph[hbaseinputformat->nulloutputformat]
gremlin> 
gremlin> g = graph.traversal().withComputer(SparkGraphComputer)
==>graphtraversalsource[hadoopgraph[hbaseinputformat->nulloutputformat], sparkgraphcomputer]
gremlin> g.V().count()
14:01:48 WARN  org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer  - class org.apache.hadoop.mapreduce.lib.output.NullOutputFormat does not implement PersistResultGraphAware and thus, persistence options are unknown -- assuming all options are possible
14:02:03 WARN  org.apache.spark.scheduler.TaskSetManager  - Lost task 6.0 in stage 0.0 (TID 6, XX.XX.XX.XX, executor 0): java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2198)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
        at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
        at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
        at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:127)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.GeneratedSerializationConstructorAccessor270.newInstance(Unknown Source)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:963)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2195)
        ... 77 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator
        at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
        at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
        ... 81 more

14:02:03 ERROR org.apache.spark.scheduler.TaskSetManager  - Task 8 in stage 0.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 0.0 failed 4 times, most recent failure: Lost task 8.3 in stage 0.0 (TID 32, 10.12.11.7, executor 1): java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2198)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
        at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
        at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
        at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
        at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:127)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.GeneratedSerializationConstructorAccessor238.newInstance(Unknown Source)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:963)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2195)
        ... 76 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator
        at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
        at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
        ... 80 more

Config File–> read-hbase-standalone-cluster-stg-bigtable-spark.properties

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#
# Hadoop Graph Configuration
#
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.hbase.HBaseInputFormat
gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat

gremlin.hadoop.jarsInDistributedCache=true
gremlin.hadoop.inputLocation=none
gremlin.hadoop.outputLocation=output
gremlin.spark.persistContext=true

#
# JanusGraph HBase InputFormat configuration
#
#janusgraphmr.ioformat.conf.storage.backend=hbase
#janusgraphmr.ioformat.conf.storage.hostname=localhost
#janusgraphmr.ioformat.conf.storage.port=8586
#janusgraphmr.ioformat.conf.storage.hbase.table=janusgraph
janusgraphmr.ioformat.conf.storage.backend=hbase
janusgraphmr.ioformat.conf.storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
janusgraphmr.ioformat.conf.storage.hbase.ext.google.bigtable.project.id=**********
janusgraphmr.ioformat.conf.storage.hbase.ext.google.bigtable.instance.id=*********
janusgraphmr.ioformat.conf.storage.hbase.table=**********
janusgraphmr.ioformat.conf.storage.hbase.ext.hbase.regionsizecalculator.enable=false

# This defines the indexing backend configuration used while writing data to JanusGraph.
janusgraphmr.ioformat.conf.index.search.backend=elasticsearch
janusgraphmr.ioformat.conf.index.search.hostname=**********

#
# SparkGraphComputer Configuration
#
spark.master=spark://spark-master:7077
spark.executor.memory=1g
spark.executor.extraClassPath=/opt/janusgraph-0.6.2/lib/*
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator```

Solution – 1

With reference to this https://li-boxuan.medium.com/configure-janusgraph-0-6-0-for-spark-64bd7628fb77 blog.
spark.executor.extraClassPath=/opt/janusgraph-0.6.2/lib/* mentioned in the gremlin reference config files, these libraries should be present in both Master and Worker nodes.

Rate this post
We use cookies in order to give you the best possible experience on our website. By continuing to use this site, you agree to our use of cookies.
Accept
Reject