When running part4rddjoins.CogroupingRDDs directly from SBT/IntelliJ with Spark 4.0.0, the app fails during the first shuffle operation.
Simpler examples like Playground.scala run fine, but this example triggers Spark shuffle/Kryo initialization through reduceByKey, join, and cogroup.
Environment:
Scala: 2.13.16
Spark: 4.0.0
JDK: 21 (Temurin)
Run command: runMain part4rddjoins.CogroupingRDDs
Error seen:
java.lang.reflect.InaccessibleObjectException:
Unable to make field final byte[] java.nio.ByteBuffer.hb accessible:
module java.base does not "opens java.nio" to unnamed module
After opening java.nio, similar errors appear for java.lang.invoke.SerializedLambda and java.util.
Workaround that fixes it locally:
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
I placed these in .jvmopts and restarted the SBT shell.
It would be helpful if the repo documented this for Java 17/21 users, or included the required .jvmopts / SBT run configuration. Spark’s own launcher adds module-opening options, but running examples directly from SBT/IntelliJ bypasses that launcher.
When running
part4rddjoins.CogroupingRDDsdirectly from SBT/IntelliJ with Spark 4.0.0, the app fails during the first shuffle operation.Simpler examples like Playground.scala run fine, but this example triggers Spark shuffle/Kryo initialization through reduceByKey, join, and cogroup.
Environment:
Run command:
runMain part4rddjoins.CogroupingRDDsError seen:
After opening java.nio, similar errors appear for java.lang.invoke.SerializedLambda and java.util.
Workaround that fixes it locally:
I placed these in
.jvmoptsand restarted the SBT shell.It would be helpful if the repo documented this for Java 17/21 users, or included the required .jvmopts / SBT run configuration. Spark’s own launcher adds module-opening options, but running examples directly from SBT/IntelliJ bypasses that launcher.