createStageMetricsDF fails with NoSuchMethodError on Spark 4.0.0 due to removed SparkSession.implicits

[fjammes]

Hi @LucaCanali , and thanks for this useful tool!

Problem description

When using spark-measure with Apache Spark 4.0.0 (Scala 2.13), the following error occurs at runtime when calling createStageMetricsDF:

java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$implicits$ org.apache.spark.sql.SparkSession.implicits()'

This is due to the fact that the method SparkSession.implicits() has been removed or made inaccessible in Spark 4. As a result, StageMetrics.createStageMetricsDF throws a NoSuchMethodError when it tries to create a DataFrame using spark.implicits._.

📄 Stacktrace (abbreviated)

java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$implicits$ org.apache.spark.sql.SparkSession.implicits()'
    at ch.cern.sparkmeasure.StageMetrics.createStageMetricsDF(StageMetrics.scala:228)

✅ Environment

Spark version: 4.0.0

Scala version: 2.13.13

spark-measure: latest commit from main branch (built locally): 4ea725a

Request

Would it be possible to:

Add support for Spark 4.0.0 via a dedicated branch or version?

Thanks again for the tool, and happy to contribute a PR if it can help.

[LucaCanali]

Thank you @fjammes for reporting this.
I’ve just pushed a fix for this issue:
eab34df

Please test it and let me know if it works for you

[fjammes]

Thanks a lot @LucaCanali!
I've run the integration tests using Spark 4 and Scala 2.13, and everything looks good:
https://github.com/k8s-school/sparkMeasure/actions/runs/16592168357/job/46930257847

Would you consider setting up a matrix build (once PR #71 is merged) to run the end-to-end tests on both Spark 3 / Scala 2.12 and Spark 4 / Scala 2.13 within the GitHub Actions workflow?

[fjammes]

Hi @LucaCanali , I have added support for the matrix build (spark v3/v4) here: #74

Thanks in advance for your review.