Prepare release 0.25

Author: LucaCanali

README.md

@@ -14,7 +14,7 @@ making it a practical choice for both developers and data engineers.
 With sparkMeasure, users can obtain a clearer understanding of their Spark job performance, 
 facilitating smoother and more reliable data processing operations.
 
-### Key Features
+### ✨ Highlights
 - **Interactive Troubleshooting:** Ideal for real-time analysis of Spark workloads in notebooks 
 and spark-shell/pyspark environments.
 - **Development & CI/CD Integration:** Facilitates testing, measuring, and comparing execution metrics
@@ -28,7 +28,7 @@ and spark-shell/pyspark environments.
 - **Language Compatibility:** Fully supports Scala, Java, and Python, making it versatile for a wide range
   of Spark applications.
 
-### Contents
+### 📚 Table of Contents
 - [Getting started with sparkMeasure](#getting-started-with-sparkmeasure)
   - [Demo](#demo) 
   - [Examples of sparkMeasure on notebooks](#examples-of-sparkmeasure-on-notebooks)  
@@ -40,7 +40,6 @@ and spark-shell/pyspark environments.
 - [Architecture diagram](#architecture-diagram)
 - [Concepts and FAQ](#main-concepts-underlying-sparkmeasure-implementation)
 
-
 ### Resources   
 - Blog on [building an Apache Spark Performance Lab](https://db-blog.web.cern.ch/node/195)
 - [TPCDS PySpark](https://github.com/LucaCanali/Miscellaneous/tree/master/Performance_Testing/TPCDS_PySpark) - A tool you can use run TPCDS with PySpark, instrumented with sparkMeasure
@@ -52,13 +51,11 @@ and spark-shell/pyspark environments.
 Main author and contact: Luca.Canali@cern.ch
 
 ---
-### Getting started with sparkMeasure
-
-#### Demo
+## 🚀 Quick start
 
 [![Watch the video](https://www.youtube.com/s/desktop/050e6796/img/favicon_32x32.png) Watch sparkMeasure's getting started demo tutorial](https://www.youtube.com/watch?v=NEA1kkFcZWs)
 
-#### Examples of sparkMeasure on notebooks
+### Examples of sparkMeasure on notebooks
 - Run locally or on hosted resources like Google Colab, Databricks, GitHub Codespaces, etc on Jupyter notebooks
 
 - [<img src="https://raw.githubusercontent.com/googlecolab/open_in_colab/master/images/icon128.png" height="50"> Jupyter notebook on Google Colab Research](https://colab.research.google.com/github/LucaCanali/sparkMeasure/blob/master/examples/SparkMeasure_Jupyter_Colab_Example.ipynb)
@@ -70,26 +67,29 @@ Main author and contact: Luca.Canali@cern.ch
 - [<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/Jupyter_logo.svg/250px-Jupyter_logo.svg.png" height="50"> Local Python/Jupyter Notebook](examples/SparkMeasure_Jupyter_Python_getting_started.ipynb)
 
 
-#### Examples of sparkMeasure on the CLI 
+### Examples of sparkMeasure on the CLI 
   - Run locally or on hosted resources
     - [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/LucaCanali/sparkMeasure)
-  ```
-  # Scala CLI
-  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
 
-  val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
-  stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
-  ```
+#### Python CLI
   ```
   # Python CLI
   # pip install pyspark
   pip install sparkmeasure
-  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 
   from sparkmeasure import StageMetrics
   stagemetrics = StageMetrics(spark)
   stagemetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
   ```
+#### Scala CLI
+  ```
+  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
+
+  val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
+  stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
+  ```
+
 The output should look like this:
 ```
 +----------+
@@ -143,11 +143,8 @@ Stage 1 duration => 411 (0.4 s)
 Stage 3 duration => 98 (98 ms)
 ```
 
-- Stage metrics collection mode has an optional memory report command
-    - this is available in sparkMeasure since version 0.21, it requires Spark versions 3.1 or higher
-    - note: this report makes use of per-stage memory (executor metrics) data which is sent by the
-      executors at each heartbeat to the driver, there could be a small delay or the order of
-      a few seconds between the end of the job and the time the last metrics value is received.
+### Memory report
+Stage metrics collection mode has an optional memory report command:
 ```
 (scala)> stageMetrics.printMemoryReport
 (python)> stagemetrics.print_memory_report()
@@ -161,12 +158,17 @@ Stage 1 OnHeapExecutionMemory maxVal bytes => 0 (0 Bytes)
 Stage 3 JVMHeapMemory maxVal bytes => 322888344 (307.9 MB)
 Stage 3 OnHeapExecutionMemory maxVal bytes => 0 (0 Bytes)
 ```
+Notes:
+- this is available in sparkMeasure since version 0.21, it requires Spark versions 3.1 or higher
+- note: this report makes use of per-stage memory (executor metrics) data which is sent by the
+  executors at each heartbeat to the driver, there could be a small delay or the order of
+  a few seconds between the end of the job and the time the last metrics value is received.
 
-#### Command line example for Task Metrics:
+### CLI example for Task Metrics:
 This is similar but slightly different from the example above as it collects metrics at the Task-level rather than Stage-level
   ```
   # Scala CLI
-  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 
   val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark)
   taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
@@ -175,41 +177,42 @@ This is similar but slightly different from the example above as it collects met
   # Python CLI
   # pip install pyspark
   pip install sparkmeasure
-  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 
   from sparkmeasure import TaskMetrics
   taskmetrics = TaskMetrics(spark)
   taskmetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
   ```
+---
+## Spark configuration
 
-### Spark configuration
-
-* Choose the sparkMeasure version suitable for your environment:
+* Choose sparkMeasure version suitable for your environment:
   * For Spark 3.x, please use the latest version
+  * For Spark 4.x, please use the latest version and scala 2.13
   * For Spark 2.4 and 2.3, use version 0.19
   * For Spark 2.1 and 2.2, use version 0.16
 
 * Where to get sparkMeasure:
     * [sparkMeasure on Maven Central](https://mvnrepository.com/artifact/ch.cern.sparkmeasure/spark-measure)
-    * Jars in sparkMeasure's [release notes](https://github.com/LucaCanali/sparkMeasure/releases/tag/v0.24)
+    * Jars in sparkMeasure's [release notes](https://github.com/LucaCanali/sparkMeasure/releases/tag/v0.25)
     * Bleeding edge jars as artifacts in [GitHub actions](https://github.com/LucaCanali/sparkMeasure/actions)
     * Build jars from master using sbt: `sbt +package`
 
 * Choose your preferred method to include sparkMeasure in your Spark environment:
-  * `--packages ch.cern.sparkmeasure:spark-measure_2.12:0.24`
-  * `--jars /path/to/spark-measure_2.12-0.24.jar`
-  * `--jars https://github.com/LucaCanali/sparkMeasure/releases/download/v0.24/spark-measure_2.12-0.24.jar`
-  * `--conf spark.driver.extraClassPath=/path/to/spark-measure_2.12-0.24.jar`
+  * `--packages ch.cern.sparkmeasure:spark-measure_2.12:0.25`
+  * `--jars /path/to/spark-measure_2.12-0.25.jar`
+  * `--jars https://github.com/LucaCanali/sparkMeasure/releases/download/v0.25/spark-measure_2.12-0.25.jar`
+  * `--conf spark.driver.extraClassPath=/path/to/spark-measure_2.12-0.25.jar`
 
 Examples:
  * Spark with Scala 2.12:
-   - **Scala:** `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24`
-   - **Python:** `pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24`
+   - **Scala:** `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25`
+   - **Python:** `pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25`
       - note: you also need `pip install sparkmeasure` to get the [Python wrapper API](https://pypi.org/project/sparkmeasure/) 
 
  * Spark with Scala 2.13:
-   - Scala: `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.24`
-   - Python: `pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.24`
+   - Scala: `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.25`
+   - Python: `pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.25`
      - note: `pip install sparkmeasure` to get the Python wrapper API
 
 * Spark 2.4 and 2.3 with Scala 2.11:
@@ -218,7 +221,7 @@ Examples:
         - note: `pip install sparkmeasure==0.19` to get the Python wrapper API
 
 ---
-### Notes on Spark Metrics
+## Notes on Spark Metrics
 Spark is instrumented with several metrics, collected at task execution, they are described in the documentation:  
 - [Spark Task Metrics docs](https://spark.apache.org/docs/latest/monitoring.html#executor-task-metrics)
 
@@ -234,7 +237,7 @@ To learn more about the metrics, I advise you set up your lab environment and ru
 A good place to start with is [TPCDS PySpark](https://github.com/LucaCanali/Miscellaneous/tree/master/Performance_Testing/TPCDS_PySpark) - A tool you can use run TPCDS with PySpark, instrumented with sparkMeasure
 
 ---
-### Documentation, API, and examples 
+## Documentation, API, and examples 
 SparkMeasure is one tool for many different use cases, languages, and environments:
   * [![API Documentation](https://img.shields.io/badge/API-Documentation-brightgreen)](docs/Reference_SparkMeasure_API_and_Configs.md) 
     - [SparkMeasure's API and configurations](docs/Reference_SparkMeasure_API_and_Configs.md)
@@ -291,11 +294,11 @@ SparkMeasure is one tool for many different use cases, languages, and environmen
     a tool for running the TPCDS benchmark workload with PySpark and instrumented with sparkMeasure
 
 ---
-### Architecture diagram  
+## Architecture diagram  
 ![sparkMeasure architecture diagram](docs/sparkMeasure_architecture_diagram.png)
 
 ---
-### Main concepts underlying sparkMeasure implementation 
+## Main concepts underlying sparkMeasure implementation 
 * The tool is based on the Spark Listener interface. Listeners transport Spark executor 
   [Task Metrics](https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_TaskMetrics.md)
   data from the executor to the driver.

build.sbt

@@ -1,58 +1,71 @@
+/****************************************
+ *  sparkMeasure – build definition     *
+ ****************************************/
+
 name := "spark-measure"
 
-version := "0.25-SNAPSHOT"
+version := "0.25"
+isSnapshot := false
 
-scalaVersion := "2.12.18"
+scalaVersion       := "2.12.18"
 crossScalaVersions := Seq("2.12.18", "2.13.8")
 
 licenses += ("Apache-2.0", url("http://www.apache.org/licenses/LICENSE-2.0"))
 
-// publishing to Sonatype Nexus repository and Maven
-publishMavenStyle := true
-isSnapshot := true
-
+// ─── Dependencies ──────────────────────────────────────────────────────────────
 val testDeps = Seq(
-  "org.scalatest" %% "scalatest" % "3.2.19" % Test,
-  "org.scalatest" %% "scalatest-shouldmatchers" % "3.2.19" % Test,
-  "org.wiremock" % "wiremock" % "3.13.0" % Test
+  "org.scalatest"      %% "scalatest"               % "3.2.19" % Test,
+  "org.scalatest"      %% "scalatest-shouldmatchers"% "3.2.19" % Test,
+  "org.wiremock"        % "wiremock"                % "3.13.0" % Test
 )
 
 libraryDependencies ++= Seq(
-  "org.apache.spark" %% "spark-sql" % "3.5.5",
-  "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.18.3",
-  "org.slf4j" % "slf4j-api" % "2.0.17",
-  "org.influxdb" % "influxdb-java" % "2.25",
-  "org.apache.kafka" % "kafka-clients" % "3.9.0"
+  "org.apache.spark"            %% "spark-sql"            % "3.5.5",
+  "com.fasterxml.jackson.module"%% "jackson-module-scala" % "2.18.3",
+  "org.slf4j"                    % "slf4j-api"            % "2.0.17",
+  "org.influxdb"                 % "influxdb-java"        % "2.25",
+  "org.apache.kafka"             % "kafka-clients"        % "3.9.0"
 ) ++ testDeps
 
-// This is for kafka-clients conflicting with Spark 3.5.5 jni dependency, remove with Spark 4.x
+// ── Override to resolve Kafka/Spark JNI clash (remove once on Spark 4.x) ──────
 dependencyOverrides += "com.github.luben" % "zstd-jni" % "1.5.5-4"
 
-// This is for tests, TaskMetricsTest and StageMetricsTest need to access these java packages
-Test / fork := true        // separate JVM so the flags are used
+// ─── Test JVM flags (needed by TaskMetricsTest & StageMetricsTest) ─────────────
+Test / fork := true
 Test / javaOptions ++= Seq(
   "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED",
   "--add-opens=java.base/java.nio=ALL-UNNAMED"
 )
 
-organization := "ch.cern.sparkmeasure"
-description := "sparkMeasure is a tool for performance troubleshooting of Apache Spark workloads."
-developers := List(Developer(
-  "LucaCanali", "Luca Canali", "Luca.Canali@cern.ch",
-  url("https://github.com/LucaCanali")
-))
-homepage := Some(url("https://github.com/LucaCanali/sparkMeasure"))
+// ─── Publishing ────────────────────────────────────────────────────────────────
+publishMavenStyle := true
 
-publishTo := Some(
+publishTo := Some {
   if (isSnapshot.value)
-    Opts.resolver.sonatypeSnapshots
+    Opts.resolver.sonatypeOssSnapshots.head
   else
     Opts.resolver.sonatypeStaging
+}
+
+// ─── Project metadata ─────────────────────────────────────────────────────────
+organization := "ch.cern.sparkmeasure"
+description  := "sparkMeasure is a tool for performance troubleshooting of Apache Spark workloads."
+
+developers := List(
+  Developer(
+    id    = "LucaCanali",
+    name  = "Luca Canali",
+    email = "Luca.Canali@cern.ch",
+    url("https://github.com/LucaCanali")
+  )
 )
 
+homepage := Some(url("https://github.com/LucaCanali/sparkMeasure"))
+
 scmInfo := Some(
   ScmInfo(
-    url("https://github.com/LucaCanali/sparkMeasure"),
-    "scm:git@github.com:LucaCanali/sparkMeasure.git"
+    browseUrl  = url("https://github.com/LucaCanali/sparkMeasure"),
+    connection = "scm:git@github.com:LucaCanali/sparkMeasure.git"
   )
 )
+

docs/Flight_recorder_mode_FileSink.md

@@ -80,7 +80,7 @@ This example collected metrics with Task granularity.
 (note: source the Hadoop environment before running this)
 ```
 bin/spark-submit --master yarn --deploy-mode cluster \
---packages ch.cern.sparkmeasure:spark-measure_2.12:0.24 \
+--packages ch.cern.sparkmeasure:spark-measure_2.12:0.25 \
 --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics \
 --conf spark.sparkmeasure.outputFormat=json_to_hadoop \
 --conf spark.sparkmeasure.outputFilename="hdfs://myclustername/user/luca/test/myoutput_$(date +%s).json" \
@@ -96,7 +96,7 @@ Example, use spark-3.3.0, Kubernetes, Scala 2.12 and write output to S3:
 bin/spark-submit --master k8s://https://XXX.XXX.XXX.XXX --deploy-mode client --conf spark.executor.instances=3 \
 --conf spark.executor.cores=2 --executor-memory 6g --driver-memory 8g \
 --conf spark.kubernetes.container.image=<registry-URL>/spark:v3.0.0_20190529_hadoop32 \
---packages org.apache.hadoop:hadoop-aws:3.3.2,ch.cern.sparkmeasure:spark-measure_2.12:0.24 \
+--packages org.apache.hadoop:hadoop-aws:3.3.2,ch.cern.sparkmeasure:spark-measure_2.12:0.25 \
 --conf spark.hadoop.fs.s3a.secret.key="YYY..." \
 --conf spark.hadoop.fs.s3a.access.key="ZZZ..." \
 --conf spark.hadoop.fs.s3a.endpoint="https://s3.cern.ch" \
@@ -115,7 +115,7 @@ To post-process the saved metrics you will need to deserialize objects saved by
 This is an example of how to do that using the supplied helper object sparkmeasure.Utils
 
 ```
-bin/spark-shell  --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+bin/spark-shell  --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 
 val myMetrics = ch.cern.sparkmeasure.IOUtils.readSerializedStageMetricsJSON("/tmp/stageMetrics_flightRecorder")
 // use ch.cern.sparkmeasure.IOUtils.readSerializedStageMetrics("/tmp/stageMetrics.serialized") for java serialization

docs/Flight_recorder_mode_InfluxDBSink.md

@@ -87,7 +87,7 @@ bin/spark-shell \
   --conf spark.sparkmeasure.influxdbURL="http://localhost:8086" \
   --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSink,ch.cern.sparkmeasure.InfluxDBSinkExtended \
   --conf spark.sparkmeasure.influxdbStagemetrics=true
-  --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+  --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 
 // run a Spark job, this will produce metrics  
 spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show

docs/Flight_recorder_mode_KafkaSink.md

@@ -67,7 +67,7 @@ bin/spark-shell \
 --conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink \
 --conf spark.sparkmeasure.kafkaBroker=localhost:9092 \
 --conf spark.sparkmeasure.kafkaTopic=metrics
---packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+--packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 ```
 
 - Look at the metrics being written into Kafka:

docs/Flight_recorder_mode_PrometheusPushgatewaySink.md

@@ -60,7 +60,7 @@ Examples:
 bin/spark-shell \
 --conf spark.extraListeners=ch.cern.sparkmeasure.PushGatewaySink \
 --conf spark.sparkmeasure.pushgateway=localhost:9091 \
---packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+--packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 ```
 
 - Look at the metrics being written to the Pushgateway

docs/Instrument_Python_code.md

@@ -54,7 +54,7 @@ The details are discussed in the [examples for Python shell and notebook](https:
 
 - This is how to run sparkMeasure using a packaged version in Maven Central
   ```
-  bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24 your_python_code.py
+  bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25 your_python_code.py
 
   // alternative: just download and use the jar (it is only needed in the driver) as in:
   bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.12-0.24.jar ...

docs/Instrument_Scala_code.md

@@ -71,7 +71,7 @@ See details at: [Prometheus Pushgateway](Prometheus.md)
 
 - This is how to run sparkMeasure using a packaged version in Maven Central
     ```
-    bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+    bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
 
     // or just download and use the jar (it is only needed in the driver) as in:
     bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.12-0.24.jar ...

docs/Prometheus.md

@@ -35,7 +35,7 @@ https://prometheus.io/docs/instrumenting/exposition_formats/
 
 1. Measure metrics at the Stage level (example in Scala):
 ```
-bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24 
+bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25 
 
 val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) 
 stageMetrics.begin()

docs/Python_shell_and_Jupyter.md

@@ -9,7 +9,7 @@ See also [README](../README.md) for an introduction to sparkMeasure and its arch
 - Use PyPi to install the Python wrapper and take the jar from Maven central: 
     ```
     pip install sparkmeasure
-    bin/pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.24
+    bin/pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.25
     ```
  - If you prefer to build from the latest development version:
     ```