Prepare release 0.27

Author: LucaCanali

.github/workflows/build_with_scala_and_python_tests.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-24.04
     strategy:
       matrix:
-        python-version: [ '3.9', '3.12' ]
+        python-version: [ '3.10', '3.12' ]
         java-version: [ 11, 17 ]
 
     steps:
@@ -25,7 +25,7 @@ jobs:
         cache: sbt
     - uses: sbt/setup-sbt@v1
       with:
-        sbt-runner-version: 1.11.5
+        sbt-runner-version: 1.11.6
 
     - name: Build using sbt for all Scala versions supported
       run: sbt +package

README.md

@@ -32,14 +32,14 @@ and spark-shell/pyspark environments.
     - [✨ Highlights](#highlights)
     - [📚 Table of Contents](#tableofcontents)
     - [Links to related work on Spark Performance](#links-to-related-work-on-spark-performance)
-  - [🚀 Quick start](#quickstart)
+- [🚀 Quick start](#quickstart)
     - [Examples of sparkMeasure on notebooks](#examples-of-sparkmeasure-on-notebooks)
     - [Examples of sparkMeasure on the CLI](#examples-of-sparkmeasure-on-the-cli)
       - [Python CLI](#python-cli)
       - [Scala CLI](#scala-cli)
     - [Memory report](#memory-report)
     - [CLI example for Task Metrics:](#cli-example-for-task-metrics)
-  - [Setting Up SparkMeasure with Spark](#setting-up-sparkmeasure-with-spark)
+- [Setting Up SparkMeasure with Spark](#setting-up-sparkmeasure-with-spark)
     - [Version Compatibility for SparkMeasure](#version-compatibility-for-sparkmeasure)
     - [📥 Downloading SparkMeasure](#-downloading-sparkmeasure)
     - [Setup Examples](#setup-examples)
@@ -48,7 +48,7 @@ and spark-shell/pyspark environments.
     - [Including sparkMeasure in your Spark environment](#including-sparkmeasure-in-your-spark-environment)
   - [Running unit tests](#running-unit-tests)
   - [Notes on Spark Metrics](#notes-on-spark-metrics)
-  - [Documentation, API, and examples](#documentation-api-and-examples)
+- [Documentation, API, and examples](#documentation-api-and-examples)
   - [Architecture diagram](#architecture-diagram)
   - [Main concepts underlying sparkMeasure implementation](#main-concepts-underlying-sparkmeasure-implementation)
     - [FAQ:](#faq)
@@ -95,7 +95,7 @@ Main author and contact: Luca.Canali@cern.ch
   # Python CLI
   # pip install pyspark
   pip install sparkmeasure
-  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
   # Import sparkMeasure
   from sparkmeasure import StageMetrics
@@ -118,12 +118,12 @@ Main author and contact: Luca.Canali@cern.ch
   # get metrics as a dictionary
   metrics = stagemetrics.aggregate_stage_metrics()
   ```
-Note: for Spark 3.x with Scala 2.12, use `--packages ch.cern.sparkmeasure:spark-measure_2.12:0.26` 
-instead of `--packages ch.cern.sparkmeasure:spark-measure_2.13:0.26`
+Note: for Spark 3.x with Scala 2.12, use `--packages ch.cern.sparkmeasure:spark-measure_2.12:0.27` 
+instead of `--packages ch.cern.sparkmeasure:spark-measure_2.13:0.27`
 
 #### Scala CLI
   ```
-  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
   val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
   stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
@@ -206,7 +206,7 @@ Notes:
 This is similar but slightly different from the example above as it collects metrics at the Task-level rather than Stage-level
   ```
   # Scala CLI
-  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
   val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark)
   taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show())
@@ -215,7 +215,7 @@ This is similar but slightly different from the example above as it collects met
   # Python CLI
   # pip install pyspark
   pip install sparkmeasure
-  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
   from sparkmeasure import TaskMetrics
   taskmetrics = TaskMetrics(spark)
@@ -229,8 +229,8 @@ This is similar but slightly different from the example above as it collects met
 
 | Spark Version  | Recommended SparkMeasure Version | Scala Version       |
 | -------------- |----------------------------------|---------------------|
-| Spark 4.x      | 0.26 (latest)                    | Scala 2.13          |
-| Spark 3.x      | 0.26 (latest)                    | Scala 2.12 and 2.13 |
+| Spark 4.x      | 0.27 (latest)                    | Scala 2.13          |
+| Spark 3.x      | 0.27 (latest)                    | Scala 2.12 and 2.13 |
 | Spark 2.4, 2.3 | 0.19                             | Scala 2.11          |
 | Spark 2.2, 2.1 | 0.16                             | Scala 2.11          |
 
@@ -244,7 +244,7 @@ To get SparkMeasure, choose one of the following options:
 
 2. **Specific Versions:**
 
-    * Download JAR files from the [sparkMeasure release notes](https://github.com/LucaCanali/sparkMeasure/releases/tag/v0.26).
+    * Download JAR files from the [sparkMeasure release notes](https://github.com/LucaCanali/sparkMeasure/releases/tag/v0.27).
 
 3. **Latest Development Builds:**
 
@@ -258,21 +258,21 @@ To get SparkMeasure, choose one of the following options:
 
 #### Spark 4 with Scala 2.13
 
-* **Scala:** `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26`
+* **Scala:** `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27`
 * **Python:**
 
   ```bash
-  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
   pip install sparkmeasure
   ```
 
 #### Spark 3 with Scala 2.12
 
-* **Scala:** `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.26`
+* **Scala:** `spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.12:0.27`
 * **Python:**
 
   ```bash
-  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.26
+  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.27
   pip install sparkmeasure
   ```
 ### Including sparkMeasure in your Spark environment
@@ -282,14 +282,14 @@ Choose your preferred method:
 * Use the `--packages` option:
 
   ```bash
-  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
   ```
 * Directly reference the JAR file:
 
   ```bash
-  --jars /path/to/spark-measure_2.13-0.26.jar
-  --jars https://github.com/LucaCanali/sparkMeasure/releases/download/v0.26/spark-measure_2.13-0.26.jar
-  --conf spark.driver.extraClassPath=/path/to/spark-measure_2.13-0.26.jar
+  --jars /path/to/spark-measure_2.13-0.27.jar
+  --jars https://github.com/LucaCanali/sparkMeasure/releases/download/v0.27/spark-measure_2.13-0.27.jar
+  --conf spark.driver.extraClassPath=/path/to/spark-measure_2.13-0.27.jar
   ```
 
 

build.sbt

@@ -4,7 +4,7 @@
 
 name := "spark-measure"
 
-version := "0.27-SNAPSHOT"
+version := "0.27"
 
 scalaVersion       := "2.12.18"
 crossScalaVersions := Seq("2.12.18", "2.13.16")

docs/Flight_recorder_mode_FileSink.md

@@ -12,7 +12,7 @@ Metrics can also be printed to stdout.
 ## Recording metrics using the Flight Recorder mode with Stage-level granularity  
 To record metrics at the stage execution level granularity add these configurations to spark-submit: 
    ```
-   --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+   --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
    --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics
    ```
 
@@ -25,7 +25,7 @@ The usage is almost the same as for the stage metrics mode described above, just
 The configuration parameters applicable to Flight recorder mode for Task granularity are:
 
    ```
-   --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+   --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
    --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics
    ```
 
@@ -51,7 +51,7 @@ A Python example
  - This runs the pi.py example script 
  - collects and saves the metrics to `/tmp/stageMetrics_flightRecorder` in json format:
 ```
-bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 \
+bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 \
 --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics \
 examples/src/main/python/pi.py
 
@@ -63,7 +63,7 @@ A Scala example
 - same example as above, in addition use a custom output filename
 - print metrics also to stdout
 ```
-bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 \
+bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 \
 --class org.apache.spark.examples.SparkPi \
 --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics \
 --conf spark.sparkmeasure.printToStdout=true \
@@ -80,7 +80,7 @@ This example collected metrics with Task granularity.
 (note: source the Hadoop environment before running this)
 ```
 bin/spark-submit --master yarn --deploy-mode cluster \
---packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 \
+--packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 \
 --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics \
 --conf spark.sparkmeasure.outputFormat=json_to_hadoop \
 --conf spark.sparkmeasure.outputFilename="hdfs://myclustername/user/luca/test/myoutput_$(date +%s).json" \
@@ -96,7 +96,7 @@ Example, use Spark 4, Kubernetes, Scala 2.13 and write output to S3:
 bin/spark-submit --master k8s://https://XXX.XXX.XXX.XXX --deploy-mode client --conf spark.executor.instances=3 \
 --conf spark.executor.cores=2 --executor-memory 6g --driver-memory 8g \
 --conf spark.kubernetes.container.image=apache/spark \
---packages org.apache.hadoop:hadoop-aws:3.4.1,ch.cern.sparkmeasure:spark-measure_2.13:0.26 \
+--packages org.apache.hadoop:hadoop-aws:3.4.2,ch.cern.sparkmeasure:spark-measure_2.13:0.27 \
 --conf spark.hadoop.fs.s3a.secret.key="YYY..." \
 --conf spark.hadoop.fs.s3a.access.key="ZZZ..." \
 --conf spark.hadoop.fs.s3a.endpoint="https://s3.cern.ch" \
@@ -105,7 +105,7 @@ bin/spark-submit --master k8s://https://XXX.XXX.XXX.XXX --deploy-mode client --c
 --conf spark.sparkmeasure.outputFormat=json_to_hadoop \
 --conf spark.sparkmeasure.outputFilename="s3a://test/myoutput_$(date +%s).json" \
 --class org.apache.spark.examples.SparkPi \
-examples/jars/spark-examples_2.13-4.0.0.jar 10
+examples/jars/spark-examples_2.13-4.0.1.jar 10
 ```
 
 
@@ -115,7 +115,7 @@ To post-process the saved metrics you will need to deserialize objects saved by
 This is an example of how to do that using the supplied helper object sparkmeasure.Utils
 
 ```
-bin/spark-shell  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+bin/spark-shell  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
 val myMetrics = ch.cern.sparkmeasure.IOUtils.readSerializedStageMetricsJSON("/tmp/stageMetrics_flightRecorder")
 // use ch.cern.sparkmeasure.IOUtils.readSerializedStageMetrics("/tmp/stageMetrics.serialized") for java serialization

docs/Flight_recorder_mode_InfluxDBSink.md

@@ -87,7 +87,7 @@ bin/spark-shell \
   --conf spark.sparkmeasure.influxdbURL="http://localhost:8086" \
   --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSink,ch.cern.sparkmeasure.InfluxDBSinkExtended \
   --conf spark.sparkmeasure.influxdbStagemetrics=true
-  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+  --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
 // run a Spark job, this will produce metrics  
 spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show

docs/Flight_recorder_mode_KafkaSink.md

@@ -44,7 +44,7 @@ This code depends on "kafka-clients". If you deploy sparkMeasure from maven cent
 the dependency is being taken care of.
 If you run sparkMeasure from a jar instead, you may need to add the dependency manually
 in spark-submit as in:
- - `--packages org.apache.kafka:kafka-clients:3.9.0`
+ - `--packages org.apache.kafka:kafka-clients:4.1.0`
 
 ## Use cases
 
@@ -69,7 +69,7 @@ bin/spark-shell \
 --conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink \
 --conf spark.sparkmeasure.kafkaBroker=localhost:9092 \
 --conf spark.sparkmeasure.kafkaTopic=metrics
---packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+--packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 ```
 
 - Look at the metrics being written into Kafka:

docs/Flight_recorder_mode_PrometheusPushgatewaySink.md

@@ -60,7 +60,7 @@ Examples:
 bin/spark-shell \
 --conf spark.extraListeners=ch.cern.sparkmeasure.PushGatewaySink \
 --conf spark.sparkmeasure.pushgateway=localhost:9091 \
---packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+--packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 ```
 
 - Look at the metrics being written to the Pushgateway

docs/Instrument_Python_code.md

@@ -11,7 +11,7 @@ You can find an example of how to instrument a Scala application running Apache
 
 How to run the example:
  ```
-bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 <path_to_examples>/test_sparkmeasure_python.py
+bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 <path_to_examples>/test_sparkmeasure_python.py
  ```
 
  Some relevant snippet of code are:
@@ -54,10 +54,10 @@ The details are discussed in the [examples for Python shell and notebook](https:
 
 - This is how to run sparkMeasure using a packaged version in Maven Central
   ```
-  bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 your_python_code.py
+  bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 your_python_code.py
 
   // alternative: just download and use the jar (it is only needed in the driver) as in:
-  bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.26.jar ...
+  bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.27.jar ...
   ```
 
 ### Download and build sparkMeasure (optional)
@@ -73,8 +73,8 @@ The details are discussed in the [examples for Python shell and notebook](https:
      pip install .
   
      # Run as in one of these examples:
-     bin/spark-submit --jars path>/spark-measure_2.13-0.27-SNAPSHOT.jar ...
+     bin/spark-submit --jars path>/spark-measure_2.13-0.28-SNAPSHOT.jar ...
      
      # alternative, set classpath for the driver (sparkmeasure code runs only in the driver)
-     bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.27-SNAPSHOT.jar ...
+     bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.28-SNAPSHOT.jar ...
      ```

docs/Instrument_Scala_code.md

@@ -13,7 +13,7 @@ How to run the example:
 # build the example jar
 sbt package
 
-bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 --class ch.cern.testSparkMeasure.testSparkMeasure <path_to_the_example_jar>/testsparkmeasurescala_2.13-0.1.jar
+bin/spark-submit --master local[*] --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 --class ch.cern.testSparkMeasure.testSparkMeasure <path_to_the_example_jar>/testsparkmeasurescala_2.13-0.1.jar
  ```
 
 ### Collect and save Stage Metrics
@@ -72,10 +72,10 @@ You have the option to export aggregated stage metrics and/or task metrics to:
 
 - This is how to run sparkMeasure using a packaged version in Maven Central
     ```
-    bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26
+    bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27
 
     // or just download and use the jar (it is only needed in the driver) as in:
-    bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.26.jar ...
+    bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.27.jar ...
    ```
 - The alternative, see paragraph above, is to build a jar from master (See below).
 
@@ -86,11 +86,11 @@ You have the option to export aggregated stage metrics and/or task metrics to:
    git clone https://github.com/lucacanali/sparkmeasure
    cd sparkmeasure
    sbt +package
-   ls -l target/scala-2.12/spark-measure*.jar  # location of the compiled jar
+   ls -l target/scala-2.13/spark-measure*.jar  # location of the compiled jar
 
    # Run as in one of these examples:
-   bin/spark-submit --jars path>/spark-measure_2.13-0.27-SNAPSHOT.jar
+   bin/spark-submit --jars path>/spark-measure_2.13-0.28-SNAPSHOT.jar
 
    # alternative, set classpath for the driver (it is only needed in the driver)
-   bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.27-SNAPSHOT.jar ...
+   bin/spark-submit --conf spark.driver.extraClassPath=<path>/spark-measure_2.13-0.28-SNAPSHOT.jar ...
    ```

docs/Prometheus.md

@@ -35,7 +35,7 @@ https://prometheus.io/docs/instrumenting/exposition_formats/
 
 1. Measure metrics at the Stage level (example in Scala):
 ```
-bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.26 
+bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 
 
 val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) 
 stageMetrics.begin()