Add benchmarks for rally runner components#2042
Open
gareth-ellis wants to merge 5 commits into
Open
Conversation
gbanasiak
reviewed
Feb 23, 2026
Comment on lines
+54
to
+59
| - name: Compare results | ||
| id: compare | ||
| working-directory: ${{ github.workspace }} | ||
| run: | | ||
| python3 - <<'PYEOF' | ||
| import json, os, sys |
Contributor
There was a problem hiding this comment.
In rally tracks backporting we went for a script stored separately. Would it make sense to do the same here? Editing Python code inside YAML is not a great experience.
gbanasiak
reviewed
Feb 23, 2026
| only_base = sorted(set(base_benchmarks) - set(pr_benchmarks)) | ||
| only_pr = sorted(set(pr_benchmarks) - set(base_benchmarks)) | ||
|
|
||
| THRESHOLD = 5 # percent |
Contributor
There was a problem hiding this comment.
This feels arbitrary. I would make it clear in the comment if that's the case.
gbanasiak
approved these changes
Feb 23, 2026
Comment on lines
+169
to
+170
| - name: Post or update PR comment | ||
| if: always() && !cancelled() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && (steps.compare.outcome == 'success' || steps.compare.outcome == 'failure') |
Contributor
There was a problem hiding this comment.
Can we avoid the repeat and do the comment in one workflow only?
Comment on lines
+129
to
+130
| def return_raw_response(self): | ||
| pass |
Contributor
There was a problem hiding this comment.
Nit: Avoidable with inheritance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently we don't track the impact of our changes to rally, risking making a change that could impact performance of rally that could then be attributed to an Elasticsearch change.
The aim of this PR is to add a github action that will run (we can perhaps change this so it runs optionally via a label?). and compare the performance of the current branch to that of master.
We have some benchmarks that existed already, that I have included, and also added some for tracking BulkIndex performance.