Skip to content

python.md

Latest commit

 

History

History
95 lines (70 loc) · 2.86 KB

python.md

File metadata and controls

95 lines (70 loc) · 2.86 KB
Raw

Python

Python support is stdlib-only: one small client library and one standalone stress example.

Files

Example

Push and pop from Python:

python3 ./examples/python/stress_runner.py --duration 2 --publishers 2 --consumers 2

Use the client library directly:

from atomic_queue import AtomicQueueClient

client = AtomicQueueClient()
client.push("jobs", b'{"job":"reindex","id":123}')
channel, message = client.pop_message(["jobs"], 5000)
print(channel, message)
client.close()

Run the standalone Python stress example:

python3 ./examples/python/stress_runner.py --duration 10 --threads 1000

JSON output mode:

python3 ./examples/python/stress_runner.py --duration 10 --publishers 500 --consumers 500 --format json

Run 100 Python stress instances with GNU parallel:

./scripts/python-parallel-stress.sh

Override with env vars:

INSTANCES=100 DURATION=10 PUBLISHERS=5 CONSUMERS=5 PARALLEL_JOBS=100 ./scripts/python-parallel-stress.sh

By default it now targets 10s and 1000 total workers across 100 instances.

Performance On This Machine

Python note:

  • The GNU parallel wrapper is the practical benchmark path for Python in this repo and should be used first when you want serious load.
  • The single-process threaded stress example is GIL-bound and does a lot of small Python-level socket/frame work, so it does not scale like the Go or PHP runners.
  • If you want higher Python-side throughput, run many processes with GNU parallel or another process-based launcher instead of piling on more threads in one interpreter.

GNU parallel, 100 instances, 5 producers and 5 consumers per instance:

❯ ATOMIC_QUEUE_SOCKET=/tmp/atomic-queue.sock ATOMIC_QUEUE_BIN=./atomic-queue \
  INSTANCES=100 DURATION=10 PUBLISHERS=5 CONSUMERS=5 PARALLEL_JOBS=100 ./scripts/python-parallel-stress.sh
stress duration: 12.318s
workers: 1000 (500 producers, 500 consumers) across 100 instances
channels: stress-a, stress-b, stress-c, stress-d
messages pushed: 2977283
messages served: 3100775
pop timeouts: 0
client failures: 0
push rate: 241701.82 msg/s
serve rate: 251727.15 msg/s

Single-process threaded stress example:

❯ ATOMIC_QUEUE_SOCKET=/tmp/atomic-queue.sock ATOMIC_QUEUE_BIN=./atomic-queue \
  python3 ./examples/python/stress_runner.py --duration 10 --threads 1000
stress duration: 10.068s
threads: 1000 (500 producers, 500 consumers)
channels: stress-a, stress-b, stress-c, stress-d
messages pushed: 447156
messages served: 45429
pop timeouts: 0
client failures: 0
push rate: 44412.50 msg/s
serve rate: 4512.11 msg/s