feat(model-service): proxy supports stream + replay, byte passthrough, ForwardBackend/ReplayBackend#935

Merged

BCeZn merged 25 commits into

alibaba:masterfrom

BCeZn:feautre/xinshi/litellm-proxy

May 13, 2026

Commits on May 12, 2026

refactor(model-service): rebuild proxy on litellm SDK with traj record/replay

BCeZn
and
claude
committed
fix(model-service): pass api_key + use custom_openai prefix + suppress cost calc

BCeZn
and
claude
committed
refactor(model-service): drop CustomLLM, serve replay directly from cursor

BCeZn
and
claude
committed
refactor(model-service): drop litellm, use httpx byte-passthrough + openai SDK as parser

BCeZn
and
claude
committed
refactor: use openai sdk
BCeZn
committed
chore(model-service): remove litellm remnants (num_retries, stale comments)

BCeZn
and
claude
committed
refactor(model-service): split proxy handler into _ReplayBackend / _ForwardBackend

BCeZn
and
claude
committed
refactor(model-service): drop _ prefix from public Backend classes, rename replay_traj_path → replay_traj_file

BCeZn
and
claude
committed
feat(model-service): restore retry on retryable_status_codes + connection errors

BCeZn
and
claude
committed
test(model-service): add e2e tests against an in-thread uvicorn mock upstream

BCeZn
and
claude
committed
fix(model-service): inject positional index into replay-stream tool_calls deltas

BCeZn
and
claude
committed
test(model-service): refactor proxy e2e into MockUpstream + TestProxyRecordReplay

BCeZn
and
claude
committed
chore: remove useless comment in pyproject.toml
BCeZn
committed
chore: remove uesless dev docs
BCeZn
committed
refactor(model-service): flatten layout — drop integrations/, rename sse_utils→sse, merge traj_*→traj

BCeZn
and
claude
committed
refactor(model-service): rename traj_file→recording_file, replay_traj_file→replay_file

BCeZn
and
claude
committed
feat(model-service): enforce recording_file/replay_file mutex via model_validator

BCeZn
and
claude
committed
refactor(model-service): move _replay_sse_iter into ReplayBackend as a staticmethod

BCeZn
and
claude
committed
refactor(model-service): move get_base_url into ForwardBackend as _resolve_base_url

BCeZn
and
claude
committed
refactor(model-service): move _forward_stream_and_record into ForwardBackend as _stream_and_record

BCeZn
and
claude
committed
docs(model-service): move + rewrite proxy README under docs/dev/model-service/

BCeZn
and
claude
committed
feat(model-service): expose --recording-file / --replay-file on rock model-service start

BCeZn
and
claude
committed
test(model-service): add CLI-layer coverage for --recording-file / --replay-file wiring

BCeZn
and
claude
committed
test(model-service): rename test_service.py → test_service_subprocess.py to fix CI collision

BCeZn
and
claude
committed
test(model-service): rename test_proxy_record_replay.py → ..._e2e.py

BCeZn
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(model-service): proxy supports stream + replay, byte passthrough, ForwardBackend/ReplayBackend#935

feat(model-service): proxy supports stream + replay, byte passthrough, ForwardBackend/ReplayBackend#935
BCeZn merged 25 commits into
alibaba:masterfrom
BCeZn:feautre/xinshi/litellm-proxy

Commits on May 12, 2026

refactor(model-service): rebuild proxy on litellm SDK with traj record/replay

fix(model-service): pass api_key + use custom_openai prefix + suppress cost calc

refactor(model-service): drop CustomLLM, serve replay directly from cursor

refactor(model-service): drop litellm, use httpx byte-passthrough + openai SDK as parser

refactor: use openai sdk

chore(model-service): remove litellm remnants (num_retries, stale comments)

refactor(model-service): split proxy handler into _ReplayBackend / _ForwardBackend

refactor(model-service): drop _ prefix from public Backend classes, rename replay_traj_path → replay_traj_file

feat(model-service): restore retry on retryable_status_codes + connection errors

test(model-service): add e2e tests against an in-thread uvicorn mock upstream

fix(model-service): inject positional index into replay-stream tool_calls deltas

test(model-service): refactor proxy e2e into MockUpstream + TestProxyRecordReplay

chore: remove useless comment in pyproject.toml

chore: remove uesless dev docs

refactor(model-service): flatten layout — drop integrations/, rename sse_utils→sse, merge traj_*→traj

refactor(model-service): rename traj_file→recording_file, replay_traj_file→replay_file

feat(model-service): enforce recording_file/replay_file mutex via model_validator

refactor(model-service): move _replay_sse_iter into ReplayBackend as a staticmethod

refactor(model-service): move get_base_url into ForwardBackend as _resolve_base_url

refactor(model-service): move _forward_stream_and_record into ForwardBackend as _stream_and_record

docs(model-service): move + rewrite proxy README under docs/dev/model-service/

feat(model-service): expose --recording-file / --replay-file on rock model-service start

test(model-service): add CLI-layer coverage for --recording-file / --replay-file wiring

test(model-service): rename test_service.py → test_service_subprocess.py to fix CI collision

test(model-service): rename test_proxy_record_replay.py → ..._e2e.py