EAGLE3.1 Support

[bluecoffee8]

Motivation

EAGLE3.1 support, based on https://github.com/lightseekorg/TorchSpec/pull/97 which was added to torchspec.

Validation: trained Qwen3-30B-A3B-Instruct-2507 draft model (EAGLE3 vs EAGLE3.1) on a single epoch of sharegpt dataset, based on https://github.com/sgl-project/SpecForge/blob/main/examples/run_qwen3_30b_a3b_eagle3_online.sh.

Modifications

Add eagle3.1 features (draft model output norm, fc norm on target hidden states).
Added example eagle3.1 config for qwen3-30b-a3b model, and script.

Related Issues

Accuracy Test

Benchmark & Profiling

Ran benchmarks for each.

Server launch (1 x H100):

SGLANG_ENABLE_SPEC_V2=1 && SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 && python -m sglang.launch_server --host 0.0.0.0 --port 8001 --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --speculative-algorithm EAGLE3 --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-path /path/to/draft_model

Client launch:

python3 -m sglang.bench_serving --backend sglang --host 0.0.0.0 --port 8001 --dataset-name random-ids --warmup-requests 80 --num-prompts 128 --random-input-len 1024 --random-output-len 256 --random-range-ratio 1.0 --request-rate 1.0 

Results:
EAGLE3

EAGLE3.1

Comparison:

Acc length 1.68 => 2.30, +36%
p50 e2e, 1550.76 => 823.57, -47%
p50 tpot, 5.75 => 2.94, -49%

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://sgl-fru7574.slack.com/archives/C09784E3EN6 to discuss your PR.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Dogacel]

compute_logits already applies norm to the hidden_states. We should gate it if the hidden states is already normed.

[Dogacel]

Let's keep assertion and assert to self.num_aux_hidden_states.

[Dogacel]

Similarly,

I think it makes sense to calculate using hidden_states, hidden_states_for_logits = get_hidden_states(...) as a pair, where the right one is normed and left is not by default. The method can return both normed based on norm_output.

[bluecoffee8]

@Dogacel i just pushed a commit that should address your comments. could you plz take a look thanks!

[Dogacel]

LGTM 👍

[xiangqianzsh]

@bluecoffee8 Hi, I have trained an eagle3.1 model using this patch. Which branch or commit id should I use when testing performance in sglang?

[bluecoffee8]

@xiangqianzsh I think you should be able to use any, the draft model trained here should just run with EAGLE-3 spec decoding option in sglang.

@Dogacel could you confirm there should be no changes to the inference engine side of things required?

[Dogacel]

@xiangqianzsh I think you should be able to use any, the draft model trained here should just run with EAGLE-3 spec decoding option in sglang.

@Dogacel could you confirm there should be no changes to the inference engine side of things required?

Yeah it should work out of the box.

[jiapingW]

Hi, thank you for implementing eagle3.1. I'd like to know why your bench is using random_ids. Comparing accept lengths with random data as input doesn't make much sense, does it? @bluecoffee8

[bluecoffee8]

@jiapingW you can try with other benchmark method like sharegpt dataset.

[jiapingW]

@jiapingW you can try with other benchmark method like sharegpt dataset.

Can you opensource your eagle3.1 draft model? We can test it.

[bluecoffee8]

@jiapingW I don't have option to open source the model, but let me try to benchmark it when I have the bandwidth. If anybody else has the GPU resource they could try to train and benchmark in the meantime as well.