Skip to content

jetstreamer-source: use firehose StatsTracking for progress logging#263

Open
senzenn wants to merge 2 commits into
rpcpool:mainfrom
senzenn:feat/jetstreamer-source-stats-tracking
Open

jetstreamer-source: use firehose StatsTracking for progress logging#263
senzenn wants to merge 2 commits into
rpcpool:mainfrom
senzenn:feat/jetstreamer-source-stats-tracking

Conversation

@senzenn

@senzenn senzenn commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

The handler kept its own blocks_processed: AtomicU64 just to log progress every 10k blocks, while firehose() already tracks all of this and we were passing stats_tracking: None.

  • new stats_interval_slots config field (default 10000, 0 disables)
  • register a StatsTracking callback that logs slots/blocks/transactions/entries/leader-skipped + tps; the callback fires per worker thread so thread_id is included
  • drop the atomic counter and its two logging sites
  • add --stats-interval-slots to the jetstreamer example

Tested against the old-faithful archive (50 slots of epoch 885, interval 10):

Firehose progress thread_id=0 slots=11 blocks=11 transactions=12435 entries=9535 leader_skipped_slots=0 tps=2902
Firehose progress thread_id=2 slots=24 blocks=24 transactions=27338 entries=20563 leader_skipped_slots=0 tps=417
Firehose progress thread_id=1 slots=47 blocks=47 transactions=53267 entries=33296 leader_skipped_slots=0 tps=3578

…StatsTracking

Register upstream `firehose` `StatsTracking` instead of hand-counting blocks in
`process_block`. Periodic progress now reports the engine's own aggregates —
slots, blocks, transactions, entries, leader-skipped slots — plus TPS, emitted
per worker thread (thread_id logged to disambiguate the global aggregates).

- Add `stats_interval_slots` to `JetstreamSourceConfig` (default 10000). `0`
  disables progress logging and avoids upstream's unguarded `slot % interval`.
- Drop the `blocks_processed: AtomicU64` field and its two debug-logging sites.
- Expose `--stats-interval-slots` on the jetstreamer example.
@senzenn senzenn changed the title feat(jetstreamer-source): replace ad-hoc block counter with firehose StatsTracking jetstreamer-source: use firehose StatsTracking for progress logging Jun 5, 2026
Comment thread examples/jetstreamer/src/main.rs Outdated
Comment on lines +697 to +702
let stats_tracking = (config.stats_interval_slots != 0).then_some(
jetstreamer_firehose::firehose::StatsTracking {
on_stats: log_firehose_stats,
tracking_interval_slots: config.stats_interval_slots,
},
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let stats_tracking = (config.stats_interval_slots != 0).then_some(
jetstreamer_firehose::firehose::StatsTracking {
on_stats: log_firehose_stats,
tracking_interval_slots: config.stats_interval_slots,
},
);
let stats_tracking = config.stats_interval_slots.map(|tracking_interval_slots| {
jetstreamer_firehose::firehose::StatsTracking {
on_stats: log_firehose_stats,
tracking_interval_slots,
}
});

If go as optional can use a map and the None to signal that logging should be disabled.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.map needs the field to be Option<u64> (it's u64 right now), and the != 0 check is load-bearing — upstream does slot % tracking_interval_slots unguarded ([firehose.rs:1575](https://github.com/rpcpool/yellowstone-vixen/blob/main/crates/jetstreamer-firehose/src/firehose.rs#L1575)), so a 0 has to become None or it panics. A plain .map passes Some(0) straight through and brings the panic back.

Happy to move to Option<u64>, just keeping the guard:

let stats_tracking = config
    .stats_interval_slots
    .filter(|&n| n != 0)
    .map(|tracking_interval_slots| jetstreamer_firehose::firehose::StatsTracking {
        on_stats: log_firehose_stats,
        tracking_interval_slots,
    });

Want it opt-out (default Some(10_000)) or opt-in (default None)? I'll fix up the defaults and tests to match.

thread_id: usize,
stats: jetstreamer_firehose::firehose::Stats,
) -> futures_util::future::BoxFuture<'static, Result<(), SharedError>> {
async move {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to return a future or can the function be made async and passed as the callback?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has to be a boxed future upstream's Handler<Data> trait requires Fn(usize, Data) -> BoxFuture<'static, HandlerResult>, so a bare async fn won't satisfy the bound. .boxed() erases the opaque future into the BoxFuture it expects.

Same shape as the existing on_block/on_tx/on_entry callbacks at lib.rs:663–681. Happy to inline it as a closure next to those if you'd prefer.

Co-authored-by: Kyle Espinola <kyle.s.espinola@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants