[Feat] Account for node traffic and stop background transfers at a monthly limit

[al-onyxprotocol]

Summary

Account for the network traffic a node generates and receives, warn at one monthly limit, and stop generating traffic at another, so a node cannot quietly run up a hosting bill.

Details

A node moves bytes on its own initiative, without a user asking for anything. Copies are pushed to peers on upload, the repair job pushes copies again for anything it considers under-replicated, and reads pull blocks from other nodes. All of that is billable on most hosting, and none of it is currently measured or capped.

The repair job is the clearest risk. It runs on replication.repairSchedule, and for every confirmed file it believes is under-replicated it calls the placement path again, which transfers the whole file to any designated peer that does not have it. There is no backoff and no memory of previous failures, so a peer that is full, unreachable, or refusing will be re-sent every file on every pass, indefinitely. With a few thousand files that is a large, repeating transfer that nobody asked for.

Retrieval has a smaller version of the same shape: a node that does not hold a popular file fetches it once and caches it, but the cache is reclaimed when space is short, after which the next read fetches it again. Under sustained pressure that becomes a loop — fetch, cache, evict, fetch — and the same bytes cross the network repeatedly for one popular file. Collection currently has no notion of which blocks were expensive to obtain.

The other cases worth measuring before deciding what to bound:

• A transfer that fails partway is retried from the start on the next pass. Blocks the peer already received stay in its blockstore unpinned, so a retry usually resumes rather than restarting, but a collection in between erases that progress.
• A peer that answers that it has room and then fails the transfer anyway is asked again on every pass. The capacity question makes the common case cheap; it does not make the failing case bounded, which is what backoff is for.
• An upload burst can name the same node as a holder for many files at once, with nothing spreading that load over time.

What to measure

Bytes are the billable unit, and they should be attributed to the reason they moved, because the answers differ:

• copies placed on peers, and copies accepted from peers
• blocks fetched to serve a read, and blocks served to other nodes
• repair, separately from ordinary placement, because it is the one that repeats

Counters have to survive a restart, and they have to roll over on a calendar month to match how hosting is billed.

What to do at the limits

Two thresholds, both from the configuration:

• a warning limit, which logs and shows up in the storage report
• a stop limit, at which the node stops generating traffic of its own accord

Stopping must degrade in the right order. Serving a user who is asking for a file is the node's purpose and should be the last thing to go. Repair and copy placement are background work and should stop first. A node that has stopped replicating is still useful; a node that has stopped serving is not.

Backoff regardless of limits

Even under the limits, repeated failure should cost less each time. A peer that refuses or times out for a file should not be retried on the very next pass, and a file that has failed repeatedly should be retried rarely. This is worth doing whether or not the byte limits are configured, because it removes the unbounded case rather than capping it.

Checklist

• Count bytes sent and received, attributed to placement, repair, retrieval, and serving
• Persist the counters and roll them over monthly
• Report them through the storage metrics
• Add monthly warning and stop limits to the configuration, off by default
• Warn once per period rather than on every event
• At the stop limit, halt repair and copy placement while continuing to serve reads
• Back off retries per peer and per file, independently of the limits
• Avoid re-fetching the same popular file after every collection, or measure how often it happens before deciding it does not matter
• Document what a node does when it reaches each limit, and how an operator resets or raises it

Notes

Comes out of the placement and repair work in #22 and #26. Related to #28: a larger and less curated node set makes background traffic harder to predict.

Verification

• Run a node with a peer that always refuses copies and confirm repair backs off instead of re-sending every file on every pass
• Confirm the counters survive a restart and reset at a month boundary
• Cross the warning limit and confirm it is reported once, not repeatedly
• Cross the stop limit and confirm placement and repair stop while reads are still served
• Confirm a node under the stop limit still answers reads for content it does not hold, or document why it cannot

Coordination with health and compatibility

Traffic limiting changes the node's ability to fulfil its storage role and must feed the bounded health state from #23. Reaching the stop limit may leave existing reads available, but the node must become degraded and its checkpoint height must stop advancing once repair or placement freshness is no longer valid.

The public health response should expose only a bounded policy state such as normal, warning, or stopped. Exact byte counters and billing details belong in storage metrics or authenticated operations output. This addition must not change the legacy /api/node/info fields required by current PWA and iOS clients.

Additional checklist

• Record traffic-policy state for the health checkpoint without exposing detailed billing data publicly
• Freeze checkpoint advancement when a traffic stop makes replication freshness invalid

Additional verification

• Cross the background-transfer stop limit and confirm reads may continue while health becomes degraded or stale and does not advance a false freshness height