Support multithreading of rpki validation with rpki-client 9.6

[fjahr]

From the release notes of rpki-client 9.6:

The parser process now uses parallel threads for object validation. The new -p option can be used to adjust the number of threads.

This sounds potentially very useful to us. Since this is a new feature it will need to tested carefully and we potentially might need to add it in a backward compatible way of just wait a bit if the performance gain isn't huge.

[fjahr]

cc @jurraca since you were eager for some further performance improvements, let me know if you want to take this, otherwise I can probably try it out soon.

[jurraca]

ooh very nice, i'll take it! already working on something performance-related.

[jurraca]

I tested this and I might be holding it wrong but I see significantly worse runtime with this compared to our current ThreadPoolExecutor() implementation.
As a reminder, the existing ThreadPoolExecutor implementation takes batches of 250 ROAs, and validates them in parallel by spawning a rpki-client subprocess that runs in a single thread.

For a given set of RPKI data as a baseline, the current implementation gives a runtime for the validation step of 3m26s.

I tested a few different tweaks:

• keeping our existing ThreadPoolExecutor implementation, and adding the -p 8 took 3m25s. So each spawned rpki-client thread is attempting to further spawn across the 8 cores, which is probably not ideal but the result is almost identical to the baseline.
• removing our existing ThreadPoolExecutor implementation, and processing each batch linearly in batches of 250 ROAs, letting rpki-client spawn 8 processes with -p 8, it took 18m10s.
• bumping thread count to -p 16 in case in could take advantage of logical cores, result is 17m12s.
• same as before with -p 8, but with a larger batch size of 1000: 17m02s and 5000 17m06s.
• Back to our initial ThreadPoolExecutor implementation, with batches of 500, gets us a negligible improvement of 3m16s runtime compared to the baseline.

Lmk if you think I should test another way, or what might be the cause of the gap here?

[fjahr]

Hm, looking at the docs it looks like the problem might be the batching. The -p mentions in the last sentence: "Has no effect in combination with -f." https://man.openbsd.org/rpki-client

So I guess the only relevant benchmark to compare would be -p without any batching just giving it the cache dir. If that doesn't beat our current performance i guess it doesn't help us.

[jurraca]

ah, thanks. Removing our batching/parallel implementation and using the flag performs the validation in about 45 seconds, which is quite a bit better! However the output format is a fair bit different and I have to adjust a few things. It also has a handful of additional metadata which may be interesting to us. PR soon 👍

[fjahr]

However the output format is a fair bit different and I have to adjust a few things.

It's probably going to be similar to how it was before the batching was introduced, you could look at the old code but not sure it still applies because there were a few other changes probably.

[jurraca]

so batching with -f gives us basically "debug" level info about the ROA that we use to handle duplicates in rpki/parse.py. But using -p gives us the validated ROA with prefix, maxLength, asn, expiry and ta, but not, for example, valid_since which we use as an ultimate tie-breaker.
The problem is that there are conflicting announcements for the same prefix and ASN pair (with the same expiry), between 40k and 70k in a given RPKI pull. They're both valid, since they're signed. Usually we would dedupe on valid_since and take the most recent one, but we can't do that with -p. Off of a few runs, it seems that majority of RPKI duplicates are handled by the "smallest ASN wins", so it's maybe relevant to 10-15k ROAs, but we're still losing precision by not using the latest announced ROA.

Brings up a few questions:

• why so many conflicts from the RPKI DBs?
• should we handle these conflicts differently? drop them and let IRR sort it out?
• should we keep the current impl?

ROA formats

current ROA in JSON format: ```json { "file": "/.../data/1755187200/rpki/cache/rpki.arin.net/repository/arin-rpki-ta/5e4a23ea-e80a-403e-b08c-2171da2157d3/2a246947-2d62-4a6c-ba05-87187f0099b2/4e95a28e-27fe-479a-b086-2cc9809d54f6/7ff7cbfc-c8df-33ab-8fc6-00715caf3de6.roa", "hash_id": "JCorwokE9OLd5xCBjNXzgcd01q7YFx6Agr60CK8UQ+E=", "type": "roa", "ski": "82:83:68:D5:B0:47:74:74:17:7A:3A:86:34:78:D7:F5:9C:DA:A2:02", "cert_issuer": "/CN=4e95a28e-27fe-479a-b086-2cc9809d54f6", "cert_serial": "010D0C9F4328584A0C05AB172C1A2F41DDD2AF54", "aki": "55:43:28:31:2B:A2:11:E9:0C:63:2A:6B:C1:7E:0D:D6:92:76:3E:61", "aia": "rsync://rpki.arin.net/repository/arin-rpki-ta/5e4a23ea-e80a-403e-b08c-2171da2157d3/2a246947-2d62-4a6c-ba05-87187f0099b2/4e95a28e-27fe-479a-b086-2cc9809d54f6.cer", "sia": "rsync://rpki.arin.net/repository/arin-rpki-ta/5e4a23ea-e80a-403e-b08c-2171da2157d3/2a246947-2d62-4a6c-ba05-87187f0099b2/4e95a28e-27fe-479a-b086-2cc9809d54f6/7ff7cbfc-c8df-33ab-8fc6-00715caf3de6.roa", "signing_time": 1748782863, "valid_since": 1748782863, "valid_until": 1756558863, "expires": 1755302400, "vrps": [ { "prefix": "73.80.70.0/23", "asid": 33659, "maxlen": 23 } ], "validation": "OK" } ``` ROA with `-p`: ```json { "asn": 9583, "prefix": "1.6.111.0/24", "maxLength": 24, "ta": "apnic", "expires": 1760430142 } ```

[fjahr]

The problem is that there are conflicting announcements for the same prefix and ASN pair (with the same expiry)

I'm not sure I am getting this right, what is the conflict exactly, i.e. where is the difference? Is it a different ASN for the same prefix? So all fields are exactly the same except for the ASN?

Maybe you can past some of these conflicts as an example. Or maybe can you upload a dump of all the conflicts, that would be even better. I think it would be interesting to drill down into some of these conflicts and see what companies are behind them to see what is really happening. Happy to help with that if you upload the dump.

why so many conflicts from the RPKI DBs?

I am assuming that the entity announces their ROAs to multiple repositories to make sure everyone receives them ASAP. But would make sure to investigate.

should we handle these conflicts differently? drop them and let IRR sort it out?

should we keep the current impl?

I think we should look a bit deeper, I really wouldn't want to drop anything since we trust RPKI data more than IRR. Depending on what the investigation into the conflicts show, we can either keep the current impl or we go with a different tie-breaker, i.e. lowest ASN probably, I think we already use that somewhere else too.

[jurraca]

Is it a different ASN for the same prefix?

yes.

ROA dump in CSV format attached, but you can take any rpki_raw.json you have locally. Here's some code to parse it and get duplicate announcements (same prefix, different ASN). The amount matches what we usually see for "count of duplicates" during RPKI parsing, and it looks like the majority are currently getting de-duped by our "lowest ASN" heuristic. I think it's working as intended, but would like to understand how to dedupe these better.

output.csv.gz

[SuvidhJ]

Building on the findings here and the approach in PR #97, a few concrete suggestions:

AKI/SKI normalization for 9.5/9.6 compatibility

The 9.6 release notes confirm that key identifiers are now emitted as bare lowercase hex without colons (previously 47:AB:CD:EF, now 47abcdef). While parse.py doesn't currently use AKI/SKI in the mapping logic beyond completeness checks, it would be worth adding a normalization step at ingestion (strip colons, lowercase) so that any future logic that touches these fields - or any comparison of rpki_raw.json across versions - doesn't silently diverge. The transform is idempotent, so it's safe to apply regardless of which rpki-client version produced the output.

On the dedup tradeoff with -p

This seems like the core open question. The current tiebreaker chain in parse.py is valid_until → valid_since → lowest ASN. Switching to -p output loses valid_since, which @jurraca noted affects roughly 10-15k ROAs that make it past the other tiebreakers. One option worth considering: could we quantify how often the valid_since tiebreaker actually changes the final prefix→ASN mapping compared to just falling through to lowest ASN? If the difference in the final output is negligible, we could accept the tradeoff for the ~4.5x speedup (45s vs 3m26s). If it's not negligible, that's useful data for deciding whether to stay with the filemode approach on the 0.4.x branch.

Version reporting in output artifacts

Kartograf already logs the rpki-client version at startup, but recording it in the output artifact (e.g. as a header or metadata field) would make it possible to diagnose divergence between participants by inspecting the files alone, without needing to ask anyone what they had installed. Low effort, no impact on the mapping output, useful for coordinated runs.

[fjahr]

AKI/SKI normalization for 9.5/9.6 compatibility

Is this still an issue if we compare CCR hashes?

On the dedup tradeoff with -p

It seems debatable whe really is negligible but feel free to run that analysis and let us know what you find.

Version reporting in output artifacts

Why would that help? We don't have their output artifact. They would need to upload it first like they paste their logs now but adding it as meta data and requiring to upload the file which is only possible when it's zipped adds a lot of extra steps that seem unnecessary.