Investivate why the stress tests with 100 registrations at a time sometimes fail

[konradkonrad]

Related to shutter-network/shutter-api#120

[konradkonrad]

Findings:

A: incoming message buffer

With the change from #714 to
https://github.com/shutter-network/rolling-shutter/blob/dev/incoming_msg_buffer/rolling-shutter/p2p/p2p.go#L45:

- messagesBufSize = 128
+ messagesBufSize = 512 

I can stably get 150 events through. At 250 events it starts breaking again.

B: ConstructDecryptionKeyShares logic

With failing stress tests, the keyper logs are filling up with these log messages:

ERR [       service.go:39] handling key share event errored error="ignoring decryption request: shares exist already: preimages already existing

in the debugging PR I've included the identities to the log line, and there is a correspondence with the failing cases. It is not 1-to-1 though; some identities that do log this line still get resolved.

However: the logs for failed identities keep getting repeated, which points to the allSharesExist test not being sufficiently sophisticated.

My theory as up now: when the message buffer overflows, the decryption key share send logic gets out of sync, which leads to failed broadcasts. The keyper stores the own keyshare in db, and incorrectly assumes the identity was fully handled.

[konradkonrad]

With the changes in 86095b3 -- i.e. with an added re-broadcast mechanism, I could scale the stress test up to 1000 identities and still eventually pass. However, the test took a long time to actually pass and the load on the keyper got pretty high (> 100% utilization during the decryption key generation phase of the test run, anecdotally 70/30 between rolling-shutter & postgres):

   event_smoke_live_test.go:62: pass: received 1000 distinct decryption keys for 1000 registrations
--- PASS: TestEventSmokeCases (190.95s)
    --- PASS: TestEventSmokeCases/transfer_like_multi (190.95s)
PASS
ok  	github.com/shutter-network/shutter-api/tests/event_smoke	190.957s