The current bottleneck when scaling a node is a single's server ability to migrate slabs. To fix this we want to introduce a repair mode for indexd that launches indexd with just the capabilities it needs to fetch slabs to migrate, migrate them and then update the metadata of those slabs. No http API, contract maintenance etc.
To perform a migration a node needs to at least know:
- the private key of a funded account (e.g. the service account)
- the list of hosts that make a sector a "healthy sector" (Good for upload hosts)
- the list of hosts that can be used to migrate a sector to (Good for append hosts)
- slab metadata
The easiest solution to achieve that is by granting an indexer in repair mode access to the database and then run the slab migration loop on that indexer.
However, we need to make sure the migration loop is safe to use from multiple threads. Meaning that 2 indexers performing migrations don't end up with the same slabs. In theory UnhealthySlabs already updates next_repair_attempt optimistically so that shouldn't be an issue but we might need a FOR UPDATE or some other update to the slab fetching query to make this atomic.
The current bottleneck when scaling a node is a single's server ability to migrate slabs. To fix this we want to introduce a repair mode for indexd that launches
indexdwith just the capabilities it needs to fetch slabs to migrate, migrate them and then update the metadata of those slabs. No http API, contract maintenance etc.To perform a migration a node needs to at least know:
The easiest solution to achieve that is by granting an indexer in repair mode access to the database and then run the slab migration loop on that indexer.
However, we need to make sure the migration loop is safe to use from multiple threads. Meaning that 2 indexers performing migrations don't end up with the same slabs. In theory
UnhealthySlabsalready updatesnext_repair_attemptoptimistically so that shouldn't be an issue but we might need aFOR UPDATEor some other update to the slab fetching query to make this atomic.