Build system to customize scout and DPU after deployment

[ajf]

Problem

Presently, it's impossible for an end-user to customize the scout image, or DPU BFB for a particular deployment (e.g. to add local authentication, or additional monitoring tools).

Requirements

• Must not be a part of carbide-core CI/CD process
• Must be thought of like an "add-on" to an existing deployment
• Must be configurable in the carbide config file, or in the database as code
• Must be able to add files to the existing OS (non-sandboxed) and overwrite existing files in the running image

Potential implementation

As an example, we can extend the Carbide database with a table that has a list of .tar.gz files that are downloaded by cloud-init and extracted over the top of /, and the build process of those .tar.gz files lives somewhere accessible to the booting image and published by the site owner.

Perhaps an OCI container could be a format? or something like a systemd-sysext/confext.

Definition of done

• We can eliminate carbide-extras, and the BFB rebuild from the build process and still add-on the internal NVIDIA SSH authentication

[ajf]

An alternative is that we can just use Debian apt packages and include sources from the site controller and have a way to upload deb packages and then install them during dpu and scout cloud-init?

[ajf]

I've seen @ianderson-nvidia and @rwthompsonii both working on this, this is assigned to @rwthompsonii - is that right? Can we get some subtasks created?