Repository navigation
Add CDEPS Dev Container to repo - #447
danrosen25 wants to merge 1 commit into
Conversation
|
@mvertens @uturuncoglu @DeniseWorthen @ekluzek @samsrabin @Katetc @slevis-lmwg - I invited @danrosen25 to open this for discussion about whether we want it. My feeling is that I'm happy to include it if at least some others would find it useful. As some of the main recent contributors to CDEPS, do any of you have feelings on this? |
|
@mvertens @uturuncoglu @DeniseWorthen @ekluzek @samsrabin @Katetc @slevis-lmwg @billsacks |
|
I could see a docker file being the kind of thing that might get out-of-date if nobody is actively maintaining it. CDEPS is most functionally useful with CMEPS. Would it make sense to build the environment within CMEPS so it could be used by a variety of configurations? And, that would be an easier place to keep up-to-date with any machine, software version or library changes needed. |
|
@Katetc |
ekluzek
left a comment
ekluzek
left a comment
There was a problem hiding this comment.
This is a really cool capability to look at and think about.
Because of licensing issues we need to have people use podman instead of Docker. I suggest one change for that. The VS Code Dev Container extension also needs to be updated to use podman rather than Dockier. So that should be explained somewhere.
I also think that the contents here should replace most of what's in the workflow for extbuild.yml. We don't want to maintain that workflow alongside this one.
I've tried getting it to work on Derecho and ran into trouble (the docker daemon wasn't running). I also tried on my Mac laptop, it got farther but also failed, but I think because of problems I'm having in getting podman-compose installed on my Mac.
So these are things I think are needed:
- Use podman rather than Docker
- Add instructions about changing the Dev Container extension in VS code
- The instructions that are in the header of this PR need to go into some README document
- Have the exbuild github workflow use this rather than it's own setup
|
|
||
| ## Prerequisites | ||
|
|
||
| - Docker or another Docker-compatible container runtime |
There was a problem hiding this comment.
For licensing reasons we've been endorsing using podman. And podman is available on Derecho and other machines, while Docker is not.
So I think it should explicitly say podman here.
There was a problem hiding this comment.
| - Docker or another Docker-compatible container runtime | |
| - Podman or another Docker-compatible container runtime |
There was a problem hiding this comment.
The podman daemon also needs to be running, which should be mentioned somewhere.
|
In addition to the testing and portability benefits of regular containers, I really like devcontainers as a way to isolate coding agents from the rest of the filesystem. See, e.g., Anthropic's instructions for a Claude devcontainer, which also uses VS Code. There's nothing LLM-focused in this devcontainer, but it existing means someone can extend it to build e.g. a Claude-focused one, rather than starting from scratch. So I give this a big thumbs up in principle, although I haven't personally reviewed it. |
|
After a quick look at the changed files: I would want this PR to include GitHub Workflows to build (on PRs) and publish (on commits to main) the devcontainer whenever it's changed. I've done this for a container in CTSM and can provide guidance if needed. |
|
@ekluzek CTNR_USER="cdepsdev"
...
# Create non-root user with sudo privileges
ENV HOME=/home/${CTNR_USER}
RUN <<SETUP_USER
groupadd ${CTNR_USER}
useradd -m -d ${HOME} -s /bin/bash -g ${CTNR_USER} ${CTNR_USER}
adduser ${CTNR_USER} sudo
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
SETUP_USER
USER ${CTNR_USER}
WORKDIR ${HOME} |
|
@samsrabin |
Ah, that's a complication I've not actually run into—I've never used a devcontainer for CI/CD, only local development. In fact, now that I think about it, I don't think I actually built or ran the model in my devcontainer. I was using it purely for editing code, then testing on Derecho. That's obviously not an ideal workflow—my idea of "human in the loop" doesn't preclude me from letting LLMs build the model or even run simple tests. |
|
@samsrabin You might be interested in this work. For ESMF I have a script for building and pushing a multi-architecture container image package. I made an attempt to build images on their native architectures, for speed. I then cache the image and join them into a single manifest. The issue became that there's no native architecture that can build an AArch64 image so I'm building that on amd64 with emulation (slow). I left the infrastructure in place so that if the Mac OS runners (or another machine with AArch64) start supporting Docker builds then I can switch it over easily. I mitigate the slow emulation builds through installing as many binaries as possible via apt. |
|
I wasn't actually building and pushing the devcontainer; that was just a local experiment. What I'm doing for CTSM is building a container for our documentation. It's actually hosted in a separate repo, ESMCI/doc-builder. I have workflows that (1) build and test on PRs and (2) build and publish on merges to master. I'm also using emulation to build the arm64 version, but because this container is small and rarely-updated, it hasn't been an issue. However, I think there's a solution for you: The native macOS runners don't have Docker, but the arm64 ones do. That would result in containers that can run without emulation on Apple Silicon Macs (everything released in the last ~6 years). |
|
@samsrabin I only have to change one line of code, perfect. |
|
Maybe this is of interest @natalie-perlin, on container in UFS subcomponent and moving more testing upstream |
Description of changes
This PR introduces a Dev Container configuration to the repository. Moving forward, anyone working on this project can spin up a fully configured, isolated development environment with a single click.
Why use Dev Containers?
How to Use It
Specific notes