installing packages with spack in eic container fails

[plexoos]

Whenever I try to install something with spack install --add inside the container, Spack attempts to build qt@5.15.12 and then fails during the configure phase.

$ docker run --rm -it eicweb/eic_cuda:25.08-stable

> spack env ls
==> No environments

> spack env status
==> In environment /opt/spack-environment/cuda

> spack install --add units
...
==> Installing qt-5.15.12-3asbnrc6c4z2h5h5asqvzm3ectbmaq6u [472/529]
==> No binary for qt-5.15.12-3asbnrc6c4z2h5h5asqvzm3ectbmaq6u found: installing from source
==> Applied patch ... (multiple qt5 patches)
==> Ran patch() for qt
==> qt: Executing phase: 'configure'
...
==> Error: ProcessError: Command exited with status 1
...
==> Warning: Skipping build of xxdiff-2023-01-10-... since qt-5.15.12-... failed

[plexoos]

I still cannot install anything in the default environment in eic containers with spack. @veprbl can you please take a look?

$ docker run --rm -it eicweb/eic_dev_cuda:26.01.0-stable
(Fri Jan-9 12:30:14pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack env ls
==> No environments
(Fri Jan-9 12:30:26pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack env status
==> In environment /opt/spack-environment/cuda/epic
(Fri Jan-9 12:30:31pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack repo ls
[+] eic        v1.0    /opt/spack-packages/repos/eic-spack
[+] k4         v1.0    /opt/spack-packages/repos/key4hep-spack
[+] builtin    v2.2    /opt/spack-packages/repos/spack_repo/builtin
(Fri Jan-9 12:30:40pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack info plog
CMakePackage:   plog

Description:
    Portable, simple and extensible C++ logging library.

Homepage: https://github.com/SergiusTheBest/plog

Preferred version:  
    1.1.10    https://github.com/SergiusTheBest/plog/archive/refs/tags/1.1.10.tar.gz

Safe versions:  
    1.1.10    https://github.com/SergiusTheBest/plog/archive/refs/tags/1.1.10.tar.gz
    1.1.9     https://github.com/SergiusTheBest/plog/archive/refs/tags/1.1.9.tar.gz

Deprecated versions:  
    None

Variants:
    build_system [cmake]        cmake
        Build systems supported by the package

    when build_system=cmake
      build_type [Release]      Debug, MinSizeRel, RelWithDebInfo, Release
          CMake build type
      generator [make]          none
          the build system generator to use

    when build_system=cmake ^cmake@3.9:
      ipo [false]               false, true
          CMake interprocedural optimization

Build Dependencies:
    c  cmake  cxx  gmake  ninja

Link Dependencies:
    None

Run Dependencies:
    None

Licenses: 
    MIT 

(Fri Jan-9 12:30:54pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack install plog
==> Error: Failed to generate environment view, because the target /opt/._local/4fhyub63g4lrrkf7fj3caetjvvtzwapk already exists or is not empty. To update the view, remove this path, and run `spack env view regenerate`
(ERROR)-(Exit Code 1)-(General error)

[veprbl]

On singularity I get:

$ spack install plog
==> Error: Can't take write lock on read-only file: /opt/spack-environment/xl/epic/.spack-env/transaction_lock

Have you tried following To update the view, remove this path, and run `spack env view regenerate` ?

[plexoos]

Like so?

$ docker run --rm -it eicweb/eic_dev_cuda:26.01.0-stable
(Fri Jan-9 1:08:40pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack install plog
==> Error: Failed to generate environment view, because the target /opt/._local/4fhyub63g4lrrkf7fj3caetjvvtzwapk already exists or is not empty. To update the view, remove this path, and run `spack env view regenerate`
(ERROR)-(Exit Code 1)-(General error)
(Fri Jan-9 1:08:56pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> rm -fr /opt/._local/4fhyub63g4lrrkf7fj3caetjvvtzwapk
removed '/opt/._local/4fhyub63g4lrrkf7fj3caetjvvtzwapk'
(Fri Jan-9 1:10:04pm)-(CPU 1.0%:0:Net 1)-(root:/)-(20K:26)
> spack env view regenerate
==> Updating view at /opt/local
==> Error: The environment view in /opt/local cannot not be created because it is a non-empty directory.
(ERROR)-(Exit Code 1)-(General error)

[veprbl]

Yeah.

I've also tried using an additional environment:

$ . /opt/spack/share/spack/setup-env.sh
$ spack env activate --create plog
$ spack add plog
==> Adding plog to environment plog
$ spack install
==> Warning: The following issues were ignored while updating the indices of binary caches
  Multiple errors during fetching:
        Error 1: FetchIndexError: Could not fetch manifest from https://ghcr.io/v2/eic/spack-v2025.11.0/manifests/index.spack, due to: GET https://ghcr.io/v2/eic/spack-v2025.11.0/manifests/index.spack returned 404: Not Found
        Error 2: FetchIndexError: Could not fetch manifest from https://ghcr.io/v2/eic/spack-v2025.11.0/manifests/index.spack, due to: GET https://ghcr.io/v2/eic/spack-v2025.11.0/manifests/index.spack returned 404: Not Found
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v1/blobs/sha256:9ac106d746fc9830acbb2e6b135a5c1748482df8fe115bee2deb1ed02e5189c4
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v1/blobs/sha256:6599ac06ade0cb3e80695f36492ea94a306f8bde0537482521510076c5981aa0
==> Installing "clingo-bootstrap@=spack~docs+ipo+optimized+python+static_libstdcpp build_system=cmake build_type=Release generator=make patches:=bebb819,ec99431 arch=linux-centos7-x86_64" from a buildcache
==> Concretized 1 spec
 -   f36xga5  plog@1.1.10~ipo build_system=cmake build_type=Release generator=make arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[^]  afjg5rc      ^cmake@3.31.9~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[^]  wgmwcod          ^curl@8.15.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[^]  iuoszrs              ^nghttp2@1.67.1 build_system=autotools arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[-]  cowagv3                  ^diffutils@3.12 build_system=autotools arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[^]  yjaypuu                      ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[^]  yofcsvj              ^openssl@3.6.0~docs+shared build_system=generic certs=system arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[^]  xrja23o              ^perl@5.42.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[^]  egmk34o                  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[^]  qlbsxo3                  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[^]  bbkmlze                  ^gdbm@1.25 build_system=autotools arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[^]  f3da5kx                      ^readline@8.3 build_system=autotools patches:=21f0a03 arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[^]  2uerkow              ^pkgconf@2.5.1 build_system=autotools arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0
[e]  iawll3b          ^gcc@14.2.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=Release languages:='c,c++,fortran' arch=linux-debian13-x86_64_v2 
[^]  nalbzqf          ^gcc-runtime@14.2.0 build_system=generic arch=linux-debian13-x86_64_v2 
[^]  yfoipfc          ^ncurses@6.5-20250705~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[^]  h3ys6q6          ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-debian13-x86_64_v2 %c,cxx=gcc@14.2.0
[-]  hyazzye      ^compiler-wrapper@1.0 build_system=generic arch=linux-debian13-x86_64_v2 
[e]  53uytb6      ^gcc@14.2.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-debian13-x86_64_v2 
 -   54ch43q      ^gcc-runtime@14.2.0 build_system=generic arch=linux-debian13-x86_64_v2 
[e]  uvk4zqr      ^glibc@2.41 build_system=autotools arch=linux-debian13-x86_64_v2 
[^]  flapcp7      ^gmake@4.4.1~guile build_system=generic arch=linux-debian13-x86_64_v2 %c=gcc@14.2.0

[+] /usr (external glibc-2.41-uvk4zqrac6mfr25exjctdyvvwtqoj4zw)
==> Fetching https://ghcr.io/v2/eic/spack-v2025.11.0/blobs/sha256:b569617f4fa8b9d2499fc953c5ea1bf3fe69b951d57bf789d3033ecf3ce77367
==> Fetching https://ghcr.io/v2/eic/spack-v2025.11.0/blobs/sha256:159bb1b34ab874dabdecb5ced4a3b276a8d7f93ffc1773040f07a9bd5c45ee68
==> Extracting compiler-wrapper-1.0-hyazzyejoc3tc2654ciifuz7lbpjsmng from binary cache
==> Warning: Skipping build of cmake-3.31.9-afjg5rcwigdxza7ncya6c52k4prpmnwk since compiler-wrapper-1.0-hyazzyejoc3tc2654ciifuz7lbpjsmng failed
==> Warning: Skipping build of plog-1.1.10-f36xga5aa7hx7geedgsnyf2ybl4nt23o since cmake-3.31.9-afjg5rcwigdxza7ncya6c52k4prpmnwk failed
==> Warning: Skipping build of nghttp2-1.67.1-iuoszrsoe36csiv2xc67f3dkewmwcrr7 since compiler-wrapper-1.0-hyazzyejoc3tc2654ciifuz7lbpjsmng failed
==> Warning: Skipping build of curl-8.15.0-wgmwcodggddnvvsvyh3whaeifguchpuj since nghttp2-1.67.1-iuoszrsoe36csiv2xc67f3dkewmwcrr7 failed
==> Warning: Skipping build of gmake-4.4.1-flapcp7m5kv7zvl4r27xs7gmjpblfvgk since compiler-wrapper-1.0-hyazzyejoc3tc2654ciifuz7lbpjsmng failed
==> Warning: Skipping build of openssl-3.6.0-yofcsvjtllmpjtkou6hspdhq76h4zvub since gmake-4.4.1-flapcp7m5kv7zvl4r27xs7gmjpblfvgk failed
==> Warning: Skipping build of ncurses-6.5-20250705-yfoipfcqdtvgvrt5sh76hhiygq6oodc4 since gmake-4.4.1-flapcp7m5kv7zvl4r27xs7gmjpblfvgk failed
==> Warning: Skipping build of zlib-ng-2.2.4-h3ys6q63bqbwgtrgd4als7ibsld2eotp since gmake-4.4.1-flapcp7m5kv7zvl4r27xs7gmjpblfvgk failed
==> Error: Failed to install compiler-wrapper due to OSError: [Errno 30] Read-only file system: '/opt/software/linux-x86_64_v2/compiler-wrapper-1.0-hyazzyejoc3tc2654ciifuz7lbpjsmng'
[+] /usr (external gcc-14.2.0-53uytb6zav6drl7ggkmeaxun5o6cxoqt)
[+] /opt/software/linux-x86_64_v2/gcc-runtime-14.2.0-nalbzqffyk2zosw5fez6txfmvf5km5yu
==> No binary for gcc-runtime-14.2.0-54ch43qqcgggrreux7snlh6h2iz3ouz6 found: installing from source
==> Installing gcc-runtime-14.2.0-54ch43qqcgggrreux7snlh6h2iz3ouz6 [5/13]
==> No patches needed for gcc-runtime
==> gcc-runtime: Executing phase: 'install'
==> gcc-runtime: Successfully installed gcc-runtime-14.2.0-54ch43qqcgggrreux7snlh6h2iz3ouz6
  Stage: 0.00s.  Install: 0.31s.  Post-install: 0.03s.  Total: 0.39s
[+] /home/veprbl/spack/linux-x86_64_v2/gcc-runtime-14.2.0-54ch43qqcgggrreux7snlh6h2iz3ouz6
==> Error: plog-1.1.10-f36xga5aa7hx7geedgsnyf2ybl4nt23o: Package was not installed
==> Updating view at /home/veprbl/.spack/env/plog/.spack-env/view
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

still doesn't work in singularity (I think we had decided it was a spack bug, but we also had a workaround?)

[plexoos]

Here is what works for me:

docker run --rm eicweb/eic_dev_cuda:26.01.0-stable bash -c '. /opt/spack/share/spack/setup-env.sh && spack env deactivate && spack install --no-cache plog'

But it feels weird that I need to do spack env deactivate

[wdconinc]

But it feels weird that I need to do spack env deactivate

The main reason (in addition to spack issues every so often) that I've always considered spack to be a behind-the-scenes tool is that it takes some user training to make sure people know what they are doing, can easily result in users installing things and not realizing it is filling their quota, and that spack is not always behaving like a regular package manager when people tend to assume it is.

In this case, the concept of spack environments is pretty fundamental to how we use it in the containers. It is a collection of packages. Adding new packages is not an atomic operation that only affects that single package but is an operation that potentially affects the entire environment. Therefore, if you do want to install an individual package, then you will have to deactivate that environment so you are explicitly in single-package mode. It will then also not be constrained by what is already installed in the container and you may end up installing dependencies multiple times.

[plexoos]

It will then also not be constrained by what is already installed in the container and you may end up installing dependencies multiple times.

That is exactly what I am trying to avoid. As a developer of software that I want included in the EIC stack, I do want to be constrained by the environment. The use case I’m advocating for is the ability to install necessary dependencies within that same environment.