[Help] Can't start created container: executable file not found in $PATH

[stevenstetzler]

I am trying to start a container using the python API from this repo:

# up log level
import logging
logging.getLogger().setLevel(logging.DEBUG)
# action
import podman
with podman.Client() as client:
    image_id = client.images.pull("jupyterhub/singleuser:latest")
    image = client.images.get(image_id)
    container = image.create()
    container.start()

this throws an error, claims that the entrypoint command tini can't be found in the container:

DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb18f0cf8>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb18f0cf8>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb12fab00>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb12fab00>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb1254e80>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb1254e80>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb12ee5f8>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb12ee5f8>
DEBUG:root:Image f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19: create config: {'image_id': 'f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19', 'command': ['start-notebook.sh'], 'env': {'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'CONDA_DIR': '/opt/conda', 'SHELL': '/bin/bash', 'NB_USER': 'jovyan', 'NB_UID': '1000', 'NB_GID': '100', 'LC_ALL': 'en_US.UTF-8', 'LANG': 'en_US.UTF-8', 'LANGUAGE': 'en_US.UTF-8', 'HOME': '/home/jovyan', 'MINICONDA_VERSION': '4.8.3', 'MINICONDA_MD5': 'd63adf39f2c220950a063e0529d4ff74', 'CONDA_VERSION': '4.8.3'}, 'image': 'docker.io/jupyterhub/singleuser:latest', 'labels': {'maintainer': 'Jupyter Project <jupyter@googlegroups.com>'}, 'net_mode': 'bridge', 'network': 'bridge', 'args': ['docker.io/jupyterhub/singleuser:latest', 'start-notebook.sh']}
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb12d4e80>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb12d4e80>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb11a3ac8>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb11a3ac8>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb11b0128>
DEBUG:root:Starting Container "76f9f218441753ffcdc12dfb35a361f91cbc89a3d58379014ce8e0b4460fc2e2"
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7feeb11b0128>
Traceback (most recent call last):
  File "/root/checkpointspawner/python-podman/podman/libs/_containers_start.py", line 24, in start
    results = podman.StartContainer(self._id)
  File "/root/checkpointspawner/env/lib64/python3.6/site-packages/varlink/client.py", line 89, in _wrapped
    return self._call(method.name, *args, **kwargs)
  File "/root/checkpointspawner/env/lib64/python3.6/site-packages/varlink/client.py", line 146, in _call
    (message, more) = self._next_varlink_message()
  File "/root/checkpointspawner/env/lib64/python3.6/site-packages/varlink/client.py", line 116, in _next_varlink_message
    raise e
varlink.error.VarlinkError: {'parameters': {'reason': 'container_linux.go:349: starting container process caused "exec: \\"tini\\": executable file not found in $PATH": OCI runtime command not found error'}, 'error': 'io.podman.ErrorOccurred'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File "/root/checkpointspawner/python-podman/podman/libs/_containers_start.py", line 82, in start
    return self._refresh(podman)
  File "/root/checkpointspawner/python-podman/podman/client.py", line 139, in __exit__
    raise error_factory(e)
podman.libs.errors.ErrorOccurred: {'parameters': {'reason': 'container_linux.go:349: starting container process caused "exec: \\"tini\\": executable file not found in $PATH": OCI runtime command not found error'}, 'error': 'io.podman.ErrorOccurred'}

I can get the container to start just fine if I use the CLI:

podman run jupyterhub/singleuser:latest

is succesful.

I tried using a different image, and it works fine:

# up log level
import logging
logging.getLogger().setLevel(logging.DEBUG)
# action
import podman
with podman.Client() as client:
    image_id = client.images.pull("alpine:latest")
    image = client.images.get(image_id)
    container = image.create()
    container.start()

I then tried doing a mix: creating the container with the CLI:

podman container create jupyterhub/singleuser:latest

which returns a container id <container-id>, and then using the Python API to start it:

# up log level
import logging
logging.getLogger().setLevel(logging.DEBUG)
# action
import podman
with podman.Client() as client:
    container = client.containers.get(<container-id>)
    container.start()

which is successful

Additionally, I can go the other way: create the container with the Python API

# up log level
import logging
logging.getLogger().setLevel(logging.DEBUG)
# action
import podman
with podman.Client() as client:
    image_id = client.images.pull("jupyterhub/singleuser:latest")
    image = client.images.get(image_id)
    container = image.create()
    print(container.id)

which outputs <container-id>, and I can use the CLI to start it:

podman container start <container-id>

which gives an error:

Error: unable to start container "a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604": container_linux.go:349: starting container process caused "exec: \"tini\": executable file not found in $PATH": OCI runtime command not found error

This tells me that there's something going wrong with how this particular image is being created as a container using the Python API.

Here's the inspect information for the image:

Image Inspect

$ podman image inspect jupyterhub/singleuser:latest
[
    {
        "Id": "f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19",
        "Digest": "sha256:e65267c652a3dea035c25bdc93d179788a61c233a662f9a5366b4142e46a7b5d",
        "RepoTags": [
            "docker.io/jupyterhub/singleuser:latest"
        ],
        "RepoDigests": [
            "docker.io/jupyterhub/singleuser@sha256:e65267c652a3dea035c25bdc93d179788a61c233a662f9a5366b4142e46a7b5d"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2020-07-31T10:22:38.17249833Z",
        "Config": {
            "User": "1000",
            "ExposedPorts": {
                "8888/tcp": {}
            },
            "Env": [
                "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive",
                "CONDA_DIR=/opt/conda",
                "SHELL=/bin/bash",
                "NB_USER=jovyan",
                "NB_UID=1000",
                "NB_GID=100",
                "LC_ALL=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "LANGUAGE=en_US.UTF-8",
                "HOME=/home/jovyan",
                "MINICONDA_VERSION=4.8.3",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74",
                "CONDA_VERSION=4.8.3"
            ],
            "Entrypoint": [
                "tini",
                "-g",
                "--"
            ],
            "Cmd": [
                "start-notebook.sh"
            ],
            "WorkingDir": "/home/jovyan",
            "Labels": {
                "maintainer": "Jupyter Project <jupyter@googlegroups.com>"
            }
        },
        "Version": "19.03.8",
        "Author": "Project Jupyter <jupyter@googlegroups.com>",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 746017989,
        "VirtualSize": 746017989,
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/6e97361a4c0f6238f3f847bbbd84e476f3d12db6a75d524cdaa9c9d3b7641b1c/diff:/var/lib/containers/storage/overlay/1a759e27e0fc9122830680f71fe9c6dc71570dadd84e65bc08a2643eff87eadf/diff:/var/lib/containers/storage/overlay/333ffdfeeda6f14ab81ec3159f26c42f5c8514bc943b684c90ed98adef0d3dca/diff:/var/lib/containers/storage/overlay/f8c412df741909af0a20553b43c0a3895cee40317fc812f44572c9f04eaa66f2/diff:/var/lib/containers/storage/overlay/618a3af6a816f4d7fe48810ce798dd43e1277c39d85c8fa3144f20e6ccaa6193/diff:/var/lib/containers/storage/overlay/e74ffc54c99cd259134e36b203c6ab005df0bfd03efb68b5e1404071b6f1fc7d/diff:/var/lib/containers/storage/overlay/1ea64673d3aed0ab4c524a06f66b12275ea9e811d83763cedcf87a643d10b2a9/diff:/var/lib/containers/storage/overlay/47d3ba77a6da5e64f5dd3f3103729c41d4ea28dd3acf4eda6aa3deabc0a6ba3e/diff:/var/lib/containers/storage/overlay/0f13289ee6047a0af2c2da72fde4b07dde2760c6bd5ffbfdff50d18964a9f3e4/diff:/var/lib/containers/storage/overlay/103b666a8482eebcd34b088eea2b8955bfdca9280b4b802c15b1a06bf72bfc9e/diff:/var/lib/containers/storage/overlay/a365215521412762b731791a4e98169577428f319cadbbaca54c6b2d857cb37f/diff:/var/lib/containers/storage/overlay/626699adc61f1d9a01899423de78c7b875d9f4fa8c156d78cca37a1e8586458e/diff:/var/lib/containers/storage/overlay/7ee3433ebdb9fcfd9e1921e98ce740895e4f7fe0d23a4ce0e7a8ad2cc898f07f/diff:/var/lib/containers/storage/overlay/c5914a774f598d300553acb5dc31da85e04f813cfab7744db900e94c289d1af2/diff:/var/lib/containers/storage/overlay/60a71c5e6c29304f84beb1a85400368c5646b0b0e01c5a10ff50f7693fe60ffa/diff:/var/lib/containers/storage/overlay/3f9c2b6d8dddbad5d70cc4ccdcb4f6f68c1b65f6718dd3efac794a5035d64183/diff:/var/lib/containers/storage/overlay/df1094fb82405e981824e9705aca12f396d8175c0bd0fd6bb97c7a4f49d1ec88/diff:/var/lib/containers/storage/overlay/d22cfd6a8b16689838c570b91794ed18acc752a08a10bce891cc64acc1533b3f/diff",
                "UpperDir": "/var/lib/containers/storage/overlay/1b89d8d85ea7f40093b690a78a0fa19d028418f41c0bfb3660fdba9ccfe5e3cc/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/1b89d8d85ea7f40093b690a78a0fa19d028418f41c0bfb3660fdba9ccfe5e3cc/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:d22cfd6a8b16689838c570b91794ed18acc752a08a10bce891cc64acc1533b3f",
                "sha256:132bcd1e0eb5c706a017ff058b68d76c24f66f84120c51c7662de074a98cbe7a",
                "sha256:cf0f3facc4a307e4c36e346ddb777a73e576393575043e89d2ea536b693c3ff5",
                "sha256:544a70a875fc8e410b8a1389bf912e9536cf8167cbbfc1457bba355d5b7ce5c4",
                "sha256:8fb12b306bc4334420f40bbf7a23f8f8943c422b298efc1dd6d625b06ec95c79",
                "sha256:4b7eb71f03afc75314a8b0a1d42c9c5ecd11d13e94d2f23f6548e648bafc8db8",
                "sha256:9118f05dbee5f16387a4e4e50c1f2d933ca5a3df40cddd1dec13a6f305973424",
                "sha256:9118f05dbee5f16387a4e4e50c1f2d933ca5a3df40cddd1dec13a6f305973424",
                "sha256:d965477b97342363defb1ea9e6687eeec40a6246982e33a5e3ad0544e06c95a4",
                "sha256:8ac0d05486fce4c52cb01f4600c9ab4b1b26eaf5e3f501a3f4da5cfb667c65b7",
                "sha256:f1f19504f0306ef4ec824d55a1d51a3cdf0bd2ef31b40e06d51f6e373ab287ce",
                "sha256:a5475e9026e715df60775ce0d5276af2cd54d064e66ffe7a5d7463ea70e57c2a",
                "sha256:b34431a5f2eb510f0816c1e4f174de999490f083cceafd71b52abf2afacff173",
                "sha256:b203203c8e43c581de93f1224e58c97d29a28f08dd31f9021e2cd16a35d33eb5",
                "sha256:d9973ced7076322f5e8d7ff998437a8e4cf1a2cedac3274dfdd2f6c7f03b24a6",
                "sha256:20d805e832088af1abeef2b1d8b463b63e502a9f5d22a965a84673724510d061",
                "sha256:5fb2706450373119170e370c0f6353ceb183bcef23ef95ae1a6d622b1f28d9e3",
                "sha256:2a3f96a718d3afff91a10160e3b75a9d4d932d199eb564ec2b3f2b30ccecc622",
                "sha256:4d77ac3106b82281a696fc6873fa57f38ec78e7f41794f7830be2aef7197554b"
            ]
        },
        "Labels": {
            "maintainer": "Jupyter Project <jupyter@googlegroups.com>"
        },
        "Annotations": {},
        "ManifestType": "application/vnd.docker.distribution.manifest.v2+json",
        "User": "1000",
        "History": [
            {
                "created": "2020-07-06T21:56:28.828661061Z",
                "created_by": "/bin/sh -c #(nop) ADD file:cf87af1f0e27aa6ffad26c57edca4ca55dc97861590a2d63475085a08d3b0063 in / "
            },
            {
                "created": "2020-07-06T21:56:29.704325263Z",
                "created_by": "/bin/sh -c [ -z \"$(apt-get indextargets)\" ]"
            },
            {
                "created": "2020-07-06T21:56:30.474974715Z",
                "created_by": "/bin/sh -c set -xe \t\t&& echo '#!/bin/sh' > /usr/sbin/policy-rc.d \t&& echo 'exit 101' >> /usr/sbin/policy-rc.d \t&& chmod +x /usr/sbin/policy-rc.d \t\t&& dpkg-divert --local --rename --add /sbin/initctl \t&& cp -a /usr/sbin/policy-rc.d /sbin/initctl \t&& sed -i 's/^exit.*/exit 0/' /sbin/initctl \t\t&& echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup \t\t&& echo 'DPkg::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' > /etc/apt/apt.conf.d/docker-clean \t&& echo 'APT::Update::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' >> /etc/apt/apt.conf.d/docker-clean \t&& echo 'Dir::Cache::pkgcache \"\"; Dir::Cache::srcpkgcache \"\";' >> /etc/apt/apt.conf.d/docker-clean \t\t&& echo 'Acquire::Languages \"none\";' > /etc/apt/apt.conf.d/docker-no-languages \t\t&& echo 'Acquire::GzipIndexes \"true\"; Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/docker-gzip-indexes \t\t&& echo 'Apt::AutoRemove::SuggestsImportant \"false\";' > /etc/apt/apt.conf.d/docker-autoremove-suggests"
            },
            {
                "created": "2020-07-06T21:56:31.295257919Z",
                "created_by": "/bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container"
            },
            {
                "created": "2020-07-06T21:56:31.471255509Z",
                "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:11.610631152Z",
                "created_by": "/bin/sh -c #(nop)  LABEL maintainer=Jupyter Project <jupyter@googlegroups.com>",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:11.846446984Z",
                "created_by": "/bin/sh -c #(nop)  ARG NB_USER=jovyan",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:12.102668011Z",
                "created_by": "/bin/sh -c #(nop)  ARG NB_UID=1000",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:12.326598156Z",
                "created_by": "/bin/sh -c #(nop)  ARG NB_GID=100",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:12.563106469Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  SHELL [/bin/bash -o pipefail -c]",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:12.810461402Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  USER root",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:13.034413063Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ENV DEBIAN_FRONTEND=noninteractive",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:26.526685715Z",
                "created_by": "|3 NB_GID=100 NB_UID=1000 NB_USER=jovyan /bin/bash -o pipefail -c apt-get update  && apt-get install -yq --no-install-recommends     wget     bzip2     ca-certificates     sudo     locales     fonts-liberation     run-one  && apt-get clean && rm -rf /var/lib/apt/lists/*"
            },
            {
                "created": "2020-07-20T16:28:29.948217104Z",
                "created_by": "|3 NB_GID=100 NB_UID=1000 NB_USER=jovyan /bin/bash -o pipefail -c echo \"en_US.UTF-8 UTF-8\" > /etc/locale.gen &&     locale-gen"
            },
            {
                "created": "2020-07-20T16:28:30.197849107Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ENV CONDA_DIR=/opt/conda SHELL=/bin/bash NB_USER=jovyan NB_UID=1000 NB_GID=100 LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:30.446482159Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ENV PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/home/jovyan",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:30.728771164Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) COPY file:2407375a36b13fb421cfc19e79bb173b917242379d7aa88e759005477f3a0de2 in /usr/local/bin/fix-permissions "
            },
            {
                "created": "2020-07-20T16:28:32.022499595Z",
                "created_by": "/bin/bash -o pipefail -c chmod a+rx /usr/local/bin/fix-permissions"
            },
            {
                "created": "2020-07-20T16:28:33.308586154Z",
                "created_by": "/bin/bash -o pipefail -c sed -i 's/^#force_color_prompt=yes/force_color_prompt=yes/' /etc/skel/.bashrc"
            },
            {
                "created": "2020-07-20T16:28:34.716756804Z",
                "created_by": "/bin/bash -o pipefail -c echo \"auth requisite pam_deny.so\" >> /etc/pam.d/su &&     sed -i.bak -e 's/^%admin/#%admin/' /etc/sudoers &&     sed -i.bak -e 's/^%sudo/#%sudo/' /etc/sudoers &&     useradd -m -s /bin/bash -N -u $NB_UID $NB_USER &&     mkdir -p $CONDA_DIR &&     chown $NB_USER:$NB_GID $CONDA_DIR &&     chmod g+w /etc/passwd &&     fix-permissions $HOME &&     fix-permissions $CONDA_DIR"
            },
            {
                "created": "2020-07-20T16:28:34.970201486Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  USER 1000",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:35.201676676Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) WORKDIR /home/jovyan",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:35.42958766Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ARG PYTHON_VERSION=default",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:36.758811113Z",
                "created_by": "|1 PYTHON_VERSION=default /bin/bash -o pipefail -c mkdir /home/$NB_USER/work &&     fix-permissions /home/$NB_USER"
            },
            {
                "created": "2020-07-20T16:28:36.988290579Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ENV MINICONDA_VERSION=4.8.3 MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74 CONDA_VERSION=4.8.3",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:28:37.25061708Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) WORKDIR /tmp",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:31:21.925151438Z",
                "created_by": "|1 PYTHON_VERSION=default /bin/bash -o pipefail -c wget --quiet https://repo.continuum.io/miniconda/Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh &&     echo \"${MINICONDA_MD5} *Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh\" | md5sum -c - &&     /bin/bash Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR &&     rm Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh &&     echo \"conda ${CONDA_VERSION}\" >> $CONDA_DIR/conda-meta/pinned &&     conda config --system --prepend channels conda-forge &&     conda config --system --set auto_update_conda false &&     conda config --system --set show_channel_urls true &&     conda config --system --set channel_priority strict &&     if [ ! $PYTHON_VERSION = 'default' ]; then conda install --yes python=$PYTHON_VERSION; fi &&     conda list python | grep '^python ' | tr -s ' ' | cut -d '.' -f 1,2 | sed 's/$/.*/' >> $CONDA_DIR/conda-meta/pinned &&     conda install --quiet --yes conda &&     conda install --quiet --yes pip &&     conda update --all --quiet --yes &&     conda clean --all -f -y &&     rm -rf /home/$NB_USER/.cache/yarn &&     fix-permissions $CONDA_DIR &&     fix-permissions /home/$NB_USER"
            },
            {
                "created": "2020-07-20T16:31:38.095195868Z",
                "created_by": "|1 PYTHON_VERSION=default /bin/bash -o pipefail -c conda install --quiet --yes 'tini=0.18.0' &&     conda list tini | grep tini | tr -s ' ' | cut -d ' ' -f 1,2 >> $CONDA_DIR/conda-meta/pinned &&     conda clean --all -f -y &&     fix-permissions $CONDA_DIR &&     fix-permissions /home/$NB_USER"
            },
            {
                "created": "2020-07-20T16:34:03.340217468Z",
                "created_by": "|1 PYTHON_VERSION=default /bin/bash -o pipefail -c conda install --quiet --yes     'notebook=6.0.3'     'jupyterhub=1.1.0'     'jupyterlab=2.1.5' &&     conda clean --all -f -y &&     npm cache clean --force &&     jupyter notebook --generate-config &&     rm -rf $CONDA_DIR/share/jupyter/lab/staging &&     rm -rf /home/$NB_USER/.cache/yarn &&     fix-permissions $CONDA_DIR &&     fix-permissions /home/$NB_USER"
            },
            {
                "created": "2020-07-20T16:34:06.129748458Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  EXPOSE 8888",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:34:06.375095567Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ENTRYPOINT [\"tini\" \"-g\" \"--\"]",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:34:06.55878801Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  CMD [\"start-notebook.sh\"]",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:34:07.019257339Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) COPY multi:e4d2d8dbcb191c5f7ca1e67bc0a68ca3efaf7ca36b4c0d5eea05d30b01820cba in /usr/local/bin/ "
            },
            {
                "created": "2020-07-20T16:34:07.239205911Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) COPY file:cd827a3a9853bdea9decd0b6548b957cbe9821532361805d99dee359cfbbd1c0 in /etc/jupyter/ "
            },
            {
                "created": "2020-07-20T16:34:07.447243845Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  USER root",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:34:08.692850612Z",
                "created_by": "|1 PYTHON_VERSION=default /bin/bash -o pipefail -c fix-permissions /etc/jupyter/"
            },
            {
                "created": "2020-07-20T16:34:08.898441783Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  USER 1000",
                "empty_layer": true
            },
            {
                "created": "2020-07-20T16:34:09.14903861Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) WORKDIR /home/jovyan",
                "empty_layer": true
            },
            {
                "created": "2020-07-31T10:22:02.804876874Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  MAINTAINER Project Jupyter <jupyter@googlegroups.com>",
                "author": "Project Jupyter <jupyter@googlegroups.com>",
                "empty_layer": true
            },
            {
                "created": "2020-07-31T10:22:03.085613338Z",
                "created_by": "/bin/bash -o pipefail -c #(nop) ADD file:bcc8599841ae3d82e27cfe7a812a57eb854c16d52c26abbc9b259da6b294d27f in /tmp/install_jupyterhub ",
                "author": "Project Jupyter <jupyter@googlegroups.com>"
            },
            {
                "created": "2020-07-31T10:22:03.301082055Z",
                "created_by": "/bin/bash -o pipefail -c #(nop)  ARG JUPYTERHUB_VERSION=master",
                "author": "Project Jupyter <jupyter@googlegroups.com>",
                "empty_layer": true
            },
            {
                "created": "2020-07-31T10:22:38.17249833Z",
                "created_by": "|1 JUPYTERHUB_VERSION=0.9 /bin/bash -o pipefail -c python3 /tmp/install_jupyterhub &&     python3 -m pip install notebook",
                "author": "Project Jupyter <jupyter@googlegroups.com>"
            }
        ],
        "NamesHistory": null
    }
]

and the container inspect for the container made with the Python API:

Python API Container Inspect

$ podman container inspect a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604
[
    {
        "Id": "a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604",
        "Created": "2020-09-08T05:54:57.063997704Z",
        "Path": "tini",
        "Args": [
            "-g",
            "--",
            "start-notebook.sh"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "configured",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19",
        "ImageName": "docker.io/jupyterhub/singleuser:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/var/run/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata/resolv.conf",
        "HostnamePath": "/var/run/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata/hostname",
        "HostsPath": "/var/run/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata/hosts",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata",
        "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata/config.json",
        "OCIRuntime": "runc",
        "LogPath": "/var/lib/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata/ctr.log",
        "LogTag": "",
        "ConmonPidFile": "/var/run/containers/storage/overlay-containers/a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604/userdata/conmon.pid",
        "Name": "stupefied_napier",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c63,c267",
        "ProcessLabel": "system_u:system_r:container_t:s0:c63,c267",
        "AppArmorProfile": "",
        "EffectiveCaps": null,
        "BoundingCaps": [
            "CAP_AUDIT_WRITE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_MKNOD",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_RAW",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/1b89d8d85ea7f40093b690a78a0fa19d028418f41c0bfb3660fdba9ccfe5e3cc/diff:/var/lib/containers/storage/overlay/6e97361a4c0f6238f3f847bbbd84e476f3d12db6a75d524cdaa9c9d3b7641b1c/diff:/var/lib/containers/storage/overlay/1a759e27e0fc9122830680f71fe9c6dc71570dadd84e65bc08a2643eff87eadf/diff:/var/lib/containers/storage/overlay/333ffdfeeda6f14ab81ec3159f26c42f5c8514bc943b684c90ed98adef0d3dca/diff:/var/lib/containers/storage/overlay/f8c412df741909af0a20553b43c0a3895cee40317fc812f44572c9f04eaa66f2/diff:/var/lib/containers/storage/overlay/618a3af6a816f4d7fe48810ce798dd43e1277c39d85c8fa3144f20e6ccaa6193/diff:/var/lib/containers/storage/overlay/e74ffc54c99cd259134e36b203c6ab005df0bfd03efb68b5e1404071b6f1fc7d/diff:/var/lib/containers/storage/overlay/1ea64673d3aed0ab4c524a06f66b12275ea9e811d83763cedcf87a643d10b2a9/diff:/var/lib/containers/storage/overlay/47d3ba77a6da5e64f5dd3f3103729c41d4ea28dd3acf4eda6aa3deabc0a6ba3e/diff:/var/lib/containers/storage/overlay/0f13289ee6047a0af2c2da72fde4b07dde2760c6bd5ffbfdff50d18964a9f3e4/diff:/var/lib/containers/storage/overlay/103b666a8482eebcd34b088eea2b8955bfdca9280b4b802c15b1a06bf72bfc9e/diff:/var/lib/containers/storage/overlay/a365215521412762b731791a4e98169577428f319cadbbaca54c6b2d857cb37f/diff:/var/lib/containers/storage/overlay/626699adc61f1d9a01899423de78c7b875d9f4fa8c156d78cca37a1e8586458e/diff:/var/lib/containers/storage/overlay/7ee3433ebdb9fcfd9e1921e98ce740895e4f7fe0d23a4ce0e7a8ad2cc898f07f/diff:/var/lib/containers/storage/overlay/c5914a774f598d300553acb5dc31da85e04f813cfab7744db900e94c289d1af2/diff:/var/lib/containers/storage/overlay/60a71c5e6c29304f84beb1a85400368c5646b0b0e01c5a10ff50f7693fe60ffa/diff:/var/lib/containers/storage/overlay/3f9c2b6d8dddbad5d70cc4ccdcb4f6f68c1b65f6718dd3efac794a5035d64183/diff:/var/lib/containers/storage/overlay/df1094fb82405e981824e9705aca12f396d8175c0bd0fd6bb97c7a4f49d1ec88/diff:/var/lib/containers/storage/overlay/d22cfd6a8b16689838c570b91794ed18acc752a08a10bce891cc64acc1533b3f/diff",
                "UpperDir": "/var/lib/containers/storage/overlay/81261f5e99a1a46fa7717881d1c44fb72e388bbd2a4a4fde0bf8f72755d2629b/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/81261f5e99a1a46fa7717881d1c44fb72e388bbd2a4a4fde0bf8f72755d2629b/work"
            }
        },
        "Mounts": [],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": ""
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/var/run/containers/storage",
            "--log-level",
            "error",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/var/run/libpod",
            "--runtime",
            "runc",
            "--storage-driver",
            "overlay",
            "--events-backend",
            "file",
            "container",
            "cleanup",
            "a07d93be42c8c13e39b4185c1fb16c9677b3f99da1c15c15728348085998d604"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "a07d93be42c8",
            "Domainname": "",
            "User": "1000",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin",
                "TERM=xterm",
                "HOSTNAME=a07d93be42c8",
                "LANGUAGE=en_US.UTF-8",
                "CONDA_VERSION=4.8.3",
                "NB_USER=jovyan",
                "MINICONDA_VERSION=4.8.3",
                "SHELL=/bin/bash",
                "LC_ALL=en_US.UTF-8",
                "container=podman",
                "NB_GID=100",
                "LANG=en_US.UTF-8",
                "DEBIAN_FRONTEND=noninteractive",
                "HOME=/home/jovyan",
                "NB_UID=1000",
                "CONDA_DIR=/opt/conda",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74"
            ],
            "Cmd": [
                "start-notebook.sh"
            ],
            "Image": "docker.io/jupyterhub/singleuser:latest",
            "Volumes": null,
            "WorkingDir": "/home/jovyan",
            "Entrypoint": "tini -g --",
            "OnBuild": null,
            "Labels": {
                "maintainer": "Jupyter Project <jupyter@googlegroups.com>"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2020-09-08T05:54:57.063997704Z",
                "io.kubernetes.cri-o.TTY": "false",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15
        },
        "HostConfig": {
            "Binds": [],
            "CgroupMode": "host",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 4194304,
                    "Hard": 4194304
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        }
    }
]

and by the CLI:

CLI Container Inspect

$ podman container inspect daaa730a2b97ca9f7bcc586812726b30bb88c81fcfa7d7ebaa9c62d13c140475
[
    {
        "Id": "daaa730a2b97ca9f7bcc586812726b30bb88c81fcfa7d7ebaa9c62d13c140475",
        "Created": "2020-09-08T06:01:22.73682913Z",
        "Path": "tini",
        "Args": [
            "-g",
            "--",
            "start-notebook.sh"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "configured",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19",
        "ImageName": "docker.io/jupyterhub/singleuser:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "",
        "HostnamePath": "",
        "HostsPath": "",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/daaa730a2b97ca9f7bcc586812726b30bb88c81fcfa7d7ebaa9c62d13c140475/userdata",
        "OCIRuntime": "runc",
        "LogPath": "/var/lib/containers/storage/overlay-containers/daaa730a2b97ca9f7bcc586812726b30bb88c81fcfa7d7ebaa9c62d13c140475/userdata/ctr.log",
        "LogTag": "",
        "ConmonPidFile": "/var/run/containers/storage/overlay-containers/daaa730a2b97ca9f7bcc586812726b30bb88c81fcfa7d7ebaa9c62d13c140475/userdata/conmon.pid",
        "Name": "zealous_shaw",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c468,c770",
        "ProcessLabel": "system_u:system_r:container_t:s0:c468,c770",
        "AppArmorProfile": "",
        "EffectiveCaps": null,
        "BoundingCaps": [
            "CAP_AUDIT_WRITE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_MKNOD",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_RAW",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/1b89d8d85ea7f40093b690a78a0fa19d028418f41c0bfb3660fdba9ccfe5e3cc/diff:/var/lib/containers/storage/overlay/6e97361a4c0f6238f3f847bbbd84e476f3d12db6a75d524cdaa9c9d3b7641b1c/diff:/var/lib/containers/storage/overlay/1a759e27e0fc9122830680f71fe9c6dc71570dadd84e65bc08a2643eff87eadf/diff:/var/lib/containers/storage/overlay/333ffdfeeda6f14ab81ec3159f26c42f5c8514bc943b684c90ed98adef0d3dca/diff:/var/lib/containers/storage/overlay/f8c412df741909af0a20553b43c0a3895cee40317fc812f44572c9f04eaa66f2/diff:/var/lib/containers/storage/overlay/618a3af6a816f4d7fe48810ce798dd43e1277c39d85c8fa3144f20e6ccaa6193/diff:/var/lib/containers/storage/overlay/e74ffc54c99cd259134e36b203c6ab005df0bfd03efb68b5e1404071b6f1fc7d/diff:/var/lib/containers/storage/overlay/1ea64673d3aed0ab4c524a06f66b12275ea9e811d83763cedcf87a643d10b2a9/diff:/var/lib/containers/storage/overlay/47d3ba77a6da5e64f5dd3f3103729c41d4ea28dd3acf4eda6aa3deabc0a6ba3e/diff:/var/lib/containers/storage/overlay/0f13289ee6047a0af2c2da72fde4b07dde2760c6bd5ffbfdff50d18964a9f3e4/diff:/var/lib/containers/storage/overlay/103b666a8482eebcd34b088eea2b8955bfdca9280b4b802c15b1a06bf72bfc9e/diff:/var/lib/containers/storage/overlay/a365215521412762b731791a4e98169577428f319cadbbaca54c6b2d857cb37f/diff:/var/lib/containers/storage/overlay/626699adc61f1d9a01899423de78c7b875d9f4fa8c156d78cca37a1e8586458e/diff:/var/lib/containers/storage/overlay/7ee3433ebdb9fcfd9e1921e98ce740895e4f7fe0d23a4ce0e7a8ad2cc898f07f/diff:/var/lib/containers/storage/overlay/c5914a774f598d300553acb5dc31da85e04f813cfab7744db900e94c289d1af2/diff:/var/lib/containers/storage/overlay/60a71c5e6c29304f84beb1a85400368c5646b0b0e01c5a10ff50f7693fe60ffa/diff:/var/lib/containers/storage/overlay/3f9c2b6d8dddbad5d70cc4ccdcb4f6f68c1b65f6718dd3efac794a5035d64183/diff:/var/lib/containers/storage/overlay/df1094fb82405e981824e9705aca12f396d8175c0bd0fd6bb97c7a4f49d1ec88/diff:/var/lib/containers/storage/overlay/d22cfd6a8b16689838c570b91794ed18acc752a08a10bce891cc64acc1533b3f/diff",
                "UpperDir": "/var/lib/containers/storage/overlay/dfc8276bb474890bb1b1befbb8c9e5503a1544b193325fb64a3927d6f33fce90/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/dfc8276bb474890bb1b1befbb8c9e5503a1544b193325fb64a3927d6f33fce90/work"
            }
        },
        "Mounts": [],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": ""
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/var/run/containers/storage",
            "--log-level",
            "error",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/var/run/libpod",
            "--runtime",
            "runc",
            "--storage-driver",
            "overlay",
            "--events-backend",
            "file",
            "container",
            "cleanup",
            "daaa730a2b97ca9f7bcc586812726b30bb88c81fcfa7d7ebaa9c62d13c140475"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "daaa730a2b97",
            "Domainname": "",
            "User": "1000",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "CONDA_VERSION=4.8.3",
                "LC_ALL=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "CONDA_DIR=/opt/conda",
                "NB_GID=100",
                "NB_USER=jovyan",
                "DEBIAN_FRONTEND=noninteractive",
                "MINICONDA_VERSION=4.8.3",
                "container=podman",
                "LANGUAGE=en_US.UTF-8",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74",
                "NB_UID=1000",
                "SHELL=/bin/bash",
                "HOME=/home/jovyan",
                "HOSTNAME="
            ],
            "Cmd": [
                "start-notebook.sh"
            ],
            "Image": "docker.io/jupyterhub/singleuser:latest",
            "Volumes": null,
            "WorkingDir": "/home/jovyan",
            "Entrypoint": "tini -g --",
            "OnBuild": null,
            "Labels": {
                "maintainer": "Jupyter Project <jupyter@googlegroups.com>"
            },
            "Annotations": {
                "io.kubernetes.cri-o.TTY": "false",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE"
            },
            "StopSignal": 15,
            "CreateCommand": [
                "podman",
                "container",
                "create",
                "jupyterhub/singleuser:latest"
            ]
        },
        "HostConfig": {
            "Binds": [],
            "CgroupMode": "host",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": 0,
            "OomKillDisable": false,
            "PidsLimit": 2048,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 4194304,
                    "Hard": 4194304
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        }
    }
]

And here is system/version info:

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

$ podman --version
podman version 2.0.6

$ python -m pip freeze | grep podman
podman==1.6.1.dev50
$ python -m pip freeze | grep varlink
varlink==30.3.0

I was previously running podman version 1.6.4, from the default Centos repositories, paired with the default python-podman version from the PyPi repository, but I was experiencing the same behavior before and after both upgrades.

I've read elsewhere that this might have to do with the container's file system not mounting correctly? Perhaps there is also something simple I am missing in how to use the Python API. Any help is appreciated!

[stevenstetzler]

I think I found a hint toward the issue by doing a diff on the container inspect outputs. It looks like the PATH environment variable gets messed up in container creation:

For the Python API created container (incorrect PATH):

            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin",
                "TERM=xterm",
                "HOSTNAME=a07d93be42c8",
                "LANGUAGE=en_US.UTF-8",
                "CONDA_VERSION=4.8.3",
                "NB_USER=jovyan",
                "MINICONDA_VERSION=4.8.3",
                "SHELL=/bin/bash",
                "LC_ALL=en_US.UTF-8",
                "container=podman",
                "NB_GID=100",
                "LANG=en_US.UTF-8",
                "DEBIAN_FRONTEND=noninteractive",
                "HOME=/home/jovyan",
                "NB_UID=1000",
                "CONDA_DIR=/opt/conda",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74"
            ],

and for the CLI created container (correct PATH):

            "Env": [
                "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "CONDA_VERSION=4.8.3",
                "LC_ALL=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "CONDA_DIR=/opt/conda",
                "NB_GID=100",
                "NB_USER=jovyan",
                "DEBIAN_FRONTEND=noninteractive",
                "MINICONDA_VERSION=4.8.3",
                "container=podman",
                "LANGUAGE=en_US.UTF-8",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74",
                "NB_UID=1000",
                "SHELL=/bin/bash",
                "HOME=/home/jovyan",
                "HOSTNAME="
            ],

This is a bit confusing as it seems to me that the PATH is being taken correctly from the image before creation: https://github.com/containers/python-podman/blob/master/podman/libs/images.py#L44-L48 in the Python API, and I can see from the log output that the config sent to the varlink and go APIs appears correct: (from above)

DEBUG:root:Image f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19: create config: {'image_id': 'f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19', 'command': ['start-notebook.sh'], 'env': {'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'CONDA_DIR': '/opt/conda', 'SHELL': '/bin/bash', 'NB_USER': 'jovyan', 'NB_UID': '1000', 'NB_GID': '100', 'LC_ALL': 'en_US.UTF-8', 'LANG': 'en_US.UTF-8', 'LANGUAGE': 'en_US.UTF-8', 'HOME': '/home/jovyan', 'MINICONDA_VERSION': '4.8.3', 'MINICONDA_MD5': 'd63adf39f2c220950a063e0529d4ff74', 'CONDA_VERSION': '4.8.3'}, 'image': 'docker.io/jupyterhub/singleuser:latest', 'labels': {'maintainer': 'Jupyter Project <jupyter@googlegroups.com>'}, 'net_mode': 'bridge', 'network': 'bridge', 'args': ['docker.io/jupyterhub/singleuser:latest', 'start-notebook.sh']}

I can actually fix the issue by setting the env keyword during container creation to something new, which overrides the default set by the Python API:

# up log level
import logging
logging.getLogger().setLevel(logging.DEBUG)
# action
import podman
with podman.Client() as client:
    image_id = client.images.pull("jupyterhub/singleuser:latest")
    image = client.images.get(image_id)
    container = image.create(
        env={
            "FOO": "BAR"
        }
    )
    print(container.inspect().config.get("env"))
    container.start()

produces

DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2c44828>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2c44828>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da264d3c8>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da264d3c8>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da25a7748>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da25a7748>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da26244e0>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da26244e0>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da263a898>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da263a898>
DEBUG:root:Image f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19: create config: {'image_id': 'f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19', 'env': {'FOO': 'BAR'}, 'command': ['start-notebook.sh'], 'image': 'docker.io/jupyterhub/singleuser:latest', 'labels': {'maintainer': 'Jupyter Project <jupyter@googlegroups.com>'}, 'net_mode': 'bridge', 'network': 'bridge', 'args': ['docker.io/jupyterhub/singleuser:latest', 'start-notebook.sh']}
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da24f4d68>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da24f4d68>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455cc0>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455cc0>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455518>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455518>
['PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'TERM=xterm', 'HOSTNAME=', 'SHELL=/bin/bash', 'CONDA_VERSION=4.8.3', 'container=podman', 'DEBIAN_FRONTEND=noninteractive', 'LANGUAGE=en_US.UTF-8', 'LC_ALL=en_US.UTF-8', 'MINICONDA_VERSION=4.8.3', 'MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74', 'LANG=en_US.UTF-8', 'HOME=/home/jovyan', 'CONDA_DIR=/opt/conda', 'NB_USER=jovyan', 'NB_UID=1000', 'NB_GID=100']
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da25c97f0>
DEBUG:root:Starting Container "79912b88ed238ade23bd67d806fe7591547cb0ab26acdb3ccbaaba3da6adcdf2"
DEBUG:root:Started Container "79912b88ed238ade23bd67d806fe7591547cb0ab26acdb3ccbaaba3da6adcdf2"

and creates the container successfully. I print out the environment from the Container inspect in the output above and see that the PATH is set correctly using the image defaults. However, the FOO=BAR environment variable is not set in the container. I think the PATH gets set from the image in the go API: ParseCreateOpts (https://github.com/containers/podman/blob/v2.0.6/pkg/varlinkapi/create.go#L504-L556) called by CreateContainer (https://github.com/containers/podman/blob/v2.0.6/pkg/varlinkapi/create.go#L184).

Is this a problem of the communication between the Python API and the Varlink API?