[question] Build all dependencies locally

[jackrosenberg]

What is your question?

Hello, i'm packaging https://github.com/mrizaln/madbfs for nixpkgs, which uses conan to build. A requirement for nixpkgs is that packages that can be built locally, are built locally. To this end I would like to be able to build all dependencies from source.

conan install . --build=always -pr:b=.github/ci/conan-profile.ini -pr:h=.github/ci/conan-profile.ini -nr

madbfs> ======== Computing dependency graph ========
madbfs> boost/1.87.0: Not found in local cache, looking in remotes...
madbfs> boost/1.87.0: Checking remote: conancenter
madbfs> Connecting to remote 'conancenter' anonymously

is there a way to build boost and other dependencies locally, and have conan use them?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @jackrosenberg

Thanks for the question.

Sure, using --build="*" builds all dependencies from source.
There are also other ways, most likely more efficients, like defining ConanCenter as a remote with --recipes-only, that inhibits the download of binaries, but if there are binaries already built in the local cache in a previous run, it will use them, not rebuilding everything all the time from source if not necessary.

[jackrosenberg]

Hey, thanks for the quick response! The idea of nixpkgs is that the builder is not allowed onto the internet (mostly). It looks like the builder is still trying to use remotes. Can I disable that too?

conan install . --build="*" -pr:b=.github/ci/conan-profile.ini -pr:h=.github/ci/conan-profile.ini -nr -vv

madbfs> ======== Computing dependency graph ========
madbfs> boost/1.87.0: Not found in local cache, looking in remotes...
madbfs> Graph root
madbfs>     conanfile.py: /build/source/conanfile.py
madbfs> ERROR: Package 'boost/1.87.0' not resolved: No remote defined. Required by ''

[memsharded]

But the recipes need to be downloaded from ConanCenter, how do you plan to get the recipes from? In your case it seems there were no remotes defined, without remotes, it will fail to retrieve recipes for dependencies.
There are other alternatives, like the local-recipes-index feature that maybe can help, but not very clear what is the issue.

[jackrosenberg]

I would like to be able to build completely locally, with (almost) no internet. I can allow programs to fetch things from the internet, like a github repo/tarball/file - provided they do not change. All access to the internet must be done explicitly.

If i can fetch all the files from an archive before build-time, and then build locally, that would be perfect. Is there a way to download binaries from the cache beforehand?

[memsharded]

There are different ways to populate a cache. For example, you could do a conan install first, with the dependencies you need. Then a conan cache save creates a tarball with a copy of the packages from the cache. Then it can be copied to an air-gapped machine, do a conan cache restore to put those packages and binaries in the cache and make them available for a build without internet.

To build from source from recipes, the local-recipes-index allows to git clone the conan-center-index remote, add the folder as a remote and then work from there.

Note however, that many recipes will download the sources from the internet when necessary to build from source, from the different origins (github, gitlab, sourceforge, etc).

[jackrosenberg]

Hmm okay, thanks for the help so far!

Using the local-recipies-index I can start the build:

madbfs> +++ export CONAN_HOME=/build/source/conan-source
madbfs> +++ CONAN_HOME=/build/source/conan-source
madbfs> +++ cp -r /nix/store/v4fqysd3sd8vbh0xvar6hqk5nyn72a5q-source /build/conan-index
madbfs> +++ chmod 777 -R /build/conan-index
madbfs> +++ conan remote remove conancenter
madbfs> Removed remotes:
madbfs> conancenter: https://center2.conan.io [Verify SSL: True, Enabled: True]
madbfs> +++ conan remote add local /build/conan-index
madbfs> +++ conan remote list
madbfs> local: /build/conan-index [local-recipes-index, Enabled: True]
madbfs> +++ conan install . --build=missing -pr:b=.github/ci/conan-profile.ini -pr:h=.github/ci/conan-profile.ini -vv -of /build/conan
madbfs> 
madbfs> ======== Input profiles ========
madbfs> Profile host:
madbfs> [settings]
madbfs> arch=x86_64
madbfs> build_type=Release
madbfs> compiler=gcc
madbfs> compiler.cppstd=23
madbfs> compiler.libcxx=libstdc++11
madbfs> compiler.version=14
madbfs> os=Linux
madbfs> [conf]
madbfs> tools.build:compiler_executables={'c': '/usr/bin/gcc-14', 'cpp': '/usr/bin/g++-14'}
madbfs> tools.cmake.cmaketoolchain:generator=Ninja
madbfs> 
madbfs> Profile build:
madbfs> [settings]
madbfs> arch=x86_64
madbfs> build_type=Release
madbfs> compiler=gcc
madbfs> compiler.cppstd=23
madbfs> compiler.libcxx=libstdc++11
madbfs> compiler.version=14
madbfs> os=Linux
madbfs> [conf]
madbfs> tools.build:compiler_executables={'c': '/usr/bin/gcc-14', 'cpp': '/usr/bin/g++-14'}
madbfs> tools.cmake.cmaketoolchain:generator=Ninja
<snip>
madbfs> spdlog/1.15.1: Not found in local cache, looking in remotes...
madbfs> spdlog/1.15.1: Checking remote: local
madbfs> Graph root
madbfs>     conanfile.py: /build/source/conanfile.py
madbfs> Requirements
madbfs>     boost/1.87.0#aca561ae3466335fe4bbe620b2dc0217 - Downloaded (local)
madbfs>     libfuse/3.16.2#a29869e9cf770d43d2a1078a3785a6d9 - Downloaded (local)
madbfs>     rapidhash/1.0#7b5258eb18e285f671aa71528ff53715 - Downloaded (local)
madbfs> Build requirements
madbfs>     b2/5.4.2#ffd6084a119587e70f11cd45d1a386e2 - Downloaded (local)
madbfs>     meson/1.10.2#9d2d10681fe7fe61c788c58626c89b25 - Downloaded (local)
madbfs>     ninja/1.13.2#c8c5dc2a52ed6e4e42a66d75b4717ceb - Downloaded (local)
madbfs> Resolved version ranges
madbfs>     b2/[>=5.2 <6]: b2/5.4.2
madbfs>     meson/[>=1.2.0 <2]: meson/1.10.2
madbfs>     ninja/[>=1.10.2 <2]: ninja/1.13.2
madbfs> ERROR: Package 'spdlog/1.15.1' not resolved: Unable to find 'spdlog/1.15.1' in remotes. Required by ''

unfortunately, I fail on spdlog. spdlog is available on my system, can i point conan towards it?

[memsharded]

This is expected, that version is no longer in conan-center-index:

λ cat recipes\spdlog\all\conandata.yml
sources:
  "1.17.0":
    url: "https://github.com/gabime/spdlog/archive/v1.17.0.tar.gz"
    sha256: "d8862955c6d74e5846b3f580b1605d2428b11d97a410d86e2fb13e857cd3a744"
  "1.16.0":
    url: "https://github.com/gabime/spdlog/archive/v1.16.0.tar.gz"
    sha256: "8741753e488a78dd0d0024c980e1fb5b5c85888447e309d9cb9d949bdb52aa3e"
  "1.15.3":
    url: "https://github.com/gabime/spdlog/archive/v1.15.3.tar.gz"
    sha256: "15a04e69c222eb6c01094b5c7ff8a249b36bb22788d72519646fb85feb267e67"
  "1.14.1":
    url: "https://github.com/gabime/spdlog/archive/v1.14.1.tar.gz"
    sha256: "1586508029a7d0670dfcb2d97575dcdc242d3868a259742b69f100801ab4e16b"

The version there is spdlog/1.15.3, use that version instead.

unfortunately, I fail on spdlog. spdlog is available on my system, can i point conan towards it?

You can try the [replace_requires] but the transparent replacement is often not possible, there is a lot of knowledge encoded in the Conan recipes that cannot be just guessed without a recipe, this is exactly what the recipes do. This is also why system wrapper recipes like the opengl/system exists in ConanCenter.

The recommendation is to use the recipe versions that exist in conan-center-index, like 1.15.3 in this case.

[jackrosenberg]

That works, thanks.

Now i need to download the remotes and put them in the place that conan expects them to be.

madbfs> ======== Installing packages ========
madbfs> b2/5.4.2: Calling source() in /build/source/conan-source/p/b205ece0725228b/s/src
madbfs> b2/5.4.2: WARN: network: Error downloading file https://github.com/bfgroup/b2/releases/download/5.4.2/b2-5.4.2.tar.bz2: 'HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /bfgroup/b2/releases/download/5.4.2/b2-5.4.2.tar.bz2 (Caused by NameResolutionError("HTTPSConnection(host='github.com', port=443): Failed to resolve 'github.com' ([Errno -3] Temporary failure in name resolution)"))'

    mkdir -p /build/source/conan-source/p/b205ece0725228b/s
    cp -r ${b2} /build/source/conan-source/p/b205ece0725228b/s/src
    chmod 777 -R /build/source/conan-source/p/b205ece0725228b/s/src

does not stop the download, where should i put the b2 (and other) downloaded sources?

[memsharded]

Now i need to download the remotes and put them in the place that conan expects them to be.

This might not be doable manually. Conan cache is read-only from teh user side, any modification from the outside would be undefined behavior, only Conan can write in the cache.

What conan has is a "download cache", that can be activated with core.sources:download_cache conf, pointing to the folder that you want, and then a conan graph info -c tools.build:download_source=True can force to download the sources of a given dependency graph into that location.

[jackrosenberg]

Is it possible to mimic what conan graph info -c tools.build:download_source=True does manually? I can fetch git repositories explicitly, but not allow conan to do it's own thing unfortunately. Every networking request must be done manually, my me. I can move all of the sources to the right places as well.

Is there a flow that conan just starts the build, after looking at its cache and finding everything it needs? What does the cache need to look like for that to happen?

[memsharded]

Is it possible to mimic what conan graph info -c tools.build:download_source=True does manually? I can fetch git repositories explicitly, but not allow conan to do it's own thing unfortunately. Every networking request must be done manually, my me. I can move all of the sources to the right places as well.

If you are working from a conan-center-index fork you can do different things:

• You can traverse the recipes you use and read the conandata.yml, containing the URLs
• Download those URLs to somewhere local
• Change the conandata.yml to point to the file:///localpath/folder/location
• Then do the consumption of those packages, with --build=missing, it will do the download from those downloaded locations.

Or you can gather those sources, download them, and store them in the core.sources:download_cache folder, with the sha256 as the name of the blob, then no need to change the conandata.yml.

But I still don't fully get it, things needs to be downloaded anyway from the internet at some point, why not doing a full conan install of the dependencies, then doing a conan cache save/restore to make it available those dependencies to the build without internet?

[jackrosenberg]

Thanks again for sticking with this :)

Or you can gather those sources, download them, and store them in the core.sources:download_cache folder, with the sha256 as the name of the blob, then no need to change the conandata.yml.

this seems like the best option to me.

But I still don't fully get it, things needs to be downloaded anyway from the internet at some point, why not doing a full conan install of the dependencies, then doing a conan cache save/restore to make it available those dependencies to the build without internet?

Unfortunately it's not that simple. I cannot enable access to the internet. I can fetch content manually, through a a fetcher which creates fixed output derivation (1). The long and short of it is that all access to the internet is restricted to fetching paths that are known at build-time. I can download manually from URL's, but not allow a 3rd party tool to do this for me.

    export CONAN_HOME="/build/source/conan-source"

    conan remote remove conancenter
    conan remote add local /build/conan-index

    mkdir -p /build/conan-cache

    # hashes from the conandata.yml
    cp ${boost} /build/conan-cache/af57be25cb4c4f4b413ed692fe378affb4352ea50fbe294a11ef548f4d527d89
    # https://github.com/gabime/spdlog/archive/v1.15.3.tar.gz
    cp ${spdlog} /build/conan-cache/15a04e69c222eb6c01094b5c7ff8a249b36bb22788d72519646fb85feb267e67
    https://github.com/bfgroup/b2/releases/download/5.4.2/b2-5.4.2.tar.bz2
    cp ${b2} /build/conan-cache/ab9ba45bc4864963da09a3a97dbfb81271dca039b20f1ef0502a38f360e3b656

    chmod 777 -R /build/conan-cache
    ls -la /build/conan-cache

    conan install . --build=missing -pr:b=.github/ci/conan-profile.ini -pr:h=.github/ci/conan-profile.ini -vv -of /build/conan -cc core.sources:download_cache="/build/conan-cache"

i am building with this:
madbfs> +++ conan install . --build=missing -pr:b=.github/ci/conan-profile.ini -pr:h=.github/ci/conan-profile.ini -vv -of /build/conan -cc core.sources:download_cache=/build/conan-cache

my directory structure looks like this:

madbfs> +++ ls -la /build/conan-cache
madbfs> total 43
madbfs> drwxrwxrwx  5 nixbld nixbld  5 Jun 10 13:23 .
madbfs> drwx------  5 nixbld nixbld  8 Jun 10 13:22 ..
madbfs> drwxrwxrwx 11 nixbld nixbld 21 Jun 10 13:23 15a04e69c222eb6c01094b5c7ff8a249b36bb22788d72519646fb85feb267e67
madbfs> drwxrwxrwx 10 nixbld nixbld 25 Jun 10 13:23 ab9ba45bc4864963da09a3a97dbfb81271dca039b20f1ef0502a38f360e3b656
madbfs> drwxrwxrwx  8 nixbld nixbld 21 Jun 10 13:23 af57be25cb4c4f4b413ed692fe378affb4352ea50fbe294a11ef548f4d527d89

hashes come from here: for example boost
recipes/boost/all/conandata.yml

 "1.87.0":
    url:
      - "https://archives.boost.io/release/1.87.0/source/boost_1_87_0.tar.bz2"
    sha256: "af57be25cb4c4f4b413ed692fe378affb4352ea50fbe294a11ef548f4d527d89"

where the contents of the directories are the unpacked compressed tarbals from the urls listed in the conandata.yml

i think the issue lies here:

madbfs> ======== Computing dependency graph ========
madbfs> boost/1.87.0: Not found in local cache, looking in remotes...
madbfs> boost/1.87.0: Checking remote: local
madbfs> local-recipes-index: Internal export for boost/1.87.0:
madbfs>     boost/1.87.0: Exporting package recipe: /build/conan-index/recipes/boost/all/conanfile.py
madbfs>     boost/1.87.0: exports: File 'conandata.yml' found. Exporting it...
madbfs>     boost/1.87.0: copy(pattern=conandata.yml) copied 1 files
madbfs>       from /build/conan-index/recipes/boost/all/
madbfs>       to /build/source/conan-source/.local_recipes_index/local/.conan/p/t/boost9b02b0500dddb/e
madbfs>       Files:
madbfs>         conandata.yml
madbfs>     boost/1.87.0: Calling export()
madbfs>     boost/1.87.0: copy(pattern=dependencies/dependencies-1.87.0.yml) copied 1 files
madbfs>       from /build/conan-index/recipes/boost/all/
madbfs>       to /build/source/conan-source/.local_recipes_index/local/.conan/p/t/boost9b02b0500dddb/e
madbfs>       Files:
madbfs>         dependencies/dependencies-1.87.0.yml
madbfs>     boost/1.87.0: Calling export_sources()
madbfs>     boost/1.87.0: Copied 2 '.yml' files: conandata.yml, dependencies-1.87.0.yml
madbfs>     boost/1.87.0: Copied 1 '.py' file: conanfile.py
madbfs>     boost/1.87.0: Copied 2 '.patch' files: 1.87.0-boost-fiber-fix-property-numa.patch, 1.82.0-locale-iconv-library-option.patch
madbfs>     boost/1.87.0: Exported to cache folder: /build/source/conan-source/.local_recipes_index/local/.conan/p/boost1982d5c1345e7/e
madbfs>     boost/1.87.0: Exported: boost/1.87.0#aca561ae3466335fe4bbe620b2dc0217 (2026-06-10 13:30:18 UTC)
madbfs> 
madbfs> boost/1.87.0: Downloaded recipe revision aca561ae3466335fe4bbe620b2dc0217

with conan not being able to find the local cache

full logs here

[memsharded]

Thanks for the feedback.

The issue seems that https://github.com/bfgroup/b2/releases/download/5.4.2/b2-5.4.2.tar.bz2 is still missing.
That is, you have populated the necessary tarballs from the "host" context, that is the libraries and transitive libraries.
But you are still missing the tools in the "build" context, such as cmake, ninja, b2, etc.

Some of them like cmake or ninja might be avoidable with a [platform_tool_requires] to tell Conan they are already in your system and no need to depend on the Conan package for it. But others like b2, you might still need the Conan package dependency as tool_requires, and consequently you should also download and populate the download cache for those sources too.