[question] Are remote commands synchronous or asynchronous?

[SzBosch]

What is your question?

Hello together,

are remote commands like "conan remove" or "conan upload" synchronous or asynchronous regarding the remote storage?

Background: Recently I got the problem on "conan remove" or "conan upload" commands, that an immediate "conan list" command afterwards did not show the removed/uploaded conan package in Artifactory. Only after some time (many minutes) running "conan list" again showing the the removed/uploaded conan package.

Versions used:
Conan Client: 2.20.0
Artifactory: 7.125.14

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @SzBosch

Thanks for your question.

From the Conan client side, the calls are synchronous. Conan issues HTTP requests to the server, and waits synchronously for a response. The intended design is that servers respond when the operation is done too, but this is out of the scope of Conan as it is a server side implementation detail and different servers might implement it differently.

What might be happening in Artifactory could be related to indexing. If you do the operation and it returns correctly, you should be able to go into the WebUI and see the artifacts there. It is very likely that some of the operations that doesn't need indexing, like direct retrieval of specific revisions will work without issues immediately.
But conan list operations on the Artifactory side use the result of indexing. This indexing is in general fast enough, but if it is taking a few minutes in your case, it could be an indicator of resources exhaustion or other performance issues on the server side, so I'd recommend investigating this on the server side.

[memsharded]

Hi @SzBosch

Did the above help? Thanks for the feedback.

[SzBosch]

Hello @memsharded, thank you for your answer.

I understand your answer as: Conan to Artifactory commands are not ent-to-end synchronous esp. regarding the index update.

I got the problem in the meanwhile for both, package upload and package deletion in Artifactory.
It seems this file is not properly updated: https://jfrog.com/help/r/artifactory-what-is-the-packages-ref-json-file-in-artifactory-conan-repositories/artifactory-what-is-the-packages.ref.json-file-in-artifactory-conan-repositories.
Up to now some workflows rely on when a conan command like "upload" and "remove" is executed any subsequent conan command sees this changed state immediately and consistently.

So even with a high performance setup, there can be this glitch/race-condition regarding the indexing, if the end-to-end behavior of the conan commands are not synchronous by design.
Just curious whether other users ran into similar issues?

[memsharded]

Yes, it might be worth following up with JFrog regarding this specific behavior, so my understanding is that there are 2 different issues here:

• Interrupted or problematic uploads, that might not transfer all the files. The package might end in a corrupted/incomplete state in the server, there is no cleanup/rollback in the server if this happens. This is generally not a big issue, we have seen users minimizing this with a retry in the CI pipelines that do the upload if they don't finish succesfully, and in other case if a package is corrupted is relatively easy to realize about it, as its consumption will be broken.
• Slow or lack of indexing. The files are trasnferred, but the server takes a while to update the packages.refs.json or the index.json files. This is a bit more subtle to detect, the client finishes successfully, but posterior actions like a conan list -r=myremote might still not retrieve the recently uploaded things (while a conan install that doesn't require version-ranges can succeed).

In both cases there is little that we can do from the client side, so better reporting to the server side.