Feature Description
Originated from Xuanwo's comment in #7535 .
OpenDAL's copy and rename has different atomicity depends on the backend support. For example, fs backend will attempt rename and set metadata. This behavior is similar to cp --preserve=xattr. But the two operations in rename are not atomic. (Nor cp --preserve=xattr is atmoic.)
A common trick for applications is to copy a file to a temporary location and then set xattr. And then move it to a destination. This approach is obviously limited in file system.
Problem and Solution
For OpenDAL, we might need a proper solution to define:
- Principle: Do we want to guarantee atomicity for
copy and `rename?
- API design: How do we define API for atomic operation?
Principle: Do we want to guarantee atomicity for copy and `rename?
Some notable implementations:
- s3 has strongly consistence for metadata operations along with objects.
- Linux doesn't set xattr with files in an atomic operation. We might be able to simulate that but this will result in a performance lost. Some file systems doesn't support copy on write.
API design
- Do we want to have
copy and rename to be atomic by default?
- If impossible, do we add an atomic option?
- Along with 2, we could have an capability hinting if using atomic property will result in performance loss.
Additional Context
Are you willing to contribute to the development of this feature?
Feature Description
Originated from Xuanwo's comment in #7535 .
OpenDAL's
copyandrenamehas different atomicity depends on the backend support. For example,fsbackend will attemptrenameand set metadata. This behavior is similar tocp --preserve=xattr. But the two operations in rename are not atomic. (Norcp --preserve=xattris atmoic.)A common trick for applications is to copy a file to a temporary location and then set xattr. And then move it to a destination. This approach is obviously limited in file system.
Problem and Solution
For OpenDAL, we might need a proper solution to define:
copyand `rename?Principle: Do we want to guarantee atomicity for
copyand `rename?Some notable implementations:
API design
copyandrenameto be atomic by default?Additional Context
Are you willing to contribute to the development of this feature?