Skip to content

Replace EFS with object storage#855

Closed
munishchouhan wants to merge 79 commits into
masterfrom
replace-efs-with-objectstore
Closed

Replace EFS with object storage#855
munishchouhan wants to merge 79 commits into
masterfrom
replace-efs-with-objectstore

Conversation

@munishchouhan

Copy link
Copy Markdown
Member

this PR will replace efs which wave uses for build, scan and mirror with object storage

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan munishchouhan self-assigned this Jun 10, 2025
@munishchouhan
munishchouhan marked this pull request as draft June 10, 2025 07:53
munishchouhan and others added 25 commits June 10, 2025 12:59
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
munishchouhan and others added 10 commits July 17, 2025 18:34
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
munishchouhan marked this pull request as ready for review July 17, 2025 22:40
@munishchouhan
munishchouhan requested a review from pditommaso July 17, 2025 23:02
Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
munishchouhan requested review from fntlnz and jordeu July 21, 2025 07:58
@munishchouhan

Copy link
Copy Markdown
Member Author

Currently, in this PR, layers are uploaded to object store by changing the stream to byte array, I am working on stream the layers to object store

objectStorageOperations.upload(UploadRequest.fromBytes(IOUtils.toByteArray(stream), target, "application/x-tar"))

objectStorageOperations.upload(UploadRequest.fromBytes(TarGzipUtils.untarGzip(stream), target, "application/octet-stream"))

objectStorageOperations.upload(UploadRequest.fromBytes(TarGzipUtils.untarGzip(stream), contextDir, "application/octet-stream"))

@fntlnz

fntlnz commented Jul 21, 2025

Copy link
Copy Markdown

@munishchouhan is this ready for review?

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan

Copy link
Copy Markdown
Member Author

I have added InputStreamUploadRequest to pass InputStream to upload method instead of a byte array
But I also found that micronaut is changing the InputStream to byte array when sending an HTTP request to AWS

https://github.com/micronaut-projects/micronaut-object-storage/blob/089269b09978b5ecadf89079c34d08e2e808ca9d/object-storage-aws/src/main/java/io/micronaut/objectstorage/aws/AwsS3Operations.java#L227

@munishchouhan

Copy link
Copy Markdown
Member Author

I have added a PR in micronaut to see if it is possible to send inputstream in http request
micronaut-projects/micronaut-object-storage#632

@munishchouhan

Copy link
Copy Markdown
Member Author

@munishchouhan is this ready for review?

@fntlnz yes its ready

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan

Copy link
Copy Markdown
Member Author

I have also added CustomAwsS3Operations to send inputstream in httprequest

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan

Copy link
Copy Markdown
Member Author

tested successfully in dev

@pditommaso

Copy link
Copy Markdown
Collaborator

Closing as not planned anymore

@pditommaso pditommaso closed this Oct 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants