Adding support for GCS S3 with Multicloud Connector

[salamianar-microsoft]

Description

• Feature / Bug Fix: (Brief description of the feature or issue being addressed)

• Related Links:

• Issues

• Team thread

• Documents

• [Email Subject]

Type of Change

• Bug fix
• New feature
• Documentation update required
• Code quality improvement
• Other (describe):

How Has This Been Tested?

Thank you for your contribution to AzCopy!

[Copilot]

Pull request overview

This PR extends AzCopy’s S3 integration to better support Google Cloud Storage via S3-compatible endpoints (including GCP V4 signed URLs when GCP credentials are available), while also adding several Mover-build optimizations around private networking transfers, sync orchestration, plan-file memory usage, and richer transfer/status reporting.

Changes:

• Upgrades MinIO usage to minio-go/v7 across core code and test suites, updating call signatures to context-based APIs.
• Adds GCS-via-S3 endpoint handling (credential selection + GCS V4 signed URL path) and introduces skipping behavior for S3 archive-tier objects that aren’t restored.
• Introduces multiple Mover-oriented changes: private-network chunk reading, sync orchestrator options/plumbing, progressive plan-file unmapping, and transfer error-message propagation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
testSuite/cmd/upload.go	MinIO v7 import + ctx PutObject
testSuite/cmd/create.go	MinIO v7 updates; ctx bucket/object ops
testSuite/cmd/common.go	MinIO v7 client creation via Options/Creds
testSuite/cmd/clean.go	MinIO v7 list/remove APIs w/ ctx
ste/xfer.go	Private-network S3->Blob uploader selection
ste/xfer-anyToRemote-fileProperties.go	Mover-specific entity-type assertion tweak
ste/xfer-anyToRemote-file.go	GCS timestamp tolerance; private-network chunk reader; archive skip handling
ste/sourceInfoProvider.go	Remote range-read interface; SMB change time plumbing
ste/sourceInfoProvider-S3.go	GCS-via-S3 support (GCP signing, range reads, archive checks) + MinIO v7
ste/sourceInfoProvider-Local_windows.go	Add FileChangeTime stub (Windows)
ste/sourceInfoProvider-Local_linux.go	Add FileChangeTime from statx (Linux)
ste/sourceInfoProvider-GCP.go	Refactor signed URL helper; stub GetObjectRange
ste/sourceInfoProvider-File.go	Add FileChangeTime adapters; stub GetObjectRange
ste/sourceInfoProvider-Blob.go	Stub GetObjectRange
ste/sourceInfoProvider_md5_test.go	MinIO v7 test updates
ste/sender-blobFS.go	Clone metadata before mutation; add GenerateCopyMetadata
ste/sender-azureFileFromLocal.go	Add GenerateCopyMetadata for Azure Files uploader
ste/sender-azureFile.go	Preserve times logic updates; SMB change time support; ParentNotFound log tweak
ste/pacer-tokenBucketPacer.go	Wait-loop when pacer target is negative
ste/mgr-JobPartTransferMgr.go	Persist transfer error message into plan
ste/mgr-JobPartMgr.go	Cached plan fields + progressive plan unmapping + diagnostics
ste/jobStatusManager.go	Track file-property transfers + skipped archive counts
ste/JobPartPlanFileName.go	Explicit fsync/fdatasync + close; plan flush logging
ste/JobPartPlan.go	Add bounded per-transfer error message storage
ste/folderCreationTracker.go	RWMutex + optional folder creation locking (Mover)
ste/folderCreationTracker_test.go	Update tracker test for RWMutex/locking flag
ste/downloader-azureFiles_windows.go	Respect updated ShouldTransferLastWriteTime signature
ste/downloader-azureFiles_linux.go	Receiver method + updated ShouldTransferLastWriteTime usage
jobsAdmin/JobsAdmin.go	Add log-level control + STE stats registration hooks
jobsAdmin/init.go	Register stats monitor; resume plumbing for provider/error message; file-property counts
go.mod	MinIO v7 + dependency updates
e2etest/scenario_helpers.go	MinIO v7 API updates in e2e helpers
e2etest/newe2e_task_validation.go	Symlink validation + metadata time normalization
e2etest/newe2e_task_runazcopy.go	Add SYNC_ORCHESTRATOR_TEST_MODE env support
e2etest/newe2e_resource_managers_local.go	EntityType alignment; mkdir exists handling; ReadLink
e2etest/newe2e_resource_managers_file.go	ReadLink stub (unsupported)
e2etest/newe2e_resource_managers_blobfs.go	Symlink body defaults; allow Download for symlink; ReadLink
e2etest/newe2e_resource_managers_blob.go	Symlink body defaults; ReadLink
e2etest/newe2e_resource_manager_mock.go	Add ReadLink stub
e2etest/newe2e_resource_manager_interface.go	Add ReadLink to interface
e2etest/newe2e_azcopy_environment_manager.go	Fix env registration; lock defer; safer cleanup + logs
e2etest/helpers.go	MinIO v7 updates; bucket/object ops w/ ctx
e2etest/declarativeScenario.go	Surface error message in auth-fail hint
common/writeThroughFile_darwin.go	Add GetExtendedProperties stub (darwin)
common/writeThoughFile.go	Add ExtendedProperties interface + defaults
common/writeThoughFile_windows.go	Windows ExtendedProperties implementation
common/writeThoughFile_linux.go	Linux ExtendedProperties via statx/stat
common/unixStatAdapter.go	Helpers to parse time metadata (ctime/modtime)
common/sysinfo_windows.go	Windows memory detection utilities
common/sysinfo_linux.go	Linux memory + cgroup-aware detection utilities
common/sysinfo_darwin.go	Darwin memory stubs + details struct
common/syncOrchestratorUtils.go	Sync orchestrator test-mode env parsing
common/s3URLParts_test.go	Add OCI + GCS endpoint parsing tests
common/s3Models.go	MinIO v7 import update
common/s3Errors.go	Add S3 XML error parsing + wrappers
common/rpc-models.go	Provider plumbing; error message + new counters in RPC models
common/retryUtils.go	Add generic network retry helpers
common/oauthTokenManager.go	SPN cert-data login path; mover authority host handling
common/mmf_linux.go	Madvise before munmap to drop cached pages
common/logger.go	Add ChangeLogLevel to logger interface/impl
common/logger_unix.go	Add ChangeLogLevel to syslog logger
common/lifecyleMgr.go	Mover-specific progress frequency behavior
common/httpErrors.go	Add generic HTTP status error parsing
common/folderDeletionManager.go	Add options; allow recursive mode to force manager
common/fe-ste-models.go	Add skipped-archive status + archive-not-restored error
common/fdlimits_windows.go	Windows fd limit stub
common/fdlimits_unix.go	Unix RLIMIT_NOFILE helper
common/fdatasync_other.go	Non-Linux fdatasync no-op
common/fdatasync_linux.go	Linux fdatasync implementation
common/environment.go	Add S3_COMPATIBLE_ENDPOINT; instance discovery flag
common/credentialFactory.go	MinIO v7 client creation; private-network S3 client logic; provider support
common/channelUtils.go	Add channel backpressure profiles (Mover)
common/buildmode/mover_buildmode.go	Mover build tag flag
common/buildmode/default_buildmode.go	Default build tag flag
common/azHttpClient.go	Global HTTP client + connection reuse tracing hooks
cmd/zt_test.go	MinIO v7 updates in cmd tests
cmd/zt_sync_file_file_test.go	Sync comparator ctor signature updates
cmd/zt_scenario_helpers_for_test.go	MinIO v7 updates in cmd scenario helpers
cmd/zt_generic_traverser_test.go	MinIO v7 + WalkWithSymlinks options struct updates
cmd/zt_generic_service_traverser_test.go	MinIO v7 RemoveBucket ctx updates
cmd/zc_traverser_s3_service.go	MinIO v7; ctx ListBuckets; bucket name passed for S3 client
cmd/zc_processor.go	Sync-orchestrator scheduling mutex; folder trailing-slash trim; new counters
cmd/zc_newobjectadapters.go	Add FileChangeTime support; add directory file-properties adapter
cmd/zc_enumerator.go	StoredObject timestamp model updates; sync orchestrator scaffolding/types
cmd/traverser_models.go	Add TraverserErrorItemInfo interface
cmd/syncOrchestratorModels.go	Add sync orchestrator options model + defaults
cmd/syncIndexer.go	Add RWMutex gating for mover+orchestrator indexing
cmd/syncDefaultHandler.go	Default sync-handler build-tag entry point
cmd/root.go	Folder init refactor; system stats monitor hooks
cmd/root_windows.go	MinIO transport/proxy wiring for Windows
cmd/login.go	Add SPN arg-based cert login path
cmd/credentialUtil.go	Allow GCP creds for S3(GCS); global HTTP client reuse
cmd/copyEnumeratorInit.go	Track skipped archive objects; sync orchestrator folder option tweak
cmd/copyEnumeratorHelper.go	Count file-property transfers
cmd/copy.go	Surface skipped archive counts in summary
azurePipelineTemplates/run-e2e.yml	Build mover+slidingwindow tags for SW suite
azure-pipelines.yml	Add separate SW E2E job
.github/CODEOWNERS	Update code owners list

Comments suppressed due to low confidence (1)

ste/sourceInfoProvider-S3.go:323

• GetObjectRange uses context.Background() for the S3 GetObject call, so range reads won't be cancelled when the job is cancelled. This can leave goroutines stuck on network I/O during shutdown.

	ctx := context.Background()
	// Get the object with the specified range
	body, err := p.s3Client.GetObject(ctx, p.s3URLPart.BucketName, p.s3URLPart.ObjectKey, options)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.