Add read-through filesystems for gradual storage migrations

src/docs/filesystem.md

@@ -6,7 +6,7 @@
     - [The Public Disk](#the-public-disk)
     - [Driver Prerequisites](#driver-prerequisites)
     - [Driver Pools](#driver-pools)
-    - [Scoped and Read-Only Filesystems](#scoped-and-read-only-filesystems)
+    - [Scoped, Read-Only, and Read-Through Filesystems](#scoped-and-read-only-filesystems)
     - [Amazon S3 Compatible Filesystems](#amazon-s3-compatible-filesystems)
 - [Obtaining Disk Instances](#obtaining-disk-instances)
     - [On-Demand Disks](#on-demand-disks)
@@ -207,7 +207,6 @@ If you need to configure a Google Cloud Storage filesystem manually, you may use
     'path_prefix' => env('GOOGLE_CLOUD_STORAGE_PATH_PREFIX', ''),
     'storage_api_uri' => env('GOOGLE_CLOUD_STORAGE_API_URI'),
     'api_endpoint' => env('GOOGLE_CLOUD_STORAGE_API_ENDPOINT'),
-    'visibility' => 'public',
     'visibility_handler' => null,
     'metadata' => ['cacheControl' => 'public,max-age=86400'],
     'throw' => false,
@@ -288,7 +287,7 @@ $result = Storage::disk('s3')->withClient(function ($client) {
 S3 and Google Cloud Storage streams are read lazily by default, which keeps memory usage bounded and makes data available before the entire file has downloaded. This applies to `readStream()` and `readStreamRange()`; methods such as `get()` retain their normal behavior. Streaming requests close their HTTP connection after the read, so applications that open many small streams may prefer connection reuse and set the disk's `stream_reads` option to `false`.
 
 <a name="scoped-and-read-only-filesystems"></a>
-### Scoped and Read-Only Filesystems
+### Scoped, Read-Only, and Read-Through Filesystems
 
 Scoped disks allow you to define a filesystem where all paths are automatically prefixed with a given path prefix.
 
@@ -360,6 +359,28 @@ Dynamic scoped filesystems fail closed when the resolved prefix is empty. Pass `
 
 Failed writes follow the scoped disk's `throw` and `report` options.
 
+Read-through disks allow you to migrate files between disks without downtime. When reading a file, Hypervel checks the primary disk first. If the file only exists on the fallback disk, Hypervel reads it from the fallback disk and copies it to the primary disk for future requests:
+
+```php
+'assets' => [
+    'driver' => 'read-through',
+    'primary' => 's3',
+    'fallback' => 'legacy-s3',
+],
+```
+
+New files and directory listings use the primary disk. URLs, file existence checks, and metadata use the disk containing the file without copying it. Deletions remove files or directories from both disks, and visibility changes apply to the disk containing the file. The `primary` and `fallback` options may also contain inline disk configurations.
+
+To scope a read-through disk per request or tenant, wrap it in `ScopedCloudFilesystemProxy`. Dynamic scoped proxies cannot be used as its primary or fallback disk.
+
+Fallback reads promote files by default. Set `copy` to `false` to read fallback files without copying them. With promotion enabled, fallback stream reads finish copying the file before returning the stream.
+
+Promoted files use the primary disk's default visibility rather than inheriting the fallback file's visibility. Fallback copy and move operations use the same default. Configure a private primary disk when migrating private files.
+
+Promotion does not lock files across the two disks. Coordinate writes and deletions to a path while it is being copied; otherwise, promotion can overwrite a concurrent write or restore a deleted file.
+
+By default, a `FilesystemException` raised while writing the promoted copy does not fail the read. Set `throw_on_promotion_failure` to `true` to treat it as a read failure; set the disk's `throw` option to `true` to receive that failure as an exception. Other errors, including pool wait timeouts, still propagate.
+
 <a name="amazon-s3-compatible-filesystems"></a>
 ### Amazon S3 Compatible Filesystems
 

src/filesystem/README.md

@@ -15,6 +15,8 @@ The configured disk name `ondemand` is reserved. Disks explicitly set under that
 
 Hypervel pools S3 and Google Cloud Storage SDK clients rather than complete disk adapters. Disks with equivalent client construction config share the expensive client pool while retaining their own bucket, root, visibility, and callback behavior. Pooled disks expose raw internals only through borrow-scoped `withClient()`, `withDriver()`, and `withAdapter()` callbacks.
 
+On read-through disks, raw `listContents()` results from pooled, S3 and Google Cloud Storage sides are fully loaded instead of streamed.
+
 Filesystem construction differs from Laravel in how it carries logical disk identity. `callCustomCreator()` accepts the logical disk name as an optional second parameter, so existing one-argument calls remain valid while overrides must adopt the parameter. The public `build()` method uses a logical-name-aware construction path rather than the protected `resolve()` method because anonymous builds must pass a null name to creators; `resolve()` remains the configured-disk seam. `createScopedDriver()` resolves its prepared descriptor directly rather than through `build()`, so override `createScopedDriver()` for scoped construction customization. Customize on-demand construction through `Storage::extend()` or the public driver creator methods. Creator callbacks may accept the nullable name as a third argument after the application and configuration, while existing two-argument callbacks remain valid. Hypervel carries the name through scoped reconstruction and whole-driver pool fingerprints. A matching explicit fingerprint declares that every construction detail is equivalent, including serving-route ownership.
 
 Hypervel registers signed file-serving routes for any configured disk whose `serve` option is exactly `true`, while Laravel limits these routes to local disks and accepts truthy values. Every served disk must use a unique URL or application boot will fail. Custom drivers that opt in must provide the filesystem response methods used by these routes.

src/filesystem/composer.json

@@ -30,6 +30,7 @@
         "hypervel/collections": "^0.4",
         "hypervel/conditionable": "^0.4",
         "hypervel/container": "^0.4",
+        "hypervel/context": "^0.4",
         "hypervel/contracts": "^0.4",
         "hypervel/coroutine": "^0.4",
         "hypervel/http": "^0.4",

src/filesystem/src/AwsS3V3Adapter.php

@@ -135,7 +135,14 @@ public function temporaryUploadUrl(string $path, DateTimeInterface $expiration,
      */
     public function readStream(string $path): mixed
     {
-        return $this->readStreamWithOptions($path);
+        try {
+            return $this->readStreamWithOptions($path);
+        } catch (UnableToReadFile $exception) {
+            throw_if($this->throwsExceptions(), $exception);
+            $this->report($exception);
+
+            return null;
+        }
     }
 
     /**
@@ -151,15 +158,35 @@ public function readStreamRange(string $path, ?int $start, ?int $end): mixed
             return $this->readStream($path);
         }
 
-        return $this->readStreamWithOptions($path, [
-            'Range' => "bytes={$start}-{$end}",
-        ]);
+        try {
+            return $this->readStreamRangeOrFail($path, $start, $end);
+        } catch (UnableToReadFile $exception) {
+            throw_if($this->throwsExceptions(), $exception);
+            $this->report($exception);
+
+            return null;
+        }
+    }
+
+    /**
+     * Open a whole-object or ranged stream without applying the disk's failure policy.
+     *
+     * @return resource
+     */
+    public function readStreamRangeOrFail(string $path, ?int $start = null, ?int $end = null): mixed
+    {
+        [$start, $end] = $this->normalizeStreamRange($start, $end);
+
+        return $this->readStreamWithOptions(
+            $path,
+            $start === null && $end === null ? [] : ['Range' => "bytes={$start}-{$end}"],
+        );
     }
 
     /**
      * Read an object while preserving configured options and operation-owned keys.
      *
-     * @return null|resource
+     * @return resource
      */
     private function readStreamWithOptions(string $path, array $operationOptions = []): mixed
     {
@@ -181,24 +208,14 @@ private function readStreamWithOptions(string $path, array $operationOptions = [
         } catch (CanceledException $exception) {
             throw $exception;
         } catch (Throwable $exception) {
-            $exception = UnableToReadFile::fromLocation($path, $exception->getMessage(), $exception);
-
-            throw_if($this->throwsExceptions(), $exception);
-            $this->report($exception);
-
-            return null;
+            throw UnableToReadFile::fromLocation($path, $exception->getMessage(), $exception);
         }
 
         if (! is_resource($stream)) {
-            $exception = UnableToReadFile::fromLocation(
+            throw UnableToReadFile::fromLocation(
                 $path,
                 'Downloaded object does not contain a file resource.',
             );
-
-            throw_if($this->throwsExceptions(), $exception);
-            $this->report($exception);
-
-            return null;
         }
 
         return $stream;

src/filesystem/src/Concerns/InteractsWithPooledFilesystem.php

@@ -9,13 +9,17 @@
 use DateTimeInterface;
 use Hypervel\Container\Container;
 use Hypervel\Contracts\Filesystem\Filesystem as FilesystemContract;
+use Hypervel\Filesystem\AwsS3V3Adapter;
 use Hypervel\Filesystem\FileResponseBuilder;
+use Hypervel\Filesystem\FilesystemOperatorAdapter;
+use Hypervel\Filesystem\GoogleCloudStorageAdapter;
 use Hypervel\Http\File;
 use Hypervel\Http\Request;
 use Hypervel\Http\UploadedFile;
 use Hypervel\Image\Image;
 use Hypervel\Image\ImageException;
 use Hypervel\Support\Traits\Conditionable;
+use League\Flysystem\FilesystemOperator;
 use Psr\Http\Message\StreamInterface;
 use RuntimeException;
 use Symfony\Component\HttpFoundation\Response;
@@ -526,6 +530,34 @@ public function withDriver(Closure $callback): mixed
         return $this->withBorrowedAccessor('getDriver', $callback);
     }
 
+    /**
+     * Get an operator that owns each operation's borrow and each stream's lease.
+     */
+    public function getOperator(): FilesystemOperator
+    {
+        return new FilesystemOperatorAdapter(
+            $this->withDriver(...),
+            fn (string $path): mixed => $this->leasedStream(static function (FilesystemContract $filesystem) use ($path): mixed {
+                if ($filesystem instanceof AwsS3V3Adapter || $filesystem instanceof GoogleCloudStorageAdapter) {
+                    return $filesystem->readStreamRangeOrFail($path);
+                }
+
+                if (! method_exists($filesystem, 'getDriver')) {
+                    throw new RuntimeException(
+                        'Pooled filesystem driver [' . $filesystem::class . '] does not support [getDriver] access.',
+                    );
+                }
+
+                return $filesystem->getDriver()->readStream($path);
+            }),
+            fn (string $path, ?int $start, ?int $end): mixed => $this->leasedStream(
+                static fn (FilesystemContract $filesystem): mixed => $filesystem instanceof AwsS3V3Adapter || $filesystem instanceof GoogleCloudStorageAdapter
+                    ? $filesystem->readStreamRangeOrFail($path, $start, $end)
+                    : null,
+            ),
+        );
+    }
+
     /**
      * Run a callback with borrow-scoped access to the Flysystem adapter.
      */

src/filesystem/src/FilesystemAdapter.php

@@ -451,7 +451,9 @@ public function putFileAs(string|File|UploadedFile $path, array|string|File|Uplo
         try {
             $result = $this->put($path, $stream, $options);
         } finally {
-            @fclose($stream);
+            if (is_resource($stream)) {
+                fclose($stream);
+            }
         }
 
         return $result ? $path : false;

src/filesystem/src/FilesystemManager.php

@@ -7,6 +7,7 @@
 use Aws\S3\S3Client;
 use Closure;
 use Google\Cloud\Storage\StorageClient as GcsClient;
+use Hypervel\Context\CoroutineContext;
 use Hypervel\Contracts\Container\Container;
 use Hypervel\Contracts\Filesystem\Cloud;
 use Hypervel\Contracts\Filesystem\Factory as FactoryContract;
@@ -58,6 +59,11 @@ class FilesystemManager implements FactoryContract
      */
     protected const string ON_DEMAND_DISK_NAME = 'ondemand';
 
+    /**
+     * The coroutine-local construction stack prefix for each manager.
+     */
+    protected const string READ_THROUGH_CONTEXT_KEY_PREFIX = '__filesystem.read-through.construction.';
+
     /**
      * Google Cloud Storage client constructor options supported by the installed SDK.
      */
@@ -244,7 +250,7 @@ private function resolveConstructionDescriptor(
             return $this->createClientPooledDisk($driver, $config);
         }
 
-        $driverMethod = 'create' . ucfirst($driver) . 'Driver';
+        $driverMethod = 'create' . Str::studly($driver) . 'Driver';
 
         if (! method_exists($this, $driverMethod)) {
             throw new InvalidArgumentException("Driver [{$driver}] is not supported.");
@@ -434,6 +440,95 @@ public function createS3Driver(array $config): Cloud
         );
     }
 
+    /**
+     * Create an instance of the read-through driver.
+     */
+    public function createReadThroughDriver(array $config, string $name = 'read-through'): Filesystem
+    {
+        if (! isset($config['primary']) || $config['primary'] === '' || $config['primary'] === []) {
+            throw new InvalidArgumentException('Read-through disk is missing "primary" configuration option.');
+        }
+        if (! isset($config['fallback']) || $config['fallback'] === '' || $config['fallback'] === []) {
+            throw new InvalidArgumentException('Read-through disk is missing "fallback" configuration option.');
+        }
+        if ($config['primary'] === $config['fallback']) {
+            throw new InvalidArgumentException('Read-through disk requires distinct "primary" and "fallback" disks.');
+        }
+
+        // Scoped inline sides can re-enter construction without resolving a named disk.
+        $contextKey = self::READ_THROUGH_CONTEXT_KEY_PREFIX . spl_object_id($this);
+        $stack = CoroutineContext::get($contextKey, []);
+        $label = $name === self::ON_DEMAND_DISK_NAME ? '(on-demand)' : $name;
+
+        foreach ($stack as $entry) {
+            if ($entry['config'] !== $config) {
+                continue;
+            }
+
+            if (count($stack) === 1) {
+                throw new InvalidArgumentException("Read-through disk [{$label}] cannot reference itself.");
+            }
+
+            $cycle = [...array_column($stack, 'name'), $label];
+
+            throw new InvalidArgumentException('Circular read-through disk definition detected: ' . implode(' -> ', $cycle) . '.');
+        }
+
+        CoroutineContext::set($contextKey, [...$stack, ['config' => $config, 'name' => $label]]);
+
+        try {
+            $primary = is_array($config['primary'])
+                ? $this->resolveWithLogicalName(self::ON_DEMAND_DISK_NAME, $config['primary'], null)
+                : $this->disk($config['primary']);
+            $fallback = is_array($config['fallback'])
+                ? $this->resolveWithLogicalName(self::ON_DEMAND_DISK_NAME, $config['fallback'], null)
+                : $this->disk($config['fallback']);
+
+            if (! $primary instanceof Cloud || ! $fallback instanceof Cloud) {
+                throw new InvalidArgumentException('Read-through disks must implement the cloud filesystem contract.');
+            }
+
+            $adapter = new ReadThroughFilesystemAdapter(
+                $this->readThroughOperator($primary),
+                $this->readThroughOperator($fallback),
+                $config['throw_on_promotion_failure'] ?? false,
+                $config['copy'] ?? true,
+            );
+
+            return new ReadThroughFilesystem(
+                $this->createFlysystem($adapter, $config),
+                $primary instanceof FilesystemAdapter ? $primary->getAdapter() : $adapter,
+                array_replace($primary->getConfig(), $config), // @phpstan-ignore method.notFound (Pooled decorators forward adapter accessors.)
+                $primary,
+                $fallback,
+                $config['prefix'] ?? '',
+                $adapter,
+            );
+        } finally {
+            if ($stack === []) {
+                CoroutineContext::forget($contextKey);
+            } else {
+                CoroutineContext::set($contextKey, $stack);
+            }
+        }
+    }
+
+    /**
+     * Get a side operator without exposing borrowed clients or losing native cloud reads.
+     */
+    protected function readThroughOperator(Cloud $disk): FilesystemOperator
+    {
+        if ($disk instanceof AwsS3V3Adapter || $disk instanceof GoogleCloudStorageAdapter) {
+            return new FilesystemOperatorAdapter(
+                static fn (Closure $operation): mixed => $operation($disk->getDriver()),
+                $disk->readStreamRangeOrFail(...),
+                $disk->readStreamRangeOrFail(...),
+            );
+        }
+
+        return $disk instanceof FilesystemAdapter ? $disk->getDriver() : $disk->getOperator(); // @phpstan-ignore method.notFound (Pooled decorators forward the borrow-safe accessor.)
+    }
+
     /**
      * Derive the S3 client construction config from a disk config.
      */