[Migration Engine Part 1] Implement the foundation for migration engine - #5894
Draft
warwickschroeder wants to merge 15 commits into
Draft
warwickschroeder wants to merge 15 commits into
warwickschroeder wants to merge 15 commits into
Conversation
rbev
reviewed
Sep 15, 2026
rbev
left a comment
rbev
left a comment
Contributor
There was a problem hiding this comment.
Partial review - adding as comments for now.
warwickschroeder
added this pull request to stack #5898
September 16, 2026 04:25
warwickschroeder
changed the title
warwickschroeder
changed the title
rbev
reviewed
Sep 16, 2026
johnsimons
approved these changes
Sep 17, 2026
johnsimons
left a comment
johnsimons
left a comment
Member
There was a problem hiding this comment.
A few question and comments but overall, lets keep going
…d ServiceControl migration
…n, clarifying strategies, goals, and migration limitations.
…on source reporting - Implemented MigrationEngineFailurePathTests to validate behavior during write failures and halts. - Created MigrationEngineHaltTests to ensure categories halt correctly on systemic failures. - Added MigrationEngineOptionsTests to verify default and environment variable configurations. - Developed MigrationEngineOrderingTests to check execution order of migration categories. - Introduced MigrationEngineResumeTests to confirm no duplicates or gaps after restarts. - Added MigrationEngineRunCategoriesTests to ensure all categories run in specified order. - Implemented MigrationEngineSkipReasonTests to validate skip reasons and their aggregation. - Enhanced MigrationEngineThrottleTests to verify pause behavior for optional categories. - Updated MigrationSourceReportCommand for improved output and clarity. - Modified Help.txt and HostArguments.cs for better command descriptions. - Refined Settings.cs and PersistenceFactory.cs for clearer migration source configuration.
…r SQL migration from RavenDB
- Introduced `BenignSkipped` property in `MigrationWriteResult` to track rows that the target would have deleted anyway. - Updated `MigrationEngine` logic to ensure benign skips do not count towards the halt threshold. - Added tests to verify that benign skips are handled correctly and do not cause category halts. - Updated skip reasons to include `PastRetention` for better clarity on skipped rows. - Created approval files for category selection tests to ensure correct order and configuration.
- Improved error logging in MigrationEngine to provide clearer context on exceptions. - Added functionality to InMemoryMigrationSource to simulate shutdown behavior during body reads. - Enhanced InMemoryMigrationTarget to allow custom exceptions for failure scenarios. - Introduced TimerRecordingTimeProvider to facilitate testing of timer-related functionality. - Expanded HaltThresholdTests to cover edge cases involving mixed benign and fault skips. - Updated MigrationEngineBodyRetryTests to ensure proper handling of shutdowns during body reads. - Added tests to verify behavior when categories are empty or already completed. - Implemented checks for checkpoint conflicts and cancellation scenarios in MigrationEngineFailurePathTests. - Enhanced MigrationEngineHaltTests to evaluate mixed skip scenarios and their impact on halting. - Improved MigrationEngineOrderingTests to ensure blocked categories behave correctly. - Added tests to MigrationEngineResumeTests to verify that restart behavior maintains original start time. - Updated MigrationEngineRunCategoriesTests to handle cases where no categories are selected. - Enhanced MigrationEngineSkipReasonTests to ensure accurate recording of multiple skip reasons. - Added tests to MigrationEngineThrottleTests to verify behavior during pauses and shutdowns.
warwickschroeder
force-pushed
the
warwick/migration-engine-1
branch
from
September 30, 2026 01:24
8949344 to
0dcf9c8
Compare
…r imports handling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The foundation for moving an error instance's data from RavenDB to SQL Server or PostgreSQL: a read-only RavenDB source, a report command and the copy engine. Nothing is copied yet, there is no SQL target, and the migration is not switched on anywhere.
--migration-source-reportprints the source's version, location, database names and row counts.docs/migration/, linked from the README, and CI gains aMigrationtest category.Tests
ServiceControl.UnitTests/Migration: the engine, category registry, options and fakes.ServiceControl.UnitTests/Hosting: parsing--migration-source-report.ServiceControl.UnitTests/ApprovalFiles: the new setting in the platform sample settings.ServiceControl.Persistence.Tests.RavenDB/DataMigration: the read-only source's lifecycle.ServiceControl.Migration.Tests(new): the report command, and both persisters in one process.