diff --git a/docs/migration/ravendb-to-sql-migration-instructions.md b/docs/migration/ravendb-to-sql-migration-instructions.md index 91b421c143..d229d03d15 100644 --- a/docs/migration/ravendb-to-sql-migration-instructions.md +++ b/docs/migration/ravendb-to-sql-migration-instructions.md @@ -1,9 +1,9 @@ # Migrating from RavenDB to SQL Server or PostgreSQL -This page covers what you can run today. How the migration works, and what is planned, is in the [migration overview](ravendb-to-sql-migration-overview.md) and the [system design diagram](migration-system-design-diagram.png). +This page covers reporting on the RavenDB source before you migrate. How the migration works is in the [migration overview](ravendb-to-sql-migration-overview.md) and the [system design diagram](migration-system-design-diagram.png). > [!NOTE] -> Copying data is not built yet. The one migration command available is the source report. It sends RavenDB only reads, but loading a database lets RavenDB's own expiration, its automatic deletion of documents past their retention date, run against it. If you are keeping the RavenDB database as a fallback, back it up before you run the report, as [Goals](ravendb-to-sql-migration-overview.md#goals) explains. +> The source report sends RavenDB only reads, but loading a database lets RavenDB's own expiration, its automatic deletion of documents past their retention date, run against it. If you are keeping the RavenDB database as a fallback, back it up before you run the report, as [Goals](ravendb-to-sql-migration-overview.md#goals) explains. ## Before you start @@ -42,10 +42,11 @@ The report prints the RavenDB server version, whether the source is embedded or ## If the report fails -The error names the setting to fix: +The error says what to fix: - **"has no database named ..."**: the database name setting it quotes is wrong. - **"refused its client certificate access ..."**: grant that certificate Read access to the database, or supply a certificate that has it. +- **"could not start a server for the embedded database ..."**: a ServiceControl instance is still running against that data directory and holds it. Stop the instance, run the report, then start it again. ## Not available yet diff --git a/docs/migration/ravendb-to-sql-migration-overview.md b/docs/migration/ravendb-to-sql-migration-overview.md index 9c167c3c0b..b1277bd1e8 100644 --- a/docs/migration/ravendb-to-sql-migration-overview.md +++ b/docs/migration/ravendb-to-sql-migration-overview.md @@ -1,8 +1,8 @@ # Moving data from RavenDB to SQL -## Problem +## Purpose -A customer can already point ServiceControl at SQL Server or PostgreSQL. They cannot bring their existing data with them. +The migration moves an error instance's data from RavenDB to SQL Server or PostgreSQL, so a customer can switch persisters and keep their data. This covers the error instance only. The audit instance has no SQL persister, so a customer who finishes this migration still runs RavenDB for audit. @@ -16,7 +16,7 @@ This covers the error instance only. The audit instance has no SQL persister, so - **Minimal downtime**. Only the required data copies with ServiceControl closed. Optional data copies in the background while it serves traffic. - **All three RavenDB sources are supported**. Embedded, a container, or RavenDB Cloud, on one code path rather than three. -- **No writes through the client**. The copier never changes the source, but RavenDB's own expiration does: the primary database already has it configured, and the sweep keeps deleting failed messages and event log items throughout the migration and for as long afterwards as the instance is left running. The old database is a fallback that degrades from the moment you start. +- **No writes through the client**. The copier never changes the source, but RavenDB's own expiration does: the primary database has it configured, and the sweep keeps deleting failed messages and event log items throughout the migration and for as long afterwards as the instance is left running. The old database is a fallback that degrades from the moment you start. - **Abandonable up to a known point, and only up to that point**. While ServiceControl is closed the copy can be thrown away at no cost, because nothing but the copier has written to SQL and the migration has written nothing to RavenDB: see [the one point you can go back](#the-one-point-you-can-go-back). Once the host opens there is no way back at all. - **No duplicates and no gaps**. Rows and the resume cursor, a marker of the last row copied, commit in one transaction, so a crash needs no reconciliation. - **Every identifier anything depends on is carried across**. The event log, historic retry operations and pending integration events are renumbered, because nothing references their keys. @@ -26,13 +26,13 @@ This covers the error instance only. The audit instance has no SQL persister, so - **Known before it starts, visible while it runs**. A dry run reports what will move and how long ServiceControl is closed, and every category transition is reported as it happens. - **Use existing functionality where possible**. Progress goes through custom checks and the activity feed, so no new client or screen is needed. -## Deliberately not built, and not currently planned +## Non-goals - **Zero downtime.** The required data is copied with ServiceControl closed, so there is a real, if short, outage. - **Reversible once ServiceControl opens.** Nothing copies SQL rows back to RavenDB, so once the host has served traffic there is no rollback of any kind. - **Steerable while running.** No pause or resume, and no abort command. Going back during the closed window means stopping and reconfiguring, and changing anything else means editing configuration and restarting. - **A general-purpose migration tool.** The source is always RavenDB and the target is always a ServiceControl EF Core persister, both at versions this build can read. -- **Custom migration UI via ServicePulse.** Custom checks and the event log will be used for progress reporting, but migration configuration and migration engine control will not be available via the UI. +- **Custom migration UI via ServicePulse.** Custom checks and the event log report progress, and migration configuration and control are not available in the UI. ## Supported migration scenarios @@ -57,7 +57,7 @@ The copier runs inside the ServiceControl host, so every row and every message b - The ServiceControl host needs network access to the RavenDB source, the SQL target and the body store simultaneously - Both RavenDB databases, primary and throughput, on one server or cluster - A SQL Server target must have Full-Text Search installed. `--setup` checks `SERVERPROPERTY('IsFullTextInstalled')` and fails if it is absent, because message search is not optional. A stock SQL Server container image does not include it. PostgreSQL needs nothing extra, since its index is a GIN over `to_tsvector` -- A managed target's transient failures are already survivable: retry on failure is on by default and there is no setting to turn it off +- A managed target's transient failures are survivable: retry on failure is on by default and there is no setting to turn it off **Not supported:** @@ -75,12 +75,12 @@ The copier runs inside the ServiceControl host, so every row and every message b 4. Run the [dry run](#dry-run). It reports what it resolved as a source, what each category holds, and an estimate of how long ServiceControl will be closed. Read [what the dry run reports](#dry-run) before booking an outage around its estimate. 5. Start ServiceControl (`MigrationMode=true`). 6. Every check runs before a single row moves. If one fails the host does not start and names which, having copied nothing, so a wrong database name or unconfigured body storage costs a restart rather than a half-finished migration. -7. The copying of [required data](#required) starts, with ServiceControl still closed. How long it takes depends on how many unresolved failures you have and how busy the last 7 days were, and the [dry run](#dry-run) gives you an estimate. If a required category halts, the host stays closed until you fix the cause and restart, or abandon that category. -8. ServiceControl opens, and whatever [optional data](#optional) they asked for is copied in the background while the instance runs normally. They can watch it from ServicePulse custom checks and events, but not steer it. +7. The copying of [required data](#required) starts, with ServiceControl still closed: the copy runs inside that same start, before the API begins listening and before any background service runs. How long it takes depends on how many unresolved failures you have and how busy the last 7 days were, and the [dry run](#dry-run) gives you an estimate. If a required category halts, the host stays closed until you fix the cause and restart, or abandon that category. +8. ServiceControl opens by itself the moment the required copy finishes, with no second restart to perform, and whatever [optional data](#optional) you asked for is copied in the background while the instance runs normally. You can watch it from ServicePulse custom checks and events, but not steer it. 9. You run the verification pass once the background job has completed, which reports row counts on both sides category by category, accounting for deliberate skips so a difference is explained rather than reported as a fault, then set `MigrationMode=false` and restart. Counts can differ in both directions without anything being wrong. SQL can hold more rows, because RavenDB keeps expiring rows the copier already took. SQL can also hold fewer, because once ServiceControl opens it sends pending integration events, removes group comments whose group has no failed messages left, and removes failed error imports once they are imported again. 10. RavenDB data can be removed. -- If `MigrationMode=false` is set while a selected category is still incomplete, the host refuses to start and names exactly what is outstanding. +- If `MigrationMode=false` is set while a selected category is still incomplete, the startup is gated: it refuses and names exactly what is outstanding, or, where the [free abort](#the-one-point-you-can-go-back) is still open, starts with a warning that says so. See [turning migration mode off is a gated startup too](#turning-migration-mode-off-is-a-gated-startup-too). - A category that ended *complete with errors* counts as complete and does not block, though its skipped count is printed so the loss is stated rather than silent. - An explicit override exists for a customer who has changed their mind and accepts leaving data behind. It marks the outstanding categories as abandoned, which is a deliberate end state rather than a failure, so the progress check settles and the guard stays armed for any later migration. - While a selected category is still unfinished, ServiceControl pauses its own clean-up: the retention sweep, the purge API, the heartbeat settings sync and throughput collection. Your SQL database grows until the copy finishes, and these restart on their own once it does. @@ -96,7 +96,7 @@ flowchart TB subgraph host["One ServiceControl host process, started with MigrationMode = true"] direction LR - raven["RavenDB persister
own AssemblyLoadContext
new read-only lifecycle"] + raven["RavenDB persister
own AssemblyLoadContext
read-only lifecycle"] engine["MigrationEngine
categories, throttle,
dry run, verification"] target["EF Core persister
SQL Server or PostgreSQL
own AssemblyLoadContext"] raven -->|"IMigrationSource"| engine @@ -104,7 +104,7 @@ flowchart TB end old[("Old RavenDB
read only, never written to")] - sql[("SQL Server or PostgreSQL
plus a new checkpoint table")] + sql[("SQL Server or PostgreSQL
plus the checkpoint table")] bodies[("Message body store
filesystem, Azure Blob or S3")] cfg --> host @@ -114,7 +114,7 @@ flowchart TB target --> bodies ``` -- **The engine and the host know no store.** They deal in categories, cursors and counts. The source maps a category to what it reads and describes itself as labelled facts, the target maps a category to where it writes and how to count it, and each contributes its own startup checks. RavenDB to SQL is the only pair built, and another pair, such as SQL to RavenDB, would add a source or a target without changing the engine. +- **The engine and the host know no store.** They deal in categories, cursors and counts. The source maps a category to what it reads and describes itself as labelled facts, the target maps a category to where it writes and how to count it, and each contributes its own startup checks. RavenDB to SQL is the only supported pair, and the engine does not depend on it. - **The source reads the instance's own RavenDB settings**, so an existing customer sets nothing new. Leave them in place when switching `PersistenceType`. - **Both persisters load into the same process**, each into its own `AssemblyLoadContext`. - **The engine references neither assembly.** It knows only `IMigrationSource` and `IMigrationTarget`, and treats the resume cursor as an opaque value it passes from one to the other, so it can be tested against fakes on either side. @@ -123,18 +123,23 @@ flowchart TB ```mermaid flowchart TB - A["Restart with MigrationMode = true"] --> B["Open the SQL target, exactly as today"] - B --> C["Open the old RavenDB, read only"] - C --> D{"All checks pass?"} - D -->|"No"| E["Host does not start.
Says which check failed.
Nothing has been copied."] - D -->|"Yes"| F["Copy what cannot be recreated.
How long depends on unresolved failures
and the last 7 days of events.
ServiceControl still closed."] - F -->|"A required category halts"| M["Host stays closed.
Fix the cause and restart,
or abandon the category."] - F -->|"Required data copied"| G["ServiceControl opens.
New failed messages go straight to SQL."] - G --> H["Copy the selected history in the background,
with a fixed pause between batches"] - H --> I["Verify row counts on both sides,
category by category"] - I --> J{"MigrationMode = false,
everything complete?"} - J -->|"No"| K["Host does not start.
Names what is outstanding.
An override exists."] - J -->|"Yes"| L["RavenDB is never opened again"] + A["ServiceControl starts on SQL"] --> M{"MigrationMode?"} + + M -->|"On"| B["Open the SQL target
and the RavenDB source, read only"] + B --> D{"All checks pass?"} + D -->|"No"| E["Host does not start.
Names the failed check.
Nothing is copied."] + D -->|"Yes"| F["Copy the required categories.
The API is not listening."] + F -->|"A required category halts"| T["Host stays closed.
Fix the cause and restart,
or abandon the category."] + F -->|"Required categories settled"| G["ServiceControl opens.
New failed messages go to SQL."] + G --> H["Copy the selected optional categories
in the background"] + + M -->|"Off"| N{"Any category
outstanding?"} + N -->|"No"| L["ServiceControl opens.
RavenDB is not opened."] + N -->|"Yes"| O{"Override set?"} + O -->|"Yes"| P["Mark each outstanding category abandoned,
log what it leaves behind,
and open."] + O -->|"No"| Q{"Has this instance
ever opened on SQL?"} + Q -->|"No"| R["Open with a warning: point PersistenceType
back at RavenDB, or carry on
and lose the way back."] + Q -->|"Yes"| S["Host does not start.
Names every outstanding category,
its counts, and every route out."] ``` **Checked before a single row moves:** @@ -147,6 +152,20 @@ flowchart TB - The selected categories are valid - `RetryHistoryDepth` is greater than zero. At zero or less, the first completed retry after the migration deletes the entire copied retry history, and no row count would ever show it +### Turning migration mode off is a gated startup too + +*The right-hand branch above is the half a customer meets last and expects least, so it is worth reading before the migration starts rather than at the end of one.* + +Every startup on a SQL Server or PostgreSQL instance looks at the checkpoint table before ServiceControl opens, whether `MigrationMode` is on or off. That is what stops a migration ending by accident, and it costs nothing on an instance with nothing outstanding: one that has never migrated holds no checkpoint rows, and one whose categories all settled has none left open. Both start normally. A RavenDB instance never reaches the gate. + +With `MigrationMode` off and at least one category still outstanding, one of three things happens, and each is said out loud at startup rather than discovered weeks later: + +- **The override is set.** Every outstanding category is recorded as abandoned, with its copied and skipped counts left as they are, and the host starts. Each one is logged saying what state it was in, how much it had copied, and that whatever it had not copied stays only in RavenDB. Abandoning is final: selecting that category in a later migration does not copy it again. +- **The override is not set, and this instance has never opened on SQL.** This is the [free abort](#the-one-point-you-can-go-back), so the host starts and warns rather than refusing. The warning names the two moves: stop now and point `PersistenceType` back at RavenDB, which discards the partial copy and costs nothing else, or carry on, which opens ServiceControl on a partly copied database and ends the free abort. It deliberately does not mention the override, because at that moment nothing is lost yet. +- **The override is not set, and this instance has already opened on SQL.** The host does not start. The error names every outstanding category, its state, its copied and skipped counts and its last error, and then the three routes out: restart with `MigrationMode=true` to let the copy finish or to resume a halted category once its cause is fixed, set the override to abandon what is outstanding and start without it, or, if RavenDB is already gone, abandon, because that is the only exit left. + +**Which is why the source stays until verification passes.** A customer who decommissions RavenDB while a category is outstanding has both doors shut: `MigrationMode=true` cannot start, because it opens the source before it copies anything, and `MigrationMode=false` refuses. Abandoning is then the only way to start the instance, and it is a real loss whose size is the counts in that message. + ## Data to be migrated (Categories) ### Required @@ -215,17 +234,17 @@ That window closes the moment ServiceControl opens. From then on new failed mess ## Reading from RavenDB -- A third RavenDB lifecycle opens the source: connect, check the version, stop. It never calls `DatabaseSetup.Execute`. +- A dedicated read-only RavenDB lifecycle opens the source: connect, check the version, stop. It never calls `DatabaseSetup.Execute`. - Both source databases must be on the same server or cluster (`LicensingDataStore.cs:35`). -- The source has to be at a ServiceControl version this build can read, and nothing in RavenDB records one today. The only version check that exists compares the RavenDB server version to the RavenDB client version, and runs only for an external source. So a marker is stamped into the database on upgrade, and a source without one, or one from a newer major version, is refused by name rather than misread. +- The source has to be at a ServiceControl version this build can read. ServiceControl stamps a version marker into the database on upgrade, because the RavenDB server version says nothing about which ServiceControl version wrote the data. A source without a marker, or one from a newer major version, is refused by name rather than misread. - Duration scales with distance to the source. The copier already holds the document from the stream, so each body costs **one** round trip rather than two, but it is one per message and they are not batched. Egress out of RavenDB Cloud is billed to the customer. See [batching and throttling](#batching-and-throttling). ## Writing to SQL -- A whole `FailedMessage` is written with its stored status intact. No existing caller does that, though the dialect upsert already accepts a status, so the gap is smaller than it looks. +- A whole `FailedMessage` is written with its stored status intact. - `UniqueMessageId` keeps its value, but converts type: the source holds a string and the target column is a `uniqueidentifier`. It is the primary key, the ServicePulse URL, the retry correlation key and the body lookup key at once. - `StatusChangedAt` is reconstructed from `@expires` for resolved and archived messages, which is the only place RavenDB sets it. Unresolved and retry-issued messages normally have no `@expires` (see [the exception](#what-does-not-come-across) for messages from version 6.18 or earlier), so the copier uses the newest processing attempt's timestamp. The column is `NOT NULL`, so it cannot be left empty, but the value is harmless for those two: the retention sweep only considers resolved and archived rows, so an unresolved message never ages out whatever is written here. -- Message bodies go through `IBodyStoragePersistence`, which owns the compression threshold and the choice of filesystem, Azure Blob or S3. The separate 102,400-byte inline threshold is not there: it lives on the ingestion path, so the copier has to apply it rather than inherit it. +- Message bodies go through `IBodyStoragePersistence`, which owns the compression threshold and the choice of filesystem, Azure Blob or S3. The copier applies the 102,400-byte inline threshold itself, because that threshold lives on the ingestion path rather than in `IBodyStoragePersistence`. - Throughput rows are written directly rather than through the collector, and the write sets each day's count rather than adding to it. Throughput is a required category, so it copies while ServiceControl is closed, before any collector has written to SQL. Setting is what makes the category safe to resume after a crash, where adding would double-count. Copying the rows is also what stops the audit and broker collectors re-gathering the same days when the host opens, because `LastCollectedDate` is derived from the newest throughput row rather than stored (`LicensingDataStore.cs:45`). The checkpoint is what stops a second pass overwriting days the collectors have written since. - Identifiers narrow on the way across, and the dry run counts every kind. What narrows, merges or cannot be stored at all is in [what does not come across](#what-does-not-come-across). @@ -380,5 +399,5 @@ A containerised instance runs all three as a one-off `docker run` of the same im ## Out of scope -- The audit instance, which has no EF Core persister at all, so a customer who finishes this migration is still running RavenDB for audit. This is stated up front under [Problem](#problem), because it changes whether the migration is worth doing at all +- The audit instance, which has no EF Core persister at all, so a customer who finishes this migration is still running RavenDB for audit. This is stated up front under [Purpose](#purpose), because it changes whether the migration is worth doing at all - The monitoring instance, which keeps its data in memory, so there is nothing to move diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260916125150_AddMigrationCheckpoints.Designer.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260916125150_AddMigrationCheckpoints.Designer.cs new file mode 100644 index 0000000000..a8a3dd5b3b --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260916125150_AddMigrationCheckpoints.Designer.cs @@ -0,0 +1,976 @@ +// +using System; +using System.Collections.Generic; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; +using ServiceControl.Persistence.EFCore.PostgreSql; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations +{ + [DbContext(typeof(PostgreSqlServiceControlDbContext))] + [Migration("20260916125150_AddMigrationCheckpoints")] + partial class AddMigrationCheckpoints + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ArchiveOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("request_id"); + + b.Property("ArchiveType") + .HasColumnType("integer") + .HasColumnName("archive_type"); + + b.Property("OperationType") + .HasColumnType("integer") + .HasColumnName("operation_type"); + + b.Property("CurrentBatch") + .HasColumnType("integer") + .HasColumnName("current_batch"); + + b.Property("GroupName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("group_name"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_id"); + + b.Property("InitiatedByName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_name"); + + b.Property("NumberOfBatches") + .HasColumnType("integer") + .HasColumnName("number_of_batches"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("operation_id"); + + b.Property("Started") + .HasColumnType("timestamp with time zone") + .HasColumnName("started"); + + b.Property("TotalNumberOfMessages") + .HasColumnType("integer") + .HasColumnName("total_number_of_messages"); + + b.HasKey("RequestId", "ArchiveType", "OperationType") + .HasName("pk_archive_operations"); + + b.HasIndex("Started") + .HasDatabaseName("ix_archive_operations_started"); + + b.ToTable("archive_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Category") + .IsRequired() + .HasColumnType("text") + .HasColumnName("category"); + + b.Property("CustomCheckId") + .IsRequired() + .HasColumnType("text") + .HasColumnName("custom_check_id"); + + b.Property("FailureReason") + .HasColumnType("text") + .HasColumnName("failure_reason"); + + b.Property("OriginatingEndpointHost") + .IsRequired() + .HasColumnType("text") + .HasColumnName("originating_endpoint_host"); + + b.Property("OriginatingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("originating_endpoint_host_id"); + + b.Property("OriginatingEndpointName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("originating_endpoint_name"); + + b.Property("ReportedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("reported_at"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("Id") + .HasName("pk_custom_checks"); + + b.HasIndex("ReportedAt") + .HasDatabaseName("ix_custom_checks_reported_at"); + + b.HasIndex("Status", "ReportedAt") + .HasDatabaseName("ix_custom_checks_status_reported_at"); + + b.ToTable("custom_checks", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EndpointSettingsEntity", b => + { + b.Property("Name") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("name"); + + b.Property("TrackInstances") + .HasColumnType("boolean") + .HasColumnName("track_instances"); + + b.HasKey("Name") + .HasName("pk_endpoint_settings"); + + b.ToTable("endpoint_settings", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EventLogItemEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Category") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("category"); + + b.Property("Description") + .IsRequired() + .HasColumnType("text") + .HasColumnName("description"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("event_type"); + + b.Property("RaisedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("raised_at"); + + b.PrimitiveCollection>("RelatedTo") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("related_to"); + + b.Property("Severity") + .HasColumnType("integer") + .HasColumnName("severity"); + + b.HasKey("Id") + .HasName("pk_event_log_items"); + + b.HasIndex("RaisedAt", "Id") + .IsDescending() + .HasDatabaseName("ix_event_log_items_raised_at_id"); + + b.ToTable("event_log_items", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ExternalIntegrationDispatchRequestEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("DispatchContextJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("dispatch_context_json"); + + b.Property("DispatchContextTypeName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("dispatch_context_type_name"); + + b.HasKey("Id") + .HasName("pk_external_integration_dispatch_requests"); + + b.ToTable("external_integration_dispatch_requests", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("Body") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("body"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("text") + .HasColumnName("exception_info"); + + b.Property("FailedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("failed_at"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_error_imports"); + + b.HasIndex("FailedAt") + .HasDatabaseName("ix_failed_error_imports_failed_at"); + + b.ToTable("failed_error_imports", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEditEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("EditId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("edit_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_message_edits"); + + b.HasIndex("EditId") + .HasDatabaseName("ix_failed_message_edits_edit_id"); + + b.ToTable("failed_message_edits", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("body_content_type"); + + b.Property("BodySize") + .HasColumnType("integer") + .HasColumnName("body_size"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("BodyText") + .HasColumnType("text") + .HasColumnName("body_text"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("conversation_id"); + + b.Property("ExceptionMessage") + .HasColumnType("text") + .HasColumnName("exception_message"); + + b.Property("ExceptionType") + .HasColumnType("text") + .HasColumnName("exception_type"); + + b.Property("FailingEndpointAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("failing_endpoint_address"); + + b.Property("FirstTimeOfFailure") + .HasColumnType("timestamp with time zone") + .HasColumnName("first_time_of_failure"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("IsSystemMessage") + .HasColumnType("boolean") + .HasColumnName("is_system_message"); + + b.Property("LastAttemptedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_attempted_at"); + + b.Property("LastModified") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_modified"); + + b.Property("LastTimeOfFailure") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_time_of_failure"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.Property("MessageType") + .HasColumnType("text") + .HasColumnName("message_type"); + + b.Property("NumberOfProcessingAttempts") + .HasColumnType("integer") + .HasColumnName("number_of_processing_attempts"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_host"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("receiving_endpoint_host_id"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_name"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_host"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("sending_endpoint_host_id"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_name"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.Property("StatusChangedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("status_changed_at"); + + b.Property("TimeSent") + .HasColumnType("timestamp with time zone") + .HasColumnName("time_sent"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_messages"); + + b.HasIndex("ConversationId") + .HasDatabaseName("ix_failed_messages_conversation_id"); + + b.HasIndex("FailingEndpointAddress") + .HasDatabaseName("ix_failed_messages_failing_endpoint_address"); + + b.HasIndex("ReceivingEndpointName") + .HasDatabaseName("ix_failed_messages_receiving_endpoint_name"); + + b.HasIndex("StatusChangedAt") + .HasDatabaseName("ix_failed_messages_status_changed_at") + .HasFilter("status IN (2, 4)"); + + b.HasIndex("TimeSent") + .HasDatabaseName("ix_failed_messages_time_sent"); + + b.HasIndex("Status", "LastModified") + .HasDatabaseName("ix_failed_messages_status_last_modified"); + + b.ToTable("failed_messages", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.Property("FailedMessageUniqueId") + .HasColumnType("uuid") + .HasColumnName("failed_message_unique_id"); + + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("group_id"); + + b.Property("Title") + .IsRequired() + .HasColumnType("text") + .HasColumnName("title"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("character varying(255)") + .HasColumnName("type"); + + b.HasKey("FailedMessageUniqueId", "GroupId") + .HasName("pk_failed_message_groups"); + + b.HasIndex("GroupId") + .HasDatabaseName("ix_failed_message_groups_group_id"); + + b.HasIndex("Type", "GroupId") + .HasDatabaseName("ix_failed_message_groups_type_group_id"); + + b.ToTable("failed_message_groups", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageRetryEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("RetryBatchId") + .HasColumnType("uuid") + .HasColumnName("retry_batch_id"); + + b.Property("StageAttempts") + .HasColumnType("integer") + .HasColumnName("stage_attempts"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_message_retries"); + + b.HasIndex("RetryBatchId") + .HasDatabaseName("ix_failed_message_retries_retry_batch_id"); + + b.ToTable("failed_message_retries", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.GroupCommentEntity", b => + { + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("group_id"); + + b.Property("Comment") + .IsRequired() + .HasColumnType("text") + .HasColumnName("comment"); + + b.HasKey("GroupId") + .HasName("pk_group_comments"); + + b.ToTable("group_comments", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.HistoricRetryOperationEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CompletionTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("completion_time"); + + b.Property("Failed") + .HasColumnType("boolean") + .HasColumnName("failed"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("request_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.HasKey("Id") + .HasName("pk_historic_retry_operations"); + + b.HasIndex("CompletionTime", "Id") + .IsDescending() + .HasDatabaseName("ix_historic_retry_operations_completion_time_id"); + + b.ToTable("historic_retry_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.KnownEndpointEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("host"); + + b.Property("HostId") + .HasColumnType("uuid") + .HasColumnName("host_id"); + + b.Property("Monitored") + .HasColumnType("boolean") + .HasColumnName("monitored"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("name"); + + b.HasKey("Id") + .HasName("pk_known_endpoints"); + + b.ToTable("known_endpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("normalized_name"); + + b.Property("ThroughputSource") + .HasColumnType("integer") + .HasColumnName("throughput_source"); + + b.PrimitiveCollection>("EndpointIndicators") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("endpoint_indicators"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("name"); + + b.Property("NormalizedSanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("normalized_sanitized_name"); + + b.Property("SanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sanitized_name"); + + b.Property("Scope") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("scope"); + + b.Property("UserIndicator") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("user_indicator"); + + b.HasKey("NormalizedName", "ThroughputSource") + .HasName("pk_licensing_endpoints"); + + b.HasIndex("NormalizedSanitizedName") + .HasDatabaseName("ix_licensing_endpoints_normalized_sanitized_name"); + + b.ToTable("licensing_endpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("normalized_name"); + + b.Property("ThroughputSource") + .HasColumnType("integer") + .HasColumnName("throughput_source"); + + b.Property("DateUtc") + .HasColumnType("date") + .HasColumnName("date_utc"); + + b.Property("MessageCount") + .HasColumnType("bigint") + .HasColumnName("message_count"); + + b.HasKey("NormalizedName", "ThroughputSource", "DateUtc") + .HasName("pk_licensing_endpoint_throughput"); + + b.HasIndex("DateUtc") + .HasDatabaseName("ix_licensing_endpoint_throughput_date_utc"); + + b.ToTable("licensing_endpoint_throughput", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MessageRedirectEntity", b => + { + b.Property("FromPhysicalAddress") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("from_physical_address"); + + b.Property("LastModified") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_modified"); + + b.Property("ToPhysicalAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("to_physical_address"); + + b.HasKey("FromPhysicalAddress") + .HasName("pk_message_redirects"); + + b.ToTable("message_redirects", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MigrationCheckpointEntity", b => + { + b.Property("CategoryId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("category_id"); + + b.Property("AlreadyPresentCount") + .HasColumnType("bigint") + .HasColumnName("already_present_count"); + + b.Property("CopiedCount") + .HasColumnType("bigint") + .HasColumnName("copied_count"); + + b.Property("Cursor") + .HasColumnType("text") + .HasColumnName("cursor"); + + b.Property("LastError") + .HasColumnType("text") + .HasColumnName("last_error"); + + b.Property("LastProgressAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_progress_at"); + + b.Property("SettledAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("settled_at"); + + b.Property("SkipReasons") + .HasColumnType("text") + .HasColumnName("skip_reasons"); + + b.Property("SkippedCount") + .HasColumnType("bigint") + .HasColumnName("skipped_count"); + + b.Property("SourceTotal") + .HasColumnType("bigint") + .HasColumnName("source_total"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("State") + .HasColumnType("integer") + .HasColumnName("state"); + + b.Property("Version") + .IsConcurrencyToken() + .HasColumnType("bigint") + .HasColumnName("version"); + + b.HasKey("CategoryId") + .HasName("pk_migration_checkpoints"); + + b.ToTable("migration_checkpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Classifier") + .HasColumnType("text") + .HasColumnName("classifier"); + + b.Property("Context") + .HasColumnType("text") + .HasColumnName("context"); + + b.Property("InitialBatchSize") + .HasColumnType("integer") + .HasColumnName("initial_batch_size"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_id"); + + b.Property("InitiatedByName") + .HasColumnType("text") + .HasColumnName("initiated_by_name"); + + b.Property("Last") + .HasColumnType("timestamp with time zone") + .HasColumnName("last"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("operation_id"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("request_id"); + + b.Property("RetrySessionId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("retry_session_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("StagingId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("staging_id"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("Id") + .HasName("pk_retry_batches"); + + b.HasIndex("Status", "RetrySessionId") + .HasDatabaseName("ix_retry_batches_status_retry_session_id"); + + b.ToTable("retry_batches", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchNowForwardingEntity", b => + { + b.Property("Id") + .HasColumnType("integer") + .HasColumnName("id"); + + b.Property("RetryBatchId") + .HasColumnType("uuid") + .HasColumnName("retry_batch_id"); + + b.HasKey("Id") + .HasName("pk_retry_batch_now_forwarding"); + + b.ToTable("retry_batch_now_forwarding", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => + { + b.Property("Key") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("key"); + + b.Property("Value") + .IsRequired() + .HasColumnType("text") + .HasColumnName("value"); + + b.HasKey("Key") + .HasName("pk_settings"); + + b.ToTable("settings", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SubscriptionEntity", b => + { + b.Property("MessageType") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("message_type"); + + b.Property("TransportAddress") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("transport_address"); + + b.Property("Endpoint") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("endpoint"); + + b.HasKey("MessageType", "TransportAddress") + .HasName("pk_subscriptions"); + + b.ToTable("subscriptions", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.UnacknowledgedRetryOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("request_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("Classifier") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("classifier"); + + b.Property("CompletionTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("completion_time"); + + b.Property("Failed") + .HasColumnType("boolean") + .HasColumnName("failed"); + + b.Property("Last") + .HasColumnType("timestamp with time zone") + .HasColumnName("last"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.HasKey("RequestId", "RetryType") + .HasName("pk_unacknowledged_retry_operations"); + + b.ToTable("unacknowledged_retry_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", null) + .WithMany() + .HasForeignKey("FailedMessageUniqueId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_failed_message_groups_failed_messages_failed_message_unique"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", null) + .WithMany() + .HasForeignKey("NormalizedName", "ThroughputSource") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_licensing_endpoint_throughput_licensing_endpoints_normalize"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260916125150_AddMigrationCheckpoints.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260916125150_AddMigrationCheckpoints.cs new file mode 100644 index 0000000000..b2b97c730a --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260916125150_AddMigrationCheckpoints.cs @@ -0,0 +1,45 @@ +using System; +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations +{ + /// + public partial class AddMigrationCheckpoints : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateTable( + name: "migration_checkpoints", + columns: table => new + { + category_id = table.Column(type: "character varying(450)", maxLength: 450, nullable: false), + state = table.Column(type: "integer", nullable: false), + cursor = table.Column(type: "text", nullable: true), + copied_count = table.Column(type: "bigint", nullable: false), + skipped_count = table.Column(type: "bigint", nullable: false), + source_total = table.Column(type: "bigint", nullable: true), + skip_reasons = table.Column(type: "text", nullable: true), + started_at = table.Column(type: "timestamp with time zone", nullable: true), + last_progress_at = table.Column(type: "timestamp with time zone", nullable: true), + settled_at = table.Column(type: "timestamp with time zone", nullable: true), + last_error = table.Column(type: "text", nullable: true), + already_present_count = table.Column(type: "bigint", nullable: false), + version = table.Column(type: "bigint", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("pk_migration_checkpoints", x => x.category_id); + }); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropTable( + name: "migration_checkpoints"); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs index 06b1a9b639..cfc74e1483 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs @@ -715,6 +715,68 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("message_redirects", (string)null); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MigrationCheckpointEntity", b => + { + b.Property("CategoryId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("category_id"); + + b.Property("AlreadyPresentCount") + .HasColumnType("bigint") + .HasColumnName("already_present_count"); + + b.Property("CopiedCount") + .HasColumnType("bigint") + .HasColumnName("copied_count"); + + b.Property("Cursor") + .HasColumnType("text") + .HasColumnName("cursor"); + + b.Property("LastError") + .HasColumnType("text") + .HasColumnName("last_error"); + + b.Property("LastProgressAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_progress_at"); + + b.Property("SettledAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("settled_at"); + + b.Property("SkipReasons") + .HasColumnType("text") + .HasColumnName("skip_reasons"); + + b.Property("SkippedCount") + .HasColumnType("bigint") + .HasColumnName("skipped_count"); + + b.Property("SourceTotal") + .HasColumnType("bigint") + .HasColumnName("source_total"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("State") + .HasColumnType("integer") + .HasColumnName("state"); + + b.Property("Version") + .IsConcurrencyToken() + .HasColumnType("bigint") + .HasColumnName("version"); + + b.HasKey("CategoryId") + .HasName("pk_migration_checkpoints"); + + b.ToTable("migration_checkpoints", (string)null); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => { b.Property("Id") diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260916125144_AddMigrationCheckpoints.Designer.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260916125144_AddMigrationCheckpoints.Designer.cs new file mode 100644 index 0000000000..5dfcc173b9 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260916125144_AddMigrationCheckpoints.Designer.cs @@ -0,0 +1,782 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ServiceControl.Persistence.EFCore.SqlServer; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations +{ + [DbContext(typeof(SqlServerServiceControlDbContext))] + [Migration("20260916125144_AddMigrationCheckpoints")] + partial class AddMigrationCheckpoints + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ArchiveOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("ArchiveType") + .HasColumnType("int"); + + b.Property("OperationType") + .HasColumnType("int"); + + b.Property("CurrentBatch") + .HasColumnType("int"); + + b.Property("GroupName") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("InitiatedByName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("NumberOfBatches") + .HasColumnType("int"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Started") + .HasColumnType("datetime2"); + + b.Property("TotalNumberOfMessages") + .HasColumnType("int"); + + b.HasKey("RequestId", "ArchiveType", "OperationType"); + + b.HasIndex("Started"); + + b.ToTable("ArchiveOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Category") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CustomCheckId") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailureReason") + .HasColumnType("nvarchar(max)"); + + b.Property("OriginatingEndpointHost") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("OriginatingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginatingEndpointName") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ReportedAt") + .HasColumnType("datetime2"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ReportedAt"); + + b.HasIndex("Status", "ReportedAt"); + + b.ToTable("CustomChecks"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EndpointSettingsEntity", b => + { + b.Property("Name") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("TrackInstances") + .HasColumnType("bit"); + + b.HasKey("Name"); + + b.ToTable("EndpointSettings"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EventLogItemEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Category") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Description") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RaisedAt") + .HasColumnType("datetime2"); + + b.PrimitiveCollection("RelatedTo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Severity") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("RaisedAt", "Id") + .IsDescending(); + + b.ToTable("EventLogItems"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ExternalIntegrationDispatchRequestEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DispatchContextJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DispatchContextTypeName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("Id"); + + b.ToTable("ExternalIntegrationDispatchRequests"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("Body") + .IsRequired() + .HasColumnType("varbinary(max)"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailedAt") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("FailedAt"); + + b.ToTable("FailedErrorImports"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEditEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("EditId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("EditId"); + + b.ToTable("FailedMessageEdits"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("BodySize") + .HasColumnType("int"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("BodyText") + .HasColumnType("nvarchar(max)"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ExceptionMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("ExceptionType") + .HasColumnType("nvarchar(max)"); + + b.Property("FailingEndpointAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("FirstTimeOfFailure") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("IsSystemMessage") + .HasColumnType("bit"); + + b.Property("LastAttemptedAt") + .HasColumnType("datetime2"); + + b.Property("LastModified") + .HasColumnType("datetime2"); + + b.Property("LastTimeOfFailure") + .HasColumnType("datetime2"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("MessageType") + .HasColumnType("nvarchar(max)"); + + b.Property("NumberOfProcessingAttempts") + .HasColumnType("int"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Status") + .HasColumnType("int"); + + b.Property("StatusChangedAt") + .HasColumnType("datetime2"); + + b.Property("TimeSent") + .HasColumnType("datetime2"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("ConversationId"); + + b.HasIndex("FailingEndpointAddress"); + + b.HasIndex("ReceivingEndpointName"); + + b.HasIndex("StatusChangedAt") + .HasFilter("[Status] IN (2, 4)"); + + b.HasIndex("TimeSent"); + + b.HasIndex("Status", "LastModified"); + + b.ToTable("FailedMessages"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.Property("FailedMessageUniqueId") + .HasColumnType("uniqueidentifier"); + + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Title") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("nvarchar(255)"); + + b.HasKey("FailedMessageUniqueId", "GroupId"); + + b.HasIndex("GroupId"); + + b.HasIndex("Type", "GroupId"); + + b.ToTable("FailedMessageGroups"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageRetryEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("RetryBatchId") + .HasColumnType("uniqueidentifier"); + + b.Property("StageAttempts") + .HasColumnType("int"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("RetryBatchId"); + + b.ToTable("FailedMessageRetries"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.GroupCommentEntity", b => + { + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Comment") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.HasKey("GroupId"); + + b.ToTable("GroupComments"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.HistoricRetryOperationEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletionTime") + .HasColumnType("datetime2"); + + b.Property("Failed") + .HasColumnType("bit"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.HasKey("Id"); + + b.HasIndex("CompletionTime", "Id") + .IsDescending(); + + b.ToTable("HistoricRetryOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.KnownEndpointEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("HostId") + .HasColumnType("uniqueidentifier"); + + b.Property("Monitored") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("Id"); + + b.ToTable("KnownEndpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("ThroughputSource") + .HasColumnType("int"); + + b.PrimitiveCollection("EndpointIndicators") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("NormalizedSanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Scope") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("UserIndicator") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("NormalizedName", "ThroughputSource"); + + b.HasIndex("NormalizedSanitizedName"); + + b.ToTable("LicensingEndpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("ThroughputSource") + .HasColumnType("int"); + + b.Property("DateUtc") + .HasColumnType("date"); + + b.Property("MessageCount") + .HasColumnType("bigint"); + + b.HasKey("NormalizedName", "ThroughputSource", "DateUtc"); + + b.HasIndex("DateUtc"); + + b.ToTable("LicensingEndpointThroughput"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MessageRedirectEntity", b => + { + b.Property("FromPhysicalAddress") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("LastModified") + .HasColumnType("datetime2"); + + b.Property("ToPhysicalAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("FromPhysicalAddress"); + + b.ToTable("MessageRedirects"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MigrationCheckpointEntity", b => + { + b.Property("CategoryId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("AlreadyPresentCount") + .HasColumnType("bigint"); + + b.Property("CopiedCount") + .HasColumnType("bigint"); + + b.Property("Cursor") + .HasColumnType("nvarchar(max)"); + + b.Property("LastError") + .HasColumnType("nvarchar(max)"); + + b.Property("LastProgressAt") + .HasColumnType("datetime2"); + + b.Property("SettledAt") + .HasColumnType("datetime2"); + + b.Property("SkipReasons") + .HasColumnType("nvarchar(max)"); + + b.Property("SkippedCount") + .HasColumnType("bigint"); + + b.Property("SourceTotal") + .HasColumnType("bigint"); + + b.Property("StartedAt") + .HasColumnType("datetime2"); + + b.Property("State") + .HasColumnType("int"); + + b.Property("Version") + .IsConcurrencyToken() + .HasColumnType("bigint"); + + b.HasKey("CategoryId"); + + b.ToTable("MigrationCheckpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Classifier") + .HasColumnType("nvarchar(max)"); + + b.Property("Context") + .HasColumnType("nvarchar(max)"); + + b.Property("InitialBatchSize") + .HasColumnType("int"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("InitiatedByName") + .HasColumnType("nvarchar(max)"); + + b.Property("Last") + .HasColumnType("datetime2"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RetrySessionId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("StagingId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Status", "RetrySessionId"); + + b.ToTable("RetryBatches"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchNowForwardingEntity", b => + { + b.Property("Id") + .HasColumnType("int"); + + b.Property("RetryBatchId") + .HasColumnType("uniqueidentifier"); + + b.HasKey("Id"); + + b.ToTable("RetryBatchNowForwarding"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => + { + b.Property("Key") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Value") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.HasKey("Key"); + + b.ToTable("Settings"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SubscriptionEntity", b => + { + b.Property("MessageType") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TransportAddress") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Endpoint") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("MessageType", "TransportAddress"); + + b.ToTable("Subscriptions"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.UnacknowledgedRetryOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("Classifier") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("CompletionTime") + .HasColumnType("datetime2"); + + b.Property("Failed") + .HasColumnType("bit"); + + b.Property("Last") + .HasColumnType("datetime2"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.HasKey("RequestId", "RetryType"); + + b.ToTable("UnacknowledgedRetryOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", null) + .WithMany() + .HasForeignKey("FailedMessageUniqueId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", null) + .WithMany() + .HasForeignKey("NormalizedName", "ThroughputSource") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260916125144_AddMigrationCheckpoints.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260916125144_AddMigrationCheckpoints.cs new file mode 100644 index 0000000000..b19e73d767 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260916125144_AddMigrationCheckpoints.cs @@ -0,0 +1,45 @@ +using System; +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations +{ + /// + public partial class AddMigrationCheckpoints : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateTable( + name: "MigrationCheckpoints", + columns: table => new + { + CategoryId = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: false), + State = table.Column(type: "int", nullable: false), + Cursor = table.Column(type: "nvarchar(max)", nullable: true), + CopiedCount = table.Column(type: "bigint", nullable: false), + SkippedCount = table.Column(type: "bigint", nullable: false), + SourceTotal = table.Column(type: "bigint", nullable: true), + SkipReasons = table.Column(type: "nvarchar(max)", nullable: true), + StartedAt = table.Column(type: "datetime2", nullable: true), + LastProgressAt = table.Column(type: "datetime2", nullable: true), + SettledAt = table.Column(type: "datetime2", nullable: true), + LastError = table.Column(type: "nvarchar(max)", nullable: true), + AlreadyPresentCount = table.Column(type: "bigint", nullable: false), + Version = table.Column(type: "bigint", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("PK_MigrationCheckpoints", x => x.CategoryId); + }); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropTable( + name: "MigrationCheckpoints"); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs index b9e41a3681..e8a1ad22cd 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs @@ -572,6 +572,54 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("MessageRedirects"); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MigrationCheckpointEntity", b => + { + b.Property("CategoryId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("AlreadyPresentCount") + .HasColumnType("bigint"); + + b.Property("CopiedCount") + .HasColumnType("bigint"); + + b.Property("Cursor") + .HasColumnType("nvarchar(max)"); + + b.Property("LastError") + .HasColumnType("nvarchar(max)"); + + b.Property("LastProgressAt") + .HasColumnType("datetime2"); + + b.Property("SettledAt") + .HasColumnType("datetime2"); + + b.Property("SkipReasons") + .HasColumnType("nvarchar(max)"); + + b.Property("SkippedCount") + .HasColumnType("bigint"); + + b.Property("SourceTotal") + .HasColumnType("bigint"); + + b.Property("StartedAt") + .HasColumnType("datetime2"); + + b.Property("State") + .HasColumnType("int"); + + b.Property("Version") + .IsConcurrencyToken() + .HasColumnType("bigint"); + + b.HasKey("CategoryId"); + + b.ToTable("MigrationCheckpoints"); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => { b.Property("Id") diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs index 4b76e5add7..6b61a58b40 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs @@ -13,6 +13,7 @@ namespace ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.EFCore.Infrastructure.Metrics; +using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.MessageRedirects; using ServiceControl.Persistence.Recoverability; using ServiceControl.Persistence.UnitOfWork; @@ -38,6 +39,8 @@ protected static void RegisterDataStores(IServiceCollection services, EFPersiste services.AddSingleton(p => p.GetRequiredService()); services.AddHostedService(p => p.GetRequiredService()); + services.AddSingleton(); + if (settings.RunRetentionSweep) { services.AddSingleton(); diff --git a/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs b/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs index 3717c6a7f3..3884ce1e95 100644 --- a/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs +++ b/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs @@ -34,6 +34,7 @@ public abstract class ServiceControlDbContext(DbContextOptions options) : DbCont public DbSet LicensingEndpoints { get; set; } public DbSet LicensingEndpointThroughput { get; set; } public DbSet ExternalIntegrationDispatchRequests { get; set; } + public DbSet MigrationCheckpoints { get; set; } protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) => optionsBuilder.EnableDetailedErrors(); @@ -68,6 +69,7 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) modelBuilder.ApplyConfiguration(new LicensingEndpointConfiguration()); modelBuilder.ApplyConfiguration(new LicensingEndpointThroughputConfiguration()); modelBuilder.ApplyConfiguration(new ExternalIntegrationDispatchRequestConfiguration()); + modelBuilder.ApplyConfiguration(new MigrationCheckpointConfiguration()); } public abstract bool IsDuplicateKeyException(DbUpdateException exception); diff --git a/src/ServiceControl.Persistence.EFCore/Entities/MigrationCheckpointEntity.cs b/src/ServiceControl.Persistence.EFCore/Entities/MigrationCheckpointEntity.cs new file mode 100644 index 0000000000..9f82eb4335 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Entities/MigrationCheckpointEntity.cs @@ -0,0 +1,26 @@ +namespace ServiceControl.Persistence.EFCore.Entities; + +using ServiceControl.Persistence.DataMigration; + +public class MigrationCheckpointEntity +{ + public required string CategoryId { get; set; } + public MigrationCategoryState State { get; set; } + public string? Cursor { get; set; } + public long CopiedCount { get; set; } + public long SkippedCount { get; set; } + public long? SourceTotal { get; set; } + public IReadOnlyDictionary? SkipReasons { get; set; } + public DateTime? StartedAt { get; set; } + public DateTime? LastProgressAt { get; set; } + public DateTime? SettledAt { get; set; } + public string? LastError { get; set; } + public long AlreadyPresentCount { get; set; } + public long Version { get; set; } + + internal MigrationCheckpoint ToCheckpoint() => new( + CategoryId, State, Cursor, + CopiedCount, SkippedCount, SourceTotal, SkipReasons, + StartedAt, LastProgressAt, SettledAt, LastError, + AlreadyPresentCount, Version); +} \ No newline at end of file diff --git a/src/ServiceControl.Persistence.EFCore/EntityConfigurations/MigrationCheckpointConfiguration.cs b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/MigrationCheckpointConfiguration.cs new file mode 100644 index 0000000000..9cc35d77f5 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/MigrationCheckpointConfiguration.cs @@ -0,0 +1,38 @@ +namespace ServiceControl.Persistence.EFCore.EntityConfigurations; + +using System.Text.Json; +using Entities; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.ChangeTracking; +using Microsoft.EntityFrameworkCore.Metadata.Builders; +using ServiceControl.Persistence.DataMigration; + +class MigrationCheckpointConfiguration : IEntityTypeConfiguration +{ + public void Configure(EntityTypeBuilder builder) + { + builder.HasKey(e => e.CategoryId); + builder.Property(e => e.CategoryId).HasMaxLength(ColumnLengths.ShortTextLength).ValueGeneratedNever(); + builder.Property(e => e.Version).IsConcurrencyToken(); + + // Null never reaches a converter: EF Core stores NULL for it, so a checkpoint with no skips has no JSON. + builder.Property(e => e.SkipReasons).HasConversion( + reasons => ToJson(reasons!), + json => FromJson(json), + new ValueComparer>( + (left, right) => left!.Count == right!.Count && !left.Except(right).Any(), + reasons => reasons.Aggregate(0, (hash, entry) => HashCode.Combine(hash, entry.Key, entry.Value)), + reasons => reasons.ToDictionary(entry => entry.Key, entry => entry.Value))); + } + + // Keyed by name, so a stored count keeps its meaning whatever order the enum's members are in. + static string ToJson(IReadOnlyDictionary reasons) => + JsonSerializer.Serialize(reasons.ToDictionary(entry => entry.Key.ToString(), entry => entry.Value)); + + // A name this build has no member for buckets into Unknown rather than throwing, because ReadAll is what + // decides whether the host may start; grouped because several unknown names collapse onto the one member. + static IReadOnlyDictionary FromJson(string json) => + JsonSerializer.Deserialize>(json)! + .GroupBy(entry => Enum.TryParse(entry.Key, out var reason) && Enum.IsDefined(reason) ? reason : MigrationSkipReason.Unknown) + .ToDictionary(group => group.Key, group => group.Sum(entry => entry.Value)); +} \ No newline at end of file diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/EFMigrationCheckpointStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/EFMigrationCheckpointStore.cs new file mode 100644 index 0000000000..3f343f44e3 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/EFMigrationCheckpointStore.cs @@ -0,0 +1,27 @@ +namespace ServiceControl.Persistence.EFCore.Implementation; + +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using ServiceControl.Persistence.DataMigration; + +// Every method opens its own scope and context through DataStoreBase, so none of them can join a +// caller's transaction. Inside one, call ServiceControlDbContext.UpsertCheckpoint instead. +public class EFMigrationCheckpointStore(IServiceScopeFactory scopeFactory) : DataStoreBase(scopeFactory), IMigrationCheckpointStore +{ + public Task> ReadAll(CancellationToken cancellationToken = default) => + ExecuteWithDbContext(async (dbContext, token) => + { + var entities = await dbContext.MigrationCheckpoints.AsNoTracking().ToListAsync(token); + return (IReadOnlyList)[.. entities.Select(entity => entity.ToCheckpoint())]; + }, cancellationToken); + + public Task Read(string categoryId, CancellationToken cancellationToken = default) => + ExecuteWithDbContext(async (dbContext, token) => + { + var entity = await dbContext.MigrationCheckpoints.AsNoTracking().SingleOrDefaultAsync(e => e.CategoryId == categoryId, token); + return entity?.ToCheckpoint(); + }, cancellationToken); + + public Task Upsert(MigrationCheckpoint checkpoint, CancellationToken cancellationToken = default) => + ExecuteWithDbContext((dbContext, token) => dbContext.UpsertCheckpoint(checkpoint, token), cancellationToken); +} \ No newline at end of file diff --git a/src/ServiceControl.Persistence.EFCore/MigrationCheckpointExtensions.cs b/src/ServiceControl.Persistence.EFCore/MigrationCheckpointExtensions.cs new file mode 100644 index 0000000000..e497fa1794 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/MigrationCheckpointExtensions.cs @@ -0,0 +1,95 @@ +namespace ServiceControl.Persistence.EFCore; + +using DbContexts; +using Entities; +using Microsoft.EntityFrameworkCore; +using ServiceControl.Persistence.DataMigration; + +public static class MigrationCheckpointExtensions +{ + /// + /// Saves a migration checkpoint on the caller's own context, so it commits with that context's transaction + /// and flushes whatever else that context is tracking, and returns it carrying the version the save landed on. + /// Throws when the stored row no longer holds + /// . + /// + public static async Task UpsertCheckpoint(this ServiceControlDbContext dbContext, MigrationCheckpoint checkpoint, CancellationToken cancellationToken = default) + { + try + { + return await dbContext.UpsertCheckpointCore(checkpoint, cancellationToken); + } + catch (DbUpdateConcurrencyException e) + { + throw new MigrationCheckpointConflictException( + $"Checkpoint {checkpoint.CategoryId} was saved from version {checkpoint.Version}, which the stored row no longer holds. Another writer advanced it, or a retried save is meeting its own committed write after a lost acknowledgement.", e); + } + } + + // Deliberately not UpsertAsync: its duplicate-key fallback reloads and saves again, which inside a + // caller-opened PostgreSQL transaction runs after the failed insert aborted it and fails with 25P02. + static async Task UpsertCheckpointCore(this ServiceControlDbContext dbContext, MigrationCheckpoint checkpoint, CancellationToken cancellationToken) + { + var entity = await dbContext.FindAsync([checkpoint.CategoryId], cancellationToken: cancellationToken); + + if (entity is null) + { + if (checkpoint.Version != 0) + { + throw new MigrationCheckpointConflictException($"Checkpoint {checkpoint.CategoryId} was saved from version {checkpoint.Version}, but no row exists for it."); + } + + var inserted = Insert(checkpoint); + dbContext.Add(inserted); + + try + { + await dbContext.SaveChangesAsync(cancellationToken); + } + catch (DbUpdateException e) when (dbContext.IsDuplicateKeyException(e)) + { + // Another writer created this category's first checkpoint between the read and the insert. Not + // retried: PostgreSQL has already aborted the caller's transaction, so a retry cannot run. + throw new MigrationCheckpointConflictException( + $"Checkpoint {checkpoint.CategoryId} was saved as a first write, but another writer had already created it.", e); + } + + return inserted.ToCheckpoint(); + } + + // The expected version goes in the UPDATE's WHERE clause, so a save from an out-of-date copy matches no row. + dbContext.Entry(entity).Property(e => e.Version).OriginalValue = checkpoint.Version; + entity.Version = checkpoint.Version + 1; + Apply(checkpoint, entity); + + await dbContext.SaveChangesAsync(cancellationToken); + + return entity.ToCheckpoint(); + } + + static MigrationCheckpointEntity Insert(MigrationCheckpoint checkpoint) + { + var entity = new MigrationCheckpointEntity { CategoryId = checkpoint.CategoryId, Version = 1 }; + + Apply(checkpoint, entity); + + return entity; + } + + // One place both branches copy the row through, so a column added to the checkpoint cannot be written on + // an update and forgotten on an insert. + static void Apply(MigrationCheckpoint checkpoint, MigrationCheckpointEntity entity) + { + entity.State = checkpoint.State; + entity.Cursor = checkpoint.Cursor; + entity.CopiedCount = checkpoint.CopiedCount; + entity.SkippedCount = checkpoint.SkippedCount; + entity.SourceTotal = checkpoint.SourceTotal; + entity.SkipReasons = checkpoint.SkipReasons; + entity.StartedAt = checkpoint.StartedAt; + entity.LastProgressAt = checkpoint.LastProgressAt; + entity.SettledAt = checkpoint.SettledAt; + entity.LastError = checkpoint.LastError; + entity.AlreadyPresentCount = checkpoint.AlreadyPresentCount; + } +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs index 354587cb5e..3b6f32ff55 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenReadOnlySourceLifecycle.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System; +using System.Diagnostics; using System.Net.Http; using System.Threading; using System.Threading.Tasks; @@ -34,7 +35,7 @@ public async Task Open(CancellationToken cancellationToken = default) try { - var serverUrl = settings.UseEmbeddedServer ? StartEmbedded() : settings.ConnectionString; + var serverUrl = settings.UseEmbeddedServer ? await StartEmbedded(cancellationToken) : settings.ConnectionString; documentStore = Connect(serverUrl); if (!settings.UseEmbeddedServer) @@ -63,6 +64,8 @@ public IAsyncDocumentSession OpenSession(string databaseName) => async Task EnsureReadable(string databaseName, string settingKey, CancellationToken cancellationToken) { + var elapsed = Stopwatch.StartNew(); + while (true) { cancellationToken.ThrowIfCancellationRequested(); @@ -72,10 +75,12 @@ async Task EnsureReadable(string databaseName, string settingKey, CancellationTo await DocumentStore.Maintenance.ForDatabase(databaseName).SendAsync(new GetStatisticsOperation(), cancellationToken); return; } + catch (DatabaseLoadTimeoutException e) when (settings.UseEmbeddedServer && elapsed.Elapsed >= EmbeddedLoadBudget) + { + throw new InvalidOperationException($"The RavenDB migration source at {Located()} has a database named '{databaseName}', from the '{settingKey}' setting, but it did not finish loading within {EmbeddedLoadBudget.TotalMinutes:N0} minutes. A data directory held by another process, or one that is corrupt, is the usual cause.", e); + } catch (DatabaseLoadTimeoutException) when (settings.UseEmbeddedServer) { - // A large embedded database routinely exceeds the load timeout on first open, which - // RavenEmbeddedPersistenceLifecycle already allows for the same way. await Task.Delay(EmbeddedLoadRetryDelay, cancellationToken); } catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) @@ -103,13 +108,26 @@ internal static string Located(RavenPersisterSettings sourceSettings, SettingsRo ? $"{sourceSettings.ServerUrl} (embedded, data directory '{sourceSettings.DatabasePath}', from '{root}/{RavenBootstrapper.DatabasePathKey}')" : sourceSettings.ConnectionString; - string StartEmbedded() + async Task StartEmbedded(CancellationToken cancellationToken) { - var configuration = new EmbeddedDatabaseConfiguration(settings.ServerUrl, settings.DatabaseName, settings.DatabasePath, settings.LogPath, settings.LogsMode); + // A dynamic query is a POST to /queries, which the request guard allows and which builds an auto-index + // on the customer's fallback database. This makes the server refuse it rather than trusting every reader. + var configuration = new EmbeddedDatabaseConfiguration(settings.ServerUrl, settings.DatabaseName, settings.DatabasePath, settings.LogPath, settings.LogsMode) { DisableAutoIndexCreation = true }; embedded = EmbeddedDatabase.Start(configuration, lifetime); - return embedded.ServerUrl; + try + { + return await embedded.WaitUntilReady(cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + throw new InvalidOperationException($"The RavenDB migration source could not start a server for the embedded database at {Located()}. A ServiceControl instance still running against that data directory is the usual cause: stop it, run the report, then start it again.", e); + } } IDocumentStore Connect(string serverUrl) @@ -171,6 +189,8 @@ public async ValueTask DisposeAsync() static readonly string[] ReadOnlyPostPaths = ["/queries", "/multi_get", "/streams/queries"]; static readonly TimeSpan EmbeddedShutdownTimeout = TimeSpan.FromSeconds(30); static readonly TimeSpan EmbeddedLoadRetryDelay = TimeSpan.FromMilliseconds(500); + // Generous because one DatabaseLoadTimeoutException already means RavenDB waited its own load timeout. + static readonly TimeSpan EmbeddedLoadBudget = TimeSpan.FromMinutes(5); IDocumentStore? documentStore; EmbeddedDatabase? embedded; diff --git a/src/ServiceControl.Persistence.Tests/EFCore/EFMigrationCheckpointStoreTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/EFMigrationCheckpointStoreTests.cs new file mode 100644 index 0000000000..b96684bb33 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/EFMigrationCheckpointStoreTests.cs @@ -0,0 +1,124 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.DataMigration; + +class EFMigrationCheckpointStoreTests : PersistenceTestBase +{ + IMigrationCheckpointStore Store => ServiceProvider.GetRequiredService(); + + [Test] + public async Task Read_returns_null_for_a_category_that_has_never_run() + { + Assert.That(await Store.Read("EndpointSettings"), Is.Null); + } + + [Test] + public async Task Upsert_then_Read_round_trips_every_field() + { + var checkpoint = new MigrationCheckpoint( + "EndpointSettings", MigrationCategoryState.CompleteWithErrors, "cursor-1", 5, 2, 40, + new Dictionary { [MigrationSkipReason.BodyUnreadable] = 2 }, Now, Now, Now, "body storage unavailable", AlreadyPresentCount: 33); + + var saved = await Store.Upsert(checkpoint); + var stored = await Store.Read("EndpointSettings"); + + // Record equality compares SkipReasons by reference, and a round-tripped dictionary is a new instance. + using (Assert.EnterMultipleScope()) + { + Assert.That(stored! with { SkipReasons = null, Version = 0 }, Is.EqualTo(checkpoint with { SkipReasons = null })); + Assert.That(stored.SkipReasons, Is.EquivalentTo(checkpoint.SkipReasons)); + Assert.That(stored.Version, Is.EqualTo(1), "the first save leaves the row at version 1"); + // The engine saves from what Upsert hands back, so a returned version that is not the stored one + // makes the next save conflict against a row nothing else touched. + Assert.That(saved.Version, Is.EqualTo(stored.Version)); + } + } + + [Test] + public async Task A_second_Upsert_for_the_same_category_updates_rather_than_duplicates() + { + await Store.Upsert(new MigrationCheckpoint("EndpointSettings", MigrationCategoryState.InProgress, "cursor-1", 5, 0, 40, null, Now, Now, null, null)); + var saved = await Store.Upsert(new MigrationCheckpoint("EndpointSettings", MigrationCategoryState.Complete, "cursor-2", 9, 1, 40, null, Now, Now, Now, null, Version: 1)); + + var all = await Store.ReadAll(); + var stored = await Store.Read("EndpointSettings"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(all, Has.Count.EqualTo(1)); + Assert.That(saved.Version, Is.EqualTo(stored!.Version), "an update hands back the version it landed on, not the one it was saved from"); + Assert.That(stored.State, Is.EqualTo(MigrationCategoryState.Complete)); + Assert.That(stored.Cursor, Is.EqualTo("cursor-2")); + Assert.That(stored.CopiedCount, Is.EqualTo(9)); + Assert.That(stored.SettledAt, Is.EqualTo(Now)); + } + } + + [Test] + public async Task A_checkpoint_saved_as_Abandoned_stores_its_counts_its_reasons_and_when_it_was_abandoned() + { + var reasons = new Dictionary { [MigrationSkipReason.BodyUnreadable] = 400 }; + var halted = Now.AddMinutes(-5); + await Store.Upsert(new MigrationCheckpoint("GroupComments", MigrationCategoryState.Halted, "g-1", 12, 400, 412, reasons, Now, Now, halted, "body storage unavailable")); + await Store.Upsert(new MigrationCheckpoint("GroupComments", MigrationCategoryState.Abandoned, "g-1", 12, 400, 412, reasons, Now, Now, Now, "body storage unavailable", Version: 1)); + + var stored = await Store.Read("GroupComments"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(stored!.State, Is.EqualTo(MigrationCategoryState.Abandoned)); + // State says which terminal state was reached and SettledAt says when, so abandoning a halted + // category moves the one timestamp on rather than filling a second column beside it. + Assert.That(stored.SettledAt, Is.EqualTo(Now)); + Assert.That(stored.SettledAt, Is.Not.EqualTo(halted)); + Assert.That(stored.SkippedCount, Is.EqualTo(400)); + Assert.That(stored.SkipReasons, Is.EquivalentTo(new Dictionary { [MigrationSkipReason.BodyUnreadable] = 400 })); + Assert.That(stored.LastError, Is.EqualTo("body storage unavailable")); + } + } + + [Test] + public async Task A_restart_after_a_halt_clears_the_fields_the_previous_save_set() + { + await Store.Upsert(new MigrationCheckpoint("EndpointSettings", MigrationCategoryState.Halted, "cursor-1", 5, 2, 40, + new Dictionary { [MigrationSkipReason.BodyUnreadable] = 2 }, Now, Now, Now, "body storage unavailable")); + + await Store.Upsert(new MigrationCheckpoint("EndpointSettings", MigrationCategoryState.InProgress, "cursor-1", 5, 2, 40, + null, Now, Now, null, null, Version: 1)); + + var stored = await Store.Read("EndpointSettings"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(stored!.SettledAt, Is.Null, "a resumed category must not look finished"); + Assert.That(stored.LastError, Is.Null); + Assert.That(stored.SkipReasons, Is.Null); + } + } + + [Test] + public void An_upsert_for_a_category_that_has_never_run_is_refused_when_it_carries_a_version() + { + var conflict = Assert.ThrowsAsync(() => Store.Upsert( + new MigrationCheckpoint("EndpointSettings", MigrationCategoryState.InProgress, "cursor-1", 1, 0, 40, null, Now, Now, null, null, Version: 1))); + + Assert.That(conflict!.Message, Does.Contain("EndpointSettings")); + } + + [Test] + public async Task ReadAll_returns_every_category_that_has_run() + { + await Store.Upsert(new MigrationCheckpoint("EndpointSettings", MigrationCategoryState.Complete, "e-1", 1, 0, 1, null, Now, Now, Now, null)); + await Store.Upsert(new MigrationCheckpoint("KnownEndpoints", MigrationCategoryState.InProgress, "k-1", 4, 0, 9, null, Now, Now, null, null)); + + var all = await Store.ReadAll(); + + Assert.That(all.Select(c => c.CategoryId), Is.EquivalentTo(new[] { "EndpointSettings", "KnownEndpoints" })); + } +} \ No newline at end of file diff --git a/src/ServiceControl.Persistence.Tests/EFCore/MigrationCheckpointTableTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/MigrationCheckpointTableTests.cs new file mode 100644 index 0000000000..4d92080e2f --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/MigrationCheckpointTableTests.cs @@ -0,0 +1,155 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; + +class MigrationCheckpointTableTests : PersistenceTestBase +{ + [Test] + public async Task The_table_round_trips_every_column_written_directly_through_the_DbContext() + { + using (var scope = ServiceProvider.CreateScope()) + { + var dbContext = scope.ServiceProvider.GetRequiredService(); + dbContext.MigrationCheckpoints.Add(new MigrationCheckpointEntity + { + CategoryId = "EndpointSettings", + State = MigrationCategoryState.InProgress, + Cursor = "cursor-1", + CopiedCount = 3, + SkippedCount = 1, + SourceTotal = 40, + SkipReasons = new Dictionary { [MigrationSkipReason.BodyUnreadable] = 1 }, + StartedAt = Now, + LastProgressAt = Now, + SettledAt = null, + LastError = null, + AlreadyPresentCount = 36, + Version = 7 + }); + await dbContext.SaveChangesAsync(); + } + + using var readScope = ServiceProvider.CreateScope(); + var readContext = readScope.ServiceProvider.GetRequiredService(); + var stored = await readContext.MigrationCheckpoints.AsNoTracking().SingleAsync(c => c.CategoryId == "EndpointSettings"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(stored.State, Is.EqualTo(MigrationCategoryState.InProgress)); + Assert.That(stored.Cursor, Is.EqualTo("cursor-1")); + Assert.That(stored.CopiedCount, Is.EqualTo(3)); + Assert.That(stored.SkippedCount, Is.EqualTo(1)); + Assert.That(stored.SourceTotal, Is.EqualTo(40)); + Assert.That(stored.SkipReasons, Is.EquivalentTo(new Dictionary { [MigrationSkipReason.BodyUnreadable] = 1 })); + Assert.That(stored.AlreadyPresentCount, Is.EqualTo(36)); + Assert.That(stored.Version, Is.EqualTo(7)); + Assert.That(stored.StartedAt, Is.EqualTo(Now)); + Assert.That(stored.LastProgressAt, Is.EqualTo(Now)); + Assert.That(stored.SettledAt, Is.Null); + Assert.That(stored.LastError, Is.Null); + } + } + + [Test] + public async Task Every_timestamp_column_comes_back_as_a_Utc_DateTime() + { + Assert.That(Now.Kind, Is.EqualTo(DateTimeKind.Utc), "the engine stamps timings from TimeProvider.GetUtcNow().UtcDateTime, so the test clock has to match"); + + using (var scope = ServiceProvider.CreateScope()) + { + var dbContext = scope.ServiceProvider.GetRequiredService(); + dbContext.MigrationCheckpoints.Add(new MigrationCheckpointEntity + { + CategoryId = "KnownEndpoints", + State = MigrationCategoryState.Complete, + Cursor = null, + CopiedCount = 1, + SkippedCount = 0, + SourceTotal = 1, + SkipReasons = null, + StartedAt = Now, + LastProgressAt = Now, + SettledAt = Now, + LastError = null + }); + await dbContext.SaveChangesAsync(); + } + + using var readScope = ServiceProvider.CreateScope(); + var readContext = readScope.ServiceProvider.GetRequiredService(); + var stored = await readContext.MigrationCheckpoints.AsNoTracking().SingleAsync(c => c.CategoryId == "KnownEndpoints"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(stored.StartedAt!.Value.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(stored.LastProgressAt!.Value.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(stored.SettledAt!.Value.Kind, Is.EqualTo(DateTimeKind.Utc)); + } + } + + [Test] + public void The_skip_reasons_JSON_is_keyed_by_reason_name_not_by_its_number() + { + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var converter = dbContext.Model.FindEntityType(typeof(MigrationCheckpointEntity))!.FindProperty("SkipReasons")!.GetValueConverter()!; + + var json = (string)converter.ConvertToProvider(new Dictionary { [MigrationSkipReason.BodyUnreadable] = 2 })!; + + // Enum.Parse accepts "0" as readily as "BodyUnreadable", so a round trip cannot tell the two encodings apart. + Assert.That(json, Does.Contain("BodyUnreadable")); + } + + [Test] + public void A_reason_name_this_build_does_not_know_is_read_as_Unknown_rather_than_throwing() + { + // The rows a newer build wrote are what a downgraded instance meets, and the read path that would + // throw is ReadAll, which is what decides whether the host may start. + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var converter = dbContext.Model.FindEntityType(typeof(MigrationCheckpointEntity))!.FindProperty("SkipReasons")!.GetValueConverter()!; + + var reasons = (IReadOnlyDictionary)converter.ConvertFromProvider( + """{"BodyUnreadable":2,"SomethingFromTheFuture":3,"AlsoFromTheFuture":1}""")!; + + using (Assert.EnterMultipleScope()) + { + Assert.That(reasons[MigrationSkipReason.BodyUnreadable], Is.EqualTo(2)); + Assert.That(reasons[MigrationSkipReason.Unknown], Is.EqualTo(4), "several unknown names collapse onto the one member, so their counts are summed rather than one of them winning"); + Assert.That(reasons, Has.Count.EqualTo(2)); + } + } + + [Test] + public void The_table_carries_exactly_the_columns_the_checkpoint_needs() + { + // The one thing the one-way door makes permanent. Adding, dropping or renaming one after this + // ships costs a migration against customer data, so it fails here rather than in a customer's + // database. Property names rather than column names, because each provider spells the columns + // its own way and the spelling is a convention the generated migrations already pin. + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var entityType = dbContext.Model.FindEntityType(typeof(MigrationCheckpointEntity))!; + + using (Assert.EnterMultipleScope()) + { + Assert.That(entityType.GetProperties().Select(property => property.Name), Is.EquivalentTo(new[] + { + "CategoryId", "State", "Cursor", "CopiedCount", "SkippedCount", "SourceTotal", + "SkipReasons", "StartedAt", "LastProgressAt", "SettledAt", "LastError", "AlreadyPresentCount", "Version" + })); + + // Underscores stripped so one assertion covers MigrationCheckpoints and migration_checkpoints. + Assert.That(entityType.GetTableName()!.Replace("_", ""), Is.EqualTo("MigrationCheckpoints").IgnoreCase); + } + } +} \ No newline at end of file diff --git a/src/ServiceControl.Persistence.Tests/EFCore/MigrationCheckpointTransactionTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/MigrationCheckpointTransactionTests.cs new file mode 100644 index 0000000000..706f41f046 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/MigrationCheckpointTransactionTests.cs @@ -0,0 +1,178 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; + +class MigrationCheckpointTransactionTests : PersistenceTestBase +{ + static MigrationCheckpoint Checkpoint(DateTime now, string cursor, long copied, long version = 0) => + new("EndpointSettings", MigrationCategoryState.InProgress, cursor, copied, 0, 40, null, now, now, null, null, Version: version); + + async Task SeedCheckpoint(string cursor, long copied) + { + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + await dbContext.UpsertCheckpoint(Checkpoint(Now, cursor, copied)); + } + + [Test] + public async Task A_failure_before_commit_leaves_neither_the_row_nor_the_checkpoint() + { + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var strategy = dbContext.Database.CreateExecutionStrategy(); + + var thrown = Assert.ThrowsAsync(() => strategy.ExecuteAsync(async () => + { + await using var transaction = await dbContext.Database.BeginTransactionAsync(); + + dbContext.Settings.Add(new SettingEntity { Key = "migration-test-row", Value = "should-not-survive" }); + await dbContext.SaveChangesAsync(); + + await dbContext.UpsertCheckpoint(Checkpoint(Now, "cursor-1", 1)); + + throw new InvalidOperationException("simulated failure just before commit"); + })); + + // Without this the test also passes when UpsertCheckpoint throws before writing anything. + Assert.That(thrown!.Message, Is.EqualTo("simulated failure just before commit")); + + using var verifyScope = ServiceProvider.CreateScope(); + var verifyContext = verifyScope.ServiceProvider.GetRequiredService(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await verifyContext.Settings.AnyAsync(s => s.Key == "migration-test-row"), Is.False); + Assert.That(await verifyContext.MigrationCheckpoints.AnyAsync(c => c.CategoryId == "EndpointSettings"), Is.False); + } + } + + [Test] + public async Task A_successful_commit_lands_the_row_and_the_checkpoint_together() + { + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var strategy = dbContext.Database.CreateExecutionStrategy(); + + await strategy.ExecuteAsync(async () => + { + await using var transaction = await dbContext.Database.BeginTransactionAsync(); + + dbContext.Settings.Add(new SettingEntity { Key = "migration-test-row", Value = "should-survive" }); + await dbContext.SaveChangesAsync(); + + await dbContext.UpsertCheckpoint(Checkpoint(Now, "cursor-1", 1)); + + await transaction.CommitAsync(); + }); + + using var verifyScope = ServiceProvider.CreateScope(); + var verifyContext = verifyScope.ServiceProvider.GetRequiredService(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await verifyContext.Settings.AnyAsync(s => s.Key == "migration-test-row"), Is.True); + Assert.That(await verifyContext.MigrationCheckpoints.AnyAsync(c => c.CategoryId == "EndpointSettings"), Is.True); + } + } + + [Test] + public async Task An_existing_checkpoint_is_advanced_by_a_commit_inside_the_callers_transaction() + { + await SeedCheckpoint("cursor-1", 1); + + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var strategy = dbContext.Database.CreateExecutionStrategy(); + + await strategy.ExecuteAsync(async () => + { + await using var transaction = await dbContext.Database.BeginTransactionAsync(); + + dbContext.Settings.Add(new SettingEntity { Key = "migration-test-row", Value = "should-survive" }); + await dbContext.SaveChangesAsync(); + + await dbContext.UpsertCheckpoint(Checkpoint(Now, "cursor-2", 2, version: 1)); + + await transaction.CommitAsync(); + }); + + using var verifyScope = ServiceProvider.CreateScope(); + var verifyContext = verifyScope.ServiceProvider.GetRequiredService(); + var stored = await verifyContext.MigrationCheckpoints.AsNoTracking().SingleAsync(c => c.CategoryId == "EndpointSettings"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await verifyContext.Settings.AnyAsync(s => s.Key == "migration-test-row"), Is.True); + Assert.That(stored.Cursor, Is.EqualTo("cursor-2")); + Assert.That(stored.CopiedCount, Is.EqualTo(2)); + } + } + + [Test] + public async Task An_existing_checkpoint_keeps_its_previous_cursor_when_the_transaction_rolls_back() + { + await SeedCheckpoint("cursor-1", 1); + + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + var strategy = dbContext.Database.CreateExecutionStrategy(); + + var thrown = Assert.ThrowsAsync(() => strategy.ExecuteAsync(async () => + { + await using var transaction = await dbContext.Database.BeginTransactionAsync(); + + dbContext.Settings.Add(new SettingEntity { Key = "migration-test-row", Value = "should-not-survive" }); + await dbContext.SaveChangesAsync(); + + await dbContext.UpsertCheckpoint(Checkpoint(Now, "cursor-2", 2, version: 1)); + + throw new InvalidOperationException("simulated failure just before commit"); + })); + + // Without this the test also passes when UpsertCheckpoint throws before writing anything. + Assert.That(thrown!.Message, Is.EqualTo("simulated failure just before commit")); + + using var verifyScope = ServiceProvider.CreateScope(); + var verifyContext = verifyScope.ServiceProvider.GetRequiredService(); + var stored = await verifyContext.MigrationCheckpoints.AsNoTracking().SingleAsync(c => c.CategoryId == "EndpointSettings"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await verifyContext.Settings.AnyAsync(s => s.Key == "migration-test-row"), Is.False); + Assert.That(stored.Cursor, Is.EqualTo("cursor-1")); + Assert.That(stored.CopiedCount, Is.EqualTo(1)); + } + } + + [Test] + public async Task A_save_built_from_an_out_of_date_copy_is_refused_rather_than_silently_winning() + { + var store = ServiceProvider.GetRequiredService(); + await store.Upsert(Checkpoint(Now, "cursor-1", 1)); + + // Both read the row at the same version, as a background copy and a required copy on two hosts would. + var first = (await store.Read("EndpointSettings"))!; + var stale = (await store.Read("EndpointSettings"))!; + + await store.Upsert(first with { Cursor = "cursor-first", CopiedCount = 2 }); + + Assert.ThrowsAsync(() => store.Upsert(stale with { Cursor = "cursor-stale", CopiedCount = 99 })); + + var stored = (await store.Read("EndpointSettings"))!; + + using (Assert.EnterMultipleScope()) + { + Assert.That(stored.Cursor, Is.EqualTo("cursor-first")); + Assert.That(stored.CopiedCount, Is.EqualTo(2)); + Assert.That(stored.Version, Is.EqualTo(first.Version + 1)); + } + } +} \ No newline at end of file diff --git a/src/ServiceControl.RavenDB/EmbeddedDatabase.cs b/src/ServiceControl.RavenDB/EmbeddedDatabase.cs index efdf66b06f..f0b60cc998 100644 --- a/src/ServiceControl.RavenDB/EmbeddedDatabase.cs +++ b/src/ServiceControl.RavenDB/EmbeddedDatabase.cs @@ -75,6 +75,11 @@ public static EmbeddedDatabase Start(EmbeddedDatabaseConfiguration databaseConfi optionalArgs.Add("--RunInMemory=true"); } + if (databaseConfiguration.DisableAutoIndexCreation) + { + optionalArgs.Add("--Indexing.DisableQueryOptimizerGeneratedIndexes=true"); + } + var serverOptions = new ServerOptions { CommandLineArgs = @@ -109,6 +114,10 @@ public static EmbeddedDatabase Start(EmbeddedDatabaseConfiguration databaseConfi return embeddedDatabase; } + // Start only queues the server up, so connecting by the configured URL instead reaches whatever already holds the port. + public async Task WaitUntilReady(CancellationToken cancellationToken = default) => + (await EmbeddedServer.Instance.GetServerUriAsync(cancellationToken)).ToString(); + void Start(ServerOptions serverOptions) { this.serverOptions = serverOptions; diff --git a/src/ServiceControl.RavenDB/EmbeddedDatabaseConfiguration.cs b/src/ServiceControl.RavenDB/EmbeddedDatabaseConfiguration.cs index 6aec695755..6184a39353 100644 --- a/src/ServiceControl.RavenDB/EmbeddedDatabaseConfiguration.cs +++ b/src/ServiceControl.RavenDB/EmbeddedDatabaseConfiguration.cs @@ -9,5 +9,8 @@ public class EmbeddedDatabaseConfiguration(string serverUrl, string dbName, stri public string LogsMode { get; } = logsMode; public bool RunInMemory { get; set; } + + /// Makes a dynamic query fail instead of building an auto-index for it. For a server started only to read, where creating an index would be a write. + public bool DisableAutoIndexCreation { get; set; } } } diff --git a/src/ServiceControl/Hosting/Commands/MigrationSourceReportCommand.cs b/src/ServiceControl/Hosting/Commands/MigrationSourceReportCommand.cs index 21446e6c5e..40d832e124 100644 --- a/src/ServiceControl/Hosting/Commands/MigrationSourceReportCommand.cs +++ b/src/ServiceControl/Hosting/Commands/MigrationSourceReportCommand.cs @@ -11,6 +11,23 @@ namespace ServiceControl.Hosting.Commands class MigrationSourceReportCommand : AbstractCommand { public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) + { + try + { + await Report(settings, cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + await Console.Error.WriteLineAsync(e.Message); + Environment.ExitCode = 1; + } + } + + static async Task Report(Settings settings, CancellationToken cancellationToken) { await using var source = await PersistenceFactory.OpenMigrationSource(settings, cancellationToken); diff --git a/src/ServiceControl/Persistence/PersistenceFactory.cs b/src/ServiceControl/Persistence/PersistenceFactory.cs index c431680696..59981ba691 100644 --- a/src/ServiceControl/Persistence/PersistenceFactory.cs +++ b/src/ServiceControl/Persistence/PersistenceFactory.cs @@ -43,7 +43,20 @@ public static IMigrationSource CreateMigrationSource(Settings settings) public static async Task OpenMigrationSource(Settings settings, CancellationToken cancellationToken = default) { var source = CreateMigrationSource(settings); - await source.Open(cancellationToken); + var opened = false; + + try + { + await source.Open(cancellationToken); + opened = true; + } + finally + { + if (!opened) + { + await source.DisposeAsync(); + } + } return source; } diff --git a/src/ServiceControl/Program.cs b/src/ServiceControl/Program.cs index 7234479cd3..b74bdc8488 100644 --- a/src/ServiceControl/Program.cs +++ b/src/ServiceControl/Program.cs @@ -39,7 +39,7 @@ await new CommandRunner(arguments.Command).Execute(arguments, settings); - return 0; + return Environment.ExitCode; } catch (Exception ex) {