Add EF Core SQL Server and PostgreSQL persisters to the audit instance

[johnsimons]

Summary

Add SQL Server and PostgreSQL persisters to the audit instance, built on EF Core. They implement the audit persistence contracts and pass the shared audit persistence and acceptance tests. The few shared tests that cover a deliberate difference from RavenDB are excluded, as listed below.

The persisters ship dormant, the same way the primary instance's EF persisters did. The installer still creates RavenDB audit instances only, and there are no public docs yet.

Configuration

Setting	Notes
ServiceControl.Audit/PersistenceType	SQLServer or PostgreSQL
ServiceControl.Audit/Database/ConnectionString	Required
ServiceControl.Audit/Database/Schema	Optional. The schema must already exist, or setup fails
ServiceControl.Audit/Database/CommandTimeout	Optional, in seconds, default 30
ServiceControl.Audit/QueryTimeoutInSeconds	Optional, default 60, bounds each read query

Setup applies the EF migrations. The migrations history is kept in its own table, __AuditMigrationsHistory, so an audit instance can share a database and schema with a primary instance. Maintenance mode is not supported.

Storage and retention

Each audit message is a plain insert, stamped with the hour it was ingested (created_on). The primary key is (created_on, id), with a database-generated id.

PostgreSQL

Both audit tables are partitioned by day, and retention drops expired partitions whole.

• A message view query locks every partition it cannot prune, and every index on those partitions, and the message views have no filter to prune on. Over a 30 day retention period, hourly partitions took 6,170 locks per query against a default lock table of 6,400. Daily partitions took 274.
• The retention period must be between 1 and 90 days. A daily partition cannot honor less than a day, and past 90 days a few concurrent queries would fill the default lock table.
• Partitions are provisioned 48 hours ahead. The "Audit partition provisioning" custom check reports when provisioning falls behind.
• Creating or dropping a partition waits at most 5 seconds for its lock, because every insert and query queues behind DDL while it waits. A sweep that times out tries again on the next run.

SQL Server

The tables are not partitioned, because a full-text index cannot be aligned to a partition scheme. Retention deletes each expired hour in batches of 4,000 rows, below the 5,000 locks at which SQL Server escalates to a table lock.

Both providers

The sweep runs hourly under a session lock (pg_try_advisory_lock or sp_getapplock) held on an unpooled connection. Only one host sweeps a schema at a time, and a host that crashes releases the lock when its connection drops.

Search and bodies

Full-text search is always on for these persisters, whatever EnableFullTextSearchOnBodies is set to.

• PostgreSQL uses a GIN expression index with the simple text search configuration, over the headers, the message type and the first 262,144 characters of the body. The cap is there because to_tsvector fails once a document's lexemes pass 1 MB. Search terms are ORed, as on RavenDB.
• SQL Server uses FREETEXT over the headers and the body. The index lives in the ServiceControlFullTextCatalog catalog, shared with the primary instance, and setup fails if Full-Text Search is not installed.
• Headers are stored as JSON written with JavaScriptEncoder.Create(UnicodeRanges.All). The default encoder would store non-ASCII header values, such as localized exception messages, as escape sequences that search cannot match.

Bodies are stored in the message row, and only text is kept:

Body	Stored	Body endpoint
None	nothing	no body URL
Text up to MaxBodySizeToStore	the whole body	200
Text over MaxBodySizeToStore	the part up to the limit, for search only	204
Binary, or not valid UTF-8	nothing	404

The body URL includes the ingestion hour, so a body lookup reads a single hour of the table.

Queries

• The total count is capped at 100,000, because an exact count is linear in the size of the table. Total-Count and the paging links report the cap when it is reached. Saga history is capped at 50,000 changes, as on RavenDB.
• Status sorts by its numeric value. The primary instance merges pages from several instances with MessageViewComparer, which compares the numeric value, so an instance sorting any other way could leave rows out of the merged page.
• Time range filters are converted to UTC, because PostgreSQL rejects a timestamptz parameter that is not UTC.
• Audit counts are per UTC day over the last 30 days, with the same semantics as RavenDB.
• Only indexed columns have a length limit, 450 characters. A conversation ID longer than that is stored as its first 385 characters, then #, then a SHA-256 hash of the whole value. A value converter applies the same function to lookups, so conversation queries stay exact. Message IDs have no limit, and every transport keeps endpoint names well under it.

Differences from RavenDB

• Audit messages are not deduplicated, so a redelivered message is stored twice.
• Binary bodies are not stored.
• Full-text search on bodies cannot be turned off.
• Status sorts numerically, where RavenDB sorts it by name.

The shared tests for the first three are excluded from the EF test projects.

Tests

• New test projects: ServiceControl.Audit.Persistence.Tests.SqlServer and .PostgreSql, and ServiceControl.Audit.AcceptanceTests.SqlServer and .PostgreSql. EF-specific tests live in ServiceControl.Audit.Persistence.Tests/EFCore/, which the RavenDB project excludes.
• SQL Server fills its full-text index in the background, several seconds behind each insert. The SQL Server persistence tests wait for it only before a search, which keeps that suite to about a minute.
• The cloud database workflow now also runs the audit persistence and acceptance suites.
• The installer engine test skips persister manifests that have a mandatory setting the installer cannot supply yet.

Local results, Release build:

Suite	Result
Audit persistence, PostgreSQL	61 passed
Audit persistence, SQL Server	48 passed
Audit persistence, RavenDB	53 passed
Audit acceptance, PostgreSQL	67 passed
Audit acceptance, SQL Server	67 passed
ServiceControl.Audit.UnitTests	103 passed
ServiceControl.UnitTests	440 passed
MultiInstance acceptance	23 passed
Packaging	18 passed

[rbev]

Looks good, a just a few minor questions

[rbev]

Is this the normal convention for the setting name?

[johnsimons]

Yes, for the audit instance, we tend to prefix settings with ServiceControl.Audit/.

[warwickschroeder]

Audit needs the same treatment as primary here i.e. ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweepCustomCheck.cs

[johnsimons]

done