Fix time range filters, non-ASCII search and oversized values in the EF Core persisters

[johnsimons]

What this fixes

Checking the audit instance's EF review fixes against the primary turned up four bugs in the primary instance's EF Core persisters. The EF Core persisters are not released yet, so no existing data needs migrating.

Time range filters return a 500 on PostgreSQL

FilterBySentTimeRange serves the from and to parameters of GET /api/messages2. FilterByLastModifiedRange serves the modified parameter of /api/errors, /api/endpoints/{name}/errors and the failure group routes. Both passed the parsed DateTime straight into the query. A value with an offset parses as Kind=Local, and a value with no zone parses as Kind=Unspecified. Npgsql only writes UTC values to timestamptz, so both kinds threw an ArgumentException such as Cannot write DateTime with Kind=Local to PostgreSQL type 'timestamp with time zone', and the request returned a 500. Only Z values worked.

SQL Server was not affected, because UtcDateTimeConverter also converts query parameters. ServicePulse always sends toISOString(), so only other API clients hit this.

The EF filters now convert both ends of a range to UTC. A value with no zone is treated as UTC, as SQL Server already does. The shared DateTimeRange is unchanged, because the RavenDB persister uses it too.

Search cannot find non-ASCII header values

MessageHeaders serialized with the default encoder, which escapes every non-ASCII character. HeadersJson stored Bestellpr\u00FCfung for Bestellprüfung, so the full text index held the escape sequence. A search for the word found nothing on either provider.

Headers are now written with JavaScriptEncoder.Create(UnicodeRanges.All). Letters outside ASCII are stored as they are, while + & < > ' and emoji stay escaped as before. This encoder was picked over UnsafeRelaxedJsonEscaping because it fixes search without anyone having to reason about HTML safety.

A value that does not fit its column fails the whole ingestion batch

A 600 character MessageId or ConversationId failed the batch it arrived in. PostgreSQL raised 22001: value too long for type character varying(450), and SQL Server raised "String or binary data would be truncated". IngestionPipeline.Fail fails every message in the batch. After three attempts, ErrorIngestionFaultPolicy stores each message as a failed import, and re-importing such a message fails the same way.

• MessageId has no index, so the column no longer has a length limit on FailedMessages or on FailedErrorImports. When GET /messages/{id}/body looks a message up by MessageId, it compares the full value.
• ConversationId is indexed and keeps its 450 limit. A value of 450 characters or fewer is stored unchanged, so GUIDs and ordinary custom ids are not affected. A longer value is stored as its first 385 characters, then #, then the SHA-256 of the whole value as 64 hex characters, without splitting a surrogate pair. Two different long ids never share a stored value, and applying the function to a stored value returns it unchanged.
• The function runs in an EF value converter on ConversationId. EF applies a converter to its own writes and to every query value compared with the property. The conversation lookup therefore finds a message by its full id or by its stored form, and a future query cannot forget the conversion.
• Ingestion writes through hand-written SQL, which skips converters, so EFRecoverabilityIngestionUnitOfWork also applies the function next to TruncateTypeName.
• ServicePulse shows the stored form of a long id, and the headers tab still shows the full value. In a deployment that mixes RavenDB and EF instances, a lookup by the stored form finds nothing on the RavenDB side.
• Endpoint names, queue addresses and MessageType keep their limits. Transports keep endpoint names and queue addresses far below 450 characters, and TruncateTypeName already shortens MessageType.

A body with too many distinct words fails ingestion on PostgreSQL

The full text index is a GIN index on a to_tsvector expression over the headers, the body and the message type. to_tsvector fails once the vector passes 1 MB. Because the index is on an expression, the error surfaces on INSERT and fails the batch. A 1.35 MB body of 150,000 distinct words failed with 54000: string is too long for tsvector (1800526 bytes, max 1048575 bytes). BodyText holds at most MaxBodySizeToStore bytes, 100 KB by default, so the failure needs that setting raised to around 1 MB.

The index now covers substring(COALESCE(body_text, ''), 1, 262144), and the search dialect renders exactly that from (message.BodyText ?? "").Substring(0, 262144). Words past the first 262,144 characters of a body are no longer searchable on PostgreSQL. SQL Server fills its full text index in the background, so a large body cannot fail an insert there, and its index is unchanged.

The index expression and the query expression must match character for character, or PostgreSQL falls back to a sequential scan without any error. FullTextSearchIndexTests pins the two together. A one-off EXPLAIN of the search query, with sequential scans disabled, showed a Bitmap Index Scan on ix_failed_messages_full_text.

[rbev]

why do html sensitive characters matter in a json serialisation?

[johnsimons]

They don't, for this column. System.Text.Json escapes them by default in case the JSON ends up inside HTML. HeadersJson never does: MessageHeaders.Read parses it, and the API serializes the headers again with its own settings.

The escaping also hurt search, because the full text index reads the stored text. Create(UnicodeRanges.All) still wrote ' as \u0027, so "The given key 'CustomerId' was not present in the dictionary." was indexed with the token u0027customerid. A search for CustomerId found nothing on either provider. The same happened to a nested type name after a +.

Headers now use JavaScriptEncoder.UnsafeRelaxedJsonEscaping. A new test searches for a word in single quotes on both providers.