fix(objectql, driver-sql, spec): a stored expression that reads an autonumber stores the value computed with the number

.changeset/25073-autonumber-stored-expression.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/spec': patch
+'@objectstack/objectql': patch
+'@objectstack/driver-sql': patch
+'@objectstack/driver-turso': patch
+'@objectstack/plugin-pinyin-search': patch
+---
+
+fix(objectql, driver-sql, spec): a stored field whose `expression` reads an `autonumber` stores the value computed with the number
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+A stored (non-`formula`) field whose `expression` reads a record number, such as `record.case_number + " - " + record.subject`, was computed before the number existed. On insert it stored `null` (or the `has()` fallback) until a later update by id computed it again. Now the insert computes it again right after the number is issued, and writes the row once, carrying both.
+
+- **`IDataDriver.issueAutonumbers` (new, optional).** A driver that owns its record-number counter can issue a row's numbers ahead of the write, in place, and answer after a unique violation whether the collision was on a number it issued (re-seeding the counter and emptying those slots). `SqlDriver` implements it over its existing counter, so `driver-sqlite-wasm` and `driver-turso` (both faces) inherit it. A driver without it is unchanged.
+- **The engine.** For an object whose stored expression reads a record number (directly or through another stored field), the engine asks the driver for the number after validation, where it already issued numbers itself, so a rejected row still burns none. It computes the readers again and hands `create()` / `bulkCreate()` the row with the number filled: one INSERT, no transaction. A collision on a single row is re-issued through the member by the engine's existing retry loop, which computes the readers again. A batch collision re-seeds the counter and is not re-issued (the batch policy the engine already had). Objects with no such field keep the driver's own issue-at-write path. Where the engine issues the number itself (driver-memory, driver-mongodb), the readers are computed again after it does.
+- **A driver that owns its counter without the member** stores the readers computed without the number, as before, and the engine now says so once per object at `error` level, with the remedy.
+- **`ObjectQL.registerInsertRecomputeListener(id, fn)` (new).** Called per row after that second computation, on the row the driver is handed. `plugin-pinyin-search` registers on it, so a `__search` companion whose name field reads the record number is stamped from the value that carries the number, and `$search` finds the record by pinyin and by its number.
+- **One change in what a refused insert reports.** On an object whose stored expression reads the number, a SQL driver's collision that survives every retry now leaves as `ERR_AUTONUMBER_COLLISION` (the engine's three attempts) instead of `DUPLICATE_RECORD` (the driver's own retries). Every other object is unchanged.

packages/drivers/driver-sql/src/sql-driver-autonumber-issue-ahead.test.ts

@@ -0,0 +1,215 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#25073] `SqlDriver.issueAutonumbers` — the record number on request, ahead
+ * of the write.
+ *
+ * `create()` / `bulkCreate()` fill an empty `autonumber` slot as the last step
+ * before the INSERT, so a stored field whose `expression` reads the number was
+ * computed by the engine before the number existed (null for a concatenation,
+ * the `has()` fallback otherwise). The member issues the same number earlier,
+ * on the caller's row, and `create()` then finds the slot filled and writes
+ * the row as it stands: one statement, carrying the number the engine computed
+ * from.
+ *
+ * What is pinned here is the member's own half, on a real SQLite database:
+ *
+ *  1. ISSUE fills what `create()` would have filled (same counter, same
+ *     organization scope), leaves a typed value alone, and `create()` of the
+ *     filled row draws no second number.
+ *  2. AFTER A COLLISION it is `create()`'s #5495 re-seed split at the point
+ *     only the caller can go on: it judges the error as `create()` does,
+ *     re-seeds the stale counter once and EMPTIES the slot, so the caller's
+ *     next issue lands on MAX + 1. It never answers for a duplicate on a
+ *     value the caller typed, for a non-unique error, or inside a caller's
+ *     transaction.
+ *  3. A batch: every row that drew from the stale counter is emptied (#6943's
+ *     per-counter reading), the counter re-seeded once.
+ *
+ * The engine's half (issue after validation, recompute, retry) is pinned in
+ * `packages/runtime/src/autonumber-stored-expression.integration.test.ts`.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { isUniqueViolationError } from '@objectstack/types';
+import { SqlDriver } from './index.js';
+
+const SEQUENCES_TABLE = '_objectstack_sequences';
+
+const CRM_CASE = {
+  name: 'crm_case',
+  fields: {
+    organization_id: { type: 'string' },
+    case_number: { type: 'autonumber', format: 'CASE-{00000}', unique: true },
+    title: { type: 'string' },
+  },
+} as any;
+
+describe('[#25073] SqlDriver.issueAutonumbers', () => {
+  let driver: SqlDriver;
+  const knex = () => (driver as any).knex;
+
+  const lastValue = async (tenant: string): Promise<number | undefined> => {
+    const rows = await knex()(SEQUENCES_TABLE).select('*');
+    const row = rows.find((r: any) => r.object === 'crm_case' && r.field === 'case_number' && r.tenant_id === tenant);
+    return row ? Number(row.last_value) : undefined;
+  };
+
+  /** Land rows by a path that bypasses the counter, as a seed replay does. */
+  const bypassInsert = async (org: string, from: number, to: number) => {
+    const rows = [];
+    for (let n = from; n <= to; n++) {
+      rows.push({ id: `${org}-s${n}`, organization_id: org, case_number: `CASE-${String(n).padStart(5, '0')}`, title: `seed ${n}` });
+    }
+    await knex()('crm_case').insert(rows);
+  };
+
+  /** The error a write raises, or a failure when it lands. */
+  const refusal = async (write: Promise<unknown>): Promise<unknown> => {
+    try {
+      await write;
+    } catch (error) {
+      return error;
+    }
+    throw new Error('expected the write to be refused');
+  };
+
+  beforeEach(async () => {
+    driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true });
+    await driver.initObjects([CRM_CASE]);
+  });
+
+  afterEach(async () => {
+    await driver.disconnect();
+  });
+
+  describe('issue', () => {
+    it('fills the empty slot in place, from the counter create() draws from, and create() draws no second number', async () => {
+      await driver.create('crm_case', { organization_id: 'orgA', title: 'by create' });
+      expect(await lastValue('orgA')).toBe(1);
+
+      const row: Record<string, unknown> = { organization_id: 'orgA', title: 'ahead' };
+      expect(await driver.issueAutonumbers('crm_case', [row])).toEqual([['case_number']]);
+      expect(row.case_number).toBe('CASE-00002');
+      expect(await lastValue('orgA')).toBe(2);
+
+      const stored = await driver.create('crm_case', row);
+      expect(stored.case_number).toBe('CASE-00002');
+      expect(await lastValue('orgA')).toBe(2);
+      // And the counter simply goes on from there.
+      expect((await driver.create('crm_case', { organization_id: 'orgA', title: 'next' })).case_number).toBe('CASE-00003');
+    });
+
+    it('issues per organization from the call options, as create() stamps the organization first', async () => {
+      await driver.create('crm_case', { organization_id: 'orgA', title: 'a1' });
+      const row: Record<string, unknown> = { title: 'b, organization from options' };
+      expect(await driver.issueAutonumbers('crm_case', [row], { tenantId: 'orgB' } as any)).toEqual([['case_number']]);
+      expect(row.case_number).toBe('CASE-00001');
+      expect(await lastValue('orgB')).toBe(1);
+      expect(await lastValue('orgA')).toBe(1);
+      // Only the number lands on the caller's row; `create()` stamps the organization.
+      expect(row).not.toHaveProperty('organization_id');
+      expect((await driver.create('crm_case', row, { tenantId: 'orgB' } as any)).organization_id).toBe('orgB');
+    });
+
+    it('leaves a typed value alone and does not report it', async () => {
+      const row: Record<string, unknown> = { organization_id: 'orgA', case_number: 'CASE-07777', title: 'import' };
+      expect(await driver.issueAutonumbers('crm_case', [row])).toEqual([[]]);
+      expect(row.case_number).toBe('CASE-07777');
+      expect(await lastValue('orgA')).toBeUndefined();
+    });
+
+    it('answers index-aligned with the rows of a batch', async () => {
+      const rows: Record<string, unknown>[] = [
+        { organization_id: 'orgA', title: 'one' },
+        { organization_id: 'orgA', case_number: 'CASE-00500', title: 'typed' },
+        { organization_id: 'orgA', title: 'two' },
+      ];
+      expect(await driver.issueAutonumbers('crm_case', rows)).toEqual([['case_number'], [], ['case_number']]);
+      expect(rows.map((r) => r.case_number)).toEqual(['CASE-00001', 'CASE-00500', 'CASE-00002']);
+      const stored = await driver.bulkCreate('crm_case', rows);
+      expect(stored.map((r) => r.case_number)).toEqual(['CASE-00001', 'CASE-00500', 'CASE-00002']);
+    });
+  });
+
+  describe('after a collision', () => {
+    it('re-seeds the counter it proves stale and empties the slot, so the next issue lands on MAX + 1', async () => {
+      await bypassInsert('orgA', 9, 9);
+      await driver.create('crm_case', { organization_id: 'orgA', title: 'first' });
+      expect(await lastValue('orgA')).toBe(10);
+      await bypassInsert('orgA', 11, 39);
+
+      const row: Record<string, unknown> = { organization_id: 'orgA', title: 'after the seeds' };
+      await driver.issueAutonumbers('crm_case', [row]);
+      expect(row.case_number).toBe('CASE-00011');
+      const error = await refusal(driver.create('crm_case', row));
+      expect(isUniqueViolationError(error)).toBe(true);
+
+      expect(await driver.issueAutonumbers('crm_case', [row], undefined, error)).toEqual([['case_number']]);
+      expect(row).not.toHaveProperty('case_number');
+      expect(await lastValue('orgA')).toBe(39);
+
+      expect(await driver.issueAutonumbers('crm_case', [row])).toEqual([['case_number']]);
+      expect(row.case_number).toBe('CASE-00040');
+      expect((await driver.create('crm_case', row)).case_number).toBe('CASE-00040');
+    });
+
+    it('answers nothing, and changes nothing, for a duplicate on a value the caller typed', async () => {
+      await driver.create('crm_case', { organization_id: 'orgA', case_number: 'CASE-00001', title: 'typed' });
+      const row: Record<string, unknown> = { organization_id: 'orgA', case_number: 'CASE-00001', title: 'typed again' };
+      expect(await driver.issueAutonumbers('crm_case', [row])).toEqual([[]]);
+      const error = await refusal(driver.create('crm_case', row));
+      expect(isUniqueViolationError(error)).toBe(true);
+
+      expect(await driver.issueAutonumbers('crm_case', [row], undefined, error)).toEqual([[]]);
+      expect(row.case_number).toBe('CASE-00001');
+    });
+
+    it('answers nothing for an error that is not a unique violation', async () => {
+      const row: Record<string, unknown> = { organization_id: 'orgA', title: 'x' };
+      await driver.issueAutonumbers('crm_case', [row]);
+      const issued = row.case_number;
+      expect(await driver.issueAutonumbers('crm_case', [row], undefined, new Error('SQLITE_BUSY: database is locked'))).toEqual([[]]);
+      expect(row.case_number).toBe(issued);
+    });
+
+    it('answers nothing inside a caller transaction, where create() retries nothing either', async () => {
+      await driver.create('crm_case', { organization_id: 'orgA', title: 'seeds the counter at 1' });
+      await bypassInsert('orgA', 2, 5);
+      const row: Record<string, unknown> = { organization_id: 'orgA', title: 'in a transaction' };
+      await driver.issueAutonumbers('crm_case', [row]);
+      const error = await refusal(driver.create('crm_case', row));
+      expect(isUniqueViolationError(error)).toBe(true);
+      const trx = await knex().transaction();
+      try {
+        expect(await driver.issueAutonumbers('crm_case', [row], { transaction: trx } as any, error)).toEqual([[]]);
+      } finally {
+        await trx.rollback();
+      }
+      expect(row.case_number).toBe('CASE-00002');
+      expect(await lastValue('orgA')).toBe(2);
+    });
+
+    it('empties every row of a batch that drew from the stale counter, and re-seeds it once', async () => {
+      await bypassInsert('orgA', 1, 1);
+      await driver.create('crm_case', { organization_id: 'orgA', title: 'seeds the counter at 2' });
+      await bypassInsert('orgA', 3, 3);
+
+      const rows: Record<string, unknown>[] = [
+        { organization_id: 'orgA', title: 'collides on 3' },
+        { organization_id: 'orgA', title: 'would be 4, did not collide' },
+        { organization_id: 'orgB', title: 'another counter' },
+      ];
+      await driver.issueAutonumbers('crm_case', rows);
+      expect(rows.map((r) => r.case_number)).toEqual(['CASE-00003', 'CASE-00004', 'CASE-00001']);
+      const error = await refusal(driver.bulkCreate('crm_case', rows));
+
+      expect(await driver.issueAutonumbers('crm_case', rows, undefined, error)).toEqual([['case_number'], ['case_number'], []]);
+      expect(rows[0]).not.toHaveProperty('case_number');
+      expect(rows[1]).not.toHaveProperty('case_number');
+      expect(rows[2].case_number).toBe('CASE-00001');
+      expect(await lastValue('orgA')).toBe(4);
+      expect(await lastValue('orgB')).toBe(1);
+    });
+  });
+});

packages/drivers/driver-sql/src/sql-driver.ts

@@ -8489,6 +8489,98 @@ export class SqlDriver implements IDataDriver {
     return reservations;
   }
 
+  /**
+   * The reservations {@link issueAutonumbers} made, keyed by the caller's row
+   * object, so its collision call can tell a number it issued from one the
+   * caller typed — the distinction `create()` draws from its own local
+   * `reservations`. Weak, so a row the caller drops takes its entry with it.
+   */
+  private readonly issuedAutoNumbers = new WeakMap<object, AutoNumberReservation[]>();
+
+  /**
+   * [#25073] `IDataDriver.issueAutonumbers`: {@link fillAutoNumberFields} on
+   * request, ahead of the write, so the engine can compute a stored field
+   * whose `expression` reads the number before it hands the row to
+   * {@link create} / {@link bulkCreate}. Those find the slot filled and
+   * reserve nothing, so the row goes in as ONE statement carrying the number
+   * the engine computed from.
+   *
+   * After a collision it is {@link create}'s and {@link bulkCreate}'s own
+   * re-seed (#5495, #6943), split at the point where only the caller can go
+   * on: which numbers collided is decided as there
+   * ({@link collidingAutoNumberReservations}, so a duplicate on a value the
+   * caller typed is never this counter's), each stale counter is re-seeded
+   * ONCE, and every slot drawn from a stale counter is emptied — on a batch
+   * that includes rows whose own value did not collide, for the reason
+   * {@link bulkCreate} gives. The caller issues again; this method never
+   * re-issues on its own, because the caller has to recompute what reads
+   * the number. Inside a caller's transaction nothing is retried, as
+   * {@link create} retries nothing there (Postgres has aborted the handle).
+   *
+   * Inherited by `driver-sqlite-wasm` and `driver-turso`: the remote face
+   * issues through its own `getNextSequenceValue` and probes through its own
+   * `autoNumberValueExists` / `resyncSequenceToDataMax`, the overrides its
+   * `create()` already calls.
+   */
+  async issueAutonumbers(
+    object: string,
+    rows: Record<string, unknown>[],
+    options?: DriverOptions,
+    afterCollision?: unknown,
+  ): Promise<string[][]> {
+    if (afterCollision === undefined) {
+      const filled: string[][] = [];
+      for (const row of rows) {
+        if (!row || typeof row !== 'object') {
+          filled.push([]);
+          continue;
+        }
+        // Filled as `create()` fills it: the caller's organization stamped
+        // first (numbers are issued per organization), on a view, so only
+        // the numbers land on the caller's row.
+        const view: Record<string, any> = { ...row };
+        this.injectTenantOnInsert(object, view, options);
+        const reserved = await this.fillAutoNumberFields(object, view, options);
+        for (const r of reserved) row[r.field] = view[r.field];
+        // A value kept from an earlier issue on this row is still this
+        // driver's, so it stays described beside the fresh ones (#6943).
+        const fresh = new Set(reserved.map((r) => r.field));
+        const kept = (this.issuedAutoNumbers.get(row) ?? []).filter((r) => !fresh.has(r.field));
+        this.issuedAutoNumbers.set(row, [...kept, ...reserved]);
+        filled.push(reserved.map((r) => r.field));
+      }
+      return filled;
+    }
+
+    const none = rows.map((): string[] => []);
+    if (options?.transaction !== undefined) return none;
+    const issued = rows.map((row) => (row && typeof row === 'object' ? this.issuedAutoNumbers.get(row) ?? [] : []));
+    const colliding = await this.collidingAutoNumberReservations(afterCollision, issued.flat(), options);
+    if (colliding.length === 0) return none;
+
+    const stale = new Set<string>();
+    for (const reservation of colliding) {
+      const key = this.autoNumberCounterKey(reservation);
+      if (stale.has(key)) continue;
+      stale.add(key);
+      await this.resyncSequenceToDataMax(reservation);
+    }
+    return rows.map((row, i) => {
+      const emptied: string[] = [];
+      const kept: AutoNumberReservation[] = [];
+      for (const reservation of issued[i]) {
+        if (stale.has(this.autoNumberCounterKey(reservation))) {
+          delete row[reservation.field];
+          emptied.push(reservation.field);
+        } else {
+          kept.push(reservation);
+        }
+      }
+      if (emptied.length > 0) this.issuedAutoNumbers.set(row, kept);
+      return emptied;
+    });
+  }
+
   /**
    * Does a row already hold `reservation.value` in the partition this
    * counter covers? (#5495)

packages/drivers/driver-turso/src/turso-driver.ts

@@ -1051,6 +1051,11 @@ export const REMOTE_FACE_ANSWERS = {
   // Deprecated aliases: they call `this.commit` / `this.rollback`, which refuse.
   commitTransaction: 'inherited',
   rollbackTransaction: 'inherited',
+  // [#25073] Record numbers ahead of the write: `fillAutoNumberFields` and the
+  // collision re-seed, which already reach the remote database through this
+  // face's own `getNextSequenceValue`, `autoNumberValueExists` and
+  // `resyncSequenceToDataMax` (see "Autonumber on the remote face" above).
+  issueAutonumbers: 'inherited',
   // Knex-only reads.
   findWithWindowFunctions: 'refused',
   analyzeQuery: 'refused',