fix(rest)!: apply an inline import mapping entry's transform, defaultValue and required

.changeset/25491-import-inline-mapping-entry-rules.md

@@ -0,0 +1,26 @@
+---
+'@objectstack/rest': patch
+---
+
+fix(rest)!: an inline import mapping entry's `transform`, `defaultValue` and `required` are applied, not dropped
+
+`POST /api/v1/data/:object/import`, the async import job and plugin-auth's admin user import (`POST /api/v1/auth/admin/import-users`) all parse their body through the same request parser, which accepts `mapping` as a `FieldMappingEntry[]`, whose entries carry `transform`, `defaultValue` and `required`. The request parser kept only the column each entry names with `sourceField` and the field it names with `targetField` and dropped the rest, so a request asking for `transform: 'uppercase'` validated, ran, and stored the cell unchanged. Each entry's rules now run on its column before the cells are coerced:
+
+- `trim`, `uppercase` and `lowercase` rewrite a string cell. `none` and `lookup` copy it; a `lookup` cell is resolved to a record id by the import's reference resolution, as before.
+- `defaultValue` fills a cell that is blank (absent, empty, whitespace or one of the request's `nullValues`).
+- `required: true` refuses the request with `400 MISSING_REQUIRED_FIELD` when a row still leaves that column blank, naming the column and the rows. The refusal comes before any row is written, on the dry run and the commit alike.
+- `transform: 'date_format'` refuses the request with `400 UNSUPPORTED_TRANSFORM`: an entry carries no date format for it to apply. A cell mapped to a date or datetime field is already parsed as a date, so drop the transform. A transform name outside the declared set is refused the same way.
+
+A plain `{ sourceField, targetField }` entry, and the compact `{ column: field }` record, map columns exactly as before.
+
+Clause-②: no (narrowing) — the import doors (the sync import, the async import job and plugin-auth's admin user import) refuse two request shapes they used to run: an inline mapping entry declaring `transform: 'date_format'` (or a transform name outside the declared set), and an entry marked `required: true` whose column a row leaves blank. Nothing is widened.
+
+Metadata-protocol: none (a REST import request's handling; nothing a metadata author writes changes)
+
+<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no spec key, authorable spelling, object definition, stored row or stored shape is added, removed, renamed or re-shaped, and no exported declaration is removed or narrowed, so there is nothing for `objectstack migrate meta` to rewrite. What changes is how the REST import doors handle a request body's inline mapping entries: their declared per-cell rules now run, and two shapes those rules make unanswerable are refused. The remedy is a request edit (drop `date_format`, fill the required cells or give the entry a `defaultValue`), never a rewrite of a package source or a stored row. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id is named or touched (not registered or already-registered); and nothing exported is removed or narrowed (not runtime-interface-only, type-surface-only or runtime-export-retired). -->
+
+**BREAKING** for a caller that relied on either request running.
+
+- **Before.** An inline entry's `transform`, `defaultValue` and `required` were ignored; the import ran with the rename alone.
+- **Now.** They are applied; `date_format` and a blank `required` cell refuse the request with `400`.
+- **Fix.** Drop `transform: 'date_format'` from the entry (the date field's own parse still applies), and fill a required column or give its entry a `defaultValue`.

content/docs/data-modeling/import-mappings.mdx

@@ -198,7 +198,7 @@ sending both is `400 CONFLICTING_MAPPING`.
 | | inline `mapping` | `mappingName` |
 |:---|:---|:---|
 | Shape | A flat `{ "<source column>": "<field>" }` rename (or a `sourceField`/`targetField` array) | A registered `mapping` artifact |
-| Transforms | None — rename only | The `fieldMapping` pipeline |
+| Transforms | Per entry of the array form: `transform` (`trim`, `uppercase`, `lowercase`; `none` and `lookup` copy the cell), `defaultValue` for a blank cell, `required`. The flat record is a rename only | The `fieldMapping` pipeline |
 | Unmapped columns | **Pass through** to the write path under their own header | **Dropped.** The artifact is a strict projection: only mapped targets survive |
 | Lives | In the one request | In a package or in metadata, reusable |
 
@@ -218,7 +218,8 @@ These are whole-request failures — nothing is written, and there is no per-row
 | 400 | `MAPPING_TARGET_MISMATCH` | The artifact's `targetObject` is not the object in the URL. |
 | 400 | `MAPPING_FORMAT_UNSUPPORTED` | The artifact declares `sourceFormat: 'xml'` or `'sql'`; the endpoint accepts csv, json and xlsx payloads only. |
 | 400 | `MAPPING_FORMAT_MISMATCH` | The declared `sourceFormat` contradicts the payload actually sent. A `csv` mapping does apply to an `xlsx` payload — those rows are tabular in the same way — but a `json` mapping does not. |
-| 400 | `UNSUPPORTED_TRANSFORM` | Some entry declares `javascript`, or a transform name the pipeline does not implement. |
+| 400 | `UNSUPPORTED_TRANSFORM` | Some artifact entry declares `javascript`, or a transform name the pipeline does not implement; or an inline entry declares `date_format` (it carries no format to apply) or a name outside `none`, `lookup`, `trim`, `uppercase`, `lowercase`. |
+| 400 | `MISSING_REQUIRED_FIELD` | An inline entry is marked `required: true` and some row leaves its column blank after its `defaultValue`. The error names the column and the rows; it applies to the dry run and the commit alike. |
 | 400 | `CONFLICTING_MAPPING` | Both `mappingName` and an inline `mapping` were supplied. |
 | 400 | `INVALID_REQUEST` | No recognizable payload, or `writeMode` resolved to update/upsert with no `matchFields` from either the request or the artifact. |
 | 413 | `PAYLOAD_TOO_LARGE` | More than 5,000 rows on the synchronous route. |

packages/rest/src/import-inline-mapping-entry.test.ts

@@ -0,0 +1,220 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#25491] An inline `FieldMappingEntry[]` mapping's per-cell rules
+ * (`transform`, `defaultValue`, `required`) are applied by the REAL
+ * `POST /data/:object/import` route before the cells are coerced: they used to
+ * be accepted and dropped, leaving only the rename. Driven through a real
+ * ObjectQL engine + protocol on sqlite `:memory:`, as
+ * `import-integration.test.ts` is.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { ObjectQL } from '@objectstack/objectql';
+import { SqlDriver } from '@objectstack/driver-sql';
+import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol';
+import { SysMetadataAuditObject, SysMetadataCommitObject, SysMetadataHistoryObject, SysMetadataObject } from '@objectstack/metadata-core';
+import { RestServer } from './rest-server';
+import { installDefaultOrganization } from './default-organization.testkit.js';
+
+const SUPPLIER = {
+  name: 'supplier', label: 'Supplier', systemFields: false,
+  fields: {
+    id: { name: 'id', type: 'text' as const, primaryKey: true },
+    name: { name: 'name', type: 'text' as const, label: 'Name' },
+    country: { name: 'country', type: 'text' as const, label: 'Country' },
+    credit_code: { name: 'credit_code', type: 'text' as const, label: 'Credit code' },
+  },
+  validations: [{
+    type: 'format' as const, name: 'credit_code_format', field: 'credit_code',
+    regex: '^[0-9A-Z]{18}$', message: 'Credit code must be 18 digits or capital letters',
+  }],
+};
+
+const CONTACT = {
+  name: 'contact', label: 'Contact', systemFields: false,
+  fields: {
+    id: { name: 'id', type: 'text' as const, primaryKey: true },
+    supplier: { name: 'supplier', type: 'text' as const, label: 'Supplier' },
+    email: { name: 'email', type: 'email' as const, label: 'Email' },
+    name: { name: 'name', type: 'text' as const, label: 'Name' },
+  },
+};
+
+const liveEngines: ObjectQL[] = [];
+afterEach(async () => {
+  while (liveEngines.length) {
+    try { await liveEngines.pop()?.destroy(); } catch { /* noop */ }
+  }
+});
+
+function makeRes() {
+  const res: any = {
+    write: () => true, end: () => {},
+    header: () => res,
+    status: (code: number) => { res._status = code; return res; },
+    json: (body: any) => { res._json = body; return res; },
+  };
+  return res;
+}
+
+async function boot() {
+  const engine = new ObjectQL();
+  liveEngines.push(engine);
+  engine.registerDriver(new SqlDriver({
+    client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true,
+  }), true);
+  await engine.init();
+  engine.registry.registerObject(SUPPLIER as any);
+  engine.registry.registerObject(CONTACT as any);
+  await engine.syncSchemas();
+  await installDefaultOrganization(engine);
+  for (const o of [SysMetadataObject, SysMetadataHistoryObject, SysMetadataAuditObject, SysMetadataCommitObject]) {
+    if (!engine.registry.getObject(o.name)) engine.registry.registerObject(o as any);
+  }
+  const protocol = new ObjectStackProtocolImplementation(engine as any);
+  const noop = () => {};
+  const server = { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} };
+  const rest = new RestServer(server as any, protocol as any, { api: { requireAuth: false } } as any);
+  (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' });
+  rest.registerRoutes();
+  const route = rest.getRoutes().find((r: any) => r.method === 'POST' && r.path === '/api/v1/data/:object/import');
+  expect(route).toBeDefined();
+  return { engine, route };
+}
+
+describe('import route — inline FieldMappingEntry rules (#25491)', () => {
+  let engine: any;
+  let route: any;
+  const call = async (object: string, body: any) => {
+    const res = makeRes();
+    await route.handler({ params: { object }, body } as any, res);
+    return res;
+  };
+
+  beforeEach(async () => {
+    ({ engine, route } = await boot());
+  });
+
+  it('uppercase: a lowercase credit code passes the ^[0-9A-Z]{18}$ rule and is stored upper-cased (the card\'s reproduction)', async () => {
+    const res = await call('supplier', {
+      format: 'csv',
+      csv: '公司全称,国家 / 地区,统一社会信用代码\n示例公司,中国,91000000ma0trial02\n',
+      mapping: [
+        { sourceField: '公司全称', targetField: 'name' },
+        { sourceField: '国家 / 地区', targetField: 'country' },
+        { sourceField: '统一社会信用代码', targetField: 'credit_code', transform: 'uppercase' },
+      ],
+      writeMode: 'upsert', matchFields: ['credit_code'],
+    });
+    expect(res._json).toMatchObject({ total: 1, ok: 1, errors: 0, created: 1 });
+    const stored = await engine.find('supplier', {});
+    expect(stored).toHaveLength(1);
+    expect(stored[0]).toMatchObject({ name: '示例公司', country: '中国', credit_code: '91000000MA0TRIAL02' });
+  });
+
+  it('lowercase: two spellings of one email merge into one contact under matchFields [supplier, email]', async () => {
+    const mapping = [
+      { sourceField: 'Supplier', targetField: 'supplier' },
+      { sourceField: 'Email', targetField: 'email', transform: 'lowercase' },
+      { sourceField: 'Name', targetField: 'name' },
+    ];
+    const first = await call('contact', {
+      format: 'csv', csv: 'Supplier,Email,Name\ns1,ZHOU@X.EXAMPLE.COM,Zhou\n',
+      mapping, writeMode: 'upsert', matchFields: ['supplier', 'email'],
+    });
+    expect(first._json).toMatchObject({ ok: 1, created: 1 });
+    const second = await call('contact', {
+      format: 'csv', csv: 'Supplier,Email,Name\ns1,zhou@x.example.com,Zhou Wei\n',
+      mapping, writeMode: 'upsert', matchFields: ['supplier', 'email'],
+    });
+    expect(second._json).toMatchObject({ ok: 1, created: 0, updated: 1 });
+    const stored = await engine.find('contact', {});
+    expect(stored).toHaveLength(1);
+    expect(stored[0]).toMatchObject({ supplier: 's1', email: 'zhou@x.example.com', name: 'Zhou Wei' });
+  });
+
+  it('trim: strips the cell even when the request turns trimWhitespace off', async () => {
+    const res = await call('supplier', {
+      format: 'json', trimWhitespace: false,
+      rows: [{ N: '  Acme  ', C: '  CN  ' }],
+      mapping: [
+        { sourceField: 'N', targetField: 'name', transform: 'trim' },
+        { sourceField: 'C', targetField: 'country' },
+      ],
+    });
+    expect(res._json).toMatchObject({ ok: 1, created: 1 });
+    const [stored] = await engine.find('supplier', {});
+    expect(stored).toMatchObject({ name: 'Acme', country: '  CN  ' });
+  });
+
+  it('defaultValue: fills a blank cell and a missing column, leaves a filled cell alone', async () => {
+    const res = await call('supplier', {
+      format: 'csv', csv: 'Name,Country\nA,\nB,JP\n',
+      mapping: [
+        { sourceField: 'Name', targetField: 'name' },
+        { sourceField: 'Country', targetField: 'country', defaultValue: 'CN' },
+        { sourceField: 'Code', targetField: 'credit_code', defaultValue: '000000000000000000' },
+      ],
+    });
+    expect(res._json).toMatchObject({ ok: 2, created: 2 });
+    const stored = await engine.find('supplier', { orderBy: [{ field: 'name', order: 'asc' }] });
+    expect(stored.map((r: any) => [r.name, r.country, r.credit_code])).toEqual([
+      ['A', 'CN', '000000000000000000'],
+      ['B', 'JP', '000000000000000000'],
+    ]);
+  });
+
+  it('required: a blank required cell refuses the request before any row, on the dry run and the commit alike', async () => {
+    const body = {
+      format: 'csv', csv: 'Name,Country\nA,CN\nB,\n',
+      mapping: [
+        { sourceField: 'Name', targetField: 'name' },
+        { sourceField: 'Country', targetField: 'country', required: true },
+      ],
+    };
+    const dry = await call('supplier', { ...body, dryRun: true });
+    const commit = await call('supplier', body);
+    for (const res of [dry, commit]) {
+      expect(res._status).toBe(400);
+      expect(res._json?.error?.code ?? res._json?.code).toBe('MISSING_REQUIRED_FIELD');
+      expect(JSON.stringify(res._json)).toContain('row 2');
+    }
+    expect(await engine.find('supplier', {})).toHaveLength(0);
+  });
+
+  it('required with a defaultValue is satisfied by the default', async () => {
+    const res = await call('supplier', {
+      format: 'csv', csv: 'Name,Country\nB,\n',
+      mapping: [
+        { sourceField: 'Name', targetField: 'name' },
+        { sourceField: 'Country', targetField: 'country', required: true, defaultValue: 'CN' },
+      ],
+    });
+    expect(res._json).toMatchObject({ ok: 1, created: 1 });
+  });
+
+  it('date_format is refused with 400 UNSUPPORTED_TRANSFORM naming it, and nothing is written', async () => {
+    const res = await call('supplier', {
+      format: 'csv', csv: 'Name\nA\n',
+      mapping: [{ sourceField: 'Name', targetField: 'name', transform: 'date_format' }],
+    });
+    expect(res._status).toBe(400);
+    expect(res._json?.error?.code ?? res._json?.code).toBe('UNSUPPORTED_TRANSFORM');
+    expect(JSON.stringify(res._json)).toContain('date_format');
+    expect(await engine.find('supplier', {})).toHaveLength(0);
+  });
+
+  it('control: a plain { sourceField, targetField } array mapping is the rename it always was', async () => {
+    const res = await call('supplier', {
+      format: 'csv', csv: 'Name,Country\n  Acme ,cn\n', trimWhitespace: false,
+      mapping: [
+        { sourceField: 'Name', targetField: 'name' },
+        { sourceField: 'Country', targetField: 'country', transform: 'none' },
+      ],
+    });
+    expect(res._json).toMatchObject({ ok: 1, created: 1 });
+    const [stored] = await engine.find('supplier', {});
+    expect(stored).toMatchObject({ name: '  Acme ', country: 'cn' });
+  });
+});

packages/rest/src/import-mapping.ts

@@ -6,8 +6,11 @@
  * pipeline to parsed rows.
  *
  * Seam with the inline request `mapping`:
- *   • inline  = a plain `{ sourceColumn: targetField }` RENAME for one-off,
- *     wizard-driven imports — unmapped columns pass through untouched.
+ *   • inline  = a `{ sourceColumn: targetField }` rename (or a
+ *     `FieldMappingEntry[]` whose entries may add `transform`
+ *     trim/uppercase/lowercase, `defaultValue` and `required`, applied in
+ *     `import-prepare.ts`) for one-off, wizard-driven imports — unmapped
+ *     columns pass through untouched.
  *   • artifact = a reusable, governed ETL projection for recurring /
  *     programmatic imports — the output row contains ONLY the mapped
  *     targets (a strict projection; source files from external systems