Elwood Roadmap

Vision

Elwood is a standalone JSON transformation DSL inspired by KQL pipes, LINQ lambdas, and JSONPath navigation. It is designed for high performance, rich error reporting, and cross-platform use.

The .NET implementation is the reference engine. A TypeScript implementation provides the same language in browsers, Node.js, and edge runtimes. Both implementations share a conformance test suite to guarantee identical behavior.

Phase 1 — Standalone DSL Engine ✅ (Complete)

Goal: A working JSON transformation engine.

Phase 1b — Cross-Platform (TypeScript Port)

Goal: Make Elwood available in browsers, Node.js, Deno, Bun, and edge runtimes (Cloudflare Workers, Vercel Edge) via a TypeScript implementation that is behaviorally identical to the .NET reference engine.

Detailed implementation plan: See docs/typescript-port-plan.md

Lazy evaluation (prerequisite — .NET engine) ✅

LazyArrayValue wraps IEnumerable<IElwoodValue> without materializing; streaming operators (where, select, selectMany, take, skip, distinct, index) return lazy arrays
$[*] wildcard returns lazy array (critical for short-circuit)
Only materializing operators call .ToList(): orderBy, groupBy, count, sum, min, max, last, batch, join, reduce
take(n) short-circuits — take(1) on 100K items: 19ms vs 400ms+ before
first short-circuits without materializing
Pipeline endpoints materialize via ToConcreteValue() for final JSON output
4 benchmark tests with results logged to tests/Elwood.Core.Tests/Benchmarks/results.log (last 10 runs)
All 95 tests pass (91 conformance + 4 benchmarks)

Pre-restructure cleanup ✅

All 48 explanation files cleaned — legacy system references replaced with generic terms
Sample data and examples updated
Source code comments cleaned

Repo restructure ✅

68 test cases extracted to spec/test-cases/{name}/ (per-directory: script.elwood, input.json, expected.json, explanation.md)
.NET code moved under dotnet/ (dotnet/src/, dotnet/tests/, dotnet/Elwood.slnx)
FileBasedTests.cs updated to discover from spec/test-cases/
CLAUDE.md updated with new structure
97/97 tests pass after restructure

TypeScript implementation (in progress)

Keeping in sync (ongoing)

New features follow the workflow: spec test case first → implement in .NET → implement in TS → update docs. CI runs both conformance suites on every PR.

Phase 1c — Publish & Package

Goal: Open-source release of Elwood with both implementations, CI, and package distribution.

Depends on: Phase 1b complete (repo is in final structure, both implementations pass conformance suite).

Open source ✅

Published to GitHub: https://github.com/max-favilli/elwood
README.md with quick start, examples, CLI download links, playground screenshot
LICENSE file (MIT)

CI/CD ✅

GitHub Actions CI: build + test on push (both .NET and TS) — .github/workflows/ci.yml
Conformance gate: both implementations must pass all spec/test-cases/
Automated release workflow (.github/workflows/release.yml) — triggered on version tags
v0.1.0 released with native binaries for Windows, macOS, Linux

.NET distribution ✅

NuGet package metadata (Elwood.Core, Elwood.Json — multi-target net8.0;net10.0)
global.json pinning SDK version
dotnet tool configured (dotnet tool install --global Elwood.Cli → elwood command)
Native AOT binaries on GitHub Release (linux-x64, macos-x64, win-x64)
Publish NuGet packages to nuget.org — Elwood.Core, Elwood.Json, Elwood.Pipeline, Elwood.Cli, Elwood.Xlsx, Elwood.Parquet (v0.3.0)

TypeScript distribution ✅

npm package published: @elwood-lang/core on npmjs.com
Browser compatibility documented (crypto limitation in known-issues.md)

Playground ✅

Browser-based interactive playground: https://max-favilli.github.io/elwood/
Monaco Editor with Elwood syntax highlighting, autocomplete, snippet placeholders
Example gallery (68 examples from spec/test-cases/) with search, categories, rendered markdown explanations
File loading (local file picker + drag-and-drop, stays in browser)
Shareable links via lz-string URL compression
GitHub Pages deployment via GitHub Actions (.github/workflows/deploy-playground.yml)
Spec: docs/playground-spec.md

API container

Elwood.Api project — minimal ASP.NET API: POST /api/evaluate + /health
Dockerfile (multi-stage, Alpine-based)
GitHub Actions workflow to build and push container on release tags (.github/workflows/docker.yml)
Usage: docker run -p 8080:8080 ghcr.io/max-favilli/elwood-api
Enables integration with any iPaaS via HTTP

Remaining engine work

iterate(seed, fn) and takeWhile — lazy sequence generation
.parseJson() — deserialize embedded JSON strings
Identify and port any remaining common JSON transformation functions not yet covered

Phase 2 — Elwood Scripts as Maps + Multi-Format I/O

Goal: Use pure Elwood scripts (.elwood files) as the transformation format, with format conversion for non-JSON inputs/outputs. No YAML maps.

Design decision: Scripts, not YAML maps

An earlier design proposed YAML documents where the tree structure mirrors the output shape and leaf values are Elwood expressions. This was abandoned for two reasons:

Deep indentation — A 20-level deep output (common in SAP IDocs, complex XML) means 40+ spaces of indent before the content. YAML's indentation-as-structure makes depth visible and painful.
Two syntaxes in one file — YAML for structure + Elwood for values means no editor can fully syntax-highlight, autocomplete, or error-check both. Every YAML-embedded-DSL (GitHub Actions + bash, Helm + Go templates) suffers from this — the embedded language gets zero tooling.

Pure Elwood scripts solve both problems:

One syntax → full editor support (highlighting, autocomplete, error reporting)
let decomposition flattens deep nesting into named, testable pieces
memo handles repeated patterns (e.g., 82 SAP segments → 30 one-liner function calls)
Already works today with elwood run script.elwood --input data.json

Script-based maps

// artmas09.elwood — replaces a 6500-line JSON map

let ausprt = memo (src, charName) => {
  "@SEGMENT": "1", FUNCTION: "009", CHAR_NAME: charName,
  MATERIAL_LONG: $.SAP_STUFF.MATERIAL_LONG,
  CHAR_VALUE_LONG: src.CHAR_VALUE_LONG
}

let ediDc40 = { "@SEGMENT": "1", TABNAM: "EDI_DC_40", MANDT: "400", ... }
let matHead = { "@SEGMENT": "1", MATL_TYPE: $.SAP_STUFF.MATL_TYPE, ... }

return {
  ARTMAS09: {
    IDOC: {
      "@BEGIN": "1",
      EDI_DC40: ediDc40,
      E1BPE1MATHEAD: matHead,
      E1BPE1AUSPRT: [
        ausprt($.C8_PRODUCTSUBTYPE, "C8_PRODUCTSUBTYPE"),
        ausprt($.JW_PRODUCTGROUP, "JW_PRODUCTGROUP"),
        // ... 30 more one-liners instead of 82 copy-pasted map nodes
      ]
    }
  }
}

Format conversion

Elwood operates on JSON internally. Format converters handle non-JSON inputs/outputs:

any format ──→ [input conversion] ──→ JSON ──→ Elwood script ──→ JSON ──→ [output conversion] ──→ any format

Format	Input (→ JSON)	Output (JSON →)
JSON	Native (no conversion)	Native (no conversion)
CSV	Rows → array of objects	Array of objects → rows
XML	Elements → objects, attributes → `@` properties	Objects → elements
XLSX	Sheet → array of objects	Array of objects → sheet
Text	Content as string or line-split array	Join array / render string

Two ways to use format conversion:

CLI flags — for simple cases where the entire input is one format and the entire output is another:

elwood run transform.elwood --input data.csv --input-format csv
elwood run transform.elwood --input data.xml --output result.csv --output-format csv

In-script functions — for full control within the script (custom options, mixed formats, converting mid-pipeline):

// Parse CSV with custom delimiter
let orders = $.rawCsv | fromCsv({ delimiter: ";", headers: true })

// Transform
let result = orders | where(.amount > 100) | select({ id: .id, total: .amount })

// Output as CSV
return result | toCsv()

// Parse XML, transform, output as JSON
let items = $.xmlPayload | fromXml()
return items.catalog.products | select({ sku: .@id, name: .title })

// Mix formats in one script
let products = $.csvData | fromCsv({ headers: true })
let categories = $.xmlCategories | fromXml()
return products | select(p => {
  ...p,
  categoryName: categories.list[*] | first(c => c.@id == p.catId) | select(.name)
})

Format functions

Function	Description	Options	Status
`fromCsv(options?)`	Parse CSV string → array of objects	`delimiter`, `headers`, `quote`, `skipRows`, `parseJson`	✅
`toCsv(options?)`	Array of objects → CSV string	`delimiter`, `headers`, `alwaysQuote`	✅
`fromXml(options?)`	Parse XML string → JSON object	`attributePrefix`, `stripNamespaces`	✅
`toXml(options?)`	JSON object → XML string	`rootElement`, `attributePrefix`, `declaration`	✅
`fromXlsx(options?)`	Parse XLSX (base64) → array of objects	`headers`, `sheet` Extension	✅
`toXlsx(options?)`	Array of objects → XLSX (base64)	`headers`, `sheet` Extension	✅
`fromParquet(options?)`	Parse Parquet (base64) → array of objects	Extension	✅
`toParquet(options?)`	Array of objects → Parquet (base64)	`schema`, `compression` Extension, .NET only	✅
`fromText(options?)`	Split text into lines or structured data	`delimiter` (default `\n`)	✅
`toText(options?)`	Join array into text	`delimiter` (default `\n`)	✅

Tasks

Phase 2b — Performance ✅

Goal: Ensure Elwood outperforms comparable JSONPath-based transformation engines.

Result: .NET interpreter is 2x faster than legacy baseline

Benchmarked in-process on 100K rows (fair comparison, same machine, no HTTP overhead):

Test	Elwood .NET	Legacy baseline	Speedup
`where active \| select name`	121ms	240ms	2.0x faster
`select` with toString + charArray concat	836ms	1,819ms	2.2x faster

Why the interpreter is fast enough

Lazy evaluation via LazyArrayValue streams items through pipeline stages without materializing intermediate arrays
LINQ integration — .NET's optimized Where/Select/Take on IEnumerable avoids per-item allocation
The JIT compiler already optimizes the hot interpreter loop after warm-up

Compiled mode (explored, removed)

Expression Tree compilation was implemented and tested but provided no speedup over the interpreter. The interpreter's lazy streaming is more efficient than compiled fused loops that materialize arrays. The compiled mode was removed to reduce complexity.

TypeScript performance

Benchmarked on 100K rows — the TS interpreter is already ~5x faster than .NET:

Test	.NET	TypeScript
where+select name	121ms	24ms
toString + charArray concat	836ms	173ms

V8's JIT aggressively optimizes native array methods (filter, map). Generator-based lazy evaluation is unnecessary — the eager approach with V8-optimized array ops is faster.

Future optimization (if needed)

Bypass IElwoodValue abstraction for direct JsonNode access in .NET hot paths — only if 100MB+ workloads need it

Phase 3 — Integration Pipeline Configuration

Goal: Use YAML to describe complete data integration pipelines — sources, transformations, and destinations — in a single document. YAML handles declarative orchestration (triggers, connections, destinations); Elwood scripts (.elwood files) handle transformation logic.

Design: hybrid approach

Real-world integration configs embed complex expressions in many YAML values — multi-line filter chains, conditional logic, method chains, etc. This creates the same two-syntax-in-one-file problem we rejected for transformation maps.

Solution: YAML for structure, external .elwood scripts for any non-trivial expression.

Inline in YAML (OK):

Static values: trigger: http, concurrency: 100, container: output
Simple paths: $.request.season, $.code
Short interpolation: `{$.request.season}-images`

External .elwood file (required for):

Anything with pipes (|)
Conditionals (if/then/else)
Method chains (.toUpper().split()...)
Filters (where, in)
Multi-line logic

Guideline (not enforced): simple $.field, short interpolation, or brief expressions stay inline. Complex logic with multiple pipes, conditionals, or long method chains goes in an external .elwood file. This is a recommendation — short pipes like $.items[*] | take(5) are fine inline. A future elwood validate command could warn (not error) when inline expressions exceed a complexity threshold.

Example

# pipeline.elwood.yaml
version: 2

sources:
  - name: api-trigger
    trigger: http
    endpoint: /api/data/{category}
    contentType: json
    map: request-map.elwood                # ← external script

  - name: file-source
    trigger: pull
    from:
      fileShare:
        connectionString: ${FILE_SHARE_CONN}
        path: /{$.request.category}/data   # ← simple interpolation, OK inline
    map: source-transform.elwood           # ← external script

join:
  path: $
  keys: []

outputs:
  - name: publish-to-fileshare
    path: filter-results.elwood            # ← complex filter → external script
    outputId: output-id.elwood             # ← method chains → external script
    contentType: $.contentType             # ← simple path, OK inline
    concurrency: 100                       # ← static value
    map: output-map.elwood                 # ← external script (reusable!)
    destinations:
      fileShare:
        - connectionString: ${FS_CONN}
          filename: output-filename.elwood # ← complex interpolation → external

  - name: publish-to-sftp
    path: filter-results.elwood            # ← same filter, reused!
    outputId: output-id.elwood             # ← same ID logic, reused!
    contentType: $.contentType
    concurrency: 50
    map: output-map.elwood                 # ← same map, reused!
    destinations:
      sftp:
        - connectionString: ${SFTP_CONN}
          filename: output-filename.elwood # ← same filename logic, reused!

External scripts are reusable — the same output-id.elwood and filter-results.elwood are shared across multiple outputs. They're also independently testable via the CLI.

What stays in YAML vs external scripts

In YAML (inline)	Inline Elwood (simple)	External .elwood (complex)
`trigger: http`	`path: $.results[*]`	`path: filter-active.elwood`
`contentType: json`	`contentType: $.contentType`	`outputId: generate-id.elwood`
`concurrency: 100`	`endpoint: /api/{$.category}`	`map: transform.elwood`
`container: output`	`filename: /{$.code}.json`	`filename: build-path.elwood`

Guideline: static config → plain YAML. Simple expressions → inline. Complex logic → external .elwood file. This is a best practice, not enforced. Short inline pipes are fine when readable.

Core concepts

IDM (Intermediate Data Model): shared JSON document built progressively by sources, consumed by outputs. Sources → IDM → Outputs → Destinations.

Script bindings: named root variables set by the executor, available in .elwood scripts:

Binding	Available in	What it contains
`$`	Source maps	Raw source payload
`$`	Output maps	Current fan-out slice
`$source`	Everywhere	Source metadata (trigger, headers, eventId)
`$idm`	Source maps (after first source)	Current IDM state from previous sources
`$idm`	Output maps	Complete IDM
`$output`	Output maps	Full array from `path` (all slices)
`$secrets`	YAML properties	Secret references loaded from provider

Scripts that don't need metadata work identically in the playground and in a pipeline — $ is always the data.

Fan-out: both sources and outputs support path — slices the IDM, processes once per slice with optional concurrency.

Full schema reference: docs/pipeline-yaml-reference.md

Design consideration: staged execution (deferred)

The current model is strictly sources → IDM → outputs: all sources complete before any output runs. This works well for most integrations, but some real-world flows need to deliver data mid-pipeline — for example, archiving the raw payload to blob storage before enrichment, or calling an API and delivering its response to a file THEN using that result to call a second API.

Today, these cases require splitting into multiple chained pipelines (pipeline A outputs to a queue, pipeline B triggers from the queue). This works but spreads a single logical flow across multiple configs, making it harder to reason about, test, and monitor end-to-end.

A staged execution model would allow interleaving sources and outputs within a single pipeline:

stages:
  - sources:
      - name: trigger
        trigger: http
    outputs:
      - name: archive-raw
        destinations:
          blob:
            - container: raw-payloads

  - sources:
      - name: enrich
        trigger: pull
        from:
          http:
            url: https://api.example.com/enrich
            body: $.request
    outputs:
      - name: api-response
        response: true

Each stage runs in order. Within a stage, sources build the IDM, then outputs consume it. The IDM accumulates across stages. The key distinction between sources and outputs is preserved: a source fetches from ONE place and writes to the IDM; an output delivers from the IDM to ONE OR MORE destinations (fan-out on the delivery side).

Status: deferred. The current sources→outputs model handles the common case. If real-world integrations repeatedly require chaining multiple pipelines for what is logically one flow, this design will be revisited. Evidence from actual pipeline authoring will inform the decision.

Executor model — three levels

Executor	Purpose	Sources	Destinations
CLI Executor	Development + testing with saved payloads	Local files (one per named source)	Local files
Sync Executor	End-to-end local execution, connects to real sources	HTTP calls, file shares, queues	Real destinations
Cloud Executors (Azure, AWS)	Production, distributed, async	Triggers + pull sources	Real destinations

All three share the same pipeline parser, script resolver, and transformation engine. They differ only in how they acquire source data and deliver outputs.

CLI Executor usage:

# Single source
elwood pipeline run pipeline.elwood.yaml --source api-trigger=payload.json

# Multi-source — provide envelope files with source metadata
elwood pipeline run pipeline.elwood.yaml \
  --source api-trigger=trigger-envelope.json \
  --source product-api=product-response.json

# Outputs written to local files (stdout or --output-dir)

Envelope file format (for CLI executor):

{
  "source": {
    "name": "api-trigger",
    "trigger": "http",
    "eventId": "evt-abc-123",
    "http": { "method": "POST", "headers": { "X-Correlation-Id": "corr-789" } }
  },
  "payload": {
    "orders": [{ "id": 1, "active": true }]
  }
}

The executor splits it: $ = envelope.payload, $source = envelope.source. Plain JSON files (no envelope) are also accepted — $ = the file content, $source = minimal defaults.

Tasks

Step 1 — Pipeline YAML schema + parser: ✅

Step 2 — CLI Executor: ✅

elwood pipeline run <yaml> --source name=file command
--source-envelope name=file for explicit envelope files (no auto-detection)
--output-dir writes each output as {name}.json
elwood pipeline validate <yaml> — validates YAML, scripts, dependencies, duplicates
5 pipeline test scenarios (single source, multi-source merge, XML→CSV, fan-out, depends chain)
Pipeline conformance test runner (discovers spec/pipelines/*)
21 CLI integration tests (including 6 pipeline tests)

Step 3 — State + persistence: ✅

ExecutionState model: per-execution, per-source, per-output step state with timestamps
IStateStore + IDocumentStore interfaces
InMemoryStateStore + InMemoryDocumentStore implementations
FileSystemStateStore + FileSystemDocumentStore for persistent local state
PipelineExecutor tracks state automatically when stores are provided
elwood pipeline status [state-dir] — shows recent executions with step details
--output-dir persists state to .state/ subdirectory

Step 4 — Sync Executor: ⚠️ partial

Step 5 — Deployment + Runtime API: ⚠️ partial

Step 6 — Cloud Executors (separate packages):

Broken into seven sub-steps. The cloud runtime supports two execution modes via a mode: field in pipeline YAML — sync (HTTP function runs end-to-end and returns one designated output) and async (HTTP function returns 202 + executionId, fan-out via Service Bus).

Step 6a — Schema additions + Azure storage adapters: ✅

Step 6b — GitPipelineStore: ✅

GitPipelineStore wraps the git CLI — every save = git commit, GetRevisions = git log, RestoreRevision = git show + commit
GitHelper utility: thin wrapper around the git CLI (no LibGit2Sharp — avoids native binary issues on newer .NET)
Backed by any git remote (Azure DevOps, GitHub, GitLab, local bare repo) — the store manages commits, the API server manages push/pull
11 tests: CRUD, revision history with limits, restore (including script add/removal), author tracking, empty repo safety

Step 6c — AsyncExecutor (step-at-a-time, queue-driven): ✅

AsyncExecutor — step-at-a-time execution engine for mode: async pipelines
- StartAsync(pipeline, payload) — creates state, stores trigger payload + pipeline content in IDocumentStore, queues stage 0 sources
- ExecuteStepAsync(message) — processes one source or output step per invocation
IStepQueue interface + InMemoryStepQueue for tests (Service Bus impl ships in 6d)
StepMessage model (ExecutionId, PipelineId, StepType, StepName, StageIndex)
Fan-in via idempotent steps: after completing a source, checks all sources in stage; duplicates are no-ops
Stage plan stored in IDocumentStore so queue workers can reload independently
Pipeline content stored in IDocumentStore — workers are stateless, no local git clone
8 tests: start + state creation, source processing, output processing, multi-stage ordering, concurrent sources, idempotency, failure handling, end-to-end

Step 6d — Elwood.Runtime.Azure Functions project:

BlobPipelineStore — implements IPipelineStore over Azure Blob Storage (folder-per-pipeline layout in a blob container). Replaces FileSystemPipelineStore for serverless environments where there is no persistent disk.
HTTP trigger (catch-all): matches route via RedisPipelineRegistry, dispatches to SyncExecutor (sync) or starts execution + queues first step (async)
Service Bus queue trigger: invokes AsyncExecutor.ExecuteStepAsync(...) (async mode only — not needed for sync-only pipelines)
DI wiring: AddElwoodAzureStorage(...) + Functions startup
HTTP method support added to SourceConfig schema
Route pattern matching with parameter extraction (e.g. /api/{category})

Step 6e — Pipeline deployment flow:

How pipelines move from editing to live execution. Two modes, detailed in docs/pipeline-deployment-modes.md:

Mode 1 (save = live): Portal saves → API writes to Blob Storage → updates Redis → pipeline is live immediately. Git commit as optional audit trail. Good for solo developers and dev/test environments.
Mode 2 (save = draft, merge = live): Portal saves to a git branch → submits PR → review + merge → webhook triggers API → updates Blob + Redis → pipeline goes live. Good for teams and production. Requires: portal branch awareness, PR creation, merge webhook handler.
elwood deploy <yaml> CLI command — uploads pipeline to blob + updates Redis
Git commit on save (Mode 1 audit trail) — optional, configured per environment
Merge webhook handler (Mode 2) — receives git merge events, updates blob + Redis

Step 6f — Terraform:

infra/azure/main.tf — Storage Account (pipelines container + Functions runtime) + Redis + Function App
Support reusing existing resources via data blocks (resource group, App Service Plan, App Configuration, Application Insights)
infra/azure/variables.tf — resource names, region, SKU, existing resource references
infra/azure/outputs.tf — function URL, storage connection, Redis connection
infra/azure/terraform.tfvars.example — sample values (real tfvars gitignored)
Getting started documentation for terraform apply → deploy → test

Step 6g — End-to-end integration test:

docker-compose: Azurite + Redis + Service Bus emulator
Run a real pipeline through the Functions runtime locally
Verify state, documents, fan-out, route matching all work end-to-end

Step 6 — AWS Executor (later):

Lambda + SQS + DynamoDB + S3 adapters (separate package: Elwood.Pipeline.Aws)
AWS executor Functions equivalent (Lambda functions package)

Infrastructure (separate repo: elwood-infra):

Terraform module: Azure (Function App + ASB + Storage + App Insights)
Terraform module: AWS (Lambda + SQS + DynamoDB + S3)
Example configurations (minimal, production)

Phase 3b — Elwood Management Portal

Goal: A web-based management UI for authoring, deploying, testing, and monitoring Elwood integration pipelines. Re-engineered from an existing enterprise integration frontend, adapted for Elwood's pipeline YAML + script architecture.

Tech stack: Next.js + React + Tailwind + Monaco Editor + Redux (separate repo: elwood-portal)

Implementation prompt: See docs/prompts/implement-portal.md

Pipeline Authoring

Browse, search, create, edit pipeline YAML files (.elwood.yaml)
Browse, search, create, edit Elwood scripts (.elwood) with full Monaco syntax highlighting + autocomplete
Version history with diff view and restore (via GitPipelineStore)
Deploy pipelines to runtime
Bulk upload (ZIP of pipeline + scripts)
Validation: elwood validate integrated in the editor

Transformation Testing

Run .elwood scripts against input data with live preview
Load payloads from execution history or file upload
Format-aware input: JSON, CSV, XML, Text with conversion preview
Compare input vs output side-by-side

Execution Monitoring

Pipeline execution dashboard (reads from IStateStore)
Real-time activity log with filtering (by pipeline, status, time range)
Execution detail view: steps, fan-out progress, errors, duration
Correlation tracing across multi-step async flows
Health status indicator for the runtime

Document & State Inspection

Browse execution state (reads from IStateStore)
Browse stored documents / IDM (reads from IDocumentStore)
Download payloads and outputs

Administration

Role-based access control (Admin, Editor, Viewer)
Authentication (Azure AD / MSAL)
Multi-environment support (dev, staging, production)

Playground & Examples

Integrated Playground — run Elwood expressions in the browser via @elwood-lang/core npm package
86 official examples bundled from spec/test-cases/ (static JSON, build-time)
Example sidebar with search, categories, click-to-load
API-backed examples (refinement): replace static bundle with GET /api/examples endpoint. Official examples embedded as assembly resources in Elwood.Pipeline (read-only, updated on NuGet upgrade). Organization examples stored server-side via POST/PUT/DELETE /api/examples/{id} (same CRUD pattern as pipelines). Portal shows two sections: "Elwood" (official) and "Organization" (user-managed). Enables each adopting organization to maintain their own shared example library alongside the official ones.

Tasks (detailed WBS to be defined later)

Phase 4 — Developer Tooling & Ecosystem

Goal: IDE support, developer tools, and community ecosystem.

IDE support

VS Code extension: syntax highlighting for .elwood and .elwood.yaml files
Language server (LSP): autocomplete, go-to-definition for let bindings, error squiggles
Hover documentation for built-in methods
Snippet library for common patterns

Developer tools

Playground multi-format expansion: input/output format selectors, conversion preview, dual-view output (see playground-spec.md Phase 2+ section)
Trace mode: elwood run --trace shows step-by-step pipeline execution with intermediate values
Visual debugger: step through pipe stages, inspect values at each step
Schema inference: analyze an Elwood expression and infer the expected input/output JSON schema

Ecosystem

Documentation site (GitHub Pages or similar)
Example library: real-world transformation patterns
Community contributions: custom function plugins

Phase 5 — Elwood DB

Goal: A JSON database where you store documents and query them with Elwood. No predefined schema, no document size limits. PostgreSQL as the storage backend.

Detailed design: See docs/architecture-vision.md Phase 5 Vision section.

What makes it unique

Elwood as the native query language — the same language for querying and transforming, no context switch
No document size limits — automatic splitting handles 100MB+ documents
Schema-on-write — define structure when storing, not upfront
PostgreSQL backend — we build a query translation layer, not a database engine

Storage

Fixed schema: 2 tables (collections + chunks), no dynamic table creation
Large documents are split along a configured path (e.g., $.orders[*] → one row per order)
PostgreSQL JSONB indexing (GIN + expression indexes) handles query performance
Elwood queries translate to SQL, with non-SQL operations (.toUpper(), complex select) executed in Elwood on the result set

Tasks

Execution Order

Phase 1   ✅  Build the .NET engine
   ↓
Phase 1b  ✅  Lazy evaluation (.NET) → repo restructure → TypeScript port
   ↓
Phase 1c  ✅  Publish everything (GitHub, NuGet, npm, CI, Playground, API container)
   ↓
Phase 2   ✅  Multi-format I/O (fromCsv, toXml, etc.) + script-based maps
   ↓
Phase 2b  ✅  Performance — 2x faster than legacy baseline (compiled mode explored, not needed)
   ↓
Phase 3       Integration pipeline configuration (Elwood Runtime + Executors)
   ↓
Phase 3b      Elwood Management Portal (web UI for authoring, testing, monitoring)
   ↓
Phase 4       IDE support, developer tools, ecosystem
   ↓
Phase 5       Elwood DB — JSON database with Elwood queries (separate repo)

Phase 3b is a separate repo (elwood-portal). Phase 5 is a separate repo (elwood-db).

Adoption Strategy

Elwood is designed for incremental adoption:

Phase 1 (done): Standalone transformation engine. Use via CLI or .NET library.
Phase 1b (done): Cross-platform reach. Use in browsers, Node.js, edge runtimes.
Phase 1c (done): Open-source launch. Available via NuGet, npm, browser playground, and self-hosted API container.
Phase 2 (done): Multi-format I/O. All formats complete. Extension API for XLSX. CLI format flags.
Phase 2b (done): Performance verified — 2x faster than legacy baseline on 100K rows. Compiled mode explored but interpreter is already optimal.
Phase 3: Integration pipelines. YAML defines sources, transforms, and destinations. Pluggable executors run them.
Phase 3b: Management portal. Web UI for authoring pipelines, testing transformations, monitoring executions.
Phase 4: IDE support, developer tools, and community ecosystem.
Phase 5: Elwood DB. Store JSON, query with Elwood. PostgreSQL backend, no size limits.

Each phase is independently useful. No phase requires adopting a later one.

How to use Elwood

{elwood}

  As a library       →  NuGet: Elwood.Core / Elwood.Json
                        npm: @elwood-lang/core

  As a CLI tool      →  Download from GitHub Releases (Windows, macOS, Linux)
                        Or: dotnet tool install --global Elwood.Cli

  In the browser     →  Playground: https://max-favilli.github.io/elwood/

  As an API          →  docker run -p 8080:8080 ghcr.io/max-favilli/elwood-api
                        POST /api/evaluate { script, input } → result
                        (integrates with any iPaaS via HTTP)

FilesExpand file tree

roadmap.md

Latest commit

History

roadmap.md

File metadata and controls

Elwood Roadmap

Vision

Phase 1 — Standalone DSL Engine ✅ (Complete)

Phase 1b — Cross-Platform (TypeScript Port)

Lazy evaluation (prerequisite — .NET engine) ✅

Pre-restructure cleanup ✅

Repo restructure ✅

TypeScript implementation (in progress)

Keeping in sync (ongoing)

Phase 1c — Publish & Package

Open source ✅

CI/CD ✅

.NET distribution ✅

TypeScript distribution ✅

Playground ✅

API container

Remaining engine work

Phase 2 — Elwood Scripts as Maps + Multi-Format I/O

Design decision: Scripts, not YAML maps

Script-based maps

Format conversion

Format functions

Tasks

Phase 2b — Performance ✅

Result: .NET interpreter is 2x faster than legacy baseline

Why the interpreter is fast enough

Compiled mode (explored, removed)

TypeScript performance

Future optimization (if needed)

Phase 3 — Integration Pipeline Configuration

Design: hybrid approach

Example

What stays in YAML vs external scripts

Core concepts

Design consideration: staged execution (deferred)

Executor model — three levels

Tasks

Phase 3b — Elwood Management Portal

Pipeline Authoring

Transformation Testing

Execution Monitoring

Document & State Inspection

Administration

Playground & Examples

Tasks (detailed WBS to be defined later)

Phase 4 — Developer Tooling & Ecosystem

IDE support

Developer tools

Ecosystem

Phase 5 — Elwood DB

What makes it unique

Storage

Tasks

Execution Order

Adoption Strategy

How to use Elwood