hw/gen: --format json, the generator's findings for tools

changes/feat-hwgen-json-diagnostics.md

@@ -0,0 +1,3 @@
+### Added (the generator's output for tools, 2026-11-13)
+
+- `emb-hwgen validate`, `generate` and `check` take `--format json`, which prints one JSON document for an IDE or a script (`hw/schemas/diagnostics.schema.json`). It gives the status and exit code, every error and warning with its file, line, node path and message as separate fields, the diff of each outdated generated file, and the files written ([hwgen](docs/engineering/hwgen.md)). A failure of the generator itself still yields a document, with exit status 3. EmbLink Studio's first stage reads it.

docs/engineering/hwgen.md

@@ -15,9 +15,45 @@ The generator needs Python 3 and PyYAML, the configuration tool's one dependency
 
 - Exit status 2 means a description error. Every error is printed as `file:line: node.path: message`, and all errors found are reported, sorted.
 - `check` exits 1 when a committed output differs from what the descriptions give, and prints the diff.
+- Exit status 3 means the generator itself failed: a bug to report, not a description error.
 
 Never edit `generated/` by hand (HW-003). Change the YAML and regenerate.
 
+### Output for tools: `--format json`
+
+`validate`, `generate` and `check` take `--format json`. The command then prints a single JSON document on standard output and nothing else, for an IDE such as EmbLink Studio or a script. `hw/schemas/diagnostics.schema.json` defines the document. The exit status is the same as for text output. An application's `--system` takes it too. When `generate` leaves a merge conflict in an edited file (A126), the document's status is `stale`, exit 1, with an HW-003 finding for each such file; `check` also counts the system descriptions it checked (`systems`).
+
+```sh
+python3 -I hw/gen/emb-hwgen.py validate --board boards/st/nucleo_f446re --format json
+```
+
+With `baud: fast` in the NUCLEO-F446RE's description, it prints:
+
+```json
+{
+  "tool": "emb-hwgen", "version": "0.1.0", "format": 1, "command": "validate",
+  "status": "errors", "exit_code": 2, "files": [],
+  "diagnostics": [
+    {"severity": "error", "rule": "HW-002", "file": "boards/st/nucleo_f446re/board.yaml", "line": 26,
+     "path": "console.baud", "message": "expected integer, found string 'fast'",
+     "text": "boards/st/nucleo_f446re/board.yaml:26: console.baud: expected integer, found string 'fast'"}
+  ]
+}
+```
+
+- **`status`** is one of four values:
+  - `ok` (exit 0): warnings are still possible;
+  - `errors` (exit 2): the descriptions are wrong;
+  - `stale` (exit 1): `check` found outdated outputs, or hardware facts outside a description;
+  - `internal-error` (exit 3): the document carries `error`, the exception's type and message, and the traceback goes to standard error.
+- **Each finding** gives its file, line, node path and message as separate fields. The file is relative to the repository root, or absolute for a board kept outside it. It also keeps the line the text output prints, in `text`.
+- **`rule`** says which requirement the finding concerns:
+  - HW-002: a description error or warning;
+  - HW-001: a hardware fact outside a description;
+  - HW-003: a generated file out of date. These findings carry `diff`, a unified diff from the committed file to the generator's output.
+- **`files`** lists what `generate` wrote. **`boards`** counts what `check` checked.
+- **`format`** is the document's own version. A new field keeps it; a changed or removed field bumps it.
+
 ## Files
 
 | Path | Kind | Content |

hw/gen/emb-hwgen.py

@@ -13,25 +13,32 @@
     emb-hwgen.py check --board-root ~/my-project   # the boards a project keeps outside the tree
     emb-hwgen.py validate --board boards/st/nucleo_f446re
     emb-hwgen.py generate --system examples/controller/system.yaml   # an application (A125)
+    emb-hwgen.py validate --board boards/st/nucleo_f446re --format json   # for tools and IDEs
 
-Exit status 0 on success, 1 when check finds a difference, 2 on a description error; each
-error is printed as 'file:line: node.path: message' (HW-002).
+Exit status 0 on success, 1 when check finds a difference, 2 on a description error, 3 when the
+generator itself fails; each error is printed as 'file:line: node.path: message' (HW-002). With
+--format json the output is one JSON document instead (hw/schemas/diagnostics.schema.json): the
+status, the exit code, each finding with its file, line, node path and message apart, and the
+files written; nothing else goes to standard output.
 """
 import argparse
 import difflib
 import os
 import sys
+import traceback
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parent.parent.parent
 sys.path.insert(0, str(Path(__file__).resolve().parent))
-from emb_hwgen import HwError, facts, generate, writer  # noqa: E402
+from emb_hwgen import HwError, diagnostics, facts, generate, writer  # noqa: E402
 
 
 def _boards(root):
     return sorted(p.parent.parent.parent for p in root.glob("boards/*/*/generated/include/hw_config.h"))
 
 
+CONFLICT = "an unresolved merge conflict (<<<<<<< yours ... >>>>>>> generated now)"
+
 _SKIP = {"build", "third_party", ".git", "node_modules"}
 
 
@@ -78,12 +85,19 @@ def _target(args):
 
 def cmd_generate(args):
     """Each file written over what is there, edits kept (A126): created, unchanged, rewritten,
-    merged with its edits, or a conflict for its author to resolve."""
+    merged with its edits, or a conflict for its author to resolve. With --format json, the files
+    written and, for each conflict, an HW-003 finding (status stale, exit 1)."""
     placements, warnings, _ = _target(args)
     outcomes = {}
     for path, text, base in placements:
         outcome = writer.write(path, text, base)
         outcomes.setdefault(outcome, []).append(path)
+    if args.format == "json":
+        written = sorted(os.path.relpath(path, ROOT) for kind in ("created", "rewritten", "merged", "conflict")
+                         for path in outcomes.get(kind, []))
+        conflicts = [diagnostics.stale(os.path.relpath(path, ROOT), CONFLICT) for path in outcomes.get("conflict", [])]
+        code = 1 if conflicts else 0
+        return _emit(args, "stale" if code else "ok", code, _warnings(warnings) + conflicts, written)
     for w in warnings:
         print(f"warning: {w}", file=sys.stderr)
     for outcome in ("merged", "conflict"):
@@ -102,11 +116,14 @@ def cmd_check(args):
     """Every baseline is what the descriptions give now (HW-003), no conflict is left, and nothing
     in a generated directory is an output of no description. With --strict, a board's files are
     exactly their baseline: a board's facts belong in its description (A126)."""
-    bad = 0
     roots = [Path(r) for r in args.board_root] or [ROOT]
+    found = []
     for r in roots:
         if not (r / "boards").is_dir():
-            print(f"emb-hwgen: {r} has no boards/ directory (a board root holds boards/<vendor>/<board>/)", file=sys.stderr)
+            message = f"{r} has no boards/ directory (a board root holds boards/<vendor>/<board>/)"
+            if args.format == "json":
+                return _emit(args, "errors", 2, [diagnostics.finding(message, "error", None)])
+            print(f"emb-hwgen: {message}", file=sys.stderr)
             return 2
     boards = [b for r in roots for b in _boards(r)]
     systems = [s for r in roots for s in _systems(r)]
@@ -124,70 +141,101 @@ def cmd_check(args):
             have = path.read_text(encoding="utf-8") if path.is_file() else None
             baseline = base.read_text(encoding="utf-8") if base.is_file() else None
             if have is None:
-                bad += 1
-                print(f"{rel}: missing; run emb-hwgen.py generate {how}")
+                found.append(diagnostics.stale(rel, f"missing; run emb-hwgen.py generate {how}"))
             elif baseline != text:
-                bad += 1
-                print(f"{rel}: its description changed since it was generated; run emb-hwgen.py generate {how}")
-                sys.stdout.writelines(difflib.unified_diff((baseline or "").splitlines(True), text.splitlines(True),
-                                                           f"{rel} (generated then)", f"{rel} (generated now)"))
+                diff = "".join(difflib.unified_diff((baseline or "").splitlines(True), text.splitlines(True),
+                                                    f"{rel} (generated then)", f"{rel} (generated now)"))
+                found.append(diagnostics.stale(
+                    rel, f"its description changed since it was generated; run emb-hwgen.py generate {how}", diff))
             elif writer.has_conflict(have):
-                bad += 1
-                print(f"{rel}: an unresolved merge conflict (<<<<<<< yours ... >>>>>>> generated now)")
+                found.append(diagnostics.stale(rel, CONFLICT))
             elif strict and have != text:
-                bad += 1
-                print(f"{rel}: edited; a board's facts belong in its description (check --strict)")
-                sys.stdout.writelines(difflib.unified_diff(text.splitlines(True), have.splitlines(True),
-                                                           f"{rel} (generated)", f"{rel} (committed)"))
+                diff = "".join(difflib.unified_diff(text.splitlines(True), have.splitlines(True),
+                                                    f"{rel} (generated)", f"{rel} (committed)"))
+                found.append(diagnostics.stale(rel, "edited; a board's facts belong in its description (check --strict)",
+                                               diff))
         if gen.is_dir():
             for path in sorted(p for p in gen.rglob("*") if p.is_file() and ".base" not in p.relative_to(gen).parts):
                 if path not in expected:
-                    bad += 1
-                    print(f"{os.path.relpath(path, ROOT)}: not an output of emb-hwgen; remove it")
+                    found.append(diagnostics.stale(os.path.relpath(path, ROOT), "not an output of emb-hwgen; remove it"))
+    bad = len(found)
     # HW-001 for the tree, or for each board root: its boards hold no hand-written files, and its
     # own SoC families' C no address-like constant (A84)
     problems = [p for r in roots for p in facts.check(r)]
+    code = 1 if bad or problems else 0
+    if args.format == "json":
+        found += [diagnostics.finding(p, "error", "HW-001", with_path=False) for p in problems]
+        return _emit(args, "stale" if code else "ok", code, found, boards=len(boards), systems=len(systems))
+    for f in found:  # the file and what to do, then the diff where there is one
+        print(f["text"])
+        if f.get("diff"):
+            sys.stdout.write(f["diff"])
     for p in problems:
         print(p)
     print(f"emb-hwgen check: {len(boards)} board(s), {len(systems)} system(s), {bad} stale file(s), "
           f"{len(problems)} hardware fact(s) outside a description")
-    return 1 if bad or problems else 0
+    return code
 
 
 def cmd_validate(args):
     _, warnings, what = _target(args)
+    if args.format == "json":
+        return _emit(args, "ok", 0, _warnings(warnings))
     for w in warnings:
         print(f"warning: {w}", file=sys.stderr)
     print(f"emb-hwgen: {what} is valid")
     return 0
 
 
+def _warnings(warnings):
+    return [diagnostics.finding(w, "warning", "HW-002") for w in warnings]
+
+
+def _emit(args, status, code, findings=(), files=(), boards=None, error=None, systems=None):
+    doc = diagnostics.document(args.cmd, status, code, findings, files, boards, error)
+    if systems is not None:
+        doc["systems"] = systems
+    sys.stdout.write(diagnostics.dumps(doc))
+    return code
+
+
 def main(argv=None):
     ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    fmt = argparse.ArgumentParser(add_help=False)
+    fmt.add_argument("--format", choices=("text", "json"), default="text",
+                     help="text (default), or one JSON document for tools (hw/schemas/diagnostics.schema.json)")
     sub = ap.add_subparsers(dest="cmd", required=True)
-    g = sub.add_parser("generate", help="write a board's or an application's generated outputs")
+    g = sub.add_parser("generate", parents=[fmt], help="write a board's or an application's generated outputs")
     which = g.add_mutually_exclusive_group(required=True)
     which.add_argument("--board", help="the board directory, holding board.yaml")
     which.add_argument("--system", help="an application's system.yaml (SPEC-014 §5, A125)")
     g.add_argument("--out", help="output directory (default: generated/ beside the board.yaml or system.yaml)")
-    c = sub.add_parser("check", help="regenerate every board and compare with the committed outputs (HW-003), "
+    c = sub.add_parser("check", parents=[fmt], help="regenerate every board and compare with the committed outputs (HW-003), "
                        "and look for hardware facts outside the descriptions (HW-001)")
     c.add_argument("--strict", action="store_true",
                    help="a board's files must be exactly what was generated: no edits (the tree's boards, in CI)")
     c.add_argument("--board-root", action="append", default=[],
                    help="check the boards under DIR/boards/ instead of the tree's (repeatable)")
-    v = sub.add_parser("validate", help="validate a board and its SoC, or a system, without writing anything")
+    v = sub.add_parser("validate", parents=[fmt],
+                       help="validate a board and its SoC, or a system, without writing anything")
     which = v.add_mutually_exclusive_group(required=True)
     which.add_argument("--board")
     which.add_argument("--system")
     args = ap.parse_args(argv)
     try:
         return {"generate": cmd_generate, "check": cmd_check, "validate": cmd_validate}[args.cmd](args)
     except HwError as e:
+        if args.format == "json":
+            return _emit(args, "errors", 2, [diagnostics.finding(line, "error", "HW-002") for line in e.errors])
         for line in e.errors:
             print(line, file=sys.stderr)
         print(f"emb-hwgen: {len(e.errors)} error(s)", file=sys.stderr)
         return 2
+    except Exception as e:  # the generator's own fault: a tool still gets a document, the trace goes to stderr
+        if args.format != "json":
+            raise
+        traceback.print_exc()
+        return _emit(args, "internal-error", 3, error={"type": type(e).__name__, "message": str(e)})
 
 
 if __name__ == "__main__":

hw/gen/emb_hwgen/diagnostics.py

@@ -0,0 +1,62 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Junior Deogracias
+#
+# This file is part of EmbLinkRTOS, a deterministic real-time kernel for microcontrollers.
+# Licensed under the Apache License, Version 2.0; see the LICENSE file at the repository root.
+
+"""The generator's findings as data, for tools that read them (`--format json`, HW-002).
+
+Every error and warning the generator raises is a line 'file:line: node.path: message' (HW-002);
+the HW-001 check's problems are 'file[:line]: message'. Here each becomes a record with those
+parts apart, and a command's outcome one JSON document (hw/schemas/diagnostics.schema.json): what
+ran, its status and exit code, its findings, and the files it wrote. A line that does not have the
+shape keeps its whole text as the message, so nothing is lost.
+"""
+import json
+import re
+
+from . import VERSION
+
+FORMAT = 1  # the document's own version: a field added keeps it, one changed or removed bumps it
+
+# a Windows drive is part of the file name; a node path has no spaces, and its segments may be
+# compatibles (bindings.st,stm32f4-iwdg.class)
+_LINE = re.compile(r"^(?P<file>(?:[A-Za-z]:)?[^:\n]+?)(?::(?P<line>\d+))?: "
+                   r"(?:(?P<path>\(root\)|\(file\)|[A-Za-z_][\w,+-]*(?:\.[\w,+-]+|\[\d+\])*): )?"
+                   r"(?P<message>.+)$", re.S)
+_FACT = re.compile(r"^(?P<file>(?:[A-Za-z]:)?[^:\n]+?)(?::(?P<line>\d+))?: (?P<message>.+)$", re.S)
+
+
+def finding(text, severity, rule, with_path=True):
+    """One finding from its line of text; with_path false for the HW-001 check's lines, which name
+    no node."""
+    m = (_LINE if with_path else _FACT).match(text)
+    if not m:
+        return _record(severity, rule, None, None, None, text, text)
+    return _record(severity, rule, m["file"], m["line"], m.groupdict().get("path"), m["message"], text)
+
+
+def _record(severity, rule, file, line, path, message, text):
+    return {"severity": severity, "rule": rule, "file": file, "line": int(line) if line else None,
+            "path": path, "message": message, "text": text}
+
+
+def stale(file, message, diff=None):
+    """A generated file that differs from what the generator writes now (HW-003)."""
+    rec = _record("error", "HW-003", file, None, None, message, f"{file}: {message}")
+    rec["diff"] = diff
+    return rec
+
+
+def document(command, status, exit_code, findings=(), files=(), boards=None, error=None):
+    doc = {"tool": "emb-hwgen", "version": VERSION, "format": FORMAT, "command": command,
+           "status": status, "exit_code": exit_code, "diagnostics": list(findings), "files": list(files)}
+    if boards is not None:
+        doc["boards"] = boards
+    if error is not None:
+        doc["error"] = error
+    return doc
+
+
+def dumps(doc):
+    return json.dumps(doc, indent=2, sort_keys=True) + "\n"