-
Notifications
You must be signed in to change notification settings - Fork 62
Add basic browser tool for agentic env gen interactive editing #803
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
qianl-nv
wants to merge
17
commits into
main
Choose a base branch
from
qianl/dev/agentic_browser
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
34a2a15
Add graph review app for UnresolvedArenaEnvGraphSpec
qianl-nv 0807491
fix copyright years
qianl-nv 07c9d7f
Move folder
qianl-nv f84bb1c
Simplify setup
qianl-nv 19ab3df
Refactor graph review tool into review_gui package.
qianl-nv 8d5ecab
Rename review_gui render module to dashboard.
qianl-nv a173200
Reorder review GUI dashboard: nodes, graph with unary sidebar, tasks.
qianl-nv ab92515
Add prompt-driven generation to the review GUI and defer pxr on asset…
qianl-nv e7579e5
Remove redundant to_yaml from ArenaEnvGraphSpecBase.
qianl-nv 846594d
Revert asset pxr import deferral.
qianl-nv 97c82d7
Revert fetch_intent_from_prompt split.
qianl-nv 2a42552
Move review GUI launcher to gui_runner.py.
qianl-nv 3dbd1ce
Add missing docstrings to review GUI panel renderers.
qianl-nv 804bf5d
Add module docstring to review GUI styles.
qianl-nv 0f77a33
Make prompt-only mode the default in launcher docs.
qianl-nv 1f77d6e
Add missing docstrings in review GUI streamlit_ui.
qianl-nv 54410ac
Simplify YAML validation in the review GUI editor.
qianl-nv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
91 changes: 91 additions & 0 deletions
91
isaaclab_arena_examples/agentic_environment_generation/gui_runner.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """CLI launcher for the ArenaEnvInitialGraphSpec live editor. | ||
|
|
||
| Spawns Streamlit with :mod:`~isaaclab_arena_examples.agentic_environment_generation.review_gui.streamlit_ui`. | ||
|
|
||
| Usage: | ||
| # Default — prompt-only (empty editor until you generate or paste YAML): | ||
| /isaac-sim/python.sh -m isaaclab_arena_examples.agentic_environment_generation.gui_runner | ||
|
|
||
| # Open an existing spec: | ||
| /isaac-sim/python.sh -m isaaclab_arena_examples.agentic_environment_generation.gui_runner \\ | ||
| --yaml isaaclab_arena/tests/test_data/pick_and_place_maple_table_init_env_graph.yaml | ||
|
|
||
| # Custom port: | ||
| /isaac-sim/python.sh -m isaaclab_arena_examples.agentic_environment_generation.gui_runner \\ | ||
| --yaml <path> --port 8600 | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import os | ||
| import subprocess | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| _REVIEW_GUI_DIR = Path(__file__).resolve().parent / "review_gui" | ||
|
|
||
|
|
||
| def main() -> None: | ||
| parser = argparse.ArgumentParser( | ||
| description=__doc__, | ||
| formatter_class=argparse.RawDescriptionHelpFormatter, | ||
| ) | ||
| parser.add_argument( | ||
| "--yaml", | ||
| type=Path, | ||
| default=None, | ||
| help="Optional ArenaEnvInitialGraphSpec YAML to open in the editor.", | ||
| ) | ||
| parser.add_argument( | ||
| "--port", | ||
| type=int, | ||
| default=8501, | ||
| help="Streamlit server port (default: 8501).", | ||
| ) | ||
| args = parser.parse_args() | ||
| serve_live_editor(args.yaml, port=args.port) | ||
|
|
||
|
|
||
| def serve_live_editor(yaml_path: Path | None, port: int = 8501) -> None: | ||
| """Spawn ``streamlit run streamlit_ui.py`` and wait.""" | ||
| app_path = _REVIEW_GUI_DIR / "streamlit_ui.py" | ||
| if not app_path.exists(): | ||
| raise FileNotFoundError(f"Streamlit app not found at {app_path} — installation is incomplete.") | ||
|
|
||
| cmd = [ | ||
| sys.executable, | ||
| "-m", | ||
| "streamlit", | ||
| "run", | ||
| str(app_path), | ||
| "--server.port", | ||
| str(port), | ||
| "--browser.gatherUsageStats", | ||
| "false", | ||
| "--server.fileWatcherType", | ||
| "none", | ||
| "--", | ||
| ] | ||
| if yaml_path is not None: | ||
| cmd.extend(["--yaml", str(yaml_path.resolve())]) | ||
|
|
||
| print(f"[review_gui] launching Streamlit live editor: {' '.join(cmd)}", file=sys.stderr) | ||
| try: | ||
| subprocess.run(cmd, env=os.environ.copy(), check=True) | ||
| except FileNotFoundError as exc: | ||
| raise SystemExit( | ||
| "Streamlit is not installed. Inside the isaaclab_arena container run:\n" | ||
| " python -m pip install --user --ignore-installed streamlit streamlit-ace" | ||
| ) from exc | ||
| except KeyboardInterrupt: | ||
| pass | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
6 changes: 6 additions & 0 deletions
6
isaaclab_arena_examples/agentic_environment_generation/review_gui/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """Live editor and HTML dashboard for :class:`~isaaclab_arena.environments.arena_env_graph_spec.ArenaEnvInitialGraphSpec`.""" |
6 changes: 6 additions & 0 deletions
6
isaaclab_arena_examples/agentic_environment_generation/review_gui/render/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """HTML rendering backend for the initial-graph review dashboard.""" |
60 changes: 60 additions & 0 deletions
60
isaaclab_arena_examples/agentic_environment_generation/review_gui/render/dashboard.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import html as html_lib | ||
|
|
||
| from isaaclab_arena.environments.arena_env_graph_spec import ArenaEnvInitialGraphSpec | ||
| from isaaclab_arena_examples.agentic_environment_generation.review_gui.render.mermaid_graph import render_mermaid_graph | ||
| from isaaclab_arena_examples.agentic_environment_generation.review_gui.render.panels import ( | ||
| render_node_cards, | ||
| render_tasks_table, | ||
| render_unary_constraints, | ||
| ) | ||
| from isaaclab_arena_examples.agentic_environment_generation.review_gui.render.styles import DASHBOARD_CSS | ||
|
|
||
|
|
||
| def render_dashboard_html(spec: ArenaEnvInitialGraphSpec) -> str: | ||
| """Render the self-contained review dashboard HTML for ``spec``.""" | ||
| initial_state = spec.initial_state_spec | ||
| return f"""<!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="utf-8"> | ||
| <title>{html_lib.escape(spec.env_name)} — graph review</title> | ||
| <script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script> | ||
| <style>{DASHBOARD_CSS}</style> | ||
| </head> | ||
| <body> | ||
| <header> | ||
| <h1>{html_lib.escape(spec.env_name)}</h1> | ||
| <p class="sub">{len(spec.nodes)} nodes · {len(spec.tasks)} tasks · initial state: <code>{html_lib.escape(initial_state.id)}</code></p> | ||
| </header> | ||
| <main> | ||
| <section class="panel nodes-panel"> | ||
| <h2>Nodes</h2> | ||
| <div class="node-grid">{render_node_cards(spec)}</div> | ||
| </section> | ||
| <section class="panel graph-panel"> | ||
| <h2>Spatial graph <span class="muted">(initial state: <code>{html_lib.escape(initial_state.id)}</code>)</span></h2> | ||
| <div class="graph-row"> | ||
| <div class="graph-mermaid"> | ||
| <pre class="mermaid">{render_mermaid_graph(spec, initial_state)}</pre> | ||
| </div> | ||
| <aside class="graph-unary"> | ||
| {render_unary_constraints(initial_state)} | ||
| </aside> | ||
| </div> | ||
| </section> | ||
| <section class="panel tasks-panel"> | ||
| <h2>Tasks</h2> | ||
| {render_tasks_table(spec)} | ||
| </section> | ||
| </main> | ||
| <script>mermaid.initialize({{ startOnLoad: true, theme: 'dark', themeVariables: {{ fontFamily: 'ui-monospace, monospace' }} }});</script> | ||
| </body> | ||
| </html> | ||
| """ |
96 changes: 96 additions & 0 deletions
96
isaaclab_arena_examples/agentic_environment_generation/review_gui/render/mermaid_graph.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import re | ||
|
|
||
| from isaaclab_arena.environments.arena_env_graph_spec import ArenaEnvInitialGraphSpec | ||
| from isaaclab_arena.environments.arena_env_graph_types import ArenaEnvGraphStateSpec | ||
|
|
||
| _MERMAID_ID_SAFE = re.compile(r"[^A-Za-z0-9_]") | ||
|
|
||
|
|
||
| def render_mermaid_graph(spec: ArenaEnvInitialGraphSpec, state: ArenaEnvGraphStateSpec) -> str: | ||
| """Emit a left-to-right mermaid graph of spatial and task constraints. | ||
|
|
||
| Binary spatial constraints (reference is set) are drawn as solid edges: | ||
| subject -->|kind| reference | ||
|
|
||
| Unary spatial constraints (no reference) are omitted from the graph and | ||
| listed to its right by :func:`render_unary_constraints` so their params are | ||
| visible. | ||
|
|
||
| Task constraints with a child are drawn as dashed edges: | ||
| parent -.->|type| child | ||
|
|
||
| object_reference nodes are drawn with a dotted edge to their parent node: | ||
| ref_node -. ref .-> parent_node | ||
| """ | ||
| lines = ["graph LR"] | ||
|
|
||
| anchor_ids: set[str] = set() | ||
| edge_nodes: set[str] = set() | ||
|
|
||
| for constraint in state.spatial_constraints: | ||
| kind = constraint.kind | ||
| if kind == "is_anchor": | ||
| anchor_ids.add(constraint.subject) | ||
| if constraint.reference is not None: | ||
| lines.append( | ||
| f" {_mermaid_id(constraint.subject)}[{_mermaid_label(constraint.subject)}]" | ||
| f" -->|{kind}| " | ||
| f"{_mermaid_id(constraint.reference)}[{_mermaid_label(constraint.reference)}]" | ||
| ) | ||
| edge_nodes.add(constraint.subject) | ||
| edge_nodes.add(constraint.reference) | ||
|
|
||
| for task_constraint in state.task_constraints: | ||
| if task_constraint.child is not None: | ||
| lines.append( | ||
| f" {_mermaid_id(task_constraint.parent)}[{_mermaid_label(task_constraint.parent)}]" | ||
| f" -.->|{_mermaid_label(task_constraint.type.value)}| " | ||
| f"{_mermaid_id(task_constraint.child)}[{_mermaid_label(task_constraint.child)}]" | ||
| ) | ||
| edge_nodes.add(task_constraint.parent) | ||
| edge_nodes.add(task_constraint.child) | ||
|
|
||
| for node in spec.nodes: | ||
| if node.id not in edge_nodes: | ||
| lines.append(f" {_mermaid_id(node.id)}[{_mermaid_label(node.id)}]") | ||
|
|
||
| nodes_by_id = spec.nodes_by_id | ||
| for node in spec.nodes: | ||
| if node.type.value == "object_reference" and node.parent is not None: | ||
| if node.parent in nodes_by_id: | ||
| lines.append(f" {_mermaid_id(node.id)} -.->|ref| {_mermaid_id(node.parent)}") | ||
|
|
||
| for anchor_id in anchor_ids: | ||
| lines.append(f" style {_mermaid_id(anchor_id)} fill:#3a7d44,color:#fff,stroke:#7fd17f,stroke-width:2px") | ||
|
|
||
| type_palette = { | ||
| "background": ("#3a4f7a", "#7aa0d8"), | ||
| "embodiment": ("#7a3a3a", "#d87a7a"), | ||
| "object": ("#7a6b3a", "#d8c47a"), | ||
| "object_reference": ("#6b3a7a", "#c47ad8"), | ||
| "lighting": ("#3a7a7a", "#7ad8d8"), | ||
| } | ||
| for node in spec.nodes: | ||
| if node.id in anchor_ids: | ||
| continue | ||
| fill, stroke = type_palette.get(node.type.value, ("#3a3d44", "#888")) | ||
| lines.append(f" style {_mermaid_id(node.id)} fill:{fill},color:#fff,stroke:{stroke}") | ||
|
|
||
| return "\n".join(lines) | ||
|
|
||
|
|
||
| def _mermaid_id(value: str) -> str: | ||
| """Mermaid node identifiers must be alphanumeric / underscore.""" | ||
| return _MERMAID_ID_SAFE.sub("_", value) | ||
|
|
||
|
|
||
| def _mermaid_label(value: str) -> str: | ||
| """Escape mermaid-significant characters inside node labels.""" | ||
| return value.replace('"', """).replace("|", "|") |
83 changes: 83 additions & 0 deletions
83
isaaclab_arena_examples/agentic_environment_generation/review_gui/render/panels.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| # Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md). | ||
| # All rights reserved. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import html as html_lib | ||
| import yaml | ||
|
|
||
| from isaaclab_arena.environments.arena_env_graph_spec import ArenaEnvInitialGraphSpec | ||
| from isaaclab_arena.environments.arena_env_graph_types import ArenaEnvGraphNodeSpec, ArenaEnvGraphStateSpec | ||
| from isaaclab_arena_examples.agentic_environment_generation.review_gui.render.thumbnails import ( | ||
| render_placeholder_thumbnail, | ||
| ) | ||
|
|
||
|
|
||
| def render_unary_constraints(state: ArenaEnvGraphStateSpec) -> str: | ||
| """List constraints without a reference beside the spatial graph.""" | ||
| rows = [] | ||
| for constraint in state.spatial_constraints: | ||
| if constraint.reference is not None: | ||
| continue | ||
| params = ( | ||
| " <code" | ||
| f' class="muted">{html_lib.escape(yaml.safe_dump(constraint.params, default_flow_style=True).rstrip())}</code>' | ||
| if constraint.params | ||
| else "" | ||
| ) | ||
| rows.append( | ||
| f'<li><span class="badge type-{html_lib.escape(constraint.kind)}">{html_lib.escape(constraint.kind)}</span>' | ||
| f" on <code>{html_lib.escape(constraint.subject)}</code>{params}</li>" | ||
| ) | ||
| if not rows: | ||
| return '<p class="muted unary-empty"><em>No unary constraints.</em></p>' | ||
| return ( | ||
| f'<h3 class="unary-heading">Unary constraints <span class="muted">({len(rows)})</span></h3>' | ||
| f'<ul class="unary-list">{"".join(rows)}</ul>' | ||
| ) | ||
|
|
||
|
|
||
| def render_tasks_table(spec: ArenaEnvInitialGraphSpec) -> str: | ||
|
qianl-nv marked this conversation as resolved.
|
||
| """Render task rows as an HTML table for the dashboard tasks panel.""" | ||
| if not spec.tasks: | ||
| return "<p class='muted'><em>No tasks defined.</em></p>" | ||
| rows = [] | ||
| for index, task in enumerate(spec.tasks): | ||
| params_str = yaml.safe_dump(task.params, sort_keys=False).rstrip() if task.params else "(empty)" | ||
| description = html_lib.escape(task.description or "") | ||
| rows.append( | ||
| "<tr>" | ||
| f"<td><code>{index}</code></td>" | ||
| f'<td><span class="badge type-task">{html_lib.escape(task.kind)}</span></td>' | ||
| f"<td>{description}</td>" | ||
| f"<td><pre>{html_lib.escape(params_str)}</pre></td>" | ||
| "</tr>" | ||
| ) | ||
| return ( | ||
| "<table class='tasks'>" | ||
| "<thead><tr><th>#</th><th>kind</th><th>description</th><th>params</th></tr></thead>" | ||
| f"<tbody>{''.join(rows)}</tbody>" | ||
| "</table>" | ||
| ) | ||
|
|
||
|
|
||
| def render_node_cards(spec: ArenaEnvInitialGraphSpec) -> str: | ||
|
qianl-nv marked this conversation as resolved.
|
||
| """Render one card per graph node for the dashboard nodes panel.""" | ||
| return "\n".join(render_node_card(node) for node in spec.nodes) | ||
|
|
||
|
|
||
| def render_node_card(node: ArenaEnvGraphNodeSpec) -> str: | ||
|
qianl-nv marked this conversation as resolved.
|
||
| """Render a single node card with placeholder thumbnail and YAML dump.""" | ||
| node_dict = node.model_dump(mode="json", exclude_none=True) | ||
| node_yaml = yaml.safe_dump(node_dict, sort_keys=False).rstrip() | ||
| thumb = render_placeholder_thumbnail(node) | ||
| return f"""<article class="node-card type-{html_lib.escape(node.type.value)}"> | ||
| {thumb} | ||
| <div class="node-meta"> | ||
| <div class="node-id">{html_lib.escape(node.id)}</div> | ||
| <span class="badge type-{html_lib.escape(node.type.value)}">{html_lib.escape(node.type.value)}</span> | ||
| </div> | ||
| <pre class="node-yaml">{html_lib.escape(node_yaml)}</pre> | ||
| </article>""" | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.