Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
388 lines (313 loc) · 14.1 KB

ARCHITECTURE.md

File metadata and controls

388 lines (313 loc) · 14.1 KB

Architecture

This document describes the system architecture of the Shiny AI Assistant.

System Overview

The Shiny AI Assistant is a multi-agent system that enables natural language interaction with R Shiny dashboards. It consists of four main layers:

  1. Widget Layer - React chat interface embedded in the dashboard
  2. Server Layer - Next.js API that orchestrates AI agents
  3. Bridge Layer - Platform-agnostic protocol for dashboard interaction
  4. Adapter Layer - Shiny-specific implementation
┌─────────────────────────────────────────────────────────────────────┐
│                    Widget Layer (packages/widget)                    │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────────────────┐  │
│  │ ChatPanel    │  │ MessageList  │  │ QuickActionsBar           │  │
│  │ FloatingBtn  │  │ InputArea    │  │ ToolCallIndicator         │  │
│  └──────────────┘  └──────────────┘  └───────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  │ HTTP POST + SSE Streaming
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Server Layer (packages/server)                    │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                       Orchestrator                            │   │
│  │   Router Agent → classifyIntent() → Specialist Agent         │   │
│  └──────────────────────────────────────────────────────────────┘   │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐       │
│  │ Explain │ │Navigate │ │ Action  │ │ Locate  │ │  Data   │       │
│  │  Agent  │ │  Agent  │ │  Agent  │ │  Agent  │ │  Agent  │       │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘       │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  │ Tool Execution
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Bridge Layer (packages/bridge)                    │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    DashboardBridge Interface                  │   │
│  │   introspect() | navigateTo() | executeAction() | queryData() │   │
│  └──────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  │ Shiny Messages
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Adapter Layer (Shiny)                             │
│  ┌──────────────────────────┐  ┌─────────────────────────────────┐  │
│  │  JavaScript (widget)     │  │  R (shinyAIAssistant)           │  │
│  │  sendToR() / onMessage() │  │  aiAssistantHandler()           │  │
│  └──────────────────────────┘  └─────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

Agent System

Router Agent

The Router Agent classifies user intent and routes to specialist agents.

Intents:

  • explain - Questions about charts, metrics, data
  • navigate - Requests to go to pages/tabs
  • action - Requests to execute actions (set filters, export)
  • locate - Requests to find/highlight UI elements
  • data - Analytical queries requiring data access
  • tour - Requests for guided dashboard tours
  • clarify - Ambiguous requests needing clarification

Implementation: packages/agents/src/router/index.ts

Specialist Agents

Each agent uses Claude Haiku 4.5 for fast responses via Vercel AI SDK.

Agent Purpose Tools
ExplainAgent Answer questions, analyze images None (streaming)
NavigateAgent Navigate dashboard navigateTo, searchNavigation
ActionAgent Execute dashboard actions executeAction, getAvailableActions
LocateAgent Find and highlight elements searchComponents, highlightElement
DataAgent Query data, provide analytics queryData, getDataSchema
TourAgent Generate guided tours getComponents, getCurrentContext

Implementation: packages/agents/src/specialists/

Bridge Protocol

The Bridge Protocol defines a platform-agnostic interface for AI-dashboard interaction.

Capability Levels

type CapabilityLevel = 'level-0' | 'level-1' | 'level-2' | 'level-3';
Level Features
level-0 Introspection, navigation, highlighting
level-1 Enhanced metadata via manifest
level-2 Action execution
level-3 Data queries

DashboardBridge Interface

interface DashboardBridge {
  // Level 0 - Always available
  introspect(): Promise<DashboardState>;
  getComponents(): Promise<ComponentDescriptor[]>;
  getNavigationStructure(): Promise<NavigationNode[]>;
  getCurrentContext(): Promise<DashboardContext>;
  navigateTo(path: string): Promise<NavigationResult>;
  highlightElement(selector: string, options?: HighlightOptions): Promise<void>;
  scrollToElement(selector: string): Promise<void>;
  searchComponents(query: string): Promise<ComponentMatch[]>;
  searchNavigation(query: string): Promise<NavigationMatch[]>;

  // Level 2 - Actions
  supportsActions(): boolean;
  getAvailableActions(): Promise<ActionDescriptor[]>;
  executeAction(action: DashboardAction): Promise<ActionResult>;
  validateAction(action: DashboardAction): Promise<ValidationResult>;

  // Level 3 - Data
  supportsDataQueries(): boolean;
  getDataSchema(): Promise<DataSchema>;
  queryData(query: DataQuery): Promise<DataResult>;

  // Events
  onEvent(event: DashboardEventType, handler: EventHandler): Unsubscribe;
}

Implementation: packages/bridge/src/protocol/interface.ts

Key Types

DashboardContext - Current state for AI awareness:

interface DashboardContext {
  currentPage: string;
  visibleComponents: ComponentDescriptor[];
  activeFilters: Array<{ id: string; value: unknown; label?: string }>;
}

DashboardAction - Action execution request:

interface DashboardAction {
  type: string;        // Action type ID (e.g., "set_filter")
  params: Record<string, unknown>;
  targetComponent?: string;
}

DataQuery - Data query specification:

interface DataQuery {
  source: string;
  select: string[];
  filters?: QueryFilter[];
  groupBy?: string[];
  aggregations?: QueryAggregation[];
  orderBy?: QueryOrderBy[];
  limit?: number;
}

Shiny Adapter

The Shiny Adapter implements the bridge protocol for R Shiny dashboards.

JavaScript Side (Widget)

Shiny Detection:

function isShinyAvailable(): boolean;
function isShinyInitialized(): boolean;

Binding Discovery:

function discoverInputBindings(): ShinyBinding[];
function discoverOutputBindings(): ShinyBinding[];
function discoverAllComponents(): ComponentDescriptor[];

Action Execution:

// Send action to R
sendToR(action: DashboardAction);

// Listen for results from R
registerMessageHandler('__ai_assistant_result__', handler);

Implementation: packages/bridge/src/shiny/

R Side (Package)

Widget Function:

aiAssistantWidget(
  apiUrl,                    # Required: Chat API endpoint
  position = "bottom-right", # Widget position
  theme = "auto",            # light | dark | auto
  manifest = NULL,           # Path to manifest YAML
  dataSources = NULL,        # Level 3 data source config
  enableDataQueries = FALSE  # Enable Level 3
)

Handler Function:

aiAssistantHandler(session, handlers = list(
  navigate_to = function(params) { ... },
  set_filter = function(params) { ... },
  set_date_range = function(params) { ... },
  reset_filters = function(params) { ... },
  query_data = function(params) { ... }
))

Implementation: packages/r-shiny/R/

Widget Architecture

The widget is built with React and Tailwind CSS.

Component Hierarchy

Widget (main container)
├── FloatingButton (toggle button)
└── ChatPanel (main interface)
    ├── MessageList
    │   └── MessageItem (per message)
    │       ├── ToolCallIndicator
    │       └── AudioPlayer (TTS)
    ├── QuickActionsBar
    └── InputArea
        ├── ImagePreview
        └── TranscriptionToggle (STT)

Key Hooks

Hook Purpose
useChat Message state, sending messages
useSSEStream Server-Sent Events parsing
useSession Session management
useQuickActions Quick action buttons
useToolExecution Bridge tool execution
useTour Guided tour state
useSpeechRecognition Voice input
useTextToSpeech Voice output

Providers

<WidgetProvider config={...}>
  <BridgeProvider>
    <Widget />
  </BridgeProvider>
</WidgetProvider>

Implementation: packages/widget/src/

Server Architecture

The server is a Next.js application with API routes.

API Endpoints

Endpoint Method Purpose
/api/chat POST Main chat endpoint
/api/chat/tool-result POST Tool execution results
/api/transcribe POST Speech-to-text
/api/speak POST Text-to-speech

Chat Flow

  1. Widget sends message via POST to /api/chat
  2. Server returns SSE stream
  3. Orchestrator classifies intent via Router Agent
  4. Specialist agent processes request
  5. Agent may invoke tools (sent as SSE events)
  6. Widget executes tools via bridge
  7. Tool results sent back via /api/chat/tool-result
  8. Final response streamed to widget

Session Management

Sessions track:

  • Message history
  • Active stream handlers
  • Rate limiting state

Implementation: packages/server/src/

Manifest System

The manifest provides enhanced metadata for Level 1+ capabilities.

Structure

name: "Dashboard Name"
version: "1.0.0"
description: "Dashboard description"
capabilityLevel: "level-3"

aiInstructions: |
  Custom instructions for the AI assistant.

glossary:
  - term: "ARR"
    definition: "Annual Recurring Revenue"
    synonyms: ["annual revenue"]

navigation:
  - id: "overview"
    label: "Overview"
    path: "/overview"
    description: "Executive summary"

pages:
  - id: "overview"
    title: "Overview"
    components:
      - id: "revenue_chart"
        label: "Revenue Chart"
        type: "chart"
        dataBinding:
          source: "sales_data"
          fields: ["date", "arr"]

dataSources:
  - id: "sales_data"
    name: "Sales Data"
    fields:
      - name: "arr"
        type: "number"
        aggregatable: true

actions:
  - id: "set_filter"
    name: "Set Filter"
    description: "Apply a filter"
    category: "update"
    parameters:
      - name: "filterId"
        type: "string"
        required: true

Implementation: packages/bridge/src/manifest/

Data Flow

Message Flow

User Input → Widget → Server → Router Agent → Specialist Agent
                                    ↓
              Tool Call ←── Agent Response
                 ↓
    Widget → Bridge → Shiny Adapter → R Handler
                 ↓
              Result → Server → Agent → Response → Widget

Tool Execution Flow

  1. Agent requests tool call (e.g., executeAction)
  2. Server sends tool-call SSE event
  3. Widget receives and executes via bridge
  4. Bridge sends Shiny message to R
  5. R handler processes and returns result
  6. Result sent back through chain
  7. Agent incorporates result in response