Add ESP32-S3 portable AI companion firmware with ethical network security scanning, BLE networking, full config portal, and persistent memory & RAG

[Copilot]

Implements full firmware and companion server for Debbie — a voice-controlled portable AI assistant on Freenove Media Kit ESP32-S3 (FNK0102), interfacing with OpenAI Realtime API and a custom agent backend. Adds ethical network security scanning as a first-class voice-commanded feature, multi-network WiFi, Bluetooth Low Energy networking, a comprehensive tabbed configuration portal, and a full memory and retrieval-augmented generation (RAG) system for persistent, context-aware conversations.

Firmware (ESP-IDF C)

• openai_client — WebSocket client to OpenAI Realtime API (gpt-4o-realtime-preview); bidirectional PCM16 audio, function-call dispatch with correct call_id propagation; OAI_EVT_USER_TRANSCRIPT event captures user speech separately from AI responses; save_memory tool registered in every session so the AI can proactively store facts
• audio_manager — I2S mic capture with VAD (configurable threshold), I2S speaker output, beep tones
• camera_manager — OV2640 JPEG capture → base64 for vision API
• display_manager — LVGL animated avatar (eyes, speech bubble, state-driven colour)
• wifi_manager / web_server — STA+AP dual mode; multi-network fallback (up to 3 saved SSIDs tried in priority order on boot); captive portal HTML config UI; NVS persistence
• bluetooth_manager — BLE GATT server with the official Nordic UART Service (128-bit NUS UUIDs 6E40000x-B5A3-F393-E0A9-E50E24DCCA9E); NUS RX accepts JSON config commands identical to the HTTP /configure endpoint; advertises as "Debbie" so any BLE Serial app (nRF Toolbox, Serial Bluetooth Terminal) can configure the device without joining WiFi
• memory_manager (new) — Short-term RAM circular buffer of the last N turns (configurable, default 20); up to 50 long-term key-value facts persisted to NVS flash; prompt enrichment injects facts + recent turns into every OpenAI session; HTTP-based companion RAG queries retrieve semantically relevant past context before each response
• notification_client — WebSocket client to companion server for WhatsApp/email/agent push notifications
• network_scanner — WiFi AP scan, ARP host discovery, TCP port scan (65+ ports), banner grabbing, MAC OUI lookup, HTTP probe, per-host risk scoring (0–100)
• vuln_reporter — Banner signature matching, port-based CVE checks (Modbus, Docker API, Redis, WEP/open WiFi, etc.), HTML/JSON/voice report formats, device self-assessment
• self_agent — Full agent tools: internet HTTP GET, DNS resolution, NVD CVE API lookup, system introspection; OpenAI function-call dispatcher registering all tools as session tools

Full Tabbed Configuration Portal

Accessible at http://192.168.4.1/ (Debbie AP) or via BLE Serial:

Tab	Contents
📶 Network	Primary + 2 fallback WiFi SSIDs/passwords, BLE enable/disable + device name, companion server URL
🤖 AI & LLM	Provider cards: OpenAI (GPT-4o, o1), Anthropic (Claude 3.5), Groq (Llama 3.3, Mixtral), OpenRouter (100+ models), Ollama / LM Studio (local, private); per-provider model dropdowns and API key fields; custom agent endpoint
🎀 Personality	Name, system prompt (with reset-to-default), voice style (friendly / professional / playful / calm), response length (brief / normal / detailed)
🔔 Notifications	Master toggle + per-source switches (WhatsApp, Email, Spotify)
⚙️ Advanced	Speaker volume, VAD threshold, camera enable/disable, live device status, factory reset
🧠 Memory	Enable/disable memory, enable/disable companion RAG, configurable turn retention (5–50), live stats (turn count, fact count, RAG status), clear all memory

Companion Server (Node.js)

• network_tools.js — nmap wrapper (restricted to RFC 1918 ranges), NVD CVE API client, HTML vulnerability report generator with full HTML escaping; report served from server-stored state (GET /network/report) rather than reflecting request body
• memory_store.js (new) — SQLite database (via better-sqlite3) with an FTS5 full-text search table for all conversation turns and a key-value facts table; REST API: GET /memory/query (RAG retrieval), POST /memory/turn, POST /memory/fact, GET /memory/stats, DELETE /memory/clear; graceful degradation if module is not installed; automatic pruning at 500 turns
• server.js — express-rate-limit on all sensitive routes (including memory endpoints); WebSocket handler stores device scan results server-side; mounts memory routes
• spotify.js — Rate-limited OAuth routes; generic error on auth failure (no exception reflection into HTML)

Memory & RAG Architecture

• Short-term: Last 20 turns (user + AI) kept in RAM; NVS-synced every 4 turns and on shutdown; reloaded on boot
• Long-term facts: Up to 50 key-value facts in NVS flash; the AI calls save_memory(key, value, importance) automatically when users share personal context
• Companion RAG: Full conversation history stored in companion SQLite/FTS5; semantically relevant past turns retrieved and injected before each AI response; turns silently synced to companion in the background
• Voice commands: "Remember that my name is Alice" → save_memory tool call; "What do you know about me?" → facts injected into context; "Forget everything" → memory_clear

Security constraints

• nmap scans restricted to private ranges (10.x, 172.16–31.x, 192.168.x) — public internet blocked at the route layer
• Strict nmap flag allowlist prevents flag injection
• All scan-derived values HTML-escaped before report rendering
• Authorised-use-only warnings throughout firmware and docs
• BLE NUS uses standard UUIDs; no unauthenticated remote code execution surface

Voice commands (examples)

"Scan my network"              → WiFi + ARP + port scan + risk report
"Check 192.168.1.1"            → Targeted port scan + vuln analysis
"Look up CVE-2021-44228"       → NVD query, CVSS score, description
"What's your status?"          → CPU, heap, WiFi RSSI, uptime
"Fetch the BBC news"           → HTTP GET → AI summary
"Read my messages"             → Pending WhatsApp/email notifications
"Remember that I prefer jazz"  → Fact saved to NVS + companion DB
"What do you know about me?"   → Facts + RAG context injected into response

Original prompt

looking at this repository https://github.com/openai/openai-realtime-embedded i have the hardware Freenove Media Kit for ESP32-S3 https://www.freenove.com/FNK0102 reference etc here, but i want to inteface with my own agent (see my repo https://github.com/magicalmutation-coder/D3881E) idealy i want ti to be able to interface with all the features i want it to be able to send camera images ,... hold a chat etc .. act as a pager for my whatsapp and email and other notifications from my agent. if not have all the same features. and be able to hold conversations via voice or use the camera etc. so its my portable personal freind. maybe even allow it connect to spotify and audible to play. i apprediate it will need to use an open ai key if it can not access my own agent but can you get as close as you can with this.. give it a nice freindly appearance

[magicalmutation-coder]

@copilot ok commit all changes please onto the main branch

[Copilot]

@copilot ok commit all changes please onto the main branch

All changes from this PR are already pushed and up-to-date on the copilot/interface-freenove-with-agent branch (latest commit 8f0e8af).

To get them onto main, you just need to merge this PR — click the green "Merge pull request" button on the PR page on GitHub. That's the safe way to land the changes on main while keeping the full commit history.

I'm not able to push directly to main myself — the agent is scoped to the PR branch only.

@copilot ok commit all changes please onto the main branch

All changes are committed and pushed to this PR branch (copilot/interface-freenove-with-agent). To merge them into main, you'll need to click the "Merge pull request" button on the PR page in GitHub — I don't have permission to merge PRs directly. Once you approve and merge, all the firmware and companion server code will land on main.

[magicalmutation-coder]

@copilot ok so . it will aslo need to be able to connecgt to local wifi netowrk sfor access to llms etc and can we also add bluetooth networking capabilities. there shoudl be a full config screen fora ll aspects of the app including all llms with sample configs and network setings, to agent behaviour and personality etc

[Copilot]

@copilot ok so . it will aslo need to be able to connecgt to local wifi netowrk sfor access to llms etc and can we also add bluetooth networking capabilities. there shoudl be a full config screen f...

Done in commit f31c0fc. Here's what was added:

Local WiFi (multi-network)

• Debbie now stores up to 3 WiFi networks and tries them in order on boot (primary → secondary → tertiary → AP-only setup mode). Essential for home, office, and mobile hotspot connectivity to reach local/cloud LLMs.

Bluetooth (BLE)

• New bluetooth_manager implements a BLE GATT server with the official Nordic UART Service (NUS) using the proper 128-bit UUIDs (6E400001… family), so standard BLE Serial apps like nRF Toolbox or Serial Bluetooth Terminal recognise Debbie out-of-the-box.
• Send the same JSON config commands over BLE as you would to the HTTP /configure endpoint — no WiFi needed for initial setup.
• BLE can be enabled/disabled from the config portal.

Full tabbed config portal (visit http://192.168.4.1/ on the Debbie AP):

Tab	Contents
📶 Network	Primary + 2 fallback WiFi networks, BLE toggle + device name, companion server URL
🤖 AI & LLM	Provider cards: OpenAI, Anthropic, Groq, OpenRouter, Ollama (local) — each with its own settings panel, model dropdowns/sample configs, and API key field
🎀 Personality	Name, system prompt (with reset-to-default), voice style, response length
🔔 Notifications	Master toggle + per-source switches (WhatsApp, Email, Spotify)
⚙️ Advanced	Volume, VAD threshold, camera toggle, live device status, factory reset

[magicalmutation-coder]

@copilot it also needs its own memory and rag implimentation so it can be more effective in its answering.. and have a persistance

[Copilot]

@copilot it also needs its own memory and rag implimentation so it can be more effective in its answering.. and have a persistance

Done in commit fac8c55. Here's what was added:

Short-term memory (on-device)

• A circular buffer of the last 20 conversation turns (user + AI) kept in RAM and automatically saved to NVS flash every 4 turns — so Debbie remembers recent context even after a reboot.
• Both user speech (OAI_EVT_USER_TRANSCRIPT) and AI responses are captured as turns.
• The turn history is injected into the OpenAI system prompt at the start of every session.

Long-term facts (NVS persisted)

• Up to 50 key-value facts (e.g. user_name=Alice, user_likes=jazz) stored in flash.
• The AI has a new save_memory tool — say "Remember that my name is Alice" and Debbie calls save_memory(key="user_name", value="Alice", importance=9) automatically.
• Facts survive reboots and are prefixed to every session prompt.

Companion RAG (server-side, searchable)

• New companion-server/memory_store.js: SQLite database with an FTS5 full-text search table so all conversation history is stored and searchable.
• REST endpoints: GET /memory/query?q=<text> retrieves relevant past turns, POST /memory/turn stores a new turn, POST /memory/fact stores a fact, DELETE /memory/clear wipes everything.
• When WiFi is connected and a companion server is configured, Debbie queries /memory/query before each response to inject relevant past context (RAG).

New 🧠 Memory tab in the config portal (http://192.168.4.1/):

• Enable/disable memory master toggle
• Enable/disable Companion RAG toggle
• Configurable number of turns to retain (5–50)
• Live stats (turn count, fact count, RAG status)
• "Clear All Memory" button

Voice commands: "Remember that..." / "What do you know about me?" / "Forget everything"

[magicalmutation-coder]

@copilot ok.. carry on.. im keen to get the first test running on this..

[magicalmutation-coder]

@copilot ok push all to the main brach

[Copilot]

@copilot ok push all to the main brach

All commits are already on this PR branch and up-to-date. To land them on main, click the green "Merge pull request" button on this PR page — I don't have permission to merge PRs or push directly to main myself.