An open-source, self-hostable multimodal memory system for smart glasses and phones.
🌐 Try in Browser · 🚀 Quick Start · 🎬 Watch the Demo · 📱 Glasses APK
LightMem-Ego is the end-to-end system; its long-term tier (M_lt) is powered by EM²Mem (EMNLP 2026 Findings), part of the ZJUNLP LightMem project family.
📑 Table of contents
- [2026-09] ✨ Multi-session support, online memory editing, and streaming answers land in the backend and web UI.
- [2026-08] 🎉🎉🎉 EM²Mem: Event-Centric Multimodal Memory for Large Language Models — the long-term memory engine behind this backend — has been accepted to EMNLP 2026 Findings!
- [2026-07-13] 📄 LightMem-Ego: Your AI Memory for Everyday Life is released on arXiv.
- [2026-07] 📦 v1.0.0 released — download the Rokid AI Glass APK and reproduce the full stack with Docker.
- [2026-05] 🎉 LightMem-Ego: Your AI Memory for Everyday Life is open-sourced.
Ask the glasses a question in the middle of your day, and get an answer grounded in what you actually saw and heard. Prefer typing? Join the same live session from the web page.
Tip
No hardware? Try it right now. The live web demo runs the full LightMem-Ego workflow in your browser — on a phone too, where it captures from the phone's own camera and microphone. No glasses, no local installation.
Hands-free on Rokid AI Glasses Ask by voice or preset question |
Answers grounded in memory Timestamps + visual evidence |
Same session on the web Type questions when speaking isn't convenient |
|
- 🎥 Always-on egocentric capture — streams first-person camera frames and microphone audio from Rokid AI Glasses or a phone.
- 🧠 Three-tier memory — a rolling current memory, short-term micro-events, and consolidated long-term episodes, routines, and preferences.
- ⏱️ One aligned timeline — frames, audio chunks, ASR transcripts and metadata all share a single session timeline.
- 🔍 Memory-grounded answers — every answer ships with the timestamped visual and transcript evidence behind it.
- 👓 Glasses and web, one session — start capture on the glasses, keep asking from the web page in the same live session.
- 🐳 Self-hostable —
docker compose up --buildbrings up the web UI and the full backend worker pipeline.
Text memory systems only know what you typed, live assistants only know the current scene, and video systems only let you search afterwards. LightMem-Ego covers all three — the system-by-system table is in How It Compares.
Important
Every path except the hosted demo needs an OpenAI-compatible LLM endpoint (base URL, API key, model names) and Xfyun ASR credentials for speech. The default stack also expects Qwen3-Embedding-4B weights under docker-data/models/.
| Path | Setup | What it adds | GPU |
|---|---|---|---|
| 🌐 Live demo | none | the full hosted workflow | — |
| 🐳 Docker | Docker + your LLM endpoint | the standard self-hosted setup | no |
| 👁 + visual retrieval | --profile models + VLM2Vec weights |
frame-level visual matching | yes |
| ⚡ + local Qwen | vLLM server on your GPU | lower first-token latency | yes |
| 👓 Rokid glasses | APK + any backend above | hands-free capture, HUD answers | no |
Note
The last two are independent add-ons, not requirements. The plain Docker setup is what we run day to day — add visual retrieval when caption and transcript evidence is not enough, and local Qwen when a remote API feels slow.
git clone https://github.com/zjunlp/LightMem-Ego.git
cd LightMem-Ego
cp deploy/.env.example .env # LLM endpoint, keys, model names, Xfyun credentials
docker compose up --buildOpen http://localhost:8080. The web container proxies /api to the backend, so no CORS setup is needed. The first build takes a few minutes.
By default EM2MEM_VISUAL_BACKEND=mock, so retrieval runs on captions and transcripts. Current memory, short-term micro-events, long-term consolidation and evidence-grounded answers all behave as in the demo — only frame-level visual matching is off.
Put VLM2Vec-V2.0 and Qwen3-Embedding-4B under docker-data/models/, then start the model services:
docker compose --profile models up --buildPoint the backend at them in .env:
EM2MEM_VISUAL_BACKEND=remote
EM2MEM_TEXT_EMBED_BACKEND=remoteOn a GPU host, add the override so the workers get the GPU as well:
docker compose -f compose.yaml -f compose.gpu.yaml --profile models up --buildThis needs the NVIDIA Container Toolkit and model directories matching the paths in .env — see deploy/DOCKER.md for details.
This is where a GPU helps most. Every memory write and every answer otherwise round-trips to a remote API; serving the LLM locally cuts first-token latency noticeably. The scripts build an isolated vLLM environment and switch the backend onto it:
cd src/backend
scripts/setup_local_qwen35_env.sh
scripts/download_local_qwen35_model.sh
scripts/select_llm_profile.sh local-qwen35
scripts/stop_server_and_workers.sh --keep-api --forceDetails and the smoke test: src/backend/README.md.
Install the released APK:
adb install -r app-release.apkOr build it (JDK + Android SDK):
cd src/ai_glass_app
./gradlew assembleDebug # Windows: .\gradlew.bat assembleDebugSet API_BASE_URL in LightMemEgoConfig.kt to your own backend — it points at our demo server by default. Details: src/ai_glass_app/README.md.
Web frontend (Node.js + npm)
cd src/frontend/online_web
npm install
npm run devPoint it at your backend by creating online_web/.env.local:
VITE_API_BASE_URL=http://127.0.0.1:8000Details: src/frontend/README.md
Backend (Python 3.10+, ffmpeg/ffprobe)
cd src/backend
python -m venv .venv && source .venv/bin/activate
python -m pip install --upgrade pip && python -m pip install -e .
cp .env.example .env # configure model paths and API credentials
scripts/start_api.sh
scripts/start_online_all_workers.shDetails: src/backend/README.md and DEPLOYMENT.md.
| Scenario | Example question | Memory used |
|---|---|---|
| Object finding | "Where did I leave my badge?" | Current + short-term |
| Conversation recall | "What did the doctor tell me after checking the report?" | Short-term + transcript |
| Day summarization | "What did I do this afternoon?" | Short-term + long-term |
| Routine discovery | "What do I usually do after arriving at the office?" | Long-term semantic |
| Live assistance | "What am I looking at right now?" | Current |
Rokid AI Glasses ─┐
├─► Stream API ─► M_cur ─► M_st ─► M_lt ─► Retrieval ─► Answer + Evidence
Browser (web) ────┘ current short long
| Memory tier | Scope | Example |
|---|---|---|
M_cur current memory |
The ongoing scene, updated as frames arrive | "What am I looking at?" |
M_st short-term memory |
Recent micro-events, actions, and conversations | "What did she just tell me?" |
M_lt long-term memory |
Consolidated episodes, routines, preferences, semantic facts | "What do I usually do on Fridays?" |
The backend divides each session into short event anchors and stores multimodal evidence per anchor. The long-term tier (M_lt) is built by EM²Mem, our event-centric multimodal memory framework (EMNLP 2026 Findings, arXiv:2609.00551): events are the retrieval unit, and episodic and semantic graphs link them across a session. At query time the system retrieves aligned event-level evidence — captions, transcripts, frames, timestamps — instead of reconstructing context at inference.
EM²Mem in one picture. A video is segmented into 30-second event anchors, and each anchor becomes a memory cell holding dense captions, transcripts, keyframes, and metadata. Episodic and semantic graphs link those cells, and retrieval reads grounded evidence from them instead of re-aligning raw fragments at query time.
Note
These numbers come from a small, intentionally balanced set — 27 queries (nine per scenario) over five everyday-life videos, about 45.7 minutes of footage — collected with the phone and glasses client profiles used in the paper. In the open-source release the phone client is the web frontend running in a mobile browser, capturing from the phone's own camera and microphone — a native app is on the roadmap. It measures the current prototype rather than a public leaderboard. Reported in the LightMem-Ego paper.
Retrieval accuracy — Recall@k over the retrieved memory entries, with MRR for the first relevant hit:
| Scenario | R@1 | R@3 | R@5 | MRR |
|---|---|---|---|---|
| Object finding | 22.2 | 66.7 | 77.8 | 0.454 |
| Conversation recall | 44.4 | 55.6 | 55.6 | 0.481 |
| Life summarization | 88.9 | 100.0 | 100.0 | 0.944 |
| Overall | 51.9 | 74.1 | 77.8 | 0.627 |
Answer accuracy — experience QA over daily scenarios:
| Scenario | LLM-Judge | Human |
|---|---|---|
| Object finding | 44.4 | 55.6 |
| Conversation recall | 33.3 | 33.3 |
| Life summarization | 77.8 | 77.8 |
| Overall | 51.9 | 55.6 |
Latency — P50 / P90 across two client profiles:
| Stage | Phone P50 | Phone P90 | Glasses P50 | Glasses P90 |
|---|---|---|---|---|
| Short-term memory QA | ||||
| Retrieval | 76 ms | 131 ms | 44 ms | 87 ms |
| Time to first token | 532 ms | 643 ms | 423 ms | 494 ms |
| Answer generation | 6.13 s | 10.11 s | 6.81 s | 9.14 s |
| End-to-end | 6.42 s | 10.34 s | 6.95 s | 9.31 s |
| Long-term memory QA | ||||
| Retrieval | 2.99 s | 3.84 s | 3.06 s | 3.44 s |
| Time to first token | 4.64 s | 5.44 s | 4.74 s | 5.07 s |
| Answer generation | 5.78 s | 9.56 s | 4.37 s | 9.16 s |
| End-to-end | 10.57 s | 13.93 s | 8.61 s | 13.60 s |
Glasses columns are the glasses-style client profile. Short-term queries stay near-interactive; long-term ones trade latency for temporal coverage.
The long-term tier (M_lt) is built by EM²Mem. Average accuracy (%) across three long-video and egocentric benchmarks, as reported in the EM²Mem paper:
| Method | EgoLifeQA | Ego-R1 Bench | Video-MME (L) |
|---|---|---|---|
| GPT-5 | 48.6 | 46.3 | 74.3 |
| HippoRAG | 59.6 | 56.0 | 52.1 |
| M3-Agent | 53.5 | 52.0 | 55.3 |
| Ego-R1 | 53.0 | 52.0 | 42.7 |
| WorldMM | 65.6 | 65.3 | 76.6 |
| EM²Mem | 66.0 | 67.7 | 76.8 |
Against the strongest baseline (WorldMM, reproduced under the same evaluation setting):
| Metric | EM²Mem | WorldMM | Gain |
|---|---|---|---|
| Avg. latency per query | 98.21 s | 459.00 s | 4.67× faster |
| Wall-clock evaluation time | 6,138 s | 229,502 s | 37.4× faster |
| Total tokens | 15.27M | 42.03M | 63.7% fewer |
EM²Mem moves multimodal alignment and graph organization into offline memory construction, so inference reads from pre-built event-indexed memory cells instead of re-aligning isolated fragments. Full per-category tables are in the backend README; reproduction scripts in experiments/egolife.
Representative commercial assistants, text-based memory systems, and egocentric multimodal assistants. This compares publicly described capabilities rather than measured performance.
| System | Platform & input | Real-time A/V stream | Current / short-term MM memory | Long-term episodic | Long-term semantic | Timestamped evidence |
|---|---|---|---|---|---|---|
| ChatGPT Memory | Text chat | — | — | — | Partial | — |
| Mem0-style memory | Text and agent memory | — | Partial | — | ✓ | Partial |
| Memories.ai | Video archives and visual memory | Partial | Partial | ✓ | Partial | Partial |
| Gemini Live | Phone | ✓ | Partial | — | — | — |
| Ray-Ban Meta AI Glasses | Glasses | Partial | Partial | — | — | — |
| Vinci | Phone or wearable camera | ✓ | ✓ | Partial | Partial | Partial |
| VisualClaw | Streaming video with agent workspace | Partial | Partial | — | Partial | Partial |
| VisionClaw | Smart glasses | ✓ | Partial | — | — | Partial |
| Egocentric Co-Pilot | Smart glasses with web agents | ✓ | ✓ | Partial | Partial | Partial |
| EgoButler | AI-glasses egocentric video and audio | Partial | Partial | Partial | Partial | ✓ |
| LightMem-Ego | Phone and glasses-style client | ✓ | ✓ | ✓ | ✓ | ✓ |
✓ implemented as an explicit first-class component · Partial limited, implicit, offline, session-level, or modality-restricted · — not explicitly supported or not publicly described. Adapted from the LightMem-Ego paper. Phone was the paper's evaluation client — the open-source clients are the browser frontend and the Rokid AI Glasses app.
| Path | What's inside | Docs |
|---|---|---|
src/ai_glass_app/ |
Android app for Rokid AI Glasses (Kotlin, Jetpack Compose, CameraX) | README |
src/frontend/ |
Vite + React web UI for capture, sessions, QA, and evidence review | README |
src/backend/ |
FastAPI service plus the online worker pipeline (ASR, memory, retrieval, QA) | README |
compose.yaml, deploy/ |
Docker Compose stack and deployment notes | DOCKER.md |
- Ship a native phone app — today the phone client is the web frontend in a mobile browser.
- Release the end-to-end evaluation dataset and reproducibility scripts.
- Pluggable ASR, VLM, and embedding backends beyond the current defaults.
- Support wearable devices beyond Rokid AI Glass.
- On-device filtering and user-controlled memory editing for privacy-sensitive capture.
- One-click deployment template for a full cloud deployment.
If you find LightMem-Ego useful, please cite our paper:
@article{chen2026lightmemego,
title={LightMem-Ego: Your AI Memory for Everyday Life},
author={Chen, Yijun and Xiao, Boyi and Zhao, Yixian and Xia, Haoting and Xu, Buqiang and Fang, Jizhan and Li, Yanya and Zheng, Yaqi and Wang, Xuehai and Xue, Zirui and others},
journal={arXiv preprint arXiv:2607.11487},
year={2026}
}The long-term memory tier (M_lt) of the backend is built by EM²Mem, which has been accepted to EMNLP 2026 Findings. Please cite it as well when you use that module:
@article{chen2026em2mem,
title={EM$^{2}$Mem: Event-Centric Multimodal Memory for Large Language Models},
author={Chen, Yijun and Zheng, Yaqi and Li, Yanya and Xiao, Boyi and Xu, Buqiang and Qiao, Shuofei and Fang, Jizhan and Deng, Xinle and Yao, Yunzhi and Wang, Xuehai and others},
journal={arXiv preprint arXiv:2609.00551},
year={2026}
}
This repository belongs to the ZJUNLP LightMem series, which targets context bloat, excessive token consumption, and low cache utilization in long-running LLM agents:
- LightMem — a lightweight and efficient memory management framework for LLMs and AI agents
- LightRSI — a modular framework for recursive improvement in long-horizon LLM agents
- EM²Mem (EMNLP 2026 Findings) — event-centric multimodal memory for long-video QA, and the long-term memory engine behind this system (code overview)
LightMem-Ego builds on the broader line of work on memory-augmented agents, egocentric multimodal understanding, and wearable AI assistants. We thank all contributors and collaborators who helped develop the system.
Released under the MIT License.
LightMem-Ego processes camera frames, microphone audio, transcripts, and generated memories. It is released for research and demonstration; a production deployment needs HTTPS, access control, encryption at rest, a data retention/deletion policy, and explicit user consent. Runtime media and memory are written to local, Git-ignored directories.
