A browser-native augmented reality engine with two modes in one repo — real-time face filters powered by Jeeliz + Three.js, and image tracking powered by MindAR + A-Frame. No frameworks, no backend, no install.
Live Demo: https://webar-studio.vercel.app
Face.AR.Runtime.Hat.filter.with.snow.accumulation.and.image.tracking.demo.mp4
Browser AR Runtime — face filter with real-time snow accumulation on 3D geometry, and image tracking AR tour
If the video doesn't autoplay, click to play or view it directly on GitHub.
Cultural institution apps need AR that works without an app store. A visitor points their phone at an exhibit image and a 3D overlay appears — no QR code, no download, no WebXR device requirement. The same runtime also powers face filter experiences in the browser. This repo isolates both modes in standalone, runnable form, distilled from a production system serving 30+ cultural institution apps across Spain, France, and Belgium.
Point your camera at a registered target image — a 3D overlay appears anchored to that image in world space.
- 7 target cards in a slider — 5 are registered trackable targets, 2 are intentional non-targets
- For each trackable target: Gaudí bust 3D model overlaid + narration audio plays on
targetFound, pauses ontargetLost - Non-targets show a "Not a registered target" badge — no overlay, no audio
- A countdown timer auto-advances through the slider; user can also swipe manually
- Desktop: no camera request — image tracking only makes sense on mobile, so the desktop view shows the target cards in browse mode only
- Mobile: camera activates automatically on tab load
Real-time face detection and 3D model overlay via webcam. No server involved — everything runs on the GPU in the browser.
- Jeeliz FaceFilter detects face position, scale, and head rotation every frame
- Three.js renders a GLB model anchored to the detected face on a transparent overlay canvas
- Two effects selectable via dropdown: Black Leather Hat and Sunglasses
- Both GLBs are fetched and cached in memory on page load — effect switching is instant, no re-download
- Per-effect position, rotation, and scale tuning so hat sits on the head and sunglasses align to the eyes
- Face detection status dot — amber while camera initialises, red when tracking is active but no face found, green when a face is actively detected
- 2D Snow effect — snowflakes fall from the top of the frame and physically accumulate on detected edges (hat rim, hair boundary, face outline). Move your head and the piled snow drops — it reacts to geometry changes in real time, not just the camera feed
- "Point your camera at your face" hint shown until face is detected, hidden once tracking begins
The 7 targets:
| # | Image | Status |
|---|---|---|
| 1 | Antoni Gaudí — young portrait (colorized) | ✅ Trackable |
| 2 | Unknown man in suit | ❌ Not registered |
| 3 | Antoni Gaudí — framed poster, 1852–1926 | ✅ Trackable |
| 4 | Young man, studio photo | ❌ Not registered |
| 5 | Sagrada Família tower — mosaic spire detail | ✅ Trackable |
| 6 | Gaudí statue with Sagrada Família and lizard mosaic | ✅ Trackable |
| 7 | Casa Batlló facade | ✅ Trackable |
The two non-targets are deliberate. They demonstrate that the system performs image-feature matching against a registered set, not category recognition. A similar-looking portrait won't trigger the overlay — only the exact registered images do.
| Layer | Tech |
|---|---|
| Face detection | Jeeliz FaceFilter (WebGL neural net) |
| 3D rendering — Face AR | Three.js r112 + GLTFLoader |
| 2D overlay effects | Canvas 2D API |
| Image tracking | MindAR 1.2.3 |
| AR scene — Image AR | A-Frame 1.4.2 |
| 3D models | GLB (fetched on load, cached in memory) |
| Audio | HTML5 Audio API |
| Deploy | Vercel (static, no build step) |
| Libraries | CDN only — nothing installed |
No install required. Clone and open.
git clone https://github.com/sujoymondal87/browser-ar-runtime.git
cd browser-ar-runtime
npx serve .Visit http://localhost:3000 in a browser with camera access.
For image tracking, use a mobile device or grant camera permission on desktop. Point the camera at any of the 5 registered target images shown in the slider.
Switching tabs triggers a page reload:
/ → Image AR (default)
/face → Face AR
This is intentional. Jeeliz and MindAR both claim the WebGL context and the camera stream. Destroying and reinitialising them in the same page lifetime causes context conflicts and camera lock errors across browsers. A page reload is the cleanest solution — the loading screen makes it invisible to the user.
Why Jeeliz instead of MediaPipe or TensorFlow.js for face tracking? Jeeliz FaceFilter is a single self-contained WebGL neural net with no dependency graph. It runs entirely on the GPU, requires no WASM, and has a stable API. At the scale of this demo — single face, position and rotation only — it outperforms heavier frameworks on mid-range mobile hardware.
Why two separate canvas elements for Face AR?
Jeeliz owns one canvas for the video feed and neural net computation. Three.js owns a second canvas with alpha: true stacked on top via CSS. Sharing the WebGL context between Jeeliz and newer Three.js versions causes rendering bugs. Two canvases eliminates the conflict entirely.
Why GLBs fetched on load rather than bundled? The models are large binary assets. Fetching them once and holding them in memory means effect switching is instantaneous — no re-download, no re-parse. The browser cache handles persistence across sessions.
Why page reload on tab switch instead of in-page teardown? Jeeliz and MindAR both acquire the camera stream at the OS level. Neither library fully releases it on destroy across all browsers — Chrome on Android is particularly unreliable. A page reload guarantees a clean state. The shared loading screen makes the transition seamless.
Why desktop shows browse-only mode for Image AR? Image tracking requires a physical camera pointed at a physical image. On desktop, that interaction model doesn't exist. Requesting camera permission on desktop produces a confusing UX and serves no demo purpose. Desktop users can still browse the target set and understand what the mobile experience does.
Why two non-targets in the slider? To demonstrate the architecture honestly. The system uses compiled image feature descriptors — it matches against a specific registered set, not a category. Including similar-looking images that deliberately fail to trigger AR shows exactly where the boundary is.
Why copy the Jeeliz WebGL canvas to a 2D canvas before sampling pixels?
WebGL canvas pixels are not reliably readable via getImageData — the buffer is GPU-controlled and returns all zeros in most browsers regardless of what's rendered. Copying the WebGL canvas to an intermediate 2D canvas first makes the pixels CPU-readable. The snow edge detection depends entirely on this step; sampling the WebGL canvas directly produced zero values and the effect did nothing.
Why use horizontal edge detection for snow accumulation rather than depth or segmentation?
The only reliable signal available at this layer is pixel brightness across the composited frame — including the rendered 3D hat. A horizontal kernel [2,2,2,0,0,0,-2,-2,-2] finds bright-to-dark transitions: hat rims, hair boundaries, face edges. Snowflakes slow and pile on those detected edges. When the head moves, the edges shift — flakes that were resting on the hat rim lose their anchor and fall. This produces physically plausible snow accumulation behaviour without a depth buffer, segmentation model, or physics engine. It's edge detection repurposed as a collision surface.
Why run 2D effects on an independent requestAnimationFrame loop rather than inside Jeeliz's render callback?
The snow effect copies the WebGL canvas, runs a getImageData call, and executes an O(n²) convolution every frame. Running that synchronously inside Jeeliz's _onTrack callback blocks the JS thread the neural net runs on — measurably slowing face detection. A separate RAF loop fully decouples 2D rendering from Jeeliz. Face detection runs at full speed regardless of which 2D effect is active.
| Decision | Upside | Limitation |
|---|---|---|
| CDN-only libraries | Zero install, instant fork | CDN downtime breaks the demo |
| Page reload on tab switch | Clean camera + context state | Brief loading screen on every switch |
Single .mind file for all targets |
One compile, one load | Recompile needed to add or remove targets |
| Same GLB for all trackable targets | Simple asset pipeline | No per-target differentiation in the overlay |
| Desktop browse-only for Image AR | Honest UX, no wasted permission prompt | Feature appears disabled on desktop |
| Two canvas elements | No WebGL context conflict | Extra DOM element, slightly more CSS to manage |
| 2D effects on independent RAF | Face detection speed unaffected by overlay cost | Two animation loops running in parallel |
Static site — no build step.
Vercel:
- Connect repo, framework preset: Other
- Build command: leave empty
- Output directory: leave empty
- No environment variables
The vercel.json sets Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers — required for MindAR's use of SharedArrayBuffer.
The AR system this repo is distilled from is deployed inside a no-code app platform serving cultural institutions across Spain, France, and Belgium. The production face filter runtime handles GLTF model loading, expression-triggered animations, and multi-language narration routing. The image tracking runtime powers museum exhibit overlays — visitors point their phone at a physical exhibit image and a 3D model appears anchored to it in world space. This repo isolates both modes in standalone, runnable form with the same core architecture.
| Asset | Author | License |
|---|---|---|
| Black Leather Hat GLB | iGauravRajput | CC BY 4.0, Sketchfab |
| Sunglasses GLB | Eric Chadwick / Darmstadt Graphics Group GmbH | CC BY 4.0, Khronos glTF Sample Assets |
| Antoni Gaudí bust GLB | Roberto Domínguez (@vmmaniac) | CC BY 4.0, Sketchfab |
targetFoundandtargetLostevent listeners in MindAR must use explicit(e) =>arrow functions — referencing the globaleventobject inside the handler silently fails in Safari- MindAR requires
Cross-Origin-Opener-Policy: same-origin+Cross-Origin-Embedder-Policy: require-corp— missing either header produces aSharedArrayBuffer is not definederror with no useful stack trace - Jeeliz and Three.js r112 share the WebGL context without conflict; r125+ breaks — pin the Three.js version
- Camera stream on Android Chrome is not reliably released by
JEELIZFACEFILTER.destroy()— always reload the page rather than attempting in-page teardown - WebGL canvas pixels are not reliably readable via
getImageDatadirectly — copy the WebGL canvas to an intermediate 2D canvas first before sampling. Sampling the WebGL canvas directly returns all zeros in most browsers regardless of what's rendered
- MindAR compiles target images into
.mindfiles at author time. How would you build a runtime that compiles targets on the fly from user-uploaded images without blocking the main thread? - The two non-targets in this demo fail silently — the user sees no overlay. What feedback mechanism would you add to make the "not registered" state explicit in a real museum deployment?
- The page-reload tab switch is a pragmatic solution to a real browser bug. At what point would you invest in a proper in-page teardown and what would that look like?
- Face AR and Image AR share a loading screen but nothing else. What would a unified AR session manager look like if both modes needed to run simultaneously?
- The snow effect uses edge detection as a collision surface — no physics engine, no depth buffer. Where does this approach break down and what would you replace it with if you needed sub-pixel accuracy?
MIT