video-matting#1378
Open
1NL1 wants to merge 12 commits into
Open
Conversation
Introduce the AdvancedMattingProcessor orchestrator and the supporting shared modules (entry index, debug helpers, timer worker, error store). Add the package dependencies required by the background matting feature.
Introduce the Segmenter abstraction along with Landscape and Multiclass implementations, the loop runner, and a benchmarker so the pipeline can pick the best model at runtime.
Build a preprocessing pipeline that tracks video frames, manages the matting canvases, and crops a region of interest from face landmarks before passing the frame to the segmenter.
Modularize the WebGL2 renderer, extract its shaders, and add a fast GPU guided filter together with the mask post-processor and segmo compositor. The render loop runner and Canvas2D fallback live next to them. The old standalone postprocessing helpers (Morphology, Sigmoid, TemporalEMA) are removed since they are now folded into the renderer pipeline.
Wire the matting pipeline into the effects configuration panel, persist the user's blur choice, and add the matching translations for de, en, fr and nl.
Cover the RoiCropper behavior with Vitest and add the matching Vitest configuration so the suite can run in the frontend workspace.
Add a README walking through the blur module's architecture, and record the feature in the changelog.
Resolve SonarCloud warnings across the matting pipeline, extract an abstract Segmenter base class to remove duplication, add an isSupported guard and module README, trim the effects configuration, drop the standalone Vitest config and consolidate room locales across languages.
… module Strip verbose inline documentation and commentary from the matting pipeline, also dropping a few duplicated try/catch blocks left over from the refactor.
Add the background blur and virtual background entry under Unreleased.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a GPU-accelerated background blur effect to the in-call effects panel. Introduces a modular matting pipeline (segmentation → preprocessing → renderer → compositor) that runs entirely client-side using MediaPipe models and a WebGL2 guided filter, with a Canvas2D fallback for unsupported browsers.
What's included:
Examples:
Background for reference:
Old pipeline:
https://github.com/user-attachments/assets/4890a97b-661d-4193-9cb0-34e007a71660
New pipeline:
With Mediapipe Multiclass (heavier but more precise model):
https://github.com/user-attachments/assets/194b0c67-3658-4f01-a9c2-330c27ff40f1
With Mediapipe Selfie Landscape (lighter but less accurate model):
https://github.com/user-attachments/assets/20d7ace2-c0d3-4d7d-a9a4-a0318f53df83