Introduces Nth-frame full face detection combined with KCF bounding
box tracking and Lucas-Kanade (LK) optical flow for keypoint (KPS)
tracking on intermediate frames. This is primarily for single-face
webcam mode to improve performance while maintaining per-frame swaps.
Key Changes:
- Modified `face_swapper.py` (`process_frame`):
- Full `insightface.FaceAnalysis` runs every N frames (default 5)
or if tracking is lost.
- KCF tracker updates bounding box on intermediate frames.
- Optical flow (`cv2.calcOpticalFlowPyrLK`) tracks the 5 keypoints
from the previous frame to the current intermediate frame.
- A `Face` object is constructed with tracked bbox and KPS for
swapping on intermediate frames (detailed landmarks like
`landmark_2d_106` are None for these).
- Experimental similar logic added to `_process_live_target_v2`
for `map_faces=True` live mode (non-many_faces path).
- Robustness:
- Mouth masking and face mask creation functions in `face_swapper.py`
now handle cases where `landmark_2d_106` is `None` (e.g., by
skipping mouth mask or using bbox for face mask).
- Added division-by-zero check in `apply_color_transfer`.
- State Management:
- Introduced `reset_tracker_state()` in `face_swapper.py` to clear
all tracking-related global variables.
- `ui.py` now calls `reset_tracker_state()` at appropriate points
(webcam start, mode changes, new source image selection) to ensure
clean tracking for new sessions.
- `DETECTION_INTERVAL` in `face_swapper.py` increased to 5.
This aims to provide you with a smoother face swap experience with better FPS
by reducing the frequency of expensive full face analysis, while the
actual swap operation continues on every frame using tracked data.
Optimizes webcam performance for face swapping by introducing
Nth-frame full face detection and using a KCF tracker for
intermediate frames in modules/processors/frame/face_swapper.py.
Key changes:
- Full face analysis (get_one_face) now runs every N frames (default 3)
or when tracking is lost in the process_frame function (for single
face mode).
- For intermediate frames, a KCF tracker updates the target face bounding
box, and keypoints are estimated by translating the last known good
keypoints.
- The actual face swap (inswapper model) still runs on every frame if a
face (either detected or tracked) is available.
- Experimental tracking logic added to _process_live_target_v2 for
map_faces=True in live mode (non-many_faces path).
- Added robustness:
- None checks for landmarks in mouth_mask and create_face_mask
functions, with fallbacks for create_face_mask.
- Division-by-zero check in apply_color_transfer.
- Reset tracker state in process_video for new video files.
This aims to significantly improve FPS by reducing the frequency of
costly full face analysis, while still providing a continuous swap.
Mouth masking will be less effective on tracked intermediate frames
due to the absence of full landmark data.
Deletes an erroneous `[end of modules/processors/frame/face_swapper.py]`
marker line from the end of the face_swapper.py file. This marker was
accidentally written into the source code by me and was
causing a SyntaxError, preventing the application from starting.
This commit ensures the file is syntactically correct.
All previous setup scripts and fixes remain on this new branch,
which is based on the prior state of 'feat/macos-setup-scripts'.
Resolves a SyntaxError ('(' was never closed) and associated
IndentationErrors in modules/processors/frame/face_swapper.py.
These errors were caused by malformed and duplicated definitions of
the helper functions _prepare_warped_source_material_and_mask and
_blend_material_onto_frame.
The fix involved:
- Removing the entire erroneous duplicated/malformed function blocks.
- Ensuring that the single, correct definitions for these helper
functions are properly indented at the top level of the module.
This critical fix addresses a major blocker that prevented the
application from starting and parsing the face_swapper.py module.
refactor: Default "Swap Hair" toggle to OFF
I've changed the default initial state of the "Enable Hair Swapping" feature to OFF.
- I updated `modules/globals.py` so that `enable_hair_swapping = False`.
- I also updated `modules/ui.py` in the `load_switch_states()` function, where the default for `enable_hair_swapping` is now `False`.
This change aligns with the current focus on perfecting the face-only swap before re-addressing hair swap features and provides a faster default experience for you.
Addresses an IndentationError at the definition of the
_blend_material_onto_frame helper function in
modules/processors/frame/face_swapper.py.
The fix ensures that the function definition line starts at
column 0 (no leading whitespace) and that the preceding
function's structure does not cause misinterpretation by the
Python parser. Duplicated/malformed definitions of related
helper functions were also confirmed to be removed in prior steps.
This resolves a syntax error that prevented your application from
starting.
Replaces Python 3.10+ type hint syntax (e.g., Frame | None)
with Python 3.9 compatible syntax (e.g., Optional[Frame])
in modules/processors/frame/face_swapper.py.
This resolves a TypeError encountered when running on Python 3.9.
Specifically, the return type of _prepare_warped_source_material_and_mask
was updated.
This commit introduces shell scripts to automate the setup process and provide convenient ways to run the application on macOS.
New files added:
- setup_mac.sh: Checks for Python 3.9+ and ffmpeg, creates a virtual environment, installs pip dependencies from requirements.txt.
- run_mac.sh: Runs the application with the CPU execution provider by default.
- run_mac_cpu.sh: Explicitly runs with the CPU execution provider.
- run_mac_coreml.sh: Runs with the CoreML execution provider.
- run_mac_mps.sh: Runs with the MPS execution provider.
The README.md has also been updated with a new section detailing how to use these scripts for macOS users.
These scripts aim to simplify the initial setup and execution of the project on macOS, similar to the .bat files available for Windows.
This commit introduces the capability to swap hair along with the face from a source image to a target image/video or live webcam feed.
Key changes include:
1. **Hair Segmentation:**
- Integrated the `isjackwild/segformer-b0-finetuned-segments-skin-hair-clothing` model from Hugging Face using the `transformers` library.
- Added `modules/hair_segmenter.py` with a `segment_hair` function to produce a binary hair mask from an image.
- Updated `requirements.txt` with `transformers`.
2. **Combined Face-Hair Mask:**
- Implemented `create_face_and_hair_mask` in `modules/processors/frame/face_swapper.py` to generate a unified mask for both face (from landmarks) and segmented hair from the source image.
3. **Enhanced Swapping Logic:**
- Modified `swap_face` and related processing functions (`process_frame`, `process_frame_v2`, `process_frames`, `process_image`) to utilize the full source image (`source_frame_full`).
- The `swap_face` function now performs the standard face swap and then:
- Segments hair from the `source_frame_full`.
- Warps the hair and its mask to the target face's position using an affine transformation estimated from facial landmarks.
- Applies color correction (`apply_color_transfer`) to the warped hair.
- Blends the hair onto the target frame, preferably using `cv2.seamlessClone` for improved realism.
- Existing mouth mask logic is preserved and applied to the final composited frame.
4. **Webcam Integration:**
- Updated the webcam processing loop in `modules/ui.py` (`create_webcam_preview`) to correctly load and pass the `source_frame_full` to the frame processors.
- This enables hair swapping in live webcam mode.
- Added error handling for source image loading in webcam mode.
This set of changes addresses your request for more realistic face swaps that include hair. Further testing and refinement of blending parameters may be beneficial for optimal results across all scenarios.
- Add explicit checks for face detection results (source and target faces).
- Handle cases when face embeddings are not available, preventing AttributeError.
- Provide meaningful log messages for easier debugging in future scenarios.
google-labs-jules[bot]
NeuroDonu
Kenneth Estanislao
Adrian Zimbran
Soul Lee
Pedro Santos
KRSHH
Rishon
Roland Pereira
pereiraroland26@gmail.com
underlines
c4fun
Joep de Jong