GitHub: 🎨 Project Page
Install Python dependencies:
git clone https://git-dev.yunwei-afs.com/AITraining/UniAnimate.git
cd UniAnimate
conda create -n UniAnimate python=3.9
conda activate UniAnimate
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements_20240805.txt
If environment setup fails, you can also try cloning my prepared environment directly: /home/ubuntu02/anaconda3/envs/unianimate/ (IP, username, and password are the same as in the "Download model files" section below).
python unianimate.py
Example usage:
# Before initializing the UniAnimate instance, define model input size. Default is [512, 768].
model_width, model_height = 448, 640 # Accelerated setting: model_width, model_height = 448, 640
# Specify model checkpoint paths
test_model_path = 'checkpoints/unianimate_16f_32f_non_ema_223000.pth'
embedder_pretrained_path = 'checkpoints/open_clip_pytorch_model.bin'
auto_encoder_pretrained_path = 'checkpoints/v2-1_512-ema-pruned.ckpt'
detection_ckpt_path = 'checkpoints/yolov8m.pt'
inpainting_ckpt_path = 'checkpoints/big-lama.pt'
matting_ckpt_path = 'checkpoints/portrate_selflabelFFinetune_fullAndCut_dp_size512_ImageANTIALIAS_adduserimage_best_ckpt.pth'
dwpose_onnx_det_path = 'checkpoints/yolox_l.onnx'
dwpose_onnx_pose_path = 'checkpoints/dw-ll_ucoco_384.onnx'
yoloface_path = 'checkpoints/yoloface_8n.onnx'
landmark_path = 'checkpoints/2dfan4.onnx'
face_embedding_path = 'checkpoints/arcface_w600k_r50.onnx'
inswapper_path = 'checkpoints/inswapper_128.onnx'
ua = UniAnimate(model_width=model_width, model_height=model_height,
test_model_path=test_model_path,
embedder_pretrained_path=embedder_pretrained_path,
auto_encoder_pretrained_path=auto_encoder_pretrained_path,
detection_ckpt_path=detection_ckpt_path,
inpainting_ckpt_path=inpainting_ckpt_path,
matting_ckpt_path=matting_ckpt_path,
dwpose_onnx_det_path=dwpose_onnx_det_path,
dwpose_onnx_pose_path=dwpose_onnx_pose_path,
yoloface_path=yoloface_path,
landmark_path=landmark_path,
face_embedding_path=face_embedding_path,
inswapper_path=inswapper_path
)
ref_img_path = 'data/images/2.jpg'
ref_img = cv2.imread(ref_img_path)
ref_img = cv2.cvtColor(ref_img, cv2.COLOR_BGR2RGB)
pose_json = 'data/saved_pose/guibuxinwu.json'
save_path = 'outputs/UniAnimate_infer_long/res.mp4'
save_mask = 'outputs/UniAnimate_infer_long/res_448x640_mask.mp4'
save_bg = 'outputs/UniAnimate_infer_long/res_bg.png'
ua.run(ref_img, pose_json, save_path, save_mask, save_bg)
Meaning of input parameters for UniAnimate.run:
ori_img: User input image, in RGB format.
pose_path: Path to the driving pose file, in JSON format.
save_path: Path to save the output video (for plain dancing output without extra effects).
save_mask: Path to save the mask video (used for adding effects).
save_bg: Path to save the background image after removing the person.
use_faceswap: Whether to use face-swap post-processing. Default is True.
input_fps: Output video FPS. Since pose_path usually includes FPS, this is generally not required.
input_max_frames: Maximum number of frames. Default is "None". Usually not required because it is handled internally.
iterations_step: Number of sampling iterations. More iterations take longer but may improve quality. Current default is 30.
pose_align_type: Keypoint alignment mode. There are two options; default is 1.