Weidu experimental neural portrait animation 0.2.0 6 November 2026 This runtime uses manual cropping, not the InsightFace detection or landmark models. It uses the LivePortrait appearance feature extractor and LP-Distilled0.1. Sources and declared licences: LP Distill JS: https://github.com/stephenperez/lp-distilled-js/tree/ce87f30dc7ce584dab1cbf3286b6fb73d7598e93 LP-Distilled model: https://huggingface.co/slperez/LP-Distilled0.1-ONNX/tree/cd9c2c31302107cf490be36320f9e16b2b722e9f Appearance extractor: https://huggingface.co/warmshao/FasterLivePortrait/tree/eb937f4bec7186598df2d1f68e1ddbb488ae1de5 Teacher: https://github.com/KlingAIResearch/LivePortrait/tree/9b294b3d0536135442ea73cb01e6cb3ca7029dd3 ONNX Runtime Web 1.20.1: accompanying ORT-LICENSE.txt and ORT-ThirdPartyNotices.txt. Exact upstream source records follow. They are preserved without editorial rewriting. lp-package.json { "name": "lp-distill-js", "version": "0.1.0", "description": "Static browser JavaScript wrapper for a distilled LivePortrait-style ONNX model.", "type": "module", "main": "./lp-distill.js", "exports": { ".": "./lp-distill.js" }, "files": [ "lp-distill.js", "examples", "serve.mjs", "README.md" ], "scripts": { "start": "node serve.mjs" }, "license": "MIT" } LP-Distilled-model-card.md --- library_name: onnx pipeline_tag: image-to-image tags: - onnx - webgpu - liveportrait - portrait-animation - image-to-image license: mit --- # LP-Distilled0.1 ONNX `LP-Distilled0.1.onnx` is an experimental/proof-of-concept browser-oriented distillation of a LivePortrait-style portrait animation pipeline. It takes a static portrait crop, a LivePortrait appearance feature volume, normalized head-pose controls, and normalized expression controls, then renders a `256x256` RGB animation frame. This release is intended for interactive portrait-animation research and demos. It is not a full replacement for the original LivePortrait pipeline and should not be used in production. ## Model Details | Field | Value | | --- | --- | | Model file | `LP-Distilled0.1.onnx` | | Format | ONNX | | ONNX opset | 17 | | ONNX IR version | 8 | | Size | `74,573,121` bytes, about `71.1 MiB` | | SHA-256 | `335F98B44E5FAB3577B15C3EC53EDDD39BE228F072D98D7CE11232BCF1C42AFD` | | Output resolution | `256x256` | | Teacher | Internal LivePortrait-based teacher pipeline | | Runtime target | Browser ONNX Runtime Web / WebGPU, with WASM fallback where needed | The released ONNX graph is a fused two-stage model: ```text Stage1: source portrait + feature_3d + pose controls -> posed neutral RGB Stage2: Stage1 RGB + same source feature_3d + expression controls -> final RGB ``` The fused graph routes the Stage1 `rgb` output into Stage2's image input. The external caller only has to feed the original aligned portrait, the source `feature_3d`, and the two control vectors. ## Inputs And Outputs | Name | Shape | Type | Description | | --- | --- | --- | --- | | `feature_3d` | `[1, 32, 16, 64, 64]` | `float32` | LivePortrait appearance feature volume extracted once from the aligned source portrait. | | `pose_control_vector` | `[1, 3]` | `float32` | Normalized `[yaw, pitch, roll]` controls. | | `input_rgb` | `[1, 3, 256, 256]` | `float32` | Aligned source portrait crop in CHW RGB format. | | `expression_control_vector` | `[1, 30]` | `float32` | Normalized expression, blink, eye-drift, mood, and viseme controls. | | `rgb` | `[1, 3, 256, 256]` | `float32` | Rendered output frame in CHW RGB format. | The model does not include face detection, crop/alignment, or appearance feature extraction. A browser demo can prepare those inputs with compatible LivePortrait-derived ONNX source-preparation models: ```text retinaface_det_static.onnx face_2dpose_106_static.onnx appearance_feature_extractor.onnx ``` The fused model does not require `motion_extractor.onnx`. ## Control Contract All controls are normalized numeric values. They are not image-space landmark coordinates. Pose controls: ```text pose.yaw pose.pitch pose.roll ``` Expression controls: ```text kp1.x kp1.y kp1.z kp2.x kp2.y kp2.z kp3.y kp5.x kp5.y kp5.z kp6.y kp6.z kp11.x kp11.y kp13.y kp14.x kp14.y kp15.x kp15.y kp16.x kp16.y kp17.x kp17.y kp17.z kp19.x kp19.y kp19.z kp20.x kp20.y kp20.z ``` The control schema deliberately excludes many LivePortrait keypoint axes. Runtime code should use the exact schema order above rather than assuming `keypoint * 3 + axis` layout. Keypoint expression controls are normalized deltas, not absolute keypoints. The training convention was: ```text raw_delta = clamp(raw_delta, -0.08, 0.08) normalized_delta = raw_delta / 0.08 ``` Pose controls are also normalized to `[-1, 1]`. The training convention was: ```text yaw_norm = yaw_degrees / 20 pitch_norm = pitch_degrees / 14 roll_norm = roll_degrees / 14 ``` Those ranges are part of the model contract. Sending much larger values can produce degraded or unstable output. ## Pose Vocabulary The pose model was trained primarily around a finite vocabulary of Catmull-Rom head-pose trajectories, plus single-axis paths. This is intentional: the runtime motion system is expected to follow, stop, reverse, or blend from learned trajectories rather than freely sampling every possible yaw/pitch/roll combination. Inputs outside the trained trajectory vocabulary may still work, but quality can degrade, especially for extreme rotations or unusual pose/expression combinations. ## Limitations - Output is fixed at `256x256`. - This proof-of-concept model was trained on only a few hundred portrait style images. In adhoc testing it can generalize some, but might not look great, depending on the specific source image. - Quality depends heavily on source crop/alignment quality. - The model can blur or distort hair, facial details, accessories, and unusual backgrounds. - Extreme poses can be softer than neutral or moderate poses. - Controls are semantic approximations learned from a teacher; they are not a physically meaningful face rig. - The model card does not assert compatibility with all upstream licenses. Check the relevant LivePortrait and dataset licenses before redistribution or commercial use. ## Ethical Considerations Portrait animation models can be misused to create misleading or non-consensual media. Applications should clearly disclose generated animation, obtain consent for likeness use, and avoid representing outputs as authentic footage. ## Example Runtime Flow ```text 1. Load a portrait image. 2. Detect and align the face to a 256x256 RGB crop. 3. Run a compatible LivePortrait appearance feature extractor once: input_rgb -> feature_3d 4. For each frame: feature_3d pose_control_vector input_rgb expression_control_vector -> LP-Distilled0.1.onnx -> rgb 5. Draw rgb to a canvas or video frame. ``` For lip-sync, map phonemes or visemes to the 30-wide expression control vector, then smooth the control curve over time. Directly snapping between discrete viseme vectors will look jerky. ## Release Notes This is an early research release. The current graph is useful for interactive demos, but quality is still evolving. Future releases may change the control schema, model capacity, training data mix, or source-prep assumptions. LivePortrait-LICENSE.txt MIT License Copyright (c) 2024 Kuaishou Visual Generation and Interaction Center Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- The code of InsightFace is released under the MIT License. The models of InsightFace are for non-commercial research purposes only. If you want to use the LivePortrait project for commercial purposes, you should remove and replace InsightFace’s detection models to fully comply with the MIT license. FasterLivePortrait-LICENSE.txt MIT License Copyright (c) 2025 warmshao Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- ADDITIONAL NOTICE FOR MODELS: This repository may contain or reference machine learning models. These models are subject to their respective licenses, which may differ from the MIT license applied to the code in this repository. Users are responsible for complying with the license terms of any models they use. This repository and its maintainers assume no responsibility for model licensing compliance. Please check the original source and license of each model before use. Deployment note, 6 November 2026 The ONNX WebAssembly software binary is requested only from this fixed public URL: https://cdn.jsdelivr.net/npm/onnxruntime-web@1.20.1/dist/ort-wasm-simd-threaded.jsep.wasm It is checked against the npm-verified byte count and SHA-256 before compilation. No portrait, voice or text is sent to the CDN. JavaScript modules remain self-hosted. The licence and Microsoft third-party notices above remain applicable.