English Version | [中文版本](https://github.com/xinyang-sun/VirtuSync/blob/main/README.zh-cn.md)

# VirtuSync This is a 3D motion capture software developed using Unity, Mediapipe, and Python, supporting real-time tracking and synchronization of the face, eyes, and hands. ## Authors - [@Xinyang-sun](https://www.github.com/xinyang-sun) ## Built With

![Unity2020.3.21f1c1](https://camo.githubusercontent.com/432aa937fc56e1b634e49c59af0100dc3561707576eef68a6e602448cc31048d/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f556e6974792d323032302e332e3231663163312d626c7565) ![Python 3.10](https://camo.githubusercontent.com/ee885d010a2d894c3c5abc7990ac37b97af3a150702c5e85f5240ce536146240/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f507974686f6e2d332e31302d626c7565) ![Mediapipe](https://camo.githubusercontent.com/8632d96e080387fcf1b3149a85e994e8cc3fad3abb6e397da5d7c03d0e57636a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4d65646961706970652d626c7565 ) [![license](https://camo.githubusercontent.com/bb4e5c0036a6a8cdbc59b38d44f09ad8f6dc722751dad34d3df5bf0ac61913c1/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d626c7565 )](https://github.com/xinyang-sun/VirtuSync?tab=MIT-1-ov-file)

## Demo - **Video demo** [Demo video](https://www.bilibili.com/video/BV1NZrtYAEaK/?share_source=copy_web&vd_source=9b95709580179b5fcb8562c82ecdfa35) - **GIF Show** ![Camera control](https://github.com/xinyang-sun/VirtuSync/raw/main/Unity/face%20and%20hand%20track/GIF/camera_en.gif) ![Face Capture Function](https://github.com/xinyang-sun/VirtuSync/raw/main/Unity/face%20and%20hand%20track/GIF/face_en.gif) ![Hand Capture Function](https://github.com/xinyang-sun/VirtuSync/raw/main/Unity/face%20and%20hand%20track/GIF/hand_en.gif) ## How to use [Download](https://github.com/xinyang-sun/VirtuSync/releases) the latest package in releases **1. Activate the camera**: Simply extract the archive and run `Tracking.exe/VirtuSync_Camera.exe`. Enter the `Port value`. The default is `5056`. Select the `camera number`. Here you need to try from 0 one by one until you find the camera you want to use. Click `Running` to turn on the camera. **2. Activate 3D motion capture software**: run `Model_test.exe/VirtuSync_Model.exe`. Enter the same `Port value`, the default is `5056`. Click `Start` and the software will stream with the camera. Click `Model` to enter the modeling interface, and start your vtuber experience. 😉 **3. Expression Management System**: - Click on the button `Edit Expressions` to open the model expression configuration page. Select the expression. The `slider` is to control the expression amplitude. The slider bar controls the range of the expression, from 0 to 100, which is convenient for displaying expression combinations. - On the right hand side, the expression binding buttons are `Alpha0` for `keyboard 0`. `Alpha1` for `keyboard 1` and so on. - To change the emoji bindings, click on the input box and enter letters or numbers. Just click on the input box and enter the alphanumeric keys directly. - The default reset button for emoticons is 0. - Once everything is set, click Save to save it. - **Note**: The keypad numeric keys and the keyboard numeric keys do not interoperate when binding. If you use the keypad numeric keys for binding, press the keypad numeric keys when using them. **4. Modify the background color of the model**: Click `Background Color` to open the color palette. Click `Close` to close the palette after modifying the color. ## License [MIT](https://github.com/xinyang-sun/VirtuSync/tree/main?tab=MIT-1-ov-file) ## Reference - [VTuber-Python-Unity](https://github.com/mmmmmm44/VTuber-Python-Unity/tree/main) - [head-pose-estimation](https://github.com/yinguobing/head-pose-estimation) - [VTuber_Unity](https://github.com/kwea123/VTuber_Unity?tab=readme-ov-file) - [Face-Pose-Estimation](https://github.com/jerryhouuu/Face-Yaw-Roll-Pitch-from-Pose-Estimation-using-OpenCV) - [kalidokit](https://github.com/yeemachine/kalidokit?tab=readme-ov-file) - [UnityHandTrackingWithMediapipe](https://github.com/TesseraktZero/UnityHandTrackingWithMediapipe) - [Demonstration of models used：Nemu Nemu_黒パーカーVer（2ndモデル） - VRoid Hub](https://hub.vroid.com/en/characters/9150908110176006593/models/2059315200011240750)