1 Overview
2 Developer Guide
3 Annex - Generic Action Units (names and neutral values)
4 Annex - Sample External Programmable Tracker

Overview

Animaze supports a programmable tracker that enables 3rd party apps to directly retarget their tracking data on the Animaze avatar (Animaze 3D, Live2D, VRM, Ready Player Me, GLB).

The system has a simple architecture, once the Programmable Tracker is enabled in Animaze, the app listens to a UDP socket (address and port are customizable) for a stream of tracking data that is applied directly on the avatar. The format of the tracking data is Animaze’s Generic Action Coding System, which is an extension of the FACS system with additional action units for head & body motions, shoulders, hands & arms, general emotions, visemes, etc.

Developer Guide

The usage should be relatively simple: open a UDP socket that matches the IP and Port of the Animaze app’s Programmable Tracker and send an array of 157 float values representing the values of the Generic Action Units presented in the Annex - Generic Action Units (names and neutral values).

The expected value of an Action Unit is in the range 0 to 1, going outside will exhibit undefined behavior on the avatar. Action unit values are not purposely clamped, rather there are underlaying systems that may clamp the values to [0, 1] range.

The values set by the Programmable Tracker will take precedence to other trackers that are active at the same time, e.g. if HeadYaw from Default webcam tracker is 0.5 (neutral, facing forward) and the Programmable Tracker receives a value of 0.0, the final avatar head yaw will be facing to the left, because the 0.0 will overwrite the Default’s webcam tracker value.

Some of the Action Units may not affect the avatar at all, due to missing skeletal animations or blendshapes in case of 3D avatars, and Parameters in case of Live2D avatars. Additionally, there are avatar-sided overrides and filtering that may reduce the effect of the sent Action Units, e.g. look-at-camera override will lock the avatar’s gaze to the render camera, if the eye’s gaze action units (Eye[L/R]PupPos[X/Y]) are in a certain interval. The overrides and filtering can be customized or altogether disabled from Animaze app Settings > Advanced Tracking Configuration > Overrides.

Annex - Generic Action Units (names and neutral values)

Index	Action Unit	Neutral Value
0	HeadPitch	0.5
1	HeadYaw	0.5
2	HeadRoll	0.5
3	BodyPitch	0.5
4	BodyYaw	0.5
5	BodyRoll	0.5
6	BodyUpDown	0.5
7	MouthSmileUnsure	0
8	Unused	0
9	MouthSmileL	0
10	MouthSmileR	0
11	MouthUnsureL	0
12	MouthUnsureR	0
13	MouthMoveUp	0
14	MouthOpen	0
15	JawDrop	0
16	PursedLips	0
17	PursedLipsLR	0.5
18	MouthUnveiledTeeth	0
19	MouthUnveiledTeethDown	0
20	MouthUpperLipMBP	0
21	MouthSmileWithTeethCovered	0
22	JawLR	0.5
23	JawF	0
24	NoseWrinklerU	0
25	NoseWrinklerD	0
26	BrowIntLeftU	0
27	BrowIntLeftD	0
28	BrowIntRightU	0
29	BrowIntRightD	0
30	BrowExtLeftU	0
31	BrowExtLeftD	0
32	BrowExtRightU	0
33	BrowExtRightD	0
34	BrowCenterLeftUp	0
35	BrowCenterLeftDown	0
36	BrowCenterRightUp	0
37	BrowCenterRightDown	0
38	BrowSqueezeInLeft	0
39	BrowSqueezeInRight	0
40	BrowStrechOut	0
41	EyeLClosed	0
42	EyeRClosed	0
43	EyeLSize	0
44	EyeRSize	0
45	EyeLSquint	0
46	EyeRSquint	0
47	EyeLPupPosX	0.5
48	EyeLPupPosY	0.5
49	EyeRPupPosX	0.5
50	EyeRPupPosY	0.5
51	EyePupilDilate	0
52	TongueOut	0
53	TongueLR	0.5
54	TongueUD	0.5
55	PuffCheekL	0
56	PuffCheekR	0
57	HandR_LR	0.5
58	HandR_UD	0.5
59	HandR_CloseFront	0.5
60	HandR_NaturalPose	1
61	HandL_LR	0.5
62	HandL_UD	0.5
63	HandL_CloseFront	0.5
64	HandL_NaturalPose	1
65	HandR_SoloLR	0.5
66	HandR_SoloUD	0.5
67	HandR_SoloTwist	0.5
68	HandL_SoloLR	0.5
69	HandL_SoloUD	0.5
70	HandL_SoloTwist	0.5
71	HandR_Finger1	0
72	HandR_Finger2	0
73	HandR_Finger3	0
74	HandR_Finger4	0
75	HandR_Finger5	0
76	HandL_Finger1	0
77	HandL_Finger2	0
78	HandL_Finger3	0
79	HandL_Finger4	0
80	HandL_Finger5	0
81	Unused	0
82	ShoulderR_Up	0
83	ShoulderL_Up	0
84	Unused	0
85	Viseme_AA	0
86	Viseme_AH	0
87	Viseme_AO	0
88	Viseme_AW	0
89	Viseme_OY	0
90	Viseme_EH	0
91	Viseme_IH	0
92	Viseme_EY	0
93	Viseme_Y	0
94	Viseme_R	0
95	Viseme_L	0
96	Viseme_W	0
97	Viseme_M	0
98	Viseme_N	0
99	Viseme_CH	0
100	Viseme_FV	0
101	Viseme_S	0
102	Viseme_TH	0
103	Lipsync_F	0
104	Lipsync_L	0
105	Lipsync_TH	0
106	Lipsync_Y	0
107	EarLeft_U	0
108	EarLeft_D	0
109	EarRight_U	0
110	EarRight_D	0
111	MouseAndStylus_X	0
112	MouseAndStylus_Y	0
113	MouseAndStylus_LeftClick	0
114	MouseAndStylus_RightClick	0
115	MouseAndStylus_Scroll	0
116	MouseAndStylus_Inclination	0
117	MouseAndStylus_Pressure	0
118	Keyboard_HandR_Finger1	0
119	Keyboard_HandR_Finger2	0
120	Keyboard_HandR_Finger3	0
121	Keyboard_HandR_Finger4	0
122	Keyboard_HandR_Finger5	0
123	Keyboard_HandL_Finger1	0
124	Keyboard_HandL_Finger2	0
125	Keyboard_HandL_Finger3	0
126	Keyboard_HandL_Finger4	0
127	Keyboard_HandL_Finger5	0
128	Gamepad_Button_A	0
129	Gamepad_Button_B	0
130	Gamepad_Button_X	0
131	Gamepad_Button_Y	0
132	Gamepad_Button_BACK	0
133	Gamepad_Button_GUIDE	0
134	Gamepad_Button_START	0
135	Gamepad_Button_LEFTSTICK	0
136	Gamepad_Button_RIGHTSTICK	0
137	Gamepad_Button_LEFTSHOULDER	0
138	Gamepad_Button_RIGHTSHOULDER	0
139	Gamepad_Button_DPAD_UP	0
140	Gamepad_Button_DPAD_DOWN	0
141	Gamepad_Button_DPAD_LEFT	0
142	Gamepad_Button_DPAD_RIGHT	0
143	Gamepad_Button_MISC1	0
144	Gamepad_Button_PADDLE1	0
145	Gamepad_Button_PADDLE2	0
146	Gamepad_Button_PADDLE3	0
147	Gamepad_Button_PADDLE4	0
148	Gamepad_Button_TOUCHPAD	0
149	Emotion_eAnger	0
150	Emotion_eDisgust	0
151	Emotion_eFear	0
152	Emotion_eHappiness	0
153	Emotion_eSadness	0
154	Emotion_eSurprise	0
155	Emotion_eNeutral	0
156	Procedural_Breath	0

Annex - Sample External Programmable Tracker

The sample Programmable Tracker is a Python script that runs the MediaPipe tracker and sends the tracking data through a UDP socket. You can get the archive from here:

Requirements:

Python 3.11+
Modules
- opencv
- mediapipe
- numpy

Animaze Docs

Animaze Programmable Tracker

Overview

Developer Guide

Annex - Generic Action Units (names and neutral values)

Annex - Sample External Programmable Tracker