Animaze Programmable Tracker

Overview

Animaze supports a programmable tracker that enables 3rd party apps to directly retarget their tracking data on the Animaze avatar (Animaze 3D, Live2D, VRM, Ready Player Me, GLB).

The system has a simple architecture, once the Programmable Tracker is enabled in Animaze, the app listens to a UDP socket (address and port are customizable) for a stream of tracking data that is applied directly on the avatar. The format of the tracking data is Animaze’s Generic Action Coding System, which is an extension of the FACS system with additional action units for head & body motions, shoulders, hands & arms, general emotions, visemes, etc.

Developer Guide

The usage should be relatively simple: open a UDP socket that matches the IP and Port of the Animaze app’s Programmable Tracker and send an array of 157 float values representing the values of the Generic Action Units presented in the Annex - Generic Action Units (names and neutral values).

The expected value of an Action Unit is in the range 0 to 1, going outside will exhibit undefined behavior on the avatar. Action unit values are not purposely clamped, rather there are underlaying systems that may clamp the values to [0, 1] range.

The values set by the Programmable Tracker will take precedence to other trackers that are active at the same time, e.g. if HeadYaw from Default webcam tracker is 0.5 (neutral, facing forward) and the Programmable Tracker receives a value of 0.0, the final avatar head yaw will be facing to the left, because the 0.0 will overwrite the Default’s webcam tracker value.

Some of the Action Units may not affect the avatar at all, due to missing skeletal animations or blendshapes in case of 3D avatars, and Parameters in case of Live2D avatars. Additionally, there are avatar-sided overrides and filtering that may reduce the effect of the sent Action Units, e.g. look-at-camera override will lock the avatar’s gaze to the render camera, if the eye’s gaze action units (Eye[L/R]PupPos[X/Y]) are in a certain interval. The overrides and filtering can be customized or altogether disabled from Animaze app Settings > Advanced Tracking Configuration > Overrides.

Annex - Generic Action Units (names and neutral values)

Index

Action Unit

Neutral Value

0

HeadPitch

0.5

1

HeadYaw

0.5

2

HeadRoll

0.5

3

BodyPitch

0.5

4

BodyYaw

0.5

5

BodyRoll

0.5

6

BodyUpDown

0.5

7

MouthSmileUnsure

0

8

Unused

0

9

MouthSmileL

0

10

MouthSmileR

0

11

MouthUnsureL

0

12

MouthUnsureR

0

13

MouthMoveUp

0

14

MouthOpen

0

15

JawDrop

0

16

PursedLips

0

17

PursedLipsLR

0.5

18

MouthUnveiledTeeth

0

19

MouthUnveiledTeethDown

0

20

MouthUpperLipMBP

0

21

MouthSmileWithTeethCovered

0

22

JawLR

0.5

23

JawF

0

24

NoseWrinklerU

0

25

NoseWrinklerD

0

26

BrowIntLeftU

0

27

BrowIntLeftD

0

28

BrowIntRightU

0

29

BrowIntRightD

0

30

BrowExtLeftU

0

31

BrowExtLeftD

0

32

BrowExtRightU

0

33

BrowExtRightD

0

34

BrowCenterLeftUp

0

35

BrowCenterLeftDown

0

36

BrowCenterRightUp

0

37

BrowCenterRightDown

0

38

BrowSqueezeInLeft

0

39

BrowSqueezeInRight

0

40

BrowStrechOut

0

41

EyeLClosed

0

42

EyeRClosed

0

43

EyeLSize

0

44

EyeRSize

0

45

EyeLSquint

0

46

EyeRSquint

0

47

EyeLPupPosX

0.5

48

EyeLPupPosY

0.5

49

EyeRPupPosX

0.5

50

EyeRPupPosY

0.5

51

EyePupilDilate

0

52

TongueOut

0

53

TongueLR

0.5

54

TongueUD

0.5

55

PuffCheekL

0

56

PuffCheekR

0

57

HandR_LR

0.5

58

HandR_UD

0.5

59

HandR_CloseFront

0.5

60

HandR_NaturalPose

1

61

HandL_LR

0.5

62

HandL_UD

0.5

63

HandL_CloseFront

0.5

64

HandL_NaturalPose

1

65

HandR_SoloLR

0.5

66

HandR_SoloUD

0.5

67

HandR_SoloTwist

0.5

68

HandL_SoloLR

0.5

69

HandL_SoloUD

0.5

70

HandL_SoloTwist

0.5

71

HandR_Finger1

0

72

HandR_Finger2

0

73

HandR_Finger3

0

74

HandR_Finger4

0

75

HandR_Finger5

0

76

HandL_Finger1

0

77

HandL_Finger2

0

78

HandL_Finger3

0

79

HandL_Finger4

0

80

HandL_Finger5

0

81

Unused

0

82

ShoulderR_Up

0

83

ShoulderL_Up

0

84

Unused

0

85

Viseme_AA

0

86

Viseme_AH

0

87

Viseme_AO

0

88

Viseme_AW

0

89

Viseme_OY

0

90

Viseme_EH

0

91

Viseme_IH

0

92

Viseme_EY

0

93

Viseme_Y

0

94

Viseme_R

0

95

Viseme_L

0

96

Viseme_W

0

97

Viseme_M

0

98

Viseme_N

0

99

Viseme_CH

0

100

Viseme_FV

0

101

Viseme_S

0

102

Viseme_TH

0

103

Lipsync_F

0

104

Lipsync_L

0

105

Lipsync_TH

0

106

Lipsync_Y

0

107

EarLeft_U

0

108

EarLeft_D

0

109

EarRight_U

0

110

EarRight_D

0

111

MouseAndStylus_X

0

112

MouseAndStylus_Y

0

113

MouseAndStylus_LeftClick

0

114

MouseAndStylus_RightClick

0

115

MouseAndStylus_Scroll

0

116

MouseAndStylus_Inclination

0

117

MouseAndStylus_Pressure

0

118

Keyboard_HandR_Finger1

0

119

Keyboard_HandR_Finger2

0

120

Keyboard_HandR_Finger3

0

121

Keyboard_HandR_Finger4

0

122

Keyboard_HandR_Finger5

0

123

Keyboard_HandL_Finger1

0

124

Keyboard_HandL_Finger2

0

125

Keyboard_HandL_Finger3

0

126

Keyboard_HandL_Finger4

0

127

Keyboard_HandL_Finger5

0

128

Gamepad_Button_A

0

129

Gamepad_Button_B

0

130

Gamepad_Button_X

0

131

Gamepad_Button_Y

0

132

Gamepad_Button_BACK

0

133

Gamepad_Button_GUIDE

0

134

Gamepad_Button_START

0

135

Gamepad_Button_LEFTSTICK

0

136

Gamepad_Button_RIGHTSTICK

0

137

Gamepad_Button_LEFTSHOULDER

0

138

Gamepad_Button_RIGHTSHOULDER

0

139

Gamepad_Button_DPAD_UP

0

140

Gamepad_Button_DPAD_DOWN

0

141

Gamepad_Button_DPAD_LEFT

0

142

Gamepad_Button_DPAD_RIGHT

0

143

Gamepad_Button_MISC1

0

144

Gamepad_Button_PADDLE1

0

145

Gamepad_Button_PADDLE2

0

146

Gamepad_Button_PADDLE3

0

147

Gamepad_Button_PADDLE4

0

148

Gamepad_Button_TOUCHPAD

0

149

Emotion_eAnger

0

150

Emotion_eDisgust

0

151

Emotion_eFear

0

152

Emotion_eHappiness

0

153

Emotion_eSadness

0

154

Emotion_eSurprise

0

155

Emotion_eNeutral

0

156

Procedural_Breath

0

Annex - Sample External Programmable Tracker

The sample Programmable Tracker is a Python script that runs the MediaPipe tracker and sends the tracking data through a UDP socket. You can get the archive from here:

Requirements:

  • Python 3.11+

  • Modules

    • opencv

    • mediapipe

    • numpy