1.1.4

• Update to Spine WebGL 4.1.24 .
• Add skeleton debug mode into scene render. The checkbox is located under "Canvas Settings" menu.
• Add debug bones checkbox under "Canvas Settings" menu. This include options for bone center color and bone line color, and bone line width ( minimum of 0 and max of 10 ).
• Add debug region attachments checkbox under "Canvas Settings" menu. This include option for region attachment line color.
• Add debug mesh triangle checkbox under "Canvas Settings" menu. This include the option for mesh triangle line color and mesh line opacity ( minimum of 0 and max 100 ).
• Add debug clipping checkbox under "Canvas Settings" menu. This include the option for clipping line color.
• Add left/right pupil pitch/yaw threshold settings to "Single Value Properties" drop-down list under "Model Settings" menu.
• Fix bug that did not load pre-multiplied alpha when it was assigned false.

https://silverstraw.itch.io/spine-vtuber-prototype

Misaki wrote

One thing I thought was it might be a good idea to add a threshold value for the eye pupils, such that subtle changes in position could be ignored. My model's pupils wobbled while winking, so I adjusted the left/right pupil pitch strength and left/right pupil yaw strength parameters, but if these values are set too low, the model’s eyes will not follow the eye movement at all and it is not ideal. FYI, in the model in the video above, left/right pupil pitch strength is set to 4 and left/right pupil yaw strength to 5. I would be happy if you could consider it!

I had to reinstate the threshold mechanism after I forgo it for moving average. You cannot export those settings to .svp yet. I want you to test it out before I make more commitments to the pupil threshold.

but somehow I can export the threshold settings to .svp. (Maybe you have updated this tool after replying to this thread?)

It has been a while since I worked on the source code. I forgot that I have a function that exports settings from a list of default setting values. I updated that list for the left/right pupil pitch/yaw threshold default values, therefore, they got exported. :lol:

Update 1.1.5

• Add "Backface culling" checkbox to "Canvas Settings" menu. The backface of attachments are invisible if checked. The back side is kept consistent with Spine editor.

• Add "Flip Horizontal" checkbox to "Canvas Settings" menu . When checked, the world X-axis is flipped.

• Add empty animation tracks: custom tracks 1 to 4.

• Add customizable expression buttons 0 to 20 underneath the rendering canvas. Button 0 resets all custom tracks thus removing all customizable expressions active. The buttons are usable once the expression slot has been setup.

• Add customizable expression setup interface under "Model Settings" menu.

  1. Add customizable expression slot drop-down menu ( 1 to 20 ) in the setup. Each slot allow you to setup multiple custom track index ( 1 to 4 ), transitional animation, and animation loop that follow after the transitional animation.
     Include a button to add a setup interface row for adding more custom tracks ( 1 to 4 ), transitional animations, and animation loops.

  2. Include a button to add a setup interface row for adding more custom tracks ( 1 to 4 ), transitional animations, and animation loops.

  3. Add a button to remove the last setup interface row. You do not want any rows with incomplete information as you will not be able to finish setting up the customizable expression slot.

  4. Add a button to assign all the custom tracks ( 1 to 4 ), transitional animations, and animation loops to the customizable expression numbered slot ( 1 to 20 ).

  5. Each setup interface row has three parts: custom track index ( 1 to 4 ), transitional animation, animation loop. Custom track index ( 1 to 4 ) and transition animation are required for setup while animation loop is optional. The transition animation and animation loop input fields are drop-down menus that lists all the animation found in file ( .json | .skel ). Transition animation does not loop and the animation loop plays after the transitional animation. There is not any mix duration between transition animation and animation loop.

  6. The customizable expression slots are saved into and can be loaded from SVP file.

• Remove animation selection from Single Value Properties drop-down list with "Model Settings"

https://silverstraw.itch.io/spine-vtuber-prototype

Updated model for testing custom expressions.

https://silverstraw.itch.io/spine-vtube-test-model

I should add another setting for 'returning to default pose' animation to the customizable expression feature. That way you save another expression slot.

After this feature, facial expression recognition AI should be as easy as setting which expression slot you want to use for the AI result.

Spine Vtuber Prototype body track test 1. Preview only.

There is an updated tracking framework but I would have break down the application. It is almost like starting over. Wish me luck .

Some possible gestures if it works .

Face gestures
1 - browDownLeft
2 - browDownRight
3 - browInnerUp
4 - browOuterUpLeft
5 - browOuterUpRight
6 - cheekPuff
7 - cheekSquintLeft
8 - cheekSquintRight
9 - eyeBlinkLeft
10 - eyeBlinkRight
11 - eyeLookDownLeft
12 - eyeLookDownRight
13 - eyeLookInLeft
14 - eyeLookInRight
15 - eyeLookOutLeft
16 - eyeLookOutRight
17 - eyeLookUpLeft
18 - eyeLookUpRight
19 - eyeSquintLeft
20 - eyeSquintRight
21 - eyeWideLeft
22 - eyeWideRight
23 - jawForward
24 - jawLeft
25 - jawOpen
26 - jawRight
27 - mouthClose
28 - mouthDimpleLeft
29 - mouthDimpleRight
30 - mouthFrownLeft
31 - mouthFrownRight
32 - mouthFunnel
33 - mouthLeft
34 - mouthLowerDownLeft
35 - mouthLowerDownRight
36 - mouthPressLeft
37 - mouthPressRight
38 - mouthPucker
39 - mouthRight
40 - mouthRollLower
41 - mouthRollUpper
42 - mouthShrugLower
43 - mouthShrugUpper
44 - mouthSmileLeft
45 - mouthSmileRight
46 - mouthStretchLeft
47 - mouthStretchRight
48 - mouthUpperUpLeft
49 - mouthUpperUpRight
50 - noseSneerLeft
51 - noseSneerRight
52 - tongueOut

Hand gestures
["None", "Closed_Fist", "Open_Palm", "Pointing_Up", "Thumb_Down", "Thumb_Up", "Victory", "ILoveYou"]

Misaki https://silverstraw.itch.io/spine-vtuber-prototype-2
I need your help breaking it the application again . This is using the new facing tracking. I haven't added any tracking smoothing and I am not sure if it is needed. You can tell me your thoughts.

Hopefully I can transfer the body animation code to the new body tracking without much problem like the face animation. The draw order won't be ready with the body tracking because I haven't figure a good solution to handle depth for Spine skeletons.

2.0.1

• Added a check box for enabling body pose tracking. It is located under "AI Settings" menu > "AI Tracking Modes" section > "Body Tracking".
• Added a button for 3d plot graph showing body pose. It is named "Update 3D Plot" and is located under "Camera Settings" menu.

https://silverstraw.itch.io/spine-vtuber-prototype-2

Test model updated.
https://silverstraw.itch.io/spine-vtube-test-model

@Misaki I noticed that this body tracking works better when the whole body is within frame from my experience. It may not completely be your skirt's fault. If you have the chance could you test the body tracking wearing a skirt but have your foot within web camera frame. I think the body tracking would predict better results that way.

The body tracking software makes more assumptive predictions, like when your legs and arms are not in frame.

I might have the software check both if the knee and ankle are out of frame and prevent them from going swaying wildly. So far, the software checks those features individually.

I mentioned this on Discord. The next tracking I should tackle is hand gesture since I think it's straightforward. Various hand gestures: closed fist, open palm, pointing up, thumbs down, thumbs up, victory sign, IU sign. If the tracker detects any of the hand gestures with a high degree than swap out the hand components. I hope it is able to distinguish left and right hands.

2.0.2

• Add a pointing downward behavior for the Spine model legs when the tracked legs are outside the video frame boundary. The leg positions are more assumptive and less reliable when outside of video frame boundary. The legs should default to pointing downward ( standing position ) to look more natural than flailing legs.
• Start to implement web parallel processing for browsers that support it. Multiple threads might speed up computation tasks.
• Defer 3d graph plotting task to optimize canvas rendering.
• Save 3d plot graph camera location when user rotates the graph. Note that the user can only rotate the graph when the graph is not being updated ("Update 3D Plot" is unchecked).
• Replace the word "body" with "torso" in the animation names of "body_rotate", "body_scale_y", and "body_scale_x". Make these animation names less ambiguous.

https://silverstraw.itch.io/spine-vtuber-prototype-2

Test model updated.
https://silverstraw.itch.io/spine-vtube-test-model