ue-realtime · MetaHuman facial expression and lip-sync prototype

Reconstruction of the true value of the public data set. Problem: The original label of the data set is dirty, sad is actually a crying face, and surprise contains an eye mask. Solution: Abandon the original label, use detection fidelity to confirm (full true glare with high glare channel), signature retrieval → manual review → clean exemplars 52-dimensional scalar vector to do the truth value.

Detection engine redundancy. Problem: A single detection engine has no signal for the FACS key AU (levator buccinator / corrugator rhinis) and always outputs 0, disgust / happy, and the accuracy collapses. Solution: Change to a detection engine that supports full channels, all key AUs are active, multi-channel projection standard 52 dictionary.

FACS signature empirical locking. Problem: The academic textbook signature fails on the self-collected real data - the calibrated wrinkled nose AU of disgust is all angry faces in the real data, and sad is not AU1+15 but AU15+14. Solution: Positive channel + negative constraint weighting (score = positive mean - 0.4 × negative mean), dead channel bypass, parameter-by-parameter comparison table to finalize six emotions.

Local self-collection of the truth value library. Problem: The existing public data sets either have coefficient ranges that are incompatible with industry standards, or are gated, or speak and open their mouths to drown out emotional signals, all of which are unavailable. Solution: Local self-collection of scalar parameters in line with industry standards through desktop cameras, automatic recording + simulated mobile real-time channels, and full process automation.

Emotional semantic judgment. Problem: Parameters match ≠ semantic pair, similar emotions (crying/grievance/fear). The parameter difference is very small but the emotions are different. Solution: Blindly feed the actual parameters without giving the target, LLM outputs the emotion category + intensity + basis, and manually judges the consistency instead of giving numerical scores.

Three-layer verification table. Problem: Universal face detection is seriously inaccurate on non-real faces (animation faces). Mouth opening is 1.0 and the actual measurement is only 0.28. Solution: The actual driving value of the direct reading model is used as an objective guarantee. The detection value is only used in real face scene annotation. The detection is changed to manual triggering to decouple the expression logic.

Local TTS lip synchronization. Problem: The phoneme timing alignment returned by TTS inference is discarded by the official script. Solution: Replicate the inference path to retain phoneme alignment, map primary vowels to finals, and drive mouth shape + smooth transition in real time.

The first version of 3D demo. Problem: The online model source was cut off by the proxy wall, and the asset names were mixed in upper and lower case, causing binding failure. Solution: Change to offline model + full asset localization, case-independent compatible matching, verify that the expression driver actually takes effect.

Full link closed loop for expressions. Problem: There is no automatic verification of the expressions adjusted by the slider, and it is difficult to evaluate the accuracy of LLM emotion → parameter translation. Solution: Add a two-way API service, display 12-channel JSON in real time on the front end, and blindly distinguish emotions and target expressions for side-by-side verification (without applying the scoring template).

Quantitative deterministic verification. Problem: It is suspected that the candidate solution is a temperature sampling output. Different parameters for the same emotion every time will invalidate the LLM addressing. Solution: Run the deterministic test continuously with the same input, the pixel difference is 0, and the slider to vector mapping is deterministically confirmed, dispelling the illusion.

Truth source route selection. Problem: The 63-dimensional implicit vector output by a candidate route has no semantics and cannot be addressed by LLM; the official repo of another reference solution has no code, no weight, and cannot be reproduced. Solution: Lock the ARKit52 blendshape dictionary as the truth source (LLM readable, 0-1 scalar, MetaHuman industry standard compatible), and reject the 2D implicit route.