Hz Unity Face Tracking

Unity Face Tracking for ARKit-Rigged Models (Meta Movement SDK)

End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Meta VR. Uses the public OVRCustomFace extension hook — no OVR_INTERNAL_CODE, ships to 3P.

The model's blendshape names must follow the ARKit naming convention (camelCase, _L/_R suffixes, e.g. eyeBlink_L, jawOpen, mouthSmile_R). An optional prefix like blendShape2.eyeBlink_L is automatically stripped.

When to use

• User has an FBX/GLB/mesh with ARKit-named blendshapes and wants it driven by Meta VR face tracking.
• User asks: "animate this head with my face", "drive these blendshapes from face tracking", "use Movement SDK A2E with my model", "wire ARKit shapes to Meta VR".
• Target device: Quest Pro, Quest 3, Quest 3S (Quest 2 is no-op — no face cameras).

Prerequisites checklist

1. Quest face tracking-capable headset (Pro / 3 / 3S).
2. Packages in Packages/manifest.json:
   • com.meta.xr.sdk.core (Meta XR Core — provides OVRFaceExpressions, OVRCustomFace)
   • com.meta.xr.sdk.movement (Meta Movement SDK — A2E + retargeting helpers)
3. Assets/Oculus/OculusProjectConfig.asset (verify via Project Settings → Meta XR):
   • faceTrackingSupport: 1 (Supported) or 2 (Required)
   • eyeTrackingSupport: 1 if the rig has gaze
4. Android manifest (Assets/Plugins/Android/AndroidManifest.xml) permissions:
   • <uses-feature android:name="oculus.software.face_tracking" android:required="false" />
   • <uses-permission android:name="com.oculus.permission.FACE_TRACKING" />
   • <uses-permission android:name="android.permission.RECORD_AUDIO" /> (required for A2E)
   • For eye gaze: oculus.software.eye_tracking + com.oculus.permission.EYE_TRACKING
5. OVRCameraRig in the scene with OVRManager.FaceTrackingDataSources including Audio (A2E) — if you skip Audio, mouth motion is visual-only.

After any change to OculusProjectConfig, regenerate the manifest (see Regenerating the manifest below).

Approach (high level)

1. Drop the ARKitOVRCustomFace script (in references/ARKitOVRCustomFace.cs) into the project.
2. Add an OVRFaceExpressions component to the OVRCameraRig (or anywhere in the scene).
3. On the GameObject that has the model's SkinnedMeshRenderer, add ARKitOVRCustomFace. Adding the component triggers Reset(), which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names.
4. Wire the component's FaceExpressions field to the OVRFaceExpressions instance.
5. For eye gaze: drop ARKitEyeGazeBlendshapeDriver in (see Eye tracking below).
6. Press Play with Meta XR Link, or build APK and side-load. On first launch, accept the Face Tracking / Eye Tracking / Microphone permission prompts.

That's the whole flow. Details below.

Eye tracking (DO THIS — face expressions alone don't work)

Don't rely on OVRFaceExpressions.EyesLook* for eye gaze. Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is OVREyeGaze, which taps the eye tracker directly.

For ARKit rigs with eyeLook* blendshapes (no eye bones), this skill ships ARKitEyeGazeBlendshapeDriver:

• Reads gaze rotation from two OVREyeGaze components (one per eye, TrackingMode = HeadSpace).
• Computes rotation relative to a head reference (e.g. CenterEyeAnchor).
• Decomposes pitch → eyeLookUp/Down_{L,R}, yaw → eyeLookIn/Out_{L,R} (with the ARKit "_In = toward nose" convention).
• Writes weights in LateUpdate, so it overrides whatever ARKitOVRCustomFace wrote in Update.

For rigs with eye bones (no eyeLook* blendshapes), skip the blendshape decomposition and just parent OVREyeGaze to each eye bone with ApplyRotation = true — the component will rotate the bone directly.

Eye gaze setup (blendshape rigs)

1. Create two empty GameObjects under CenterEyeAnchor (or your head transform): LeftEyeGaze, RightEyeGaze.
2. Add OVREyeGaze to each. Set Eye = Left / Right, TrackingMode = HeadSpace, ApplyPosition = false, ApplyRotation = true, ConfidenceThreshold = 0.5.
3. Add ARKitEyeGazeBlendshapeDriver to the head's SkinnedMeshRenderer GameObject (alongside ARKitOVRCustomFace). Wire leftEye, rightEye, and referenceFrame (= CenterEyeAnchor).
4. Tweak maxAngleDeg (default 30°) and smoothing (default 0.4) to taste.

Step-by-step

1. Install the script

Copy references/ARKitOVRCustomFace.cs into Assets/Scripts/ARKitOVRCustomFace.cs. It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a MapBlendshapes() method that scans SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i), strips any prefix before the last ., lowercases, and matches against the table. Unmatched mesh blendshapes are set to OVRFaceExpressions.FaceExpression.Max (sentinel — skipped at runtime).

2. Enable Movement SDK + permissions

Project settings:

Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)

Then regenerate the manifest.

Regenerating the manifest

Against a live Editor, call the SDK's generator through run_script — the type resolves directly, so no reflection is needed:

// AgentScripts/UpdateManifest.cs
public static class UpdateManifest
{
    public static string Run()
    {
        OVRManifestPreprocessor.GenerateOrUpdateAndroidManifest(true);   // silentMode: true
        return "ok";
    }
}

unity command run_script --file AgentScripts/UpdateManifest.cs --entry UpdateManifest.Run --format json

silentMode: true is required — without it the generator opens a modal dialog that blocks the request. Then confirm the face-tracking entries from the checklist above actually landed in Assets/Plugins/Android/AndroidManifest.xml. See hz-unity-meta-core-sdk for the full manifest workflow and the rule against hand-editing managed entries.

Alternatives: the Editor menu Meta → Tools → Update AndroidManifest.xml, or the meta_update_android_manifest tool if you are on a Unity MCP server with the Meta Unity extension.

3. Scene setup

Scene Hierarchy
├── OVRCameraRig                         (OVRQuickActionsAPI.AddOVRInteractionRig(), or the
│   │                                     OVRCameraRig prefab; meta_add_camerarig under MCP)
│   └── (add) OVRFaceExpressions         component
└── YourHeadModel
    └── ...SkinnedMeshRenderer GO...
        ├── SkinnedMeshRenderer          (existing)
        └── (add) ARKitOVRCustomFace     component
             └── FaceExpressions = the OVRFaceExpressions ref
             └── retargetingType = Custom (set automatically by base when overriding)
             └── Mappings[] = auto-filled on Reset()
             └── BlendShapeStrengthMultiplier = 100 (default; OVR weights are 0–1, mesh wants 0–100)

If the head has multiple SkinnedMeshRenderers (e.g. separate teeth/tongue meshes), add ARKitOVRCustomFace to each one.

4. OVRManager — enable Audio as a data source (A2E)

On OVRCameraRig's OVRManager component:

• Face Tracking Data Sources → check Visual AND Audio (Audio = A2E; produces mouth shapes from microphone when the visual face cameras can't see something — talking, occlusion, etc.).

5. Trigger the mapping if the component already existed

MapBlendshapes() runs automatically when the component is added (Reset()) and from OnValidate() when Mappings is empty. If you need to remap manually (e.g. after re-importing the FBX), use the component's inspector context menu → Map Blendshapes, or from a script:

go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();

Against a live Editor via run_script, the project script's type is directly referenceable (it lives in Assets/), so just call it:

// AgentScripts/Remap.cs
using UnityEngine;

public static class Remap
{
    public static string Run(string modelName)
    {
        var smr = GameObject.Find(modelName).GetComponentInChildren<SkinnedMeshRenderer>();
        smr.gameObject.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();
        return $"Remapped {smr.sharedMesh.blendShapeCount} blendshapes on {smr.name}";
    }
}

unity command run_script --file AgentScripts/Remap.cs --entry Remap.Run --args '["YourHeadModel"]' --format json
unity command save_scene --format json

Under a Unity MCP server the OVR/MSDK types are not visible to the dynamic assembly, so the reflection-free workaround there is SendMessage:

GameObject.Find("YourHeadModel")
    .GetComponentInChildren<SkinnedMeshRenderer>().gameObject
    .SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);