Sofdec Cue Points [Part 2]
In the first part of this tutorial, we exported markers from our After Effects project into a text file. Then, we used this data in the Sofdec Encoding Wizard to embed cue points in our video. In this part, we will explain how to create a Unity script that reacts when the cue points from the .usm file we created are reached during video playback.
To showcase this, we will be using the same scene, representing a virtual fashion show.
Playback
The full Unity script that starts the video and reacts to the cue points is described below.
When playing back a video using CRI Sofdec, there is a time lag between the command to start the playback (Start function) and when the first rendering can actually be performed.
Therefore, it is recommended to first call the playback preparation command (PrepareForRendering), which also includes drawing preparation, and then call the Start function, and enable drawing at the same time.
Callback
By registering a CuePointCallback() function with CriMana (Sofdec’s runtime library), the script will be notified when a specific cue point has been reached during the video playback. You can find more information about this callback in the official documentation:
https://game.criware.jp/manual/unity_plugin_en/latest/contents/classCriWare_1_1CriMana_1_1Player.html#a52f34523349d55e89bccb13db73f8c98
In our callback function, the script switches between two light objects and updates the pose of a Humanoid character.
For our example, we kept the character animation relatively basic. It simply cycles through various poses, one each time “Action” is triggered, as shown in the diagram below.
Conclusion
Using cue points to synchronize 3D content or visual effects with a movie can add a whole new dimension to the playback of song videos in Unity, for example.
If a song and VFX are prepared in the video file, and cue points are set to match the song’s timing, it becomes possible to synchronize them with the movement of a Humanoid character (including mouth movements, choreography…), and various objects or lights present in the Unity scene.