Sofdec Cue Points

In this 2-part tutorial, we will see how to synchronize Unity scenes with Sofdec videos. Indeed, when encoding your videos with Sofdec, it is possible to embed multiple cue points. Once in Unity, a special callback function can be added to your script that will be triggered when one of the cue points is reached during playback, allowing you to synchronize various objects in your scene with the events in the video.

Introduction

To showcase this feature, we will create a demo in Unity that simulates a virtual fashion show, where spotlights and character poses change in sync with the shockwave effects displayed in the video.

In this first part, we will introduce techniques for automatically generating cue point files using After Effects, and in the next post, we will introduce the script used to synchronize the objects in Unity with the video.

The Sofdec Encoding Wizard is a user-friendly tool to encode a video into a .usm file that can then be played back by the Sofdec run-time library. The tool offers many options, including the possibility to embed cue points by providing a text file containing their time stamps.

We will now introduce some tips for a smooth workflow when preparing the cue points, using Adobe After Effects, and then we will import them into the Sofdec Ending Wizard, along with the video.

Getting the cue points

Below is an example of a background video composition we made in Adobe After Effect for a virtual fashion show.

We added some shockwave-like visual effects, accompanied by braaam sounds, and we inserted markers where we want our cue points to be.

When placing a large number of markers on the timeline, manually checking the individual positions can be cumbersome and time-consuming. Therefore, we decided to write a script to automate the work in After Effects. Below is a jsx script that will collect the timestamps of the markers and save them in a text file.

Once saved with the .jsx extension, you can run the script from After Effects and save the .txt file in the same directory as your project file.

The following page describes the format of the cue point information file. It is possible to include additional information if needed: https://www.criware.cn/sdk_manual/SOFDEC_pc/contents/usr_cmew_data4.html

The Sofdec Encoding Wizard

Once the video and the cue point information file are ready, it is time to import them into the Sofdec Encoding Wizard. This is done on step 2 of the process. The .usm file that will be generated will contain whatever cue point information you put in the .txt file. In Unity, place the video data in the StreamingAssets folder of your project, so that it is easier to play it back with CriMana (the Sofdec run-time library).

In the second part of this tutorial, we will switch to Unity and learn how to trigger a callback function when the cue points embedded in the .usm video file are reached during playback. This will allow us to synchronize lights and animations with the video.