Variable audio drift from screen captured VR project in Unity

[slopate]

Hi guys, I'm new to video editing, and I need to sync up the audio and video from two different sources for a Virtual Reality game. Basically, I have the VR itself running in Unity on one computer, with the game output being screen captured (I have screen captures both from OBS and and in-game Unity screen capture). There is no audio output on this computer. All of the audio output happens in a program called Max/MSP on my MacBook, where the audio is being recorded internally. These two computers are linked over an ethernet cable, so there is essentially 0 latency. All of the audio was recorded at 44.1 KHz, and all of the video was recorded at 30 fps, with a constant frame rate.

When I pull up the footage in Premiere, there are two things that are confusing me.

The video files are shorter than the audio files by about 7% (though it varies, so it’s not a consistent error), even though everything lined up exactly when I played it in real time. This would lead me to believe it’s a frame rate or sample rate issue, however I recorded everything consistently at 44.1 and at a constant 30 FPS. Just to be sure, I converted the audio again and encoded the video at a constant frame rate with Handbrake, and still got the same results.
Even when I’ve slowed down the video to sync up with the audio at one point in the video, it’s still not synced at other points. If I choose a point to sync it, it will go out of sync usually within 10-30 seconds, though again this varies.

Inspecting the audio files individually, there is definitely no speeding up, slowing down or buffers being dropped; the audio was recorded correctly and I believe accurately represents the length of the take. This leads me to believe there was some sort of issue with the screen capture, maybe because my CPU and GPU usage were very high during the recording. Maybe some frames are being dropped (Which would make sense having the video being shorter), though I would think programs like premiere and final cut would account for that. Or it could be a Unity bug. Although again, I don’t think this is the case because I have screen captures with both Unity and OBS and both have issues. Beyond that I really have no clue.

The only solution I've been able to muster is adjusting the speed of the video, but since the audio goes out of sync every 20 or 30 seconds, I have to constantly clip the videos and readjust. The videos are 20-30 minutes in length, and I have about 30 to edit, so it would be a long and tedious process to do this for all of them (One video took me several hours).

I would be extremely grateful to hear any input on what the issue might be, and possible solutions. Unfortunately I'm not able to record any more gameplay, so I have to fix everything I have in post. Thank you!

[Selur]

Screen captures are often VFR so unless you also take the time codes into account when reencoding or remuxing you will get async output.
What does Mediainfo about to the video source?

[slopate]

Hi Selur, thanks for the quick reply. Here's what MediaInfo says:


[Attachment 53723 - Click to enlarge]

When I pull it up in Premiere, it does not give me the VFR detected flag. And again just to be sure I re-encoded it, but yeah I'm not sure about the timecodes. Is it possible that it is VFR and these media viewers just can't tell?

[Selur]

MediaInfo also has some detail/debug view where it shows way more data.
And yes, MediaInfo cna be wrong since it only checks the first may be hundred time codes and decides from them whether the source is vfr or not.

[slopate]

I see, thank you. Here's the detailed info from MediaInfo:

General
Count : 331
Count of stream of this kind : 1
Kind of stream : General
Kind of stream : General
Stream identifier : 0
Inform : MPEG-4 (Base Media / Version 2): 1.18 GiB, 20 min 54 s
Count of video streams : 1
Count of audio streams : 1
Video_Format_List : AVC
Video_Format_WithHint_List : AVC
Codecs Video : AVC
Audio_Format_List : AAC LC
Audio_Format_WithHint_List : AAC LC
Audio codecs : AAC LC
Complete name : /Users/samlopate/Desktop/VR Recordings/UnityCapture007.mp4
Folder name : /Users/samlopate/Desktop/VR Recordings
File name extension : UnityCapture007.mp4
File name : UnityCapture007
File extension : mp4
Format : MPEG-4
Format : MPEG-4
Format/Extensions usually used : braw mov mp4 m4v m4a m4b m4p m4r 3ga 3gpa 3gpp 3gp 3gpp2 3g2 k3g jpm jpx mqv ismv isma ismt f4a f4b f4v
Commercial name : MPEG-4
Format profile : Base Media / Version 2
Internet media type : video/mp4
Codec ID : mp42
Codec ID : mp42 (mp41/isom)
Codec ID/Url : http://www.apple.com/quicktime/download/standalone.html
CodecID_Compatible : mp41/isom
File size : 1264640183
File size : 1.18 GiB
File size : 1 GiB
File size : 1.2 GiB
File size : 1.18 GiB
File size : 1.178 GiB
Duration : 1254633
Duration : 20 min 54 s
Duration : 20 min 54 s 633 ms
Duration : 20 min 54 s
Duration : 00:20:54.633
Duration : 00:20:54:19
Duration : 00:20:54.633 (00:20:54:19)
Overall bit rate : 8063809
Overall bit rate : 8 064 kb/s
Frame rate : 30.000
Frame rate : 30.000 FPS
Frame count : 37639
Stream size : 441200
Stream size : 431 KiB (0%)
Stream size : 431 KiB
Stream size : 431 KiB
Stream size : 431 KiB
Stream size : 430.9 KiB
Stream size : 431 KiB (0%)
Proportion of this stream : 0.00035
HeaderSize : 64
DataSize : 1264198999
FooterSize : 441120
IsStreamable : No
Encoded date : UTC 2020-06-01 22:10:30
Tagged date : UTC 2020-06-01 22:10:30
File last modification date : UTC 2020-06-01 22:32:12
File last modification date (local) : 2020-06-01 17:32:12

Video
Count : 379
Count of stream of this kind : 1
Kind of stream : Video
Kind of stream : Video
Stream identifier : 0
StreamOrder : 0
Inform : 7 996 kb/s, 1920*1080 (16:9), at 30.000 FPS, AVC (Baseline@L4) (1 Ref Frames)
ID : 1
ID : 1
Format : AVC
Format : AVC
Format/Info : Advanced Video Codec
Format/Url : http://developers.videolan.org/x264.html
Commercial name : AVC
Format profile : Baseline@L4
Format settings : 1 Ref Frames
Format settings, CABAC : No
Format settings, CABAC : No
Format settings, Reference frames : 1
Format settings, Reference frames : 1 frame
Format settings, GOP : M=1, N=30
Internet media type : video/H264
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 1254633
Duration : 20 min 54 s
Duration : 20 min 54 s 633 ms
Duration : 20 min 54 s
Duration : 00:20:54.633
Duration : 00:20:54:19
Duration : 00:20:54.633 (00:20:54:19)
Bit rate : 7996121
Bit rate : 7 996 kb/s
Width : 1920
Width : 1 920 pixels
Height : 1080
Height : 1 080 pixels
Stored_Height : 1088
Sampled_Width : 1920
Sampled_Height : 1080
Pixel aspect ratio : 1.000
Display aspect ratio : 1.778
Display aspect ratio : 16:9
Rotation : 0.000
Frame rate mode : CFR
Frame rate mode : Constant
FrameRate_Mode_Original : VFR
Frame rate : 30.000
Frame rate : 30.000 FPS
Frame count : 37639
Color space : YUV
Chroma subsampling : 4:2:0
Chroma subsampling : 4:2:0
Bit depth : 8
Bit depth : 8 bits
Scan type : Progressive
Scan type : Progressive
Bits/(Pixel*Frame) : 0.129
Stream size : 1254024963
Stream size : 1.17 GiB (99%)
Stream size : 1 GiB
Stream size : 1.2 GiB
Stream size : 1.17 GiB
Stream size : 1.168 GiB
Stream size : 1.17 GiB (99%)
Proportion of this stream : 0.99161
Encoded date : UTC 2020-06-01 22:10:30
Tagged date : UTC 2020-06-01 22:10:30
mdhd_Duration : 1254633
Codec configuration box : avcC

Audio
Count : 280
Count of stream of this kind : 1
Kind of stream : Audio
Kind of stream : Audio
Stream identifier : 0
StreamOrder : 1
Inform : 66.2 kb/s (96.0 kb/s), 48.0 kHz, 2 channels, AAC LC
ID : 2
ID : 2
Format : AAC
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Commercial name : AAC
Format_AdditionalFeatures : LC
Codec ID : mp4a-40-2
Duration : 1254613
Duration : 20 min 54 s
Duration : 20 min 54 s 613 ms
Duration : 20 min 54 s
Duration : 00:20:54.613
Duration : 00:20:51:13
Duration : 00:20:54.613 (00:20:51:13)
Bit rate mode : CBR
Bit rate mode : Constant
Bit rate : 66150
Bit rate : 66.2 kb/s
Nominal bit rate : 96000
Nominal bit rate : 96.0 kb/s
Channel(s) : 2
Channel(s) : 2 channels
Channel positions : Front: L R
Channel positions : 2/0/0
Channel layout : L R
Samples per frame : 1024
Sampling rate : 48000
Sampling rate : 48.0 kHz
Samples count : 60221424
Frame rate : 46.875
Frame rate : 46.875 FPS (1024 SPF)
Frame count : 58810
Compression mode : Lossy
Compression mode : Lossy
Stream size : 10174020
Stream size : 9.70 MiB (1%)
Stream size : 10 MiB
Stream size : 9.7 MiB
Stream size : 9.70 MiB
Stream size : 9.703 MiB
Stream size : 9.70 MiB (1%)
Proportion of this stream : 0.00804
Encoded date : UTC 2020-06-01 22:10:30
Tagged date : UTC 2020-06-01 22:10:30


I'm not exactly sure how to decode all of this info, but one thing I noticed is that the reported number of frames (37639) seems to be the correct number when you multiply the length of the video by the frame rate ((1254 * 30) + 19 = 36639). Then again I don't know if they generated this frame number simply by doing that calculation rather than counting all the frames (I'm guessing this is the case).

The other thing I noticed is that is says the Frame_rate_mode is CFR, but the Frame_rate_mode_original is VFR. I'm not sure what this means either.

[Selur]

usually:
Frame_rate_mode = frame rate mode indicated by the container
Frame_rate_mode_original = frame rate mode indicated by the stream
-> so the stream was once vfr and either the time codes got removed during the muxing or they are still there and indicate vfr just not early enough to MediaInfo to report it.
iirc

ffmpeg -i "PATH TO FILE" -debug_ts -vcodec copy -an -sn -y -f null -

should show time codes during playback


Cu Selur

[slopate]

Thank you for all the help. I ran the ffmpeg command, and this is extracted from one entry in the stream.

demuxer -> ist_index:0 type:video next_dts:1254533333 next_dts_time:1254.53 next_pts:1254533333 next_pts_time:1254.53 pkt_pts:37636000 pkt_pts_time:1254.53 pkt_dts:37636000 pkt_dts_time:1254.53 off:0 off_time:0
demuxer+ffmpeg -> ist_index:0 type:video pkt_pts:37636000 pkt_pts_time:1254.53 pkt_dts:37636000 pkt_dts_time:1254.53 off:0 off_time:0
muxer <- type:video pkt_pts:37636000 pkt_pts_time:1254.53 pkt_dts:37636000 pkt_dts_time:1254.53 size:34064

And here's the info at the end of the stream:

frame=37639 fps=36131 q=-1.0 Lsize=N/A time=00:20:54.60 bitrate=N/A speed=1.2e+03x
video:1224634kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown


So it seems like the time codes are there. One thing is that it says frame=37639 and fps=36131. I'm not exactly sure what this discrepancy is.

It's definitely possible that VFR wouldn't be detected until later in the video. But in this case, wouldn't encoding it at CFS with Handbrake solve the issue? Are there any other ways which I can fix this?

[Selur]

If the original vfr video plays synch, then yes reencoding it properly to cfr should fix problems.
If the original vfr video already isn't synch, then reencoding to cfr won't help.

DTS is decode time stamp
PTS is the presentation time stamp
So the interesting thing about the output are the 'pkt_dts_time' times in the 'demuxer+ffmpeg' lines and their differences. If their differences are constant your got cfr otherwise you got vfr.

if you call:

ffmpeg -i "PATH TO FILE" -f mkvtimestamp_v2 "timecodefile.txt"

you should get a better readable output in the timecodefile.txt

[slopate]

So I just checked the timecodes on the original, unconverted file, and they indicate that it's CFR for the entire length of the video. So now it's even more perplexing as to why my videos are significantly shorter than my audio files.

Is it possible that the timecodes are wrong and indicate CFR when in actuality the video is VFR? One possible scenario is that when I selected my settings for the screen capture, I selected CFR at 30 fps. When I ran the game, the CPU/GPU usage was too high so it had to drop the frame rate, but just didn't indicate it in the timecodes.

Is there any way to fix this? If the timecodes are constant I have no way of knowing at which points the frame rate was dropped...

[poisondeathray]

Whenever you have 2 separate pieces of equipment, you have high chance of drift unless they are synced up by some mechanism like timecode or genlock . But the magnitude - 7% seems a lot


"no audio output on this computer." - is that AAC audio track "blank" ? Where is that AAC audio from ? Did you mux the audio in separately or is that different audio ?

Why the weird duration?
Duration : 00:20:54.613
Duration : 00:20:51:13
Duration : 00:20:54.613 (00:20:51:13)

[slopate]

Yes, it is blank. The audio I'm using comes from a different computer. To be honest I'm not sure about the duration.

I essentially had two computers running at the same time, one recording the screen capture, and one recording internal audio. My VR game on my windows computer was sending messages over an ethernet cable to my Macbook, which was triggering the audio. The connection was wired and had no latency at all in real time. Both started and stopped at the same time.

But the length of the recorded video is shorter. And even if I time-adjust it in an NLE it is still not synced. So I'm guessing there must be some VFR going on.

[poisondeathray]

What do you mean by "triggering" the audio ? Just at the beginning or was there stop/starting going on , or was it a continuous single take ?

Audio can have timecodes too . Audio can have pauses, gaps governed by audio timecodes. What format is the recorded audio from the macbook ?

In your NLE, does the video look "correct" in terms of frames? Are there gaps or discrepancies in motion when you check a section frame by frame ? If you check the frame count in the editor, does it match with the mediainfo reported value ?

[slopate]

I have C# scripts, which are sending OSC messages over the ethernet cable when the player touches or moves certain objects in the game. These OSC messages are received by Max/MSP (where my audio is coming from) and trigger sounds in the program, in real time, with the VR game. Like I said, when you're playing the game, the audio and video are perfectly synced. Everything was recorded in one continuous take.

The audio was recorded at 44.1 KHz to an .aif file. I'm quite sure the audio is continuous and and was recorded correctly.

The frame count in adobe premiere matches the frame count in mediainfo. I would say the video "looks" correct in that in looks continuous, but there may be discrepancies; it's hard to tell if a one or two frames is skipped every so often. That is what I suspect though I'm not sure how to verify that, or how I'd go about fixing it.

[poisondeathray]

Not sure what is going on, but this is not going to be fun to fix.

You don't have internal recorded audio on the same computer, so you can't use "automatic" methods like plural eyes that sync and correct for drift

[slopate]

Yeah, that's what I feared; so far I've been able to sync the video by changing the speed until it' in sync. Then when it inevitably goes out of sync, I chop the video and sync it again. It takes an hour or two per video but at least it works...

If there's another solution I'd love to hear it

[slopate]

Ah, so I've actually confirmed what the issue is. So I used OBS screen capture to record about half of my videos, and the built in Recorder in Unity to record the other half. Both the OBS videos and Unity videos were out of sync initially. After converting to CFR with handbrake, the Unity videos are still out of sync (and still VFR). The OBS videos, on the other hand, were successfully converted to CFR and sync perfectly. So I suspect what's going on is that the Unity Recorder (which is in a beta version) attempted to capture CFR at 30 fps, but was unable to. So the result is that some frames are missing, but instead of accurately reflecting that in the timecode, it just assumes CFR, and encodes every frame at an even interval. Then when mediainfo/handbrake or whatever other software looks at the time codes, it looks like CFR already, so the conversion does nothing.

I can't think of any way in which I could recover the lost frames since the time codes were lost in the encoding. Maybe someone has another idea, but at least I know what the issue is now.