Error when encoding VFR video

[naoto89]

The source is a MPEG2 Transport Stream video (TV capture dump with set top box) and after indexing it says is a hybrid video (<85% VIDEO). This is annoying because the content broadcasted by a TV channel (NTSC) is an anime series created at 24fps (23.976fps). In theory, TIVTC should be applied to have the 29.97 fps video at the original 23.976 fps but encoding it like this I only get a video that doesn't play at the same audio speed. In the same captured video there are commercials and according to the indexing it's 100% VIDEO (NTSC), so I don't understand why with that anime series it's not a similar case. Previously I encoded anime captured from other TV channels (NTSC) that when indexing says that it's 100% VIDEO and I encode them at 23.976 fps without problems. Encoding at 29.97 fps would be an easy escape for anyone but it's not what I have in mind and not having many options I decided for VFR.

I followed this gide and the first pass works but the second pass I get this error.

avs [error]: TDecimate:  input error (mode 5 and 6, all frames must have entries)!

This is the Avisynth script for the first pass.

MPEG2Source("D:\TS\INDEX\0001.d2v", cpu=0)
Trim(0,39008).Spline36Resize(640,480)
TFM(mode=1,output="tfm.txt")
TDecimate(mode=4,output="stats.txt")

And this is the script for the second pass.

MPEG2Source("D:\TS\INDEX\0001.d2v", cpu=0)
Trim(0,39008).Spline36Resize(640,480)
TFM(mode=1)
TDecimate(mode=5,hybrid=2,dupthresh=1.0,input="stats.txt",tfmin="tfm.txt",mkvout="timecodes.txt")

[jagabo]

Did you build the index file in "honor pulldown flags" mode?

[naoto89]

Did you build the index file in "honor pulldown flags" mode?

I hadn't noticed that but, yes, that option was activated. Should I choose another mode for this?

I had forgotten that the indexer has "Field Operation" in the Video menu although honestly I rarely used its modes, like when I had to rip DVD episodes and I used "Forced Film" because it's "100% Film". Most of the time I use NTSC videos and that's why "Honor Pulldown Flags" is always on.

[hello_hello]

I've bumped into a similar problem before. It was a fair while ago but if I remember correctly the metrics file created by TDecimate had an additional entry at the end, causing a similar error. It sounds like in your case though, you've opened the first pass script after it's been run and the output files have been over-written, so the first pass would have to be run again. Once it's run make sure you don't open the first pass script again.

I tend to put both passes in the same script and just comment out the appropriate lines, but it's personal preference. I'd also refrain from putting any filters before TFM/TDecimate. Telecined or interlaced video should never be resized anyway, but converted to progressive first (although it's probably okay as long as the height isn't resized). Filtering isn't needed for the first pass though and will likely slow it down. Trim() before TFM/TDecimate would be the exception to that rule, because it's not a filter as such.

MPEG2Source("D:\TS\INDEX\0001.d2v", cpu=0)

Trim(0,39008)

# Comment out the next two lines for the second pass

TFM(mode=1,output="tfm.txt")
TDecimate(mode=4,output="stats.txt")

# Uncomment the remaining lines for the second pass

# TFM(mode=1)
# TDecimate(mode=5,hybrid=2,dupthresh=1.0,input="stats.txt",tfmin="tfm.txt",mkvout="timecodes.txt")

# Spline36Resize(640,480)

The guide you linked to says to "play through" the video. For the first pass though, you'd use the "File/Run Analysis Pass" menu. That'll be much faster and ensure the video is played from beginning to end, so the metrics file should contain the correct number of frames.

Since that guide was created, the x264 encoder has been given the ability to encode as VFR. If that's what you're using, the "timecodes.txt" file can be added to the x264 command line and the encoder will use it to encode in VFR mode. For example:

--level 4.1 --preset slow --tune film --crf 19.0 --tcfile-in "C:\Wherever\timecodes.txt"

The timecodes file doesn't need to exist to add it to the command line. It'll be created as soon as the 2nd pass script is opened so you only need to ensure it's given the correct name in the command line. It doesn't have to be done that way, but it'll change the way the video is encoded a little as x264 is frame rate aware.

By the way, applying TFM/TDecimate for IVTC and a constant frame rate of 23.976fps should never mess with the audio sync. TDecimate should always drop/remove one frame in every five. If you use the appropriate hybrid mode, for each group of five frames TDecimate will either drop a frame if it sees them as telecined, or if it sees them as video it'll convert them to 23.976fps by reducing the number of frames to four using frame blending. The difference between CFR and VFR modes is the number of frames aren't reduced for the video sections and they stay as 29.97fps. I'm not sure why either method would result in a loss of audio sync, unless the audio stream has gaps that aren't being accounted for when it's extracted, or something like that.

PS DGIndex reports both hard (encoded) telecine and interlaced video as VIDEO, as it has no way of knowing which it is. TFM has to work that out. It only reports FILM for sections with soft telecine (using repeat field flags).

[naoto89]

I'd also refrain from putting any filters before TFM/TDecimate. Telecined or interlaced video should never be resized anyway, but converted to progressive first (although it's probably okay as long as the height isn't resized). Filtering isn't needed for the first pass though and will likely slow it down. Trim() before TFM/TDecimate would be the exception to that rule, because it's not a filter as such.

Ok, I think that was my big mistake, having put resize before the TFM/TDecimate in the first and second pass. The encoding works after modifying that.

The guide you linked to says to "play through" the video. For the first pass though, you'd use the "File/Run Analysis Pass" menu. That'll be much faster and ensure the video is played from beginning to end, so the metrics file should contain the correct number of frames.

For the first pass I use "Queue Analysis Pass" of meGUI, it shouldn't make difference if I do with VirtualDub, right?

Since that guide was created, the x264 encoder has been given the ability to encode as VFR. If that's what you're using, the "timecodes.txt" file can be added to the x264 command line and the encoder will use it to encode in VFR mode.

I didn't know that I could add timecodes to the x264 command line, that would avoid having to add it in mkvmerge at the time of mux with the other files.

The timecodes file doesn't need to exist to add it to the command line. It'll be created as soon as the 2nd pass script is opened so you only need to ensure it's given the correct name in the command line. It doesn't have to be done that way, but it'll change the way the video is encoded a little as x264 is frame rate aware.

That leads me to another question. AvsPmod as such will always preview the video as CFR even though it will later be VFR and the framerate (which can be any non-common number) will always be the same. If I am going to create chapters and preview with the script of the second pass to search the frames and get the times for the chapter file (.txt) and create the QPF file of the frames that will be converted into keyframes. Won't there be problems with the time set in the chapter file if the video is VFR?

By the way, applying TFM/TDecimate for IVTC and a constant frame rate of 23.976fps should never mess with the audio sync. TDecimate should always drop/remove one frame in every five. If you use the appropriate hybrid mode, for each group of five frames TDecimate will either drop a frame if it sees them as telecined, or if it sees them as video it'll convert them to 23.976fps by reducing the number of frames to four using frame blending. The difference between CFR and VFR modes is the number of frames aren't reduced for the video sections and they stay as 29.97fps. I'm not sure why either method would result in a loss of audio sync, unless the audio stream has gaps that aren't being accounted for when it's extracted, or something like that.

When I change the container of the original TS file to MKV the sync is the same but if I encode even with the timecodes, the new video doesn't match the audio. The audio file is unchanged even after uncompressed trims but since it's an MP2 file I decided to convert it to FLAC which shouldn't be a problem either.

Edit:
After modifying the AVS scripts of each pass, the first attempt to encode worked, but after trying again for the second time, the same TDecimate error appeared.

[jagabo]

Did you build the index file in "honor pulldown flags" mode?

I hadn't noticed that but, yes, that option was activated. Should I choose another mode for this?

No, you always want to use "honor pulldown flags" mode with mixed hard and soft telecine.

[hello_hello]

For the first pass I use "Queue Analysis Pass" of meGUI, it shouldn't make difference if I do with VirtualDub, right?

Either is fine. I just assumed you were using VirtualDub because you linked to the quide.

I didn't know that I could add timecodes to the x264 command line, that would avoid having to add it in mkvmerge at the time of mux with the other files.

There's definitely no need to add the timecodes when muxing if the encoder is writing directly to an MKV as it's output. There's probably no need if you're outputting a raw stream and muxing later either, but I can't remember.

That leads me to another question. AvsPmod as such will always preview the video as CFR even though it will later be VFR and the framerate (which can be any non-common number) will always be the same. If I am going to create chapters and preview with the script of the second pass to search the frames and get the times for the chapter file (.txt) and create the QPF file of the frames that will be converted into keyframes. Won't there be problems with the time set in the chapter file if the video is VFR?

For a VFR encode the script always runs at a constant frame rate because Avisynth doesn't support VFR (obviously), and the frame rate used by TIVTC is always the average frame rate, so it'll be different each time.

I'm not sure I've ever created a QPF file, but you'd be correct about not being able to do it as you would for CFR video. I don't think it's necessary anyway. If the chapter's not on a keyframe the worst that should happen is the player might take a fraction longer to display the first frame after navigating to a chapter.

As the chapters correspond to a particular point in time rather than a particular frame, working out where they should go should be do-able if you're creating them yourself. You could either encode the video first and use a media player to find the correct times for each chapter, or if you want to do it with the script first, you could use TimecodeFPS to temporarily convert back to a constant frame rate, or just do it without TFM/TDecimate in the script so it's still the original 29.97fps.

When I change the container of the original TS file to MKV the sync is the same but if I encode even with the timecodes, the new video doesn't match the audio. The audio file is unchanged even after uncompressed trims but since it's an MP2 file I decided to convert it to FLAC which shouldn't be a problem either.

Gaps in the audio stream would be the most likely reason. If you've remuxed the original TS file as an MKV and the sync is still okay, instead of extracting the audio, open the encoded video with MKVToolNix, add the original TS file or remuxed MKV, deselect the original video stream and remux. That should create a new MKV with the original A/V sync. If you need to extract the audio, use MeGUI's HD Streams Extractor to extract it with eac3to. It'll attempt to replace any gaps with silence and the extracted stream will hopefully be okay. eac3to creates a log file so you can check that to see if it found anything to fix.

If the A/V sync is out by the same amount from start to finish, it might be because the original video stream has a delay applied. When you re-encode, any video delay is usually lost. You can check that with gMKVExtractGUI. When you open an MKV it'll display any video delay, along with two delay values for the audio stream. One is any normal audio delay, the other is the audio delay relative to the video. Most demuxing programs ignore any video delay when extracting audio, so if a video delay exists, the delay written to the file name for the extracted audio will be wrong if the intention is to re-mux it with an encoded version of the video. gMKVExtract writes the same delay value, but at least the GUI can be used to check if there's a delay relative to the video that should be used instead.

Edit:: After modifying the AVS scripts of each pass, the first attempt to encode worked, but after trying again for the second time, the same TDecimate error appeared.

Once you've added the first pass script to the job queue remove it from MeGUI, just to be safe. If you reload it into the GUI or use MeGUI to preview it, or add it to the job queue again for the second pass, MeGUI will do it's usual checking of the script and any existing metrics files will be over-written, so if you're using the same script for both passes be careful not to preview it or add it to the GUI or job queue after the analysis pass has run until you've modified it for the second pass. That's the most likely reason it'll happen.

If you have quite a few files to do, separate first and second pass scripts might be easier. You can add all the first pass scripts to the GUI for analysis, then add the second pass scripts for encoding and let MeGUI run the lot. To have unique file names for the metrics and timecodes files, put the attached function in the Avisynth auto-loading plugins folder, if you have Avisynth installed, or load it manually at the top of each script, then create first and second pass scripts that look like this:

# 1st pass
TFM(Output=FTFM())
TDecimate(Mode=4, Hybrid=2, Output=FTDec())

# 2nd pass
TFM(Input=FTFM())
TDecimate(Mode=5, Hybrid=2, Input=FTDec(), TFMIn=FTFM(), mkvOut=FTC())

The metric files created are given names based on the script name.
If you don't want to change the x264 command line to modify the timecodes file name for every job you add to the queue, you can use the same name each time instead of mkvOut=FTC(), as any existing timecodes file will be over-written by each second pass script when it's opened, or when MeGUI starts the encoding job.

PS. As the the function below uses the script name to create the metrics files, it won't work with AvsPmod or MeGUI's script creator until the script exists. In other-words, when you create a new script from scratch in a GUI, you'll probably need to save it to your hard drive before the function can work.

[naoto89]

Either is fine. I just assumed you were using VirtualDub because you linked to the quide.

I do it in meGUI because it generates the tfm.txt file with info while in VirtualDub the tfm.txt file is empty. However, meGUI generates the stats.txt file with little info while VirtualDub generates more info on the first pass but I imagine both have basically the same thing. Elsewhere I read that adding batch=true for TDecimate mode 5 on the second pass, it ignores certain important data from stats.txt like CRC32 and when VirtualDub generates the empty tfm.txt file. I did the encoding with batch=true but I only get a 23.976 CFR video which wouldn't be bad if it wasn't for what I said earlier about the DGindex info.

If the A/V sync is out by the same amount from start to finish, it might be because the original video stream has a delay applied. When you re-encode, any video delay is usually lost. You can check that with gMKVExtractGUI. When you open an MKV it'll display any video delay, along with two delay values for the audio stream. One is any normal audio delay, the other is the audio delay relative to the video. Most demuxing programs ignore any video delay when extracting audio, so if a video delay exists, the delay written to the file name for the extracted audio will be wrong if the intention is to re-mux it with an encoded version of the video. gMKVExtract writes the same delay value, but at least the GUI can be used to check if there's a delay relative to the video that should be used instead.

Not even MediaInfo shows that about the delay but it's just as you mentioned, gMKVExtractGUI shows me that the video stream has those 2 delays.

Once you've added the first pass script to the job queue remove it from MeGUI, just to be safe. If you reload it into the GUI or use MeGUI to preview it, or add it to the job queue again for the second pass, MeGUI will do it's usual checking of the script and any existing metrics files will be over-written, so if you're using the same script for both passes be careful not to preview it or add it to the GUI or job queue after the analysis pass has run until you've modified it for the second pass. That's the most likely reason it'll happen.

The meGUI that I am now using is a current and portable version, no custom changes were made to it either, so it does not queue the Analisys Pass before or after.

If you have quite a few files to do, separate first and second pass scripts might be easier. You can add all the first pass scripts to the GUI for analysis, then add the second pass scripts for encoding and let MeGUI run the lot. To have unique file names for the metrics and timecodes files, put the attached function in the Avisynth auto-loading plugins folder, if you have Avisynth installed, or load it manually at the top of each script, then create first and second pass scripts that look like this:

# 1st pass
TFM(Output=FTFM())
TDecimate(Mode=4, Hybrid=2, Output=FTDec())

# 2nd pass
TFM(Input=FTFM())
TDecimate(Mode=5, Hybrid=2, Input=FTDec(), TFMIn=FTFM(), mkvOut=FTC())

The metric files created are given names based on the script name.
If you don't want to change the x264 command line to modify the timecodes file name for every job you add to the queue, you can use the same name each time instead of mkvOut=FTC(), as any existing timecodes file will be over-written by each second pass script when it's opened, or when MeGUI starts the encoding job.

There are several TS files from the same source that I want to encode, so what you suggested might work for me. So far I have used separate scripts for each pass but I wasn't sure how I was going to deal with the metrics and timecodes files. The command line I made for x264 is more or less like this.

set extAVS=.avs
set extVID=.mkv
set extQPF=.qpf
set "x264=%~dpn1x264_86.exe"
set inp=%~dp1.\AVS
set out=%~dp1.\MKV
set kframe=%~dp1.\QPF
ECHO.
for %%I in ("%inp%\*%extAVS%") do "%x264%" ^
  --crf 16.0 ^
  --preset slow ^
  --tune animation ^
  --ref 4 ^
  --level 4.1 ^
  --qpfile "%kframe%\%%~nI%extQPF%" ^
  -o "%out%\%%~nI%extVID%" ^
  "%%I"

For the mux with mkvmerge, I did this command line.

set extVID=.mkv
set extAUD=.flac
set mkvmerge=%~dpn1mkvmerge.exe
set invid=%~dp1.\MKV
set inaud=%~dp1.\AUDFLAC
set chap=%~dp1.\CHAPTERS
set outvd=%~dp1.\NEW
set tc=%~dp1.\AVS
FOR %%f in ("%invid%\*%extVID%") DO (

	"%mkvmerge%" --output ^"%outvd%\%%~nf%extVID%^" ^
		--language 0:und ^
		--timestamps ^"%tc%\timecodes.txt^" ^
		^"%%f^" ^
		--language 0:und ^
		^"%inaud%\%%~nf%extAUD%^" ^
		--chapters ^"%chap%\%%~nf.txt^" ^
		--track-order 0:0,1:0
)

For timecodes I didn't use variable because at the moment I was only doing tests with a single video.

[hello_hello]

I just gave VirtualDub2 a try and the metrics files created were identical to the ones created by MeGUI's analysis pass.

TIVTC appears not to write the metrics files until the 1st pass has finished, and for it to know it's finished the script can't be open, so after running the 1st pass in VD you'll still have empty metrics files until you close the script. Likewise it appears you have to be careful if you're using VD's batch mode too. If the script is loaded into the GUI when the batch job runs the metrics files will still be empty when it's finished, and when you close the script in the GUI it's as though the analysis pass has started again and immediately aborted so the metrics files are written with very little info, so you have to open the first pass script, run the 1st pass, then close the script. Don't play the 1st pass script in-between the "open script", analysis pass", "close script" steps either or TIVTC will either think it's part of the 1st pass or it's starting from scratch again. Likewise re-loading the 1st pass script into MeGUI will do a similar thing.

I can't think of another reason for ending up with incorrect metrics files. I ran a test 1st pass over 500 frames of video and that's what showed when checking the TFM output file (remembering Avisynth starts counting from zero instead of one). I'm pretty sure the TDecimate output file should always have two more lines than the number of video frames before any decimation, counting the first frame as number one instead of zero.

Near the end of the TFM output file.



The end of the TDecimate output file.



The timecodes file can be very short, or very long, depending on how many individual film and video sections there are.

While I'm thinking about it too, some tools don't like the first line in the timecodes file being this:
# timecode format v1
If there's any error message along the way about it not being a valid timecodes file, change the first line to this:
# timestamp format v1
You might have to change it to keep x264 happy but I haven't done any VFR encoding for a while and my memory is terrible, so I'm not sure.

I've never used the Batch option for TDecimate. Probably more due to habit, but when adding the second pass scripts to a GUI such as MeGUI, to get around the fact that the metrics files don't exist yet (if the 1st pass hasn't been run) I've added them with the 2nd pass TFM/TDecimate lines commented out, then uncommented them after adding the scripts to the job queue. Using the Batch option would probably be simpler.

It's quite possible that after the analysis TFM/TDecimate will decide the source is 100% film or 100% video and the timecodes file will give you a constant frame rate output anyway. My test video was 100% 29.97fps progressive video because I used a script I had handy, and that's what the timecodes file shows. The total number of frames remained the same at 29.97fps.

# timecode format v1
Assume 29.970030
# TDecimate v1.0.8 by tritical
# Mode 5 - Auto-generated mkv timecodes file
# vfr stats:  00.00% film  100.00% video
# vfr stats:  0 - film  500 - video  500 - total
# vfr stats:  longest vid section - 0 frames
# vfr stats:  # of detected vid sections - 0

Edit: On the subject of MKV delays.... the audio delay displayed by MediaInfo is always the audio delay relative to the video. As MKV streams can't have a negative delay, if MediaInfo reports a negative delay for an audio stream it's a sure sign there's a positive delay for the video. For example, a 20ms video delay with a 12ms audio delay would be reported by MediaInfo as an audio delay of -8ms.

[naoto89]

I was doing more tests but there are no good results, I don't know more things I'm failing. Can I share here the TS from an external site (GD) for them to check?

[jagabo]

You're working with a transport stream? That's often a problem. Try remuxing to a program stream and working with that.

[naoto89]

You're working with a transport stream? That's often a problem. Try remuxing to a program stream and working with that.

In previous message I mentioned that I had copied the audio and video streams to MKV just to verify that it runs at the same speed as the TS. What am I supposed to do with the MKV? Index it with DGIndexNV or load it directly with LSMASH like meGUI? What plugin is right for this?

[jagabo]

DGIndex/Mpeg2Source doesn't work well with MKV or TS streams. You should remux to MPG of VOB. You can use ffmpeg to do this with a command like:

ffmpeg -i input.ts -vcodec copy -acodec copy output.vob"

Then index the VOB file With DGIndex (honor pulldown flags) and use Mpeg2Source("output.d2v").

[naoto89]

DGIndex/Mpeg2Source doesn't work well with MKV or TS streams.

I never said that I used DGIndex to index MKV. If I want to index an MKV with an MPEG2 video stream I would use DGIndexNV because I know it has that support. I said that I was using DGIndex and Mpeg2Source to process a TS because I have done it before with other TS files and I didn't have that framerate problem that I have now.

You should remux to MPG of VOB. You can use ffmpeg to do this with a command like:

ffmpeg -i input.ts -vcodec copy -acodec copy output.vob"

Then index the VOB file With DGIndex (honor pulldown flags) and use Mpeg2Source("output.d2v").

If I can directly open the TS with DGIndex and Mpeg2Source, I don't understand why I should copy the MPEG video and audio streams to another container supported by DGIndex and Mpeg2Source, it's a TV capture, not a DVD.

[hello_hello]

I was doing more tests but there are no good results, I don't know more things I'm failing. Can I share here the TS from an external site (GD) for them to check?

You can split off a section with TSMuxer to use as a sample and attach it to a post or upload it somewhere and post a link.

I can't say I've had any problems opening TS files with DGIndex myself. Now and then I have to convert mpeg2 video in an MKV (a DVD ripped with MakeMKV for example) and I usually remux something like that as a TS file first so I can open it with DGIndex.

[naoto89]

Well, I leave a fragment (6 minutes, 88 MB) from TS file but if it's not enough I can also put the complete file.

[hello_hello]

The sample is pretty much entirely film. The only place I could see where it could be considered video is near the end where there's a fade to black (there may be more though small sections though as I didn't inspect it thoroughly). I assume the fade was overlayed at 29.97fps. You could probably get away with decimating it all to 23.976fps though.

It'd probably pay to enable TDecimate's denoising so it's making decisions without taking the noise into account. It doesn't denoise the output video, just a copy of the clip it uses for creating the metrics file etc. Without denoising, the result according to the timecodes file was:

# vfr stats: 75.12% film 24.88% video

with TDecimate's denoising enabled it was

# vfr stats: 98.50% film 01.50% video

Apart from the fade to black, the rest of what it's seeing as video is probably caused by picture wobble or artifacts appearing for a single frame.

The audio sync isn't right for the sample you uploaded, and it seems to get worse after muxing. I just opened the encoded MKV and fiddled with the audio delay until I thought it was about right, then muxed the MKV again while applying the delay. I used a -1000ms delay for the audio for both samples. If you don't want to lose a second from the beginning of the audio, you should be able to use a positive 1000ms delay for the video instead.

jagabo might have some ideas for cleaning up the video if he returns. He's good at that sort of thing. I just added some standard stuff to the script. They're probably ideas borrowed from jagabo anyway.

The script is below. The CFR version was the same aside from only using the TFM/TDecimate defaults with it's denoising enabled.
I used Stab to fix some of the picture wobble, but unfortunately the wobble was there before the ETC logo was added, so the end result is the logo seems to wobble a little instead.

DGDecode_mpeg2source("D:\0001.[0].d2v")

# TFM(Output=FTFM())
# TDecimate(Mode=4, Hybrid=2, Output=FTDec(), Denoise=true)

# TFM(Input=FTFM())
# TDecimate(Mode=5, Hybrid=2, Input=FTDec(), TFMIn=FTFM(), mkvOut=FTC(), Denoise=true)

# For CFR
TFM()
TDecimate(Denoise=true)

Stab()
CropResize(624,468, 2,2,-2,-2, InDAR=4.0/3.0, Resizer="Resize8")
MAA()
TemporalDegrain()
FastLineDarkenMod4()
MergeChroma(aWarpSharp(Depth=30))
GradFun3()
Prefetch(12)

Two sample encodes, including the timecodes and metrics files for the VFR version.

[naoto89]

The correct thing would be that the processed video is FILM to have it at 23,976 fps but due to the delay and that in some frames the frame rate drops to 17 fps (according to timecodes), sync is lost no matter what. Even so, I imagined that a fragment was not going to be enough so I uploaded the entire TS. Something I may not have mentioned before is that most of the TS like this that I try to encode, these don't have commercials, these were cut with a program like Smart Cutter and shared like this by the person who captured them. I have other TS captured from the same TV channel but shared by another person, these were also edited, although to a lesser extent, so some do keep the commercials.

[hello_hello]

The 18fps sections are usually only three frames in duration and generally infrequent, but they'd maintain audio sync. If you do the math:

1000 / 18000 * 1001 * 3 = 166.833333333 ms
1000 / 24000 * 1001 * 4 = 166.833333333 ms
1000 / 30000 * 1001 * 5 = 166.833333333 ms

Three frames at 18fps has the same total duration as four frames at 24fps and five at 30fps, so it'd just mean now and then there must be enough duplicates in a group of five for TDecimate to drop two of them. Because TIVTC always makes it's decisions on each group of five frames from beginning to end (at least in this mode), it means the A/V sync is effectively reset every 166.83 ms, or just under six times per second.

I'm downloading your full sample but I won't get a chance to play with it for a little while as the real world needs me to spend some time in it.

[hello_hello]

Deleted double post.

[hello_hello]

Well it'll take someone smarter than me to work out what's happening. There needs to be an audio delay applied as before, but the correct delay needs to be determined manually. I don't know why, but when comparing the quick and dirty encodes I've attached with the original TS file (using a media player and adjusting the audio sync), it required an audio delay of -320ms for the A/V sync to be the same, give or take a few milliseconds.

I muxed the audio with the encoded video by adding the original TS file to MKVToolNix, adding the new video, deselecting the old video, and then applying a -320ms delay to the audio. The A/V sync should be the same for the CFR and VFR versions, although TIVTC says it's 99.01% film, so maybe it's not worth bothering with VFR encoding.

The metric files are included with the samples.

Edit: I've deleted the samples because I labelled them wrong. I'll upload them again in my next post.

[hello_hello]

Full samples, take two.

[naoto89]

It's strange how when copying the A/V streams from TS to MKV everything still works correctly but when these are encoded everything gets out of sync and you even have to add more delay.

Something that is not seen in mkvmerge but in ffmpeg is that when copying the audio/video streams this appears in color.

[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735402326, dts=1735402382, size=256814e+03x
[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735408342, dts=1735408386, size=2808
[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735420374, dts=1735420394, size=1512
[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735426390, dts=1735426398, size=1808
[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735429334, dts=1735429400, size=2376
[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735435350, dts=1735435404, size=2840
[mpegts @ 080aa680] Invalid timestamps stream=0, pts=1735444310, dts=1735444410, size=2312

That must be indicative of something.

[jagabo]

I'm not seeing any significan't A/V sync problems after encoding. Simple TFM(), TDecimate(), BilinearResize(), quick dirty x264 encoding, mux with original audio with -300 ms audio advance. The downscale was just to make a smaller file for upload here.

[hello_hello]

Edit: jagabo, which source filter did you use to decode? Did you apply any audio delay when muxing yourself?

I don't fully understand what ffmpeg is showing there, but the issue seems to be gaps in the video stream. I hadn't used eac3to to extract the video until now, but when I did the log file had a warning about it. The big gap is right near the beginning. eac3to also created a file with a .gaps extension and the log file suggested running eac3to a second time. When I did it updated the log file to say it'd used the gaps file to create a new MKV from the extracted video, so I assume what it does is use the gaps file to put the gaps back when muxing, as it can't just add empty frames.

TS, 1 video track, 1 audio track, 0:21:43, 60i /1.001
1: MPEG2, 480x480 60i /1.001 (4:3)
2: MP2, 2.0 channels, 128kbps, 48kHz, -23ms
[v01] Video gap description file detected, will be used for muxing...
[v01] Extracting video track number 1...
[v01] Muxing video to Matroska...
[a02] Extracting audio track number 2...
[a02] Applying MPx delay...
[a02] A remaining delay of +1ms could not be fixed.
[a02] Creating file "D:\0001_T2_Audio - English.mp2"...
[a02] Audio has a gap of 23ms at playtime 0:14:12.  <WARNING>
[a02] Starting 2nd pass...
[a02] Realizing MPx gaps...

DGIndex and FFMS2 (I haven't tried Lsmash yet) appear to ignore the gaps in the video stream, causing the A/V de-sync.
By default FFMS2 and Lsmash operate in VFR mode and assume you'll use the original timecodes to create a VFR video after encoding (if the original isn't CFR). For DVD video they ignore the repeat flags by default, so to get them to work like DGIndex does by default you have to tell them not to. For FFMS2 you'd use rffmode=1. For lsmash the option is repeat=true. The trouble is, enabling the repeat flags doesn't cause FFMS2 to account for the gaps, and I suspect lsmash will be the same.

Both ffms2 and lsmash can convert the frame rate by adding or dropping frames, so they can convert VFR to CFR. The arguments are the same for both, but for FFMS2 you'd need to do this:

FFVideoSource("D:\0001.ts", cachefile="D:\0001.ffindex", FPSNum=30000, FPSDen=1001)

If you open the ts file and step through the frames, just before the one second mark you'll see the beginning of a bunch of repeated frames in a row, where ffms2 is filling in the gap.

rffmode mode and frame rate conversion can't be used together, but it's mostly hard telecine anyway, so you should be able to use the frame rate conversion method while decoding with ffms2 followed by TIVTC doing it's usual thing for a constant 23.976fps. Now I know what the problem is, I can see the video pause for a split second near the beginning when playing the TS file.

I haven't had a chance to encode it that way myself yet, but it should work. You'll probably have to extract the audio with eac3to to fix any gaps it has, or add the whole TS file to MKToolNix along with the encoded video to retain any audio gaps that way.

So it appears that's why remuxing the TS file produces an MKV with the A/V sync still intact, because MKVToolnix retains the audio and video gaps, but the video gaps weren't being accounted for when re-encoding. For the record, here's a section of the timecodes file I extracted from the remuxed TS file (as MKV). You can see where a gap is at line 28. It looks like jagabo was right about not using DGIndex with TS files, or at least with problem ones.

[jagabo]

The sample I uploaded was with LWlibavVideoSource("0001.ts"). I've done the same with DgIndex+Mpeg2Source and got essentially the same result.

Oops, I made a mistake in my earlier post: I muxed the new video with the original MP2 audio with a -300 ms audio delay, ie a 300 ms advance. The audio was demuxed from the ts file as an MP2 file when I used DGIndex.

<edit>
I played the original TS file and my encoded video side by side with two instances of VLC. I synced them up (video and audio) near the beginning. After 20 minutes my encoded video was about 1/10 second (~2 frames) ahead of the original. The audio of the two videos was still in sync.
</edit>

[naoto89]

The sample I uploaded was with LWlibavVideoSource("0001.ts").

If your sample was processed with LWLibavVideoSource, I don't understand how you got 23.976 fps. When I use LWLibavVideoSource I get 28.759 (TFM) and 23.007 fps (TDecimate).

After everything hello_hello said about eac3to in previous posts, I did a demux test and a container change test. As eac3to uses Haali for MKV remux and so it couldn't copy gap data like mkvmerge does, this appeared to me.

[v01] Extracting video track number 1...
[a02] Extracting audio track number 2...
[v01] Muxing video to Matroska...
[a02] Applying MPx delay...
[a02] A remaining delay of +1ms could not be fixed.
[v01] Video has a gap of 70 frames at playtime 0:00:01.  <WARNING>
[v01] Video has a gap of 8 frames at playtime 0:14:11.  <WARNING>
[v01] Video has a gap of 21 frames at playtime 0:14:11.  <WARNING>
[v01] Video has a gap of 27 frames at playtime 0:14:12.  <WARNING>
[v01] Video has a gap of 8 frames at playtime 0:21:32.  <WARNING>
[v01] Video has a gap of 7 frames at playtime 0:21:32.  <WARNING>
[v01] The MKV file was created without making use of the gap/overlap information.  <WARNING>
[v01] Please check whether audio is in sync. If it is in sync everything is fine.  <WARNING>
[v01] Otherwise ask eac3to to repeat the muxing. It will then automatically make  <WARNING>
[v01] use of the detailed gap/overlap information.  <WARNING>
[a02] Audio has a gap of 23ms at playtime 0:14:12.  <WARNING>

That it detects gaps at 0:00:01, 0:14:11, 0:14:12, 0:21:32 is because of the cuts applied to TS file after capture and before sharing it. In those parts there should be commercials, but as I had already suspected, those parts were cut by a software like Smart Cutter. I don't doubt that those cuts had their counterproductive effects and that's why I have problems with encoding but there should be some effective solution.

[poisondeathray]

If your sample was processed with LWLibavVideoSource, I don't understand how you got 23.976 fps. When I use LWLibavVideoSource I get 28.759 (TFM) and 23.007 fps (TDecimate).

Your LSmash version might be old/buggy ?

https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

[jagabo]

Try:

LWlibavVideoSource("0001.mkv", fpsnum=30000, fpsden=1001)

[hello_hello]

If your sample was processed with LWLibavVideoSource, I don't understand how you got 23.976 fps. When I use LWLibavVideoSource I get 28.759 (TFM) and 23.007 fps (TDecimate).

It's because by default LWLibavVideoSource decodes in VFR mode, but outputs the average frame rate for Avisynth. Each gap decreases the average frame rate a little. If the video was progressive you could extract the timecodes and use them for a VFR encode (retaining the gaps when encoding) but because you need to apply IVTC the number of frames will change so the original timecodes won't work.
As jagabo said, use FPSNum and FPSDen to force decoding at a constant frame rate of 29.970fps and LWLibavVideoSource will fill in the gaps by repeating frames. Unless you remux the TS file as an MKV while retaining the gaps (remuxing with MKVToolNix should be fine) you'll have to use the TS file as the source.