Is there an easy way to find broken cadence in 2:3 pulldown of TS files?

[Ronstang]

I unknowingly created a problem I don't know how to fix.

I didn't find out about 2:3 pulldown and using IVTC to correct it in my recorded TV until recently. I have totally changed the way I handle my recorded content and it's encoding and it is fantastic now, and I have corrected all my problems....except this one that I created.

I have a bunch of movies I recorded from the MGM channel and edited out the the lead in and lead out fluff....but MGM also inserts a long commercial break somewhere in the movie. I edited those out too but at that time I had no idea about 2:3 pulldown and the importance of maintaining the cadence. I now have to re-encode these movies PROPERLY but I don't know what to do about the possible broken cadence in the middle of the movie due to my edit. I assume it will screw up everything and result in unusable encoded files.

I need to fix all these movies as most are extremely hard to find and since I no longer have the MGM channel since Suddenlink sucks I cannot simply re-record them when they come back on again.

Luckily these movies only have one edit point that I have to find but I don't know an easy way to do that. Scanning through an entire movie one frame at a time looking for it is not practical. Is there an easy way using some software to find my edit points or at least the broken cadence?

If I can find the edit points I can break the movie into two parts at that point and encode them separately and then simply put them back together and they should be fine at that point or if that doesn't work I could always edit out a frame or two and deal with the skip in the movie....that would be better than not having them.

Thanks

[jagabo]

Why do you need to reencode? TVs that have 3:2 pulldown removal will adapt to changes in cadence. Other TVs will just deinterlace on-the-fly.

[Ronstang]

These movies are edited TS files....so they are large, about 6 times larger than my compressed h.265 versions of them and they are not encoded at the moment....just raw recordings that have the fluff edited out. I don't want to leave them interlaced as I have them on my emby server and if someone is watching one of them it will need to transcode....if two are watching one of these types of movies then two need to transcode. I don't have a real powerful server yet so I am using this machine to encode two movies at a time so even one transcode brings my machine to it's knees.

If I leave them as TS files I can get about 400+ on a 4TB hard drive but encoded with my current h.265 settings I can get almost 2300+ on a single 4 TB hard drive with no noticeable loss in quality....so I really don't want to leave them uncompressed if I don't have to.

[poisondeathray]

You can use IVTC with comb deinterlacing fallback ie. "adaptive IVTC"

[jagabo]

So you're basically looking to compress the video.

A basic ffmpeg command line (you may need to change tff to bff):

ffmpeg -i input.ts -vf fieldmatch=order=tff:mode=pc,decimate=cycle=5 c:v libx264 -preset slow -crf 18 c:a copy output.mp4

Basic AviSynth script:

LWlibavVideoSource("input.ts")
AssumeTFF() # or bff
TFM()
TDecimate()

Encode with your choice of encoder.

[Ronstang]

Thanks for always being here for everyone. If it wasn't for you two guys I would never have known that my content was telecined in the first place so I had a feeling one or the both of you would have some suggestions. You both are a great asset to this community.

poisondeathray, the following is the command string I use in FFmpeg Batch AV Converter

-map 0 -c:v hevc_nvenc -vf "fieldmatch,bwdif=mode=0:deint=1,decimate,format=yuv420p10le" -c:a copy -c:s copy -preset hq -rc constqp -rc-lookahead 60 -spatial_aq 1 -temporal_aq 1 -nonref_p 1 -cq 26

If you remember we were having problems getting ffmpeg to properly IVTC and that is the filter you suggested I use, and it has been working fantastic on everything but when I ran a series of movies I recorded from MGMHD and had saved all of them failed in FFmpeg Batch. I assumed it was because of the IVTC but I could be incorrect. Is this still the filter string you suggest to do IVTC and 10bit color?

jagabo, I'm not sure what tff or bff are but I could try that string...will it work with HEVC? I don't know how to use AviSynth either so if that is the only solution it would be something I have to learn.

Thank you

[jagabo]

tff and bff specify the field order: top field first, vs bottom field first. Interlaced frames consists of two half pictures, intended to be seen separately and sequentially. tff and bff specify which of the two pictures is supposed to be seen first.

The simple ffmpeg command line I gave should work with any codec.

[poisondeathray]

poisondeathray, the following is the command string I use in FFmpeg Batch AV Converter

-map 0 -c:v hevc_nvenc -vf "fieldmatch,bwdif=mode=0:deint=1,decimate,format=yuv420p10le" -c:a copy -c:s copy -preset hq -rc constqp -rc-lookahead 60 -spatial_aq 1 -temporal_aq 1 -nonref_p 1 -cq 26

If you remember we were having problems getting ffmpeg to properly IVTC and that is the filter you suggested I use, and it has been working fantastic on everything but when I ran a series of movies I recorded from MGMHD and had saved all of them failed in FFmpeg Batch. I assumed it was because of the IVTC but I could be incorrect. Is this still the filter string you suggest to do IVTC and 10bit color?

Look for a log file - What is the error message ? It should provide clues . Or run 1 and look at the error message

[Ronstang]

I ran a whole bunch and over 75% failed. I looked at the log files and it was huge and hard navigate so I ran the two files that failed the quickest and I have the log file for those and it simply says it failed, no reason why. I have uploaded it but it tells nothing. Is there another log file I need to look for somewhere. Last time I had a failure it actually told my why but not this time.

[poisondeathray]

That log does not tell you anything useful, and it's missing information that is supposed to be there. A full log should have ffmpeg build, commmand line, and console error messages

If 1/4 work, 3/4 don't work, something is wrong with either your files , or that specific ffmpeg build, or some other issue like hardware maybe overheating etc.... I know, not helpful.... You have to go through the process of elimination

If some syntax was wrong with the command line, then they should all fail (not producing some % that complete or work)

I would choose one that "failed" and run the command line individually. You can add -report for more info

[jagabo]

I don't know much about bwdiff -- does it handle video field blending well? As is common in a PAL/NTSC conversions.

The OP should upload a sample of his "bad" source for inspection.

[poisondeathray]

I don't know much about bwdiff -- does it handle video field blending well? As is common in a PAL/NTSC conversions.

bwdif is a deinterlacer like yadif - so not well for blended material (as expected). But bwdif is faster and the quality a bit higher on most sources than yadif. A notable feature of bwdif is it does a good job of weaving static content (not relevant for what he's doing)

The way bwdif being used in that commandline is for marked combed scene changes only from fieldmatch, since combmatch is set to default, instead of "full", and deint=1 . I wouldn't expect bwdif applied on combed scene changes to be a cause for incomplete encoding, but he can try yadif instead

[jagabo]

I don't know much about bwdiff -- does it handle video field blending well? As is common in a PAL/NTSC conversions.

bwdif is a deinterlacer like yadif - so not well for blended material (as expected).

I wouldn't expect it to remove the blending. I was just wondering if it would "give up" under some circumstances and leave the frames interlaced. That sounds like a "no."

[Ronstang]

I will try to run ffmpeg from the commandline when the current batch of files is done in about 24 hours. Strange thing is running content recorded from other stations do have some failures but only a few percent...but the one thing all these files have in common is they were recorded a long time ago and have been sitting on hard drives in my cabinet for a very long time. Some of these files are a year and a half old. Is there a possibility there are transmission errors in the stream that are interrupting the encoding. I used to have problems with blips here and there, especially on MGM until I reconfigured my cables and changed the splitters. All the files play fine as TS files off of the HD. If I run a batch of content recorded this week from all the same channels I do not get one singe failure even on a batch of 130+ files being queued at the same time....all complete fine.

I have two files that fail rather quickly, like a few minutes into encoding. I need to test them with some other settings like just running them through using HEVC default settings and see if they fail and then onto a different build of ffmpeg.

I will note that I tried to upgrade to the latest 5.0 build of ffmpeg last week and every single file would fail but I think the failure was immediate. I do not remember as I switched back to 4.4.1 after a short time.

[Ronstang]

I have been monitoring my hardware temps and such and nothing is out of the ordinary and I have only had one failure in the current batch. Once again files recorded about a year and a half ago. I have quality coolers on everything and they run under temp normals. I worked hard to get them there as I run this machine 24/7 encoding and did experience problems a year and a half ago so I had to upgrade.

[Ronstang]

There was an option to turn on more verbose logs in FFmpeg Batch so I did and I reran the two files that fail the quickest and I have attached the logs....even though I cannot find anything in there that tells me anything.

Interesting note. I am currently running the file that failed in the first 30 seconds again, with HEVC default settings and BWDIF and the file is encoding fine, already at 20% whereas before it would not even make it to 1%. I assume this means there is something in my parameters that is causing these files to fail. Now the question becomes which parameter is the problem.

[Ronstang]

I kept running the same file over and over, each time taking one more parameter out of my ffmpeg settings. I took out the IVTC it still failed, took out 10 bit color still failed....I then replaced IVTC and 10 bit with the BWDIF only and started taking other parameters out one by one and each time it failed. Until I took out -rc-lookahead 60.

It is my understanding that this is one of the settings that improve quality so I really don't want to take it out but I would like to understand what is happening and maybe find a way to fix this.

Any ideas?

[poisondeathray]

Not sure, it might be changes to ffmpeg hevc_nvenc ; sometimes new commits that break things. You can try NVEncC to help rule it out . It's less likely an explanation because you said some actually completed

It might be that your prior edits cause an issue, maybe timestamps have errors. A longer lookahead might cause a problem because frames get mixed up with non monotonic timestamps. If you index the video it would probably help in that case (e.g. avisynth dgindexnv)

Or you can try "fixing" the file if there was a problem , e.g. videoredo's quickstream fix , or maybe remuxing it would help (just rewrap it in mkvmerge)

[Ronstang]

If I set -rc-lookahead to 0 the file that fails before 1% encodes fine. I don't know that that means but I doubt it is the edit points as some of the files that fail have no edit points. They simply have the fluff trimmed off the beginning and end and so they are only movie.

It could be timestamps as you say. I have had trouble with those a lot when I edit. Sometimes Avidemux will not save the edit because of the time stamps and I have to demux and remux to fix them. I have found that ProjectX is the only real solution even though it is old an no longer updated as it truly fixes the issues upon demuxing....but it also strips out the CCs so I have to pull them out first. Then using TSmuxer the files edit fine.

Even though these edited and saved in Avidemux I will try and demux and remux. If you remember we had this same issue back in October where the one file I picked to test IVTC kept encoding but failed upon playback and I finally figured out it was a timestamp issue and used the procedure above to get it working.

If none of that works I will test these files with my current settings using an older build of ffmpeg. I tried the newest version of ffmpeg, build 5, the other day and all of my files failed. This ia a head scratcher.

[Ronstang]

The demux remux route seems to have solved the problem. I will go back and do this to all the files that have failed and see if it works on all of them. It is going to be tedious because I have thousands of files to encode so I will just have to let them fail, sort the failed files and then only demux and remux them. Not perfect because not all the files fail right away, some almost complete before the fail so I will waste a lot of encoding time but that looks like the only solution.

I am happy with the settings you guys helped me solve my issues. I really appreciate your help. Unfortunately using IVTC and 10bit color has to be done in software so my encode times are about 2X running two files....so basically real time....so this will take some time to sort out but it is worth it because the resulting files are smaller than I have ever had yet higher quality too.