DePALifying TV Episodes

[koberulz]

25.0/23.976024 gets me the same result, 25000/23976 works.

[jagabo]

Odd. The docs say floating point is supported.

[koberulz]

So that works for 25p material, but what about 25i? I have a series here that seems to be sped up looking at IMDB episode runtimes, but the footage is definitely interlaced.

[jagabo]

It may be phase mis-aligned. Try just TFM(pp=0) and see what happens.

[koberulz]

I don't know what either of those things mean.

[koberulz]

To phrase it in a less-rushed way, now that I've got some time:

I'm not sure what "phase mis-aligned" means. TFM seems to be some sort of IVTC thing or something, but it's difficult to understand what you're getting at without understanding the mis-alignment thing, nor do I know what the full script would look like. Just Source, TFM, done? Or would I also need to adjust the frame rate and audio as per the previous script?

I'm not sure what "see what happens" involves either, as I have no NTSC source to compare with. Thankfully season three of Kyle XY never got a PAL release so I had to import, and was able to use the closing credits music to figure out which was correct for part one of season two, which IMDB had listed at the same runtimes as my PAL DVD - but also 2-3 minutes shorter than every other episode of the series. Turns out IMDB is wrong and it did need to be slowed down.

The 25i DVD in question is for The West Wing. I have the whole thing on PAL DVDs, so nothing to compare to even if I did run TFM on it blindly.

I've also discovered that White Collar's Australian DVD releases were in NTSC anyway, which is excellent. Why wasn't/isn't this more common? I know American TVs often can't play PAL content but the reverse isn't true. All of WWE's Australian DVD releases are NTSC.

[jagabo]

I'm not sure what "phase mis-aligned" means.

All analog PAL broadcast (and analog tape recordings) was interlaced. Say a 24 fps film is sped up to 25p:

1  2  3  4  5... (film frames at 25 frames per second)

and broadcast (or recorded on analog video tape) as top-field-first interlaced analog video:

1T  1B  2T  2B  3T  3B  4T  4B  5T  5B...

The broadcast is a series of fields, not frames, at 50 fields per second. When digitized to 25i pairs of successive fields are woven into frames. The process can be captured starting with a top field then weaving in the next bottom field to complete a frame (a top-field-first digital capture) or it can be captured starting with a bottom field and weaving the next top field to complete a frame (a bottom-field-first capture).

Suppose that is digitized as top-field-first video; successive pairs of fields are woven together into frames, starting with a top field:

1T+1B  2T+2B  3T+3B  4T+4B  5T+5B...

Each 25i digital frame contains two fields from the same film frame -- the frames are essentially progressive even though the broadcast signal was interlaced.

But what if that signal was captured as bottom field first frames?

(1T discarded) 1B+2T  2B+3T  3B+4T  4B+5T  5B+6T...

Every 25i digital frame now contains fields from two different film frames. Any time there is motion the frames appear to be interlaced. That is, it looks like interlaced digital video. This is out-of-phase digital 25i PAL video.

TFM seems to be some sort of IVTC thing or something

Yes, TFM() is part of the NTSC inverse telecine process. Those out-of-phase interlaced video frames can still be restored to the original progressive film frames by recombining the correct fields. First split the frames back into fields:

1B  2T  2B  3T  3B  4T  4B  5T  5B  6T...

Then search for matching fields and combine them:

(1B discarded)  2T+2B  3T+3B  4T+4B  5T+5B  6T+6B...

You have now restored the progressive film frames. That is what TFM() does.

TFM() is part of the TIVTC package -- designed to restore 24p progressive film frames from 30i NTSC telecined video. With NTSC video TFM() is usually followed TDecimate() to remove duplicate frames, but that step is not necessary here. The TFM() part works with this type of out-of-phase PAL video. If your video is truly interlaced video (for example: live sports, live news, daytime soaps) where every field is from a unique point in time, TFM() will resort to deinterlacing since it can't match fields. Setting the pp parameter to 0 prevents that deinterlacing -- interlaced frames will pass through without being deinterlaced. It's a surefire way to determine if you have an out-of-phase PAL capture.

If your video looks progressive after TFM(pp=0) (except for maybe at scene changes) you have out-of-phase PAL video. If it still looks interlaced you have something else.

[koberulz]

The combing seems to disappear when using TFM(pp=0). So do I just insert that as the first step of the previous script, or does it need different handling?

[jagabo]

The combing seems to disappear when using TFM(pp=0). So do I just insert that as the first step of the previous script?

Yes. But check your video -- you may find that there are still some interlaced frames at scene changes. These are caused by orphaned fields (fields with no matching field to complete the frame), typically because the video was cut/pasted after being digitized. If that happens you should leave out the pp=0 so that TFM() deinterlaces those frames. You can also supply your own deinterlaced frames for those residual interlaced frames using the clip2 parameter. For example: TFM(clip2=QTGMC(FPSDivisor=2)). QTGMC is a better deinterlacer than TFM's internal deinterlacer so that will deliver better results.

[koberulz]

I've started another 25p series in the meantime, so I'll do that when I get back to it.

In the meantime though, ffmpeg always shows the frame number, the number of frames per second it's working at, a q value, file output size, the timecode of the current frame, and the bitrate.

What's the q value? When I was using qp encoding it would vary, generally within two of the qp number in either direction. With crf it's holding steady at 23.

[koberulz]

And what about subtitles? mkvmerge should be able to handle them, right? I'm just not sure how to go about it; I don't really understand enough to make sense of the documentation.

[jagabo]

I don't know what the q value on that status line is when using CRF encoding. You would think it's the current average quantizer or the crf value you specified. But it's neither of those.

ffmpeg and mkvmerge can certainly mux subtitles along with the audio and video. That's about all I've ever done with them.

[koberulz]

Right, but they'd need to be time-converted same as chapters, yeah? I'm fairly sure the mkvmerge documentation indicates it can do that, but I'm less sure of what is actually involved in terms of the batch file or what have you.

[jagabo]

I don't think MkvMerge actually changes the timestamps of the subs. But, like it does for audio and video, it just sets a flag in the header telling the player to play the subs faster or slower. Outside of a computer, few players respond to such flags. I would guess ffmpeg can change the timing of the subs but I've never done it or looked into it.

[koberulz]

If it can change the timestamps of the chapters, why couldn't it change the timestamps of subs?

From the documentation I'm looking at it seems like --sync should do it, it has exactly the same syntax and whatnot as chapter-sync and even says you can use it instead if you specify the chapter track. But it doesn't say where to specify the track and I'm not sure how to pull the subs out of the old one, or handle both subs and chapters in one go (or multiple sub tracks, where applicable).

[jagabo]

MkvMergeGUI can show you the MkvMerge command line for what it does. So just use the GUI to get the arguments for the command line utility.

[koberulz]

for %%F in (*.mkv) do (
echo A = LWLibAvVideoSource("%%F"^) > script.avs
echo B = LWLibAvAudioSource("%%F"^) >> script.avs
echo AudioDub(A,B^) >> script.avs
echo Trim(475,0^) >> script.avs
echo AssumeFPS(24000,1001,sync_audio=true^) >> script.avs
echo ResampleAudio(48000^) >> script.avs
ffmpeg\ffmpeg.exe -i script.avs -aspect 4:3 -acodec ac3 -vcodec libx264 -preset slow -crf 18 "Output\%%F"
mkvextract "%%F" chapters "%%F.txt"
rem mkvmerge -o "ChapterVersions\%%F" --chapter-sync 0,25000/23976 --chapters "%%F.txt" "Output\%%F"
mkvmerge -o "ChapterVersions\%%F" --chapter-sync -19000,25000/23976 --chapters "%%F.txt" "Output\%%F"
del "%%F.lwi"
del script.avs
del "%%F.txt"
del "Output\%%F"
)

pause

Have I done the math correctly here? The chapter points seem to be a split-second late in the final output file. I trimmed off a 19-second modern studio logo from an old film it really isn't suited to.

[koberulz]

Also, a question about batch files. I've written a demux batch, like so:

for %%F in (*.mkv) do (
ffmpeg -i "%%F" -map v:0 -c:v copy "%%F".mp4 -map a:0 -c:a copy "%

%F".ac3
)
pause

But I'm getting filename.mkv.ac3 and filename.mkv.mp4 as output. Is there a way to omit the mkv extension?

[jagabo]

for %%F in (*.mkv) do (
echo A = LWLibAvVideoSource("%%F"^) > script.avs
echo B = LWLibAvAudioSource("%%F"^) >> script.avs
echo AudioDub(A,B^) >> script.avs
echo Trim(475,0^) >> script.avs
echo AssumeFPS(24000,1001,sync_audio=true^) >> script.avs
echo ResampleAudio(48000^) >> script.avs
ffmpeg\ffmpeg.exe -i script.avs -aspect 4:3 -acodec ac3 -vcodec libx264 -preset slow -crf 18 "Output\%%F"
mkvextract "%%F" chapters "%%F.txt"
rem mkvmerge -o "ChapterVersions\%%F" --chapter-sync 0,25000/23976 --chapters "%%F.txt" "Output\%%F"
mkvmerge -o "ChapterVersions\%%F" --chapter-sync -19000,25000/23976 --chapters "%%F.txt" "Output\%%F"
del "%%F.lwi"
del script.avs
del "%%F.txt"
del "Output\%%F"
)

pause

Have I done the math correctly here? The chapter points seem to be a split-second late in the final output file. I trimmed off a 19-second modern studio logo from an old film it really isn't suited to.

The equations look right. How are you checking the chapter points? With a media player? The player may be set to "fast seek" -- always seek to the nearest keyframe rather than the exact frame specified. Turning off that feature may fix the problem. Another way to fix is it to force the encoder to create keyframes at the chapter points (-force_key_frames in ffmpeg) but then you need to know the chapter times before encoding. You can check your chapter points by exporting them as an XML file with MkvExtract.

[jagabo]

Also, a question about batch files. I've written a demux batch, like so:

for %%F in (*.mkv) do (
ffmpeg -i "%%F" -map v:0 -c:v copy "%%F".mp4 -map a:0 -c:a copy "%

%F".ac3
)
pause

But I'm getting filename.mkv.ac3 and filename.mkv.mp4 as output. Is there a way to omit the mkv extension?

F is the full drive, path, filename and extension. If you want just the filename in the current folder use n, as in "%~nF.ac3".

By the way the %~ format supports:

d = drive:
p = path \path\to\folder\
n = name (no extension)
x = extension

So "%~dpnx" is the full drive:\path\name.extension. Type "Help Call" or "Help For" at the command line to get a full list of options.

[koberulz]

I checked the chapter points in VLC. The easist one to judge is out by under a second - originally starts on "how much longer's that man going to stay?" and my new version has the chapter start "man going to stay?" So comparing the actual chapter time, as displayed in the toolnix chapter editor, with the time in VLC's progress bar isn't helpful. toolnix claims 1:05:51.845845845, VLC shows 1:05:51 at both the start of the line and the start of the chapter...although that it's eight-tenths into the second makes it more likely than not VLC is correctly jumping to an incorrectly placed chapter, I should think

I initially trimmed at 500 and had the delay at -20000 but it cut off a bit of the start of the film. I think from memory I got different results, although by frames rather than seconds and I'm not sure if it was better or worse. Which seems to point to it being a math issue. I wonder if it's an order of operations thing? ie when is it applying the -19000 relative to the 25000/23976? Maybe I'd get better results with two chapter-syncs, one to lop off the 19 seconds and one to convert the "frame rate". EDIT: Yep, that's the ticket. Although I muddled my way through it and ended up with an MKV that had two "editions" each with completely different chapters - I manually removed the completely wrong one but I'm not sure how it got there or how to make this efficient enough for a batch. But it's a start.

Further question regarding that 25i show that's phase mis-aligned...is there any reason I need to make it progressive instead of just running the same change of frame rate as the progressive shows? I'm not actually editing the frames at all, so unless I'm missing something it shouldn't matter that it's interlaced. If I were resizing from 720x576 to 720x480 it'd make sense to deinterlace, but I'm not bothering with that because I don't need to be DVD-compliant.

Also: multiple audio tracks. I assume the current script simply picks the default and bins the rest. What if I wanted to keep them?

[jagabo]

Turn off "fast seek" in VLC. It's in the Input/Codecs section of the Preferences dialog. Chapter seeking any better?

Regarding whether you should encode your out-of-phase PAL video interlaced or progressive, my preference is to encode progressive whenever possible. Progressive encoding is more efficient and has sharper chroma. But you can leave the video interlaced and encode interlaced if you want.

[koberulz]

The two-step chapter-sync process solved the chapter problem without re-encoding the video at all, so I consider that issue resolved. I have to account for the trim first, then separately account for the PAL-to-NTSC conversion.

What about multiple audio tracks?

[jagabo]

What about multiple audio tracks?

What about them? In ffmpeg you can use the map options to include multiple audio tracks.

https://trac.ffmpeg.org/wiki/Map

[koberulz]

But I'd need to get them through the AviSynth steps first.

[jagabo]

There are tricky ways you can do that. For example, load two stereo audio tracks in AviSynth and combine them into a quad track with MergeChannels(). Then in ffmpeg separate the quad track back into two stereo tracks and encode them as separate audio streams.

[koberulz]

Attempting to use %~nF just gets me an error.

Subtitle-wise, I gave this a shot:

for %%F in (*.mkv) do (
echo A = LWLibAvVideoSource("%%F"^) > script.avs
echo B = LWLibAvAudioSource("%%F"^) >> script.avs
echo AudioDub(A,B^) >> script.avs
echo AssumeFPS(24000,1001,sync_audio=true^) >> script.avs
echo ResampleAudio(48000^) >> script.avs
ffmpeg\ffmpeg.exe -i script.avs -aspect 16:9 -acodec ac3 -vcodec libx264 -preset slow -crf 18 "Output\%%F"
mkvextract "%%F" chapters "chapters.txt" tracks 2:subs.idx
mkvmerge -o "ChapterVersions\%%F" --chapter-sync 0,25000/23976 --chapters "chapters.txt" "Output\%%F" 

subs.idx --sync 2:0,25000/23976
del "%%F.lwi"
del script.avs
rem del "chapters.txt"
rem del "subs.idx"
rem del "Output\%%F"
)

pause

But the subtitle sync hasn't done anything, all the subs have exactly the same timing as before. Not sure if my script is right.

[jagabo]

I think the problem is that you are specifying track 2 of subs.idx to be have its timing changed. But it only has one track, 0.

[koberulz]

Ah, the wonders of ambiguity. I was figuring it would attack track 2 of the merged file. Perhaps not. Basically the difference between:
(sync the chapters) + (chapter file) + (video + audio) + (subs) + (sync track 2)
and
(sync the chapters + chapter file) + (video + audio) + (subs + sync track 2)

EDIT: Changing it to --sync 0:0,25000/23976 doesn't change anything. The timings are still exactly the same as the PAL version.

EDIT 2: Yes, --sync only modifies the subs track, but it has to be called before the subs track. --sync -1:25000/23976 subs.idx works perfectly.

Why doesn't %~nF work?

[jagabo]

Yes, this is what I did when I tested the process:

mkvmerge -o newvid.mkv nosub.mkv --sync 0:0,25000/23976 subs.idx

Why doesn't %~nF work?

Did you put it in quotes?