Problem NTSC & PAL interlaced source MPEG2

[TubeBar]

I've gotten use to dealing w/ most newer formats, not many errors, no interlacing and I do everything in FCP/Compressor and Premier Pro/After Effects. I can do anything in there pretty fast and easy. God forbid you get these old MPEG2 files sourced from old DVDs w/ incorrect authoring? Then they do next to nothing.

I assume this is going to be an ffmpeg fix - still trying to discern the source errors.

I have 2 sources I want to encode ripped from DVD, both have problems and not sure which is the better one.

These are the DVD MPEG2 files, I just losslessly trimmed them out TMPGEnc Smart Ender, so these are the exact files I am dealing with. Just first 2 min of same episode.

1. The NTSC isn't flagged properly, mediainfo will tell you it's 29.97 but Adobe and Apple claims it's 23.976 progressive. I get nice "pull downs" remaining as part of the video when I try to deinterlace to progressive. If I tell it that's it 29.97 Top upper field it SPEEDS up the video. The is encoded wrong, not sure it's worth fixing.

2. The PAL source (by Madman) is better, obviously better Pal colors better over all. This is my choice to encode for my home library. It has some weird issues too. The main one being it is running at 25fps but the time and pitch seem to match the NTSC time. It's not running fast. Did they add a pulldown (never heard of that w/ PAL) or did they add an extra frame blend? You can clearly see in PAL same there are 2 fields binded together just like an NTSC pulldown. Uh what a mess.

3. The third file I attached was what the studio did for YouTube. Besides running the video w/ a BIG 16:9 stretch it's 23.976 and seems to deinterlaces perfectly. It's missing content, is of lower quality h264, not for me. I just used it as a ref source.

What advice does anyone have to fix this? Ideally I want to use the PAL DVD source, encode it to correct speed is what? 24? Or is 23.976? I am stumped w/ the PAL interlaced pulldown looking frames in there.

Again this sounds like a job for ffmpeg.

[jagabo]

1) The NTSC clip is mostly normal 3:2 pulldown (a mix of soft and hard pulldown, so you have to perform the soft pulldown). TFM().TDecimate() takes care of that. ffmpeg has similar fieldmatch and decimate filters. The video also has some chroma blending here and there. You can probably address that with some strong temporal filter of the chroma. Or just ignore it.

2) This is the usual analog NTSC to analog PAL conversion with field blending. QTGMC().SRestore() will get rid of most of the blending and restore the original 23.976 fps film frames.

[ProWo]

Your PAL source, with jagabo's QTGMC().SRestore(), crop and resize.

[TubeBar]

1) The NTSC clip is mostly normal 3:2 pulldown (a mix of soft and hard pulldown, so you have to perform the soft pulldown). TFM().TDecimate() takes care of that. ffmpeg has similar fieldmatch and decimate filters. The video also has some chroma blending here and there. You can probably address that with some strong temporal filter of the chroma. Or just ignore it.

2) This is the usual analog NTSC to analog PAL conversion with field blending. QTGMC().SRestore() will get rid of most of the blending and restore the original 23.976 fps film frames.

Oh so that's what's going on. Thanks for clearing up the confusion, seemed really messy to me.

Your PAL source, with jagabo's QTGMC().SRestore(), crop and resize.

That came out perfect! What syntax did you use for that clip? Sorry I am an ffmpeg dummy. I tried:

ffmpeg -i pal.mpg -vf QTGMC().SRestore() out.mkv

Says filter doesn't exist.

[ProWo]

What syntax did you use for that clip? Sorry I am an ffmpeg dummy. I tried:

ffmpeg -i pal.mpg -vf QTGMC().SRestore() out.mkv

Says filter doesn't exist.

You have to use avisynth.

[TubeBar]

I thought so as I couldn't find anything. Looking for a tut on that, thinking about following this guide: https://macilatthefront.blogspot.com/2021/01/deinterlacing-with-avisynth-and-qtgmc.html

[ProWo]

I thought so as I couldn't find anything. Looking for a tut on that, thinking about following this guide: https://macilatthefront.blogspot.com/2021/01/deinterlacing-with-avisynth-and-qtgmc.html

Monday I'll send you more info about.

[TubeBar]

I thought so as I couldn't find anything. Looking for a tut on that, thinking about following this guide: https://macilatthefront.blogspot.com/2021/01/deinterlacing-with-avisynth-and-qtgmc.html

Monday I'll send you more info about.

OK thanks

[lollo]

@TubeBar:

All for 32-bit version

• download AviSynth_2.60.exe from https://sourceforge.net/projects/avisynth2/ and install it
• create AviSynth_plugin_dir directory
  
• download QTGMC from http://avisynth.nl/index.php/QTGMC saving the file QTGMC.avsi inside AviSynth_plugin_dir
• download Zs RF Shared from http://avisynth.nl/index.php/Zs_RF_Shared saving the file Zs_RF_Shared.avsi inside AviSynth_plugin_dir
• download MaskTools2 from http://avisynth.nl/index.php/MaskTools2 unpacking the file masktools2_v2.2.30.7z in directory masktools2_v2.2.30 inside AviSynth_plugin_dir
• download RgTools from http://avisynth.nl/index.php/RgTools unpacking the file RgTools-v1.2.7z in directory RgTools-v1.2 inside AviSynth_plugin_dir
• download MVTools from https://github.com/pinterf/mvtools unpacking the file mvtools-2.7.45-with-depans20210608.7z in directory mvtools-2.7.45-with-depans20210608 inside AviSynth_plugin_dir
• download Nnedi3 from http://avisynth.nl/index.php/Nnedi3 unpacking the file NNEDI3_v0_9_4_60.7z in directory NNEDI3_v0_9_4_60 inside AviSynth_plugin_dir
• download LoadDLL from http://avisynth.nl/index.php/LoadDLL unpacking the file LoadDLL10.7z in directory LoadDLL10 inside AviSynth_plugin_dir
• download FFTW 3.3.5 from http://www.fftw.org/install/windows.html unpacking the file fftw-3.3.5-dll32.zip in directory fftw-3.3.5-dll32 inside AviSynth_plugin_dir
  
• download Srestore from http://avisynth.nl/index.php/Srestore saving the file https://pastebin.com/raw/JYBjMBvx as Srestore.avsi in directory Srestore inside AviSynth_plugin_dir
• download GRunT from http://avisynth.nl/index.php/GRunT unpacking the file GRunT-v1.02.7z in directory GRunT-v1.02 inside AviSynth_plugin_dir
  
  to open the file
• download FFmpegSource from http://avisynth.nl/index.php/FFmpegSource unpacking the file ffms2-2.40-msvc.7z in directory ffms2-2.40-msvc inside AviSynth_plugin_dir

as alternative I uploaded the full dir for you in https://drive.google.com/file/d/1fMuGnVIUncUWOn028D_UhjFPshf59TfZ/view?usp=drive_link

create this script with a text editor (already present in the zip file):

# plugins directory
plugins_dir="AviSynth_plugin_dir\"

	# QTGMC
Import(plugins_dir + "QTGMC.avsi")
	# Zs_RF_Shared
Import(plugins_dir + "Zs_RF_Shared.avsi")

	# MaskTools2
loadPlugin(plugins_dir + "masktools2_v2.2.30\x86\masktools2.dll")
	# RgTools
loadPlugin(plugins_dir + "RgTools-v1.2\x86\RgTools.dll")
	# MVTools
loadPlugin(plugins_dir + "mvtools-2.7.45-with-depans20210608\x86\mvtools2.dll")
	# Nnedi3
loadPlugin(plugins_dir + "NNEDI3_v0_9_4_62\x86\Release_W7\nnedi3.dll")
	# FFTW
loadPlugin(plugins_dir + "LoadDLL10\LoadDLL\LoadDLL32.dll")
loadDll(plugins_dir + "fftw-3.3.5-dll32\libfftw3f-3.dll")

	# Srestore
Import(plugins_dir + "Srestore\Srestore.avsi")
loadPlugin(plugins_dir + "GRunT-v1.02\x86\grunt.dll")

	# FFmpegSource
loadPlugin(plugins_dir + "ffms2-2.40-msvc\x86\ffms2.dll")

video_org=FFmpegSource2("S1E2 (PAL) 25i.mpg")

video_restored=video_org.AssumeTFF().QTGMC().Srestore()

return(video_restored)

Open the script in VirtualDub to watch the result.

Feed ffmpeg with the avs script to create a h264/aac compressed file in a mp4 container:

ffmpeg.exe -i script.avs -c:v libx264 -crf 17 -preset slow -aspect 4:3 -c:a aac -b:a 128k output.mp4

The plugins will evolve during time, so if needed upgrade to last version of them once available

If you prefere to use the common AviSynth plugins directory instead of loading them one by one (I do that to have full control), they will be automatically loaded (check the AviSynth doc), and the script will reduce to:

FFmpegSource2("S1E2 (PAL) 25i.mpg").AssumeTFF()

QTGMC()

Srestore()

3 lines code for such powerful result, 1 being the opening of the file!

You may want to use AvsPmod to watch the result as alternative to VirtulDub

You may want to upgrate to AviSynth+ and 64-bit filters once you are familiar with the whole.

If you feel more confortable with a GUI, ProWo will help for sure, otherwise enjoy the direct usage of AviSynth and ffmpeg tools!

[poisondeathray]

NTSC looks better to start with overall - but it depends on what your goals are

PAL is a field blended conversion with more texture and detail loss, worse compression artifacts (some frames are severely worse), cropped more (smaller active image area). Part of the process of deblending requires (global) deinterlacing which degrades it even more

PAL version has slightly elevated Y levels, washed out black level - but colors are easy to adjust to whatever, do not base your decision based on that. You can easily make one look like the other, or something else (e.g if you had a reference cel) in terms of levels and color

Both have rainbows, you might consider applying derainbow filter as well

[ProWo]

I thought so as I couldn't find anything. Looking for a tut on that, thinking about following this guide: https://macilatthefront.blogspot.com/2021/01/deinterlacing-with-avisynth-and-qtgmc.html

Monday I'll send you more info about.

OK thanks

Here we go:

Step 1) Install everything according to lollo's excellent post.

Step 2) Start clever FFmpeg GUI, load your S1E2 (PAL) 25i.mpg file, click main, click encode video stream, click crop detect, check the crop with preview and then click OK, done.


[Attachment 72671 - Click to enlarge]

Set all values as in picture 1, click on Avisynth support, click on create script, the script is created, click on edit script, the script is opened in notepad, add QTGMC().SRestore() at the position like in picture 2 and save the script.


[Attachment 72674 - Click to enlarge]

Click Test script to test the script with virtualdub2.
This may take some time because the file is indexed. If it can be played without problems, close the Vitualdub window and the Notepad window and click encode.

When that is done, click multiplex, the newly created video stream (deinterlaced and progressive, 23.976 fps) is shown in green and already selected, select still the original audio stream, choose the container of your choice (mkv, mp4) and click multiplex (see picture 3).


[Attachment 72673 - Click to enlarge]

Done.

[Alwyn]

@Lollo, minor thread drift. Your QTGMC instructions in post #9 don't seem to include this, from the QTGMC wiki (which is included in Andrew's tute):

Dfttest, FFT3DFilter and MVTools2 need the FFTW3 library (windows builds). On a 64-bit Windows OS, extract the 32-bit libfftw3f-3.dll. Make a copy of it and rename it as "FFTW3.dll". Place the files "libfftw3f-3.dll" and "FFTW3.dll" in the SysWow64 folder.

Could you explain?

[lollo]

I find that wiki explanation confusing for the newbies (although correct). So instead, I specified to download the requested filter

download FFTW 3.3.5 from http://www.fftw.org/install/windows.html unpacking the file fftw-3.3.5-dll32.zip in directory fftw-3.3.5-dll32 inside AviSynth_plugin_dir

and manually load it in the script with the help of loadDLL plugin:

	# FFTW
loadPlugin(plugins_dir + "LoadDLL10\LoadDLL\LoadDLL32.dll")
loadDll(plugins_dir + "fftw-3.3.5-dll32\libfftw3f-3.dll")

instead of downloading/copying/manually placing it in the windows system folders, that are not clear to inexperienced people.

[Alwyn]

Thanks.

[Alwyn]

@Lollo, while the thread is quiet, if you're just loading a DLL from your plugins folder, why can't we just drop libfftw3f-3.dll into the normal AVISynth plugins folder? I don't understand why you're just calling a DLL from a folder that you've created but normally we have to copy "libfftw3f-3.dll" into Syswow64 and do a copy>rename to "FFTW3.dll" as well.

What's the difference between "loadPlugin" and "loadDll"?

[davexnet]

libfftw3f-3.dll is a Windows DLL, not an Avisynth plugin dll - hence the different load options

[Alwyn]

Thanks Dave, so it appears to me that the AVISynth wiki writer thinks it is, on balance, easier to set up the libfftw3f-3.dll and FFTW3.dll files into syswow64 just once and not have to call that DLL via a "LoadDLL" line in every QTGMC script we write. Does that sound right?

[davexnet]

Thanks Dave, so it appears to me that the AVISynth wiki writer thinks it is, on balance, easier to set up the libfftw3f-3.dll and FFTW3.dll files into syswow64 just once and not have to call that DLL via a "LoadDLL" line in every QTGMC script we write. Does that sound right?

Yes. Once it's in the proper location you can forget about it

[lollo]

My choice is to do not mess around with Windows system dirs and to have full control of what is loaded in AviSynth.

In my experience this allows a better understanding of the required additional plugins and, in case of an upgrade of a primary filter, quickly find the solution when the dependencies are broken (errors like green screens, system exceptions, module not found, etc.) or installed the first time.

I am the exception, most just indiscriminately puts plugins or different revisions of the same plugin inside the main folder; it works as well, just more difficult to manage.

[TubeBar]

I apologize for my late response, life got in the way. Lot to digest here so give ma few. Thank you so much @lollo and @ProWo for this info.

I will start downgliding and having a crack at this. I am sure I will run into questions lol

[TubeBar]

NTSC looks better to start with overall - but it depends on what your goals are

PAL is a field blended conversion with more texture and detail loss, worse compression artifacts (some frames are severely worse), cropped more (smaller active image area). Part of the process of deblending requires (global) deinterlacing which degrades it even more

PAL version has slightly elevated Y levels, washed out black level - but colors are easy to adjust to whatever, do not base your decision based on that. You can easily make one look like the other, or something else (e.g if you had a reference cel) in terms of levels and color

Both have rainbows, you might consider applying derainbow filter as well

Yes perhaps except S1E1 for NTSC is seriously "messed up". The audio uses PAL speed-up pitch but the video is much slower. Unlike the other NTSC files it's flagged at 29.976i (others are flagged at 23.976) and appear to be same as the PAL source. Also S1E1 NTSC is clearly taken from a lower quality source, it can't be fixed.

I supposed I can use S1E1 for PAL and rest use NTSC as seems easier to deinterlace.

[TubeBar]

@TubeBar:
[*]download QTGMC from http://avisynth.nl/index.php/QTGMC saving the file QTGMC.avsi inside AviSynth_plugin_dir[*]download Zs RF Shared from http://avisynth.nl/index.php/Zs_RF_Shared saving the file Zs_RF_Shared.avsi inside AviSynth_plugin_dir[*]download MaskTools2 from http://avisynth.nl/index.php/MaskTools2 unpacking the file masktools2_v2.2.30.7z in directory masktools2_v2.2.30 inside AviSynth_plugin_dir[*]download RgTools from http://avisynth.nl/index.php/RgTools unpacking the file RgTools-v1.2.7z in directory RgTools-v1.2 inside AviSynth_plugin_dir[*]download Nnedi3 from http://avisynth.nl/index.php/Nnedi3 unpacking the file NNEDI3_v0_9_4_60.7z in directory NNEDI3_v0_9_4_60 inside AviSynth_plugin_dir[*]download LoadDLL from http://avisynth.nl/index.php/LoadDLL unpacking the file LoadDLL10.7z in directory LoadDLL10 inside AviSynth_plugin_dir
[*]download Srestore from http://avisynth.nl/index.php/Srestore saving the file https://pastebin.com/raw/JYBjMBvx as Srestore.avsi in directory Srestore inside AviSynth_plugin_dir[*]download GRunT from http://avisynth.nl/index.php/GRunT unpacking the file GRunT-v1.02.7z in directory GRunT-v1.02 inside AviSynth_plugin_dir

to open the file[*]download FFmpegSource from http://avisynth.nl/index.php/FFmpegSource unpacking the file ffms2-2.40-msvc.7z in directory ffms2-2.40-msvc inside AviSynth_plugin_dir[/LIST]

The avisynth.nl domain seems to be down. Can you opload the files to your googledrive, unless they are already in the pack you made. I am just about going to look and set things up.

[TubeBar]

I thought so as I couldn't find anything. Looking for a tut on that, thinking about following this guide: https://macilatthefront.blogspot.com/2021/01/deinterlacing-with-avisynth-and-qtgmc.html

Monday I'll send you more info about.

OK thanks

Here we go:

Step 1) Install everything according to lollo's excellent post.

Step 2) Start clever FFmpeg GUI, load your S1E2 (PAL) 25i.mpg file, click main, click encode video stream, click crop detect, check the crop with preview and then click OK, done.


[Attachment 72671 - Click to enlarge]

Set all values as in picture 1, click on Avisynth support, click on create script, the script is created, click on edit script, the script is opened in notepad, add QTGMC().SRestore() at the position like in picture 2 and save the script.


[Attachment 72674 - Click to enlarge]

Click Test script to test the script with virtualdub2.
This may take some time because the file is indexed. If it can be played without problems, close the Vitualdub window and the Notepad window and click encode.

When that is done, click multiplex, the newly created video stream (deinterlaced and progressive, 23.976 fps) is shown in green and already selected, select still the original audio stream, choose the container of your choice (mkv, mp4) and click multiplex (see picture 3).


[Attachment 72673 - Click to enlarge]

Done.

I figured I would start w/ yours since it's pretty straight forward and your test results worked good and as expected already his a problem. @lollo I will go to your more advanced stuff as well.

When I go to test the script w/ Virtualdub2 I get these errors. I tried using both VirtualDub.exe and VirtualDub64.exe neither work but give slightly different error messages. See the attached screen shots.

[ProWo]

Virtualdub2 is ok.
The problem is with Avisynth.

Download
https://www.videohelp.com/download/AviSynthPlus_3.7.3_20230715_vcredist.exe
and install it.
Then post the contents of your AviSynth+\plugins64+ folder (or AviSynth+\plugins+folder if you use the 32bit variant) here.

[TubeBar]

Virtualdub2 is ok.
The problem is with Avisynth.

Download
https://www.videohelp.com/download/AviSynthPlus_3.7.3_20230715_vcredist.exe
and install it.
Then post the contents of your AviSynth+\plugins64+ folder (or AviSynth+\plugins+folder if you use the 32bit variant) here.

I tried it, same error. I installed it and unsitalled twice w/ both options (merge and uninstal). Guess I'll have to play w/ it later. Been a long day.

[TubeBar]

Virtualdub2 is ok.

Then post the contents of your AviSynth+\plugins64+ folder (or AviSynth+\plugins+folder if you use the 32bit variant) here.

That's what I missed. Not sure what you mean by that? WHat dir are you talking about? I see 2 Avysynth dir in Program files, not sure what to put there?

[lollo]

The avisynth.nl domain seems to be down. Can you opload the files to your googledrive, unless they are already in the pack you made. I am just about going to look and set things up.

All plugins are in the pack. They refer to AviSynth 2.60 32-bit.

If you are experimenting ProWo approach with AviSynth+ 64-bit and his GUI, better stay there.

Mine was just a starting point, allowing you to fully understand the whole process and being able in the future to write your own scripts.

[TubeBar]

The avisynth.nl domain seems to be down. Can you opload the files to your googledrive, unless they are already in the pack you made. I am just about going to look and set things up.

All plugins are in the pack. They refer to AviSynth 2.60 32-bit.

If you are experimenting ProWo approach with AviSynth+ 64-bit and his GUI, better stay there.

Mine was just a starting point, allowing you to fully understand the whole process and being able in the future to write your own scripts.

I'll have a look at your pack later, really tired now. Very long and stressful day w/ real life problems. Thanks will have a look later.

[TubeBar]

@lollo thanks you have everything setup for me. I am getting there but still not quite there yet.

I couldn't figure out how to open the script w/ VirtualDub but dragging it onto it seemed to work. I can see and play the video and see 2 screens. Guess one original and other deinterlaced. At least it's there.

The problem is how I use ffmpeg at this point.

I put ffmpeg.exe in the QTGMC dir and ran that command.

ffmpeg.exe -i script.avs -c:v libx264 -crf 17 -preset slow -aspect 4:3 -c:a aac -b:a 128k output.mp4

I get this:

C:\>cd C:\lollo files\QTGMC

C:\lollo files\QTGMC>ffmpeg.exe -i script.avs -c:v libx264 -crf 17 -preset slow -aspect 4:3 -c:a aac -b:a 128k output.mp4
ffmpeg version N-111575-g74474a551b-20230721 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.1.0 (crosstool-NG 1.25.0.196_227d99d)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20230721
  libavutil      58. 14.100 / 58. 14.100
  libavcodec     60. 22.100 / 60. 22.100
  libavformat    60. 10.100 / 60. 10.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9.  8.102 /  9.  8.102
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
[in#0 @ 000002e3f8800900] Error opening input: Unknown error occurred
Error opening input file script.avs.
Error opening input files: Unknown error occurred

C:\lollo files\QTGMC>ffmpeg.exe -i script.avs -c:v libx264 -crf 17 -preset slow -aspect 4:3 -c:a aac -b:a 128k output.mp4

Not sure what is wrong? Not sure where else I can put ffmpeg.exe.

[davexnet]

If the script is not in the same folder as FFmpeg, you have to use a fully qualified file name in the call so the system can find it:

ffmpeg.exe -i "c:\your_folder\script.avs"