Removing MPEG2 compression artefacts from commercial DVDs?

[Master Tape]

I've been ripping some DVDs lately and would like to know if there's a way of removing the compression artefacts that come with the format, in AviSynth or such?

This particular animated DVD for example, exhibits some horrid artefacts in a few frames.



[Attachment 45973 - Click to enlarge]


[Attachment 45974 - Click to enlarge]

[pandy]

Search for mosquito noise and deblocking - for example http://avisynth.nl/index.php/Categoryeringing_%26_Mosquito_Noise - cartoons should be relatively easy to dering and to deblock - some wavelet noise reduction may works nicely, NLE should also improve perceived quality. Perhaps gradfun will work for you.

[jagabo]

Start by using DgDecode's Mpeg2Source() to open the video using its de-ringing option. That will reduce the DCT ringing artifacts substantially. Don't use its deblocking filters as they are too strong and will blur some of the details. For deblocking use some thing like deblock_qed().

the remaining DCT ringing can be greatly reduced with a filter like Deen() or TNLMeans().

Mpeg2Source("2DTV_track1_eng.d2v", CPU2="ooooxx", Info=3) 
deblock_qed()
TNLMeans(Ax=4, Ay=4, Az=0, h=4.0)

[Master Tape]

Cheers fellas! Getting a much better picture now Regarding deblock_qed(), tried so many times, but can't seem to get it to load. The avsi file is in the Plugins folder but keep getting the following error..



And doing Import("C:\Program Files (x86)\AviSynth\plugins\Deblock_QED_MT2Mod.avsi") doesn't work either..



Also had to resize to 1024x576 widescreen as it was anamorphic 720x756, so loaded in AvsPmod in the wrong ratio! There's also an annoying dancing line at the bottom of the screen, particularly near the start, and the outputted video has no sound.

[jagabo]

Note the other filters required by deblock_qed(): http://avisynth.nl/index.php/Deblock_QED#Requirements

[Master Tape]

Have installed the missing DCTfilter but still getting the same error, despite the dll being in the Plugin folder.

Do i have to remux the seperate m2v and ac3 files from the disc in order to restore the audio?

It's also strange you have to keep the DVD in the drive when loading the d2v file.

SetMTMode(5, 4)
Mpeg2Source("C:\Users\Desktop\VTS_01_3.d2v",CPU2="ooooxx", Info=3)
TNLMeans(Ax=4, Ay=4, Az=0, h=4.0)
Deblock_QED()
SetMemoryMax(1000)
#Return Info() 
SetMTMode(2)
AssumeTFF()
QTGMC( Preset="Medium", SourceMatch=3, Lossless=2, EdiThreads=2)
Trim(42560,44000)
Spline36Resize(1024,576)
Crop(14,0,-8,0)
AddBorders(14,0,8,0)

[davexnet]

Best thing to do is to copy the video_ts folder from the optical drive to the hard disk and access it from there.
The d2v is just an index to the data, not the data itself (didn't you notice how small the d2v is relative to the original data?)

[jagabo]

Have installed the missing DCTfilter but still getting the same error, despite the dll being in the Plugin folder.

I have dctfilter.dll in AviSynth's plugins folder. Make sure it's the right "bitness". Try loading it manually in the script and see if you get an error.

Do i have to remux the seperate m2v and ac3 files from the disc in order to restore the audio?

Yes. Or you could import the audio into your AviSynth script with NicAudio.

It's also strange you have to keep the DVD in the drive when loading the d2v file.

The d2v file is just an index file. It doesn't include the video. So the source video has to remain where it is. Rip to your hard drive first if you don't want the DVD spinning all day.

SetMTMode(5, 4)
Mpeg2Source("C:\Users\Desktop\VTS_01_3.d2v",CPU2="ooooxx", Info=3)
TNLMeans(Ax=4, Ay=4, Az=0, h=4.0)
Deblock_QED()
SetMemoryMax(1000)
#Return Info() 
SetMTMode(2)
AssumeTFF()
QTGMC( Preset="Medium", SourceMatch=3, Lossless=2, EdiThreads=2)
Trim(42560,44000)
Spline36Resize(1024,576)
Crop(14,0,-8,0)
AddBorders(14,0,8,0)

I would deblock before tnlmeans. You might want to set Az=2 or so in tnlmeans for some temporal noise reduction. If you find tnlmeans too slow try using deen instead. I don't think it will work quite as well but it may be good enough for you. Or, if you have a supported GPU, use nlmeanscl. Why qtgmc? Do other parts of the video show comb artifacts? Personally, I'd leave the video at 720x576 (or crop to 704x576) and use SAR flags to set the resolution.

[Master Tape]

Best thing to do is to copy the video_ts folder from the optical drive to the hard disk and access it from there.
The d2v is just an index to the data, not the data itself (didn't you notice how small the d2v is relative to the original data?)

I tried that but keep getting the error "MPEG2Source: Could not open one of the input files"

The d2v file is just an index file. It doesn't include the video. So the source video has to remain where it is. Rip to your hard drive first if you don't want the DVD spinning all day.

I demuxed the DVD in DGIndex to extract the m2v (video) and ac3 (audio) files. It also created a d2v file, so i don't know why it wants the original DVD still.

I have dctfilter.dll in AviSynth's plugins folder. Make sure it's the right "bitness". Try loading it manually in the script and see if you get an error.

Hurray it finally works.. had copied and pasted the wrong dctfilter, there were several of them (no clue why) and had to figure out the one i just downloaded among the mess that is my downloads folder..

Yes. Or you could import the audio into your AviSynth script with NicAudio.

I'm having problems getting NicAudio to work. What's the best program for muxing the m2v and ac3 files together?

I would deblock before tnlmeans. You might want to set Az=2 or so in tnlmeans for some temporal noise reduction. If you find tnlmeans too slow try using deen instead. I don't think it will work quite as well but it may be good enough for you. Or, if you have a supported GPU, use nlmeanscl. Why qtgmc? Do other parts of the video show comb artifacts? Personally, I'd leave the video at 720x576 (or crop to 704x576) and use SAR flags to set the resolution.

Skimming through, i don't seem to notice any comb artifacts with this particular source but don't you always have to deinterlace for online distribution? as i do plan to upload some clips. I think it's QTGMC that's causing the flickering line at the bottom of the video (or exaggerating it more)

Would it be wise to add a sharpening filter to counteract the somewhat softer picture, or just leave it as is?

[jagabo]

I demuxed the DVD in DGIndex to extract the m2v (video) and ac3 (audio) files. It also created a d2v file, so i don't know why it wants the original DVD still.

If you did that with "Save Project and Demux Video" the d2v file still references the DVD, not the demuxed m2v. If you want to use the demuxed m2v you have to run DgIndex on the m2v file and produce a new index.

What's the best program for muxing the m2v and ac3 files together?

I use MkvToolNix because I usually create MKV files.

Skimming through, i don't seem to notice any comb artifacts with this particular source but don't you always have to deinterlace for online distribution? as i do plan to upload some clips.

Your source is encoded interlaced but the content of the frames is progressive. You can just treat it as progressive and encode it progressive.

I think it's QTGMC that's causing the flickering line at the bottom of the video (or exaggerating it more)

Yes. QTGMC() can be confused by very thin horizontal lines.

Would it be wise to add a sharpening filter to counteract the somewhat softer picture, or just leave it as is?

If you want. Sharpen(), aWarpSharp2(), FastLineDarkenMod(), Hysteria(), etc. Be aware that sharpening tends to increase noise (including DCT ringing) and create/exacerbate oversharpening halos.

[SaurusX]

I think it's QTGMC that's causing the flickering line at the bottom of the video (or exaggerating it more)

Use the "border=true" parameter in QTGMC to stop that flickering bottom line.

[Master Tape]

If you did that with "Save Project and Demux Video" the d2v file still references the DVD, not the demuxed m2v. If you want to use the demuxed m2v you have to run DgIndex on the m2v file and produce a new index.

Have done so now, but AviSynth really doesn't want to play nice "MPEGSource: Could not open one of the input files"

I use MkvToolNix because I usually create MKV files.

Worked like a charm. Though how do i extract a d2v file from the mkv?

Your source is encoded interlaced but the content of the frames is progressive. You can just treat it as progressive and encode it progressive.

How do i tell AviSynth it's a progressive source without having to deinterlace?

Also i've just noticed there is one part in the video that exhibits comb artifacts, only when the end credits roll, but i guess this isnt part of the same source as the cartoons are in a window box next to them and don't show any signs of it.

If you want. Sharpen(), aWarpSharp2(), FastLineDarkenMod(), Hysteria(), etc. Be aware that sharpening tends to increase noise (including DCT ringing) and create/exacerbate oversharpening halos.

Yeah i don't want to overdo it with the sharpening, just a tiny bit to restore some that has been lost. Doesn't Spline64Resize apply a bit of sharpening too?

[Master Tape]

I think it's QTGMC that's causing the flickering line at the bottom of the video (or exaggerating it more)

Use the "border=true" parameter in QTGMC to stop that flickering bottom line.

Cheers, will come in useful if that problem crops up again.

[jagabo]

If you did that with "Save Project and Demux Video" the d2v file still references the DVD, not the demuxed m2v. If you want to use the demuxed m2v you have to run DgIndex on the m2v file and produce a new index.

Have done so now, but AviSynth really doesn't want to play nice "MPEGSource: Could not open one of the input files"

I use MkvToolNix because I usually create MKV files.

Worked like a charm. Though how do i extract a d2v file from the mkv?

I'm not sure what you're doing here. I recommend you copy the contents of the DVD to your hard drive, then use DgIndex to build a d2v file from that copy, finally use Mpeg2Source to open that d2v file. Once you are done cleaning and encoding your new video mux with the AC3 audio.

Or, if you don't have enough hard drive space to copy the entire DVD to your hard drive (or can't for some other reason): use DgIndex to open your VOB files directly from the DVD, mark the episode you want, then "Save Project and Demux Video". Then discard the d2v file and run DgIndex again, open the m2v file created by the first step, and "Save Project" to create a new d2v file. Finally open that d2v file with Mpeg2Source() to do your cleaning and encoding.

How do i tell AviSynth it's a progressive source without having to deinterlace?

You don't have to do anything. AviSynth doesn't really even know if your video is progressive or interlaced. Just skip the QTGMC step.

Also i've just noticed there is one part in the video that exhibits comb artifacts, only when the end credits roll, but i guess this isnt part of the same source as the cartoons are in a window box next to them and don't show any signs of it.

What you're seeing is common with broadcast video. Stuff overlaid by the broadcaster may not match the main program. If you really want to get rid of the combing in those sections use QTGMC().SelectEven() just on those sections (SelectEven() to restore the 25 fps frame rate).

Doesn't Spline64Resize apply a bit of sharpening too?

Yes, many of the resizers sharpen: Bicubic, Lanczos, Spline16/36/64, etc.

[Master Tape]

I'm not sure what you're doing here. I recommend you copy the contents of the DVD to your hard drive, then use DgIndex to build a d2v file from that copy, finally use Mpeg2Source to open that d2v file. Once you are done cleaning and encoding your new video mux with the AC3 audio.

Or, if you don't have enough hard drive space to copy the entire DVD to your hard drive (or can't for some other reason): use DgIndex to open your VOB files directly from the DVD, mark the episode you want, then "Save Project and Demux Video". Then discard the d2v file and run DgIndex again, open the m2v file created by the first step, and "Save Project" to create a new d2v file. Finally open that d2v file with Mpeg2Source() to do your cleaning and encoding.

Ah that's where i went wrong.. i assumed 'Save Project' was just saving it for later, can finally take the DVD out of my drive now lol. I'm not encoding the entire DVD in this instance, just specific clips, so not sure how to mux the 25 minute audio file to a 1 minute clip?

Have dragged just the VOB were the special features section is stored, so working with that.

What you're seeing is common with broadcast video. Stuff overlaid by the broadcaster may not match the main program. If you really want to get rid of the combing in those sections use QTGMC().SelectEven() just on those sections (SelectEven() to restore the 25 fps frame rate).

So would i have to render those parts seperate then later join them up in ffmpeg? Just incase this situation pops up again in future.

[Atak_Snajpera]

To remove noise and other artefacts in animations I just use MDegrain2+HQDN3D

Loadplugin("C:\Users\Dave\Documents\Delphi_Projects\RipBot264\_Compiled\Tools\AviSynth plugins\mvtools\mvtools2.dll")
super=MSuper(video,pel=2)
fv1=MAnalyse(super,isb=false,delta=1,overlap=4)
bv1=MAnalyse(super,isb=true,delta=1,overlap=4)
fv2=MAnalyse(super,isb=false,delta=2,overlap=4)
bv2=MAnalyse(super,isb=true,delta=2,overlap=4)
video=MDegrain2(video,super,bv1,fv1,bv2,fv2,thSAD=400)
LoadPlugin("C:\Users\Dave\Documents\Delphi_Projects\RipBot264\_Compiled\Tools\AviSynth plugins\hqdn3d\hqdn3d.dll")
LoadPlugin("C:\Users\Dave\Documents\Delphi_Projects\RipBot264\_Compiled\Tools\AviSynth plugins\flash3kyuu_deband\flash3kyuu_deband.dll")
video=hqdn3d(video,6,0,0,0).f3kdb

Example before and after

[Master Tape]

One more thing, how do you get rid of this annoying artifact, seen at the beginning of these two clips, when it usually pans into something, like these buildings for example?

[jagabo]

QTGMC cleans that up a bit:

TurnRight().QTGMC(InputType=3).TurnLeft().QTGMC(InputType=3)

The first one cleans vertical lines, the second horizontal lines. before and after:

[Master Tape]

Thanks, that's great! It's amazing what all these different filters can do/clean up. A big thank you for all your help on this jagabo, without would not have been possible

[jagabo]

Be aware that two consecutive QTGMC calls will be very slow.

[Master Tape]

Back again! I have since upgraded to AviSynth+ but having trouble getting MPEG2Source to work. I have downloaded the required D2VSource plugin and placed it in the plugins64+ folder, but getting nowhere


[Attachment 54546 - Click to enlarge]

[poisondeathray]

Back again! I have since upgraded to AviSynth+ but having trouble getting MPEG2Source to work. I have downloaded the required D2VSource plugin and placed it in the plugins64+ folder, but getting nowhere


[Attachment 54546 - Click to enlarge]

The new one has changed syntax to D2VSource()

https://github.com/Asd-g/MPEG2DecPlus

Or you can use old version such as MPEG2DecPlus-0.1.2 . There are no other material changes

[Master Tape]

Oh i see. Thanks once again

[Master Tape]

I'm trying to apply the deringing option again but i'm guessing the new syntax works differently..


[Attachment 54547 - Click to enlarge]

[davexnet]

I'm trying to apply the deringing option again but i'm guessing the new syntax works differently..


[Attachment 54547 - Click to enlarge]

Why don't you read the readme file included in the package?

[Master Tape]

Why don't you read the readme file included in the package?

There was no reference to deringing, and now i know why. CPU is a disabled parameter in MPEG2DecPlus/D2VSource. Just seems more like a downgrade to me than a successor.

# D2VSource

This is a project (previously named as MPEG2DecPlus) to modify DGDecode.dll for AviSynth+.

# Usage

 ```
 D2VSource(string "d2v", int "idct", bool "showQ", int "info", int "upConv", bool "i420", bool "iCC")
 ```

## Parameters:

- d2v\
    The path of the dv2 file.
    
- idct\
    The iDCT algorithm to use.\
    0: Follow the d2v specification.\
    1,2,3,6,7: AP922 integer (same as SSE2/MMX).\
    4: SSE2/AVX2 LLM (single precision floating point, SSE2/AVX2 determination is automatic).\
    5: IEEE 1180 reference (double precision floating point).\
    Default: -1.
    
- showQ\
    It displays macroblock quantizers..\
    Default: False.
    
- info\
    It prints debug information.\
    0: Do not display.\
    1: Overlay on video frame.\
    2: Output with OutputDebugString(). (The contents are confirmed by DebugView.exe).\
    3: Embed hints in 64 bytes of the frame upper left corner.\
    Default: 0.
    
- upConv\
    The output format.\
    0: No conversion. YUV420 output is YV12, YUV422 output is YV16.\
    1: Output YV16.\
    2: Output YV24.\
    Default: 0.
    
- i420\
    It determinates what is the output of YUV420.\
    True: The output is i410.\
    False: The output is YV12.\
    Default: False.
    
- iCC\
    It determinates how YUV420 is upscaled when upConv=true.\
    True: Force field-based upsampling.\
    False: Forse progressive upsampling.\
    Default: Auto determination based on the frame flag.

[davexnet]

Why not revert to the old DGindex? Works fine under Avisynth Plus

[poisondeathray]

Why not revert to the old DGindex? Works fine under Avisynth Plus

DGIndex works for either to index; it's dgdecode.dll that does the actual decoding and post processing for x86 . MPEG2DecPlus64.dll and D2VSource.dll do not have all the functions as the original


@Master Tape -
You can use x86 filter (even a source filter) in x64 host with mp_pipeline . You can mix/match pass back and forth between x86/x64 in the same script . Some filters do not have native x64 yet, eg. hdragc, jpegsource (recon passes are useful) , dedup, a few others...so this means everything is usable

And yes, there are examples and instructions

[jagabo]

The 64 bit version of the old DgDecode.dll works in AviSynth+. But it's also missing the deblocking and deringing options. I use the 32 bit version when I need that.

[Master Tape]

Why not revert to the old DGindex? Works fine under Avisynth Plus

Thanks, all working fine now. Though looks like i had a different kind of problem with my source, aliasing and the saw-tooth effect. Resolution was 720x576, but the source was widescreen. So i kept the resolution the same and just used a 16:9 flag in the encoder, to stretch out the picture again like a DVD player would but the aliasing remained.

I've attached an example. You'll notice it on slanted edges, like a nose, or the bar taps, or even a sideway view of a head.

Spline64Resize(1024,576) fixes the problem mostly, but is there a way to get rid of this saw-tooth effect without having resize the video?