So after processing a bunch of clips and reading a bunch of forum threads I think I've confused myself and just wanted to check to see if I did everything right. I have a DVD recorded off television, DGindex says the VOB's are SMPTE 170M, the D2V has PC scale for YUV to RGB conversion. I loaded the D2V into avisynth with DGDecode_mpeg2source, I did not use colormatrix(), followed by QTGMC and encoded to Lagarith in YV12. After trimming all the AVI's I load them back into avisynth with AVISource, UnalightedSplice and convertToYV12(interlaced=false, matrix="Rec601") for final output to x264 via MeGUI. If I look at either the Histogram or ColorYUV(analyse=true) it's showing colors outside the 16-235 range. From everything I read this is because PC scale expands the range from 16-235 out to 0-255. If I just load up the VOB in something like Mediaplayer Classic or VLC it seems like that's what they do anyway so I'm confused if I should keep it like this or add something like Levels(0, 1, 255, 16, 235, coring=false,dither=true) to bring things back to where they should be before export or is that only is outputting to MPEG2 for DVD?
+ Reply to Thread
Results 1 to 20 of 20
Last edited by mrwhitethc; 12th Feb 2020 at 10:27.
[Attachment 51922 - Click to enlarge]
That's what I thought as well but I still get this. I even went back to the original D2V file, loaded it into avisynth with DGDecode_mpeg2source("Part_1_1.d2v").convertToYV12 (interlaced=true, matrix="Rec601").ColorYUV(analyze=true) and I get the same thing:
[Attachment 51923 - Click to enlarge]
It's not unusual for DVD sources to have values outside the Y=16-235 U/V=16-240 range, especially home recordings. Use Histogram for a better view of what's going on.
The output of Mpeg2Source() should already be interlaced YV12 with rec.601 colors. The ConvertToYV12() shouldn't be needed. On the other hand it won't hurt because it shouldn't do anything.
Maybe upload a small sample of your source (cut and demuxed with DgIndex). Include both bright and dark areas. And post your entire scripts so one can see where there might be problems. Note that sharpening filters can produce halos that extend outside the legal range. And automatic levels filters often produce illegal values too.
Attached a zip with the d2v, m2v and ac3 cut and demuxed in DGIndex
As for the scripts, I used this to load into VirtualDub to make the lossless AVI:
SetMemoryMax(700) # This line may not be necessary - depends on your settings, number of threads and available memory. Try 400,600,700,800,1000,1200... Import("D:\Program Files\Apps AV\QTGMC\QTGMC-3.32.avsi") Import("D:\Program Files\Apps AV\QTGMC\Dither\dither.avsi") Import("D:\Program Files\Apps AV\QTGMC\Dither\mt_xxpand_multi.avsi") Import("D:\Program Files\Apps AV\QTGMC\GrainFactory\Grainfactory3.avsi") LoadPlugin("D:\Program Files\Apps AV\MEGUI_2836\tools\dgindex\DGDecode.dll") LoadPlugin("D:\Program Files\Apps AV\MEGUI_2836\tools\avisynth_plugin\TIVTC.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\AddGrainC.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\dfttest.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\EEDI2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\eedi3.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\FFT3DFilter.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\mvtools2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\nnedi.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\nnedi2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\nnedi3.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\RemoveGrainSSE2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\RepairSSE2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\SSE2Tools.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\TDeint.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\VerticalCleanerSSE2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\2.6x Plugins\mt_masktools-26.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\chromashift27\ChromaShift.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Dither\win32\dither.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Dither\win32\avstp.dll") A=WAVSource("Part_1_1 T80 2_0ch 384Kbps 48KHz.wav") DGDecode_mpeg2source("Part_1_1.d2v") V=AssumeTFF().QTGMC( Preset="Slower", MatchPreset="Slow", MatchPreset2="Slow", TR2=2, Sharpness=0.6, Denoiser="dfttest", NoiseProcess=2, GrainRestore=1, NoiseRestore=0.2, DenoiseMC=true, NoiseDeint="Generate", StabilizeNoise=true).GradFun3().Crop(12, 0, -8, -4).grainfactory3(g1str=4, g2str=6, g3str=8).AddBorders(10,2,10,2) AudioDub(V,A)
then I trimmed the AVI in VirtualDub to get rid of any problem areas and saved the resulting AVI with direct stream copy for the video and audio. Then I'm going to load this back into MeGUI to make the final h264 MKV
A=AVISource("01_Part_1_1.avi") B=AVISource("02_Part_1_2.avi") C=AVISource("03_Part_1_3.avi") D=AVISource("04_Part_1_4.avi") UnalignedSplice(A,B,C,D).Crop(0,6,0,0).AddBorders(0,4,0,2).convertToYV12(interlaced=false, matrix="Rec601")
I ended up cropping a 2nd time to get rid of some problematic areas I missed the first time around and then recenter the video.
"so I'm confused if I should keep it like this or add something like Levels(0, 1, 255, 16, 235, coring=false,dither=true) to bring things back to where they should be before export or is that only is outputting to MPEG2 for DVD?"
Depends on your output target.
Ntsc usa dvd etc will use the 16-255 range, so if you're trying to make an edited ntsc dvd, stick to that range else the dark areas will be too dark.
If you're outputting to ntsc Japan dvd, 0-255 is the standard there, so if you use 16-255, the dark areas will look to light/gray.
All PCs use 0-255, but usually, software dvd players adjust for the difference.
If you use a media player like mphc with 16-255 video, you'll need to add a video SHADER that adjusts for this (right click, menu, find the shades submenu, add the 16-255 to 0-255 shader and add it to the pre-... Section.).
Bluray can handle 0-255.
Other output devices, check specs if they use 0 or 16 as black.
[Attachment 51926 - Click to enlarge]
There's still a little later in the clip:
[Attachment 51927 - Click to enlarge]
That's using TurnRight().Histogram().TurnLeft() on the source video, with no other processing. The brownish bars at the top and bottom of the histogram (really a waveform monitor) indicate the super whites (Y>235, none here) and super blacks (Y<16). Perfect limited range video would have nothing in those areas. But in the real world you often see a little there.
The front and back porches (the black bars at the extreme left and right edges) at Y=0 may indicate a Japanese video tape was recorded on a North American recorder (the NTSC setup issue babygdav allueded to -- but pretty much everything else he said was wrong) . Regardless of the cause you should fix it. Almost all video devices use limited range YUV (Broadcast TV, Satellite TV, DVD, Blu-ray, Youtube, etc.).
How to address it here? There are only two shots here so it's hard to say for sure. But brights in the clip rarely exceed ~190. So adding 16 to Y and applying a small gain may be appropriate. Something like ColorYUV(gain_y=40, off_y=16) will get the black level up and keep the brightest brights from exceeding 235 (except for a few oversharpening halos). But I would look at a lot of other shots to verify those limits. For example, I wouldn't care the brights in the early part of the clip were blown out if the rest of the video looked good.
The processing you listed indicates there were not YUV to RGB conversions. The ConvertToYV12 in your second script didn't do anything since the video was already YV12. That's why you continued to have super blacks.
I also looked at it in Virtualdub2; I tried to increase the gamma a little to brighten the image
and I didn't see the incursion into the super black. Then I realized that the levels command
was crushing the black (I increased the gamma to 1.09 from 1.0)
MeGUI and possibly avoid the lossless intermediate file?
After you've created your script, save it and open it with MeGUI's AVS Cutter from the Tools menu. Use the preview to add the "cuts". The cuts are the sections you want to keep. When you're done you save the cuts to the script, and you can also save a separate cuts file. The cuts file is for loading into the audio section so the audio can be re-encoded to match the video (the area for loading the cuts file is labelled Cuts).
In the example below, there's no frames left out because I wasn't editing as such, but obviously there'd be gaps in the frame numbers if you were editing out particular sections of video. This is how MeGUI adds the cuts to the script though.
In this case I was using the cuts to apply different filtering to different sections of the script, or you can apply different cropping etc, regardless of whether you're using the cuts to edit as such.
__film = last
__t0 = __film.trim(0, 924)
__t1 = __film.trim(925, 25492)
__t2 = __film.trim(25843, 27635)
__t3 = __film.trim(27636, 28889)
__t4 = __film.trim(28890, 36950)
__t0 ++ __t1 ++ __t2 ++ __t3 ++ __t4
Used to apply different cropping (and making up numbers to illustrate).
__film = last
__t0 = __film.trim(0, 924).Crop(12,4,-6,-8).Spline36Resixe(640,480)
__t1 = __film.trim(925, 25492).Crop(4,4,-12,--8).Spline36Resixe(640,480)
__t2 = __film.trim(25843, 27635).Crop(12,4,-6,-8).Spline36Resixe(640,480)
__t3 = __film.trim(27636, 28889).Crop(4,4,-12,--8).Spline36Resixe(640,480)
__t4 = __film.trim(28890, 36950).Crop(12,4,-6,-8).Spline36Resixe(640,480)
__t0 ++ __t1 ++ __t2 ++ __t3 ++ __t4
I just thought I'd mention it in case you weren't aware of the AVS Cutter as it can be quite useful. If you're de-interlacing with QTGMC it can pay to replace it with Yadif(mode=1) temporarily while you add the cuts, and comment out any really slow filtering temporarily so you can navigate easily, but by using the AVS Cutter to edit first you can just load the script and encode, no intermediate lossless file for editing required.
Last edited by hello_hello; 10th Feb 2020 at 03:19.
Levels(0, 1, 255, 16, 235, coring=false,dither=true)
I did add this:
#Crop the black addborders
comma = ","
WriteFileStart("D:\temp\MinMax.txt", """ "FrameNumber,YMIN,YMAX,UMIN,UMAX,VMIN,VMAX" """, append=true)
WriteFile("D:\temp\MinMax.txt", "current_frame", "comma", "YPlaneMin", "comma", "YPlaneMax", "comma", "UPlaneMin", "comma", "UPlaneMax", "comma", "VPlaneMin", "comma", "VPlaneMax", append=true)
WriteFileEnd("D:\tempMinMax.txt", """ "End,End,End,End,End,End,End" """, append=true)
Thank you for sharing this. I did briefly play with this but there are so many clips the needed trimming and rearranging it was just easier to batch output the whole thing, trim and output new clean AVI's from Virtualdub, then number and rearrange the pieces as I needed them. I have the hard drive space so might as well use it.
After scanning through I pulled out 2 clips where the Luma hits 255.
Also looking at them I tried both
ColorYUV(gain_y=20, off_y=8).Levels(0, 1, 255, 16, 235, coring=false,dither=true)
Here I cropped 4 lines off the bottom, to avoid Avisynth complaining about mod4 height.
[Attachment 51975 - Click to enlarge]
[Attachment 51976 - Click to enlarge]
[Attachment 51977 - Click to enlarge]
[Attachment 51978 - Click to enlarge]
DPS-470AV, Reference: SDI 10-bit → FFVideoSource 8-bit images:
(Warning: do not click if you have a data limit! Link auto-loads MBs of PNGs.)
[EDIT: Come to think of it, I don't know whether looking at the SDI capture is appropriate since I don't know whether SDI includes pedestal in the same way. But the XCard captures, for example, show the same thing.]
Last edited by Brad; 12th Feb 2020 at 09:53. Reason: Thought of something last night
I'll have to dive into that link later, thanks.
I guess my biggest thing is trying to learn what to look for since I'm in the process of converting a wide range of vhs tapes and home made DVD-R's.
I'm in the process of converting a wide range of vhs tapes and home made DVD-R's.
[Attachment 51984 - Click to enlarge]
Which pixels are at 255? Let's use mt_binarize(254, chroma="-128") to find out:
[Attachment 51985 - Click to enlarge]
Pixels with values 254 and below are black, pixels with value 255 and above (but nothing can be above 255) are white. The single white pixel is at the bottom of the frame, in the head switching noise. Ok, so which pixels are above 235 (the max legal Y value)?
[Attachment 51986 - Click to enlarge]
Aside from the head switching noise at the bottom of the frame there's only one pixel above 235. Which pixels are above 200?
[Attachment 51987 - Click to enlarge]
But even those are all oversharpening halos. It's not until you get down to ~190 that you start seeing real picture details turn white.
Let's look at another frame with Histogram() (which, again, is really a waveform monitor) using TurnRight().Histogram().TurnLeft():
[Attachment 51988 - Click to enlarge]
The white bar near the top of the waveform monitor is from the bright sky. The Y values there are around 200 plus or minus a few units. If you scrub through the video you'll see that the big bright areas are all around that level. So that's close to the actual white level -- which would normally be at 235, the bottom of the top brown bar. The black levels look about right, very close to the top of the bottom brown bar.
Unless you plan to go through the video and adjust levels shot by shot you should look for a compromise setting that keeps the black and white levels close to the expected 16 and 235 values. This is where you have to use your own judgement. Auto levels adjustment filters don't work very well.
Beyond that have have things like hue, saturation, white balance, gamma etc. And just because all Y values fall between 16 and 235, and all U and V values fall between 16 and 240 doesn't mean every pixel is legal. For example, at Y=16 (full black) there is only one legal value for U and V, 128. The same is true at Y=235 (full white). In fact, most YUV combinations between the stated min/max ranges are illegal. Here's a picture of the valid RGB values within the YUV (YCbCr) cube:
(from https://software.intel.com/en-us/node/503873, "RGB Colors Cube in the YCbCr Space")
You can that most YUV values are outside the inner RGB cube. There are filters that can test for this too.
Thank you everyone for the help, after looking at the raw numbers and waveforms from ColorYUV, Histogram and mt_binarize, a bunch of many test encodes and finally looking at the playback on a few TV's and my monitors at home I've come up with:
This seems to give me just the right amount of blacks and brightness to the image to make out what the hell is going on during the night scenes along with fix the small color shift. Thanks to everyone who took the time, I now have a new obsession when making my transfers to look over
Last edited by mrwhitethc; 18th Feb 2020 at 21:16.