View Profile
View Forum Posts
So after processing a bunch of clips and reading a bunch of forum threads I think I've confused myself and just wanted to check to see if I did everything right. I have a DVD recorded off television, DGindex says the VOB's are SMPTE 170M, the D2V has PC scale for YUV to RGB conversion. I loaded the D2V into avisynth with DGDecode_mpeg2source, I did not use colormatrix(), followed by QTGMC and encoded to Lagarith in YV12. After trimming all the AVI's I load them back into avisynth with AVISource, UnalightedSplice and convertToYV12(interlaced=false, matrix="Rec601") for final output to x264 via MeGUI. If I look at either the Histogram or ColorYUV(analyse=true) it's showing colors outside the 16-235 range. From everything I read this is because PC scale expands the range from 16-235 out to 0-255. If I just load up the VOB in something like Mediaplayer Classic or VLC it seems like that's what they do anyway so I'm confused if I should keep it like this or add something like Levels(0, 1, 255, 16, 235, coring=false,dither=true) to bring things back to where they should be before export or is that only is outputting to MPEG2 for DVD?
Try StreamFab Downloader and download from Netflix, Amazon, Youtube! Or Try DVDFab and copy Blu-rays! or rip iTunes movies!
+ Reply to Thread
Results 1 to 20 of 20
Thread
-
Last edited by mrwhitethc; 12th Feb 2020 at 10:27.
-
But convertToYV12(interlaced=false, matrix="Rec601") recompresses the 0-255 RGB range to 16-235 YUV.Originally Posted by mrwhitethc
-
[Attachment 51922 - Click to enlarge]
That's what I thought as well but I still get this. I even went back to the original D2V file, loaded it into avisynth with DGDecode_mpeg2source("Part_1_1.d2v").convertToYV12 (interlaced=true, matrix="Rec601").ColorYUV(analyze=true) and I get the same thing:
[Attachment 51923 - Click to enlarge] -
It's not unusual for DVD sources to have values outside the Y=16-235 U/V=16-240 range, especially home recordings. Use Histogram for a better view of what's going on.
The output of Mpeg2Source() should already be interlaced YV12 with rec.601 colors. The ConvertToYV12() shouldn't be needed. On the other hand it won't hurt because it shouldn't do anything.
Maybe upload a small sample of your source (cut and demuxed with DgIndex). Include both bright and dark areas. And post your entire scripts so one can see where there might be problems. Note that sharpening filters can produce halos that extend outside the legal range. And automatic levels filters often produce illegal values too. -
Attached a zip with the d2v, m2v and ac3 cut and demuxed in DGIndex
As for the scripts, I used this to load into VirtualDub to make the lossless AVI:
Code:SetMemoryMax(700) # This line may not be necessary - depends on your settings, number of threads and available memory. Try 400,600,700,800,1000,1200... Import("D:\Program Files\Apps AV\QTGMC\QTGMC-3.32.avsi") Import("D:\Program Files\Apps AV\QTGMC\Dither\dither.avsi") Import("D:\Program Files\Apps AV\QTGMC\Dither\mt_xxpand_multi.avsi") Import("D:\Program Files\Apps AV\QTGMC\GrainFactory\Grainfactory3.avsi") LoadPlugin("D:\Program Files\Apps AV\MEGUI_2836\tools\dgindex\DGDecode.dll") LoadPlugin("D:\Program Files\Apps AV\MEGUI_2836\tools\avisynth_plugin\TIVTC.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\AddGrainC.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\dfttest.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\EEDI2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\eedi3.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\FFT3DFilter.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\mvtools2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\nnedi.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\nnedi2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\nnedi3.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\RemoveGrainSSE2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\RepairSSE2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\SSE2Tools.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\TDeint.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\VerticalCleanerSSE2.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Avisynth 32-bit Plugins\2.6x Plugins\mt_masktools-26.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\chromashift27\ChromaShift.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Dither\win32\dither.dll") LoadPlugin("D:\Program Files\Apps AV\QTGMC\Dither\win32\avstp.dll") A=WAVSource("Part_1_1 T80 2_0ch 384Kbps 48KHz.wav") DGDecode_mpeg2source("Part_1_1.d2v") V=AssumeTFF().QTGMC( Preset="Slower", MatchPreset="Slow", MatchPreset2="Slow", TR2=2, Sharpness=0.6, Denoiser="dfttest", NoiseProcess=2, GrainRestore=1, NoiseRestore=0.2, DenoiseMC=true, NoiseDeint="Generate", StabilizeNoise=true).GradFun3().Crop(12, 0, -8, -4).grainfactory3(g1str=4, g2str=6, g3str=8).AddBorders(10,2,10,2) AudioDub(V,A)
then I trimmed the AVI in VirtualDub to get rid of any problem areas and saved the resulting AVI with direct stream copy for the video and audio. Then I'm going to load this back into MeGUI to make the final h264 MKV
Code:A=AVISource("01_Part_1_1.avi") B=AVISource("02_Part_1_2.avi") C=AVISource("03_Part_1_3.avi") D=AVISource("04_Part_1_4.avi") UnalignedSplice(A,B,C,D).Crop(0,6,0,0).AddBorders(0,4,0,2).convertToYV12(interlaced=false, matrix="Rec601")
I ended up cropping a 2nd time to get rid of some problematic areas I missed the first time around and then recenter the video. -
"so I'm confused if I should keep it like this or add something like Levels(0, 1, 255, 16, 235, coring=false,dither=true) to bring things back to where they should be before export or is that only is outputting to MPEG2 for DVD?"
Depends on your output target.
Ntsc usa dvd etc will use the 16-255 range, so if you're trying to make an edited ntsc dvd, stick to that range else the dark areas will be too dark.
If you're outputting to ntsc Japan dvd, 0-255 is the standard there, so if you use 16-255, the dark areas will look to light/gray.
All PCs use 0-255, but usually, software dvd players adjust for the difference.
If you use a media player like mphc with 16-255 video, you'll need to add a video SHADER that adjusts for this (right click, menu, find the shades submenu, add the 16-255 to 0-255 shader and add it to the pre-... Section.).
Bluray can handle 0-255.
Other output devices, check specs if they use 0 or 16 as black.
https://www.provideocoalition.com/luminance_ranges/ -
Your source has lots of blacks below Y=16. It's worst in the early part of the clip where the titles are flickering:Originally Posted by mrwhitethc
[Attachment 51926 - Click to enlarge]
There's still a little later in the clip:
[Attachment 51927 - Click to enlarge]
That's using TurnRight().Histogram().TurnLeft() on the source video, with no other processing. The brownish bars at the top and bottom of the histogram (really a waveform monitor) indicate the super whites (Y>235, none here) and super blacks (Y<16). Perfect limited range video would have nothing in those areas. But in the real world you often see a little there.
The front and back porches (the black bars at the extreme left and right edges) at Y=0 may indicate a Japanese video tape was recorded on a North American recorder (the NTSC setup issue babygdav allueded to -- but pretty much everything else he said was wrong) . Regardless of the cause you should fix it. Almost all video devices use limited range YUV (Broadcast TV, Satellite TV, DVD, Blu-ray, Youtube, etc.).
How to address it here? There are only two shots here so it's hard to say for sure. But brights in the clip rarely exceed ~190. So adding 16 to Y and applying a small gain may be appropriate. Something like ColorYUV(gain_y=40, off_y=16) will get the black level up and keep the brightest brights from exceeding 235 (except for a few oversharpening halos). But I would look at a lot of other shots to verify those limits. For example, I wouldn't care the brights in the early part of the clip were blown out if the rest of the video looked good.
The processing you listed indicates there were not YUV to RGB conversions. The ConvertToYV12 in your second script didn't do anything since the video was already YV12. That's why you continued to have super blacks. -
I also looked at it in Virtualdub2; I tried to increase the gamma a little to brighten the image
and I didn't see the incursion into the super black. Then I realized that the levels command
was crushing the black (I increased the gamma to 1.09 from 1.0)
According to the doc the method to correct this, to remap without crushing is this .Code:levels(0,1.09,255,0,255)
Not seen this mentioned much around here: any downside to this?Code:levels(0,1.09,255,16,235,coring=false)
-
I haven't looked at the video, but as a side note are you aware you can edit with MeGUI and possibly avoid the lossless intermediate file?Originally Posted by mrwhitethc
After you've created your script, save it and open it with MeGUI's AVS Cutter from the Tools menu. Use the preview to add the "cuts". The cuts are the sections you want to keep. When you're done you save the cuts to the script, and you can also save a separate cuts file. The cuts file is for loading into the audio section so the audio can be re-encoded to match the video (the area for loading the cuts file is labelled Cuts).
In the example below, there's no frames left out because I wasn't editing as such, but obviously there'd be gaps in the frame numbers if you were editing out particular sections of video. This is how MeGUI adds the cuts to the script though.
In this case I was using the cuts to apply different filtering to different sections of the script, or you can apply different cropping etc, regardless of whether you're using the cuts to edit as such.
__film = last
__t0 = __film.trim(0, 924)
__t1 = __film.trim(925, 25492)
__t2 = __film.trim(25843, 27635)
__t3 = __film.trim(27636, 28889)
__t4 = __film.trim(28890, 36950)
__t0 ++ __t1 ++ __t2 ++ __t3 ++ __t4
Used to apply different cropping (and making up numbers to illustrate).
__film = last
__t0 = __film.trim(0, 924).Crop(12,4,-6,-8).Spline36Resixe(640,480)
__t1 = __film.trim(925, 25492).Crop(4,4,-12,--8).Spline36Resixe(640,480)
__t2 = __film.trim(25843, 27635).Crop(12,4,-6,-8).Spline36Resixe(640,480)
__t3 = __film.trim(27636, 28889).Crop(4,4,-12,--8).Spline36Resixe(640,480)
__t4 = __film.trim(28890, 36950).Crop(12,4,-6,-8).Spline36Resixe(640,480)
__t0 ++ __t1 ++ __t2 ++ __t3 ++ __t4
I just thought I'd mention it in case you weren't aware of the AVS Cutter as it can be quite useful. If you're de-interlacing with QTGMC it can pay to replace it with Yadif(mode=1) temporarily while you add the cuts, and comment out any really slow filtering temporarily so you can navigate easily, but by using the AVS Cutter to edit first you can just load the script and encode, no intermediate lossless file for editing required.Last edited by hello_hello; 10th Feb 2020 at 03:19.
-
No downside. The default of coring=true is retarded, due to the way it works.Originally Posted by davexnet
-
So I looked through and noticed I started to get more areas with 255. I did keep the original small, I can upload a second sample where this would push the whites over?Originally Posted by jagabo
I did play around with the level command before but wasn't sure if I was using it correctly, this is what I had some across, I added the dither since it seemed like the histogram had some weird artifacts:Originally Posted by davexnet
Code:Levels(0, 1, 255, 16, 235, coring=false,dither=true)
Good to know.Originally Posted by hello_hello
I did add this:
And was able to throw that into excel and extract some data, however, it takes forever to output the text file, I was hoping I could output to null or something but I ended up just doing a "Scan stream for errors" in virtualdub.#Crop the black addborders
UnalignedSplice(A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q, R).Crop(12,4,-12,-4)
comma = ","
WriteFileStart("D:\temp\MinMax.txt", """ "FrameNumber,YMIN,YMAX,UMIN,UMAX,VMIN,VMAX" """, append=true)
WriteFile("D:\temp\MinMax.txt", "current_frame", "comma", "YPlaneMin", "comma", "YPlaneMax", "comma", "UPlaneMin", "comma", "UPlaneMax", "comma", "VPlaneMin", "comma", "VPlaneMax", append=true)
WriteFileEnd("D:\tempMinMax.txt", """ "End,End,End,End,End,End,End" """, append=true) -
Originally Posted by hello_hello
Thank you for sharing this. I did briefly play with this but there are so many clips the needed trimming and rearranging it was just easier to batch output the whole thing, trim and output new clean AVI's from Virtualdub, then number and rearrange the pieces as I needed them. I have the hard drive space so might as well use it. -
After scanning through I pulled out 2 clips where the Luma hits 255.
Also looking at them I tried both
which had the whites looking whiter but maybe a touch yellow andColorYUV(gain_y=20, off_y=8).Levels(0, 1, 255, 16, 235, coring=false,dither=true)
which seems to make the colors richer but seems to be reddish. I'm guessing it's all up to how it looks and there's no way to look at it based purely on numbers?levels(0,1.09,255,16,235,coring=false) -
1.09 in the levels command was just my guess at a modest gamma change. Feel free to experiment, 1.0 is the defaultOriginally Posted by mrwhitethc
-
Formerly 'vaporeon800'
- Apr 2001
- Vancouver, Canada
You need to crop 2 lines off the bottom to eliminate head-switching noise. The white you're reading is a blip at the head-switch point. After doing that, the highest level I see is 251, which is just a spurious single pixel in two frames (probably block noise?).Originally Posted by mrwhitethc
Here I cropped 4 lines off the bottom, to avoid Avisynth complaining about mod4 height.
[Attachment 51975 - Click to enlarge]
[Attachment 51976 - Click to enlarge]
[Attachment 51977 - Click to enlarge]
[Attachment 51978 - Click to enlarge]
This is normal. Test signal generator function of DPS-470AV, Reference: SDI 10-bit → FFVideoSource 8-bit images:Originally Posted by jagabo
https://forum.videohelp.com/threads/360704-2013-my-video-capture-device-comparison-scr...=1#post2285745
(Warning: do not click if you have a data limit! Link auto-loads MBs of PNGs.)
[EDIT: Come to think of it, I don't know whether looking at the SDI capture is appropriate since I don't know whether SDI includes pedestal in the same way. But the XCard captures, for example, show the same thing.]Last edited by Brad; 12th Feb 2020 at 09:53. Reason: Thought of something last night
-
I think it does look better, so thank you for that davexnet.Originally Posted by davexnet
You're right, I had 2 different sheets I was looking at, one with only 255 and one with anything above 235. While they have trace amounts of 255 they also have longer segments of above 235. Since the luma was being adjusted and like jagabo had mentioned about looking to see any out of range specs I wanted to have clips that might be out of range with different adjustments.Originally Posted by vaporeon800
You need to crop 2 lines off the bottom to eliminate head-switching noise. The white you're reading is a blip at the head-switch point. After doing that, the highest level I see is 251, which is just a spurious single pixel in two frames (probably block noise?).Originally Posted by mrwhitethc
I'll have to dive into that link later, thanks.
I guess my biggest thing is trying to learn what to look for since I'm in the process of converting a wide range of vhs tapes and home made DVD-R's. -
Formerly 'vaporeon800'
- Apr 2001
- Vancouver, Canada
-
The thread I linked? That was just for jagabo and others interested in the topic he brought up. You can safely ignore it.Originally Posted by mrwhitethc
Would you mind changing the thread title of your original post to say something like VHS-DVDR or similar instead of just DVD? I initially ignored the thread since my interest is mainly tape conversion; from the title I assumed it was just a straightforward retail DVD ripping question.I'm in the process of converting a wide range of vhs tapes and home made DVD-R's. -
Originally Posted by vaporeon800
The thread I linked? That was just for jagabo and others interested in the topic he brought up. You can safely ignore it.Originally Posted by mrwhitethc
Would you mind changing the thread title of your original post to say something like VHS-DVDR or similar instead of just DVD? I initially ignored the thread since my interest is mainly tape conversion; from the title I assumed it was just a straightforward retail DVD ripping question.I'm in the process of converting a wide range of vhs tapes and home made DVD-R's.
Gotcha.
I see how that can be confusing, I've updated the title. -
You need to consider what parts of the picture matter. Here's one frame where ColorYUV(analyze=true) shows max luma 255:Originally Posted by mrwhitethc
[Attachment 51984 - Click to enlarge]
Which pixels are at 255? Let's use mt_binarize(254, chroma="-128") to find out:
[Attachment 51985 - Click to enlarge]
Pixels with values 254 and below are black, pixels with value 255 and above (but nothing can be above 255) are white. The single white pixel is at the bottom of the frame, in the head switching noise. Ok, so which pixels are above 235 (the max legal Y value)?
[Attachment 51986 - Click to enlarge]
Aside from the head switching noise at the bottom of the frame there's only one pixel above 235. Which pixels are above 200?
[Attachment 51987 - Click to enlarge]
But even those are all oversharpening halos. It's not until you get down to ~190 that you start seeing real picture details turn white.
Let's look at another frame with Histogram() (which, again, is really a waveform monitor) using TurnRight().Histogram().TurnLeft():
[Attachment 51988 - Click to enlarge]
The white bar near the top of the waveform monitor is from the bright sky. The Y values there are around 200 plus or minus a few units. If you scrub through the video you'll see that the big bright areas are all around that level. So that's close to the actual white level -- which would normally be at 235, the bottom of the top brown bar. The black levels look about right, very close to the top of the bottom brown bar.
Unless you plan to go through the video and adjust levels shot by shot you should look for a compromise setting that keeps the black and white levels close to the expected 16 and 235 values. This is where you have to use your own judgement. Auto levels adjustment filters don't work very well.
Beyond that have have things like hue, saturation, white balance, gamma etc. And just because all Y values fall between 16 and 235, and all U and V values fall between 16 and 240 doesn't mean every pixel is legal. For example, at Y=16 (full black) there is only one legal value for U and V, 128. The same is true at Y=235 (full white). In fact, most YUV combinations between the stated min/max ranges are illegal. Here's a picture of the valid RGB values within the YUV (YCbCr) cube:
(from https://software.intel.com/en-us/node/503873, "RGB Colors Cube in the YCbCr Space")
You can that most YUV values are outside the inner RGB cube. There are filters that can test for this too. -
Thank you everyone for the help, after looking at the raw numbers and waveforms from ColorYUV, Histogram and mt_binarize, a bunch of many test encodes and finally looking at the playback on a few TV's and my monitors at home I've come up with:
I left off convertToYV12(interlaced=false, matrix="Rec601") since all the footage is in YV12 to begin with and Info() confirms it as well even after QTGMC to lossless AVI.Code:ColorYUV(gain_y=2.5,off_y=5,off_u=2, off_v=-2).Levels(0,1.06,255,3,235,coring=false,dither=true)
This seems to give me just the right amount of blacks and brightness to the image to make out what the hell is going on during the night scenes along with fix the small color shift. Thanks to everyone who took the time, I now have a new obsession when making my transfers to look over
Last edited by mrwhitethc; 18th Feb 2020 at 21:16.
Quote
Similar Threads
-
color correct megui
By nomaher in forum Blu-ray RippingReplies: 2Last Post: 31st Mar 2018, 16:27 -
How to color correct a video like this?
By Hypersonic1 in forum EditingReplies: 29Last Post: 14th Dec 2017, 20:38 -
How the "see" correct BT.709 color using VLC in a PC?
By marcorocchini in forum Newbie / General discussionsReplies: 2Last Post: 25th Jun 2017, 10:57 -
x264 conversion with Virtualdub/sony vegas 13 Color issues
By MajorFoley in forum Video ConversionReplies: 23Last Post: 20th Dec 2016, 07:11 -
Sony Vegas x264wfv output changes red color
By LeoKac in forum Video ConversionReplies: 2Last Post: 25th Mar 2015, 09:17