Windows Color Profiles ignored by TMPGenc, VirtualDub, etc. ?

[sanlyn]

After a month of re-calibrating three monitors, testing DVD renders, watching retail and home-made DVDs, VCR recordings, etc., etc., my eyes and a number of graphics plug-ins are telling me that the following applications pay no attention whatever to calibration software from Colorvision, Samsung, Sony, etc.:

TMPGenc Plus 2.5
VirtualDub (any version)
Windows Media Player (any version)
PowerDVD player (any version)
Sony Vegas

Hopefully someone has more info on this matter, but my eyes and my DVD's tell me the above applications pay no attention to Windows color profiles (.icm). My experience is:

VirtualDub - Too dark, definitely too red
PowerDVD - default settings = too much contrast, pre-set choices = color off the wall.
Windows Media Player - too bright, too green (on all 3 of my monitors)
TMPGenc Plus 2.5 - too dark, too much red, blue looks weird
SONY Vegas - image looks different every time it's opened.

I use Photoshop to test color casts, RGB levels, whites, blacks, grays. If I go by screen captures in Photoshop and correct color using its controls, I mimick those controls in various graphics programs and use Photoshop's corrections despite what the above applications are telling me. The result is that my DVD's have vastly improved
in contrast and color, even in b+w. I tried to remove a color cast from a B&W recording using what my eyes told me in TMPGenc, VirtualDub, etc., and the results had color casts noted above. Photoshop tells me no color cast existed in the original AVI recording, so I applied no corrections and made a DVD. Result: Photoshop was correct.

[jagabo]

Virtually all media players use video overlay which I assume doesn't pay any attention to color profiles because overlay has it's own brightness, contrast, etc. controls.

With editors the screen output is generally via Windows' GDI but you will have to be careful about the colorspace of the source video and the matrix used to convert it to RGB for display. My guess: if you feed VirtualDub or TMPGEnc Plus an uncompressed RGB video (thereby avoiding any YUV to RGB conversion) it will appear exactly as you expect.

[sanlyn]

Yup, jugabo, I capture in huffyuv compressed RGB to AVI, but as soon as I start any color work in VirtualDub or TMPGenc I'm working with uncompressed clips from the original AVI capture (i,.e., I get a short scene from the huff AVI, cut it out with Vdub and save it as uncompressed AVI in 24bit RGB). Ultimately TMPGenc gets an uncompressed clip to work into MPEG, the very last step.

I find that pretty much ignoring what all these editors tell me visually is working OK, if I keep capturing images and throwing them into Photoshop. The actual color and levels, etc., is done in Photoshop and I use Adobe's numbers and histograms to feed VirtualDub filters (sometimes it's a guess; is 10 points of green in Adobe the same as 10 points in an RGB or gradation filter? Only approximately, I've learned, but I seem to be getting it pretty close). It's working OK so far. But making a final judgement from what my eyes alone tell me in VirtualDub, etc., usually leads me astray. I've been trusting Photoshop's numbers and previews more than the other apps, and it seems to work much better.

I never really know until I get a few minutes of uncorrected video onto a DVD and watch on my tv's. It's a real pain but, after all, previously I had to re-correct and re-burn those DVD tests anyway. But now it's more predictable, with fewer reworks.

I've set Photoshop to "see" thru my calibration profiles. Amazingly, many clips that look dreadful improve miraculously just by increasing overall gamma a bit and taming the very darks and very brights with Adobe. Often, after that I don't even have to fiddle with individual colors. It's amazing what a correct gamma setting can do.

VHS color was, and still is, such a drag. Back in 1996 I thought it was spectacular on some players. Thank the gods for for this forum and the bright people in it.

[jagabo]

making a final judgement from what my eyes alone tell me in VirtualDub, etc., usually leads me astray... I never really know until I get a few minutes of uncorrected video onto a DVD and watch on my tv's.

Therein lies your problem. TVs have very different brightess, contrast, and especially gamma than computer monitors. You need to either calibrate your video overlay so that video on the monitor (as displayed by a media player using overlay) looks like TV (you don't want to calibrate your desktop this way or the rest of Windows will look like crap), or you need to have a TV hooked up to your computer (and calibrated) via s-video to see the result.

[sanlyn]

I used Spyder2 to calibrate my pc monitors and spyderTV to check my alread-calibrated tv's. I was skeptical of the results I might get, but now I'm still amazed at how easy it was to match pc-to-tv. Obviously the two types of monitors don't "look alike", but Colorvision's gear has paid off with astounding results. I'll never go back to test patterns by eye again, that's for sure.

[edDV]

Monitoring off the display card is difficult unless the monitor and edit programs support color profiles (ICM, etc.). Overlay settings from ATI and NVidia are difficult for video setup since they are working in RGB space.

Premiere, Vegas, etc. allow monitoring via the IEEE-1394 connector keeping the full path YCbCr from capture to monitoring. You then use a camcorder or DV transcoder to S-Video or analog component for TV connection. The TV gets calibrated via SMPTE/BellNuit/THX/etc. test patterns for levels, gamma and color balance. This works like a charm for DV/HDV/YCbCr workflows.

Vegas 6 added DVI/HDMI monitoring using color profiles provided by broadcast standard LCD monitors. Premiere and Vegas also support SDI monitoring via AJA and BlackMagic YCbCr SDI I/O cards.

[sanlyn]

edDv, camcorders and DV are out of the picture. The sources are old (and often damaged) VHS tapes with material that isn't available elsewhere (I've replaced many of my old VHS issues with new commercial DVD's). I realize what you're saying about YUV -vs- RGB, but the very effective filters I use work only in RGB, and so do my encoders. Anyway, I don't own a camcorder and wouldn't tolerate a cheap one.

My two tv's were calibrated professionally with internal PCB card mods and other adjustments by a video engineer at WLIW in Roslyn, NY (and at a healthy price, too, not to mention voiding the warranties!). I now use Colorvision software and gear to calibrate PC monitors and to keep tabs on how close my tv calibrations have stayed. True, my ATI graphics cards can display to tv, but I've never found it to be worth the trouble and I'd have to keep re-setting my tv picture controls to correct for what the ATI cards are doing (time-consuming pain the neck).

With the help of Colorvision, Photoshop, and some VDub filters, I've pretty much solved my old color balance and IRE problems. But, yes, saturation levels and midtones look entirely different between PC and TV, and of course the RGB 16-235 correction makes everything look "thinner" on the PC. But it's now reached the point where saturation and midtones are just about the only corrections I've needed after viewing the test DVD. Often I need no corrections at all.

One question, though, edDV: I'm setting NTSC levels on my video work in my PC, and MPEG's for DVD play back with grayed blacks. But if I play a retail DVD on my PC's DVD drive thru PowerDVD, the darks and brights look normal (retail DVD's look too bright thru Windows Media Player). So I'm wondering if PowerDVD is making some IRE corrections when playing commercial DVD's, but not when playing MPEGs from my TMPGenc and CCE encoders.

Something amazing: I transfer many of my DVD recordings off cable to my PC to make custom menus and remove commercials. Cable is broadcasting stuff that is so far out of whack with NTSC luma and RGB levels! Thru histograms I see blacks smashed against the left edge of the histogram (and they look pretty grimy on my TV, too). I have a good cable amp on that line, so the signal's not underpowered. Couple of days ago I cleaned up a piece of recorded video using my new methods, and they play a lot cleaner now.

[jagabo]

Most Windows programs, including VirtualDub and TMGPEnc use a rec.601 matrix when converting RGB to YUV and vice versa. This means that 16-235 Y is expanded to 0-255 in RGB. Conversely, 0-255 RGB is compressed to 16-235 Y.

TMPGEnc Plus and CCE both have the option of not compressing the luma range when converting RGB to YUV for encoding.

You can use AviSynth to open YUV video sources and convert to RGB without the luma expansion using ConvertToRGB(matrix="PC.601").

[sanlyn]

Thanks, jagabo, I figured the players and VirtualDub-MPEG (for converting recorded MPEG(DVD) to AVI) were playing tricks in the background. The captures from VHS that I'm woprking with were made with VirtualDub directly to RGB (the color space I'm using for the capture is UYVY for Huff, though sometimes I use YUY2. I haven't noticed any difference between them once I start using those AVI's in VirtualDub).

But I thought TMPGenc Plus 2,5 encoders render to MPEG using CCIR.601, not REC.601. TMPGenc has an option to "Output YUV data as Basic YCbCr not CCIR601". I leave that option unchecked and let TMPGenc use CCIR601. I used to know what the difference was between "CCIR" and "REC", but that was so long ago I don't remember. CCE Basic has a "16-235" switch, which I usually use, but I don't know which "601" CCE is using for the render.

I've often used AviSynth's "PC->TV" command to convert 0-255 RGB to 16-235 RGB. But I doubt its usefulness with TMPGenc or CCE, because using AviSynth's PC->TV and then feeding it to TMPGenc or CCE just gave me a lame video that had the same washed-out look that I saw on my PC.

I think the problem with AviSynth was that the conversion commands involve a color space change that sorta mucks things up. The Avisynth commands are:

ConvertToYUY2(interlaced=true)
ColorYUV(levels="PC->TV")

TMPGenc will convert that color space to its own RGB space, then reconvert it to YUY2 again for DVD. So AVI's that I load into TMPGenc after using the above commands have some "dirty" color afterward. If you use that same clip in CCE, you can't click CCE's "16-235" switch on because you'll end up with IRE 15 or so.

In any case, I begin by avoiding crushed blacks and blown-up whites to begin with. Works OK so far.

[edDV]

Where to start? I guess we first need to agree on goals.

The goal of broadcast production is to output a standardized product that can be loaded and played to air as hundreds of clips with no black shifts (level 16), consistent gamma and whites that won't peg the the transmitter (235 nominal with small energy spikes up to 255 max).

Ad agencies are more anal with demands for product color match to an approximation of Pantone standards used in print*. These arguments can get heated but the first step is to agree on a evaluation monitor (usually a factory calibrated Sony BVM that is likely to be found in both production houses and TV stations).

From the home theater perspective, basic need is all sources to the display should produce acceptable levels and color match without adjusting the display. Some sophisticated TV sets have separate calibration adjustments for each input. Most have one overall setting requiring levels adjustment at each source.

OTA/Cable/Sat tuners - these usually have fixed level 7.5-100 IRE analog or 16-235 digital outputs (no user adjustments) so the TV display is usually adjusted to the tuner.

DVD players - good ones have user preset levels adjustments. In this case the DVD player is adjusted to match the tuner.

VCR - most output close to 7.5-100 IRE** from a factory tape. If they don't, you need to make internal adjustments or invest in a proc amp.

DV/HDV camcorders - unfortunately for the North American user, these output Japanese/Euro 0-100 IRE (for 16-235 digital levels) that do require a custom levels setting at the display.

Computer display card - Usually overlay settings are tweaked to best match the tuner reference path. I find it difficult to get a match.

Computer IEEE-1394 out via DV transcoder (e.g. Canopus ADVC, ADS Pyro, etc.) - If these can be set to 7.5-100 IRE out they should closely match the tuner reference path.


If you have a pro DV/HDV software editor with waveform monitor (e.g. FCP, Premiere, Vegas), you can get tighter levels match using the DV standard (SMPTE or Bell Nuit) test patterns to burn a test DVD, DV tape or VHS tape that can be looped to input capture to verify levels match on the scope. The idea is to get a tight calibration at the computer I/O that matches at the editor timeline and reference display.


* Pantone has qualified video color match to 24 bit RGB and YCbCr (4:4:4) but NTSC, 4:2:0 and even 4:2:2 have a more limited color space. This often results in a signoff on the 4:2:2 master followed by complaints when viewed on-air over NTSC or 4:2:0 MPeg2.

** VHS has an even more constained color space due to 0.5MHz IQ (UV) chroma component bandwidth vs. 1.3/0.5 MHz over air.
http://en.wikipedia.org/wiki/YIQ

[sanlyn]

edDV, the problem always discussed in these threads about IRE/RGB etc., is video that has ben captured at too bright an IRE level. I have two proc amps I use during capture to avoid luma and RGB level problems. I usually use the SignVideo PA-1 for its level meters.

But what about cases where a proc amp introduces severe problems with Macrovision on VHS tapes? I have a handful of VHS tapes with copy protection signals that really screw up the first half of the movie. A proc amp inserted anywhere into the playback/record circuit places highly visible gray bars at the bottom and right of the image. The semi-transparent bar is about 32 pixels high at the bottom and 8 pixels wide at the right edge. The bars gradually fade during the recording until they disappear or become too faint to notice after 30 to 40 minutes.

For these sources I have to make 2 captures, one with a proc amp and one without. The capture made without a proc amp is recorded with crushed darks and blown-out whites. So the first 30 or 40 minutes of captured video needs correction, especially at the black levels. I know of no easy way to raise darks at RGB 10 or 8 or 4 to RGB 16. No matter what color space is used for capture or processing, the original uncorrected capture is RGB 0-255. Every attempt to rehabilitate the video to 16-235 simply destroys contrast; the result is washed out on the PC and looks just as bad on tv. DVD's made from these captures play just as badly on the TV as they do on the PC.

So I've had to do some fancy acrobatics with VirtualDub and Avisynth filters. The least useful filter is VirtualDub's histogram (levels) filter. It's not possible to use this filter to raise darks and lower whites; all it does is destroy both extremes. VirtualDub's coring and "TV" filters are just as destructive.

Far more useful is the newer version of VirtualDub's gradation curves filter. You can do some fancy footwork with curves; raise midtones without affecting darks and brights, work only with darks without affecting midtones, etc. But all this work and angst isn't needed on the latter half of the video, because Macrovision effects from the proc amps aren't present. That part of the capture has correct luma and color levels; color cast and other corrections can be made without drastic means, and final contrast, etc., looks "normal" on both the PC and TV.

But how the heck do you avoid the gray bars due to Macrovision effects? I've recorded these tapes with and without proc amps, with and without full-frame TBC's (and, wow, you should see the disturbances when Microvision isn't excluded!!! Completely useless video). Four different VCR's play the tape, including a JVC with its onboard TBC turned on and off (the external TBC is still required to remove Macrovision), etc., etc., ad nauseam.

The minute you insert a proc amp, however, the gray bars appear.

There oughtta be an easier way ! !

[jagabo]

But I thought TMPGenc Plus 2,5 encoders render to MPEG using CCIR.601, not REC.601.

The RGB/YUV conversion matrices are the same.

I've often used AviSynth's "PC->TV" command to convert 0-255 RGB to 16-235 RGB. But I doubt its usefulness with TMPGenc or CCE, because using AviSynth's PC->TV and then feeding it to TMPGenc or CCE just gave me a lame video that had the same washed-out look that I saw on my PC.

I think the problem with AviSynth was that the conversion commands involve a color space change that sorta mucks things up. The Avisynth commands are:

ConvertToYUY2(interlaced=true)
ColorYUV(levels="PC->TV")

ConvertToYUY2() by default compresses the luminance range when going from RGB to YUV. ColorYUV(levels="PC->TV") will do it again. This gives you two luminance range compressions. RGB 0-255 will become Y 32-216. You definitely don't want to do this.

In any case, I begin by avoiding crushed blacks and blown-up whites to begin with. Works OK so far.

You've already crushed blacks (below digital IRE 0) and blown out whites (over digital IRE 100) by capturing with VirtualDub as RGB. No capture card captures as RGB. They all capture in YUV. If you specify RGB in VirtualDub it (or the capture driver) will convert the YUV to RGB using the rec.601 matrix. Y 16-235 will become RGB 0-255. Y below 16 will all be RGB 0, Y above 235 will all be RGB 255. Any detail in those areas will be lost.

[edDV]

You have Vegas so I'll show you how I use Vegas with Virtualdub and avisynth.
I always test my capture chain with test signals before I trust it. A color bar like the
BelleNuit evaluated on a waveform monitor quickly exposes problems with YCbCr to RGB
conversion.

YCbCr Broadcast and VHS-DVD source captures usually scale mostly within the 16-235 levels
with some peaking into 236-255. Some older cable systems manage black poorly with about +/-
3 IRE around 7.5 IRE (digital 16) as you change channels. Prosumer camcorders capture with
similar levels but consumer camcorders typically use all of 16-255 levels with some clipping
at 255. Here is a typical low end Digital8 with a high contrast scene. Consumer 8mm, Hi8,
MiniDV, MiniDVD and VHSC camcorders will look similar.



Prosumer camcorders such as this Hi8 Sony V5000 managed black and white levels better.
This Hi8 tape was captured from a Digital8 camcorder to DV.



Vegas renders to so called "studio RGB" which preserves the 0-15 and 236-255 overshoot
levels. If you cap the BelleNuit color pattern, the luminance overshoot regions are
preserved at the timeline and after MPeg2 encoding.



If you cap DV to Virtualdub RGB using the Panasonic DV codec, your RGB will be expanded in
contrast and overshoots will be clipped. This is what happens when DV is capped to
Virtualdub using the Panasonic codec. The resulting 0-255 RGB output was imported to
Vegas and put on the scope.



If you output the above from Virtualdub back to DV using the Panasonic DV codec, the
resulting waveform looks like this. Note white highlights and sub-blacks are clipped.



To compare, this shows a post MPeg2 encode from Vegas where the test slide was capped to a
DV project from DV tape, then output to Virtualdub as uncompressed RGB*, then output from
Virtualdub as RGB back to import on the Vegas DV timeline and then encoded to DVD MPeg2. As
you can see the overshoot regions were preserved and the waveform quality was maintained
with only a few edge errors. I use this method to access Virtualdub and AVISynth RGB filters
from the Vegas timeline.

This method uses the Sony DV codec (inside Vegas) for all RGB conversions.




* I repeated this test exporting huffyuv from Vegas to Virtualdub and back from Virtualdub to Vegas import as huffyuv and the quality was similar.

[sanlyn]

jagabo: thanks for catching my slip-up on the original VirtualDub capture. I just realized I had VirtualDub's capture set to RGB24 instead of YUV2 thru huffyuv. I also had all 3 copies of NeatVideo set to work in RGB instead of NV's default YUV color space (!@#$%). After I made those corrections, I took your advice and tried AviSynth's rec.601 conversion again when opening in VirtualDub for correction, and this time it worked (opening via AviSynth did move my NeatVideo speed from 3.2 fps to 2.8 fps, but the results were worth it). I avoided further problems by capturing with huffyuv but working scene-by-scene with uncompressed AVI (Sorry, I have only 3 320-GB hard drives, so making the original capture as uncompressed just uses up too much space!)

Big problem now: I have to throw out the original capture and record those three tapes again. Shucks.

edDV: a partial solution to the problem of Macrovision causing gray bars along two borders on VHS captures:

The main reason I need a line-level TBC somewhere in the capture circuit is that many old VHS tapes have chroma smearing and wiggly verticals on playback, and occasionally some top-of-frame tearing; an outboard full-frame TBC corrects a few errors, but won't help wiggles, edged smearing, or some types of tearing.

So, to repeat the circuit I used for capturing these "problem tapes": VCR -> Toshiba RD-XS34 (as a pass-thru line-level TBC and mild 3d-Y/C noise filter) -> outboard full-frame TBC -> proc amp -> VirtualDub.

I had been avoiding gray bars by leaving the proc amp out of the circuit. But this gave me badly crushed darks and blown-out brights because of luma expansion on playback and the tendency of Panasonic and SONY vcr's to overly juice up contrast on playback.

Solution: keep the proc amp, but leave out the frame-level TBC. I had always assumed that the outboard TBC destroyed Macrovision and made the VHS recordable. But VirtualDub is capturing these copy-protected tapes anyway! I still find that Macrovision won't let me record directly to ATI MMC or a DVD recorder without an outboard TBC. But it works in VirtualDub.

Once I make a capture without the full-frame TBC, I do see some results of bypassing Macrovision in this way; the color balance is screwed up either heavily red or heavily green, and brights and midrange are depressed. The proc amp's color controls can't be used, becaue you don't see those effects until AFTER the capture and they're totally unpredictable. But these are easily corrected with level or gradation filters, no problem. Anyway, Macrovision effects apparently fade away after the first half of tape or so. Using the SignVideo proc amp's luma meters is keeping me well within the correct IRE and 16-235 RGB range without wiping out detail in darks and brights -- which was the major problem to begin with.

It's possible that Toshiba's line TBC is helping to cure some problems from Macrovision effects, including jitter and tearing. Also, verticals are straight, chroma smearing is often invisible, noisy high-contrast edges are greatly improved, etc., and there seem to be far fewer color fluctuations and streaking within a scene.

The RD-XS34 has IRE and brightness adjustments that work on pass-thru, but setting them to other than "Normal" produced extremely washed-out or dark, grimy images (an IRE change that's really too tough to cope with). So I'm leaving it at Normal.

Why didn't I just use a VCR with a built-in line TBC? I did -- but Macrovision was giving me gray bars on those tapes before the image even left the VCR; I can see them on playback connected directly with a single wire from VCR to TV. I don't see them using RF cable or with the VCR's TBC turned off -- but I think you know what an image looks like thru RF connections(!), and turning off TBC defeats the purpose of using that VCR, and one of those damaged tapes won't play at all on that VCR.

I don't know how many users have encountered these Macrovision annoyances on some (but not all) VHS tapes. But maybe this info will be useful for those who have these problems when capturing VHS to their PC.