x264 or other option for encoding Fraps output with no chroma subsampling?

[Cogitation]

I've got about 80 GB of 1920x1200, 60fps avi's from output of Fraps. I've been using x264 to encode them, and am disappointed with the large, visible reduction in color resolution that happens automatically after conversion to YV12 (which x264 requires, though the h264 standard does not.) Does anyone know of an x264 branch or patch that enables encoding without chroma subsampling or the high 4:4:4 predictive modes of the h264 standard? Or just as good, some encoder for theora, dirac/schrodinger or wmv9/vc1 that will do the same? I'm not interested in stand-alone compatibility or uploading to youtube... I want to encode the files for my own use and maybe to send to a few other people. I've been looking for a while and I can't find squat. I think Main Concept's pro level encoder would do it, but it's only like $4 grand, and requires Adobe CS4, which is like another $2 grand or so, right? Another thought I had, is that I could encode the files at 3840x2400, so (one might hope) no actual subsampling of color information occured, but not only would this be massive massive size overkill, it won't decode fast enough to play back smoothly.

[poisondeathray]

Nope, most compressed formats are planar 4:2:0. x264 does not support anything by 4:2:0 at this time. Most decoders don't support AVC High 422 or 444 even if you encoded it.

You could try MPEG2 4:2:2 , but MPEG2 compression isn't that great, and you still have sampling. At least it offers long GOP

DNxHD (free) has a 8-bit and 10-bit 4:2:2 mode, and should be able to play in real time, but high data rates. I-frame only

UT Video Codec (free) is lossless and has a RGB444 mode, but high data rates, and you need RAID0 array or SSD's to play in realtime. I frame only.

Morgan MJPEG2000 (not free), has a 4:4:4 mode, and variable levels of compression , you can try that out. It's I-frame only.

Eitherway, it takes a lot more bitrate with no chroma sub sampling and no long GOP

[Cogitation]

About what I figured, but thanks for the quick reply with good information.

[Cogitation]

Another question along the same vein (color/brightness distortion after encoding from Fraps output/FPS1 to x264) is do any decoders actually support --fullrange? I can't seem to even preserve the full 255 shades of brightness. Which really hurts, because a lot of the video is very dark, and everything gets shoved down to almost black. I think I might be doing something wrong here, but I haven't been able to figure it out yet.

[poisondeathray]

The full range flag may or may not be ignored by your software player and/or decoder

What you're describing is more often a playback issue with levels (PC vs TV), and there are number of factors that might be involved including graphics card settings, renderer, software playback settings, monitor calibration.

[nonoitall]

Actually a little while back there was some talk in the developers' circle about making x264 work without chroma subsampling. I don't know how much (if any) progress has been made toward that end, or when/if the feature will ever be available, but there's at least been some discussion about it. (Of course, since it obviously isn't implemented yet, this is just FYI.) Would definitely be a nice feature for recording pre-3D game footage.

[Cogitation]

The full range flag may or may not be ignored by your software player and/or decoder

What you're describing is more often a playback issue with levels (PC vs TV), and there are number of factors that might be involved including graphics card settings, renderer, software playback settings, monitor calibration.

Well, I do know about PC versus TV levels though. Using ffdshow, one of the typical filters I have constantly on is the levels switch.

But that's not what I meant. I meant, whenever I encode with x264, even with a --fullrange flag on, the output is truncated (maybe compressed, but it just looks truncated to me) to that 12-235 range. So even if I convert TV to PC levels, and darken what is now shade 12 gray to shade 0 black... I've lost all the shadow detail that used to be between level 0 and level 12, and everything is *way* too dark. To the point that what was easy to see in the original fraps output (if dim) is close to black now and impossible to see.

I've tried using ffdshow and coreavc, with multiple render engines in MPC-HC and also tried in VLC. They all look like they're ignoring the fullrange flag (or it's not happening in the first place.)

Also, just to be circumspect, I did go check my Catalyst Control Center settings. Everything is set to "application preference" and all video affecting settings like denoise and dynamic contrast are turned off.

This is what originally made me want to find out if there's a way to encode in RGB or YUV with no subsampling and fullrange, in fact.

@nonoitall:

I thought I remembered seeing that, but a few pokes around Google didn't find anything. I thought I might ask on a forum and see if anyone knew anything. Thanks.

[poisondeathray]

The fullrange flag doesn't do anything really. It's like metadata. It's up to the software to ignore or accept it (I don't know of any that accept it)

what matrix are you using for RGB=>YV12 conversion?

If you just specify ConvertToYV12() , avisynth will assume Rec.601 (16-235)

If you use PC.709 for HD content it will be BT.709 with 0-255

The renderer make a huge difference on how your video appears. e.g. Try switching from haali , to overlay, evr, or vmr9/7 renderless (close & restart your player each time). Big difference between each

[nonoitall]

@nonoitall:

I thought I remembered seeing that, but a few pokes around Google didn't find anything. I thought I might ask on a forum and see if anyone knew anything. Thanks.

Yeah, no problem. Here's the thread on Doom9 where I saw it mentioned (see Dark Shikari's post further down).

[Cogitation]

poisondeathray... check out nonoitall's link there, post #20. That's exactly what I'm talking about. It doesn't just reduce chroma resolution, it's also screwing up the colors. The first one is the original. The second, everything is too dark. And the third, everything is washed out and black is actually dark gray. I've had both.

Just for curiosity's sake, I tried recording with fraps at half resolution, so it came out 960x600, then I encoded with x264 at 1920x1200. It's playable, but the color and black level distortions are still happening. It did help me realize that the actual subsampling (at least on my high resolution source; it's probably much worse on lower resolution screen captures) isn't that big an impact on the picture, it's just the poor conversion from RGB to YV12. Either that or encoding at 1920x1200 isn't accomplishing what I think it is.

To answer your question about the color matrix: This is 1920x1200 60fps source; my understanding was that BT.701 is automatically assumed for anything over 1280 width resolution. Of course my evidence for that would be "I read it somewhere... can't remember" So...

About trying other renderers, I did already say: "I've tried using ffdshow and coreavc, with multiple render engines in MPC-HC and also tried in VLC. They all look like they're ignoring the fullrange flag (or it's not happening in the first place.)"

[poisondeathray]

check out nonoitall's link there, post #20. That's exactly what I'm talking about. It doesn't just reduce chroma resolution, it's also screwing up the colors. The first one is the original. The second, everything is too dark. And the third, everything is washed out and black is actually dark gray. I've had both.

His comparison might problematic. How did he take screenshots? For example, you cannot take screenshots in overlay renderer mode.

The chroma subsampling exists. But that manifests as blocky colors on edges - they aren't as crisp , but subsampling should have NO effect on the brightness/contrast. That issue is either with encoding or decoding levels, or software display (renderer or graphics driver)

To answer your question about the color matrix: This is 1920x1200 60fps source; my understanding was that BT.701 is automatically assumed for anything over 1280 width resolution. Of course my evidence for that would be "I read it somewhere... can't remember" So...

For DEcoding with some graphics drivers, and settings, not necessarily with ENcoding.

Are you encoding directly to x264, or with avisynth? If you do it in avisynth, you have to specfiy the matrix, otherwise it assumes BT.601 16-235 (i.e. Rec.601)



I'm going to demonstrate that what you're experiencing is likely a playback or configuration issue. I have captured Fraps (in RGB Mode) of a game. The avs script was just ConvertToYV12() , no matrix specified. I've included screenshots in the zip file below

1) Original RGB Fraps - this will be what you compare to. Notice how clean the edges are

2) ConvertToYV12 - this is .avs script before being fed to x264 , taken in AvsP, so without the effects of renderer. Note, the luminance doesn't change, only the choma sampling and color edges are less distinct

3) x264 - this is taken from the x264 encode, crf16 --slower in AvsP (again eliminated inconsistency between renderer issues). Script was directshowsource(), haali media splitter, ffdshow default settings. Notice how it looks almost identical to (2) above, same chroma sampling, but no levels shift

4) MPC vmr9 renderless (viewing the x264 encode) - haali splitter, ffdshow same settings, but vmr9 renderless renderer - notice how levels are changed. Note you could use a different renderer and get different results, but the point is the underlying video is the same, just how it's viewed is different.

5) flv player (viewing the x264 encode) - taken through applian flv player. Notice levels are the same as the original RGB, but the chroma upsampling from Y'CbCr=>RGB is lower quality (you can see blockier edges in the reds for example), this uses the same mechanism as Adobe flash player, youtube etc... Because this doesn't use your graphics card overlay or other configuration which may be different between systems, it's pretty standardized/consistent (same decoder and display mechanism) and thus should look the same between computers.

In other software, there are higher quality chroma upsampling mechanisms (for Y'CbCr=>RGB display) so you don't get as blocky color edges e.g. ffdshow has one for decoding, madVR for the renderer is another (see Doom9 thread) - but these are done through directshow, not what flash uses

To summarize, what you have is likely a configuration / playback problem. The issue is not with x264. Play your x264 encoded file in a FLV player or a flash application, and proove this for yourself.

[nonoitall]

Yeah, when you take RGB material and convert it to YV12, YUV2 or YV24, you're always going to see two colorspace conversions: The one that takes place when you convert it in the first place, and the one that takes place when you're playing back that material on a (RGB) monitor. If the two methods use different ranges, you'll end up with too little or too much contrast when you view the encoded material. This is a completely separate issue from chroma subsampling.

[Cogitation]

+1 to poisondeathray's rep. That was a helpful and informative post. I *was* using MadVR, and was still having the problem. I've since deleted those encodes, but it must have been encoding in BT.601 and then getting decoded with 701. Fixed by learning a bit more about avisynth instead of using HDConvertToX's standard settings. The chroma subsampling is still visible, but dark scenes at least retain shadow details and contrast.