YUY2 vs RGB24 vs Y800

[201flyer]

I'm not at all sure where I should post this problem, and I'm not sure if I will find anyone who knows the answer to this question as the topic is geared towards Astrophotography.

The issue is that using my CCD camera - Image Source DFK31AU03.AS - there are several choices in the method of capturing avi files. My question is why the color looks so different on playback, particularly when I capture in YUY2? If I capture in RGB24 or Y800 (and then use software to Debayer) the colors look very similar. But with all the same settings, capturing in YUY2 looks quite odd. I took some shots through my telescope of a distant tree to illustrate. The top photo is Y800 and looks correct (after Debayering). The bottom photo is YUY2 and has a purple cast. One thing I didn't try (and should have perhaps) was to change the white balance after changing codecs, but I can't see why that ought to be necessary. Any ideas are greatly appreciated!

[jagabo]

Try adjusting your graphics card's video processing amp (where the YUY2 to RGB conversion is taking place).

Shouldn't Y800 be grayscale?

[edDV]

The Image Source DFK31AU03.AS is an astronomy camera. The sensor is a Sony ICX204AK single 1/3" 1024 (H) × 768 (V) effective square pixel CCD. Max frame rate seems to be 15fps.

http://www.astronomycameras.com/en/products/usb-cameras/colorir/dfk31au03as/
http://www.theimagingsource.com/downloads/icx204ak.en_US.pdf

Bayer pattern favors green with red and blue half sampled H and V as shown here.



The DeBayer filter is probably optimized for RGB24 out. Suggest you check with Astronomy forums.

[201flyer]

Try adjusting your graphics card's video processing amp (where the YUY2 to RGB conversion is taking place).

Shouldn't Y800 be grayscale?

The Y800 is grayscale when playing back the .avi. In this case, however, I have Debayered it after capture to restore the color. It looks identical to the BY8/RGB24 color.

Would you mind elaborating on your first sentence, please. My monitor is an ATI Radeon X600 SE. I looked at all the settings and could find nothing about YUY2. The hardware acceleration is on full.

Thanks!

[jagabo]

The Y800 is grayscale when playing back the .avi. In this case, however, I have Debayered it after capture to restore the color. It looks identical to the BY8/RGB24 color.

Ah, I see. Y800 is a raw dump of the camera's CCD sensors with 8 bits per primary.

Would you mind elaborating on your first sentence, please. My monitor is an ATI Radeon X600 SE. I looked at all the settings and could find nothing about YUY2. The hardware acceleration is on full.

Use ATI's CCC to adjust the video color settings. Not the Desktop color settings, the video color settings. I don't
have a computer with an ATI card right now so I can't give you explicit instructions. But you'll see two areas for adjusting on-screen colors (Desktop, video). There won't be mention of YUY2, just settings for brightness, contrast, hue, saturation, etc.

If you upload a few small clips I'll verify that they should look the same.

There are some other possible issues like ITU Rec vs PC contrast stretch, and 709 vs 601 color.
https://forum.videohelp.com/threads/329866-incorrect-collor-display-in-video-playback?p...=1#post2045481
But the differences aren't as large as in your sample images. So I suspect your major problem is the video proc amp settings.

I just noticed that the framing and lighting of the two sample images wasn't the same. Are you sure the colors are supposed to be the same?

[201flyer]

Hi Jagabo,

Many thanks for your help with this! I really appreciate it!

OK, the photos posted are aligned/stacked frames of the video using a program called Registax. The tree was actually dancing around quite a bit during the capture which is why the framing isn't exactly the same.....but I shot several videos under the exact same conditions so I would expect the colors to look the same. But, I have to say I just noticed something rather obvious that I overlooked. The camera shot the YUY2 video at 30fps (which is a default setting), but the capture of BY8 defaulted to 15fps. Exposure, however, was on automatic, but maybe this accounts for the different color. Not sure. Perhaps I should repeat the test with everything at 15 fps.

Anyway, two small samples attached.

EDIT: I ran the test again.....15 fps for both captures. The colors remain decidedly different. Everything was identical for both captures except one was YUY2, the other BY8 - both to codec RGB24. I looked at both captures in Mediainfo. Acording to the capture program I captured the same number of frames, and captured both videos for 30 seconds. Yet, Mediainfo reports that YUY2 was for 30 sec at a bit rate of 116Mbps, while the BY8 capture was for 12 sec, 56ms, at a bit rate of 283Mbps. Now I'm more confused than ever. If the bit rate of the BY8 capture is more than 2x the bit rate of the YUY2 I guess it isn't surprising to see "differences."

[jagabo]

The differences between the two aren't just a normal YUY2 conversion issue. The colors ARE very different.

Normally, converting your RGB to YUY2 would cause a barely detectable difference (very slightly blurred colors and a tiny loss of precision). There must be something else going wrong. How much processing is going on here? The YUY2 sample you provided is uncompressed RGB24, not YUY2 (the BY8 file is also uncompressed RGB24). Which image looks right compared to what you saw live? I suspect the conversion from RGB (from the camera) to YUY2, or YUY2 to RGB (in the file you uploaded) is going wrong. More likely the former.

By the way, could you provide a short sample of Y800 video? I've never seen any and I'd like to add it to my collection of odds and ends. Thanks!

[201flyer]

The samples I posted are straight from the capture program supplied with the camera. They are unprocessed from that point forward, save I clipped out the 10-20- frames using VDUB (no reproccessing). I will upload a Y800 raw sample. It will show no color, but will show color if you can figure out how to Debayer it. I use a free program called Registax. If you load the file into the program it will ask you if you want to process the file in B&W. Select "no." Then look in the far left column and go to options...select Debayer, then GB. The color you will see is very very close to what I can see in the capture software, or by eye through the telescope. It also looks exactly like the BY8 capture I already posted.

But there is now more to the story. I posted this question at an astronomy forum in Australia. It was suggested that I not specify RGB24 as the codec when capturing YUY2, but instead select the codec as "unspecified" I did a test and the video looked better but the color was still off compared to capturing either Y800 or BY8. I hit the "autowhite" in the software and the image in YUY2 immediately looked very close to normal. I guess this is a pecularity of the capture software - i.e., needing to white balance every time the capture method or codec is changed. I don't think I'd run into that before. If this is all that is involved I'm sorry to have stirred up this discussion!

You might be interested to learn why Y800 is used in this camera; it allows a much higher frame rate and smaller file size putting less stress on the computer while capturing given that the Debayering is delayed until after the frames are captured. For example BY8 can only achieve a frame rate of 15FPS at 1024x768, but Y800 will go up to 30FPS.

Again, thanks for your assistance!

[jagabo]

Thanks for the Y800 sample. I understand it's a raw dump of the camera's sensor.

Maybe you forgot to select Video -> Direct Stream Copy when you created the YUY2 sample? That would have converted it to uncompressed RGB24. The Y800 sample does appear to be Y800, not RGB24.

Attached is your BY8 avi converted to YUY2. I made this in VirtualDub by opening the BY8 file, selecting Video -> Color Depth... -> Output Format To Compressor/Display -> 4:2:2 YCbCr (YUY2), then saving as a new AVI. If you compare this to the BY8 cap you'll see the differences are almost imperceptible. Ie, proper YUY2 shouldn't look much different from RGB. So something is going wrong in your software's YUY2 processing.

YUY2 uses YUV instead of RGB to encode colors. Like RGB, YUV uses three components to encode color. The luma (Y) value is a grayscale image, U and V are the color components. YUY2 stores the color information at 1/2 the horizontal resolution of the grayscale information. So your 1024x768 image would have a 1024x768 Y channel and 512x768 U and V channels.

If we take your BY8 cap, convert to YUY2, and separate the Y, U, and V components into separate images we get this (top = Y, bottom left = U, bottom right = V, all at half size):



Doing the same with your YUY2 sample:



The latter has less contrast in the luma channel but even if we were to adjust for that the U and V channels don't look at all alike. Especially the U channel.

[201flyer]

Sorry to be slow to respond today. This is interesting detective work you have done!

I have redone the original YUY2 capture in case I made an error the first time. I am positive this one is "direct stream copy." (In the original YUY2 capture I set the codec to RGB24)

I am also posting the new YUY2 I captured where I set the codec as "unspecified" in the capture software. This was after pressing "auto white balance." To me this video looks as expected.

[jagabo]

Both those videos are RGB. The colors in "New-video..." are closer, but still has very different than the earlier BY8 source. My guess is the software simply isn't doing the RGB -> YUY2 conversion properly.

<later>

I wrote a quick and dirty debayering function in AviSynth:

AviSource("Y800.avi", pixel_type="RGB24")
R=PointResize(512,384).Subtitle("R")
Gr=Crop(1,0,-0,-0).AddBorders(0,0,1,0).PointResize(512,384).Subtit le("Gr")
Gb=Crop(0,1,-0,-0).AddBorders(0,0,0,1).PointResize(512,384).Subtit le("Gb")
B=Crop(1,1,-0,-0).AddBorders(0,0,1,1).PointResize(512,384).Subtit le("B")
MergeRGB(R,Gr,B)

And it delivered an image with colors very similar to your BY8 sample:



Note that the layout of the data in the sample video differs slightly from that Sony document -- remove the top row of the bayer pattern example:



The R at the top left now corresponds to the top left pixel in your Y800 video.

[201flyer]

Jagabo!!!

Thank you very much for the AviSynth Debayering script. I've been searching for an easy way to Debayer the Y800 captures and I have found that using Registax is not that reliable. I'm far from an expert in using AviSynth so I'm not sure I will succeed using your script and I don't have time to play with this today. * I also didn't quite follow your comment about the Layout of Sample being different from the Sony document. Are you implying that the camera manufacturer has got the Debayering wrong in the camera, or that the Sony document is incorrect. What is the implication of that intriguing statement?

Again thank you for this invaluable help!

* Is there a way to use AviSynth to debayer the avi file into a second "colored" avi file that is not recompressed but more of less a "direct stream copy" + color? I know I can frame serve various encoders, but I don't want to re-encode and lose any quality.

[jagabo]

The AviSynth script above is very crude. It gives you a half sized image and treats the RGB sub-pixels as coincident rather than spacially offset. This script is a slight improvement:

AviSource(&quot;Y800.avi&quot;, pixel_type=&quot;RGB24&quot

R=PointResize(512,384)
Gr=Crop(1,0,-0,-0).AddBorders(0,0,1,0).PointResize(512,384)
Gb=Crop(0,1,-0,-0).AddBorders(0,0,0,1).PointResize(512,384)
B=Crop(1,1,-0,-0).AddBorders(0,0,1,1).PointResize(512,384)

R=BicubicResize(R, 1024,768)
Gr=BicubicResize(Gr, 1024,768).Crop(0,0,-1,-0).AddBorders(1,0,0,0)
Gb=BicubicResize(Gb, 1024,768).Crop(0,0,-0,-1).AddBorders(0,0,0,1)
B=BicubicResize(B, 1024,768).Crop(0,0,-1,-1).AddBorders(1,1,0,0)

MergeRGB(R, Merge(Gr,Gb).Sharpen(1.0), B)

It restores the original locations of the subpixels and creates a full size image but it blurs the Gr and Gb images together. So it's less sharp (even with the Sharpen() function) than your BY8 clip. Note I'm using a trick with PointResize() to extract every other pixel (both dimension) to create R, Gr, Gb, and B planes which are later recombined into an RGB image. It may give some color fringing at very sharp edges.

I also found a debayering filter for AviSynth:

http://forum.doom9.org/showpost.php?p=1072167&postcount=8

Which can be used with your Y800 file like this:

AviSource(&quot;Y800.avi&quot

FlipVertical()
FlipHorizontal()
DebayerFilter(swap=0)
FlipHorizontal()

I don't know what the swap=0 argument does.

Regarding the Sony document: is the image from the camera upside down? If you flip the frame vertically the data in the file matches the original document. Since my colors match your BY8 captures I'm pretty sure your software is handling the captures properly -- except the YUY2 caps. You could verify this by shooting something with obvious colors.

The output from AviSynth is uncompressed RGB frames. You can store that as uncompressed RGB AVI (just like your BY8 file) if you want. If you want YUY2 frames from the scripts just add ConvertToYUY2() to the end.

[201flyer]

Sorry to need to ask for help. I've tried this, but can't get a result. I get this error message: "The video decompressor couldn't produce YUY2 or RGB output"

This is a copy of my script: AviSource("H:\test\Y800.avi").FlipVertical()FlipHo rizontal()DebayerFilter(swap=0)FlipHorizontal()

(the space in the word horizontal is not in my script, but I can't eliminate it in this post for some reason!)

Assuming I can eventually get this to work, do I open this in VirtualDub? Then save avi, direct stream copy?

Would you happen to know the method this AviSynth Debayering script uses to Debayer? I know there are many ways to do this, some much better than others. I've attached a very intelligently written article on the subject you might be interested to read.

Again, I appreciate your help tremendously!!!

[Gavino]

Regarding the Sony document: is the image from the camera upside down? If you flip the frame vertically the data in the file matches the original document.

That's because RGB data is stored upside-down in memory, and so PointResize in Avisynth works from bottom to top, eg when downsizing by a factor of 2 it selects every second line starting at the bottom (hence including the bottom one and omitting the top one).

R=PointResize(512,384)
Gr=Crop(1,0,-0,-0).AddBorders(0,0,1,0).PointResize(512,384)
Gb=Crop(0,1,-0,-0).AddBorders(0,0,0,1).PointResize(512,384)
B=Crop(1,1,-0,-0).AddBorders(0,0,1,1).PointResize(512,384)
...
Note I'm using a trick with PointResize() to extract every other pixel (both dimension) to create R, Gr, Gb, and B planes

So R contains every other pixel starting at bottom left, as in original Sony diagram.
However, for the same reason, your Gb and B will be missing the top line and have an extra black line at the bottom - instead of cropping the top and adding borders at the bottom, you need to do it the other way round:

Gb=Crop(0,0,-0,-1).AddBorders(0,1,0,0).PointResize(512,384)
B=Crop(1,0,-0,-1).AddBorders(0,1,1,0).PointResize(512,384)

Note too that instead of using Crop and AddBorders, you can simply use the additional offset parameters of the resizer to do this sort of thing:

R=PointResize(512,384)
Gr=PointResize(512,384, 1,0)
Gb=PointResize(512,384, 0,-1)
B=PointResize(512,384, 1,-1)

[jagabo]

Regarding the Sony document: is the image from the camera upside down? If you flip the frame vertically the data in the file matches the original document.

That's because RGB data is stored upside-down in memory

Thanks for that. So AviSynth always stores RGB upside down? But YUV rightside up? I know some image formats can have the frame stored either way and the order is flagged by a positive or negative frame height.

[jagabo]

Note too that instead of using Crop and AddBorders, you can simply use the additional offset parameters of the resizer to do this sort of thing:

I prefer to explicitly shifted the frame. Having been a programmer for 30 years, I hate relying on rounding error behavior of algorithms.

[Gavino]

So AviSynth always stores RGB upside down? But YUV rightside up?

Yes, that's right. (I think it might be a legacy of VfW).
However, it's not clear to me whether the differing behaviour of PointResize between RGB and YUV is intentional or an oversight (the code could instead have been written differently to make the results come out the same for both).
I guess it depends on whether you think PointResize should be defined in terms of memory layout or visual layout. In any case, it's a long-existing 'feature' and is unlikely to be changed because that would break compatibility with existing scripts.

I prefer to explicitly shifted the frame. Having been a programmer for 30 years, I hate relying on rounding error behavior of algorithms.

I understand what you mean, but there is no rounding here - the two forms are defined to be equivalent and I think the shorter one expresses the intention more clearly with less opportunity for getting it wrong.

[jagabo]

Sorry to need to ask for help. I've tried this, but can't get a result. I get this error message: "The video decompressor couldn't produce YUY2 or RGB output"

I had that problem initially too. I think I solved by enabling Y800 decoding in ffdshow -- the "Raw Video" codec.

Assuming I can eventually get this to work, do I open this in VirtualDub? Then save avi, direct stream copy?

Yes. You might want to check the output of AviSynth (use Info() at the end of the script) and verify it's RGB, not YUY2 or YV12.

Would you happen to know the method this AviSynth Debayering script uses to Debayer?

No. I think the source code was mentioned earlier in the thread I linked to.

[Gavino]

There is no source code with that plugin, but the author refers to using the method by Menon, Andriani et al, "Demosaicing with directional filtering and a posteriori decision".

At the end of the thread, there is a different plugin which does have source code.
(See http://forum.doom9.org/showthread.php?p=1108152#post1108152.)
It gives you the choice of either bilinear or nearest neighbour interpolation - for a description of the parameters, see the top of source file demosaic.cpp.

[201flyer]

I'm still having trouble getting the Debayer to work for me. I enabled the raw video codec to "all supported." When I play the Y800 video directly in GOM Player it reports the video to be YU12 Raw and plays it in B & W as YU12. (Playing Media Player Classic the video is reported to be Y800, but it also will not play the AviSynth file)

*** DIRECTSHOW FILTER LIST ***
1. Video Renderer
2. Overlay Mixer
3. Gretech Video
4. ffdshow Video Decoder
5. AVI Splitter
6. Gretech File Stream

*** VIDEO INFO ***
Input Type : YV12(RAW)
Input Size : 1024 x 768
Output Type : YV12
Output Size : 1024 x 768
FrameRate(Frame/sec) : 0.00 (30.00)

When I try to frame serve the avisynth I get the error message earlier reported + a brief message "Cannot use overlay mixer."

Sorry I'm such a beginner with all this. But, I really want to solve this, and I know there are a lot of people using this camera that would like a different way to Debayer as well. Thanks again!!

[jagabo]

Did you enable the VFW raw drivers? AviSource uses VFW, not DirectShow (media players use DirectShow or their own built in drivers). Use Start -> All Programs -> ffdshow -> VFW Configuration.

I just checked, if I disable the raw decoder in ffdshow I get the message "AviSynth open failure: AviSource: couldn't locate a decompressor for fourccY800..." So maybe your problem is different. In the VFW configuration dialog highlight the Output option in the left box, make sure that YV12, YUY2, RGB32, and RGB24 are enabled in the right side. The debayer filter requires YV12.

[201flyer]

Still no joy! I found the VFW dialog but it wasn't where you indicated, probably because I installed ffdshow as part of the k-lite codec pack. I set the RAW to "all supported" which seems to include Y800, and also tried just Y800. I checked the output and all your suggestions were already checked including Yu12. Maybe I'll try uninstalling k-lite and reinstalling ffdshow by itself.

I also found a program called ffvfw installed on my computer. I'm not sure what this is or if it could be related to my problems.


Here is perhaps a stupid obvious question. Is the Debayer function built into AviSynth, or is it a filter I need to download from somewhere?

[jagabo]

Try using DirectShowSource() instead. Or maybe ffVideoSource(). Are you running 32 bit Windows or 64 bit Windows?

Debayer is an AviSynth filter. You can get it at the link I posted. Put it it AviSynth's plugins folder.

[201flyer]

YES YES YES!!!! I put the filter in the plugin folder (da, I should have read your earlier post more carefully), and changed to DirectShowSource, and it works!!!

Hopefully I'm now getting to the end of my questions!

I ran the .avs through VirtulaDub saving the .avi as a direct stream copy. The output is about 3X larger in file size than the original Y800 file. GOM Player reports the following:

*** DIRECTSHOW FILTER LIST ***
1. Video Renderer
2. Overlay Mixer
3. Gretech Video
4. AVI Splitter
5. Gretech File Stream

*** VIDEO INFO ***
Input Type : RGB24(RAW)
Input Size : 1024 x 768
Output Type : YUY2
Output Size : 1024 x 768
FrameRate(Frame/sec) : 0.00 (30.00)

GSpot reports the output is: 4cc: DIB (_RGB), Name: BI_RGB Raw Bitmap

Is this as expected? Is there any way to save a Debayered avi that is smaller in file size but not compressed? I think I remember reading somewhere that the Huffyyuv codec might allow that. I ran a test with Huffyyuv. The output was only slightly smaller and GOM reports the output as RGB32.

Thanks again!

[jagabo]

Before debayering you have one byte per pixel. That byte encodes either red, green, or blue. The process of debayering interpolates those spatially distributed RGB data points to produce a full RGB value for each pixel -- three bytes per pixel. So each frame is three times larger, and hence the resulting file is three times larger. You could use a lossless codec like HuffYUV or Lagarith to reduce the file size without losing any accuracy. The amount of compression you get with these codecs varies with the nature of the videos. The less noise and detail, the more they can compress. With the Y800 sample you provided HuffYUV gave very little compression -- as you saw. Lagarith was able to reduce the video to 2/3 the uncompressed size (still twice as large as the Y800 file).

If you allow Lagarith to convert from RGB to YV12 you'll get files about the same size as your Y800 source. YV12 encodes the grayscale information at the source's resolution (1024x768 in your case) and the colors information at half resolution (512x384). But that's not really so different from your source before debayering.

[201flyer]

Thank You!

I think there will be more than a few planetary astrophotographers interested in this Debayering outcome using AviSynth. I don't think it is common knowledge that Y800 can be dealt with in this manner. Most everyone seems to use either Registax or a program called Ninox. I hope you don't mind if I pass along this solution. I'll post a link to this thread and also give you full credit for coming up with this. You have been a great help! (ps, I'll give Lagarith a try!)

[jagabo]

Feel free to pass the method along. It was an interesting exercise for me too.

By the way, you can convert the Y800 image to uncompressed RGB simply by opening it in VirtualDub and saving as uncompressed RGB. That will be easy to open in AviSynth with AviSource and won't require ffdshow (VirtualDub has a built in Y800 handler).

[201flyer]

By the way, you can convert the Y800 image to uncompressed RGB simply by opening it in VirtualDub and saving as uncompressed RGB. That will be easy to open in AviSynth with AviSource and won't require ffdshow (VirtualDub has a built in Y800 handler).

Running into little problems:

1. I tried your alternate way and created RGB uncompressed. When I tried to run it through the Debayer filter I got the message that "input to filter must be YV12"

2. Tried original method with actual Y800 capture .avi with dimension of 392 x 288. Error message "image pitch and width doesn't match" I don't really understand what this means, but I suspect it has something to do with the fact that this video was captured in a small window drawn around the planet (in this case Jupiter). This cuts down greatly on the file size (given that the native capture dimensions of this camera is 1024x768) and consequently helps the computer not to drop frames during the capture. I'm not sure how I could get the box exactly in the correct proportions (if that is the problem), or maybe it's just a problem created by the way the camera crops during capture.

Gavino mentioned this earlier in this thread:

"At the end of the thread, there is a different plugin which does have source code.
(See http://forum.doom9.org/showthread.ph...52#post1108152.)
It gives you the choice of either bilinear or nearest neighbour interpolation - for a description of the parameters, see the top of source file demosaic.cpp."

Is there any way to use this filter. I tried but failed. I just don't know enough about writing the script....

[Gavino]

1. I tried your alternate way and created RGB uncompressed. When I tried to run it through the Debayer filter I got the message that "input to filter must be YV12"

Use AviSource("xxx.avi", pixel_type="YV12"), or if that doesn't work, use AviSource("xxx.avi").ConvertToYV12()

2. Tried original method with actual Y800 capture .avi with dimension of 392 x 288. Error message "image pitch and width doesn't match"

Try with a width that is a multiple of 16, eg 384 or 400 instead of 392.

Gavino mentioned this earlier in this thread:
"At the end of the thread, there is a different plugin which does have source code.
(See http://forum.doom9.org/showthread.ph...52#post1108152.)
It gives you the choice of either bilinear or nearest neighbour interpolation - for a description of the parameters, see the top of source file demosaic.cpp."

Is there any way to use this filter. I tried but failed. I just don't know enough about writing the script....

Take a look at the sample test.avs included in the plugin zip.
The plugin needs to be loaded with LoadCPlugin, not LoadPlugin() and so can't be loaded automatically from the plugins folder. Try, for example:
LoadCPlugin("demosaic.dll")
AviSource("y800.avi")
ConvertToYV12()
Demosaic()

If demosaic.dll is not in the same folder as the script file, you will need to use the full path in the call to LoadCPlugin.