Bizarre vertical lines

[Asterra]

So I'm staring at my captured footage, and I am no longer able to tell myself that I can live with the odd phenomenon I am seeing.

Here's what's going on. I'm capturing 1080i from HDMI as lossless YUY2. Goodbye, half of the color resolution (and half of the 1080p), but that's the state of the art in capture cards for now. After I tell After Effects to separate the fields, the resulting image has a manifestation of vertical lines wherever there is a fine color gradient. Looking closely, the vertical lines show up because the pixels are changing brightness at regular intervals from left to right:

BBDDBBDDBBDDBBDD
BBDDBBDDBBDDBBDD
BBDDBBDDBBDDBBDD
(B=brighter, D=darker)

This phenomenon is not apparent when viewing a raw interlaced frame because the alternating brightness/darkness is, well, interlaced, which conceals the phenomenon adequately.

Now, I probably understand why this is happening. The YUY2 downscale of the chroma resolution is generating an undesirable result which happens to be particularly bad after the image has been deinterlaced. What I would like to know is what I can do to reduce or eliminate the vertical lines. What process exists that might alleviate the problem? Much as I'd like to keep the video raw, it's just not acceptable under these circumstances.

I would almost be willing to capture at 720p60 just to avoid this issue, even though that would thoroughly defeat my purposes.

[poisondeathray]

some more information, like a video sample, a screenshot would help

also what is the source? does this occur on all types of captures?

capture card?

[jagabo]

Now, I probably understand why this is happening. The YUY2 downscale of the chroma resolution is generating an undesirable result which happens to be particularly bad after the image has been deinterlaced.

No. What are you capturing? No broadcast or Blu-ray sources have 4:4:4 color. Most have 4:2:0 (YV12). Even if you are capturing RGB 4:4:4 reduction to YUY2 shouldn't cause that type of problem. This is more likely a problem with your capture device. You should post a few second sample that shows the problem.

[Asterra]

some more information, like a video sample, a screenshot would help

Yeah, probably should have gone ahead and done that. (Click to enlarge.)

On the left, interlaced. On the right, deinterlaced only with AE's field separation. As described, the vertical lines are apparent in the fine (green) color gradient (it's also apparent in all other colors, not just green), but it's well hidden in the interlaced image, although objects in motion obviously exhibit interlace artifacting.

also what is the source? does this occur on all types of captures? capture card?

Xbox 360 1080i60 HDMI -> AverTV HD DVR capture card, all digital. I have not tested this card's analog capture possibilities, and I'm in no hurry or need to. Since I am supposing this is happening most particularly because of the 50% loss in horizontal color resolution, I would guess that every kind of interlaced capture would have the same problem upon deinterlacing, although the effect would be muted on anything not both largely static and digitally perfect.

[Asterra]

No. What are you capturing? No broadcast or Blu-ray sources have 4:4:4 color. Most have 4:2:0 (YV12). Even if you are capturing RGB 4:4:4 reduction to YUY2 shouldn't cause that type of problem. This is more likely a problem with your capture device. You should post a few second sample that shows the problem.

To elaborate on my speculation.. It is possible that the original 4:4:4 video already uses a kind of dithering to achieve the color gradients seen, when this dithering is first halved in resolution and then deinterlaced, what was originally a sort of checkerboard of color brightness now becomes an all-too-conspicuous series of vertical color lines two pixels thick separated by black lines two pixels thick.

I should also mention that the color lines and black lines exchange places every frame in a 59.94fps deinterlaced render, as one would expect.

[poisondeathray]

The vertical lines are already in your source, you can see them if you look at individual fields

here is your "original" , cropped, then fields separated, and stacked on top of each other

I think because the "black/white" are offset in each consecutive even/odd field, they essentially cancel each other out when looking a single interlaced frame

But when you deinterlace, each field becomes a frame (+/- other processing like antialiasing), so the vertical bands become readily visible

You should take closer look at your capture card and the workflow

[Asterra]

Of course they're there. I trusted AE's field separation. ;p I never doubted it. I wasn't trying to figure out why the lines were magically appearing. I was looking for some sort of process that might help reduce the phenomenon.

Before you ask, yes, I've tried capturing with several different codecs. You're looking at UT Video Codec. I've also tried Lagarith, HuffYUV and raw. All the same. I don't think it's a problem with the card so much as an unfortunate consequence of the reduction in color resolution, compounded by the interlaced nature of the video.

[poisondeathray]

I don't have an xbox . Do you think your capture card or the xbox is the culprit here ?

It's unlikely chroma sampling (lines occur everywhere, even in black) or interlaced issue

It's unlikely to be codec issue (They record what signal they are given)



You can probably improve it with some filters

e.g. Left is your AE sample, Right is the "original" processed

But I need to examine temporally, and how pattern behaves over time - Can you post a video sample?

But it would be better if you could fix it upstream before the capture (less processing, less quality deterioration)

[Asterra]

I slapped together a two second clip of a 640x480 section of the frame, at 59.94fps. Not sure where to put it.

The vertical lines are not terribly apparent at 60fps, but that observation is almost irrelevant, because it is my intention to give Youtube the 60fps video so that it will be future-proofed against the day Youtube finally supports that framerate, just as vintage Youtube videos which had stereo audio (or exceeded the Youtube spec in other ways) can now be viewed in all their glory, without having necessitated their re-upload. To digress, since Youtube currently squishes framerates down to 30, this would result in the vertical lines being just as obvious as they are in a snapshot.

[poisondeathray]

How did you "slap it together" ?

A few seconds of unprocessed clip would be far better with fields intact (you can zip it up to reduce file size)

You can upload it directly here if <30MB or use a free hosting site e.g. mediafire.com, sendspace.com etc...


And I agree with you about youtube, it will only improve as bandwidth, codec advancements etc...

But if this is important to you , archive the original

[Asterra]

I don't have an xbox . Do you think your capture card or the xbox is the culprit here ?

What I suspect is that the game is the culprit (or just how the Xbox renders fine color gradients in this case). This game happens to make heavy use of color blending effects, and it was probably determined that the most effective way to achieve this without, say, unwanted false contouring artifacts, was to dither the colors a bit.

It's unlikely chroma sampling (lines occur everywhere, even in black)

That did strike me as odd, that there would be lines in the black areas. I am compelled to guess that it's just the nature of the way the game is handling color, and that it is affecting pretty much the entire screen, but I will be doing some captures of other games to see what kind of results I get.

[Asterra]

Incidentally, how did you achieve that processed result you posted? I would say it's more than good enough to further justify my pursuit of a 1080p result rather than falling back on 720p.

[poisondeathray]

I don't have an xbox . Do you think your capture card or the xbox is the culprit here ?

What I suspect is that the game is the culprit (or just how the Xbox renders fine color gradients in this case). This game happens to make heavy use of color blending effects, and it was probably determined that the most effective way to achieve this without, say, unwanted false contouring artifacts, was to dither the colors a bit.

It's unlikely chroma sampling (lines occur everywhere, even in black)

That did strike me as odd, that there would be lines in the black areas. I am compelled to guess that it's just the nature of the way the game is handling color, and that it is affecting pretty much the entire screen, but I will be doing some captures of other games to see what kind of results I get.

That is why I asked about other sources earlier. You can narrow down the issue.

If other games do not exhibt the problem - presto, you have your answer

If it occurs with everything, including non-xbox sources , that suggests something else, like your capture card

Also, I would have guessed that other xbox users would have reported this issue by now if it was an xbox issue

Incidentally, how did you achieve that processed result you posted? I would say it's more than good enough to further justify my pursuit of a 1080p result rather than falling back on 720p.

I used some avisynth filters. I can post the script but it's not complete yet. I need to examine the temporal relationships. This is only a static frame. How it appears in motion is important. That's why I asked for video sample

[Asterra]

How did you "slap it together" ?

In detail, cropped a 640x480 section of a post-Fieldskit raw video at 59.94, encoded with H.264 VBR 1-pass (all AE allows).

A few seconds of unprocessed clip would be far better with fields intact (you can zip it up to reduce file size)

Done. New clip, raw 1080i cropped to 640x480, UT Video Codec.

pm_ex3_ut.avi

[poisondeathray]

unprocessed means unprocessed. not cropped or anything. ie. original capture

The reason is, any manipulation you do, can complicate the analysis of what is going on. For example, if you cropped in AE or vdub, you would have incurred an RGB conversion=>back to YUY2 when exporting UT 422, and extra sampling issues might have popped up.

unprocessed please

[Asterra]

Right, I realized what you were getting at when I was doing my other test captures. (I have to boot between different OSes to access the capture card or the editing software, hence the delays.) Here's as big of a clip as I could manage. UT codec again, straight from VirtualDub.

pm01secs.avi

Incidentally, quick analysis of other video from the 360, including the menu, reveals that the vertical lines are present irrespective of the software. I'm going to quickly test whether removing my HDMI switch from the equation makes any difference.

[Asterra]

Removing the HDMI switch made no difference. Wasn't expecting it to, but since that was the only variable I have any control over, it was worth a shot. So either 1) the card is adding a sort of 4x2 dither to its own input, or 2) the Xbox 360 does this 100% of the time, at least with 1920x1080, and it's only made apparent in uncommon scenarios like this one, or 3) the Xbox 360 does this color dither and it's partly the card's fault for subsampling the color resolution.

For laughs, I will see what can be seen with 720p60 captures, including if I treat them as interlaced.

Edit: Not that it proves anything one way or another, but treating a 720p video as 720i results in the familiar vertical lines. And prior to "deinterlacing", a checkerboard pattern can be seen even in the ostensibly black areas. The video has very much the look of something that has gone through at least one A/D conversion. My guess on the culprit: The card's subsampling hardware.

[jagabo]

YUY2 subsampling does not cause the problem you are seeing. It causes blurring of the colors. Look at the flat black areas. There is just as much banding there. A big gamma shift makes it obvious (one field):



Interlaced video is usually sent as YUY2 over HDMI. I suspect the problem is in the Xbox 360. So the Colussus is simply saving what it receives.

[Asterra]

Anyway, looks like I'm stuck with it. I can do 720p and mostly avoid the issue thanks to there being no need to deinterlace, or 1080p, for which I'll basically have to have a filter/deinterlace script handed to me on a plate because I am far from well-versed with Avisynth. Third option is to triple my expenses and buy something besides the AverTV HD DVR, with no real guarantee that the results will be better since they're all basically made by the same people in the same country.

[Asterra]

YUY2 subsampling does not cause the problem you are seeing.

You're right. However, a poor implementation of the process could. By that I mean the card may have some chip that does a thoroughly inexact in/out in addition to its job as the subsampling hardware. It's otherwise difficult to explain the mess that's made so apparent in your enhanced image. If it's the Xbox 360, even I am convinced that people would have said something by now. But it's far less likely that somebody buying the cheapest HDMI capture card they can get their hands on would be quick to notice.

Edit: It's not the Colossus. It's the AverTV HD DVR.

[Brad]

The card doesn't cause lines on a black background. Does the Xbox have a photo viewer or something you could use to test?



And an image that was converted from the PS3's RGB output by the card: http://img51.imageshack.us/img51/8986/cinaviamute.png

I don't feel like hooking it up at the moment and I don't have an Xbox so hopefully these suffice.

[poisondeathray]

Well hopefully you can do some more tests and figure out the underlying problem. Did you try other game titles ?

If you can't figure out the problem, then you can use this script (adjust to your own tastes of course)

The function "DeStripe" is written by *.mp4guy who is another avisynth guru like didee. It selectively blurs the bands by targetting spatial frequencies, so you don't degrade the rest of the image, and it's not deteriorated as much as a generic horizontal blur (e.g. you could have tried using the various blurs in AE using horizontal dimensions, like box blur etc....).

rad is the thickness or width, offset is the distance between them, and thr is the threshold . You can play with the values and get better or worse results

You can find QTGMC information, links to plugins here. The newest versions work with YUY2
http://forum.doom9.org/showthread.php?t=156028

EDIT: I think Destripe won't play nicely with YV24, so you have to use YV12

Import("PATH/DeStripe.avs")
AssumeTFF
AVISource("video.avi")
ConvertToYV12(interlaced=true)
QTGMC(preset="faster", sharpness=0.5)
DeStripe(rad=2, offset=1, thr=16)

[Asterra]

Well. After all of that considerable effort, for which I am very thankful, I wish I could come back and say that's all I needed. ;p Now I'm hoping this doesn't turn into some sort of Avisynth newbie thread. Regardless, I'll mention my results.

UT Video Codec doesn't seem to play nice with Avisynth. This complicates things but it's nothing a few raw renders can't fix.

First attempt to point Avisynth to a raw render revealed the fact that VirtualDub defaults to RGB raw, rather than same-as-input. Nice to know. Will keep it in mind.

I finally found a color format the script seems to like enough to process. But it seems that QTGMC is giving a bizarre result (I have narrowed it down to QTGMC). It properly generates a 59.94fps file with twice the frames. However, it splits the frame in half horizontally. The top half seems to be the correct output. The bottom half cycles between several different states: All green, mostly black & white, heavily interlaced in random patches, etc. (It's also a little odd that I can't see frame updates in VirtualDub unless I grab its window and move it.)

Edit: Like this. (Unavoidably converted to RGB 24-bit.)


Something fishy. ;p In any event, the top half looks so great compared to the vertical lines thing, I could almost jump for joy. I guess the only thing left is to figure out why the bottom half is doing what it's doing, and clear up a lot more drive space.

[poisondeathray]

UT Video Codec doesn't seem to play nice with Avisynth. This complicates things but it's nothing a few raw renders can't fix.

It works fine. If you're using a non mod-4 resolution, then it can cause problems. 1920x1080 won't cause any issue. If there are other issues describe exactly what is wrong

First attempt to point Avisynth to a raw render revealed the fact that VirtualDub defaults to RGB raw, rather than same-as-input. Nice to know. Will keep it in mind.

video=>fast recompress to bypass vdub RGB conversion

I finally found a color format the script seems to like enough to process. But it seems that QTGMC is giving a bizarre result (I have narrowed it down to QTGMC). It properly generates a 59.94fps file with twice the frames. However, it splits the frame in half horizontally. The top half seems to be the correct output. The bottom half cycles between several different states: All green, mostly black & white, heavily interlaced in random patches, etc. (It's also a little odd that I can't see frame updates in VirtualDub unless I grab its window and move it.)

Sorry, I think it's a problem with YV24 and Destripe . You probably have to use YV12. I cleaned up the post above (this message board inserts weird random spaces, so try importing the function instead)

Import("PATH/DeStripe.avs")
AVISource("video.avi")
AssumeTFF
ConvertToYV12(interlaced=true)
QTGMC(preset="faster", sharpness=0.5)
DeStripe(rad=2, offset=1, thr=16)

[Asterra]

UT Video Codec doesn't seem to play nice with Avisynth. This complicates things but it's nothing a few raw renders can't fix.

It works fine. If you're using a non mod-4 resolution, then it can cause problems. 1920x1080 won't cause any issue. If there are other issues describe exactly what is wrong

The new script fixes this, apparently.

Sorry, I think it's a problem with YV24 and Destripe . You probably have to use YV12.

The new script works flawlessly. (Well, after I move AssumeTFF below AVISource, as you do later.) I guess the only remaining quibbles are the fact that DeStripe definitely only likes 4:2:0 (tried every permutation), and the curious if irrelevant phenomenon of needing to drag VDub's window if I want to see a newly-processed frame.

I guess it's time to scrutinize my captures for dropped frames (wouldn't it be great if there was an app for that) and see if AE wants to play nice with Avisynth. Terrific help. Can't thank you enough!

[poisondeathray]

I don't know what's going on. It's working now with YV24 for me (as it did last night). But when I saw your message checked this morning for some reason it didn't work...

info() reports YV24 , no screwed up frames (You have to convert back to YUY2 at the end if you want YUY2 export, there aren't very many codecs that support YV24, or you can export RGB)

EDIT: yes destripe definitely working with YV24, even with other 422 sources converted to YV24 . It just needs a planar YUV format , and YUV 422 => YUV 444 => YUV 422 is less lossy than going YUV 422 => YUV 420 => YUV 422



Don't know of any drop frame detectors, but there are several dupe detectors

Earlier versions of AE accept avs import directly through avisynth importer you can download from sourceforge; CS5 doesn't

If you're going into AE, you probably want to control the RGB conversion in avisynth anyways, import a lossless intermediate RGB . If you let AE do the conversion it will use Rec601 (usually wrong for HD) , unless you use color management and have tagged files

ConvertToRGB(matrix="rec709")

[Asterra]

EDIT: yes destripe definitely working with YV24, even with other 422 sources converted to YV24 . It just needs a planar YUV format , and YUV 422 => YUV 444 => YUV 422 is less lossy than going YUV 422 => YUV 420 => YUV 422

Well, you got me. Two different OSes with fresh installs of all the relevant software, and the only time DeStripe doesn't generate unusual corruption is when I feed it YV12. YV16 causes the bottom half to be corrupted and YV24 corrupts all but the upper-left quadrant.

If you're going into AE, you probably want to control the RGB conversion in avisynth anyways, import a lossless intermediate RGB . If you let AE do the conversion it will use Rec601 (usually wrong for HD) , unless you use color management and have tagged files

ConvertToRGB(matrix="rec709")

I plugged that line in to see what it would do to the image. Seems pretty destructive (overall gamma appears to be raised for some reason). But the most odd thing is that it generates unusual behavior in the chroma channel. Observe the moving hamburger or the digit which changes from 7 to 6 in this extremely short clip:

pm_sv2t3_xp_yv12torgb.avi

These oddities are definitely not in the pre-ConvertToRGB version (already deinterlaced and DeStriped). I enhanced the levels to make certain. The alternating nature of the phenomenon is particularly baffling to me since the video is no longer interlaced.

I may want to see if I can import to AE using color management with tagged files, as you suggest, although those two criteria are currently unknown exercises to me.

[poisondeathray]

Are you sure you have avisynth 2.6 alpha 2 installed (from sourceforge) ?


Color management is a huge topic, you should read up on it when you have time. It's too much to go into here

When you view Y'CbCr material, the matrix used determines how the conversion is done to RGB for display. The matrix Rec601/709 (or other matrices) specifices the coefficients used for Y'CbCr<=>RGB conversions

If you are using vdub, it will use Rec601 (usually incorrect for HD sources). For example if you open up a fraps capture, blu-ray , anything HD, they usually have Rec709 coefficients. If you don't use color management or are aware of how Y'CbCr<=>RGB conversions are done, your end results won't look proper

All Y'CbCr imports into AE get the Rec601 treatment (except v210) . 99% of the time that is correct for SD material. 99% of the time its wrong for HD material.





When uploading uncompressed files, please zip them up. This last one could have been 1/3 of the size



I think you have memory flushing issues (frames are cached in avisynth buffer), if you look at the time display 24.7 , the "hamburger" beside it looks "ghosted" like a double image. Close all your applications, even reboot, and run the script again

If you get into more avisynth stuff, a good application to use is AvsPmod . It has multiple tabbed previews f5 to preview but (hit number keys to flip) so you can compare various scripts , macro support, sliders, it's just the best tool for avisynth

[jagabo]

ConvertToRGB(matrix="rec709")

I plugged that line in to see what it would do to the image. Seems pretty destructive (overall gamma appears to be raised for some reason). But the most odd thing is that it generates unusual behavior in the chroma channel. Observe the moving hamburger or the digit which changes from 7 to 6 in this extremely short clip:

Did you make that with VirtualDub? Taking an interlaced YV12 source and making an RGB clip? If so, that's VirtualDub's fault. It doesn't handle interlaced YV12 chroma channels correctly. Convert to RGB in AviSynth before feeding VirtualDub: ConvertToRGB(interlaced=true). Or maybe you did a conversion in AviSynth and neglected so specify interlaced=true.

[poisondeathray]

jagabo , I think he added that after the script (ie. after deinterlacing, after destripe etc...)

To me it looks like a frame misorder mixup with temporal filtering, or memory flush issue