Converting YV12 to anything else causes inexplicable blockiness

[Asterra]

Like this:



The version on the right is the non-DirectX frame from in VirtualDub, straight from a for-certain YV12 video file. Edges are nice and smooth. I discovered that AE seems incapable of importing YV12, which is probably for the best, because I was able to quickly determine that no matter what conversion I threw at it, the result was that blocky mess on the left.

Then I determined that the same results were happening in VirtualDub. Even if I was simply converting the YV12 to YUY2 in Avisynth. And that baffles me.

Yes, 4:2:0 color is expected to be blocky. I understand that. But that begs the question of why it isn't blocky (certainly nothing like the left image) in its YV12 iteration. Or more to the point, how can I get this thing converted to something besides YV12 while maintaining YV12's apparently smooth appearance?

[Asterra]

As an aside, I'll add that I've noticed that on a couple of video files where I accidentally forgot to choose "fast recompress" (and which were therefore apparently rendered as 24bit RGB, though I'll have to confirm), After Effects imports them and they are not blocky. So I know it's possible to get these YV12 files in AE without the blockiness and it's not an issue with the chroma resolution.

[poisondeathray]

I explained this in the other thread, AE uses point resize (nearest neighbor algorithm) for the chroma upsamping , hence one of the reasons behind the suggestion to convert to RGB in avisynth before importing into AE

The preview you see is RGB (it has been converted to RGB for display somewhere; you're not viewing Y'CbCr directly, you're viewing the RGB converted representation of it) . HOW that conversion and upsample was done, affects what the preview looks like.

In avisynth 2.6 , you have many more options for the chroma resample algorithm. You could use a spline36 instead of a bicubic resize for example

[Asterra]

I explained this in the other thread, AE uses point resize (nearest neighbor algorithm) for the chroma upsamping , hence one of the reasons behind the suggestion to convert to RGB in avisynth before importing into AE

I remembered this suggestion and tried importing the YV12 files to AE through .avs with ConvertToRGB, ConvertToRGB24 and even ConvertToYUY2. All produced identical results. So if this is the solution, then there are some flags missing from the () that will be needed in order to reproduce the apparently flawless result VirtualDub gave when it exported in Full Processing mode.

[vhelp]

are you leaving out other details ?

and, can you provide samples of the actual avi ? since there are many variables in each persons computer, from software to codecs and internal configurations, etc etc., what you see on yours may not be seen on some one elses. i'm just saying, there are so many variables to contend. but here's a short list of details you may have left out that you could fill in and someone could help you further.

1. origin of the source
2. did you capture it or D/L it
2a. if captured, what device did you capture from and what connections, ie composite or s-video
2b. if captured, what codec and what settings in the codec
3. can you list the exact detail settings/configurations inside vdub, include all menus, panes and dialogs
4. did you feed source directly into virtualdub, ie avi->vdub, or through proxy, ie avi->avisynth->vdub
4a. if avisynth script, post the script
4b. if avi->vdub, can you determine exact codec vdub uses from the avi, (sometimes a subsit codec is used)

depending on the source device, composite can out-perform s-video and vise versa, or the capture cards composite can out-perform s-video or vise versa depending on the content and source device.

sometimes switching to a different codec (to capture with) will solve any issues. huffy has the least issues if any and is highly recommended. with 1TB HDD's, filesize shouldn't be an issue.

[poisondeathray]

I wouldn't import .avs into AE, because you're using heavy settings for QTGMC - this will make it almost unusable and unresponsive in AE. I would use an RGB intermediate

(BTW , I don't think you need to use the very slow presest, there might be modifed settings you can use instead - if you post a sample in the QTGMC thread, one of the gurus will probably whip something up for you )

If you think Rec601 is ok, then use ConvertToRGB24() .

If you want to try other chroma sampling methods you can specify that. You can also experiment with chroma placement
e.g. ConvertToRGB24(chromaresample="spline36")
http://avisynth.org/mediawiki/Convert


2.6 actually shifts the chroma differently than 2.58. There is a higher quality conversion available by converting to YUY2 first with . If you pixel peep you will notice there is more blurring and chroma shifted off in a direction unless you use this function

YV12toYUY2_26()
ConvertToRGB24(some parameters)

Search Doom9 for the YV12toYUY2_26 script by "Gavino"

Even better, if you could stay in YUY2 instead of going to YV12 (I though you said YV24 was working for you now with destripe ?)

[vhelp]

oh, i was typing response while others were already responding.

i see that this topic is a branch from another thread that pda was helping you in already. i didn't realize.

i would have suggested the same pda, keep in the RGB and feed that through AE. but it seems AE is just screwing up things anyway. maybe its time to find an alternate tool to replace AE, just a suggestion.

[Asterra]

I wouldn't import .avs into AE, because you're using heavy settings for QTGMC - this will make it almost unusable and unresponsive in AE. I would use an RGB intermediate

Fear not. I'm only importing raw uncompressed renders.

(BTW , I don't think you need to use the very slow presest, there might be modifed settings you can use instead - if you post a sample in the QTGMC thread, one of the gurus will probably whip something up for you )

Well, after banishing six hours of effort to the multithread gods, I went ahead and let QTGMC have its one-core fun, and I'm now only a couple of hours away from being done with those renders, so it's moot. Truly, though, the artifacts never went completely away until I went as low as Very Slow.

If you think Rec601 is ok, then use ConvertToRGB24()
[...]
e.g. ConvertToRGB24(chromaresample="spline36")
[...]
YV12toYUY2_26()
ConvertToRGB24(some parameters)

None of this resulted in anything visibly different in AE. This isn't a huge deal since, at the very least, I should be able to re-import these YV12 raws into VirtualDub and render them as raw again, only with Full Processing on this time. Whatever VirtualDub does in this mode seems to be correct. It's an untidy waste of time and space but it is an option.

Even better, if you could stay in YUY2 instead of going to YV12 (I though you said YV24 was working for you now with destripe ?)

Heh. YV12 remains the only thing that DeStripe likes. For me at least.

[Asterra]

i would have suggested the same pda, keep in the RGB and feed that through AE. but it seems AE is just screwing up things anyway. maybe its time to find an alternate tool to replace AE, just a suggestion.

Don't I wish! That reminds me of the time I made a Mac OS boot drive for my PC, figuring I'd be leaving behind the kludgy Windows world for the shiny Mac. And then I realized that there was only one reason to even have that drive (Logic), and that everything I wanted to do was really only doable on Windows. AE is like that. I even did my entire home video Bluray in AE because Premiere Pro was just too constrictive.

[poisondeathray]

Even better, if you could stay in YUY2 instead of going to YV12 (I though you said YV24 was working for you now with destripe ?)

Heh. YV12 remains the only thing that DeStripe likes. For me at least.

I thought you did some testing by removing some other filters and it worked with YV24? It was a problem with QTGMC preceding that or something ? (anyways I guess it doesn't matter now...)

As I said in the other thread I'm sure YV24 works with destripe (I did several tests, different sources, different resolutions)

[Asterra]

I thought you did some testing by removing some other filters and it worked with YV24? It was a problem with QTGMC preceding that or something ? (anyways I guess it doesn't matter now...)

Not even when it was by itself did it work. However I have more things to try (in the future, if it matters), now that I've recently been made aware of another pack of DLLs to fiddle with after installing Avisynth 2.6. DLLs which were needed for SetMTMode, but which also somehow broke Avisynth, making it unable to render more than a couple of frames before complaining about not being able to read the source file. I figure I'd have to expend a lot of time juggling DLLs before I found the magic combination, but the fact that this was even necessary, combined with the half- and/or three-quarters-corrupted nature of DeStripe's output, suggests to me some sort of issue with multithreading, or the multithreaded nature of Avisynth 2.6, in combination with DLL obfuscation.

[poisondeathray]

Which MT version? Set's 2.6 MT version has been reported to cause corruption. That could be it. I was using the vanilla 2.6 alpha 2 from sourceforge

[Asterra]

Heh. SEt's was the only one I was even aware of. So perhaps that's one mystery solved.

[Gavino]

Looking at DeStripe, I now remember what it is - it's just a (very clever) script function which uses various MaskTools functions. So it should work fine with YV24, as long as you are using the 26 version of Masktools. It 'should' also work multithreaded, but as you've seen there are no guarantees with MT Avisynth.