Re-compressing many times with lossless codec in Vdub introduces colorbleed

[Mephesto]

When doing post-processing, especially lengthy ones, I usually save in x264lossless format in Vdub and continue from there, usually saving again. But each time I save in that format, the color bleeds more and more. I thought this was a color depth conversion issue to I changed color depth settings to virtualdub to Autoselect on "Decompression format" and "Same as decompression format" on "Output format to compressor/display" but I'm still having problems.

How do you fix this?

[poisondeathray]

are you in the same colorspace? what kind of x264 lossless? Y'CbCr 4:2:0 ?

any vdub filters ? (many work in RGB) or are you using video=> "fast recompress"

[Mephesto]

I believe I am, I think YV12 but not sure. How do I verify?

No filters, using "Save AVI". No fast recompress.

[poisondeathray]

Why are you using vdub at all if you're not using filters? Just to encode ?

Did you load the x264 lossless directly into vdub? File=>file information should report colorspace . Or load it with avisynth and use info()

If you use video=>fast recompress, that will prevent vdub to converting to RGB

The reason why this matters is colorspace conversions are lossy (even if you use a lossless format) , and rounding errors can can lead to bleeding around color edges if you do it a few times, more easily visible on animation than live action footage


Did you do any manipulations or colorspace conversions while it was interlaced? If you do RGB<=> YUV colorspace conversions in avisynth , you have to specify interlaced=true

[poisondeathray]

Summarize your workflow steps and filters used

If everything else seems to be ok, list your x264 version and command line. Try a short test with something else like lagarith or ut video codec instead - do you get the same problems ? (ie is this a buggy build you're using or something else)?

[Mephesto]

Why are you using vdub at all if you're not using filters?

To remove that variable. As it turns out, the color bleed remains. Now I know it can't be the filters.

Did you load the x264 lossless directly into vdub? File=>file information should report colorspace.

It doesn't.

Or load it with avisynth and use info()

Ah nice, says YV12, like I thought.

If you use video=>fast recompress, that will prevent vdub to converting to RGB

I've used Vdub for a long damn time, always knew about this feature but never tried it. What does it do? Will I be able to use filters with it? How's it different from Save as AVI?

The reason why this matters is colorspace conversions are lossy (even if you use a lossless format) , and rounding errors can can lead to bleeding around color edges if you do it a few times, more easily visible on animation than live action footage

I know, but I thought I fixed this on Colordepth options which in the past used to be RGB24 for output/display which would make sense, but now that I fixed that it still persists.

Did you do any manipulations or colorspace conversions while it was interlaced? If you do RGB<=> YUV colorspace conversions in avisynth , you have to specify interlaced=true

No, but that doesn't matter. It can be any input of any type. As soon as it's saved as x264lossless, color bleeds. If you re-save that output again with the same codec, color bleeds even more.

[Cornucopia]

This is something that many people get confused about, pro and amateur alike. Just because your codec may be lossless DOES NOT mean that your process(es) is/are lossless. And that matters greatly! Both have to be for there to be zero loss.

Scott

[Cornucopia]

This is something that many people get confused about, pro and amateur alike. Just because your codec may be lossless DOES NOT mean that your process(es) is/are lossless. And that matters greatly! Both have to be for there to be zero loss.

Scott

[Mephesto]

Summarize your workflow steps and filters used

If everything else seems to be ok, list your x264 version and command line. Try a short test with something else like lagarith or ut video codec instead - do you get the same problems ? (ie is this a buggy build you're using or something else)?

Okay, I take a video, B1.avi, a short cut of 50 frames, open it in Vdub, go to Video > Compression and select x264 with my custom settings to make it lossless. Go to File > Save as AVI and save as B2.avi.

Open B2.avi, repeat last steps, save as B3.avi. B3 has worse color bleed than B2, as B2 has worse than B1.

My x264 version is r1629 and I don't have the command line since it's Vdub. Here are encoding settings:

cabac=0 / ref=2 / deblock=0:0:0 / analyse=0x1:0x31 / me=dia / subme=3 / psy=0 / mixed_ref=1 / me_range=12 / chroma_me=0 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=0,0 / fast_pskip=0 / chroma_qp_offset=0 / threads=9 / sliced_threads=1 / slices=9 / nr=0 / decimate=0 / interlaced=0 / constrained_intra=0 / bframes=0 / weightp=2 / keyint=50 / keyint_min=5 / scenecut=40 / intra_refresh=0 / rc=cqp / mbtree=0 / qp=0

[poisondeathray]

Did you do any manipulations or colorspace conversions while it was interlaced? If you do RGB<=> YUV colorspace conversions in avisynth , you have to specify interlaced=true

No, but that doesn't matter. It can be any input of any type. As soon as it's saved as x264lossless, color bleeds. If you re-save that output again with the same codec, color bleeds even more.

Did you try lagarith, or UT Video Codec in 420 ? This will help narrow down the problem if it's tjat particular x264 buld problem or something else

What x264 build? is it VFW or CLI ?

How are you viewing the export from vdub? What decoder ?

I've used Vdub for a long damn time, always knew about this feature but never tried it. What does it do? Will I be able to use filters with it? How's it different from Save as AVI?

No, fast recompress means your are unable to use vdub filters (you could use filters in avisynth script and keep same colorspace)

In full processing mode, old vdub versions used to convert to RGB automatically (= bad) . New vdub versions depend on the how color depth options are set

Eitherway, you want to avoid uncessary colorspace conversions, and fast recompress will do this for old or new versions

[poisondeathray]

Summarize your workflow steps and filters used

If everything else seems to be ok, list your x264 version and command line. Try a short test with something else like lagarith or ut video codec instead - do you get the same problems ? (ie is this a buggy build you're using or something else)?

Okay, I take a video, B1.avi, a short cut of 50 frames, open it in Vdub, go to Video > Compression and select x264 with my custom settings to make it lossless. Go to File > Save as AVI and save as B2.avi.

Open B2.avi, repeat last steps, save as B3.avi. B3 has worse color bleed than B2, as B2 has worse than B1.

My x264 version is r1629 and I don't have the command line since it's Vdub. Here are encoding settings:

cabac=0 / ref=2 / deblock=0:0:0 / analyse=0x1:0x31 / me=dia / subme=3 / psy=0 / mixed_ref=1 / me_range=12 / chroma_me=0 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=0,0 / fast_pskip=0 / chroma_qp_offset=0 / threads=9 / sliced_threads=1 / slices=9 / nr=0 / decimate=0 / interlaced=0 / constrained_intra=0 / bframes=0 / weightp=2 / keyint=50 / keyint_min=5 / scenecut=40 / intra_refresh=0 / rc=cqp / mbtree=0 / qp=0

With video=>fast recompress , correct ?

If lagarith and/or UT works out ok, Try using an updated x264 vfw . r1629 is from June 2010....

http://komisar.gin.by/

[Mephesto]

Did you try lagarith, or UT Video Codec in 420 ? This will help narrow down the problem if it's tjat particular x264 buld problem or something else

I tried lagarith YV12 and... it REVERSED the already-colorbled test clip I tried. This is weird.

What x264 build? is it VFW or CLI ?

VFW, durrrr. CLI doesn't work on Vdub.

How are you viewing the export from vdub? What decoder ?

Vdub

Eitherway, you want to avoid uncessary colorspace conversions, and fast recompress will do this for old or new versions

Filters are a requirement.

With video=>fast recompress , correct ?

NO, I said "Save as AVI".

If lagarith and/or UT works out ok, Try using an updated x264 vfw . r1629 is from June 2010....

I can't, later x264vfw versions don't let me customize encoding settings.

[poisondeathray]

Eitherway, you want to avoid uncessary colorspace conversions, and fast recompress will do this for old or new versions

Filters are a requirement.

That is why many people prefer to do filtering in avisynth, you don't get the extra quality loss from color space conversions

And if you think the loss is neglible, it's easy to demonstrate - here is an RGB image, and it's gone through 3 rounds of RGB=> YV12 (and back to RGB for the screenshot)

View this at 100%


[Attachment 11027 - Click to enlarge]


Note this uses avisynth 2.6's default bicubic algorithm for the chroma upsampling, but you will get similar loss with other algorithms. The bleed will be less if you used something like nearest neighbor algorithm, but the color edges will look more pixellated and blocky

The moral of the story is to avoid colorspace conversions

[poisondeathray]

Somehow my earlier reply got deleted so I'll repost

Did you try lagarith, or UT Video Codec in 420 ? This will help narrow down the problem if it's tjat particular x264 buld problem or something else

I tried lagarith YV12 and... it REVERSED the already-colorbled test clip I tried. This is weird.

Not sure what's going on here, maybe upload some samples to illustrate

How are you viewing the export from vdub? What decoder ?

Vdub

With the file loaded in vdub, file=>file information what decoder ?

Eitherway, you want to avoid uncessary colorspace conversions, and fast recompress will do this for old or new versions

Filters are a requirement.

This is why some people prefer to filter in avisynth instead of vdub, to avoid the colorspace issues with vdub filters

With video=>fast recompress , correct ?

NO, I said "Save as AVI".

If you had it on full recomprss mode, with output to RGB, that at least partially explains the problem. If you had it manually set to YV12 4:2:0 then newer versions of vdub should be ok

If lagarith and/or UT works out ok, Try using an updated x264 vfw . r1629 is from June 2010....

I can't, later x264vfw versions don't let me customize encoding settings.

Yes, you can

There are 2 GUI versions. One from sourceforge, one from the other link. Both have custom command line boxes

The other link has options broken out into tabs, the one from sourceforge has 1 page only and uses "presets". Both you can customize with CLI other options box

[Mephesto]

[QUOTE=poisondeathray;2142493]Not sure what's going on for that. Maybe upload some samples[quote]

http://www.sendspace.com/file/e65svm

with the video loaded, vdub file=>file information what decoder is being used

Oh, x264vfw. Lagarith Codec with the lagarith AVIs... Seems like a redundant question to me.

That is why many people prefer to do filtering in avisynth, you don't get the extra quality loss from color space conversions

Too bad some filters aren't found on Avisynth. And you lose the extra quality anyway because you can't run everything in one big script, especially if shit goes wrong and you gotta redo some of the long-ass operations without then having to re-render the whole video with all the previous slow scripts. This is why I do denoising separate from TGMC deinterlacing. Did I mention I can't run two TGMC scripts at once or it runs way slower than if I ran them one at a time?

If you used full processing mode, then save as avi, and have the output set to RGB24, then you are converting to RGB, that explains at least part of the problem.

But I don't.

There are 2 different VFW GUI's. Both allow custom encoding settings in the command line box.

You mean two UI's? I don't even know how to do lossless mode in command-line, how would I transfer my current settings to CL? Would that custom command-line be saved for future use?

The one I linked to has many options broken/laid out with several tabs.

Bullshit, I'll have to see this to believe it. Screenshot?

[poisondeathray]

with the video loaded, vdub file=>file information what decoder is being used

Oh, x264vfw. Lagarith Codec with the lagarith AVIs... Seems like a redundant question to me.

It's not, because you might be using a different decoder for x264 lossless. It might be ffdshow for example. Some builds have known bugs

That is why many people prefer to do filtering in avisynth, you don't get the extra quality loss from color space conversions

Too bad some filters aren't found on Avisynth. And you lose the extra quality anyway because you can't run everything in one big script, especially if shit goes wrong and you gotta redo some of the long-ass operations without then having to re-render the whole video with all the previous slow scripts. This is why I do denoising separate from TGMC deinterlacing. Did I mention I can't run two TGMC scripts at once or it runs way slower than if I ran them one at a time?

No - you don't lose quality with lossless formats in the same colorspace - that's the whole point! It's easy to prove, that decoded image will be bit for bit identical. You can break the script up into 1000 sub scripts if you want. As long as you aren't doing some colorspace conversions, the compression will be lossless with x264 lossless or lagarith etc...

Usually there is some equivalent filter in avisynth. What filter are you using that only vdub has , and there is no avisynth equivalent for ?

Most of the "slow" scripts and filters are single threaded. You can try to run MT, and some people do successfully, but it can also cause problems

Another method to speed things is to run parallel - so multiple instances of vdub or whatever program encoding

If you used full processing mode, then save as avi, and have the output set to RGB24, then you are converting to RGB, that explains at least part of the problem.

But I don't.

But you also said you aren't using filters on this run.

So if you try fast recompress does the problem still occur ?

There are 2 different VFW GUI's. Both allow custom encoding settings in the command line box.

You mean two UI's? I don't even know how to do lossless mode in command-line, how would I transfer my current settings to CL? Would that custom command-line be saved for future use?

CLI won't save anything, but you can copy & paste a text line for example

VFW GUI will save settings

The one I linked to has many options broken/laid out with several tabs.

Bullshit, I'll have to see this to believe it. Screenshot?

How about a little trust here?

[poisondeathray]

I see the shift with the lagarith bmp screnshots in the sendspace sample. Did you use video=>fast recompress for those runs ? What settings was lagarith configuration in ?

Thereotically, If you loaded lagarith YV12 , saved out lagarith YV12, reloaded lagarith YV12, rinse & repeat - there should be no shift at all between subsequent runs...it would be bit for bit identical....If you did it properly. Actually I'm 100% sure this happens.

Can you upload the x264 samples to compare?

Actually upload the clips from the videos instead if you can

[jagabo]

Yes, it's the RGB/YV12 colorspace conversions in VirtualDub that cause the color blurring. You can force the color depth (Video -> Color Depth) to YV12 to avoid the problem. But as soon as you start filtering the problem will come back because most VirtualDub filters only work in RGB.

[Mephesto]

I'm back. The problem doesn't occur with Fast recompress.

Many advanced filters with no equivalents, such as NeatVideo only work with Vdub. The avisynth version of NV blows and customizing settings with a script is really unintuitive and annoying.

About running in parallel, that is a lot easier said than done. It is limited by RAM as well as hard disk seeks, and I'm dealing with 4-5 GB per episode. I can only run two in parallel and get the full benefits. With 3, the third instance runs twice as slower than the first two. With 4, the third instance runs a lot slower and the fourth is in the range of seconds per frame rather than frames per second.

And holy shit, they restored the setting tabs finally? This can't be BugMaster's build from DTS. Oh well, I will upgrade immediately. Do you know where the x264vfw settings are saved so I don't lose them?

I set color depth in Virtualdub to YV12 by default and this fixed the problem on full processing mode but it sucks to hear that some filters only work with RGB.

Is there a way to reverse the artifacts caused by faulty colorspace conversion? With the VHS filter in Vdub I can correct color bleed by shifting the color back the other direction.

[poisondeathray]

I'm back. The problem doesn't occur with Fast recompress.

That confirms your problems are from YUV<=> RGB colorspace conversions

Many advanced filters with no equivalents, such as NeatVideo only work with Vdub. The avisynth version of NV blows and customizing settings with a script is really unintuitive and annoying.

You can get similar results in avisynth. Only in rare cases that you might not be able to.

One benefit of the new neat video version is GPU accelleration (very few avisynth filters can benefit from GPU)

And holy shit, they restored the setting tabs finally? This can't be BugMaster's build from DTS. Oh well, I will upgrade immediately. Do you know where the x264vfw settings are saved so I don't lose them?

Not sure, I don't really use x264vfw that often. I try to avoid it, it doesn't have full functionality of x264CLI (even if you use the command line box) , and is slower and more buggy

I set color depth in Virtualdub to YV12 by default and this fixed the problem on full processing mode but it sucks to hear that some filters only work with RGB.

The majority work in RGB. A few like brightness can work in YUV. If you checkmark the "show image formats" box in the video filters box, it will show you what colorspace you're currently in (if you use null), or after the filter

Is there a way to reverse the artifacts caused by faulty colorspace conversion? With the VHS filter in Vdub I can correct color bleed by shifting the color back the other direction.

No, it's an irreversible loss. You can shift the chroma around, or use different chroma in/out or sampling techniques, but invariably all will cause quality loss. And it's not "faulty". You're upscaling, downscaling (subsampling) the chroma each time you go back & forth. If you enlarge and shrink a photo in photoshop multiple times in photoshop - that's analogous to what is happening to the color information. The point is to avoid it in the first place, it's a lossy transformation. Period. Problems are more visible more in animation/cartoons because of clear color borders.

So if you're going to use RGB filters, then do it only once. Don't go back & forth. Plan your filters and workflow around that accordingly.

[Mephesto]

You can get similar results in avisynth. Only in rare cases that you might not be able to.

I don't accept "similar". Besides, I cant get a similar interface.

Not sure, I don't really use x264vfw that often. I try to avoid it, it doesn't have full functionality of x264CLI (even if you use the command line box) , and is slower and more buggy

Nvm I got it. I used to use x264vfw exclusively but now with advanced options like MB-tree it is buggy and unreliable. What I always hated about x264 in AVI is that I couldn't pack it with HE-AAC audio or even regular AAC with weird samplerates.

However, when it comes to editing, VFW is mandatory. Today I only use VFW for its lossless mode so I can prepare my videos for the final encode which are done with x264CLI.

The majority work in RGB. A few like brightness can work in YUV.

Still says RGB32 by the Brightness & Contrast filter. Damn. Can x264 work with RGB? It says valid depths are 24 and 32, but when I select color depth output to RGB32 in Vdub, x264vfw refuses to compress. What up with that bullshtick?

No, it's an irreversible loss. You can shift the chroma around, or use different chroma in/out or sampling techniques, but invariably all will cause quality loss. And it's not "faulty". You're upscaling, downscaling (subsampling) the chroma each time you go back & forth. If you enlarge and shrink a photo in photoshop multiple times in photoshop - that's analogous to what is happening to the color information. The point is to avoid it in the first place, it's a lossy transformation. Period. Problems are more visible more in animation/cartoons because of clear color borders.

The loss looked like color bleed to me, which I can fix. If the colors bleed in both directions however...

So if you're going to use RGB filters, then do it only once. Don't go back & forth. Plan your filters and workflow around that accordingly.

And I do, but because of this recent shimmering problem, I had to apply QTGMC once again in progressive mode (couldn't do it in one script) so this is the only time I had to color-rape my video more than once. But now that you told me about fast recompress, this won't be a problem anymore.

[jagabo]

now that you told me about fast recompress, this won't be a problem anymore.

You can't do any filtering in fast recompress mode.

To sharpen the chroma a bit after a YV12/RGB/YV12 conversion cycle you can can try something like:

 MergeChroma(last, Sharpen(last, 0.2))

Shifting chroma in AviSynth can be done with ChromaShift(). But that's not your problem.