Video Colorization AI (PC Software) Question

[magnu]

Hi there

I have noticed a problem on my PC when trying to use two different pieces of software I have which do video colourisation. One is Vegas Pro & the other is Pixbim Video Colorize AI.

Without going into any of the usual arguments I see on here about the pros & cons of both AI software and colourising B&W video, can anyone give me a possible reason and solution as to why I see a ghosting effect (a lag in the frames of the colourising) and how to remedy it?

I have tried converting the original B&W MKVs & MP4s to ProRes to see if it makes a difference but to no avail?

[poisondeathray]

Do you have duplicate frames ?


A) Is it really a lag in the colorization of frames ? ie. frame 5's color clearly appears in frame 6 ?

Or

B) is it a mistake in segmentation ? ie. "coloring outside the lines" ?


If it's really lag, and the lag is constant, you can shift the chroma +/- "x" frames to match

If it's segmentation issue, then it's like a problem with the underlying colorization algorithm

[magnu]

I'm pretty sure it is A, so will have a go at what you suggest (though the PixBim software probably doesn't give an option to shift Chroma frames?).

[poisondeathray]

I would have guessed (B) because the likelihood of 2 programs getting it wrong in the same manner is lower

What is the source video, does it have duplicate frames ? Is it really blurry or low quality ?


To shift the chroma, it's easier to work in a program that can handle YCbCr (also known as YUV) directly, such as avisynth

When you work in YCbCr , the Y channel is separate from the Cb,Cr channels (the latter are color difference channels and hold the color information). For example, most types of prores, such as prores HQ use YCbCr.

So you would offset the CbCr channels to match the "Y" channel so they align up temporally

[magnu]

I have tried it from a few sources, but all so far have the same problem? They are all B&W TV shows from the 1960s, and are all very good copies that were recently restored professionaly. Some are from DVD and some Bluray.

[magnu]

I have Vegas Pro 19, as well as Twixtor & Vegasaur. Can I use any of these to shift the Chroma?

[magnu]

The effect I see is with the original being one or two frames ahead of the colourising, and very occasionally B&W when changing scenes.

[poisondeathray]

I have Vegas Pro 19, as well as Twixtor & Vegasaur. Can I use any of these to shift the Chroma?

Not very easily, or let me think about it. There might be a workaround. Internally Vegas works in RGB

The effect I see is with the original being one or two frames ahead of the colourising, and very occasionally B&W when changing scenes.

The problem is if it's not constant, or pattern changes, then a constant shift will not help you. You'd have to apply the shift differently to different sections. It's difficult to accurately automate the color matching to the correct frame

I have tried it from a few sources, but all so far have the same problem? They are all B&W TV shows from the 1960s, and are all very good copies that were recently restored professionaly. Some are from DVD and some Bluray.

I don't really use colorization software, but if faulty scene change detection, or faulty temporal analysis is used, that migth shift the chroma over frames

It just seems sounds unrealistic that a bunch of different sources ( good quality) would have the same issue with 2 different colorization methods

Did you try asking in the Vegas and the Pixbim Video Colorize forums ?

[magnu]

I assume I can do the same in Hybrid? If so, what would I do?

[magnu]

I am planning to ask in the forums you mention shortly.

[poisondeathray]

I assume I can do the same in Hybrid? If so, what would I do?

I don't really use hybrid, but there should be a place where you can edit a script

A basic script that corrected for the color lagging 1 frame behind would look like this

src=LWLibavVideoSource("video.ext")
Y=ExtractY(src)
U=ExtractU(src).trim(1,0)
V=ExtractV(src).trim(1,0)
CombinePlanes(Y,U,V, planes="YUV", source_planes="YYY",  sample_clip=orig)

If it was 2 frames you would use trim(2,0) instead of trim(1,0) , for the U,V lines

If it's shifting the other way, you can use loop(2,0,0) for 1 frame, which just repeats frame zero

I am planning to ask in the forums you mention shortly.

That should be your priority. Better to get it correct in the first place, than to try to fix it afterwards (and maybe not correctly)

[Past Colours]

Thhis has been a very interesting thread, especially for someone that doesn't realy understand the minutae, but it is becoming clearer all the time. Have you guys found work arounds, fixes or updates to this issue ?

Regards
Peter

[jagabo]

A basic script that corrected for the color lagging 1 frame behind would look like this

src=LWLibavVideoSource("video.ext")
Y=ExtractY(src)
U=ExtractU(src).trim(1,0)
V=ExtractV(src).trim(1,0)
CombinePlanes(Y,U,V, planes="YUV", source_planes="YYY",  sample_clip=orig)

Or simpler:

LWLibavVideoSource("video.ext")
MergeChroma(Trim(1,0))