[Solved]Color space RGB vs. YV12 with Lossless videos

[SnapPop]

hey what's up guys ?!
i 'm new in fansub stuffs ( i guess i mentioned that before xD) anyway...
i have anime sources like Fairy Tail 720p (encoded before off course as the files rounded between 290 and 480MB with duration 24:45)
i'm about to add my subtitles to these raw sources but there's a problem that their quality can't allow me to re-encode without a hardcore filtering
well no problem then, all i have to do is tiny script with Avisynth and VirtualDubMod with Lagarith lossless codec can satisfy my desire
first of all lemme show the beginning of the process with Avisynth script please keep in mind that i have the latest version of it and most of it's plugins and scripts except Anime Restoration just added Debanding with GradFunkMirror()

FFVideoSource("D:\Fairy tail\[Leopard-Raws] Fairy Tail (TX 1280x720 x264 AAC)\[Leopard-Raws] Fairy Tail - 14 RAW (TX 1280x720 x264 AAC).mp4", threads=1)
Trim(0, 2876) ++ Trim(3116, 35361)
Remation(Xres=1280, Yres=720, scale=1.5, linestr=0.15, dQuant=33, bOffset=0, dBlur=12, qBlur=4, thBlur=0.3, shrpstr=200, Deblock=true, AA=true, Sharp=true, Dehalo=true, Deband=true)

(takes about 5~6h with CPU i7 3770k and 16GB ram how horrible! xD)
well forget about trim cause i just wanted to cut about 20 useless seconds from the video xD anyway
now let's get back to the core which is the lossless rendering with VDM and Lagarith, the source is YUV color space and i heard that if the process like this YV12>>RGB>>YV12 again, this gonna be lossy. and yeah i forget to mention that i still have to add the lossless output to After Effects to add such stuffs some kind of lossless avi watermark and karaoke or whatever that means 2 lossless rendering one with VDM and second with AE (Dam it the 2 outputs exceeds 100GB xD well who cares)
Please note that if i render with VDM and Lagarith in YV12 mode it gets Color Banding (output size kinda 14GB) i don't know how that can happen after using a debanding filter and a lossless rendering while i don't have it in RGB mode like this pic



so i need your suggestions guys all i want is to retain the final lossless output with the maximum quality as possible as i can
sorry if my english is freaky

[SnapPop]

up up need your experience guys

[jagabo]

The output of AviSynth is probably YV12. Put VirtualDub in Video -> Fast Recompress mode and use a lossless codec that works in YV12, like UT Video Codec. That will avoid conversion to RGB.

[newpball]

I am a bit confused about your workflow, you got a source in CODEC A, you want to add subtitles and a watermark and then render it in CODEC B. What is CODEC A and CODEC B and what are their resp bitrates? By the way why do you need to add a watermark?

[poisondeathray]

Fansubs want to protect their "work" , so many of them choose to deface videos with annoying animated logos and other crap

If you're using AE , the output is going to be RGB only .

Another way to do this would be to export the AE derived effects (karaoke, watermark) with a luma matte, so you can overlay them onto the avisynth part in YV12 using overlay() or mt_merge() - so only the effects incur the colorspace losses instead of the entire video

[SnapPop]

first of all i'm really grateful for responding guys

@jagabo
i did what you said and used UTVideo YUV420 BT.709 VCM well the output was pretty much cool (doesn't have that banding at least) didn't try to import it to AE yet, will see about that later

@newpball
no buddy you didn't get it well, i want to filter the first source which is [Leopard-Raws] episode then import the filtered output to AE so that i can add the lossless avi watermark and karaoke on it and then its output will be rendered to the final Qualities like 720p(Hi10Bits~8Bits) or 480p whatever it is

@poisondeathray you crashed my day pal xD your saying that the output of AE will be just RGB ? it's means after all it's gonna be lossy cause at last it will be converted to YV12 actually the watermark must be added every 5 minutes among the episode as i know only AE can do this (which is video on video every 5 Min) kinda hard for any other programs but if you know something can be done with it i will be grateful this will spare time Quality and HDD usage xD

[poisondeathray]

AE can render out YV12 formats, but internally it works in RGB. So you would be going YV12 (source) => RGB in AE => YV12 out . AE will be doing the RGB=>YV12 conversion if you render out a YV12 format

Why can "only AE do this ?" What are you doing that requires AE ?

The other option is to use RGB or YUV444 as final render, but filesizes will be larger. Doesn't really make sense since your "source" was YV12 to begin with

[newpball]

@newpball
no buddy you didn't get it well, i want to filter the first source which is [Leopard-Raws] episode then import the filtered output to AE so that i can add the lossless avi watermark and karaoke on it and then its output will be rendered to the final Qualities like 720p(Hi10Bits~8Bits) or 480p whatever it is

So you are going from CODEC A to a lossless CODEC B, edit it and then you render to CODEC C.

So I like to understand why you do not want to edit straight from CODEC A, e.g. what is the point in going to CODEC B?

[poisondeathray]

He wants to apply the avisynth filters to a lossless intermediate (codec B) before importing into AE .

It's difficult to import .avs scripts in AE CC/CS6/CS5 directly

(it's possible indirectly with avfs, and old AE versions could import .avs scripts directly because they were 32bit with the avs importer; whereas avisynth 64bit is unstable and very limited in terms of plugins )

If it's just a simple overlay, don't bother with AE. Even if it's some goofy complex animated overlays, "my name is here, I did this fansub, look at me , look at me, look at me" , I would still do it as suggested above, so you don't incur additional quality loss on the entire video, only the logo overlays

[newpball]

He wants to apply the avisynth filters to a lossless intermediate before importing into AE .

Oh I see.

[SnapPop]

@poisondeathray
"it works internally in RGB" so let me say that i finished filtering and i have a lossless YV12 and imported to AE edited it and so on the render as lossless this wont loss any of the video's detail? and like i said the watermark itself is sort of a lossless avi file not just a pic or an AE project so it's like merging 2 videos or more using layers i only know AE easy to to such job

@newpball
didn't get you buddy if "CODEC A" represent the source video well check this pic from the same frame like the pic above but without filtering

so if i applied my editing in AE directly without cleaning up the source first the final outputs (720p or 480p) will be kinda miserable

[racer-x]

I'm pretty sure that ffmpeg allows intermittent watermark, check the docs.

[poisondeathray]

"lossless" codec means no loss in the same colorspace. If you convert YV12=>RGB=>YV12 you will incur some losses from the colorspace conversion, but you won't incur addition compression losses

avisynth has mt_merge and overlay() ; you probably don't need AE for this (overkill, slower, and means you have to to use RGB)

[newpball]

didn't get you buddy if "CODEC A" represent the source video well check this pic from the same frame like the pic above but without filtering

Well presumably all that stuff you do not like is very likely due to the 8bit video limit and the encoding used. So if you stay with 8 bits and encode the same way it will simply be back again.

There is no way you can simply provide the subtitles? And why do you want to butcher the video with a logo?

[SnapPop]

"lossless" codec means no loss in the same colorspace. If you convert YV12=>RGB=>YV12 you will incur some losses from the colorspace conversion, but you won't incur addition compression losses

avisynth has mt_merge and overlay() ; you probably don't need AE for this (overkill, slower, and means you have to to use RGB)

ok i got it and i will try the mt_merge or overlay

@newpball
it's just the way they acting here to protect their copyrighted xD but it's only with 8bits 720p & 480p not with 720p 10bits actually don't know the reason like you xD

[SnapPop]

i can't merge them with overlay since opacity option have some issues makes the main video darken
do i have any other options ?
in case if i have nothing else but AE to merge the Watermark, what is the best choice then ?
YV12(Source)>>RGB(VirtualDub)>>RGB(AE)>>YV12(x264)
or
YV12(Source)>>YV12(VirtualDub)>>RGB(AE-internally)>>YV12(x264)

[poisondeathray]

You use a mask, that's the "best" choice

What is the "watermark video" ? Is it RGBA? (does it have alpha channel? ) or how were you doing it in AE, what blend mode ? or what procedure ?

[jagabo]

i can't merge them with overlay since opacity option have some issues makes the main video darken

You probably have a rec. vs. pc. luma range problem with your masks. Use the PC matrix when converting masks between RGB and YUV, for example ConvertToYV12(matrix="pc.601") or ConvertToYV12(matrix="rec709").

[SnapPop]

You use a mask, that's the "best" choice

What is the "watermark video" ? Is it RGBA? (does it have alpha channel? ) or how were you doing it in AE, what blend mode ? or what procedure ?

the watermark video is 720p with a transparent background only the main animated watermark on the top left corner and yes it's RGBA and yes have alpha channel, actually i'm not the one who created it, it's the team that i'm gonna work for and that's how they use it with AE, so i'm 2 days trying all the options on overlay and looks like i fail and fail

You probably have a rec. vs. pc. luma range problem with your masks. Use the PC matrix when converting masks between RGB and YUV, for example ConvertToYV12(matrix="pc.601") or ConvertToYV12(matrix="rec709").

thanks for the info buddy i will give it a try ^^

[poisondeathray]

To overlay it in RGBA , you would use showalpha()

Here, the overlayclip is your RGBA overlay

e.g
Overlay(baseclip, overlayclip, mask=overlayclip.showalpha())

But this is no different than doing it in RGBA (ie. same math operation as AE, except this will be a straight alpha) . Earlier I was suggesting to do it in YUV, with a YUV mask

[SnapPop]

well i would prefer AE then to do all the stuff at one stage (watermark, karaoke, anime crest simulation bla bla bla)
so back to the previous question in case i'm doing it with after effects

YV12(Source)>>RGB(VirtualDub)>>RGB(AE)>>YV12(x264)
or
YV12(Source)>>YV12(VirtualDub)>>RGB(AE-internally)>>YV12(x264)

or just both the same

[poisondeathray]

well i would prefer AE then to do all the stuff at one stage (watermark, karaoke, anime crest simulation bla bla bla)
so back to the previous question in case i'm doing it with after effects

YV12(Source)>>RGB(VirtualDub)>>RGB(AE)>>YV12(x264)
or
YV12(Source)>>YV12(VirtualDub)>>RGB(AE-internally)>>YV12(x264)

or just both the same

There is no functional differece between those two , if you do the RGB<=>YUV conversions in the same manner (e.g. if you control them in avisynth). If you let one of the tools do the conversions, then you'll get slightly different results

[jagabo]

If you're going to filter in VirtualDub you should convert to RGB yourself in the AviSynth script where you have control over the conversion. It's hard to get VirtualDub to convert YUV to RGB (where it does most of its filtering) with anything other than a rec.601 matrix. And it handles interlaced YV12 incorrectly.

[SnapPop]

okay guys i'm really appreciate for your help
last thing (maybe a dumb question)
if i'm about to manage the conversion from YUV>RGB with VirtualDub which of the below codecs should i use


and in the avs script what ConvertToRGB() should look like ?

[poisondeathray]

Why are you using vdub? What operations are you doing ? If you convert to RGB in the script beforehand, use lagarith RGB or ut video RGB

Use Rec709 (standard for HD) for everything (converting to RGB, and back to YV12) , and you will be ok

The SD version requires an extra colormatrix conversion (709=>601), buty ou can do that in the script with ConvertToYV12(matrix="Rec601"), if you're coming from the AE RGB export

[SnapPop]

my bad i didn't clarify it well i mean that and the avs script which will contain the filtering and the YUV>RGB conversion, it needs rendering right ? whatever the program i use VD or VDM or AviUtl or whatever it needs a codec to be render with so which of these lossless codecs should i use
and uhm so i use in the script ConvertToRGB(matrix="Rec709") ? okay sounds good

Edit: okay thanks i'm really grateful buddy

[poisondeathray]

I see, you're using vdub to encode only.

Yes, ConvertToRGB(matrix="Rec709") is good for HD source material.

The YUV method is higher quality (less quality loss because you're up and down sampling the chroma), but if you *need* AE, then the RGB trip is unavoidable . The quality loss is visible only with clean colored lines , like clean animation, clean titles, and only to some people (averge joe won't be able to tell). Your audience is anime fans, so they might be able to tell; they tend to be very picky about lines. With live action content is usually difficult to see the degradation unless you zoom in .

[SnapPop]

you completely right about that i knew before i write the thread about the lossy of converting between those colorspaces but i thought there might be a chance to avoid the loss or at least reduce it that's why i asked for your help
again thank you guys you're really a great community keep it up

[poisondeathray]

you completely right about that i knew before i write the thread about the lossy of converting between those colorspaces but i thought there might be a chance to avoid the loss or at least reduce it that's why i asked for your help
again thank you guys you're really a great community keep it up

Maybe I wasn't clear enough - there is a way you can avoid the loss to the main video, by keeping it in YUV .

Since the "overlay" is RGBA, you convert the alpha channel to a YUV matte (a mask, or equivalent to a "luma matte" in AE terminology). Eitherway, if your final destination is YV12 4:2:0 , the overlay portions will be converted from RGBA => YV12 , so that loss of those small sections is unavoidable. But you can avoid loss on the main video by not exposing the main video to RGB conversion

ie. don't export the final composite from AE. You can import the main video if you need timing elements as a guide to do the effects, but the overlay operation won't actually be done in AE. You only export the transparent overlay, complete with " (watermark, karaoke, anime crest simulation bla bla bla)" , but on a transparent background (disable the visibility of the main video)

Is it more work or slower ? No , not really . You need to type an extra line, and will be faster to render in AE (you're mostly rendering blank alpha channel) and actually faster overall (overlay operations in mt_merge are usually faster than in AE)

[newpball]

The YUV method is higher quality (less quality loss because you're up and down sampling the chroma), but if you *need* AE, then the RGB trip is unavoidable . The quality loss is visible only with clean colored lines , like clean animation, clean titles, and only to some people (averge joe won't be able to tell). Your audience is anime fans, so they might be able to tell; they tend to be very picky about lines. With live action content is usually difficult to see the degradation unless you zoom in .

poisondeathray do you have any test results actually showing that YUV -> RGB -> YUV shows quality loss in Premiere Pro/AE?

May I ask what you base your statement on?

Nobody disagrees that if you convert a video from YUV to RGB and then take that converted video and convert it back from RGB to YUV you get quality loss but I was under the impression that in Premiere Pro/AE RGB values are kept at 32 bits per channel RGB accuracy thus converting that back to YUV would not degrade the quality at all.