Field-muxing / demuxing

[Sharc]

Attachment "field-muxing" is a field-muxed YV12 video, i.e one field is video1 and the other field is video2.
Attachment "field-demuxing" are various attempts to recover the 2 videos, either resulting in blends or deinterlacing artifacts.

Any suggestions how to recover the 2 videos cleanly? Thanks.

[lollo]

even fields are contaminated by odd fields, I do not see a simple solution

edit: it appears to me that only the red ball appears in the even fields, maybe a masking with a subtract(even, odd) limited to that region may help, no idea how to implement it

[Sharc]

Thanks. There seems to be no easy solution. Just scratching my head where the root cause of the problem is:
- the source(s)
- the "field-muxing" of the YV12 videos
- the YV12 AVC encoding/compression
- the deinterlacers (apparently they behave differently)
- alltogether ....

[lollo]

How has the file "field-muxing.mp4" been created?

The "sources" for even and odd fields are completely different, so how have them been separated into fields and mixed together?

[jagabo]

The chroma of the two fields has been partially blended together. Here's the two fields with the chroma unblended.


[Attachment 69999 - Click to enlarge]

LWLibavVideoSource("field-muxing.mp4", cache=false, prefer_hw=2) 
SeparateFields()

v1 = SelectEven()
v2 = SelectOdd()

c1 = MergeChroma(v1, Overlay(v1, v2, mode="subtract", opacity=0.33))
c2 = MergeChroma(v2, Overlay(v2, v1, mode="subtract", opacity=0.33))

# c1 and c2 are the cleaned fields

# enable one of the following lines to see a single field, double height
#return(c1.nnedi3(dh=true))
#return(c2.nnedi3(dh=true))

# otherwise, show them stacked
StackVertical(c1, c2)
AssumeFrameBased()

There may still be a tiny bit of the blended chroma left. Adjusting the opacity parameters might help. Or maybe there's some chroma from adjacent frames.

[Selur]

Out of curiosity: How do such files come to be? I mean, is someone creating such stuff by hand or is there some camera or similar creating such content?

[jagabo]

Probably mishandling of interlaced YV12 video. But this one is different than the usual 50:50 blend of the chroma. It only has a ~33 percent blend. I don't know how that came to be.

[Sharc]

How has the file "field-muxing.mp4" been created?
The "sources" for even and odd fields are completely different, so how have them been separated into fields and mixed together?

Out of curiosity: How do such files come to be? I mean, is someone creating such stuff by hand or is there some camera or similar creating such content?

Probably mishandling of interlaced YV12 video. But this one is different than the usual 50:50 blend of the chroma. It only has a ~33 percent blend. I don't know how that came to be.

There exist video surveillance systems using this technology to double the number of cameras and saving storage space. Probably these systems are shipped with their proprietary decoding software, I don't know.
Out of curiosity I made an attempt to emulate such a system using the script:

v1=AVISource("video1.avi") #4:2:2 interlaced YUY2 source1
v1=v1.assumeTFF().converttoYV12(interlaced=true) #converting to YV12

v2=AVISource("video2.avi") #4:2:2 interlaced YUY2 source2
v2=v2.assumeTFF().converttoYV12(interlaced=true) #converting to YV12

f1e=v1.separatefields().selecteven() #even field of video1
f2o=v2.separatefields().selectodd() #odd field of video2
frame=interleave(f1e,f2o).weave() #reinterlace field1 (top) of video1 and field 2 (bottom) of video2
return frame

Maybe the script is flawed, but jagabo's decoding script produces a very clean recovery of the 2 videos.


Edit:
No idea what went wrong with the chroma blending yesterday.
Just did it again and it seems to be ok now regarding the chroma blending:

[lollo]

The new files are ok, no contamination. What happened in your first example causing the (partial) chroma contamination is then unknow. Just don't do it again

In any case, master jagabo was able to repair the problem (looking to his script, the "subtract" idea we had yesterday was not too bad)

[Sharc]

The new files are ok, no contamination. What happened in your first example causing the (partial) chroma contamination is then unknow. Just don't do it again

I found that the strange chroma blending occurs when encoding the script using an NLE. Using ffmpeg or X264 produces clean results. No idea what's happening under the hood of the NLE.

[lollo]

Good to know...

[Cornucopia]

@Sharc,

Can you tell us,
Which NLE was it? (might be good for some to avoid when having to deal with these kinds of footage).

Scott

[Sharc]

@Sharc,

Can you tell us,
Which NLE was it? (might be good for some to avoid when having to deal with these kinds of footage).

Scott

Shotcut, and less blending with Kdenlive. Enabling the filter "color grading" in Shotcut without any adjustments reduced the effect but didn't eliminate it.
I served the script to the NLEs via AVFS.

[jagabo]

Do the NLEs recognize the source is interlaced?

[Sharc]

Do the NLEs recognize the source is interlaced?

In Shotcut I have to set the video properties of the source explicitly to interlaced, TFF as it defaults to progressive. It seems however to have little to nil effect on the chroma blending unless I enable the 'Color Grading' filter as well.
(Btw. I guess that the color grading filter makes a YUV to RGB conversion. Shotcut works in YUV colorspace AFAIK, unless a filter requests the conversion to RGB).
Some more tests pending .....

[Sharc]

Here some results from chroma blending/crosstalk/leakage tests of YV12 interlaced AVC encodes using different encoding tools.

The source 'source_422i.avi' for the tests is interlaced YUV 4:2:2, one field RED the other field GREEN, which was converted to YV12 by the script

LWLibavVideoSource("source_422i.avi")
ConvertToYV12(interlaced=true)

for providing the YV12 4:2:0 input to the AVC encoders.

The attached file 'YV12 encodes.avi' shows the result of the encodes. Left side is the AVC encoded YV12 frame, right side are the separated fields.
Surprisingly, only the commandline x264 and ffmpeg libx264 encodes returned clean separated fields in the original colors (cases 1 and 2).
The NLE Shotcut (case 3) darkened the colors, and the RED field is actually out of RGB gamut. Enabling the 'Color Grading' filter in Shotcut (case 3a) gave fields colors closer to the original RED and GREEN fields, but with a faint horizontal line pattern. The NLE Kdenlive (case 4) is identical to the Shotcut case 3a.

It seems to me like the NLEs added their own magic, or I did something totally wrong .........

[lollo]

Thanks for the interesting results.

I ignore if you did something wrong, but anyhow I always try to avoid a GUI or a NLE for encoding and simply use a ffmpeg command line

[Sharc]

VEGAS recovers the RED and GREEN fields pretty well. Still some horizontal line artefacts with slightly out of gamut RGB values (clipped to 0) though.

[poisondeathray]

Vegas should be ok. The line artifacts are likely because you forgot to interpret the video as "interlaced", so it performs the chroma upsampling (internally vegas works in RGB) and then to 4:2:0 as progressive chroma sampling instead of interlaced.

Shotcut and Kdenlive do not have proper interlaced support the last time I checked

EDIT: yes - vegas is ok when you interpret "source_422i.avi" as "interlaced", and the timeline and export settings match. (It comes in interpreted as progressive and you need to manually assign)

[Sharc]

Shotcut and Kdenlive do not have proper interlaced support the last time I checked

Thanks. It's just too bad. It seems still to be the case.

EDIT: yes - vegas is ok when you interpret "source_422i.avi" as "interlaced", and the timeline and export settings match. (It comes in interpreted as progressive and you need to manually assign)

This is what I am getting then for the 2 fields. No line artefacts, but the the separated fields are far from the original RED and GREEN ....


[Attachment 70191 - Click to enlarge]

[poisondeathray]

I get red and green fields in vegas h264 interlaced 4:2:0 export

Check your 1) file interpretation settings 2) project settings 3) export settings

[Sharc]

I get red and green fields in vegas h264 interlaced 4:2:0 export

Check your 1) file interpretation settings 2) project settings 3) export settings

Thank you for the hints and for the sample.
I did all this, but the problem seems to be my Vegas Movie Studio version 16 which has its limitations and cannot properly convert the 4:2:2 interlaced source to 4:2:0 AVC interlaced.
I have to convert 4:2:2 to 4:2:0 with an avisynth script (ConvertToYV12(interlaced=true)), and because Vegas cannot read Avisynth scripts I use AVFS to produce a virtual .avi file for Vegas (same applies for Shotcut and Kdenlive btw.). The result is that the colors (fields) are eventually about right but with the scanline artefacts, see post#18.
My Vegas Movie Studio can do the conversion without the AVFS workaround using its mpeg2 encoder. The outcome is correct, but mpeg2 instead of AVC.
Anyway, a simple script + commandline (x264 or ffmpeg) does the job perfectly.

[Sharc]

Just a brief update:
With the current Beta of Shotcut the issue of chroma bleeding between the fields of interlaced 4:2:2 and 4:2:0 video is solved.