Editing Mix of 576i and 720p

[Sumsaris]

So it's been awhile since I last dropped by. Well, I've got an issue that likely will be a 'first and only' affair. I've seen a few posts that deal with one of my options but not the either/or. In short I've got an old project completed and stored in 'smart rendered' DV shot with a legacy 3CCD SD Panasonic in pretty decent quality except for perhaps the inherent extra bit of noise that I've learned to deal with. I want to re-edit incorporating new material shot with an I-Phone. The finale output will roughly be 70-80% from the original mini-DV source. My initial gut reaction was to QTGMC and upscale to near 'production' quality x264 (10 crf or so) and place it on a 720/25p timeline, add in the 720/30p I-Phone material and let the program do the rest. I've done plenty of down-scales with Lanczos3-4 but have never up-scaled. It appears the utilities aren't necessarily the same. One thread listed 10 options including two called VideoEnhancer and Instant HD. The only other one I've used is Spline. The new-fangled AI methods are not an option due to hardware concerns. Is there a recommended route to go on this that at least would give me the impression (false?) of maintaining overall quality (extra de-noising followed by your preferred sharpener after the resizing)? I'm especially concerned because there are a fair number of stills in the DV edit. Will those be adversely affected by the upscale and/or will I eventually have to redo them on the timeline? The other possibility would be to load the 576i original edit as is into the editor. But it seems to me it would be disastrous to downscale the I-Phone footage from 720p to 576i. Could a compromise be a halfway method: only convert the DV material to 576p, and then load that along with the I-Phone shoot onto an SD progressive timeline? In short, what would be the ideal trade-off in my case. I've no problem with the back and forth between 4x3 and 16x9. Thanks in advance.

[jagabo]

I would use a 1280x720 50p project. 25p will be flickery and jerky. 50p will keep the smooth motion of the interlaced PAL DV. If you're going to use QTGMC() to deinterlace also use nnedi3_rpow2() to upscale. And maybe a little pre and/or post sharpening. The 1280x720 30p sources will be a little jerky at 50p. You might try using RIFE() (very slow) to convert them to 1280x720 50p.

[Sumsaris]

Ok, thanks jagabo for the quick reply. Unfortunately, you blind-sided me a bit with your advice. 50p project? I'm embarrassed to admit that my legacy NLE will not accept 50p. Only 1440/1920 50i. So perhaps I'm finally forced to go to a more modern editor even if it means throwing out a lot of old habits, filters and dedicated add-ons. I'd prefer not to. Is there a workaround? How bad will this 'flicker and jerk' be? I can of course test it for myself, but I ask because a relatively new Sony HD camera I've begun to use led me to discover this problem with my NLE editor. My work-around (mistake?) was to transcode the 50p Sony files with Handbrake to 25p mp4. I've yet to edit this stuff, but they seem to play back Ok. I realize one drawback will be using any slow motion filters (where I certainly will run up against the jitter problem.) I'm wondering if the avs below could help on the processing of the editor's 25p lagarith output to the final x264 mp4 file. Thanks for any other suggestions.

# Cores=4
# SetMemoryMax(512)
# SetMTMode(3,Cores)
PluginPath = "C:\Program Files (x86)\avisynth 2.5\plugins\"
LoadPlugin(PluginPath+"svpflow1.dll")
LoadPlugin(PluginPath+"svpflow2.dll")
Import(PluginPath+"InterFrame2.avsi")
directshowsource ("f:\xxxxxxx.avi",audio=false)
ConvertToYV12()
#SetMTMode(2)
InterFrame(cores=4,gpu=true)
crop (0,2,0,0)
crop(0,2,0,0)

[poisondeathray]

I'm embarrassed to admit that my legacy NLE will not accept 50p. Only 1440/1920 50i.

What "legacy" NLE are you using ?

[Sumsaris]

Pinnacle Studio 15

[Sumsaris]

Ok, so today I had one of those Homer Simpson 'Doh' moments. Thinking back over my past DV encodes, although most were finalized with qtgmc 50p, many were also done in plain vanilla 25p. And, well, I don't recall any having problems with jerkiness. Motion blur, yes. But the image was basically stable. So with all due respect, jagabo, I'm wondering what your concerns were all about? Anyway, thanks for the tip on nnedi3_rpow2().

[jagabo]

The problem is most visible with bright, high contrast material. Watch these two clips on a 50 Hz TV or monitor.

[Alwyn]

Jagabo, that is amazing.

[Sumsaris]

Hmm, quite right. Yet another reason not to execute rapid pans of prison cell walls at point blank range. Fortunately, I don't have many occasions to do so. But your point is well taken. Again, isn't the avs script I posted meant to precisely minimize this type of phenomenon?

[jagabo]

Hmm, quite right. Yet another reason not to execute rapid pans of prison cell walls at point blank range. Fortunately, I don't have many occasions to do so.

Yes, that was a worst case type of scenario. But it happens with any bright, high contrast, motion.

Again, isn't the avs script I posted meant to precisely minimize this type of phenomenon?

The script synthesizes in-between frames to convert 25p to 50p. That type of motion interpolation often makes very ugly mistakes. But your DV sources are most likely 25i -- they already have 50 different images per second. Using QTGMC will create progressive 50 fps out of them. And if you use a 25p project you'll be throwing away half the 50p frames to make 25p out of them. Or worse, blending the 50p to 25p.

[Sumsaris]

Ok, after a few days in the country to clean out the brain dust, I'm back to attack the problem. To the extent that all my possibilities are compromises of sorts, I think I've found a niche between quality and practicality. Thanks jagabo for nnedi3_rpower2 that got me from 25i SD>25p HD (720p). Compared to my older QTGMC>50p x264 file, the upscale retains identical quality--except for the jitter/motion blur business which I'll return to in a second. As for the Iphone material, it indeed showed (even more) jitter when imported to a 25 progressive timeline. There were four possible choices of action. Jagabo proposed Rife(). If I understand its workings it probably isn't suitable for me since a) it relies on manual user input and selection of the areas to be treated (big hassle) and b) it only works on a 2x basis and thus is perhaps a no-go for 29.97>25. SVPFlow (which I've already used) doesn't seem to have any effect (??). Then I discovered MVTools2 (MSuper + MFlow) which really did a nice job on a couple of pans. Some say the last option, FrameRateConverter, is even better than MVTools. I could not get it to work. Systematically, I got an "MT_inpand: wrong color space, only planar yuv" error in line 164. ConvertToYv12 does nothing. Any ideas? I'm using avisynth 2.6, Masktools2, MVtools2 and FRC version 1.3 (even if inside it seems to sign off as 1.21). An info() returned YUY2 (but also showed 25p for the Iphone) Why did MFlow work but not FRC?

Lastly my intention is to export the 25p project to lagarith and then re-encode using MVTools2 again to x264 and 50p. To this end, I did a sample head on comparison test of SVPFlow and MFlow in 25p>50p (previously upscaled) alongside my 'untouched' DV 25i>50p deinterlaced copy. I chose a static shot with action moving across the screen in three speeds viewed from above: people walking, a trotting horsecart, and a motorbike. Across the board, on the interpolated images MFlow produced a superior result next to SVPFlow. But surprisingly it also produced sharper results compared to the corresponding 50p frames (perhaps due to the added sharpening following the upscale). As regards the speeding motorbike, however, MFlow produced an artifact that was absent in the 50p file. But since the alternating non-interpolated frames of the bike in the three samples were blurred nearly just as much, I wonder overall if it really becomes noticeable in the end. I mean a blurred apple that turns into a blurred pear is finally just a blurred fruit. I also did a test panning over a static subject. More or less the same impression, although less marked due to the inherent blur. Another caveat: the MVTools2 fps conversion seems to have led to a color space/shift of sorts (brighter, with reds going towards orange). The DV>Lagarith>x264 transformations never suffered this alteration.

Again, the preferred route would have been editing in 50p, but... In the end, it seems it's all about balancing out between the improvements in the vast majority of scenes vs the occasional creative blob. Your feedback would be welcome and especially any input on getting FRC to work and color shifts.

[poisondeathray]

a) it relies on manual user input and selection of the areas to be treated (big hassle)

How so ? You convert the clips to the desired framerate. Use an intermediate like prores,cineform,etc.. import to your editor

b) it only works on a 2x basis and thus is perhaps a no-go for 29.97>25.

No, 4.x models are faster and more flexible - they can achieve other framerates - at the expense of slightly lower quality (still better than mvtools2 in general) . RIFE has a fps_num, fps_den argument that works when using the 4.x models

Some say the last option, FrameRateConverter, is even better than MVTools.

It uses mvtools2 too . The main difference is it other options such as artifact masking. If it detects a bad interpolation it can mask, or duplicate, or blend. It has many customizable options over a "basic" mvtools2 script. Rife is still better overall on most material, but slower

Systematically, I got an "MT_inpand: wrong color space, only planar yuv" error in line 164. ConvertToYv12 does nothing. A

Where did you add the ConvertToYV12 ? Post your script

Avisynth 2.6 might be too old, you might need avs+ versions (you definitely need avs+ to use RIFE)

the MVTools2 fps conversion seems to have led to a color space/shift of sorts (brighter, with reds going towards orange).

Sounds like a colormatrix issue. Post your script and workflow details

SD to HD usually involves a Rec.601 to Rec.709 conversion, because "HD" material uses 709 by default .

[Sumsaris]

Did a few more tests to partial resolution. Yes, the color matrix is responsible for the red shift. Very subtle but it's there. I'll probably have to redo the upscales. Or if I applied the 601>709 on the final 720p to 720p export? I did a little test file and it did deepen the reds a bit. Or is this not the kosher way to do it? As for the rest, PoisonDeathRay, I could swear i saw something to the effect about needing to select specific areas in one of the development threads. Maybe it was resolved or it was a side-topic. I admit I only skimmed the vast topic. In passing, how much slower is rife really supposed to be compared with MVtools? Twice, thrice, more?

My script is pretty basic. Oh, and yes, I know DSS is not frame accurate, but since I'm throwing out the audio anyway I felt it wouldn't matter.

DirectShowSource ("d:\movie folder\Img 6849-13.mp4")
Converttoyv12()
FrameRateConverter(25)

Could this be an avisynth issue? I'm content with 2.6 and mvtools as regards the present project (I've less than 10% iphone material in the end). I've got something more ambitious coming up next where maybe rife will be worth it. I would at least attempt to take a stab at FRC though. Thanks

[jagabo]

On my computer FrameRateConverter() is about 5 times faster than RIFE() (without encoding). Some other MVtools' based frame rate converters are more than 30x faster. Of course, it depends on the settings used. And other filtering you're doing. And how slow your encoder is.

[poisondeathray]

Or if I applied the 601>709 on the final 720p to 720p export?

That's the way most people would do it , or if you're going to RGB you can convert the matrix there

If your MP4 is already "YV12" , you probably don't need ConvertToYV12. Or what pixel format is your MP4 ?

how much slower is rife really supposed to be compared with MVtools? Twice, thrice, more?

Much more slower, but worth it IMO . If you have an Nvidia card with tensor cores, the Vapoursynth version is much faster (between 1-4x the cf. avs version depending on HW). Vulkan is a significantly slower for RIFE processing

DirectShowSource ("d:\movie folder\Img 6849-13.mp4")
Converttoyv12()
FrameRateConverter(25)


Could this be an avisynth issue? I'm content with 2.6 and mvtools as regards the present project (I've less than 10% iphone material in the end). I've got something more ambitious coming up next where maybe rife will be worth it. I would at least attempt to take a stab at FRC though. Thanks

MT_impand issue could be an outdated masktools2.x or outdated avs issue . That script should work as-is

[Sumsaris]

I think I'm close to seeing the end of the tunnel. From both jagabo's and PDR's comments regarding processing time, it seems RIFE is clearly out of my league. Especially with my mid-range computer of ten years. And whereas my present project is in SD/720p territory, the next one being full 1080p the hours will quickly turn into days and maybe even a short work week.

As for FRC, as I mentioned, an avs info() script tells me that my source file (Iphone 6) apparently is in YUY2. And the pixels as far as I know are square. But I'm wondering (per PDR's comment) if a masktools version is the culprit. Once in the distant past I believe I had a problem with planar yuv but I don't remember how I solved it. And I think I had to mix and match masktools and/or mvtools as part of the solution. I just checked more closely and in fact I've got avisynth260 MT.dll enabled in the programs folder with standard avisynth260.dll being disabled by renaming it. Also I'd removed both versions of mt_masktools (25 & 26) from the plug-ins folder. Any clues? I'll have to check out different combos with these files.

Otherwise, in lieu of all else I'll just go with the MFlow solution and live with the artefacts. Thanks again for your help.

[poisondeathray]

As for FRC, as I mentioned, an avs info() script tells me that my source file (Iphone 6) apparently is in YUY2.

Probably a mistake with DirectShowSource inserting some filters. No Iphone shoots YUY2 video. You can use mediainfo(view=>text) . If it were me, I would use LSmash . It can open MOV,MP4 without indexing, and is more accurate than Directshow. You can even use GPU acceleration for decoding

https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

And the pixels as far as I know are square. But I'm wondering (per PDR's comment) if a masktools version is the culprit. Once in the distant past I believe I had a problem with planar yuv but I don't remember how I solved it. And I think I had to mix and match masktools and/or mvtools as part of the solution. I just checked more closely and in fact I've got avisynth260 MT.dll enabled in the programs folder with standard avisynth260.dll being disabled by renaming it. Also I'd removed both versions of mt_masktools (25 & 26) from the plug-ins folder. Any clues? I'll have to check out different combos with these files.

Possibly a conflicting version of masktools. I would seriously consider upgrading to avs+ and masktools versions to the pinterf branch . Hundreds of bug fixes, speed improvements in the avs core since then

https://github.com/pinterf/masktools/releases