Is it worth upscaling video?

[sophisticles]

The commonly accepted answer among the more knowledgeable members of this forum is no, you can't add information that is not there,

I am going to prove it.

https://media.xiph.org/video/derf/

I downloaded crowd_run, ducks_take_off, in_to_tree, old_town_cross and park_joy and combined them in Shotcut and exported the quintet as an MJPEG master.

This mimics a traditional workflow where a director oversees the acquisition of various shots and a producer takes the scenes and decides what goes into the movie. when all the editing is done a high quality master is exported and sent out to the various authoring houses for creating the final products, whether it's UHD BD, BD, DVD, MXF or something else.

For this scenario after I exported the MJPEG master, I ingested it into Avidemux and created 2 files, one is supposed to be similar to a UHD BD, encoded as a 2-pass 10-bit 3840x2160p50 x265 and the other is supposed to be similar to a BD, encoded as a 2-pass 8-bit 1920x1080p50 x264.

A few notes, neither files is compliant with the respective specs, I believe UHD BD can be up to 60fps, so the frame rate is fine but the spec calls for HDR10, which this file is not.

The BD spec only allows up to 24fps progressive for NTSC or 29.97 interlaced, but since the source was 50fps and I did not want to introduce additional obstacles to upscaling, I kept the 1080p frame rate at 50.

After each of these had been encoded I loaded the 1080p50 back into Avidemux and upscaled it back to 3840x2160p50.

In order to make the upscale as efficient as possible I used the same filter for downscaling as upscaling, namely Lanczos-3.

I then used ffmpeg to create a video of the native 2160p vs the upscaled version using the blend filter to show the differences between the two.

I think the results speak for themselves.

[johns0]

Nope.

[dellsam34]

The first sample played all green for me, the second and third looked identical to me despite having different resolutions, the 4th sample has more details and more dynamic. So what is the 4th sample supposed to represent? The original file?

[victoriabears]

I have a 14 year old 46inch Sony TV 1080p, does the best job I have seen of showing old material , has a list of filters and to my 70 year old eyes, tested every yearm looks great. Not everything, as said you cannot make HD out of something which does not have the info, but this TV I am very pleased with. I typically use handbrake to create MKV and really difficult material do my best with tmpg mastering works 7.

[sophisticles]

The first sample played all green for me, the second and third looked identical to me despite having different resolutions, the 4th sample has more details and more dynamic. So what is the 4th sample supposed to represent? The original file?

The first sample is a video that uses ffmpeg's blend filter to subtract the similarities and show only the differences.

If two videos are exactly the same you will not see anything, just green, if there are differences you will see the green screen with differences between the two.

It is a way of visually examining how similar two videos are. Here is the command line:

ffmpeg -i HEVC10.mkv -i AVC_upsize.mkv -filter_complex "blend=all_mode=difference" -c:v libsvtav1 -preset 10 -crf 30 -pix_fmt yuv420p10le -color_range pc upsize_vs_original.mkv

The video in question does play green but most definitely does show significant differences between the two videos meaning the upscale was a waste of time.

Keep in mind that these videos have lots of repetitive areas per frame, meaning that it should be very easy to upscale and even still the limitations are stark.

In terms of what the files are:

upsize_vs_native.mkv - this shows the differences between HEVC10.mkv and AVC_upsize.mkv

AVC.mkv - this is supposed to be similar to a commercial BD, but it has a frame rate of 50fps

AVC_upsize.mkv - this is supposed to represent the video most people end up with when they take a source like AVC.mkv and try to upsize it.

HEVC10 - this is supposed to represent a commercial UHD BD.

When people try to upsize video they are normally starting with a commercial DVD, commercial BD, or something they downloaded, but in all cases they are starting with something that has been created from a master that was sent to an authoring house and processed accordingly.

That video in turn has been encoded a number of times if it's sourced from a VOD service or torrent site or similar origins.

Further it is common for a DVD, BD or UHD BD to have been cropped and resized because the master was 4096x1716 or 4096x2160.

You did touch on something I failed to consider, situations where people start with a BD or similar source and during the upsizing they encode with 10-bit hevc.

Here is a file to represent that instance and a comparison video of the results.

I will be uploading another pair of videos, this time upsized using AI, encoded using av1 and a comparison video.

This one is very computationally intensive and will take a while to create.

[Selur]

I only get a green mess for 'HEVC10_vs_HEVC10_upsize.mkv' for both hardware and software decoding. Seems like the file is broken,..
(same for upsize_vs_native.mkv)

[sophisticles]

I only get a green mess for 'HEVC10_vs_HEVC10_upsize.mkv' for both hardware and software decoding. Seems like the file is broken,..
(same for upsize_vs_native.mkv)

What do you mean by "green mess"? Do you mean you each frame is predominantly green with barely visible remnants of a scene, like the attached screenshot?


[Attachment 78549 - Click to enlarge]

If this is what you are seeing then this is correct, the ffmpeg blend filter with difference subtracts the similarities and leaves the differences.

If two videos are exactly alike all you will see is green with nothing else. the more dissimilar the files are the more of what appears to be a scene you will see.

I also did the same test using the so-called AI software upscaler that Avidemux has, namely fast-FSRCNN x2, the results were much the same.

Maybe one day someone will design a lossless downscale filter which by definition would allow us to losslessly upscale.

This experiment has made me wonder about VP9's and AV1's scaling feature, where certain frames are encoded at a lower resolution for storage and upscaled during decode as a way of improving compression efficiency.

I need to look at the code to see what algorithms they are using for the scaling.

[Selur]

Yes, at least to me, it wasn't clear that it should look like it.
I usually prefer to look at it another way and don't use ffmpegs blend.
I usually use something like:

but interleaved: https://imgsli.com/MjU4MTE5
to compare stuff. I'm not used to spot differences of all the color components in a diff over all planes at the same time.

Maybe one day someone will design a lossless downscale filter which by definition would allow us to losslessly upscale.

in general that even theoretical is not really possible (side from edge cases, like using svg where the scaling is the lost information).

This experiment has made me wonder about VP9's and AV1's scaling feature, where certain frames are encoded at a lower resolution for storage and upscaled during decode as a way of improving compression efficiency.

The point there is that the process is not lossless and thus saves data,.... (iirc. either nearest neighbor, or bilinear was used)

I need to look at the code to see what algorithms they are using for the scaling.

do that and let us know

Cu Selur

[dannyboy48888]

Only time I upscale is for encoder inefficiency. Example is ffmpeg for mpeg-2 defaults to a linear quant so high frequency noise ain't represented good with it. If I upscale 480p to. 1080p the dots get more "room" and uses the lower part of the DCT keeping more detail at the cost of bitrate. Mainly use avc now and days as it doesn't have this issue as much but when cpu power is low ($200 laptop ) mpeg-2 is still fast to encode, easy to play, and works on my Blu-ray, roku, and android boxes.

[Bencuri]

The commonly accepted answer among the more knowledgeable members of this forum is no, you can't add information that is not there,

Modern TVs tend to defy this theory. It depends on the source material of course, but some devices can do a pretty good job. It would be useful to know how they do it. Both the upscaling and the frame multiplication tricks.

[Tracer93]

In the year 2001 ... This video was recorded on to a VHS tape and I used my Panasonic EH50 to make a DVD-RAM to transfer the info to my Panasonic DVD burner.
It was a 320x240 avi video with stereo mp3 audio and then a couple of months ago ... I redid it and I used AI on it
Sherrie Austin - Jolene https://www.youtube.com/watch?v=kjwpPI72QiM
I am realizing ... to my right 18 inches away I still have one of the original Panasonic DVD burners in my 2010 MSI back up PC.
My other pc has a ASUS x570-P mobo with a Ryzen 5900x in it with a RTX 2070 Super VGA card.

[sophisticles]

Sorry, but I see nothing in that video that either looks good or even makes AI look like a viable option.

Nearly everything has a very artificial, smoothed out, cartoon like look to it. In fact I suspect I could accomplish nearly the same thing using a cartoonize filter in most NLEs.

What "AI" program did you use? Topaz?

[Selur]

Not knowing the source, either the source is really messed up or the restoration wasn't good.

[Bencuri]

In the year 2001 ... This video was recorded on to a VHS tape and I used my Panasonic EH50 to make a DVD-RAM to transfer the info to my Panasonic DVD burner.
It was a 320x240 avi video with stereo mp3 audio and then a couple of months ago ... I redid it and I used AI on it
Sherrie Austin - Jolene https://www.youtube.com/watch?v=kjwpPI72QiM
I am realizing ... to my right 18 inches away I still have one of the original Panasonic DVD burners in my 2010 MSI back up PC.
My other pc has a ASUS x570-P mobo with a Ryzen 5900x in it with a RTX 2070 Super VGA card.

The question whether worth upscalaing or not depends on the purpose of the project. In case you intend to upload things to Youtube, upscaling is a must as I experienced. Videos with DVD screen sizes are downgraded very drastically by Youtube. A decent DVD video can appear as complete junk. So in case your source material looks kind of okay, and you want it to be watchable on Youtube, you should upscale it with proper methods. Maybe there is a different solution for this, preserving the original screen size of the video on Youtube while maintaining a certain quality, but I have not met any yet. So in such cases upscaling is definately useful, no matter if actually in theory you cannot add data that is missing .

[sophisticles]

Not knowing the source, either the source is really messed up or the restoration wasn't good.

I'm assuming that this was the source:

https://www.youtube.com/watch?v=BfqLnQxqjFY

Edit: I realized this can't be, his was 320x240, so probably worse than this.

[dellsam34]

In another thread he said it's a VHS recorded from probably heavily loaded SD ATSC broadcast transmission where they send 20 digital channels in one single frequency, captured from VHS into PC in 320x240 authored into DVD, ripped from DVD and edited from mpgeg-2 into this mess.