Is Intel really superior to AMD for video editing?

[nic2k4]

Interesting point Jagabo, I think in my case it depends more on the clientele I serve; the cost conscious kind that would rather fix it than ditch it. I should add that I get Macs too (though less of the recent models; better cooling), again people spent so much money on them they'd rather fix them. Their only problem is finding someone to do it at a decent price. Same deal with all the iGizmos and Android crap.

[themaster1]

I´m currently considering assembling a new PC for editing (basically AVCHD video shot with a Panasonic camcorder, editing in Premiere CS6 and authoring DVDs from it and maybe later, bluray discs) I´m aware that the AMD CPUs use more power, run hotter and wind up a few notches behind equivalent Intel CPUs in most benchmarks, but considering that I don´t see them as insurmountable obstacles (I don´t edit 24/7, far from it, so the power consumption is not that important, I also plan to install a better/larger CPU fan so hi temps will also be dealt with and about performance,

There is a benchmark here for x264 encoding

Personnally i have a amd phenom II X4 955 BE, i'm happy with it (playing MGS5 right now ,high settings )with a gtx660 oc.
The down side is that even this "old" processor need a good cooling i use corsair hydro h60.
If you do advanced video edit i guess you should go intel otherwise keep your money

[sophisticles]

Despite it´s title, this thread doesnt intend to provoke a fanboy war. ...................

Need I write what it did provoke ????

I would hardly call this a "fanboy war", at least not on my part. I'm a "fanboy" of a great deal, if I could walk into a store and buy an i5 4590 for $125 with motherboard, then I would be touting the benefits of those processors.

There definitely seems to be some band loyalty present in this thread but nothing too crazy, I have seen some Intel/AMD flame wars in other forums that were epic.

In the end everyone is going to buy what they think is best for them, if they think the AMD "octo-cores" are the best deal, they'll pick that.

On a side note, that 5960x system the other poster put together is sweet, wish I could afford such a beast.

[racer-x]

I've owned both AMD and Intel over the years with good results from both. I'm also a fan of value and make my decisions accordingly. The OP really has small demands (editing AVCHD footage from Panasonic camcorder), any modern PC can easily handle that. So it's a simple matter of getting the best for what you want to spend.

[poisondeathray]

I don't see any fanboyism going on here. It's pretty tame so far. You should visit some tech forums if you want to see a good show

There are no technical reasons why any processor in the segment he is looking at cannot do the tasks he wants

Another thing - I would be wary when looking at synthetic benchmarks, because some are optimized with the Intel compiler, and favor Intel chips. If you're looking at performance data, make sure it's real life situation and applications

[jagabo]

Another thing - I would be wary when looking at synthetic benchmarks, because some are optimized with the Intel compiler, and favor Intel chips.

Yes, Intel has a long history of cheating on benchmarks. For example, their runtime libraries check for "Genuine Intel" in the CPUID (rather than just whether the CPU has support for SSE) to decide whether or not to use SSE optimizations. They are also highly influential over some of the benchmark producers and force them to weigh the sub tests on which Intel performs better more heavily.

http://www.extremetech.com/computing/193480-intel-finally-agrees-to-pay-15-to-pentium-...ng-shenanigans
http://arstechnica.com/gadgets/2008/07/atom-nano-review/6/
http://www.theinquirer.net/inquirer/news/1558372/intel-caught-dodgy-gpu-drivers

Other manufactures do much the same when they can. You should always look at real world benchmarks, especially of the software you will be using.

[sophisticles]

Yes, Intel has a long history of cheating on benchmarks. For example, their runtime libraries check for "Genuine Intel" in the CPUID (rather than just whether the CPU has support for SSE) to decide whether or not to use SSE optimizations. They are also highly influential over some of the benchmark producers and force them to weigh the sub tests on which Intel performs better more heavily.

http://www.extremetech.com/computing/193480-intel-finally-agrees-to-pay-15-to-pentium-...ng-shenanigans
http://arstechnica.com/gadgets/2008/07/atom-nano-review/6/
http://www.theinquirer.net/inquirer/news/1558372/intel-caught-dodgy-gpu-drivers

Other manufactures do much the same when they can. You should always look at real world benchmarks, especially of the software you will be using.

I knew that ATI back in the day and NVIDIA had each engaged in benchmark shenanigans and part of me suspected that most cpu reviews were skewed but I always thought the reviewers were being paid to purposely slant he results a particular way. For me the most telling part is when you see a review of a specific processor, like an Intel and the ads on the main page are for Intel and invariably the review paints the processor in a favorable light.

The counter from the author is always that they have no influence over the ads, they are generated based on content but if you know the algorithm will generate ads based on content and you get paid by the click, then it's obvious that unless your review is positive that people won't click on the ad.

I've also noticed that most websites seem to use almost the exact same testing methods, they all use that silly x264 benchmark from Techarp, they all use Handbrake and they all invariably use encoding tests that make little to no sense.

The sad thing is that the average consumer can do nothing about it, all these review sites have been bought by big companies, Tom's and Anandtech are now owned by the same company, and they have become big business with the advertisements, they control the flow of information and they can slant it anyway they want.

Real world benchmarks can also be manipulated easily, for instance if you do a x264 test with 2 pass, it's been shown that the first pass is strongly influenced by memory bandwidth, whereas the second pass is cpu limited. If you use crf encoding, since crf supposedly uses the same algorithm as 2 pass (another silly claim in and of itself) one would expect a crf test encode to be influenced by available memory bandwidth and thus if you configure an AMD test system with DDR3-1333 instead of DDR-2133 or at least DDR3-1600, you've effectively put it at a disadvantage.

Similarly, my tests have shown that the preset you use heavily skews the encoding results as to which processors are faster, with the faster presets of x264 Intel cpu's are far ahead of AMD's offering but if you crank up the settings and up the workload, the AMD cpu turns the table convincingly.

[TimA-C]

There are pros and cons with both Intel & AMD based systems. If you're building one/having one built then just make sure that, as others have said here earlier, you go for a decent quality Motherboard (I stick pretty much exclusively to ASUS), decent Memory (I use mainly Corsair, but I have had more timing issues with Corsair and AMD systems than any other combination. Having said that, the guy over on the Corsair website is VERY helpful!) and a decent power supply (I've had both good and bad experiences with both cheap 'n cheerful brands and some more expensive ones - the more expensive ones, in my experience, may last longer but often die in a more spectacular fashion!). If you haven't got a decent Motherboard, Memory and PSU you might as well not bother! Once you've got those three sorted, spend a little on the case - airflow and upgradeability is important. Expect to spend a little extra on a better CPU cooler (especially if it's an AMD CPU) and on better quality case fans. Try to make your system as upgradeable as possible - if you're building on a limited budget, buy decent basics first and then upgrade over the following weeks, months and years.

The thing that I don't think anyone has mentioned on the thread yet is that most modern Intel based systems can utilise Intel's QuickSync GPU acceleration which apparently gives MUCH better quality than NVidia's CUDA GPU acceleration. (Not sure how AMD/ATI GPU acceleration compares either in quality or take up by Video software?) So, I would advise spending some time and investigate whether you think this might be a relevant factor to you as (as far as I'm aware) there's no way of adding Intel QuickSync to an AMD based system.

[sophisticles]

The thing that I don't think anyone has mentioned on the thread yet is that most modern Intel based systems can utilise Intel's QuickSync GPU acceleration which apparently gives MUCH better quality than NVidia's CUDA GPU acceleration. (Not sure how AMD/ATI GPU acceleration compares either in quality or take up by Video software?) So, I would advise spending some time and investigate whether you think this might be a relevant factor to you as (as far as I'm aware) there's no way of adding Intel QuickSync to an AMD based system.

Quick Sync is exclusive to Intel as it utilizes portion of the Intel gpu and also some dedicated fixed function hardware built into the cpu. AMD has there own version called VCE, now in version 2 but there is little software that makes use of it and fewer benchmarks or reviews with regards to quality. Version 2 is supposed to be of much higher quality, when I tried to test it with A's Video and an A10-7850k I couldn't get it to work.

The Quick Sync included with Haswell cpu's are supposed to be of better quality than older versions and approach x264 quality wise while being much faster but I have never actually tested this myself.

The CUDA encoders based on the NVIDIA SDK sample were of poor quality when they came out but the Elemental people kept developing it and now it's supposed to be very good, albeit very expensive. The Main Concept CUDA encoder wasn't that bad but Main Concept has dropped support for gpu powered encoders with the latest versions of Total Code.

NVIDIA has added a hardware h264 encoder to it's video cards since their Kepler gpu family and improved on it with Maxwell. With the high end and mid range Maxwells NVIDIA added hardware HEVC encoding as well but there is no software that currently exploits this feature so it's anyone's guess as to how fast it is or what the quality is like.

AMD and Intel are strongly rumored to be including hardware HEVC encoding with their upcoming refreshes of Quick Sync and VCE respectively.

In all honesty, based on what I have seen with my tests of x265 and how well it's coming along development wise as far as quality and speed are concerned, I don't see too much demand for hardware HEVC encoders, unless they happen to be of very quality and are stupid fast.

[jagabo]

for instance if you do a x264 test with 2 pass, it's been shown that the first pass is strongly influenced by memory bandwidth, whereas the second pass is cpu limited. If you use crf encoding, since crf supposedly uses the same algorithm as 2 pass (another silly claim in and of itself) one would expect a crf test encode to be influenced by available memory bandwidth and thus if you configure an AMD test system with DDR3-1333 instead of DDR-2133 or at least DDR3-1600, you've effectively put it at a disadvantage.

No, a CRF encode is much more like the second pass of a 2-pass encode. The first pass of a 2-pass encode is an abr encoding at something close to the veryfast or superfast preset, just to get an idea of the relative bitrate each frame will require during the second pass.

[sophisticles]

No, a CRF encode is much more like the second pass of a 2-pass encode. The first pass of a 2-pass encode is an abr encoding at something close to the veryfast or superfast preset, just to get an idea of the relative bitrate each frame will require during the second pass.

I would like to know where you got this from? According to this:

This mode is also known as the “CRF Mode” or “Constant Quality” mode. It basically works similar to the QP Mode (see above), but it will encode with an average quantizer instead of a constant one. To be more precise, this mode encodes at a constant “rate factor”, which is derived from the specified quantizer. Internally CRF mode uses the same ratecontrol algorithm as x264's ABR mode, only without a target bitrate. The advantage of the CRF mode is that it suits the human perception much better than the QP mode. For example it will raise the quantizers in “fast” scenes where the loss won't be visible anyway and lower the quantizers in “slow” scenes. Therefore the CRF mode should give the same subjective quality as QP mode, but it usually achieves a significant higher compression. It's recommended to prefer CRF mode over QP mode, although CRF is a bit slower. When switching from QP to CRF mode, you may want to slightly lower the quantizer. This should give approximately the same file size as before, but better visual quality! Another important advantage of CRF mode is that it will benefit from adaptive quantization, something that QP mode can't do

I also would like to know how the average quantizer is related to quality, it seems like a rather arbitrary metric and more importantly how exactly do they determine quality since the x264 developers have promoted the notion that metrics such as PSNR and SSIM are not reliable measurements of quality. If they are using some other metric what makes that metric more valid than the two above and why isn't that one used in video comparisons.

It also seems to me that QP mode should effectively eliminate the need for Weighted P, AQ and MB-Tree, the last 2 which are inactive during QP encodes.

BTW, I think we may have highjacked this thread nicely, so let me try and tie this all into the topic...can't do it.

[jagabo]

No, a CRF encode is much more like the second pass of a 2-pass encode. The first pass of a 2-pass encode is an abr encoding at something close to the veryfast or superfast preset, just to get an idea of the relative bitrate each frame will require during the second pass.

I would like to know where you got this from?

Look at the output of the first pass. The settings used are very close to the veryfast or superfast presets.

[poisondeathray]

or use x264 --help

      --slow-firstpass        Don't force these faster settings with --pass 1:
                                  --no-8x8dct --me dia --partitions none
                                  --ref 1 --subme {2 if >2 else unchanged}
                                  --trellis 0 --fast-pskip

I also would like to know how the average quantizer is related to quality

Did you mean in the general sense ? The lower the quantizer (we say "frame" but they are actually mb's), the higher the bitrate, thus the higher the quality. This relationship has been the same for previous encoding formats , MPEG4-ASP (xvid), MPEG2 etc....

since the x264 developers have promoted the notion that metrics such as PSNR and SSIM are not reliable measurements of quality. If they are using some other metric what makes that metric more valid than the two above and why isn't that one used in video comparisons.

This is a big discussion too. In a nutshell , PSNR and SSIM have a fair positive correlation with human perception of "quality". Thus it's good for moderate to large deviations and noting trends. However, it's not a very strong positive correlation. You can have deviations where human subjective assessment of "quality" is very different than what SSIM or PSNR values would have predicted. Also, for encoders that use psy related rate distortion models - they tend to score lower on SSIM and especially PSNR, because of larger mathematical deviation. So even if it "looks" better to human eye, the SSIM and PSNR values caluclated might be lower. Finally, there are slightly different formulas to calculate these values, so there can be bias introduced. But there aren't very many better objective metrics with higher correlation to the human visual system - so they are still used, despite their many flaws

[sophisticles]

PDR:

Thanks for the info with regards to first pass, I never realized that it basically simulates an encode with fast settings.

Did you mean in the general sense ? The lower the quantizer (we say "frame" but they are actually mb's), the higher the bitrate, thus the higher the quality. This relationship has been the same for previous encoding formats , MPEG4-ASP (xvid), MPEG2 etc....

I meant as to how they pick an arbitrary number to define quality, I would like to know what those number actually represent, it seems like they are pulled out of thin air, like someone just said 'I'll use a quantizer of 10' and there doesn't seem to be any rhyme or reason as to how that applies to bit rate or quality.

With regards to PSNR and SSIM here are the formal definitions of each:

Peak signal-to-noise ratio is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation.

The structural similarity (SSIM) index is a method for measuring the similarity between two images. The SSIM index is a full reference metric; in other words, the measuring of image quality based on an initial uncompressed or distortion-free image as reference.

Now if both PSNR and SSIM both say that image x is closer to the original than image y, then doesn't it stand to reason that whichever method produced image x is the more accurate method?

I am familiar with the impact psychovisual rate distortion has on perceived image quality but the same could be said of a really high quality sharpening filter or a high quality noise reduction filter or color correction filter. Simply because the output looks better doesn't mean that it is better in so far as being a more accurate representation of the source.

But it does seem that most people don't want to actually reproduce the source as accurately as possible, it seems that most prefer to get what they see as an improved image. Of course this also begs the question as to why they don't start applying different filtering techniques to their source but since I have derailed this thread enough I'll just stop here.

[jagabo]

Now if both PSNR and SSIM both say that image x is closer to the original than image y, then doesn't it stand to reason that whichever method produced image x is the more accurate method?

In a mathematical sense? Yes. In a visual sense, no. Suppose an encoder was lossless except it shifted the image to the left by one column. To a human viewer that's undetectable. By PSNR and SSIM it's a gigantic error (assuming a detailed picture, not a flat grey plane).

[poisondeathray]

I meant as to how they pick an arbitrary number to define quality, I would like to know what those number actually represent, it seems like they are pulled out of thin air, like someone just said 'I'll use a quantizer of 10' and there doesn't seem to be any rhyme or reason as to how that applies to bit rate or quality.

Quantizer of zero is lossless, quality gets worse and more lossy as the value gets larger. That's the relationship. Yes it is an arbitraray scale. They could have picked a different range like 0-100 instead. Note the scale has be "rescaled" several times in x264 development history. Also the scale for 10bit is different than the one used for 8bit , so not translatable

I am familiar with the impact psychovisual rate distortion has on perceived image quality but the same could be said of a really high quality sharpening filter or a high quality noise reduction filter or color correction filter. Simply because the output looks better doesn't mean that it is better in so far as being a more accurate representation of the source.

But it does seem that most people don't want to actually reproduce the source as accurately as possible, it seems that most prefer to get what they see as an improved image. Of course this also begs the question as to why they don't start applying different filtering techniques to their source but since I have derailed this thread enough I'll just stop here.

"more accurate representation of the source" is a very difficult measurement.

PSNR and SSIM do not necessarily accurately measure "more accurage representation of the source" . Keep in mind they are only two problematic ways of measuring "representation of the source" . A higher SSIM and PSNR values does not always have strong positive predictive value . This is what I mean by only moderate correlation with "quality". Similar to jagabo's example, I can show you many examples where higher values that would predict something would have looked "closer to the source" , in fact clearly look more dissimilar than another encode with lower values. If we had a metric that was perfectly correlated 1:1, then we would be using it. You have to understand the limitations and how they are supposed to be interpreted with other measures of "quality" .

Lossy encoding involves tradeoffs. You have the option to disable or reduce psy if you want.

And you have the option to apply different filtering techniques. You can do whatever you want. Many encoders actually do, with pre processing noise removal

[racer-x]

I finally got my RAM in for a total of 12 GB in my i7 4790. I wanted to stress it a little and decided to edit some 4k HEVC stock (4096 x 2160). What a difference the extra 8 GBs makes!

Here is the live screen recording. https://forum.videohelp.com/attachments/29436-1420238277/editing_4k_hevc.mp4

[sophisticles]

Suppose an encoder was lossless except it shifted the image to the left by one column. To a human viewer that's undetectable. By PSNR and SSIM it's a gigantic error (assuming a detailed picture, not a flat grey plane).

I think you should have a look at the formula for calculating PSNR, I see nothing in the formula to support such a claim:

http://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio

[sophisticles]

I finally got my RAM in for a total of 12 GB in my i7 4790. I wanted to stress it a little and decided to edit some 4k HEVC stock (4096 x 2160). What a difference the extra 8 GBs makes!

Here is the live screen recording. https://forum.videohelp.com/attachments/29436-1420238277/editing_4k_hevc.mp4

Pretty nice rig.

[poisondeathray]

Suppose an encoder was lossless except it shifted the image to the left by one column. To a human viewer that's undetectable. By PSNR and SSIM it's a gigantic error (assuming a detailed picture, not a flat grey plane).

I think you should have a look at the formula for calculating PSNR, I see nothing in the formula to support such a claim:

http://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio

There are slightly different variations on calculating PSNR, but they will all show a huge quality difference with a 1 pixel shift. It' s easy to demonstrate. Did you want a hard example ?

Or maybe a mod would move these last few posts, because they probably don't have much to do with the original thread

[sophisticles]

There are slightly different variations on calculating PSNR, but they will all show a huge quality difference with a 1 pixel shift. It' s easy to demonstrate. Did you want a hard example ?

Or maybe a mod would move these last few posts, because they probably don't have much to do with the original thread

I would love a concrete example because the formula I found for computing PSNR has nothing in it to support such a contention.

And I agree, a mod should move our posts into another thread, we have thread jacked enough.

[jagabo]

I think you should have a look at the formula for calculating PSNR, I see nothing in the formula to support such a claim

Maybe you should put a little thought into it. When the image is shifted to the left by one pixel everywhere there are two adjacent pixels that are different there will now be an error when comparing the two images. Every vertical edge will show an error. The more detail there is in the image the more errors there will be. You can see this with a simple Subtract(last, Overlay(last, x=-1, y=0)).

[sophisticles]

Maybe you should put a little thought into it. When the image is shifted to the left by one pixel everywhere there are two adjacent pixels that are different there will now be an error when comparing the two images. Every vertical edge will show an error. The more detail there is in the image the more errors there will be. You can see this with a simple Subtract(last, Overlay(last, x=-1, y=0)).

You are attempting to apply linear reasoning to a complex calculus formula, look at the explanation on this page:

http://www.mathworks.com/help/vision/ref/psnr.html

I'm sorry but it appears the formula and methodology used to calculate MSE and PSNR takes into account the possibility that the image may have been shifted by a row of pixels.

But if you want proof based on linear reasoning that your contention is wrong consider the possibility of a source image that is 1920x1080 and that is cropped and resized down to 1280x544. Using your logic the calculated MSE between the two images should be infinite and the PSNR between the two should be 0.

I'm sure you are clever enough to test your hypothesis and see if in fact PSNR does in fact go to zero when you crop and resize an image.

[poisondeathray]

There are slightly different variations on calculating PSNR, but they will all show a huge quality difference with a 1 pixel shift. It' s easy to demonstrate. Did you want a hard example ?

Or maybe a mod would move these last few posts, because they probably don't have much to do with the original thread

I would love a concrete example because the formula I found for computing PSNR has nothing in it to support such a contention.

It's pretty easy to test it yourself... take a source, shift it over 1 pixel. In this example I took an image and added 4px borders to it. That was my "original". Then for the test video I added 5 to the right and 3 to the left, instead of 4 and 4

Originally I just did this example the full resolution, but you can't check with the free version of MSU VQMT (it's limited to lower resolutions), so I resized it smaller . So you can use multiple checks with multiple tools. I've included everything in a zip file below, including the original files, encoded files, PSNR logs from ffmpeg

Proper check should aways be done in same colorspace and bitdepth, or at least use the same transformation if checking in a different colorspace or bitdepth. (This example used 8bit RGB, this is important, because of the 1pixel odd crop - it's problematic doing 1px manipulations with chroma subsampled YV12). These tests uses lossless compression (ut video codec, in RGB mode)

Commonly used free methods to test PSNR include avisynth PSNR (Compare() http://avisynth.nl/index.php/Compare , ), MSU VQMT, and ffmpeg

The avs script for a 1 frame test of the "original" with 4px borders, or "1.avi" which can be considered the "original"

ImageSource("Cancun,Mexico.jpg")
bicubicresize(640,480)
addborders(4,4,4,4)
trim(0,-1)

2.avi has 1 extra pixel on the left, 1 fewer on the right. ie. the image is shifted 1 pixel to the right

ImageSource("Cancun,Mexico.jpg")
bicubicresize(640,480)
addborders(5,4,3,4)
trim(0,-1)

You can actually plug in the avs script directly into msu vqmt, and test PSNR-R, G, B, and it will already tell you it's not equivalent

To test the in ffmpeg , the command would be:

2vs1

ffmpeg -i 2.avi -vcodec rawvideo -vf "movie=1.avi,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=2vs1.log" [out]" -f rawvideo -y /NUL

1vs1

ffmpeg -i 1.avi -vcodec rawvideo -vf "movie=1.avi,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=1vs1.log" [out]" -f rawvideo -y /NUL

The benefit of ffmpeg method, is it supports many colorspaces , bitdepths, can batch process many files and is free

These were the aggregate R,G,B results from ffmpeg, for 1vs1 and 2vs1 respectively:
[Parsed_psnr_2 @ 02d487e0] PSNR average:inf min:inf max:inf
[Parsed_psnr_2 @ 02d487e0] PSNR average:22.34 min:22.34 max:22.34

If you 've never used PSNR before, some rough guidelines
"50dB" is considered mastering quality.
"42dB" is considered high quality for broadcast

So that 22.34dB is pretty low. It's a log scale. MSU VQMT will give slightly different results, because their formula is slightly different , but still will be a huge quality difference compared to "infinity" or lossless

[poisondeathray]

It's pretty easy to rationalize in common sense terms too - if you have 2 videos, but one is shifted slightly, if a certain metric said there were both "identical" to each other, it would be wrong, wouldn't it ? If it wasn't sensitive to a simple shift it would be a pretty poor metric

[jagabo]

I'm sorry but it appears the formula and methodology used to calculate MSE and PSNR takes into account the possibility that the image may have been shifted by a row of pixels.

It compares only pixels in the same location: I1(m,n) - I2(m,n)

[sophisticles]

Reread the evolution of this discussion, the argument put forth was that PSNR was a poor measurement of quality because it doesn't track with perceived quality and the example given was that if an image were shifted one pixel width to either side that this would result in a worse PSNR but our eyes wouldn't be able to tell the difference.

I pointed out that the formula takes into account such things but more importantly I would like to flesh out my position with these two possibilities for you guys to consider:

1) Perhaps you would notice a 1 pixel shift if you looked closely enough

2) More importantly, it would seem to me that the PSNR metric is more reliable than our eyes. As you guys have pointed out, most people probably wouldn't notice a 1 pixel shift yet the PSNR formula does. In the case of a pixel shift our eyes would be telling us that everything was fine, the formula would be telling us that something was off. The formula would be right.

It's interesting to note that all codec developers, all of them, including the x264 and x265 developers, use PSNR and SSIM in their internal testing to see how code changes and feature additions effect each metric and more importantly the highly regarded Moscow State University codec test uses both PSNR and SSIM, as well as their own custom metric for determining a winner.

If PSNR and SSIM are good enough for MSU, then they are good enough for me.

[poisondeathray]

Reread the evolution of this discussion, the argument put forth was that PSNR was a poor measurement of quality because it doesn't track with perceived quality and the example given was that if an image were shifted one pixel width to either side that this would result in a worse PSNR but our eyes wouldn't be able to tell the difference.

I pointed out that the formula takes into account such things but more importantly I would like to flesh out my position with these two possibilities for you guys to consider:

1) Perhaps you would notice a 1 pixel shift if you looked closely enough

2) More importantly, it would seem to me that the PSNR metric is more reliable than our eyes. As you guys have pointed out, most people probably wouldn't notice a 1 pixel shift yet the PSNR formula does. In the case of a pixel shift our eyes would be telling us that everything was fine, the formula would be telling us that something was off. The formula would be right.

It's interesting to note that all codec developers, all of them, including the x264 and x265 developers, use PSNR and SSIM in their internal testing to see how code changes and feature additions effect each metric and more importantly the highly regarded Moscow State University codec test uses both PSNR and SSIM, as well as their own custom metric for determining a winner.

If PSNR and SSIM are good enough for MSU, then they are good enough for me.

Ok, but if you plug in that example into any PSNR formula they will typically arrive at something like 25-30dB. That is low low quality. Have you seen what 25-30dB looks like ? Typical youtube video gets ~36-42dB! Remember it's a log scale. The point is the dB value given is not proportional to the "quality" that the average person would say. Sure there is a 1pixel shift that you might or might not "see", but is the quality so much "worse" for the 2nd image?






Nobody is telling you to not use them. As metioned earlier - people still use PSNR, SSIM - I do too. That's fine, they are useful as long as you are aware of how to use them properly , how to interpret them properly, and in what situation to use them. Nobody uses them "blindly" alone as the sole measurement or "proof." You have to correlate with other measures and look at the actual video. Everyone that uses these properly (MSU and x264 devs included) are aware of the limitations in real world usage. Heck, even broadcast engineers, who rely heavily and almost solely on PSNR, know of the limitations. It's a objective measure attempting to quantify a subjective quality - tell me that' s not a problematic idea already At least it's an "objective" and repeatable measure - That's much better in many respects than "he thinks", "she thinks" ...

You can adjust the encoding settings to score higher values on PSNR or SSIM testing, but the picture actually looks worse in many cases. So what do you do? Well you have the option to do whatever you want. Do you want your testing to reflect actual usage and look better, or artificially higher "scores" ?

[poisondeathray]

BTW I uploaded the wrong example for the PSNR, it was a mona lisa picture. I changed everything to reflect the proper "cancun" picture. I must have mixed up examples when trying to resize it smaller so you could test with MSU VQMT. The avg dB for R,G,B was 22.34 accorging to ffmpeg

For reference this is 23.54 dB , supposedly "more similar to #1", or "higher quality" than the 2nd picture above that "only" has 22.34 dB

[sophisticles]

Perhaps ffmpeg's method of computing PSNR is poorly implemented, that third picture looks like garbage. How did you create it? Is it from an I, P, or B frame?

One thing to consider, if your source is all I frames and your encode is of varying length gop chances are the frame that correlates with the time stamp may now be something other than an I frame and since P and B frames are usually encoded with a higher quantizer you are likely to get poor samples.