VideoMetricsLab - An advanced video quality comparison tool

[4KVCD]

VideoMetricsLab - Calculate VMAF and VMAF NEG on NVIDIA GPUs, and SSIMULACRA2, Butteraugli and ColorVideo VDP on NVIDIA, AMD and Intel GPUs, alongside PSNR, SSIM and XPSNR. Compare encodes with frame-exact playback that switches between the source and each encode instantly.

Here's an app I wrote to analyze video quality between different encodes. This was inspired by FFMetrics, but I wanted many more features for comparisons, like advanced statistics and an ability to easily and instantly subjectively A/B compare encoded videos with the source. This makes subjective video quality comparisons so much easier when you can compare on a frame by frame basis and while the video is playing, and instantly switch to the source or a different encoded video.

So I decided to write my own app, and I'd like to share it with all of you. I hope you all enjoy

Let me know what you think, have any issues, or have feature requests.

Features:

- Calculate VMAF, VMAF NEG, PSNR, SSIM, XPSNR, SSIMULACRA2, Butteraugli, and ColorVideo VDP for multiple test videos.
- Vship integration for GPU acceleration for ColorVideo VDP, SSIMULACRA2, and Butteraugli, on NVIDIA (CUDA), AMD (HIP) and any other GPU with a Vulkan driver
- GPU acceleration for VMAF v0.6.1 and VMAF NEG on NVIDIA GPUs (GeForce GTX 16 and RTX 20 series or newer), with a bundled libvmaf built with CUDA
- libjxl integration for CPU fallback for SSIMULACRA2, and Butteraugli (no CPU support for ColorVideo VDP)
- VMAF v0.6.1 and v1 models for standard, phone, 4K, and HFR viewing scenarios.
- Compare metric curves, statistics, and per-frame scores (per second for ColorVideo VDP).
- Switch instantly between the source and test videos during playback, hold **S** to show the source, or inspect exact frames.
- Inspect bitrate independently by frame, second, or GOP.
- Detect black bars and handle resolution mismatches automatically.
- Use GPU decoding with independent software fallback per input.
- Calculate two test videos in parallel on many-core CPUs.
- Auto save completed results and auto load results when matching videos are loaded again.









The app can be downloaded below.

https://github.com/4KVCD/VideoMetricsLab/releases

[Sharc]

Nice, thanks!

[4KVCD]

v1.1 is released!

VMAF v0.61 and v1 models for standard, phone, 4K, and HFR viewing scenarios are added along with some internal code refactoring for adding future metrics

https://github.com/4KVCD/VideoMetricsLab/releases/tag/v1.1

[4KVCD]

v1.1.1 released. Needed to fix some quick bugs

- Fixed the metric graph clipping VMAF v1 scores above 100.
- Show the application version in the window title: VideoMetricsLab 1.1.1.
- Omit calculation-library version metadata from non-VMAF cache entries

[4KVCD]

v1.2 released. This is a big one!

- Added GPU support for ColorVideo VDP, SSIMULACRA2, and Butteraugli with Vship integration.
- Added CPU fallback for SSIMULACRA2 and Butteraugli with libjxl. ColorVideo VDP has no CPU fallback.
- Added CVVDP per-second scores and graphing alongside its whole-video score.
- Added an Add/remove metrics control to the Videos tab so metric columns can be shown or hidden, and new metrics can be added to completed analyses without recalculating existing results.
- Included various small under-the-hood bug fixes and reliability improvements.

https://github.com/4KVCD/VideoMetricsLab/releases#release-v1.2

[4KVCD]

v1.3 is released!

- Added the window in 20 languages: Simplified and Traditional Chinese, Spanish, Brazilian Portuguese, German, French, Japanese, Russian, Korean, Italian, Polish, Turkish, Arabic, Indonesian, Vietnamese, Ukrainian, Thai, Czech, Hungarian and Dutch. The app opens in Windows' display language, and Settings > Window > Language chooses another. The log, saved results and exported files stay in English.
- Added Intel GPU support for the GPU metrics with Vship's Vulkan build, which runs on any GPU with a Vulkan driver. Settings > GPU metrics > GPU backend chooses the build: Auto (the default) uses CUDA on NVIDIA, HIP on AMD and Vulkan on other GPUs. On Vulkan, SSIMULACRA2 is calculated on the CPU, because Vship 5.1.1's Vulkan build scores it too high (up to 17 points at 4K).
- Updated the GPU metrics to Vship 5.1's API and color handling. Videos tagged BT.2020 SDR, SMPTE 170M (NTSC/DVD), Display P3, ICtCp or YCgCo, and monochrome and alpha videos, are now scored on the GPU instead of falling back to the CPU, and CVVDP no longer fails on them. RGB videos without a transfer tag are now read as sRGB, as Vship's own FFVship does, and their saved GPU scores are recalculated.
- Added a setting to calculate SSIMULACRA2, Butteraugli and CVVDP together in one pass per video (Settings > GPU metrics, off by default). It's better GPU utilization for faster metrics calculations, and it decodes each video once instead of once per metric, but needs much more GPU memory.
- Each metric's score now appears and is saved as soon as it is done, instead of when the whole video is finished. Stopping a run or closing the app loses at most the metric in progress.
- Fixed test frames stamped slightly earlier than the source's being compared with the previous source frame, which gave clusters of near-zero VMAF NEG and XPSNR scores. Each test frame is now compared with the source frame nearest in time. Saved scores of affected videos are kept until they are recalculated.
- Hovering a red Failed cell now says why that metric failed.
- Added a session log under Settings > Storage > Log files, with Export log... (the log files as one .zip) and Copy log (the latest run) buttons.
- The app now checks GitHub for a newer release when it starts, and says so only if there is one. Turn it off in Settings > Window.
- Windows no longer goes to sleep while metrics are being calculated. The screen can still turn off.
- Included various small bug fixes and reliability improvements.

[4KVCD]

v1.4 is released! This brings NVIDIA GPU accelerated VMAF. Enjoy

- VMAF v0.6.1 and VMAF NEG can now be calculated on NVIDIA GPUs.
- GPU metrics now take frames straight from the GPU's video decoder (NVIDIA, Intel and AMD) instead of through FFmpeg, using much less CPU and running noticeably faster.
- Updated Vship's Vulkan build to 5.1.2, with SSIMULACRA2 on Intel GPUs.
- A crash in Vship, libvmaf or the GPU driver while calculating metrics no longer closes the app.
- Reworked the progress line, with each metric's progress and time remaining.
- VMAF NEG now sits beside VMAF v0.6.1.
- Fixed wrong scores for videos with an odd width or height when decoded on NVIDIA GPUs.
- Fixed SSIMULACRA2, Butteraugli and CVVDP comparing mismatched frames when a video has dropped or extra frames.
- Fixed CPU SSIMULACRA2 and Butteraugli scores being far from the GPU's on SDR video. Saved CPU scores are recalculated once.
- Fixed 24 fps videos being accepted against 23.976 fps ones (and 30 against 29.97, 60 against 59.94).
- Fixed GPU metrics on PCs whose only GPU is Intel's.
- Fixed FFmpeg's GPU decoding not being used on AMD GPUs.

[4KVCD]

v2.0 is a major performance update. VMAF, VMAF NEG and VMAF v1 now run on
any GPU, the other metrics and Video Compare are several times faster, and the
app uses far less memory.

### Metric performance

- Added Vulkan support for VMAF v1, so it now runs on any GPU (NVIDIA, AMD or
Intel), up to **17x** faster than on a 24 core CPU.
- Added Vulkan support for VMAF v0.6.1 and VMAF NEG on AMD and Intel GPUs,
**2.3x-2.7x** faster than on a 24 core CPU. On NVIDIA GPUs, choose Vulkan in Settings > GPU metrics > GPU backend.
- PSNR, SSIM and XPSNR are now calculated in the app with libvmaf-fast,
**2.7x-11x** faster than with FFmpeg.
- Added software decoding in the app for videos the GPU can't decode, such as
VVC, making VMAF **1.8x-2.7x** faster for them.
- SSIMULACRA2 and Butteraugli on the CPU are now **7x** faster.

### System performance

- Video Compare now plays 4K HDR at 120 frames a second instead of 14, using a
fifteenth of the CPU.
- Greatly reduced memory use: the app now idles at 180 MB instead of 840 MB,
and VMAF v1 at 4K uses 2.6 GB instead of 13.8 GB.
- Metric Graphs, Video Compare and Settings now open instantly, and the window
stays responsive with hundreds of videos.
- Video Compare no longer uses any CPU while idle.
- Detected black bars are now remembered between sessions.

### New features

- Added a command line, `VideoMetricsLab-cli`, for running comparisons from
scripts.
- Video Compare can now play HDR video in Display P3 colours on the GPU.
- Added the ability to zoom in Video Compare.
- Added an option to highlight each metric's best and worst results in the
Videos tab.
- Added a dark theme, which follows Windows or can be chosen in Settings.
- Added drag and drop for videos, pasting a reference path, reordering rows by
dragging, and sorting by any column.
- File dialogs now open in the folder your videos last came from, and the app
can reopen your last session's videos.