Using Libplacebo Shaders with Rigaya’s NVENC, QSVENC, and VCEENC for Encodi

[dietboby]

Hello everyone,

I wanted to share some exciting information regarding Rigaya’s encoders: NVENC, QSVENC, and VCEENC.
You can enhance your encoding processes by using libplacebo shaders.
These shaders are widely used for video processing tasks and offer a flexible and powerful way to manipulate video streams.

For those unfamiliar, libplacebo shaders are based on GLSL (OpenGL Shading Language). They allow advanced filtering and transformations on video, such as denoising, upscaling, debanding, and more. These shaders work by leveraging GPU acceleration, enabling high-quality output.

How to Use Libplacebo Shaders with Rigaya's Encoders
To utilize libplacebo shaders, you need to specify them in the option file or the command used by Rigaya’s NVENC, QSVENC, or VCEENC tools.
This allows you to apply a sequence of filters during the encoding process.
Each shader will be processed in the order they are listed, and you can even run the same shader multiple times if needed.

Example for Anime upscaling and enhancement:

--output-res -4x1080
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Denoise_Bilateral_Median.glsl"
--vpp-libplacebo-shader shader="D:\rigaya\shader\hdeband.glsl"
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Thin_HQ.glsl"
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Upscale_CNN_x2_L.glsl",res=960x720
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Upscale_CNN_x2_L.glsl",res=1440x1080

Using the --vpp-libplacebo-shader option, you can define the shaders and any specific parameters, such as resolution or other settings.
For example, depending on your desired outcome, you might apply denoising first, followed by upscaling or thin-line enhancement.

You will need to download the shaders separately, as they are not included in the encoder packages, and ensure you set the path to the directory where they have been extracted.

Performance Considerations
It’s important to note that the encoding process can slow down significantly depending on the complexity of the shaders and the settings applied. Shaders like those in the Anime4K library or other advanced processing tools can be resource-intensive, especially when performing tasks such as upscaling or multiple filtering layers.

The performance impact will vary based on your hardware and the specific shaders you use.
For instance, simpler shaders may have a minor effect on speed, while advanced enhancements might substantially increase the encoding time.

If you’re new to using shaders, I recommend starting with one or two simpler ones to observe their effect on output quality and encoding speed.

Disclaimer
Please note that I’m just a fellow enthusiast sharing this information.
While I’ve done my best to ensure everything is accurate, I’m only human, and there could be errors in my explanation or understanding.
If you encounter any issues or discrepancies, please point them out or refer to the official documentation for further clarification.

I hope this helps you get started using libplacebo shaders to encode with Rigaya’s tools.

Happy encoding!

[dietboby]

Hello everyone,

I was recently asked how I handle improving the quality of HD anime and upscaling SD anime to HD, so I thought I’d share my approach here.
Please note that I’m not an expert—just a fellow user exploring these tools.
Feel free to adjust the settings based on your preferences and needs.

First, here are the settings I use for improving HD material:

 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Clamp_Highlights.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Denoise_Bilateral_Median.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\hdeband.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Restore_CNN_S.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\KrigBilateral.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Thin_HQ.glsl"

And here are my settings for upscaling SD material to HD:

 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Clamp_Highlights.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Denoise_Bilateral_Median.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\hdeband.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Restore_CNN_L.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\KrigBilateral.glsl" 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Upscale_CNN_x2_L.glsl",res=1440x1080 
--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Thin_HQ.glsl"

Download Links for Shaders

Anime4K Shaders
The Anime4K shaders can be found here:

https://github.com/bloc97/Anime4K/blob/master/md/GLSL_Instructions_Windows_MPV.md

Choose optimized shaders based on your GPU:

Lower-end GPUs: (e.g., GTX 980, GTX 1060, RX 570)
Higher-end GPUs: (e.g., GTX 1080, RTX 2070, RX 590, Vega 56, 5700XT, 6600XT)

Guided Chroma Upsampling Shader (KrigBilateral)
For high-quality chroma upsampling, download the igv/KrigBilateral.glsl shader here:

https://gist.github.com/igv/a015fc885d5c22e6891820ad89555637

Debanding Shader (hdeband.glsl)
The hdeband shader for removing banding artifacts can be downloaded here:

https://github.com/AN3223/dotfiles/blob/master/.config/mpv/shaders/hdeband.glsl

Explanation of Settings and Filter Order

Anime4K_Clamp_Highlights.glsl
This shader clamps image highlights to prevent overshoot and reduce ringing artifacts. It ensures that the final output looks clean and smooth, especially when multiple shaders are applied. It’s a great starting point for managing luminance consistency.

Anime4K_Denoise_Bilateral_Median.glsl
Denoising is crucial for improving the clarity of the source material, especially for older or low-quality videos. This shader helps reduce noise without sacrificing too much detail, creating a cleaner base for further processing.

hdeband.glsl
Banding artifacts can ruin the visual quality, especially in gradients. This shader effectively removes banding, ensuring a smoother and more polished output.

Anime4K_Restore_CNN_S.glsl (HD) / Anime4K_Restore_CNN_L.glsl (SD)
These shaders restore fine details and help recover lost features in the image. The _L variant is more suitable for SD material, while the _S variant works better for HD sources.

KrigBilateral.glsl
This shader performs guided chroma upsampling. By using the luma channel as a guide, it ensures more accurate chroma detail in the final output. It’s especially useful for upscaling SD material, where chroma quality might be poor.

Anime4K_Upscale_CNN_x2_L.glsl (SD only)
This is where the actual upscaling happens. It doubles the image resolution using GPU-based super-resolution techniques, maintaining as much detail as possible. If you want a 1920x1080 output, you should modify the resolution setting to 1920x1080 instead of 1440x1080.

Anime4K_Thin_HQ.glsl
Finally, this shader refines line art by making the lines thinner. It’s particularly effective for older anime with thick or uneven lines after upscaling. The HQ version is recommended for its balance of quality and performance.

Notes

Resolution Adjustment for SD Upscaling:
If your goal is to achieve a 1920x1080 resolution, you need to change the resolution parameter in this line:

--vpp-libplacebo-shader shader="D:\rigaya\shader\Anime4K\Anime4K_Upscale_CNN_x2_L.glsl",res=1920x1080

Hardware Considerations:
The performance impact of these settings varies depending on your GPU. The shaders are optimized for different GPU tiers, so choose accordingly (e.g., lower-end or higher-end GPU shaders). For example, the Anime4K project offers several versions of upscaling shaders to fit various hardware capabilities.

Why This Order Matters:
The order of shaders is intentional: luminance clamping prevents issues with subsequent filters, denoising ensures clean input for restoration, and debanding removes artefacts before detail restoration and upscaling. Finally, line thinning is applied last to polish the visuals.

Final Thoughts
These settings work well for me, but they may not be perfect for everyone. Experimenting with different shaders and orders can help you find what suits your specific needs best. I hope this guide is helpful for anyone looking to enhance their anime videos!

Happy encoding! 😊