Hybrid: Looking for adjustments to improve quality

[Compositor]

I'm doing some reencoding of an anime series I have, and I'm trying to improve the quality of my encode. To add to the uncertainty, the only means of judging quality I have is watching the encoded video and deciding, "this looks worse/better than the last attempt." Suggestions here would be appriciated too.

My encodes are HEVC, the "baseline" is crf=14.

So far, I've found that lowering the crf until the resulting bitrate is higher than the source definitely "looks" better.

I've also tried adjusting the curve compression up and down a bit, it "seems" to "look" better at the same crf, but I am uncertain if changing it from default makes a difference if I lower the crf as well.

Another thing I tried is disabling SAO. The source was encoded with SAO disabled, but Hybrid had it enabled by default and the description said it improved quality, so I kept it at first. Now that I'm running tests, I see that an encode at the same crf but with SAO disabled does look "worse" than with SAO enabled. As with curve compression, lowering the crf again seems to improve the appearance.

Pushing the ME and subME higher also seems to make the image sharper. I hope I'm not just imagining things there.

So, anyone have suggestions on what adjustments I can make to improve quality?

[redwudz]

You might try posting some short examples of before and after the encode to help with the question.

[Compositor]

I would, if I could figure out which scenes would make good examples to show you guys. The good, clear scenes aren't an issue in all my tests so far. The not-so-good scenes are like... mirages. I look at them and think, "this doesn't look so good." but I can't tell what's not good about it, sometimes it seems the source isn't much better at that scene anyway, so I can't complain if the encode is not-so-good there. Perhaps I can have something in two or three days.

It doesn't help that my last pair of glasses were badly scratched and I am still getting used to my new pair.

[Selur]

General suggestion, especially when encoding animes: don't try to tweak your x265 settings too much, use some filtering (denoising, deringing, line darkening,...)

[Compositor]

General suggestion, especially when encoding animes: don't try to tweak your x265 settings too much, use some filtering (denoising, deringing, line darkening,...)

I know I'm asking blind since I have yet to provide any samples, but any suggestions on what settings for which filters to use? I've only got the standard Hybrid installation by the way, no extra stuff.

I heard "veryslow" and "grain" are the presets to use for encoding anime, should I give those a try? Though, those were posts from years ago and referring to x264, I think.

[Selur]

• 'no extra stuff' on Windows that means Avisynth and Vapoursynth are both there,... only thing extra there are some audio encoders and some source filters from DG.
• do not assume that settings for x264 are good in x265 (also try googling for 'x265 anime encoding' throws out a bunch of useful threads about this)
• about filtering and animes: you need to set the filters for each series (sometimes episode), the older the anime the more problems there usually are. Deringing, debanding, anti-aliasing are usually needed for all animes unless your source already is really high quality. On old animes derainbowing, dot-crawler removal and denoising is also nearly always needed. Depending on the personal choice color corrections and line darkening is often needed. -> Basically any of the larger message boards has threads about anime filtering and there are also a bunch of sites which help to identify artifacts in animes and suggest filters for Avisynth and Vapoursynth.
  see for example:
  • https://www.aquilinestudios.org/avsfilters/
  • https://kageru.moe/blog/
  • http://www.l33tmeatwad.com/vapoursynth101/using-filters-functions
  .. (simply search for 'Anime filtering Avisynth' and/or 'Anime filtering Vapoursynth' and your should be able to find additional guides.

If you have a hard time with a specific series search whether someone else already filtered it to collect ideas and if you really don't get further ask here or in other boards with helpful users to help with the problem.

In general: If you can show that you at least got an idea of what you are doing and what you want people are more inclined to help. Especially more knowledgeable users, try to avoid posts where the user is to lazy to at least to some search on his own and doesn't even provide details.

-> much luck with you endeavors

Cu Selur

Ps.: If you got old animes/cartoons you will learn to hate interlacing and how most of the old animes/cartoons were mastered.