VideoHelp Forum
+ Reply to Thread
Results 1 to 11 of 11
Thread
  1. So, I'm trying to encode an anime with x264, but can't preserve lines satisfyingly for the life of me. It's not that the encoded anime looks bad (compared to the source), but lines in darker areas are distorted.

    What are the things I could try to make x264 allocate more bits to these lines and preserve them better? (Not using any filtering is preferable)
    Quote Quote  
  2. Use more bitrate, higher quality settings (for example the slow preset vs. the veryfast preset), aq-mode=2 or 3, --no-psy.
    Quote Quote  
  3. Lower aq strength should put more bits to the edges. (--tune animation lowers it from default 1.0 to 0.6.)

    Is x265 an option for you? IMHO it's much better for your use case if speed and compatibility aren't problems.
    Quote Quote  
  4. @jagabo
    As I said the general quality is good so I don't think I'm using too low a bitrate, It's not high but not low either. My problem is that I want x264 to allocate more bits to these areas without increasing the bitrate, making it distribute more bits to areas I want basically.

    I'm using "veryslow film 10bit". Mind you, I've done some 20 test encodes on this particular anime till now, testing aq-mode 3, other aq methods, aq strengths etc. and I've been able to improve the quality, but it's still not distributing bits exactly where I want it to.
    Could you please explain your reason for suggesting --no-psy? Also I think I've read on doom9 that psy-rd 0 should be used instead of --no-psy.

    @sneaker
    I did test Animation tuning, but it looked worse, I'm gonna test tune film with lowered aq strength next. These are lines within flat areas that get distorted or disappear while other areas look almost identical to the source.
    If I were doing 1080p I'd go x265, but I'm doing 720p, and x265 looks too blurred for my taste for 720p, that was at least my experience with it in the past.
    Quote Quote  
  5. Originally Posted by leonsk View Post
    As I said the general quality is good so I don't think I'm using too low a bitrate
    Low contrast detail in dark areas is the first place you'll see problems with too little bitrate in x264.

    Originally Posted by leonsk View Post
    My problem is that I want x264 to allocate more bits to these areas without increasing the bitrate, making it distribute more bits to areas I want basically.
    If your source is noisy apply some noise reduction before encoding. The less bitrate is used to encode noise the more is available for real details.
    Quote Quote  
  6. Member
    Join Date
    Aug 2017
    Location
    United States
    Search PM
    Originally Posted by sneaker View Post
    Lower aq strength should put more bits to the edges. (--tune animation lowers it from default 1.0 to 0.6.)

    Is x265 an option for you? IMHO it's much better for your use case if speed and compatibility aren't problems.
    From everything I've seen x265 is much more difficult to use when it comes to animation. If there is any grain in the picture then it's even more difficult to find the right tune. x264 will beat it for quality 99% of the time.
    Quote Quote  
  7. It's been some time since I tested it. Back then old/grainy anime (Totoro) were difficult with x265. With modern/clean anime (One Punch Man, Attack on Titan, ...) x265 was much better than x264.

    There is even some x265 modding by the Anime community with fan-made tunings for anime.
    https://down.7086.in/x265-Yuuki-Asuna/
    https://github.com/msg7086/x265-Yuuki-Asuna
    I haven't tested them personally but they look worth checking them out.
    Last edited by sneaker; 18th Jun 2019 at 10:56.
    Quote Quote  
  8. Originally Posted by jagabo View Post
    Low contrast detail in dark areas is the first place you'll see problems with too little bitrate in x264.
    OK, but how would go about improving those dark areas without increasing the bitrate, I mean take bits from other areas?
    I don't think anything I do achieves this in a significant way.

    Originally Posted by jagabo View Post
    If your source is noisy apply some noise reduction before encoding. The less bitrate is used to encode noise the more is available for real details.
    Good idea, have to try this. Care to suggest an appropriate de-noiser for this type of content? One that doesn't destroy much detail.
    Quote Quote  
  9. Originally Posted by sneaker View Post
    ...One Punch Man...
    That's what I'm testing, maybe I'll give x265 another chance after all.

    There is even some x265 modding by the Anime community with fan-made --tune animation.
    https://down.7086.in/x265-Yuuki-Asuna/
    https://github.com/msg7086/x265-Yuuki-Asuna
    I haven't tested them personally but they look worth checking them out.
    Thank you for the links. I had a cursory look, but didn't see no explanation of changes/patches and how they improve anime encoding in particular.

    What about modded x264? Has anybody tested them?
    Quote Quote  
  10. Probably most discussion/documentation is either Chinese or Japanese. If you run --help:
    Code:
    -t/--tune <string>	Tune the settings for a particular type of source or situation:
    	(mid bitrate anime) littlepox/lp,
    	(slower) littlepox++/lp++,
    	(high bitrate anime BD / film) vcb-s/vcbs,
    	(slower) vcb-s++/vcbs++,
    	psnr, ssim, grain, zerolatency, fastdecode
    Code:
    else if (!strcmp(tune, "animation"))
            {
                if (param->bframes + 1 < param->lookaheadDepth) param->bframes++;
                if (param->bframes + 1 < param->lookaheadDepth) param->bframes++;
                param->psyRd = 0.4;
                param->rc.aqStrength = 0.4;
                param->deblockingFilterBetaOffset = 1;
                param->deblockingFilterTCOffset = 1;
            }
            else if (!strncmp(tune, "littlepox", 9) || !strncmp(tune, "lp", 2) ||
                     !strncmp(tune, "vcb-s", 5) || !strncmp(tune, "vcbs", 4)) {
                param->searchRange = 25; //down from 57
                param->bEnableAMP = 0;
                param->bEnableRectInter = 0;
                param->rc.aqStrength = 0.8; //down from 1.0
                if (param->rdLevel < 4) param->rdLevel = 4;
                param->rdoqLevel = 2; //force rdoq to be effective
                param->bEnableSAO = 0;
                param->bEnableStrongIntraSmoothing = 0;
                if (param->bframes + 1 < param->lookaheadDepth) param->bframes++;
                if (param->bframes + 1 < param->lookaheadDepth) param->bframes++; //from tune animation
                if (param->tuQTMaxInterDepth > 3) param->tuQTMaxInterDepth--;
                if (param->tuQTMaxIntraDepth > 3) param->tuQTMaxIntraDepth--;
                if (param->maxNumMergeCand > 3) param->maxNumMergeCand--;
                if (param->subpelRefine < 3) param->subpelRefine = 3;
                param->keyframeMin = 1;
                param->keyframeMax = 360;
                param->bOpenGOP = 0;
                param->deblockingFilterBetaOffset = -1;
                param->deblockingFilterTCOffset = -1;
                param->maxCUSize = 32;
                param->maxTUSize = 32;
                param->rc.qgSize = 8;
                param->cbQpOffset = -2; //better chroma quality to compensate 420 subsampling
                param->crQpOffset = -2; //better chroma quality to compensate 420 subsampling
                param->rc.pbFactor = 1.2; //down from 1.3
                param->bEnableWeightedBiPred = 1;
                if (tune[0] == 'l') {
                    // Mid bitrate anime
                    param->rc.rfConstant = 20;
                    param->psyRd = 1.5; //down
                    param->psyRdoq = 0.8; //down
    
                    if (strstr(tune, "++")) {
                        if (param->maxNumReferences < 2) param->maxNumReferences = 2;
                        if (param->subpelRefine < 3) param->subpelRefine = 3;
                        if (param->lookaheadDepth < 60) param->lookaheadDepth = 60;
                        param->searchRange = 38; //down from 57
                    }
                } else {
                    // High bitrate anime (bluray) or film
                    param->rc.rfConstant = 18;
                    param->psyRd = 1.8; //down
                    param->psyRdoq = 1.0; //same
    
                    if (strstr(tune, "++")) {
                        if (param->maxNumReferences < 3) param->maxNumReferences = 3;
                        if (param->subpelRefine < 3) param->subpelRefine = 3;
                        param->bIntraInBFrames = 1;
                        param->bEnableRectInter = 1;
                        param->limitTU = 4;
                        if (param->lookaheadDepth < 60) param->lookaheadDepth = 60;
                        param->searchRange = 38; //down from 57
                    }
                }
            }
    (from param.cpp)
    Last edited by sneaker; 18th Jun 2019 at 10:55.
    Quote Quote  
  11. Originally Posted by leonsk View Post
    Care to suggest an appropriate de-noiser for this type of content?
    I haven't actually seen the content but something like SMDegrain() in AviSynth:
    https://forum.videohelp.com/threads/393441-Grain-reduction#post2553114
    Quote Quote  



Similar Threads

Visit our sponsor! Try DVDFab and backup Blu-rays!