Avisynth - Anime Blu-ray with downscale fails

[Ninelpienel]

Hey, I have here a Blu-ray.
On this Blu-ray are downscale fails (the contours be pixelated on downscaled content (a mistake from the animation studio): http://vgy.me/MOncWN.png).

Does anyone of you now, how I can eliminate this error in Avisynth?

[manono]

What's failing? Are you talking about those jagged lines in the black letters or whatever they are? And you're saying it's in the source and even the Blu-Ray has it?

Better would be for you to provide your script and an untouched 10 second sample of the video where this happens.

[Ninelpienel]

What's failing? Are you talking about those jagged lines in the black letters or whatever they are? And you're saying it's in the source and even the Blu-Ray has it?

Better would be for you to provide your script and an untouched 10 second sample of the video where this happens.

I mean this: http://vgy.me/SxKlp3.png http://vgy.me/fiWaoL.png http://www94.zippyshare.com/v/KE7XzhaU/file.html

I only load the m2ts file (LWLibavVideoSource), no other filter.

[poisondeathray]

Use an antialiasing filter. There are dozens to choose from. Play with the settings until you find something you like

Try AAA

[Ninelpienel]

Use an antialiasing filter. There are dozens to choose from. Play with the settings until you find something you like

Try AAA

No, that's no aliasing. I want a better solution. I want a "smart" solution. A antialiasing filter destroy details.

[poisondeathray]

Use an antialiasing filter. There are dozens to choose from. Play with the settings until you find something you like

Try AAA

No, that's no aliasing. I want a better solution. I want a "smart" solution. A antialiasing filter destroy details.

Fact: Anything you do that will "fix" this artifact will destroy detail. A smart mask will destroy detail even if you paint it frame by frame

Like everything in avisynth, it's about pros/cons. Nothing will fix this without destroying at least some detail. There is no filter that is perfect. This is temporal aliasing (watch it frame by frame, the pattern changes over time)

Or what did you have in mind exactly ? Maybe your expectations are a bit high ?

[LMotlow]

Hey, I have here a Blu-ray.

That's not a BluRay. MKV and 29.97fps at 1080p aren't valid for BluRay. It's just a 1080p HD MKV. Period. It's been changed from the original, if the original was really "BluRay". How did it get to be 29.97fps progressive in an MKV container? Where it did it come from? What resolution are you downscaling to?

I agree, the rough pixel edges shown in your images is aliasing. Whether it was in the original or not is anyone's guess.

[Ninelpienel]

Hey, I have here a Blu-ray.

That's not a BluRay. MKV and 29.97fps at 1080p aren't valid for BluRay. It's just a 1080p HD MKV. Period. It's been changed from the original, if the original was really "BluRay". How did it get to be 29.97fps progressive in an MKV container? Where it did it come from? What resolution are you downscaling to?

I agree, the rough pixel edges shown in your images is aliasing. Whether it was in the original or not is anyone's guess.

What? I muxed the Blu-ray (and splitt the part it with MkvMergGUI) and it's 23,976 FPS: http://vgy.me/mqCc74.png

[LMotlow]

What does "muxed it" mean? Are you referring to your sample? If you muxed it, what happened to the audio? The MKV has 25-frame GOP's, which is not used for official 29.97 BluRay. BluRay uses 1-second and 2-second GOP's. 25-frame GOP's are for 25 1080i PAL BluRay. What you submitted is closer to to AVCHD.

Here are the official BluRay specs: https://www.videohelp.com/hd#tech.

Here are the specs on the sample you submitted:

General
Unique ID                                : 197010173666297412467321356186497337617 (0x9436C77EFA724174BB13BF10FC784911)
Complete name                            : E:\forum\pixelate\00014.mkv
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 21.7 MiB
Duration                                 : 6s 90ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 29.9 Mbps
Encoded date                             : UTC 2015-08-25 20:13:19
Writing application                      : mkvmerge v8.3.0 ('Over the Horizon') 64bit
Writing library                          : libebml v1.3.1 + libmatroska v1.4.2
DURATION                                 : 00:00:06.090000000
NUMBER_OF_FRAMES                         : 146
NUMBER_OF_BYTES                          : 22780238
_STATISTICS_WRITING_APP                  : mkvmerge v8.3.0 ('Over the Horizon') 64bit
_STATISTICS_WRITING_DATE_UTC             : 2015-08-25 20:13:19
_STATISTICS_TAGS                         : BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 6s 89ms
Bit rate mode                            : Variable
Bit rate                                 : 29.3 Mbps
Maximum bit rate                         : 39.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.590
Stream size                              : 21.3 MiB (98%)
Default                                  : Yes
Forced                                   : No

All that aside, how are downscaling? I'll echo the request for the script you're using.

[Ninelpienel]

What does "muxed it" mean? Are you referring to your sample? If you muxed it, what happened to the audio? The MKV has 25-frame GOP's, which is not used for official 29.976 BluRay.

Here are the official BluRay specs: https://www.videohelp.com/hd#tech.

Here are the specs on the sample you submitted:

General
Unique ID                                : 197010173666297412467321356186497337617 (0x9436C77EFA724174BB13BF10FC784911)
Complete name                            : E:\forum\pixelate\00014.mkv
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 21.7 MiB
Duration                                 : 6s 90ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 29.9 Mbps
Encoded date                             : UTC 2015-08-25 20:13:19
Writing application                      : mkvmerge v8.3.0 ('Over the Horizon') 64bit
Writing library                          : libebml v1.3.1 + libmatroska v1.4.2
DURATION                                 : 00:00:06.090000000
NUMBER_OF_FRAMES                         : 146
NUMBER_OF_BYTES                          : 22780238
_STATISTICS_WRITING_APP                  : mkvmerge v8.3.0 ('Over the Horizon') 64bit
_STATISTICS_WRITING_DATE_UTC             : 2015-08-25 20:13:19
_STATISTICS_TAGS                         : BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 6s 89ms
Bit rate mode                            : Variable
Bit rate                                 : 29.3 Mbps
Maximum bit rate                         : 39.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.590
Stream size                              : 21.3 MiB (98%)
Default                                  : Yes
Forced                                   : No

I have put the m2ts in MkvmergeGUI, disabled the audio track, set the Split mode on Global to "split by parts based on timecodes", enter 00:00:00-00:00:06 (i wanted only the first 6 seconds from the video) and clicked on "Start muxing". In your code stands "Frame rate: 23.976 fps" btw.

This is my Avisynth script:

LWLibavVideoSource("X:\X\BDMV\STREAM\00014.m2ts")

[LMotlow]

In your code stands "Frame rate: 23.976 fps" btw.

Yes, I made a typo in my post, which I just correceted to 29.97. But the long text code is a copy MediaInfo's data, not something I made up. MediaInfo says 29.976, as shown. Two HD editors that I use report the MKV's frame rate as 97.976. That's easy enough to correct in processing, but.......

This is my Avisynth script:

LWLibavVideoSource("X:\X\BDMV\STREAM\00014.m2ts")

That's your entire Avisynth script? All it does is open the original video and decode it. It doesn't do anything else.

You haven't told us enough. What are you using to downsample? What are you downsampling to? What final format do you want for output? "Downsample" usually means to resize to a different resolution or sampling rate than the original. Is that what you're trying to do? Poisondeathray's post is correct. Downsampling and anti-aliasing always involve a detail cost to some extent. There is no way around it. The final results depend on the problem and how it's handled.

[Ninelpienel]

In your code stands "Frame rate: 23.976 fps" btw.

Yes, I made a typo in my post, which I just correceted to 29.97. But the long text code is a copy MediaInfo's data, not something I made up. MediaInfo says 29.976, as shown. Two HD editors that I use report the MKV's frame rate as 97.976. That's easy enough to correct in processing, but.......

This is my Avisynth script:

LWLibavVideoSource("X:\X\BDMV\STREAM\00014.m2ts")

That's your entire Avisynth script? All it does is open the original video. It doesn't do anything else.

You haven't told us enough. What are you using to downsample? What are you downsampling to? What final format do you want for output? "Downsample" usually means to resize to a different resolution or sampling rate than the original. Is that what you're trying to do? Poisondeathray's post is correct. Downsampling and anti-aliasing always involve a detail cost to some extent. There is no way around it. The final results depend on the problem and how it's handled.

I think it doesn't matter, what I do as next. There are these fails in the video and i want to fix this.
This are my next steps:

dither_convert_8_to_16()
ly = DebilinearM(1280, 720, lsb_inout=true, cubic=false)
lc = dither_resize16(1280*2, 720*2, src_left=0.25, kernel="blackman")
lu = lc.utoy()
lv = lc.vtoy()
ytouv(lu,lv,ly)
DitherPost()
GradFun3mod(thr=0.6,elast=4,radius=8,ampo=1,mode=6 ,ampn=1.8)

[LMotlow]

Thanks for that, but why are doing it? Do you really want 1280x720 or 1280*2x720*2? That's not BluRay, either, so I guess you want HDD-only playback and generic MKV. That script won't fix rough edges or line twitter.

There are hundreds of possible Avisynth errors. What error are you getting?

[Selur]

Reading:

This filter is designed to reverse the effects of bilinear upsampling. Parameters and sampling pattern try to mimic AviSynth's BilinearResize. Debilinear is only effective on "ideal" or close to ideal conditions. The conditions are that the material has to be a pure bilinear upscale, and you have to know exactly what was the original resolution and the possible sub-pixel shifts. Any other material overlayed on top of the upscale (like native 1080i/1080p credits) will cause ringing or visual defects. Same with compression artefacts, which will be amplified to death, so debilinear is clearly not suited for TV streams or any other low quality sources.

source: http://avisynth.nl/index.php/Debilinear
it doesn't sound like a good idea to use DebilinearM

I'm not really familiar with the whole dither tool set, but downscaling to 1280x720, then the source to 2560x1440 and then merging the color planes, seems like a messed up version of what folks do when they use their 8bit 4k 4:2:0 (or was it 4:2:2) captures to get 1080p 10bit 4:4:4 content.
-> I doubt that the whole:

dither_convert_8_to_16()
ly = DebilinearM(1280, 720, lsb_inout=true, cubic=false)
lc = dither_resize16(1280*2, 720*2, src_left=0.25, kernel="blackman")
lu = lc.utoy()
lv = lc.vtoy()
ytouv(lu,lv,ly)

part will really help and not distorting stuff.

---
Personally I too would use something like EdgeSmoothFast and Hysteria,...

[LMotlow]

@Selur, thanks for more details on DebilinearM. No wonder I didn't like it. Wrong source material.

[Ninelpienel]

Reading:

This filter is designed to reverse the effects of bilinear upsampling. Parameters and sampling pattern try to mimic AviSynth's BilinearResize. Debilinear is only effective on "ideal" or close to ideal conditions. The conditions are that the material has to be a pure bilinear upscale, and you have to know exactly what was the original resolution and the possible sub-pixel shifts. Any other material overlayed on top of the upscale (like native 1080i/1080p credits) will cause ringing or visual defects. Same with compression artefacts, which will be amplified to death, so debilinear is clearly not suited for TV streams or any other low quality sources.

source: http://avisynth.nl/index.php/Debilinear
it doesn't sound like a good idea to use DebilinearM

I'm not really familiar with the whole dither tool set, but downscaling to 1280x720, then the source to 2560x1440 and then merging the color planes, seems like a messed up version of what folks do when they use their 8bit 4k 4:2:0 (or was it 4:2:2) captures to get 1080p 10bit 4:4:4 content.
-> I doubt that the whole:

dither_convert_8_to_16()
ly = DebilinearM(1280, 720, lsb_inout=true, cubic=false)
lc = dither_resize16(1280*2, 720*2, src_left=0.25, kernel="blackman")
lu = lc.utoy()
lv = lc.vtoy()
ytouv(lu,lv,ly)

part will really help and not distorting stuff.

---
Personally I too would use something like EdgeSmoothFast and Hysteria,...

It looks a little bit better, when I use DebilinearM with chroma upscaling. The anime was produced in 720p. When I use Debilinear(1280, 720) for downscale and BilinearResize(1920, 1080) for upscale, it looks identical to the untouched Blu-ray. Thanks for the filter offers.