Xvid creates blocky artifacts! Help!

[david.dgc]

Hi everyone,

Xvid is creating blocks in my avi files. it appears in the highmotion frames and i'm not seeing the cause of it.

i removed all the filters that i was using to see if it was the cause, or where comes fom the blocking artifacts, but i don't guess.
I did a search on the forum and i found that it can be solved by activating "closed GOP" but i don't find that option in xvid settings...



The purpose is create backup file's -- AVI 720p Xvid AC3 with bitperpixel ~0.200 -- of my bluray collection wich can fit cheap dvd-r's. (I know that x264 is better but it's a question of standalone compatibility)


I created a sample of 30sec - 17mb with only crop and resize where you can see the blocks:

http://www.crocko.com/59E30EB512E44D329245CA4202A3ACBD/test.avi



The screenshots of the Xvid settings:



















The logfile of the rip is attached




this problem is hauting me for days.. please help me. i shearched a lot in the forum but i don't seem to find a solution.
thanks to all that can help me.

[david.dgc]

After some new tests i found that the blocks cames from the resize filter.
I removed the resizefilter and the blocks gone away. now, i tested lanczos, spline, bicubic, etc. and all of them re-introduces the blocks...

Anyone knows how i should use the resizer to avoid blocking?




here it is the script that i used for this movie before removing the filters to avoid blocks:



Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
FFT3DFilter(sigma=2.0, plane=0, bw=48, bh=48, bt=3, ow=16, oh=16, sharpen=0, interlaced=false)
Sharpen(1)
Crop(0,4,-0,-4)
Lanczos4Resize(1280,720)
Tweak(sat=1.10)


I'm newbie in using filters, so probably i'm making stupid mistakes..

[davexnet]

Open the script in Virtualdub and look through the movie. (hit the space bar).
Do you see the blocks?
Have you tried encoding at single pass target quantizer = 2 or 3.
Perhaps try the HD profile in Xvid, although I'm not sure it will make too much difference.

[david.dgc]

Open the script in Virtualdub and look through the movie. (hit the space bar).
Do you see the blocks?
Have you tried encoding at single pass target quantizer = 2 or 3.
Perhaps try the HD profile in Xvid, although I'm not sure it will make too much difference.

when i open the .avs on virtualdub the blocks aren't there beacuse the video isn't processed yet... But as i said, they appear after i use the resize. I prefer the 2 pass because i want to target the 0,200 bitperpixel.

i think that the problem is from the settings of xvid or with the resizer that introduces blocks...

[davexnet]

Yes, but if you do the resize in the script, it is active during the preview in Virtualdub.
So in theory, you should be able to see it, if something in the script was the culprit.

I suggest you encode a short section in CQ mode to see if it affects the problem.

[poisondeathray]

The purpose is create backup file's -- AVI 720p Xvid AC3 with bitperpixel ~0.200 -- of my bluray collection wich can fit cheap dvd-r's. (I know that x264 is better but it's a question of standalone compatibility)

x264 is actually more compatible for standalone blu-ray players

And if you are talking DVD players, most won't play 1280x720p. Most are limited to 720x576 dimensions

[jagabo]

Use higher max quantizers. Don't sharpen the video so much. You will enhance noise and grain which will make the video harder to compress. If you're playing on a computer turn on the Xvid deblocking options (and make sure Xvid is doing the decoding). But in the end, you're probably just using too little bitrate. Poisondeathray is right about DVD and Blu-ray compatibility.

[hello_hello]

If you encode using x264 and high profile, level 4.1, then as poisondeathray said you're probably more likely to make a Bluray player compatible encode than using Xvid, the file sizes will be smaller for a given quality and it probably won't be anywhere nearly as blocky.

I'd be thinking it's not the resizer as such causing blocking. Resizers can cause ringing etc but I'm not sure about blocking as such. It's more likely to be a combination of resizing and using Xvid. Persoanlly I'd try a x264 encode using a CRF value of around 20 (or lower). Most of the time at 720p the resulting file size will be small enough to burn to DVD (assuming you convert DTS audio to AAC). If you run an x264 encode on the same small section of video and it looks better, then Xvid is the problem.

[Selur]

Not sure if it helps much, but:
1. restricting the quantizers that much in Xvid can really be a problem for the rate control. (+ max quantizer for p and b frames should normally allowed to be higher than the i frame quantizer, if the quantizers are restricted)
2. I would allow quantizer 1 as minimum
3. I would use bicucic resizing because it is faster (and doesn't make a difference when downsizing)
4. I would use higher Overflow treatment settings
5. if the problem only occurs in high motion scenes, changing the CurveCompression settings might help

Cu Selur

[david.dgc]

The purpose is create backup file's -- AVI 720p Xvid AC3 with bitperpixel ~0.200 -- of my bluray collection wich can fit cheap dvd-r's. (I know that x264 is better but it's a question of standalone compatibility)

x264 is actually more compatible for standalone blu-ray players

And if you are talking DVD players, most won't play 1280x720p. Most are limited to 720x576 dimensions

My bluray player doesn't read x264. neither in mkv, mp4 or even avi. doesn't work. how is that possible? because i tested first.
It only reads HD Divx ou Xvid HD.

[Selur]

you did choose blu-ray compatible settings when encoding to x264, right? (--bluray-compat + resolution&co are compliant, see: http://forum.doom9.org/showthread.php?t=154533)
(+ m2ts might be a better container for blu-ray player compatibility than mkv or mp4)

[david.dgc]

Use higher max quantizers. Don't sharpen the video so much. You will enhance noise and grain which will make the video harder to compress. If you're playing on a computer turn on the Xvid deblocking options (and make sure Xvid is doing the decoding). But in the end, you're probably just using too little bitrate. Poisondeathray is right about DVD and Blu-ray compatibility.

What quantizers do you recommend? I said that because i tested with 2,31 and the xvid encoder emphasis even more the blocks and undersizes the final avi. when I set high max quantizers am i not telling to xvid that he can manage more freely the areas where to apply the bitrate? and this will enhancing the blocks no?

[david.dgc]

If you encode using x264 and high profile, level 4.1, then as poisondeathray said you're probably more likely to make a Bluray player compatible encode than using Xvid, the file sizes will be smaller for a given quality and it probably won't be anywhere nearly as blocky.

I'd be thinking it's not the resizer as such causing blocking. Resizers can cause ringing etc but I'm not sure about blocking as such. It's more likely to be a combination of resizing and using Xvid. Persoanlly I'd try a x264 encode using a CRF value of around 20 (or lower). Most of the time at 720p the resulting file size will be small enough to burn to DVD (assuming you convert DTS audio to AAC). If you run an x264 encode on the same small section of video and it looks better, then Xvid is the problem.

I tested a sample with no filters. i loaded the 1920x1080 into staxrip and converted to xvid 720p and there was no blocks! even with a bitperpixel of 0,180! and the quantizers set to 2-7 to all. when i re-introduce the resizer (tested lanczos, bilinear, bicubic, spline), all of them reintroduces the blocks...

[hello_hello]

Not sure if it helps much, but:
1. restricting the quantizers that much in Xvid can really be a problem for the rate control.

I've read that quite a few times, yet never understood exactly why. AutoGK for example uses quite restrictive quantizer values which it changes according to the compression test result. For a compressibility of around 70%, it'll set all the min and max quantizers to 2 and 3 respectively. At 60% it's 2 and 4 etc. AutoGK has never had issues hitting target file sizes. At least not unless you replace the version of Xvid it installs with a version which is broken in that department.
I can see how if you restrict the quantizers too much it might increase the minimum file size Xvid can achieve to more than the desired file size, and I guess in that respect it could be seen as a problem with rate control, but if you prefer to restrict the quantizers rather than be assured of hitting a tiny file size, maybe that's a good thing?

(+ max quantizer for p and b frames should normally allowed to be higher than the i frame quantizer, if the quantizers are restricted)

What about the i frame boost setting (I can't remember, but I think it's set to 10% or 20% by default)? Wouldn't that boost the number of bits assigned to i frames while still allowing P and B frames to use the same min and max quantizers? And if memory serves me correctly Xvid has some formula for B frame quantizers which can over-ride the values entered. I can't remember exactly what it is, but if for example you set all the min and max quantizers to 2 & 3, you'll still mostly have B frames with a quantizer of 5.

2. I would allow quantizer 1 as minimum

Isn't it really only used for hitting unrealistically large file sizes? I thought it stopped Xvid from producing undersized file sizes when too large an output file size is set, without actually doing anything to increase quality. In that respect, doesn't it just waste bits?

if the problem only occurs in high motion scenes, changing the CurveCompression settings might help

If you're already restricting the quantizers, wouldn't that stop Xvid increasing the quantizer too much in motion scenes?

I'm not trying to be argumentative, just trying to understand the "why" when people recommend certain settings.

[Selur]

I normally use 1-31, since I need the content to be compliant to a specific Level@Profile and this level Xvid with all the choices it wants to make. (if you don't trust Xvid to do a proper quantizer distribution, anything but 1pass cq is really just a waste of time)
To influence the distribution between complex and simple scenes, change the CurveCompression.

which it changes according to the compression test result.

That is the point, unless you adjust the ranges for each source restricting them is a bad idea.
If you just want the quantizers to stay in a specific range and you do not aim for a specific file size you could would normally better use 1pass average bitrate with restricted quantizers, 2pass really is a waste of time than,..

Wouldn't that boost the number of bits assigned to i frames while still allowing P and B frames to use the same min and max quantizers?

this would mainly cause a plusing effect (good quality on I frame, degrades till the next I frame and then it jumps to good quality),.. -> I really would not recommend to change it from the defaults.

And if memory serves me correctly Xvid has some formula for B frame quantizers which can over-ride the values entered. I can't remember exactly what it is, but if for example you set all the min and max quantizers to 2 & 3, you'll still mostly have B frames with a quantizer of 5.

you probably mean:

Bquant = (QdV+QnV)/2 * qR + qO  
 QdV (= forward reference quantizer), QdN (= backward reference quantizer), qR (= QuantizerRatio),
 qO (= QuantizerOffset) 

I'm not sure if this can overwrite the min/max for b-frames (I hope not).
From what I remember the whole thing should be:

Bquant = (QdV+QnV)/2 * qR + qO;
if (Bquant < minB) {
  Bquant = minBquant;
} else if (Bquant > maxBquant) {
  Bquant = maxBquant;
}

[david.dgc]

Not sure if it helps much, but:
1. restricting the quantizers that much in Xvid can really be a problem for the rate control. (+ max quantizer for p and b frames should normally allowed to be higher than the i frame quantizer, if the quantizers are restricted)
2. I would allow quantizer 1 as minimum
3. I would use bicucic resizing because it is faster (and doesn't make a difference when downsizing)
4. I would use higher Overflow treatment settings
5. if the problem only occurs in high motion scenes, changing the CurveCompression settings might help

Cu Selur

1. what quantizers do you suggest (min-max) for the i, p, b frame?

4. where can i set the "higher overflow"?

5. The same thing for the curve compression settings? where is that option?

[david.dgc]

you did choose blu-ray compatible settings when encoding to x264, right? (--bluray-compat + resolution&co are compliant, see: http://forum.doom9.org/showthread.php?t=154533)
(+ m2ts might be a better container for blu-ray player compatibility than mkv or mp4)

My bluray is a philipsbdp2500. i little bit older but if it was possible i would surely use x264 because it's better but when i try to play the x264 in various differents containers, always the same message "not suppported"...

[david.dgc]

you did choose blu-ray compatible settings when encoding to x264, right? (--bluray-compat + resolution&co are compliant, see: http://forum.doom9.org/showthread.php?t=154533)
(+ m2ts might be a better container for blu-ray player compatibility than mkv or mp4)

How can i use this command line on staxrip?

http://www.x264bluray.com/home/720p-encoding

[Selur]

I guess you should be able to use it in StaxRip, but no clue how StaxRip needs to be configured. + you probably need to add some of the settings as some sort of custom command line addition, since I guess the last StaxRip update does not know all of them.

[Selur]

1. what quantizers do you suggest (min-max) for the i, p, b frame?

1-31

4. where can i set the "higher overflow"?
5. The same thing for the curve compression settings? where is that option?

[david.dgc]

I guess you should be able to use it in StaxRip, but no clue how StaxRip needs to be configured. + you probably need to add some of the settings as some sort of custom command line addition, since I guess the last StaxRip update does not know all of them.

Ok, i will try it.

For the quantizers you recommend 1,31 to all. (you already have answered )

what values to you recommend to curve compression and higher overflow (supossing that i maintain 2,7 do quantizers)?

[hello_hello]

I tested a sample with no filters. i loaded the 1920x1080 into staxrip and converted to xvid 720p and there was no blocks! even with a bitperpixel of 0,180! and the quantizers set to 2-7 to all. when i re-introduce the resizer (tested lanczos, bilinear, bicubic, spline), all of them reintroduces the blocks...

Yeah, but that doesn't eliminate the problem being a combination of resizing and encoding using Xvid. I only suggest trying x264 while resizing instead of Xvid as I've re-encoded numerous 1080p videos to 720p using x264 and can't say I've noticed a problem with blockiness. At least not when using a low enough CRF value. While I can't say I've ever encoded a HD video using Xvid, you said it yourself.....
"when i open the .avs on virtualdub the blocks aren't there beacuse the video isn't processed yet"
Therefore logically, resizing isn't causing the blockiness..... at least not on it's own.

Without being able to play with the source I'll admit it's just a suggestion but looking at your sample much of the blockiness I'm seeing looks like typical Xvid to me. Mind you I don't think it's overly bad, but Xvid appears to have turned much of the film grain into small blocks (or "noise"), while some of the faster motion is blocky, which I can't imagine is due to the resizing. For example when Van Damme runs across the road, if you pause the video just before he exits to the right, his yellow jacket is very blocky. It's something you'd probably not notice unless you were looking for it as the motion is quite fast, but I think that's purely Xvid.

One other thought.... are you comparing the original video to the encode using the same monitor/TV and are you 100% sure they're both displaying using the same levels? One of the common causes of complaints regarding blockiness, especially in darker areas, is viewing video with "TV" levels on a TV/monitor which expects "PC" levels. It's even possible you're previewing the video using different levels than when you're viewing the encoded video, and it can be fairly easy not to notice.
For example, the scene in the alley..... on my TV when the correct levels are used the wall on the right looks very dark brown and the blocking is virtually unnoticeable. When sending the TV the wrong levels the wall looks a light brown and the blocking on it is quite noticable. I'm not saying you must be viewing the video with the wrong levels, but it's just another thought as level problems is a common cause of complaints regarding blockiness of encodes.

[Selur]

Damn, I also just realized normal 'curve compression' didn't allow to boost quality in high motion scenes. :/ (I had a modified version to do this, but that one is gone for years now,.. damn)
If it's just a few scene that have this problem boosting bitrate through Zones should help,..

[hello_hello]

this would mainly cause a plusing effect (good quality on I frame, degrades till the next I frame and then it jumps to good quality),.. -> I really would not recommend to change it from the defaults.

Maybe that's why AutoGK changes it to 0%.
I'll admit being lazy I tend to use AutoGK most of the time as I've always been happy with the output, and if I was to use another program I doubt I'd venture away from the Xvid defaults, but I sometime get curious as to why AutoGK does it "this way" while others do it "that way".
Anyway, what you're saying makes perfect sense. I won't sidetrack the thread any further.....
Cheers.

[david.dgc]

I tested a sample with no filters. i loaded the 1920x1080 into staxrip and converted to xvid 720p and there was no blocks! even with a bitperpixel of 0,180! and the quantizers set to 2-7 to all. when i re-introduce the resizer (tested lanczos, bilinear, bicubic, spline), all of them reintroduces the blocks...

Yeah, but that doesn't eliminate the problem being a combination of resizing and encoding using Xvid. I only suggest trying x264 while resizing instead of Xvid as I've re-encoded numerous 1080p videos to 720p using x264 and can't say I've noticed a problem with blockiness. At least not when using a low enough CRF value. While I can't say I've ever encoded a HD video using Xvid, you said it yourself.....
"when i open the .avs on virtualdub the blocks aren't there beacuse the video isn't processed yet"
Therefore logically, resizing isn't causing the blockiness..... at least not on it's own.

Without being able to play with the source I'll admit it's just a suggestion but looking at your sample much of the blockiness I'm seeing looks like typical Xvid to me. Mind you I don't think it's overly bad, but Xvid appears to have turned much of the film grain into small blocks (or "noise"), while some of the faster motion is blocky, which I can't imagine is due to the resizing. For example when Van Damme runs across the road, if you pause the video just before he exits to the right, his yellow jacket is very blocky. It's something you'd probably not notice unless you were looking for it as the motion is quite fast, but I think that's purely Xvid.

One other thought.... are you comparing the original video to the encode using the same monitor/TV and are you 100% sure they're both displaying using the same levels? One of the common causes of complaints regarding blockiness, especially in darker areas, is viewing video with "TV" levels on a TV/monitor which expects "PC" levels. It's even possible you're previewing the video using different levels than when you're viewing the encoded video, and it can be fairly easy not to notice.
For example, the scene in the alley..... on my TV when the correct levels are used the wall on the right looks very dark brown and the blocking is virtually unnoticeable. When sending the TV the wrong levels the wall looks a light brown and the blocking on it is quite noticable. I'm not saying you must be viewing the video with the wrong levels, but it's just another thought as level problems is a common cause of complaints regarding blockiness of encodes.

Thanks for the explanation.


when i try to encode the same sample with x264 the blocks are unnoticeable, that's true. you maybe right it's a combination of xvid limitations with the resizer...the noisy problem that the sample i uploaded has, was treated by the FFT3DFilter and 90% of the noise dissapeared. due to my standalone compatibility issues, mainting xvod as encoder, how can i minimize the blocky artifacts? curve compression higher overflow suggested by Selur?

the comparison with the source video (that i ripped from the bluray to my pc) is made in my pc. And i don't see much difference in terms of levels and noise, only the blocks on my encode.

[Selur]

Maybe that's why AutoGK changes it to 0%.

Probably.

Cu Selur

[david.dgc]

Damn, I also just realized normal 'curve compression' didn't allow to boost quality in high motion scenes. :/ (I had a modified version to do this, but that one is gone for years now,.. damn)
If it's just a few scene that have this problem boosting bitrate through Zones should help,..

How can i do that? boost bitrate through zones? sorry for all that questions but i'm newbie to all this stuff..

[Selur]

create a zone (http://www.divx-digest.com/articles/xvid_setup_page3.html) which spans over the scene and set a weight (W) > 1.

[david.dgc]

create a zone (http://www.divx-digest.com/articles/xvid_setup_page3.html) which spans over the scene and set a weight (W) > 1.

thanks

[hello_hello]

the comparison with the source video (that i ripped from the bluray to my pc) is made in my pc. And i don't see much difference in terms of levels and noise, only the blocks on my encode.

The correct answer would be "I don't see any difference in terms of levels".

Easiest way to know for sure.... encode something with black bars, without cropping them. The black bars should be completely black when viewing the original video (as black as if nothing is being displayed, not dark grey) and they should be equally black when viewing the encode. If they're not.....

I'm not saying levels are your problem, but they could be part of it, so I'd be remiss not to point out the potential there while you're trying to fix blockiness. Where in your sample are you seeing blocky problems? Just so I know if we're seeing the same thing.