cuda h264 very bad quality

[azor]

hi I was trying the cuda h264 encoder with mediacoder, and I got a very bad video quality as the result.

I encoded min 5:00 to 6:00 from free movie "elephants dream" and then I compared it to x264.

screenshots:
cuda: http://img37.imageshack.us/img37/4140/vlcsnap2011022400h10m25.png
x264: http://img843.imageshack.us/img843/3776/vlcsnap2011022400h10m57.png

settings:
cuda: http://img193.imageshack.us/img193/8945/cudasettings.png
x264: http://img824.imageshack.us/img824/1130/x264settings.png
x264 command line generated by mediacoder:

# ".\codecs64\x264.exe"  --no-progress --profile high --preset medium --weightp 2 --b-pyramid strict --level 41 --keyint 250 --min-keyint 25 --non-deterministic --bframes 3 --psy-rd 1:0 --bitrate $(VideoBitrate) --qcomp 0.6 --threads auto -o "$(DestFile)" "$(SourceFile)"

the cuda h264 quality is that bad throughout the entire clip. the x264 clip is good.

on other movies and clips, especially with real pictures(no animation movies) I get even worse results with cuda! I even tried ridiculously high bitrates like 6000 or 8000 for movie clips that were in 1280x720p @25fps and had an original bitrate of 4000 kbit/s and I still got really really bad and useless results.

why is the quality so bad? is there something wrong with the settings ? (I'm using the default settings of mediacoder, except that I picked the bitrate, used level 4.1 for x264, and for cuda I choose a higher profile and enabled dynamic gop and cabac).

another questioin: is the cuda h264 encoder built in the GPU or nvidia driver, or are there several cuda h264 encoders?

[RabidDog]

What is source? Try it on a bd-rip. Update nvidia drivers. All presets should be set for maximal quality, @ a much lower bitrate.
What is your decoder? what is your hardware specs? What were the timings for the two encodes?
Animation is far too easy a source .. try a babbling brook with bears and Salmon, or an explosion with fire and ploding oil barrels.

[azor]

I'm using the most recent stable nvidia driver, 266.58_notebook_winvista_win7_64bit_international_ whql.
cuda settings are at very high, maybe max. quality as you can see on my screenshot, I also tried the cuda preset "AVCHD", same result.
lower bitrate (1000kbit/s for instance) obviously makes it look even much much worse.

hardware: intel i3-M330 dual core/4 threads @2.13 Ghz, nvidia GT 330M, 4 GB, win7 64bit
software: mediacoder, decoder MEncoder, I also tried Avisynth (ffdshow) as decoder, again, same, bad result

timings? for x264 medium if that is what you mean. for cuda I don't know if there is a timing settings, but you find all the few options on that screenshot.

I also uploaded 1 min a sample of Big Buck Bunny, encoded with cuda, same settings, 2000 kbit/s, level 5.1 and so on
http://www.mediafire.com/?k0au8wqi9h2b86u

screenshots:
http://img806.imageshack.us/img806/9615/vlcsnap2011022408h00m23.png
http://img69.imageshack.us/img69/313/vlcsnap2011022408h00m35.png

source of the other movie, "Elephant Dream" is the in best quality, it's the original source:

General
Complete name : ED_1280.avi
Format : AVI
Format/Info : Audio Video Interleave
File size : 566 MiB
Duration : 10mn 53s
Overall bit rate : 7 267 Kbps
Writing library : VirtualDub-MPEG2 build 18294/release

Video
ID : 0
Format : MPEG-4 Visual
Format profile : Advanced Simple@L5
Format settings, BVOP : 2
Format settings, QPel : No
Format settings, GMC : No warppoints
Format settings, Matrix : Default (H.263)
Muxing mode : Packed bitstream
Codec ID : XVID
Codec ID/Hint : XviD
Duration : 10mn 53s
Bit rate : 6 810 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 24.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.308
Stream size : 531 MiB (94%)
Writing library : XviD 1.1.0 (UTC 2005-11-22)

I don't have a blu-ray rip available but I tried a 1920x1080p@29.970 fps MPEG2-TS Main@High with a 13.9 Mbps bitrate:
http://img30.imageshack.us/img30/3219/vlcsnap2011022408h32m38.jpg
http://img30.imageshack.us/img30/3185/vlcsnap2011022408h35m53.jpg

[Munster1]

I had a brief look at the Mediacoder website forums, and it seems you are not the only person having CUDA issues. There might be a bug with the encoder using low quality settings. If you have a look over at http://forum.mediacoderhq.com/ you might find more useful info.

[jagabo]

CUDA isn't a very good encoder but your samples are much worse than normal. Obviously, something is going wrong. TomsHardware did a comparison of various h.264 encoders a few weeks ago. I seem to recall them having problems with CUDA initially. You might look there for a fix.

I'm downloading the Big Buck Bunny clip right now. I'll see what I get out of MediaCoder.

After getting the Big Buck Bunny clip (1080p)... I resized to 1280x720 in MediaCoder and compressed with CUDA and the results were normal. Ie, not quite as good as x264 cli but not full of gross errors like your sample.

I used CUDA in single pass VBR mode with the quality slider all the way up. I started with the BD template then manually changed the GOP size to 250 (the default in x264), the I/P/B quantizers to 18/20/22, and set b-frames to 3. I tried using the ffmpeg reader but it crashed after about 20 seconds (the partial result looked ok though). So I switched to DirectShow and that encoded all the way through (Mencoder worked too). The average bitrate of the resulting file was about 2900 kbps. I'm running on a quad core Q6600 with a NVIDIA GTX 460.

Here's a short sample that overlaps with yours.

[deadrats]

another questioin: is the cuda h264 encoder built in the GPU or nvidia driver, or are there several cuda h264 encoders?

the cuda powered encoder that ships with media coder is the reference encoder that comes with the sdk, there are however other cuda powered encoders that are custom coded encoders, main concept features just such an encoder (very good), elemental make a lower end encoder featured in badaboom and an enterprise encoder that ships as a plug in with quadro workstation cards for adobe premiere software (also very good). arcsoft, cyberlink and a few others also have custom coded cuda encoders.

elephants dream, is a very poor test source, unless you actually downloaded the uncompressed source (which is huge), the 1080 samples on the site are previously encoded using either apple's h264 or xvid (in an ogm container) and are very low bit rate to begin with.

if you want a fair test actually download the uncompressed sources:

http://media.xiph.org/ED/

the file labeled "ed-pixlet.mov" at 3.6 gigs is the file you want, in fact i am downloading it right now to perform a test comparison for you.

for future reference, the best settings i have seen with the cuda encoder included with media coder are as follows:

cabac = on

cbr - you want to use a constant bit rate to ensure even quality between I/P frames and B frames, the reference cuda encoder does not produce the same quality B frames that x264 or main concept's software encoder produce.

deblocking = on, the h264 spec allows for inloop deblocking, both the leading h264 encoders have this feature enabled by default

there an option for IDR period (i'm doing this from memory, so it may be slightly different), set this to somehwere between 25 and 30 (it basically set the gop length, with the cuda encoder you don't really want long gop's)

force intra = on

force idr = on

and the reference cuda encoder also has an additional option which for some odd reason no app developer seems to feel is necessary to give access to, namely the -psnr switch, i haven't seen all that documentation on said option but i believe it's a boolean switch, i.e. true/false, i believe you would add -psnr 1 or =1 to the command line.

as soon as i'm done downloading the source mentioned above i will explore it in detail and test against x264's option for enabling psnr.

[deadrats]

just a quick update, i downloaded the latest 2011 RC3 build 5050 x64 and poked around but i can't find any place to modify the command fed to the encoder, so i couldn't manually add the -psnr switch, i'm assuming the cli version of media coder should accept said parameter.

while looking around the option i did find something very interesting that evidently stan is hiding for a surprise as i haven't seen any mention of it anywhere on his site and when i requested support for it in the media coder forums i received no response:

there is support for intel's quick sync encoder!!!

it doesn't support all the features that the intel media sdk offers but it's a start, it's not listed within the drop box that allows you to choose which encoder you want but of you go into "settings" under "video encoder" sure enough you will see it, and if you go into codecs64 you will see the encoder named imsdkenc.exe.

this is great news, a free, open source app that supports intel's quick sync encoder, hopefully others will look at the code and expand on it.

[jagabo]

elephants dream, is a very poor test source, unless you actually downloaded the uncompressed source

Yes, but did you look at the "bad" clip he posted? The problem isn't one of less than optimal settings or a marginal source. There are huge errors. The encoder is screwing up big time.

[sum_guy]

Cuda may attempt to accelerate the encoding process, but does not improve quality, Jagabo is correct, the testing done by Toms Hardware shows an across the board loss in quality with Cuda, the key to good encoding has always been a good source, proper quantizer, good bitrate, and slow encoding, no magic bullets. I know I'm simplifying.

[jagabo]

the testing done by Toms Hardware shows an across the board loss in quality with Cuda

In my own experiments with CUDA I haven't been getting results as bad as Tom's (though I haven't tried to match their methods and settings). There may some problem with different software and hardware setups.

[deadrats]

elephants dream, is a very poor test source, unless you actually downloaded the uncompressed source

Yes, but did you look at the "bad" clip he posted? The problem isn't one of less than optimal settings or a marginal source. There are huge errors. The encoder is screwing up big time.

the uncompressed elephants dream hasn't finished downloading yet but i did test out the latest 64 bit media coder build cuda encoder on a dvd and i don't get anything remotely like the encodes he's getting, there's something else wrong, perhaps the driver version he's using.

[deadrats]

ok, finished downloading the uncompressed elephant's dream, however i am unable to get any player to play it smoothly and media coder chokes on it (it's composed of uncompressed png's put in sequence within a mov container), on teh bright side i did discover that the latest build of media coder does in fact allow you to choose the intel media sdk encoder (for h264) and have been doing some test encodes using 8-9 mb/s samsung 1080p demos as sources and converting to 4 mb/s 720p, must say i'm getting some really good encodes, though slow (under 10 fps) which is to be expected.

can't wait to see what a sandy bridge cpu can do...

[RabidDog]

I thought the encoder test on Toms hardware showed that the speed ups from gfx cards were because of the inbuilt decoding ability. The reason that I wanted timings (of encodes) was that if the cuda was taking say 25secs and the CPU was taking 20mins that might show where the problem lies..
The acid question for any encoder using inbuilt sandy bridge facility's, is if they can be used whilst the inbuilt gfx core is unused (external gfx card) ?

*I personally am very wary of the badaboom encoder, as I think it may have been a very early build of this that did for my Nvdia gfx card (8600GT)(overheating shaders??) I might be completely wrong but

[deadrats]

I thought the encoder test on Toms hardware showed that the speed ups from gfx cards were because of the inbuilt decoding ability. The reason that I wanted timings (of encodes) was that if the cuda was taking say 25secs and the CPU was taking 20mins that might show where the problem lies..
The acid question for any encoder using inbuilt sandy bridge facility's, is if they can be used whilst the inbuilt gfx core is unused (external gfx card) ?

*I personally am very wary of the badaboom encoder, as I think it may have been a very early build of this that did for my Nvdia gfx card (8600GT)(overheating shaders??) I might be completely wrong but

here's the deal:

decoding is separate from encoding, the so called gpu decoders really use a separate ASIC chip that handles decoding duties, this holds true for both nvidia's pure video and amd's uvd.

with such a dedicated hardware decoder, under certain circumstances, the decoder could become a bottleneck, primarily because the chip is only capable of decoding a video stream so fast (i think it's limited to about 60 fps), so if you have a video stream that's relatively easy to decode, such as a dvd spec mepg-2, then you could conceivably hold a transcode back with a really fast cpu. on the other hand, with a demanding video stream, such as a 1080p h264 or vc-1, with MBAFF frames or some other advanced features, you really want to take the decoding workload off the cpu's shoulders.

as for timings of encodes, i did do a pretty thorough test in another thread that showed encoding and decoding cpu usage.

with regards to sandy bridge, the motion estimation calculations are performed by the integrated gpu, that's why if you want to use quick sync you need to have the on board gpu active, however the new chipsets due out soon will allow both the integrated and a discrete gpu to be active at the same time so you'll be able to use QS while you have a third party graphics board installed.

as for overheating the cards shaders, something must be wrong with that video card, there is no way that running a gpu powered transcoding application is over heating the shaders but running a 3d game doesn't (i'm assuming you could play games just fine?).

[azor]

hi thanks for the replies.

I just tried different versions of mediacoder. Initially I used v2011 RC2 64bit, then RC3 64bit which came out today, and afterwards the last non-2011 version, 0.7.5.4799 64.bit.
All have of them have that issue. Then I decided to give the NT CUDA version a shot. This version doesn't work at all, it says that my GPU has no cuda, but this is not true.
http://img819.imageshack.us/img819/9985/gpuzr.png

I almost gave it up but then trying the x86 version came to my mind, and it's working!

so there must be something wrong with the mediacoder h264 64bit cuda encoder, or a compatibility issue, maybe related to the most recent nvidia drivers.
I also found a thread on the mediacoder forum with a guy who has the same issues with the cuda encoder.

[TBoneit]

I thought the encoder test on Toms hardware showed that the speed ups from gfx cards were because of the inbuilt decoding ability. The reason that I wanted timings (of encodes) was that if the cuda was taking say 25secs and the CPU was taking 20mins that might show where the problem lies..
The acid question for any encoder using inbuilt sandy bridge facility's, is if they can be used whilst the inbuilt gfx core is unused (external gfx card) ?

*I personally am very wary of the badaboom encoder, as I think it may have been a very early build of this that did for my Nvdia gfx card (8600GT)(overheating shaders??) I might be completely wrong but

FWIW Badaboom has now released Version 2 that they claim has better speed and quality.

[azor]

I'm still wondering how the cuda encoder works. Is it it built in the GPU or in the driver or does every software has it's own cuda h.264 encoder?

In other words, would I get different results if I would try for instance Badaboom ?

[jagabo]

The GPU has specialized instructions that are good for dealing with video. NVIDIA provides the CUDA library that performs h.264 compression with many different options. As far as I know, everyone uses that library, adding their own user interface giving control over different subsets of the available options. But anyone could write their own h.264 encoder using the GPU's instruction set.

[azor]

ok, so I guess I'll just have to try them.