Quad Core Encoding, Xvid or X264

[kidcash]

Hello,

I have looked at some Xvid benchmarks and seen that from 2 cores to 4 cores there is no advantage, I have looked at some X264 benchmarks and seen that from 2 cores to 4 cores there is an advantage, however I have also read that X264 does take longer to encode than Xvid.

With a Intel Q6600, what would encode the fastest, Xvid or X264?, I am leaning toward X264 but I am not sure.

Thanks,
Kidcash.

[ofbarea]

I have not encoded using X264, however I use DivX all the time and it does provide support for multicore. Since this code is compatiblee with Xvid it could be a good option for you.

[jman98]

Xvid is ALWAYS faster to encode with than X.264. X.264 will give better results at lower bit rates than Xvid, but you will have fewer playback problems if you encode to Xvid. Some standalone DVD players will play Xvid files, depending on what options you used to make it. None of the current ones will play X.264 encoded files.

[Squash]

@kidcash,

I'm running a stock 2.4GHz Q6600 on Vista HP. x264 runs very well on it. I've been backing up DVDs to h264 mp4 using MeGUI with Sharktooth's HQ-Slow profile, adusting bitrate as necessary. On the first pass I get about 90 to 100 fps, on the second, about 50 to 60. This assumes I've not added anything to the avisynth script MeGui generates, which would slow things down. Average 2-pass encode takes about an hour, maybe a little more, for a 90-minute movie.

Since I've already ripped the DVD to my hard drive, I go ahead and back it up again, this time to a divx AVI, with AutoGK (generally using DivX Pro as the encoder rather than xvid mainly to avoid the problems associated with conflicting xvid installs. Seems some of the programs I install bring their own xvid which can cause problems).

I make the DivX backups for compatibility with my current standalone player. I make the H264 backups for future needs. (I expect cheap H264-capable standalones are probably on the way, given that DivX just bought MainConcept. Divx does pretty well from their certification program). Or I may go the HTPC route.

The Divx backups (2-pass) take probably 40 to 45 minutes; about half that if you 1-pass quantizer. But, AutoGK adds a few things like light de-noising and color correction to the avisynth script which are not used by default in MeGui.

Anyway, a head-to-head comparison on encoding speeds of xvid vs. x264 would be very difficult to accurately quantify due to all the encoding options you have with x264 that you don't have with xvid. And as you noted, multiple-processor capability is a big factor, and I haven't noticed much work being done on that lately as regards xvid, but I could have missed it. The xvid version included with AutoGK can use multiple cores, but I don't know if it handles more than 2. Last time I used it, speed seemed to be about as fast as DivX, but I wan't timing anything.

Wrapping up: from what I've seen, DivX/xvid beats x264 on speed on a Q6600, but not by that much (to my tastes, yours may vary). Quality-wise, x264 blows them both out of the water at the bitrates I'm using (~800 to 2000kbps). If it didn't, I wouldn't bother with it.

Of course, on my old AthlonXP single-core computer, no way would I use x264. Too slow.

Try them both, see which you like.

[kidcash]

Yeap sure will,

When I get my Quad I will do a comparison between them, and then post the results up.

[Valnar]

Quality-wise, x264 blows them both out of the water at the bitrates I'm using (~800 to 2000kbps). If it didn't, I wouldn't bother with it.

Very true. To the OP, unless (current) standalone player compatibility is important, x264 buries XVID.

Note that there are some devices out there which can playback H.264/MPEG4/MKV files quite nicely from a PC, like the SageTV HD Extender.

Robert

[kidcash]

Hi,

Well finally got my Quad, it's a Core i7 920.

When encoding to Xvid with the latest version of avidemux it only uses about 30% CPU of only 4 threads of the 8 I have.
See the screenshot below

http://img15.imageshack.us/img15/4773/multithread.png

As we see only 4 threads are being used, and only at about half or 1/3 of it's capacity.

Is there some setting I have not enabled?

Maybe the xvid codec cant use more than 4 threads? But what surprises me is the threads that are being used, are not being used at 100%, only at about 30-50 or so from looking at the graphs.

I am using Windows 7 64 bit Ultimate

Thanks.

[redwudz]

Have you seen this box in the XviD codec settings?:

[kidcash]

Hi,

Not that exact window, But I found the one inbuilt into avidemux under "Configure" for the xvid codec.

Under "Configure" on Avidemux to set the bitrate etc there is a option for the multithreading, default 4, when set to 8 it makes no difference.
Only 4 threads are still being used on task manager, and 30% Load.

Thanks.

[Soopafresh]

It's possibly bottlenecking on the source file decode. FWIW, my CPU (Core2 Duo 8400) peaks at 67%with Avidemux.

[Calidore]

I treated myself to an i7 860 for Christmas, an upgrade from a Q6600 like you. Even setting x264 for 8 threads and telling Virtualdub to multithread, I was still using at best <>60% of processor capacity. The real advantage showed up when encoding multiple files at once, as I'm able to simultaneously run 5 or 6 instances of Virtualdub configured as above before hitting 100% CPU usage.

One file will still encode much faster, but if you're doing a bunch, try running several instances of your software at once.

Best,

Calidore

[jagabo]

With x264 set the thread count to ~1.5 times the number of cores to get maximum throughput. So use 12 instead of 8.

[Calidore]

With x264 set the thread count to ~1.5 times the number of cores to get maximum throughput. So use 12 instead of 8.

Now that you mention it, I remember having heard that before. One possible caveat: The x264 docs say, "The quality loss from multiple threads is mostly negligible unless using very high numbers of threads (say, above 16)." Since we're getting pretty close to that number, should I start paying attention?

And while I'm asking that, what form does this multi-threading quality loss take--noise, detail, etc.? And why does parallel processing affect quality anyway?

Should I start another thread for these questions? I don't want to be a hijacker.

Best,

Calidore

[kidcash]

So really there is no way to make Xvid use 100% CPU unless I run multiple jobs at the same time? Why isn't it loading the CPU to 100%, is it just the coding or something wrong with my settings?

As for the x264, why would the amount of threads used decrease video quality?
Is it coded badly that it has problems using multiple threads that it loses quality?

Thanks.

[jagabo]

So really there is no way to make Xvid use 100% CPU unless I run multiple jobs at the same time?

That's right.

Why isn't it loading the CPU to 100%, is it just the coding or something wrong with my settings?

Settings can make some difference but for the most part Xvid simply wasn't written in a way that lets it use multiple threads very effectively.

Say you have a job to do and it takes you three hours to perform the job. You'd like to get the job done faster so you work out a way to split it into two independent parts, part A and part B. Each part can only be worked on by one person but if you hire someone to help you the job will be done faster as the two of you can work simultaneously. Unfortunately, part A of the job takes one hour to perform and part B takes two hours. No matter how many guys you hire the job will always take two hours to complete. To get the job done faster you'd have to figure out a way to split part B into more parts that can be worked on simultaneously.

When Xvid was originally written single core, single threaded CPUs were the rule. Few programmers were thinking about multithreading in those days. Xvid would have to be largely rewritten to take better advantage of multiple threads. Since there are better compression schemes now nobody wants to invest time in rewriting Xvid.

As for the x264, why would the amount of threads used decrease video quality? Is it coded badly that it has problems using multiple threads that it loses quality?

As I understand it, x264 multithreads by splitting each frame into separate pieces for each thread. For example, a 720x480 frame might (I dont' know exactly how x264 splits the frame) be treated as two 720x240 frames when using two threads, four 720x120 frames when using four threads, etc. To understand why this is a problem you need to know a little bit about how MPEG encoding works...

A significant part of MPEG encoding is motion vectors. Whenever the encoder determines that a block of pixels has moved from one location to another between frames it simply says "move this block of pixels from here to there." That saves bitrate in the output file -- it takes fewer bits to say "move these pixels" than it does to recompress the block of pixels.

When the full frame is split into sections each thread is unaware of the other threads' sections. You lose the ability to have blocks move from one section to another. You also can't use motion vectors as effectively at the very edge of each section (say an object is entering into the section, the new part of the object that appears in the section may have to be encoded because it didn't exist previously). The more sections you have the more motion vector opportunities you lose and the less effectively you can compress.

[kidcash]

Thanks for the explanation, Couple of more questions.

Does xvid do the same thing when using multiple threads? So would quality decrease in xvid using it's scheme of multithreading?

Is there any workaround for the x264 problem for the programmers that is, to modify it so the threads are aware of the other blocks of video? or does it have to be completely re-writen to use another way of splitting the workload (instead of splitting a certain section of video to each thread.

Thanks again.

[Calidore]

Thanks from me also.

The threads vs. instances thing is at least easy to check. Since it's Super Bowl day and I'll be home anyway, I hereby volunteer to do some testing. I'll post the numbers later on tonight or tomorrow.

Best,

Calidore

[jagabo]

I don't think Xvid multithreads with the same technique as x264. Xvid's results appear to be the same no matter how many threads you use (I can only set it up to 4 threads. I don't know if it sets that limit because that's the number of cores I have or if that's its absolute limit).

I'm not an expert on x264. I don't know exactly what could be done with x264. Obviously, the technique I described will not continue to work in the long run. They may be able to use overlapping blocks (it seems they may be doing some of this already, see the "deterministic" setting) or split the job in a different way, say, each thread working on a different group of frames. But brute forcing the job like that won't work with large numbers of threads either. When we get dozens, hundreds or thousands of cores on a CPU you have to start thinking about other issues. You can't just fire up 1000 threads and have the first working on frames 0 to 999, the next on frames 1000 to 1999, etc. 1000 threads trying to access the source file on the hard drive all at once will cause hard drive thrashing, a total disaster as far as performance goes. 1000 threads each accessing large amounts of different memory all at once will cause cache thrashing, another disaster. These are the kind of problems that high performance computing engineers have to think about. And those HPC programs have to be optimized for each particular system's peculiarities.

[kidcash]

Looks like xvid only supports 4 threads max, since my i7 has 8, when changing the setting from 4 to 8, I notice no performance increase.

As for the performance issues with assigning certain frames to the threads. Why would that be a problem, cant it first be read from the drive, cached into the RAM and then accessed from there to avoid any performance issues?

[hydra3333]

Looks like xvid only supports 4 threads max, since my i7 has 8, when changing the setting from 4 to 8, I notice no performance increase.

Hmm, did you take into account your disk performance ? Thrashing ? Using 2 disks 1 for input, 1 for output ? Please consider the range of factors.

[jagabo]

Looks like xvid only supports 4 threads max, since my i7 has 8, when changing the setting from 4 to 8, I notice no performance increase.

If I set the thread count to 8 and close the dialog, then reopen the dialog, it says 4 again. I have a quad core (non hyperthreading) CPU so I don't know if Xvid is limiting the thread count to the number of cores or if 4 is its max.

As for the performance issues with assigning certain frames to the threads. Why would that be a problem, cant it first be read from the drive, cached into the RAM and then accessed from there to avoid any performance issues?

I'm not a programmer for x264 but I suspect that with all the backward and forward prediction going on it's too hard to coordinate multiple threads working in the same area -- ie, what is done with one frame depends on what is done to nearby frames.