Xvid dual core encoding speed problem...

[SignedupGuest]

Note: I used one tv episode as a test.

My old CPU: 3200+ 2GHZ
New CPU: X2 3800+ (both 2GHZ) with SSE3.

On my old cpu it took about 6:46 to encode.
On my new CPU it took 6:50 to encode.
(o and using 2 VDM encoding the same episode at same time: 7:20)

I am encoding using Xvid (Nov 1 release). It should be working with dial core but I get about 50% CPU usage (Divx gives me about 75%). What is going on?

I got AMD optimizer and drivers. Xvid should both be using SSE3 and both cores...

[edDV]

...

It should be working with dial core but I get about 50% CPU usage (Divx gives me about 75%). What is going on?

A sure flag that it is working one not both cores. At least you can do somthing else.

[jagabo]

Go to the Xvid config dialog, press the Other Options button at the bottom, On the Encoder tab set Number of Threads to 2.

[redwudz]

jagabo, I'm using Koepi's version 11.08.22, and I don't see that selection anywhere in 'Configure Encoder' when I access the program from the 'Start' menu. Which version has that? BTW, with my dual core 3800+ AMD, it uses about 65% CPU total. split between the two CPUs. One is about 95% and the other about 30-40%. There are some web programs running at the time. It seems to go toward 100% with some settings, mainly higher bitrates.

[SignedupGuest]

Number of Threads is grayed out for me... It set at 0...

edit: SSE3 is not even listed as one of the common in Xvid config...

[jagabo]

I think I'm using this version:

http://www.koepi.org/XviD-1.2.-127-25022006.exe



On my A64 X2 3800+, using VirtualDubMod to convert an MPEG file to Xvid in Fast Recompress mode, went from 78 seconds with 1 thread, to 56 seconds with 2.

[SignedupGuest]

ah tnx... that version did it...

Gave me about 5:30... A little over a minute off (about 20% improvement)...

There still needs improvement for dual core tho...

[celtic_druid]

If you want the multiple threads then it needs to be a 1.2cvs build and the above build from Koepi is old. I'd suggest my most recent build instead.

[jagabo]

It's hard to find a newer build.

[handyguy]

Dual cores just means you have two computers in one, not that your computer is twice as fast. As mentioned, you could do something else with the other core.

[SignedupGuest]

Dual cores just means you have two computers in one, not that your computer is twice as fast. As mentioned, you could do something else with the other core.

No you do not have two computers in one. You have 2 cores in a single CPU. They do have some of there own components, while others are shared (ex: the 4MB cache in the due 2), while others each core needs to take it turn (small example: They cannot use the same bus’es at the same time). Becase these cpu do not have "everything of there own" they will never be 2 times faster. BUT, you can see (for example) 70-80% improvements vs a single core. The perfect example of that is video encoding. The reason you do not see speed increases with your normal applications is because they are not coded 2 use both cores.

But I can guaranty you, it is not like having two computers. Who ever told you that is wrong.

[edDV]

I've recently upgraded to a Core2Duo CPU and find that it rarely revs up both engines due to PCI bus, HDD and USB2 speed bottlenecks. USB hard drives in particular bring the system to a crawl forcing both cores into low % use.

I've even upgraded my network to Gigabit Ethernet. The strange issue there is the CPU load gigabit transfers put on the other computer's CPU. It seems both computers need a fast CPU to get fast transfers.

If you have an application that uses both cores like an encoder and if you use dedicated fast PATA/SATA drives, you can get both cores operating near 100%.

For example, the core loads for an MPeg2 encode from a remote disk on the gigabit ethernet are blocked due to the remote computers CPU maxing at 100%.

[deadrats]

Note: I used one tv episode as a test.

My old CPU: 3200+ 2GHZ
New CPU: X2 3800+ (both 2GHZ) with SSE3.

On my old cpu it took about 6:46 to encode.
On my new CPU it took 6:50 to encode.
(o and using 2 VDM encoding the same episode at same time: 7:20)

I am encoding using Xvid (Nov 1 release). It should be working with dial core but I get about 50% CPU usage (Divx gives me about 75%). What is going on?

I got AMD optimizer and drivers. Xvid should both be using SSE3 and both cores...

you need to start using an application that can use both cores to their max. avidemux has xvid and x264 built in and can launch up to 4 threads (if you had a quad core). what it does is it "slices" the video stream into 2 equal sections (or as many threads as you launch) and encodes each section on a seperate core.

having said this, xvid is not the best codec when it comes to multithreading as some find that the data tends to get ping-ponged back and forth between cores.

i recommend going with x264 with either megui or avidemux as either option will make better use of both the cores in your system.

[SerbianBoss]

I have also dual cire amd 3800+ but that option in xvid is locked. What to do to unlock ?

[SerbianBoss]

I succeed with older version 1.2-127 Februar 25.2006.

Bit with latest version 1 Novembar 2006 this option is locked. But i dont know why ?

[jagabo]

The Nov 1 2006 release is a bug fix realease of version 1.1.x. It does not support multithreading. You need 1.2.x (still in "unstable relase" status) to use multithreading. The latest 1.2 I could find was the one I linked to earlier. celtic_druid says he has a later build but didn't supply a link.

[SignedupGuest]

edDV:
Yeah both HDD will probably not go really high unless you go out of your way to get a dual core application. I just did a test with a 4GB file and I found I transferred 40.9MBps from one HDD to another. Also apparently USB2 is 60MBps (I do not have time 2 test if it is slower then 40.9MBps)…

I cannot see the HDD being a limit at 40.9MBps… My video for my test was 180MB, so in theory it would take 5 seconds to load into ram (not that it would ever load the whole thing at once (or else you could only edit vids as big as your available free memory))… I am guessing these videos editors load big chunks of the video before it even needs them so it does not have long as wait times for the HDD (I wonder how many cycle it takes to access HDD to ram)....

deadrats:
Might give that a try later... VDB is so convenient it would have 2 be a good application for me 2 switch... I also edit 720p/1080i... but xvid does a good job with them...

[jagabo]

Hard drives aren't a limiting factor unless you are using uncompressed or losslessly compressed video. Xvid is just poorly multithreaded.

[jagabo]

avidemux has xvid and x264 built in and can launch up to 4 threads (if you had a quad core). what it does is it "slices" the video stream into 2 equal sections (or as many threads as you launch) and encodes each section on a seperate core.

That seems like a rather poor multithreading technique for any codec that uses motion vectors.

[edDV]

I was pointing out that Gigabit ethernet and USB2 seem to demand a lot of CPU capacity. With Gigabit ethernet, the CPU at the other end of the network seems to hold down transfer rate to far less than Gigabit rates. The Core2 CPU is loafing.

I'm also finding that internal PATA/SATA drives allow much higher utilization of the Core2Duo CPU compared to USB2 drives.

Is what I'm seeing normal? I was expecting the system to be CPU bound but instead I'm finding the CPU is system bound.

[jagabo]

I ran some tests. I converted a DV AVI file to XVID (default settings) with VirtualDubMod (fast recompress). In one case both the input and output files were across the network (100 Mb/sec) from/to the same computer/drive. In another case both files were on the same local drive:

DV to Xvid, Net to Net: 101 seconds
DV to Xvid, HD to HD: 96 seconds

The same experiment with an MPEG2 (6 Mb/sec) file:

MPEG2 to Xvid, Net to Net: 54 seconds
MPEG2 to Xvid, HD to HD: 53 seconds

[deadrats]

avidemux has xvid and x264 built in and can launch up to 4 threads (if you had a quad core). what it does is it "slices" the video stream into 2 equal sections (or as many threads as you launch) and encodes each section on a seperate core.

That seems like a rather poor multithreading technique for any codec that uses motion vectors.

it's actually the easiest to implement and the most efficient. i don't know how much you know about programming, but properly multithreading an app is alot of hard work, balancing the threads and making sure the data isn't ping-ponged back and forth between cores/cpu's, hammering the fsb, is an extremely difficult thing to implement and debug.

however, simply bisecting the video stream into 2 equal sections and having cpu 0 encode the first half while cpu 1 encodes the second half then splicing them together is a great way to realize a nearly 100 percent benefit from adding a second core/cpu, it doesn't require a ton of complex coding and the technique can be expanded to as many cpu's as you want.

most of the better applications, such as procoder that can use up to 16 cores, do it this way. the alternative is a nightmare.

[deadrats]

I was pointing out that Gigabit ethernet and USB2 seem to demand a lot of CPU capacity. With Gigabit ethernet, the CPU at the other end of the network seems to hold down transfer rate to far less than Gigabit rates. The Core2 CPU is loafing.

I'm also finding that internal PATA/SATA drives allow much higher utilization of the Core2Duo CPU compared to USB2 drives.

Is what I'm seeing normal? I was expecting the system to be CPU bound but instead I'm finding the CPU is system bound.

yeah, your experiences are not unusual. believe it or not, some I/O operations are extremely cpu bound and in some server enviroments it actually makes sense to upgrade to a better NIC (depending on features, some of them can cost close to $500 and it's possible to set up a psuedo-RAID array of 2 NICs) than it is to add a second cpu.

as for the cpu being system bound, i would say that's also about right under certain circumstances. i stopped using the onboard SATA/PATA controllers and instead went with a SATA PCI addin card and a PATA RAID PCI card (though i didn't set up a RAID array) and found that "smoothness" definately improved more than going from a Pentium D 820 to a Pentium D 945 to a E6400.

[jagabo]

avidemux has xvid and x264 built in and can launch up to 4 threads (if you had a quad core). what it does is it "slices" the video stream into 2 equal sections (or as many threads as you launch) and encodes each section on a seperate core.

That seems like a rather poor multithreading technique for any codec that uses motion vectors.

it's actually the easiest to implement

Obviously.

[celtic_druid]

Xvid may not have the most efficient multithreading, but it does give exactly the same quality as when encoding as a single thread. x264 on the other hand uses slices, which require some overhead, meaning lesser quality per the same bitrate when compared to single threaded encoding. Anyway as per the above screenshot, Xvid works fine. ~95% of both cores in use here.

Haven't really looked that closely at the code of avidemux, but surely it just uses the native multithreading of Xvid, x264, libavcodec, etc.?

[jagabo]

as per the above screenshot, Xvid works fine. ~95% of both cores in use here.

I find it varies from about 70 to 90 percent depending on the source (DV, MPG, etc), how it's opened (directly with VirtualDubMod, indirectly with AVISynth, etc), and the Xvid settings (motion precision, etc).