We Need An Encoders Benchmark!!!!

[freak_in_cage_10k]

Hi everyone

i am not sure if this has been mensioned before etc, but it has recent;y occured to me that us encoders need a benchmark! idealy there would be some freeware and we would have to encode approx 10 mins or film to SVCD. if everyone used the same standard settings then it would be interesting to compare results. Not to start an amd vs intel debate but it would be interesting to compare hardware etc, as well as windows tweaks, overclocking, upgrading components. this standard bechmark could be included along with the pcs details in your profile! it would also solve newbie questions regarding cpu speed, good encoding pc setups, and also whether their hardware is performing to what it should be or whether there is something wrong with their pc.

hope this hasn;t been mensioned before! i use 3d mark 2001 for graphics benching and i just thought that it would be good for us encoders etc too.

so, if any one knows of a suitable encoder (preferably free) and a sample source which we could encode then i would be most interested!!!

i reckon that if a set version of tmpeg was used, with a standard template (chosen by sum 1 in the know and avaliable from vcdhelp.com). this would not cost anyone anything and also many ppl have tmpeg already!

please feel free to make any comments etc everyone!!!

[Baldrick]

This no off topic discussion...it is very "on topic" so I have moved it.

(I deleted your old topic....remember that you can bump your topic if noone replies)

[bugster]

This is an interesting idea. However, if you know anything about benchmarking you will also know that they cannot be relied on to give truly meaningful or worthwhile results. They give a useful indication in some cases buat as so many things affect PC performance and no benchmark program could possibly cater fairly for all different configurations accross a wide range of PC's.

Useful for checking for performance changes after changing settings or modifying a PC, yes, reliable for comparing performance of a wide range of systems, personally I don't think so, but thats just my opinion.

[acid512]

In theory, i think this is a good idea. As it woulnt cost anything to set up, why not give it a go (set up a free web site, upload your video, and outline the rules, then post a link to the page)

However, here are some probs that may occur:

*you might need original video (got a handycam or simular?)as you are freely distributing it.

*unless you restrict it to people who you know (which would cut out me ), you will get results from people using the wrong versions of, say, TMPGenc, cause they dont understand

*the AMD/INTEL debate has seemed to produce some extremely opinionated people, who would quite happily lie about the results to promote there choosen system

All in all, a good idea. PM me if you want help making a web site to launch it all (please remember as i am in the uk, i will be in bed/ at work/pissed at seemingly unusual times)

good luck

[acid512]

just noticed your in the uk as well.

I am still pissed at unusuall times though!

[freak_in_cage_10k]

bugster- if we used a popular encoder as a benchmark then it would give a very accurate result to CPU & Computer performance as it is using the very software that would be used in real life!!! benchmarks such as 3d mark 2003 etc are much less accurate as they tend to stress the components in different ways which stray a little from a usual game! What could be more accurate than using software directly designed to encode!?

acid512- cheers for the offer of help man, but to tell you the truth i know NOTHING of we site design etc and so you would effectivly be trying to help a complete newbie!! i have a friend who can write in all the html lark and so i will try to ask himn to help me, although it may not be for 2 or 3 weeks.

with regards to ppl cheating to up their score (eg using different parameters) or a different version of tmpeg, that would be difficult to ovoid, however, if someone who is so obsessed with their processor (because it is so good) that they feel the need to uncrease their score by cheating, surley it means that they are happy with their score in the first place eg THEIR PROCESSOR AINT THAT GOOD!! if we made it so that the correct parameters could only be downloaded once the correct version of tmpeg was downloaded (from our site) then at least the would cut out confusion for newbies??

the original video source may be a problem, perhaps we could use a movie trailer? if this is not possible (for legal reasons) then possibly a memeber of vcd help could give some of their own video??? What length of video would be suitable to give an accurate result? i think the encoder should be working for at least 20 mins, even on the fastest machine.

so, if anyone has any ideas as to how we could make this thing happen then plase just comment, i do think it would be a good idea as i see many posts from newbies asking questions like "whats the best PC setup for encoding" or "is there something wrong with my pc or is it too slow?" all they would need to do would be to find someone with a pc of similar spec to them and compare encoding speeds. if they wanted to find a good setup they could look at ppls pcs and find out just how much they want to spend!

So, do ppl think this is a good idea or is it too hard to put into practise???

[Gazorgan]

Actually it could be fairly straightforward if it's done 'Relatively'

Full system specs and encoder versions, precision settings, and clip source. Then you time the encode for say, CCE/TMPGEnc/Others, then give the subjective results. You may get results where CCE 2-pass took 2 hours, but the TMPG 2-pass 4 hours but looked better on high precision, looked the same at 3 hours on normal precision. Subjectively you evaluate facial color, background crawling, motion lines in action scenes.

Statiscally you need to track PIII's, PIV's, and Athlons, and OS.

I've done this myself, and CCE is faster than TMPG. Quality is another issue, they do things differently in side by side clips. Realistically I encode overnight, then re-queue and encode all day. As long as the total project is under 10 hours, I don't care if it takes 2 hours or 9 hours. time isn't always the factor.

[freak_in_cage_10k]

Gazorgan-i agree with also recording OS, and also your type of processor, but the main point is that as long as the same software and settings were used it would be a fair test!

Any more ideas as to how we could make this happen ppl??

[hustbaer]

hmmm....
i think the main-problem is the video-source...
downloading a "realistic" (means: high-res, low-compression) 10 or 20min video would take long and stress the server a little too much.

bittorrent could help with this.

or use a DVD that many people own - like "the matrix" or "lord of the rings" part 1 or something like that...

then give detailed instructions how to rip & decode, like:
* DVD-Decryptor x.y.z - select chapters 3 to 7, ...
* use dvd2avi with settings (...), then VFAPI-Converter to frameserve...
etc.
and forbid any further processing like avisynth/vdub etc. - even forbid the use of the dvd2avi-vfapi plugin for tmpgenc...

next somebody would have to setup means to publish the results...
maybe via email to somebody, a sticky thread that is heavily moderated - meaning anything off-topic will be deleted/moved or something similar to the dvd-player review page...

personally i'd suggest a ~5 minute huffyuv compressed avi with VCD/SVCD resolution and IMA-ADPCM compressed 44.1khz/2ch stereo audio.
download via bittorrent.
huffyuv codec to use should be v2.1.1 - it would have to be the exact same version to make "fair" comparsions...

bye,
--hustbaer

[baker]

right time to butt in on your wee conversation.

Heres how I would go about settings it up.

First of all a difffernt clip would produce different results so we would need to make sure everybodys using the same source and same version of your chosen encoder.For the encoder choose tmpegenc, I dont like it but it will do our situation lovely. Make a templet with all options locked out.

For the video source....

Well I have a few ideas on this but there all problematic. Right first off let me tell you how I would go about this if we were all pal users.
The film to use would of course be the matrix as everybody has a copy of it or knows somebody who has.
INstructions for people who want to participate in the test would be to rip files off the dvd using dvd decrypter to the folder "c:\MDT\" Then they would have to open an avs file made here by you in tmpegenc version whatever.open your templet and click encode. When encodings finnished

note the time it took etc...
Just a bit more on the avs file I mentioned above, it would automatically select the vob files from "c:\MDT" and select the folling parts for enocoding:

The scene where morpheus is talking to agent smith, from when he takes off his earpiece to the scene cuts to neo and trinity comiong into the lobby.

The scene where morpheus gets shot to just after the helicopter smashes all the windows.

These scenes are bright and colurfull and will abuse any CPU, this avs file can be easily created but if you have problems go to the doom9 forum and ask for help.

Now you read the above and say great method, should always produce the same results etc....

Well not everybody has the same copy of the matrix! Nstc people are going to hit a proble here. I aint sure how ntsc works but perhaps they can undergo some form of ntsc to pal to get similar results... actually seeing how PAL is superior it might be better if the pal people try to imitate the nstc people and not the other way around...

Note that this is as accurate as you can get but different people will have different versions of avisytnh etc... perhaps a quick setup file which installs all the needed codecs for an accurate testing is needed???

Hope you took something from this,
Baker

[SaSi]

I am afraid it's not totaly clear to me what exactly would such a benchmark measure. Compare two different encoders (e.g. CCE vs Tmpgenc)? Or measure the performance of a particular, say "Reference" encoder with a specifc source video and precice settings/template against a system configuration?

If the target is the second one, then a benchmark should be a carefully selected load for the PC to simulate an encoder. For example:

MPEG-2 encoding is CPU intensive, that's well known, but what % would one give to the CPU load in such a benchmark, that's a question.

It's also about moving data around. The encoder reads a file and writes to a file. Approximatelly 3-4 Gb of data will be written (for a DVD simulation) while the amount of data read is usually unpredictable. It can happen that I re-encode MPEG-2 streams with MainConcept (say 5-6 Gb) or I encode a huffyuv capture (40~60Gb), etc. In the first case, reading 5Gb over 40~60minutes (minimum) could get 0.01% of the total load. Reading 50Gb and decompressing huffyuv would certainly get a few orders of magnitude more. So then we would be measuring Disk I/O and the huffyuv decoder.

What I'm saying is that when you want to measure anything, you must be able to isolate all other elements that don't belong to the measured object. To give a simple example, we take off our clothes before stepping on the weighscale, don't we?

So, before running the benchmark, should one defragment the Hard Drive? Or should one select a target disk different to the source one? Or should one carefully select the target disk to be empty and the fastest one on their system? (Write performance in the start of ANY HD is about double the write performance at the end of the drive. And divide any speed by 2 if the disk is fragmented).

Based on observations over the past 3 months I've been involved in this "sport", I can suggest that the simplest way to benchmark the "encoding process" is to create a program that reads a source file (just like an encoder would do), do some heavy CPU processing (like divide each 4-byte word with the previous one and discard the result) and write dummy data to a selected output file of a user-selected size.

This program would run as fast as the CPU allows and be (perhaps) blocked by slow disk I/O. Would never have a direct relation with the actual time CCE would need to encode a file, but repeated runs of this test program and CCE in the same environment would allow someone to make useful correlations.

By the way, this is more or less the same principle on which the database TPC benchmarks are based. Perform dummy transactions (not real life ones) and use the Trans/sec index as a speed measurement of the overall system's performance.

[Thorn]

I think this is a really cool idea. It could be made more accurate than 3Dmark, but it might be hard to complete. The best way would be specific scenarios: DV to DVD using TMPGenc; Huffyuv to DVD using TMPGenc; the same using CCE; the same using Mainconcept.

The reason I say separate tests to simulate different encoders is this: They encode differently... so different machines will have different *relative* performances (i.e. hyperthreading helps TMPGenc a lot, but I don't know if it does CCE basic any good). I think synthetic benchmarks won't be very useful, because you *want* to know if defragging helped or not, or if your new memory sucks (etc). You almost just want an interface to automatically control TMPGenc and time the output automaticaly (you can't include the full versions of all 3 programs but you could just interface with what's already installed...),

Now for the delivery: Yes, the files would be really large (at the very least a sample DV file and a Huffyuv file). They could go on the VCDHelp sampler CD that was put out... or they could be shared by file sharing or other means. Certainly not very difficult.

All in all it could be a very fun project, but I don't know if anyone would pay for it... and that would mean not as many people would want to work on it.

[acid512]

Right, its great the amount of input that everyone has put into this, but it seems to be running along different lines.

What i gather, is that the results of this test would be used to show neebies what kinda of machines get what results. It would not be a test of encoders, or the particular type of video being encoded.

So heres some outlines i think are worth starting from:

Same encoder (would suggest the latest version of tmpg availabe when we descide to go ahead, as most neewbies will start with tmpg, plus its free.)

Same encoder template (bloody good idea baker)

Same sourse video (this is still gona be a pain in the arse unless everyone has broadband, and we get some generous guy to lend us web space with a fair amount of bandwidth - intersted Baldric? - if we do choose a popular DVD, we will probably have to seperate NTSC and PAL results)

Providing we get enough testers willing to partisipate, then variables such as defragged HD's and such can simply be part of your 'system spec', giving people an idea of the advantage of defragging your HD.

REMEMBER this is just an experiment, and the outcome will not destroy a CPU manufacturer, or disullution the population, it will just make intersting reading, and help a few neebies that wonder whether there 286 will rip a dvd in 5minutes. It wont be 100% accurate, but will be useful to this community.

Keep the feedback coming.

[monoxide77]

this is a great idea. i just don't see the use of the benchmark for SVCD. Why not 2-pass VBR for DVD? this process seams painfully slow for me on my Celeron 1.1, and it'd be interesting to see how long it takes people with different systems. (don't hate me for this but...) i'm not a fan of SVCD.

[freak_in_cage_10k]

Thanks a lot for the responses everyone, all good ideas/comments!

hustbaer- acquiring the video source would be a problem, putting stress on the servers + large download times. All i can think is that the main members of vcd help (i would be happy to do it also, although with 100 posts i aint a main member!) could offer a postal service of the (free) software + video source on a cheap CD-R at cost price? Say £1 for the CD, 30p for a CDR, anoither 30p for the stamp, and the profit going to dvdrhelp.com!! this may put put ppl off due to the need to give out their address so perhaps it would be better for administrators only to do it, as they would be trusted by all (eg sum 1 like me with only 100 posts may not be trusted, which is no probs) or perhaps there would be objections for legal/tax reasons??

baker- good ideas man, a littlle too complex for me to understand sum of it thou but it sounded like a practical solution!

SaSi- the idea of the benchmark would be to copare hardware setups and possibly operating systems (although i doubt it makes much of a difference). All other elements would be isolated, with the variable being the hardware setup. hopefully if enough ppl submitted their results then there would be almost all of the major combinations avaliable!

i think your comments about hard drive setups (eg raid) + defrag are very important, perhaps a small comments section next to the benchmark result commenting on when the last defrag was performed etc?? i think it should be required that only 1 hard drive could be used in the benchmark (for os,source& output) because the majority of ppl dont have 2 HDs. however, is ppl could have several results then multiple HDs eg could be used.

i think you idea about designing a "special" CPU benchmark is a good one, but wouldn't this mask any benefits from multiple HDs etc? I alkso think it would (probably) mean more work that just using tmpeg??

acid512- well done m8, summed it up perfectly there!

in short, you keep the encoder the same, vary PC hardware spec, include detailed info on PC spec & allow for comparason.

thanks for everyones input, but do ppl REALLY think the idea will take off? if so then i will be happy to help work the project, but i aint too technical so i doubt i will be much use:S:S

Any more thoughts n e 1????

[freak_in_cage_10k]

Could sum 1 please post a Poll asking ppl if they thing an encoders benchmark is a good idea? if we get a good response then perhaps we could start working on it?