Giving raw Fraps footage some extra compression

[Virnec]

EDIT: Just scroll down to the last post here, only gt a few audio questions left. THanks

I've been doing alot of frapsing lately, and the files are just flooding my harddrive.
Since I'm clueless on when I'll be making a movie out out these clips, I'm thinking of archiving them for later.

Basically, I want to compress the clips as much as possible without losing any quality, so I can use these later for making the movie. I'm thinking of using AVC/H-264 - since this is new and fresh, and looks effective.
Currently, I'm capturing @ 1280x1024 40fps. The average bitrate is around ~30mbps. The plan is to dump the raw fraps files one my server, and use batch files to decrease the size of the clips, and then archiving them on external harddrives and DVDs.
I've been testing the command-line based x264, but still got a way to go before mastering it. Before spending more time on it, I've got a few questions.

1. How much can one expect to compress the files without losing quality? (percentage)
2. How many percent can be expected from lossless compressing? (lossless music has has a ratio of ~ 70%)
3. Which codec is recommended for this kind of compression?
4. Does x264 handle the fraps files properly?

Other guidelines or redirections are also much appriciated!
Thanks.[url=https://forum.videohelp.com/images/guides/p1896029/[/url]]

[Baldrick]

30mpbs 1280x1024 40fps is already heavily compressed and you will lose quality by compressing it even more. But you can always try and see if it looks okey for you. I would try use constant quality/constant quantizer H264 using megui and also try different h264 profiles(megui uses x264).

[Virnec]

Thanks for the reply, Baldrick.
When I wrote 30mbps, I meant MB/s - ergo 240mbit/s. Fraps does some lossless compression on-the-fly, but I can't imagine more than 5% decrease in filesize.

The idea is to dump the raw files on my server and run them through x264-batches. I get errors when running simple commands involving fraps-files - but I guess that this is the wrong forum for that.

As I understand, all lossy compression degrades the video - so compressing the clips over and over isn't such a great idea. Most people say that one should work with raws all the way, which makes sense. But the files are so huge that this is practically impossible if one is to build up some material over time.

I'll note constant quantizer on my list.
Thanks!

[Baldrick]

Yep, 240mbit/s is a bit more...

x264 has a lossless option you can try also, it is "-qp 0". I don't know how much it will reduce the file size though. See https://forum.videohelp.com/topic356590.html

I haven't tried just the command line x264 so I'm not sure if it will support fraps files directly. Maybe you have to use it together with avisynth. Check also doom9 forums, http://forum.doom9.com

[Virnec]

I've been testing different compression settings, and things are looking good!
I still got a great deal of testing and comparing to do before I'm ready, though.

So far, my profile is based on one of the default profiles that comes with meGUI: Unrestricted 1pass Const. Quality HQ.
I've only changed b-frames to 0, tested out different quality-levels.

My 1,679 MB clip got compressed down to 93,4 MB - with almost no difference in quality! That is really outstanding.
Going from about 38MB/s (without audio) to 2,33 - with no real visible difference - is just above all my expectations. t It took 16 minutes to encode @ Intel dualcore 2GHz, about 2fps - but once I get my quad 3GHz up and running, this will decrease drastically.

I've been watching through several clips the last two hours, trying to compare raw fraps vs compressed. The only thing I can say, is that the colors seems abit pale, or washed out. I've tried to capture frames of the two files, for closer inspection, but I just end up with two identical images - with almost no difference at all. I figure this can't be right. Is there a program I can use for better comparisment?
Also, please do recommend avs filters/settings.


I simply used the inbuilt video-preview in meGUI and used print-screen to capture the images.
Take a look at the two images in the zip. It has been scanned for viruses.
images.zip

edit: The colors really are washed out, they just a lot less 'fresh' and solid.
Which avs filter could improve this?
Which x264 setting, in particular, may improve this?

[Virnec]

Alright, got most of the video-related sorted out now. Turns out, the differences in color I mentioned, had something to do with the player used. I guess it treats different colorsets differently, since the color seemed much paler when playing compressed footage compared to raw (RGB32).

This now seems to be a three-step operation, compress video and audio separately, and then mux it together.
I've got a commandline-routine on video compression, now I need a routine for audio-compression and the muxing.

New questions:

1. Does it really matter what container I use? MKV seems like a nice container - how would it work as a source later? Any complications?
2. Which lossless codec to use, compitable with MKV with cmdline support?
3. What program can I use for muxing it togehter, cmdline?

Any help is highly appreciated!

[45tripp]

Basically, I want to compress the clips as much as possible without losing any quality,

Also, please do recommend avs filters/settings.

As I understand, all lossy compression degrades the video - so compressing the clips over and over isn't such a great idea. Most people say that one should work with raws all the way, which makes sense. But the files are so huge that this is practically impossible if one is to build up some material over time.

uhmmm...
what exactly are you trying to do...

first you mention lossless,
then you go into highly lossy.
you say you'll back up the originals,
but actually work with the compressed...



there's good reason for the recommendation to work with raws.

Fraps does some lossless compression on-the-fly, but I can't imagine more than 5% decrease in filesize.

uncompress and see.

i'm interested in a sample btw.

I haven't tried just the command line x264 so I'm not sure if it will support fraps files directly.

it won't

x264 has a lossless option you can try also, it is "-qp 0".

which only coreavc can decode.
just so you know.

Is there a program I can use for better comparisment?

avisynth.

[*]Does it really matter what container I use? MKV seems like a nice container - how would it work as a source later? Any complications?

depends on usage.
it is a nice cntainer. will wrap around anything.

if you mean to actually edit it,
good luck finding an editor with mkv import support.
mp4 would be a slightly more sane choice.

[*]Which lossless codec to use, compitable with MKV with cmdline support?

don't see how there can be any other answer to that than flac.
which will mux into mkv.

if you're to use an mp4 container,
there's but one lossless choice.
alac.

[*]What program can I use for muxing it togehter, cmdline?

mkvmerge
mp4box



tripp

[poisondeathray]

if you mean to actually edit it,
good luck finding an editor with mkv import support.
mp4 would be a slightly more sane choice.

avidemux supports visual editing of both h.264 in .mkv and .mp4, however the limitation is editing on keyframes and there is no "smart rendering" of h.264 (yet), and mbaff/paff content is still handled poorly.

The limitations on editing with avidemux are because of h.264, not because of the container choice

[Virnec]

Thanks alot for replying.

uhmmm...
what exactly are you trying to do...

first you mention lossless,
then you go into highly lossy.
you say you'll back up the originals,
but actually work with the compressed...

The raw footage has an average bitrate of 41MBps. One hour of footage would take up 148GB of storage, which is insane. I'm trying to decrease the bitrate as much as possible, with as little quality loss possible. The footage will be archived for later use, sometime in the future.
Using lossless would only cut the filesize in half, at best, which is still too much.

Fraps does some lossless compression on-the-fly, but I can't imagine more than 5% decrease in filesize.

uncompress and see.

i'm interested in a sample btw.

I'm not quite sure how to decompress the files.
What I did, was opening the files in VirtualDub, chose the (Uncompressed RBG/YCbCr) and save as avi But the filesize increased Enormously! From 41MBps to staggering 160MBps! Is this right?
Btw, what kind of sample are you interested in? I posten a rar file a couple posts up, comparing a frame of raw versus compressed. I could make a small 10sec clip or something, just to show the quality. I'd be happy to post some raw stuff too, but I need somewhere to upload it to. Are you on UseNet?

I'm evaluating different compression methods and checking the files for quality, and it looks pretty good. I've posted my profile for download at the bottom.
Basically, it's a unrestricted 1pass with no b frames, and const. quality/quantizer of 18.
Constant quality gives a bitrate of 3,35MBps.
Constant quantizer gives 6MBps - almost the double.
Per hour, 12 GB versus 21 GB. I'm aiming for something between 10 and 20, so this looks good.
Audio is not included, but won't account for much.

if you're to use an mp4 container,
there's but one lossless choice.
alac.

mkvmerge
mp4box

Alright, great - checkin' it out!
I do like mkv. At worst, I could just demux the stuff before editing. Maybe avs has a solution.
Flac is also nice, since I use it for all my lossless music.

Great, thanks alot, tripp! I'll be reading some tutorials for muxing mkv and batch-compressing flac.

if you mean to actually edit it,
good luck finding an editor with mkv import support.
mp4 would be a slightly more sane choice.

avidemux supports visual editing of both h.264 in .mkv and .mp4, however the limitation is editing on keyframes and there is no "smart rendering" of h.264 (yet), and mbaff/paff content is still handled poorly.

The limitations on editing with avidemux are because of h.264, not because of the container choice

Great, thanks for the input
I won't be making any movies till spring, or somewhat later. But that depends on the footage and the ideas I come up with.

x264%20profile.rar

[45tripp]

avidemux supports visual editing of both h.264 in .mkv and .mp4,
The limitations on editing with avidemux are because of h.264, not because of the container choice

obviously the first limiter is avc.
but mkv furthers that.

in this context i'd not call avidemux, vdub, or any other simple stitcher an editor.
we're talking nles

I'm not quite sure how to decompress the files.
What I did, was opening the files in VirtualDub, chose the (Uncompressed RBG/YCbCr) and save as avi But the filesize increased Enormously! From 41MBps to staggering 160MBps! Is this right?

yes

Btw, what kind of sample are you interested in?
but I need somewhere to upload it to. Are you on UseNet?

just a few seconds to look at.
no
mediafire.com

At worst, I could just demux the stuff before editing.
Maybe avs has a solution.

and decompress.
there are solutions,
they just hog and cripple your machine.

Flac is also nice, since I use it for all my lossless music.
I'll be reading some tutorials for muxing mkv and batch-compressing flac.

i doubt you'd find any,
but all you need is a manual to construct a single commandline for each.


tripp

[PuzZLeR]

Tried HuffYUV, Lagarith, SheerVideo, x264 lossless, MSU lossless, LJPG, FFV1 and maybe some others as well as DV (pseudo lossless). All had their advantages. All produced great quality. However, all were HUGE.

If only I could find a lossless codec that compresses comparably to H.264, DivX/Xvid, or MPEG-2 (or even with a file size only a bit bigger), life would be grand for me in this hobby... GRAND.

Unfortunately, due to the need for certain compression attributes (ex: B frames, motion estimation, etc), which by definition are lossy, this is (at least currently) not possible.

...sigh.

[Virnec]

I'm not quite sure how to decompress the files.
What I did, was opening the files in VirtualDub, chose the (Uncompressed RBG/YCbCr) and save as avi But the filesize increased Enormously! From 41MBps to staggering 160MBps! Is this right?

yes

Pretty insane, considering that fraps does this on-the-fly.

just a few seconds to look at.

I'll have a clip ready tonight, or early tomorrow.

Flac is also nice, since I use it for all my lossless music.
I'll be reading some tutorials for muxing mkv and batch-compressing flac.

i doubt you'd find any,
but all you need is a manual to construct a single commandline for each.

I've dropped the thought of muxing the clips, and now I instead link the audio and video with avs scripts.

By decreasing the captured FPS from 40 to 30 - not sure why I was capturing at 40, maybe because of the smoothness - the filesize also decreased by 20%. The encoding time varies from 0,7 to 3 FPS (one thread), and filesizes varies between 30 and 120MB per minute of footage - all depending, naturally, on the action and variation in the clips. I've estimated that 15-20GB per hour of footage is the limits of my spendings. I've therefore still got some more spending to do on the clips - and want to reduce the lossiness further, while keeping the encoding speed at a sane level.

Which basic settings should I focus on, to decrease the lossiness?
The only thing I've really done, is excluding b-frames and use a const.quantizer of 18 (as const.qual seems more lossy). What else should I focus on, except from decreasing the quantize level?

Tried HuffYUV, Lagarith, SheerVideo, x264 lossless, MSU lossless, LJPG, FFV1 and maybe some others as well as DV (pseudo lossless). All had their advantages. All produced great quality. However, all were HUGE.

If only I could find a lossless codec that compresses comparably to H.264, DivX/Xvid, or MPEG-2 (or even with a file size only a bit bigger), life would be grand for me in this hobby... GRAND.

Unfortunately, due to the need for certain compression attributes (ex: B frames, motion estimation, etc), which by definition are lossy, this is (at least currently) not possible.

...sigh.

I'm in pretty much the same situation, except I went straight for the AVC codec.
Recording fraps with current settings, it takes up ~120GB per hour of footage. As this is a long-term project - with no current plans/ideas for a movie - it would be extremely costly to keep all that footage in raw state. The fish would go bad, so to speak. And while you can put the fish in the fridge for a few months, the fridge would eventually fill up, forcing you to start eating of it to free up space. Now you have two options, since you only got room for one fridge: either slice up the best parts to fillets and throw the bones, skin and stuff you don't eat - pr simply stop fishing. This is exaclty what I'm trying to do with all the footage that piles up.
I've gone through lots of different clips, comparing compressed against uncompressed. I've looked for errors, artifacts and visible differences. And they are barely noticeable. There are differences, sure, but the codecs are designed to remove the stuff you in most cases wouldn't notice.
But it all depends on what you are filming/capturing. Real-life footage is more sensitive to compression that anything else. If you are frapsing, the overall quality on textures etc. plays a major role. As I'm playing WoW(warcraft), which is very cartoon-like and not very detailed, it allows me to compress it alot, without losing too much quality. The stuff I notice the most, is the softening of sharp contours, like bricks on houses, details of things in the distance and distant landscape. Atleast that's how I see it now - have I missed anything?

Which purpose do you have, Puzzler. And what kind of material are you working on?

[PuzZLeR]

Hi Virnec, I agree with you totally.

I have tons of video that needs editing/processing. Cuts and joins are fine, but the crops, resizing, overlaying, IVTC/deinterlacing, deblocking, adjusting A/R, etc. needs a re-encode. But for the first time in my "encoding life" I have totally stopped encoding anything during these times of transition.

DvD is on its way out. Or is it? Shall I encode to MPEG-2/DvD today?

Blu-ray, which is supposed to replace DvD, has yet to hit prime-time if at all. Nevertheless, encoding to BD compliant content is currently premature - troublesome and/or expensive and suffers from a dilution of equality since it may not play on all units...

Encoding to DivX (or even Xvid) would have been nice if not for the fact that DivX 7 is on its way. What will that, or its adoption, be like?

AVC, the "next MPEG-2", is great, however it has yet to establish itself. MeGUI currently has over 50 AVC (x264) profiles. Which one will be the commonplace "standard" after the chaos is sorted out? Encoding to AVC today is like encoding to MPEG-2 with no knowledge of the DvD spec - would you encode to MPEG-2 without DvD specs?

So I am encoding nothing. I'm in the "Void Which Binds" much like you.

Ideally it would be nice to archive to lossless "Masters", instead of taking a chance on an encode format now that may need a re-encoding again later (with loss of quality). But lossless formats are HUGE. It doesn't seem practical to buy 10 1TB hard drives just to store just a couple of hundred hours of video (camera stuff, personal/family stuff, TV stuff, music videos, sporting events, etc.).

It's not the one encode that bothers me, since, as you say, if done right it would be indistinguishable to the naked eye. But it's the possibility of error, format changes and multiple re-encodes generation-to-generation, a series of degradation of video (since I know I wouldn't keep the source after the encode).

I don't have a suitable recipe to cook my fine meal. A wrong recipe would ruin it if I have to redo it. So my fridges keep getting full of unprocessed raw food. I'm adding fridges unnecessarily and my fish are starting to stink.

All this fine food, yet I'm starving...