Re-Encoding for Archival

[lexluthor5]

I take some family videos using my Canon SD-780. It takes 720p video.

The video gets saved in a .mov file (looks like it's H.264 protocol, 22.4Mbps, 30 Frame rate) and compression is awful.

The 1 min, 39 second video I took today came out at 265MB.

Using Handbrake (or equivalent), what should I re-encode this to in order to get the filesize down without reducing quality too much.

Let me know your suggested handbrake settings.

I'll probably be deleting the original .mov file, so I do want to keep a fairly "high quality" copy, but I know I can do a lot better than 165MB/minute).

Thanks!

[hech54]

The words Re-Encoding and Archival don't go together in the same sentence.

[lexluthor5]

Well, I'm not looking to make commercial movies out of them someday.

I just want to get the files sizes to be something more reasonable without killing quality too much and have them in a "standard" format that will still be around years from now.

Can anyone suggest some settings that would do that for me?

[Vidd]

You could use x264 with a lower bitrate, but I wouldn't recommend it.

The cost of your time spent encoding would pay for a 1TB hard drive in no time.
You could pick up an internal 1TB drive for $50 or an external 1TB drive for $75.

After formatting the 1TB drive will be 931GB. At 165MB a minute, thats over 90 hours of raw video that could fit on the drive. Buy two drives and have a backup!

[lexluthor5]

You could use x264 with a lower bitrate, but I wouldn't recommend it.

The cost of your time spent encoding would pay for a 1TB hard drive in no time.
You could pick up an internal 1TB drive for $50 or an external 1TB drive for $75.

After formatting the 1TB drive will be 931GB. At 165MB a minute, thats over 90 hours of raw video that could fit on the drive. Buy two drives and have a backup!

I guess that all makes sense.

Still, if I encode at H.264, what's a suggested bitrate to use for 720p video?

[ron spencer]

I do not get this...hard drives are soooo cheap now...don't do anything...easier, faster. Just keep what you have.

[davexnet]

Is that the default bitrate of the camera? Is there anything in the settings that let you adjust it ?

If you really want to try something, give StaxRip a go at the default settings.
(I've been using the 1153 Beta, may be a bit further ahead by now). Default settings are device "divx plus",
quality 23.

[lexluthor5]

No, can't change anything on the camera.

All I really want to know is what's a good bitrate to encode 720p video to?

Maybe I'll get another hard drive, but hese aren't commercial videos, it's not that big of a deal if they look perfect. I just want to cut down the file size some.

Plus, these are videos from a digital camera. It's not professional recording equipment, not even a dedicated video camera.

[davexnet]

Bitrate is not usually advised or used directly, rather a version of constant quality is usually used.
Very difficult to advise on a bitrate because it would depend on the complexity of the material.
Try CQ of 23 as I mentioned, its a good starting point.

[lexluthor5]

Try CQ of 23 as I mentioned, its a good starting point.

Something doesn't seem right. I took my 265MB file and used CQ 23 and got a 9MB file that looks like it's at about a 700k bitrate.

That just seems to be too low, no?

When I do an 4000K average bitrate conversion I get a 50MB file.

Won't the quality be significantly worse with the CQ 23 file?

[davexnet]

I'm not an expert on the X264 settings. However, I think that setting of 23 has been carefully considered as a good starting place.
Perhaps somebody with more x264 experience can comment on it.
Is this the 1 minute 39 second file you're referring to? Did you change the resolution at all ?

What did the file look like when you perused it? Did it look as good as the 4000k version?
Does mediainfo provide any useful info ?
Does your original file contain audio? ? What format ?

Can you post 5 or 6 seconds of the original video ? I'd be interested in seeing it.

I encoded a test file, a 720p movie trailer, 5000kbps, processed it without changing anything
@ quality 23 and received a file approx half the size - not the drastic change you got,but more
in line with what I'd have expected.

[larryc]

@lexluthor5, you are exactly in the same boat as me. I have a Panasonic camera that takes 720p videos that look great. However, the camera saves them using a MJPG codec to a .MOV container, and the files are huge, even by today's standards. Just like you, I want to compress these so they take less space, but I don't want to kill the quality of the HD video.

After playing around with lots of tools and lots of settings, I've settled on using Handbrake using a CQ of 50. Compressing using this setting shrinks the files to about 15% of their original size, but I've found the quality is still good. If you want higher quality, try a CQ of 55 or 60.

If you can post a small sample somewhere, I will try running it through the tools I use.

[lexluthor5]

Larry,

Thanks for posting.

Just to clear something up first, the poster above referenced CQ 23 and you reference CQ 50. I'm going to assume, the first poster is talking about the number after RF: in Handbrake and you are talking about the percentage number that comes first.

I need to do a little more testing with this first. I have one video where the PQ is very bad in low light and using RF: 20, it dropped the file from 265MB to 14MB. But, I have another video from the same camera where, using the same RF: 20 setting, it dropped the file from 890MB to 394MB.

Not nearly as much compression on the 2nd video.

I'll to do some more tests with this to see if it really varies that much depending on the original video.

One other minor issue is that my HTC Incredible won't play a video if I encode it to 800x480 with the constant quality option. If I encode it to 800x480 using average bitrate, it plays.

It doesn't really matter since I can always re-encode it if I really want to watch it on my phone, but I'm more concerned that if I use CQ and try to play the file 10 years from now, I might have an issue.

But, it does play fine in multiple programs on the computer, so I probably shouldn't worry about it too much.

I'm starting to think the best thing is either to use an RF setting around 23 (55%) or just to do it at around 4000K variable.

Not sure which one of those is best (mix of compression and quality) yet though.

[larryc]

Lex, I'm totally with you on the '10 years from now...' issue. That is exactly my concern as well. I'm less concerned with the specifics of getting the videos to work on specific devices (ipods, cellphones, etc) and more concerned with using a general enough format that I can be hopeful I'll be able to play it 10 or 20 years down the road.

To clarify, the setting I use in Handbrake is CQ 50/RF 25.5. I usually try this first and it generally works well. I'll spot check the video; sometimes I'll compress it using different parameters if I want better quality or more compression.

[edDV]

The file that comes out of the camcorder is your "archive". The file is already highly compressed h.264. Anything you do to recode will lower quality.

[larryc]

If lexluthor's Canon camera is spitting out a 1:30 file that consumes 265Mb, then I would not call that highly compressed h264. My Panasonic generates uncompressed (well, almost uncompressed MJPG) videos at pretty much the same bitrate. Just because the Canon uses H264 doesn't mean it's highly compressed. I completely understand the desire to want to compress these for archive purposes

[edDV]

If lexluthor's Canon camera is spitting out a 1:30 file that consumes 265Mb, then I would not call that highly compressed h264. My Panasonic generates uncompressed (well, almost uncompressed MJPG) videos at pretty much the same bitrate. Just because the Canon uses H264 doesn't mean it's highly compressed. I completely understand the desire to want to compress these for archive purposes

Depends if you define "archive" as preserving quality or as shrinking for minimum disk space.

Consumer camera/camcorder presets usually assume the user isn't going to be using a tripod, will shoot in arbitrary light and doesn't know what f-stop or ASA mean.

In other words they defer to beta user studies to do minimal harm. At the same time they know these idiots will only buy the cheapest flash RAM so that limits the upper limit on bit rate. They want to minimize store returns and service calls.

265MByte = 2120 Mbit ... 8 bits to the to the byte.

Divide by 90 seconds and you get 23.5 Mb/s. That is low for hand shot video in arbitrary light.

If you are comparing to pro shot, highly tuned DVD rips, you just don't understand the fundamentals.

[lexluthor5]

I probably misspoke in my first post.I'm pretty sure now it's the same format as larryc's camera.

[larryc]

@edDV, are you saying Lex & I are idiots (your word, not mine)? I don't understand your point. Both our cameras (not camcorders) can record HD video in addition to photos. The videos created by the cameras have little to no compression. This can be an issue for long term storage & backup since the files are very large (over 150Mb per minute). It is reasonable to consider compressing these files using H264; at the right CQ level there should be little degradation in video quality. Why do you have a problem with this?

@lexluthor5, you were correct right off the bat. Your Canon does indeed encode using H264, but as we've seen it doesn't compress it very much. Quote from http://www.cameralabs.com/reviews/Canon_PowerShot_SD780_IS_IXUS_100_IS/ : "the IXUS 100 IS / SD780 IS compresses the video using the modern H.264 format and encodes mono audio as Linear PCM at 16 bit / 44.1KHz, then stores the result in a QuickTime MOV wrapper."

[edDV]

I probably misspoke in my first post.I'm pretty sure now it's the same format as larryc's camera.

No. Your camera encodes with hardware to 15 frame GOP based h.264. Larryc's camera encodes to frame based MJPEG which is a good intermediate codec but may offer some temporal compression depending on how it was shot.

Still for "archive" Larryc should save the original MJPEG file if picture quality is a concern.

[edDV]

@edDV, are you saying Lex & I are idiots (your word, not mine)? I don't understand your point. Both our cameras (not camcorders) can record HD video in addition to photos. The videos created by the cameras have little to no compression. This can be an issue for long term storage & backup since the files are very large (over 150Mb per minute). It is reasonable to consider compressing these files using H264; at the right CQ level there should be little degradation in video quality. Why do you have a problem with this?

I was implying the target customer of these cams fall into the "idiot" range (knowledge or excessive frugality). I'd expect people on this website to be more serious users.

These cameras have enormous to excessive compression. MJPEG even under the best scenario has >35x compression using only intraframe DCT compression. H.264 has similar compression but uses intraframe and interframe (15 GOP NTSC, 12 GOP PAL) compression. I can do the math later if you want.

Mb means megabit

MB means megabyte

"Archive" generally means preserving as much quality as possible for future codecs.

Alternate is compressing more now to limits of current codecs. There is no way to reverse this decision*.

Disk storage is cheap. IMO you wouldn't compress these camera files further today if picture quality is a concern.


*My only dog in this fight is to persuade you to maintain quality to archive if you care about these videos. I'm assuming personal videos not media rips.

[ron spencer]

I agree with edDV..any change you make now is not as good as the original...I have seen 2TB hard drives for under $100...seen 1TB for $50...get some and copy *.*

[larryc]

Folks, I don't think we're arguing about the facts here, rather, we're debating the implications.
Clearly if your goal is to preserve the best quality then you don't want to compress any further. But as the OP (Lex) wrote:

All I really want to know is what's a good bitrate to encode 720p video to? ... These aren't commercial videos, it's not that big of a deal if they look perfect. I just want to cut down the file size some.

Balancing file size for ease of storage & backup is also a consideration.

I believe that with a reasonable selection of bitrate / CQ setting, you can compress these videos so they still look good while taking up far less storage space. The OP was looking for help on how to do this and I tried to help.

[lexluthor5]

I was implying the target customer of these cams fall into the "idiot" range (knowledge or excessive frugality). I'd expect people on this website to be more serious users.

Well, that's why I came here (serious users), but if this is the wrong forum for me, I guess I'm sorry.

I know my camera takes relatively "crappy" videos, but that's fine for me and it doesn't make me an "idiot". I have a tiny device that I can take anywhere with my and take ok pictures and ok video. I'd rather have an "ok" video from a camera that I carried with me than no video from a larger device that I didn't bring along at all.

Larry, thanks for finding the link on my camera specs, I should have looked it up first. Thanks also for jumping in and assisting.

I guess I kind of came to a place where everyone feels PQ is #1 and there are really very few other considerations. Personally, I'm not the type that can tell the difference (or really care, if I could) between audio at 320kbps and 128kbps and I'm sure the same goes for video. Shockingly, I'm not the only person who thinks like that. That's probably blasphemy around here though, but you guys should realize there are people who want that. Nothing wrong with educating them of the ramifications of not considering enough about PQ, but you could still help answering the question.

I have a 500GB drive for backup, I have a 1TB drive for my DVR and another 250GB drive for my other DVD. I really don't want to have external drives lying all around the house. Right now, I have these files on a laptop with a small HD and I really don't care if I take lose a small percentage of PQ if I can get a file 1/3 to 1/2 the original filesize as long as it's also in a format that should be around many years from now (which I'm thinking H.264 will be).

Sounds like encoding in Handbrake to H.264 with a CQ of 50% to 60% is probably the way to go here.

Maybe some of the "more important" videos I'll archive the originals and the "everyday" videos, I won't bother. That make you all happier?

Seriously, this was an interesting discussion either way.

[usually_quiet]

My brother-in-law has a ton of family videos on VHS tape that nobody will ever watch again. IMO if some of your family videos aren't good enough or meaningful enough to be worth archiving them in their original form to have the best possible material to watch again or edit in later years, then maybe those are not worth archiving at all. Save yourself the time and effort of re-encoding those and delete them now. You'll be doing that later anyway.

[lexluthor5]

Wow, I tried to be nice, but you are just an a**hole.

Nice way to have a friendly discussion. I doubt I'll be coming back here anytime soon. I'm glad you are a member of some of the forums I host.

[lexluthor5]

double post

[usually_quiet]

Wow, I tried to be nice, but you are just an a**hole.

Nice way to have a friendly discussion. I doubt I'll be coming back here anytime soon. I'm glad you are a member of some of the forums I host.

Sorry, I was not trying to be nasty, just truthful, after observing my brother-in-law accumulate a bunch of tapes over 20 years, some of which nobody cares about any more, and all of which now need to be reviewed and some thrown out. The difference is that he did not put any time or effort into converting the stuff that nobody cares about seeing again, but even afterwards would still have needed to be reviewed and gotten rid of at some point.

[larryc]

Wow, this is really incredible. The OP asked a pretty simple question in the first post: what should I re-encode this to in order to get the filesize down without reducing quality too much? That's a perfectly valid question to ask, and, unless you're a perfectionist, there is indeed an answer beyond 'don't do it'.

As for answers, three people replied 'don't do it' and one person posted 'don't even bother taking videos since you'll just delete them in 20 years'. I thought this website was VideoHelp? There's not much helping going on in this thread. Unfortunately, this is starting to remind me of some other sites where you ask about a windows problem and the linux fanatics jump in with 'drop windows and switch to linux'.

@Usually_quiet, your example actually validates the OP's question. Your brother-in-law presumably just threw out all those old videos because they 1) were in an obsolete format (VHS), and 2) took up too much space.

Again, the OP asked for help in 1) choosing a long-term archiving format, and 2) how to reduce the space taken up by his video files. If he chooses these correctly, he can enjoy these videos for many years to come. H264/MP4 is likely to be around for a while, and with optimal compression settings the videos will still look good without taking up too much space.

[usually_quiet]

As for answers, three people replied 'don't do it' and one person posted 'don't even bother taking videos since you'll just delete them in 20 years'. I thought this website was VideoHelp? There's not much helping going on in this thread. Unfortunately, this is starting to remind me of some other sites where you ask about a windows problem and the linux fanatics jump in with 'drop windows and switch to linux'.

@Usually_quiet, your example actually validates the OP's question. Your brother-in-law presumably just threw out all those old videos because they 1) were in an obsolete format (VHS), and 2) took up too much space.

Again, the OP asked for help in 1) choosing a long-term archiving format, and 2) how to reduce the space taken up by his video files. If he chooses these correctly, he can enjoy these videos for many years to come. H264/MP4 is likely to be around for a while, and with optimal compression settings the videos will still look good without taking up too much space.

Talk about exaggeration... I said my brother-in-law is throwing out some tapes that nobody cares about anymore, NOT all of them.He's still has the most important ones. I bet he knew shortly after watching them the first time which tapes should have been recycled for time-shifting TV, but kept them anyway. Hopefully, some day he will have the best ones converted into DV before the tapes degrade too much, but as long as the tapes are playable, those are his archive.

The OP gains absolutely nothing in terms of permanence of the format by converting from a better quality H.264 to worse, but he is loosing forever the opportunity to convert them more competently later on, when presumably he will know more on this subject. What looks good now will not look good later, especially when he has a better eye and even less forgiving video equipment to use for watching them.