Lagarith vs HuffYUV, archiving and file size

[HDClown]

Before I get too far along into my VHS -> computer work, I want to make sure that my current choice of using Lagarith isn't going to be a bad one. The main goal of this project is to capture my VHS tapes into a digital format and make that digital file my archive copy. I want this digital file to be the master, and done in a way that I would never have to go back to the VHS tape again. Being the master, I would use this as a source to make re-encodes from for other distribution needs. As a comparative example, I am ripping and encoding all my CD's to FLAC as a lossless digital archive format, so I don't need to mess with CD's again.

I read some things here and there that indicated Lagarith is more CPU intensive, but it makes smaller files. My 2.4ghz C2D has no issues playing backing the Lagarith AVI's in WMP11 on Windows 7, and the capturing only consumes 50% of each core, so the CPU issue isn't a concern to me.

With Lagaith, I'm seeing roughly 30 gigs per hour. How does HuffYUV compare when captured YUV2 ?

In terms of redistribution of the footage, I would re-encode to some other format to put online, give to others, etc.

In terms of longevity, no digital format could be considered permanent, but if we're looking at "what we know" about HuffYUV and Lagarith, and I better served with one or the other as my digital archive format?

Something I'm particularly interested in is web use. I want to put a good bi to fthis video on sites like YouTube, Vimeo, Facebook, and SmugMug. Does anyone know if these sites will accept an AVI encoded in Lagarith as valid, or will it say it's an "unknown format". What about HuffYUV?

[vhelp]

These are two of the best lossless (compression) codecs around. It is wise to incorporate either of these codecs in archival type projects. But even more important when the source medium is priceless or comes from an already poor medium, vhs. You want to retain as much of the original detail as practicle and possible. I say "practical" because for some people it is better to use another method, like hardware mpeg or dv, because their system is either not strong enough or their time is limited, and so on and so forth.

I am currently using huffy as my final archival master. When I saw the different relations of lagarith 47.0 Mbps vs huffy 52.2 Mbps, I decided it was best over all to just stay with huffy. Although not tested by me and from that number it seems to indicate that huffy retains more information than lagarith though that number could be based on other principles like space * compression or something like that, i'm not sure.

But whether true or false, i don't think it would be as easy to diferentiate between the two when generally looking at the videos, side-by-side. It would have to take zooming and actual measuring of pixel values to determine that, (though easy enough) and I just don't think it is necessary in my immediate work and time frame. Perhaps someone else would be willing to run that test.

I also chose huffy over DV because in my source mediums (satellite) this produced no additional artifacts. With satelite (directv) they heavily compress (mpeg4) which unfortunately is full of rainbow (or in other terms, posteriaztion) and if I used dv on this source, it would just increase these errors even further.

Still, I would rather use lagarith because it produces the smaller filesize over huffy, but it is slower during capturing and tends to eat up a lot of cpu. On my setup, it is quite high, 98%, 100%, even higher. And this could affect most peoples finished captures, like dropped frames. But in my system even these high numbers did not have any adverse effects to my captures. But still, to keep things safe and simple, I choose huffy in my system.

I also capture to external 1TB hdds over usb2, and with lossless codecs, the ride is smooth sailing.

Statistically speaking, 1 min capture avi resulted in:

capture 361.5MB, copy 00:48, 47.0 Mbps, lagarith->huffy -- YUY2
capture 400.0MB, copy 01:04, 52.2 Mbps, huffy->huffy -- YUY2 predict median best, predict gradient best

copy 361.5MB, 01:03, 47.0 Mbps, lagarith->lagarith
copy 400.0MB, 00:44, 52.2 Mbps, lagarith->huffy

Choose what is practicle for your case: if you have plenty of hdd(s) and space, either will do and there would be no wrong.. but if you are strapped limited hdd space then lagarith would be the wise move.

-vhelp 5365

[creamyhorror]

I don't know if Youtube accepts Lagarith, but I don't think so. It's going to be huge, and you'd probaby have to recompress to lossy anyway.

In terms of longevity, no digital format could be considered permanent

Digital formats endure as long as the hardware and software is around to support them, and given the need for lossless encoding, I'm quite sure there'll be decoders around for them in the foreseeabe future.

but if we're looking at "what we know" about HuffYUV and Lagarith, and I better served with one or the other as my digital archive format?

What are your requirements? Is your priority size on disk or decode speed? HuffYUV is geared towards decode speed, while Lagarith prioritizes smaller sizes.

If you're looking for higher decode speed, I should mention Ut codec. It's generally as fast as HuffYUV at encoding and decoding but is more size-efficient. Current version is 7.1.1, available at the end of the linked thread.

I am currently using huffy as my final archival master. When I saw the different relations of lagarith 47.0 Mbps vs huffy 52.2 Mbps, I decided it was best over all to just stay with huffy. Although not tested by me and from that number it seems to indicate that huffy retains more information than lagarith though that number could be based on other principles like space * compression or something like that, i'm not sure.

No, both codecs are lossless! They are bit-identical, as long as the input color formats and other variables are the same. If you want to do a comparison, encode the same clip in both codecs and then decode to a raw format and do a checksum or bitwise comparison. They'll be exactly the same format.

These are not "visually lossless" codecs like ProRes or Cineform, these are true lossless codecs.

[HDClown]

If it's really as simple as CPU vs Disk, I would default to taking Disk over CPU. Processors just keep getitng faster, and my 2.4ghz C2D is getting to be dated in terms of processing performance. Given that my current processor has no performance related issues with Lagarith, and that I'll just be getting faster CPU's in the future, the savings on disk space seems like a smarter decision.

[jagabo]

The difference in compression between HuffYUV and Lagarith varies. But I would guess your VHS caps will grow from about 30 GB/hr with Lagarith, to 40 GB/hr with HuffYUV. It's easy enough for you to convert a small sample and see how much bigger it gets.

I'd be a bit wary about Lagarith for long term archiving. There is only one source of the Lagarith codec and, as far as I know, it's only available for Windows. HuffYUV has much wider support. So it's more likely you'll be able to easily access HuffYUV AVI than Lagarith AVI 30 years from now. Of course, since you're talking about lossless compression, you can reencode with another lossless codec and container any time between now and then without losing any quality. And you'll probably be able to run Windows in a VM in 2040 if you really need to use the lagarith codec to access your videos! You can even do that now.

[HDClown]

As you pointed out, one of the benefits of using lossless is the ability to re-encode to another lossless codec. That's why I pointed out that no digital format is likely to last forever, and that I'm archiving to lossless so I can encode to whatever the new lossless format du-jor is when the time comes. An extra 10 GB/hr would probably be another 1 TB of disk space by the time my project is complete. In reality, this is probably a year long project, and 12 months from now, disk sizes will probably be doubled. I may need re-evaluate the choice and go to HuffYUV if it has a wider base of development and sourcing.

[themaster1]

lagarith is awfully hard to decompress in my experience whereas huffyuv isn't
MSU has made a test back in 2007 for lossless codecs: see here

[2Bdecided]

I am currently using huffy as my final archival master. When I saw the different relations of lagarith 47.0 Mbps vs huffy 52.2 Mbps, I decided it was best over all to just stay with huffy. Although not tested by me and from that number it seems to indicate that huffy retains more information than lagarith

You might want to think about what the word "lossless" means before writing silly things like this!


LAGS support is rare, so it must be HuffYUV for "long term" archiving. Which is a problem if you want to archive YV12 (not supported by most HuffYUV implementations).

If you really care about long term access, archive a decent lossy copy. MPEG is far more convenient, is far more widely played today, and is probably going to outlive, those lossless formats.

Cheers,
David.

[HDClown]

[QUOTE=2Bdecided;1981752]

If you really care about long term access, archive a decent lossy copy. MPEG is far more convenient, is far more widely played today, and is probably going to outlive, those lossless formats.

For me personally, that would be a complete waste of time and space, and defeat the purpose of what I am doing.

Digitally archiving for the consumer means being aware that you are going to have to migrate their digital copies to new storage and new formats as decades go by. No one should assume that they can simply archive once and never touch it again. You need to progress your encoding and storage mechanisms with technology. As an example, In the not very distant future, you'll have 2 TB on a thumb drive or SSD, and no moving parts at a very cheap price. Far superior to the HDD"s you need to store them on today in terms of longevity of the storage medium. Likewise, codec options will change.

My goal with any kind of digital archiving is to do the original "ripping" or conversions ONCE. That is, by far, the most time consuming part of the process due to limitations in playback devices (8mm, VHS, DV, CD, DVD, BD, etc.) because it's bound by physical limitations. Once you have that lossless digital file, your only time consumption is bound by the speed of your computer, and we know how fast those are these days. There's no good reaosn to spend the time (and space) today to archive a lossy format if you have no use for that format otherwise. Just be cogniscent of the fact you in 5 years, you may need to fire up a batch computer and let your computer re-encode your entire collection for a few hours (yea, computers will be stupidly fast by then, it probably won't take more time then that).

Now, as to the Lagarith vs HuffYUV decision. My only concern here is that if Lagarith only has one support channel, it's more likely that this codec doesn't last as long as HuffYUV, causing me to have to re-encode sooner. Aside from HuffYUV being more widely ported, I don't see an official support team or conglomerate of developers who maintain the open source codec. I've seen things like this end up completely dead in the past, because no one decided they wanted to keep the technology current with modern operating systems. On the other side of that, Lagarith has much more recent release, and seems to be more actively maintained. Sure, that could end any time, since it's just one guy, but at least his code base is more current, which could mean others are more likely to pick up from that base to keep it pogressing.

[creamyhorror]

Transcode to x264 lossless then. MPEG-4 and lossless, best of both worlds. :V

[jagabo]

Transcode to x264 lossless then. MPEG-4 and lossless, best of both worlds. :V

Does x264 have a lossless YUY2 mode? Otherwise he'll be loosing half his vertical color resolution. There are workarounds (point resize to double the height) but that will be time consuming and generate larger files. And it may be difficult to convert back to YUY2 losslessly in the future.

[2Bdecided]

@HDClown,

I'm glad you're happy with your chosen strategy.

Yet a lossy copy certainly has its uses (playing on almost any device being one!), and takes away all worry about having access to the codec in the foreseeable future.

If time and storage space aren't an issue for you, then you've either not shot much video, or edited it meticulously. I've shot far too much, and only rarely get the time to edit it to the point that I'm happy with it.

So even with DV and HDV, I've got too much. The idea that I'd store all the S-VHS as lossless isn't practical yet.

Drives may be cheap - but it's still a lot of data - and you've got to back them up, which means extra cost, space, and time. Then migrate them in five years - more cost, space, and time.

The most dangerous thing for the longevity of your video is that you (or your heirs) will just not bother to make the codec/storage transition at a vital point. Having less data, and having it in a codec with better longevity, makes this less likely. Less data = less of a hassle. Codec longevity = more chance they can play it easily and realise that it's worth copying it.

Cheers,
David.

[creamyhorror]

Transcode to x264 lossless then. MPEG-4 and lossless, best of both worlds. :V

Does x264 have a lossless YUY2 mode? Otherwise he'll be loosing half his vertical color resolution. There are workarounds (point resize to double the height) but that will be time consuming and generate larger files. And it may be difficult to convert back to YUY2 losslessly in the future.

It probably doesn't, but I was mostly being facetious. I wasn't even sure about VHS's chroma format in the first place.

[Khaver]

For archival purposes, the best solution would be to output each frame to a lossless image format such as PNG, TIF or TGA and the audio as RAW or WAV and include a text file describing the resolution, frames per second and audio format so years from now when the current list of codecs you've been talking about here won't work on your fusion powered pocket main-frame anyone will be able to reconstitute the frames and audio back to viable movie format.

[HDClown]

Yet a lossy copy certainly has its uses (playing on almost any device being one!), and takes away all worry about having access to the codec in the foreseeable future.

I don't need most of this content anywhere except digitally archived. It will be years before I need to play it for others on something other than my computer. When the time comes, I'd much rather encode it with the popular choice of the time. There is no guarantee that ANY codec (lossy or lossless) you choose today is around 10 years from now. If we're worried about our children keeping up with it, there's high proability that whatever is chosen today will not exist when we pass on anyway.

If time and storage space aren't an issue for you, then you've either not shot much video, or edited it meticulously. I've shot far too much, and only rarely get the time to edit it to the point that I'm happy with it.

So even with DV and HDV, I've got too much. The idea that I'd store all the S-VHS as lossless isn't practical yet.

Drives may be cheap - but it's still a lot of data - and you've got to back them up, which means extra cost, space, and time. Then migrate them in five years - more cost, space, and time.

VHS I am dealing is all personal home movies kind of stuff, memories of my childhood. Most of the current video This stuff is once in a lifetime. You can't go back and re-shoot. With that kind of content, I am a firm believer in using the best quality possible and doing things in away tha cause no loss to that data.

I don't have a huge amount by any means. My VHS/8mm collections will be somewherei n the realm of 2-4 TB once captured. That's $300 in two external hard drive, one for primary use, one for a backup that can sit in the safe. To only save $100 or $150 at the sacrifice of quality just does't make sense in my book.

It's A LOT easier to perpetuate digital archives with changing technology then it is making sure my VHS tapes stay safe (and that there is something that I could play them on decades from now.

[edDV]

For archival purposes, the best solution would be to output each frame to a lossless image format such as PNG, TIF or TGA and the audio as RAW or WAV and include a text file describing the resolution, frames per second and audio format so years from now when the current list of codecs you've been talking about here won't work on your fusion powered pocket main-frame anyone will be able to reconstitute the frames and audio back to viable movie format.

You are ignoring the losses associated with deinterlace or are you suggesting field by field PNG, TIF or TGA?

A major reason for lossless YUY2 (4:2:2) or YV12 archiving is to access future deinterlace technology.

[jagabo]

I believe all those image formats are RGB to. So you will get losses from the YUV to RGB conversion. Then again when you convert back to YUV.

[poisondeathray]

I believe all those image formats are RGB to. So you will get losses from the YUV to RGB conversion. Then again when you convert back to YUV.

TIF actually has many variations, including a YCbCr 4:4:4 or 4:2:2 format option

Aside from HuffYUV being more widely ported, I don't see an official support team or conglomerate of developers who maintain the open source codec. I've seen things like this end up completely dead in the past, because no one decided they wanted to keep the technology current with modern operating systems. On the other side of that, Lagarith has much more recent release, and seems to be more actively maintained. Sure, that could end any time, since it's just one guy, but at least his code base is more current, which could mean others are more likely to pick up from that base to keep it pogressing.

There is no worry about Huffyuv going obsolete, since there are versions included in open source software such as ffmpeg , avidemux etc. There is even an ffdshow version (encode and decode); it even has a mac port. ffmpeg is cross platform including linux. Conversely, lagarith isn't included in ffmpeg or has any other open source support or decoders

[jagabo]

You are ignoring the losses associated with deinterlace or are you suggesting field by field PNG, TIF or TGA?

I don't think interlacing is an issue. You can just tell your encoder the frames are interlaced once you import them.

There is no worry about Huffyuv going obsolete, since there are versions included in open source software such as ffmpeg , avidemux etc.

But that doesn't mean anyone will be using HuffYUV 30 years from now. It may be hard to find a decoder. Or you may have to find the source and compile your own -- outside most people's ability.

[creamyhorror]

But that doesn't mean anyone will be using HuffYUV 30 years from now. It may be hard to find a decoder. Or you may have to find the source and compile your own -- outside most people's ability.

As long as people are using HuffYUV to encode their precious home videos, I'm pretty confident whatever incarnation of free software decoding packages existing then will include the HuffYUV decoder. There's always going to be demand for some kind of free lossless codec after all. It's when people start switching to another lossless codec that there's some risk of obsolescence, but you'd probably have quite a long window period to do a switchover. It's not like a codec can become totally unusable in 5 years - our OSes don't change that quickly.

[2Bdecided]

As long as people are using HuffYUV to encode their precious home videos...

...but they're not. I was going to say it's just you and HDClown, but he's chosen Lagarith, so basically it's just you!

Seriously, millions of people will have home movies stored on film, tape (analogue and digital), and DVD-R. Many of them still won't have transferred them in 5, 10, or even 20 years time, or the interim transfers will be dead while (just maybe!) they still have the original.* So they'll be a commercial demand to carry out these transfers, and they'll be a hobbyist demand to try and improve them.

Whereas the number of people who have stored their home movie collections in lossless files on HDDs (and number of those files that have actually survived!) is going to be tiny in comparison. Maybe they'll be enough people interested that at least one person digs up the source code, and complies it for Windows 2035. But maybe most other people transcoded to MPEG-53 lossless in 2020, and by 2035 no one is interested in getting HuffYUV to work on a "modern" machine except for you.

Cheers,
David.

P.S. * - I find it amazing that people transferred 8mm film to VHS, and then dumped the original film.

[2Bdecided]

To only save $100 or $150 at the sacrifice of quality just does't make sense in my book.

...but high bitrate MPEG from a decent VCR is going to be higher quality than lossless from a lesser VCR.

Whereas high bitrate MPEG vs lossless is near impossible to tell apart.

Cheers,
David.

[Khaver]

Yes, if needs be, you could save each field to an image if he captures to an interlaced format. Archiving to image frames/fields will insure future ability to re-encode to any available future codec. The only worry then is your storage media.

[jagabo]

Archiving to image frames/fields will insure future ability to re-encode to any available future codec.

Can you name a software package that can EASILY take two series of YUV 4:2:2 encoded TIF files and weave them into interlaced video frames today?

[creamyhorror]

As long as people are using HuffYUV to encode their precious home videos...

...but they're not. I was going to say it's just you and HDClown, but he's chosen Lagarith, so basically it's just you!

Nah, I'm not hung up enough about quality to want to "archive" anything. Preserving every last bit perfectly doesn't matter to me. But there'll always be some hobbyists doing lossless captures around, and I don't see them dwindling as capacities and bandwidths get ever greater.

Maybe they'll be enough people interested that at least one person digs up the source code, and complies it for Windows 2035. But maybe most other people transcoded to MPEG-53 lossless in 2020, and by 2035 no one is interested in getting HuffYUV to work on a "modern" machine except for you.

If MPEG-53 became a popular lossless format in 2020, then you'd have quite a few years to do a transcode from HuffYUV/whatever to MPEG-53. No sweat, not with the speed of computers we'll have then; probably no need to muck around with settings either, since it's lossless to lossless conversion. Who knows, maybe in 2030 we'll have a program to automatically keep track of our videos and upload them to grid storage, where they'll be stored losslessly and made available for friends to see.

Whatever happens, I don't see a real risk in choosing a decently support lossless codec now. It's just a matter of migration when/if the time comes. Of course, if you're a lazy bum, or forgetful, then you may want to stick to a lossy common format to avoid the chance of 'missing the boat'.

[EddyH]

I think we may be missing something dreadfully simple here.

If you're so bothered about the codec going missing in the future - and the programs to play it with - why not, in the middle of all your many, many gigabytes (terabytes?) of lossless video .... also save a few copies of the raw codec, its installer, something like VLC or whatever (+Vdub, AVS?) and maybe a copy of a winXP installer (one ISO, one folder with raw files), so if the time comes and for some reason we're no longer able to convert these files, you can still go back to that extra stuff saved with them and load it into a virtual machine / emulator?

"well heck, those old 2013 PCs were primitive, but it's still got the code to dump that little 1080p video out to something our current machines can understand, and we can vastly overclock it in the VM to get it out quicker..."

going on the lessons of history, existing emulators etc, data files are FAR easier to lose / see disappear into the ether than the programs to read them with, and even then the ideas behind them can be preserved or reverse engineered... so long as you either have the skills or can coerce / pay a programmer into doing it for you. And surely Huff or Lag are small enough that you could probably put a copy of the installer on every single disc you save the video data to without losing more than a couple frames' worth of space?

along with maybe a lower rez copy in highly compressed form so that there's at least SOME flavour of it surviving.

Used to, e.g. quite often include a copy of DivFix and the DivX codec on CDRs full of AVIs back in the day when MPG4 was fairly novel. Often a friend wouldn't have the codec or easy internet access to get it, and the disc would get scratched or their drive wouldn't read it correctly so the file ended up with glitch frames or a missing index...

Can you name a software package that can EASILY take two series of YUV 4:2:2 encoded TIF files and weave them into interlaced video frames today?

So long as you don't mind doing 3 or 4 lines of code, I think the AVIsynth ImageReader function is able to do at least SOMETHING like that? Without going back to read the full manual, it's got an option for reading in YUV-coded bitmaps at least, so there may be some provision for TIF. If not, then Windows Bitmap has survived over 20 years so far - there's a good chance it'll remain a recognisable format for some time yet, and whizzing up a transcoder to autoconvert the images into something that it could read is something I could probably manage as a beginner programming project (the result wouldn't be particularly fast or bug free, but would probably work) if indeed Paintshop Pro/etc's batch image conversion couldn't do some of the legwork.
Making it so it reads the two file streams in the correct order may be the tricky part, but there's other functions that can do that, I just haven't had a need to learn how to use them. Then you do a common or garden field weave and away you go.

Commercial packages, well, I dunno, but image importing isn't exactly a rare thing.

[Khaver]

Using Avisynth, I would separate the fields, convert to RGB using the proper matrix, stretch the levels to full (0,255) and save using ImmaWrite as progressive PNG, TIF or TGA frames. In the future you could weave these back together as fields, or more likely, Bob them to double-the-framerate progressive frames. I think in the future interlacing will be a thing of the past.

[jagabo]

Using Avisynth, I would separate the fields, convert to RGB using the proper matrix, stretch the levels to full (0,255) and save using ImmaWrite as progressive PNG, TIF or TGA frames. In the future you could weave these back together as fields, or more likely, Bob them to double-the-framerate progressive frames. I think in the future interlacing will be a thing of the past.

That's beyond the abilities of many people. And will AviSynth be available on a typical platform in 2040?

[poisondeathray]

There is no worry about Huffyuv going obsolete, since there are versions included in open source software such as ffmpeg , avidemux etc.

But that doesn't mean anyone will be using HuffYUV 30 years from now. It may be hard to find a decoder. Or you may have to find the source and compile your own -- outside most people's ability.

I know you're just playing devil's advocate here, but couldn't you make that argument for any format? It may be hard to find a decoder for any format in 30 years, etc...

Will we even be around in 30 years? Mayan calendar....oohhhhh

But seriously, with open source and free pre compiled binaries for ffdshow, ffmpeg, I highly doubt you will ever have problems - or at least - fewer problems that lagarith or other formats that are not included with free/open source/cross platform software. With pre complied builds and GUI's, you don't have to know any coding at all. Also, you don't have to worry that one day you will have to pay a licensing fee suddenly (ala mpegla)

At least if you transfer it now to some digital format, you won't have to worry about finding a working VHS machine in the future.

And if something else better comes along in the mean time (in terms of compression and/or decoding performance), you can always encode to that along the way

[jagabo]

with open source and free pre compiled binaries for ffdshow, ffmpeg, I highly doubt you will ever have problems - or at least - fewer problems that lagarith or other formats

I agree that the more widespread and simpler the codec/container is the longer you will be able to access the video easily. Maybe we should use uncompressed RGB/YUY2 in AVI in ZIP files?