Lagarith vs HuffYUV, archiving and file size

[2Bdecided]

It's hard to predict the future.

A few decades ago, people were crying about the loss of self-destructing nitrate film stocks. How should they be preserved before they were all lost?

1. One school of thought was to transfer it onto the new hi-tech "video".

2. Another school of thought was to transfer it onto modern safety film stock. When video became mainstream, then making real film copies looked expensive in comparison.

3. Yet another school of thought (and a minority one at the time) was simply to store the stuff properly at low temperature and moderate humidity.

Guess which one of the three worked best? Number three!

Cheers,
David.

[jagabo]

Let's see how good those film copies look 1,000 years from now. 10,000 years? 100,000 years?

[Khaver]

Using Avisynth, I would separate the fields, convert to RGB using the proper matrix, stretch the levels to full (0,255) and save using ImmaWrite as progressive PNG, TIF or TGA frames. In the future you could weave these back together as fields, or more likely, Bob them to double-the-framerate progressive frames. I think in the future interlacing will be a thing of the past.

That's beyond the abilities of many people. And will AviSynth be available on a typical platform in 2040?

Use Avisynth now to export to progressive image frames. In the future, read the included text file to find image res, aspect and fps. Then use legacy2DImageto3DVideoHologram to encode the video to something watchable.

[creamyhorror]

Guess which one of the three worked best? Number three!

If you just mean that the future is hard to predict, sure, anything could happen. But we make the best decisions we can with our current knowledge. Archiving with HuffYUV/Lags now is conservative and relatively safe in terms of future accessibility, for reasons we've been giving.

[2Bdecided]

LAGs is neither conservative nor safe, for reasons that have been given!

If you don't care about file size, store raw uncompressed video in an AVI. That is safe. Even an alien species could decode that from scratch in less than an hour.

Cheers,
David.

[magillagorilla]

This thread has been helpful. I am about to archive a bunch of family VHS and SVHS-C. I had a thread about it last week, where I was trying to use 25Mbps MPEG2 as my archive format. I am thinking about this differently now because I cannot predict how I will want to use the footage in the future.

I do a lot of audio archiving and use lossless FLAC. I don't know why I considered treating my video any differently. Using MPEG for video would be the equivilant of using MP3 for audio, and I very much dislike MP3 for high fidelity applications, even at 320Kbps.

I'm having trouble getting Huffyuv to work. I right-click/install on the ini file and see the install script flash quickly. The codec is not an option in any of my capture applications, virtualdub for example. I am running Windows 7 64bit.

Any help on getting Huffyuv to work would be appreciated.

[jagabo]

I'm having trouble getting Huffyuv to work... I am running Windows 7 64bit.

https://forum.videohelp.com/threads/319943-VDub-setup-for-VHS-Capture?p=1981191&viewful...=1#post1981191

Be sure to use the 32 bit HuffYUV if you are using 32 bit VirtualDub. If you are using 64 bit VirtualDub you'll need a 64 bit HuffYUV. The above instructions are for the 32 bit version.

[magillagorilla]

I'm having trouble getting Huffyuv to work... I am running Windows 7 64bit.

https://forum.videohelp.com/threads/319943-VDub-setup-for-VHS-Capture?p=1981191&viewful...=1#post1981191

Be sure to use the 32 bit HuffYUV if you are using 32 bit VirtualDub. If you are using 64 bit VirtualDub you'll need a 64 bit HuffYUV. The above instructions are for the 32 bit version.

Easy enough. I'll try that tonight. Thanks!

[2Bdecided]

Using MPEG for video would be the equivilant of using MP3 for audio

Only in theory, because one is lossless and the other is lossy - but in such a comparison you're ignoring the information content of the original (or lack of it).

In that sense, copying VHS to 25Mbps MPEG-2 is like copying vinyl to CD.

Copying VHS to lossless is like copying vinyl to DSD-wide (352.8kHz 8-bit).


Another thought: lossless audio is ubiquitous. Anyone can burn a CD for a few pence. Lossless video is quite rate, especially in broadcasting where all current studio video formats are lossy (not to mention the broadcasts, which are very lossy).

Of course using lossless does no harm in the short term - for intermediate storage when processing it often does a little good. I have hundreds of GBs of LAGS-AVI files on my HDD for this very purpose. But I won't keep them forever.

Cheers,
David.

[magillagorilla]

Only in theory, because one is lossless and the other is lossy .

Maybe I am misunderstanding you. MPEG is most definately lossy as is MP3. Your analogy of VHS is to DVD as Vinyl is to CD is good except I would argue that Vinyl, in some circumstances sound better than CD. And in the case of VHS, the original VHS can look better than a DVD copy of the same tape.

On the audio side I can use bad DACs, sample at 16bit, filter, EQ, de-noise and burn to CD and get something that sounds pretty bad. On the other hand, I can use nice DACs, sample at 24bit, process, downsample to 16bit and often get a better sound.

This I am assuming is the same logic of capping video in raw or huffyuv AVI and processing before recodeing to your codec of choice.

What is your codec of choice for archiving video long term?

[2Bdecided]

I wasn't disputing the mathematical truth of lossy vs lossless.

I was trying to make the point that in some circumstances it makes a visible / audible difference, and in others it doesn't. In some circumstances lossless is an easy, safe, widely supported solution, in others it isn't. If space really doesn't matter, don't use compression at all - then there's no worry that the codec will be obsolete.


For very important short-edits, I've (so far) kept the lossless versions (either Lagarith, or HuffYUV, depending on whether I rendered YV12 or YUY2 at the end).

I always keep a lossy version too - if its progressive, then I'm happy with MPEG-1 (really - you try finding something that won't play it!) or MPEG-4 AVC if I'm trying to save space. If it's interlaced, then I haven't found anything that I'm 100% happy with, but I usually use DV. High bitrate MPEG-2 would be OK too, but DV is quicker to encode.

Cheers,
David.

P.S. to my knowledge, no one has ever ABXed an LP from a CD-R made from that LP.

[lordsmurf]

I've seen loss in "lossless" video captures. I've been saying this for years now. It's become more rare, however, as the versions of codecs progressed.

[magillagorilla]

I've seen loss in "lossless" video captures. I've been saying this for years now. It's become more rare, however, as the versions of codecs progressed.

That is just an odd concept to me. All of my archiving background is in the audio realm. If I record a WAV file from an analog source then compress it losslessly in FLAC, then uncompress the FLAC, the resulting WAV file is mathmatically identical to the original.

What you are suggesting is that codecs, like huffyuv, are not actually lossless. If you cap an analog source in RAW AVI then run it through HuffyUV, then convert the same file back to RAW AVI data is lost? I apologize if that questions sounds naive, I'm just trying to get a grip on all of this.

It's all a quite different than my friendly WAV/FLAC world. Digital video seems to be a world of art and science.

[poisondeathray]

What you are suggesting is that codecs, like huffyuv, are not actually lossless. If you cap an analog source in RAW AVI then run it through HuffyUV, then convert the same file back to RAW AVI data is lost? I apologize if that questions sounds naive, I'm just trying to get a grip on all of this.

no, they are mathematically lossless , just like flac for audio

but how you use or handle them can cause losses

e.g. if you have a capture in YUY2 4:2:2 color sampling but import it into an editor such as premiere pro or vegas , they will do an internal RGB conversion , and there is slight loss in that colorspace conversion - so how the "lossless" file is interpreted/used is important. If you stay in the same colorspace (e.g. do your editing in avisynth or an editor that works in YUY2 for huffyuv) , there is no loss

If you capture uncompressed AVI , convert to huffyuv, you can go back and forth infinite times without any loss - as long as you don't use another program that "interprets" it differently or does a colorspace conversion. You can do difference masks to compare versions (it would be like comparing audio waveforms and digitally subtracting the differences - you will see the image is bit for bit identical)

[lordsmurf]

Somebody I know has been running tests on this -- boredom, I guess -- and there are definitive losses among "lossless" formats. It's so minute to almost be not worth arguing, but it is most definitely present. It can vary from version to version, too. I don't have all the details, and I really don't care right now -- maybe later. Again, very small.

I've not seen noticeable loss in years now. Noticeable loss was uncommon anyway.

This excludes colorspace conversion.

By it's very essence, any program can interpret it difference. And all of them generally do. For example, "DV" is not a single format, but a basis for many versions of the format. Canopus v Canon v Sony v Matrox, etc.

The math may "prove" losslessness, but it's not necessarily accurate. Don't get dogmatic about it, that's a mistake.

[poisondeathray]

Somebody I know has been running tests on this -- boredom, I guess -- and there are definitive losses among "lossless" formats. It's so minute to almost be not worth arguing, but it is most definitely present. It can vary from version to version, too. I don't have all the details, and I really don't care right now -- maybe later. Again, very small.

I've not seen noticeable loss in years now. Noticeable loss was uncommon anyway.

This excludes colorspace conversion.

By it's very essence, any program can interpret it difference. And all of them generally do. For example, "DV" is not a single format, but a basis for many versions of the format. Canopus v Canon v Sony v Matrox, etc.

The math may "prove" losslessness, but it's not necessarily accurate. Don't get dogmatic about it, that's a mistake.

WTF? "Lossless it's not necessarily accurate?" Actually by definition it is perfectly accurate.

Perhaps that somebody is doing things a bit differently (i.e. incorrectly), or in a specific context? or using a buggy codec?

Even an RGB capture may be interpreted differently in Premiere or editing software (PP will clamp levels to 16-235 on RGB imports for most formats , even though there is no colorspace conversion)

When done properly, the decoded image is lossless when using lagarith or huffyuv. Bit for bit identical over infinity generations. This can , and has been proved with difference masks, psnr, ssim etc..

[jagabo]

I've tested HuffYUV 2.1.1 with thousands and thousands of frames. The YUY2 video that comes out is always exactly the same as the YUY2 video that went it.

The huffman compression scheme is lossless:

http://en.wikipedia.org/wiki/Huffman_coding

Huffman coding (and variations thereof) are used in most archiving programs (like WinZip, WinRar, etc.) where absolute losslessness is required. Any losses in HuffYUV would be due to a bug in the codec or mishandling of the video outside HuffYUV. I've seen no evidence of a bug like this in HuffYUV. Also note that HuffYUV hasn't been updated since 2003. So it's not possible that it has improved with time. I can't vouch for other HuffYUV implementations (ffdshow, for example) but I've seen no evidence of problems other than Lordsmurf's vague "I knew some guy had had a problem once" allegations.

[lordsmurf]

There are a number of unknowns. I just know what I've seen and what others have said. I'm just not prepared to accept absolutes as easily as others are. If there's any doubt, it's worth looking into, that's my only point. While it should be perfectly lossless, there's a chance it may not be under some circumstances. What those are, I don't know -- it's not something that matters to me right now. I'd say it's at least 95-99% to the original, and that's fine enough for anybody at this site.

[2Bdecided]

For example, "DV" is not a single format, but a basis for many versions of the format. Canopus v Canon v Sony v Matrox, etc.

But DV isn't lossless!

More importantly, DV and MPEG decoding isn't perfectly defined - there are rounding errors in DCT implementations. So the same encoded file won't give you exactly the same decoded raw video on all systems.

The lossless codecs don't have these issues. They use bit-perfect defined algorithms.

It's possible that some early version, compiled incorrectly, didn't work properly. I've never seen evidence of that, but I'm giving you the benefit of the doubt.

Even if this did happen, I can't imagine it stayed around for long. One of the most basic tests of a lossless codec is that it's lossless!

Cheers,
David.

[creamyhorror]

If there are any differences at all in encoding and decoding a video using a lossless codec, then it's absolutely a critical bug. Lossless means lossless, bit-identical, absolutely no difference, etc. and a bitwise comparison will prove it.

If Lordsmurf's friend really found differences, he should report it to the codec developers. Like the rest of you, I'm doubtful that it was more than an isolated bug or simple user error.

[lordsmurf]

Isolated bug would be my guess, too.

But the fact that such a bug can even exist is more of what I was getting at. My whole point is to question the dogma that "lossless is always lossless," as I've seen or heard about cases when it's not the case. There are a lot of variables at play. And again, the variance was about 1-2% at most. As with anything else, test. Don't just assume.

[asuranii]

LAGs is neither conservative nor safe, for reasons that have been given!

If you don't care about file size, store raw uncompressed video in an AVI. That is safe. Even an alien species could decode that from scratch in less than an hour.

Cheers,
David.

There are a number of unknowns. I just know what I've seen and what others have said. I'm just not prepared to accept absolutes as easily as others are. If there's any doubt, it's worth looking into, that's my only point. While it should be perfectly lossless, there's a chance it may not be under some circumstances. What those are, I don't know -- it's not something that matters to me right now. I'd say it's at least 95-99% to the original, and that's fine enough for anybody at this site.

both of you have a serious problem.

[isidro]

Isolated bug would be my guess, too.

But the fact that such a bug can even exist is more of what I was getting at. My whole point is to question the dogma that "lossless is always lossless," as I've seen or heard about cases when it's not the case. There are a lot of variables at play. And again, the variance was about 1-2% at most. As with anything else, test. Don't just assume.

Lossless is always lossless, chance of loss due to tested codec is 0%. Chance of loss are higher by other causes, I have seen faulty IDE cable which introduced one or to wrong bits per CD/DVD recorded, thats why I always bit compare backups. FLAC is better than WAV because it has inner CRC integrity check. It's also always advisable to have the CRCs of all backuped files, you can check integrity and use the alternative copy if necesary. HDs are not perfect either, chance of bit loss is around 10^-15 but total HD bits are near 10^14, so you could have a wrong bit every 10 hard disks...

I think chosen codec will last more than hardware needed for reading stored media (HD, DVDR, etc), it would be advisable to make an extra copy every 5-10 years, and recode to newer codec if necesary then.

Using a lossless codec for VHS is probably arguable (I would use constant Q xvid to a quality higher of the original), but it's a personal choice. For lossless, anybody knows if FFV1 faster and with better compression than both Lagarith or Huffyuv?

[themaster1]

huffyuv is perfect for capture whereas lagarith is an excellent intermediary codec with it's YV12 support.
Example: you have a slow ass avisynth script you want encode in mpeg2 with hcenc. Going avisynth> lagarith first will make hcenc happy

[jagabo]

For lossless, anybody knows if FFV1 faster and with better compression than both Lagarith or Huffyuv?

Better compression? Yes. Faster? No. On my i5 2500K both compress at more than twice the speed of FFV1.

[Kereellis]

Sorry about necromancing the thread, I just thought lordsmurf raised an interesting point earlier (as one might expect from him). I've taken quite a few screenshots of huffyuv encodes (in .png) of various captured videos, and while all the detail was still preserved down to the pixel, upon closer inspection in Corel Photopaint the individual rgb values of the pixels in screenshots would apparently randomly deviate from the original by 1 or 2 (most often observed was an increase in the reds by a value of 1 out of 255). Someone with a more technical background could have dug into something like this, but it simply seems the case that there hardly exists much doubt beyond the assumption that lossless = lossless. However, knowing one was more saturated I was able to discern between certain screenshots of the original and the huffyuv version with the naked eye at some point in Windows image viewer (of course I was being overly scrutinous, but I could still tell without looking at the file name which one was which), so this gives me reason to doubt Photopaint was at wrong. The results were also too consistent (that is the increase in reds on encodes) to assume that they were coincidentally introduced by the image viewers used. Mind that I used the original huffyuv as a decoder, not the directshow one. Screenshot methods were constant and identical. Compress -> decompress-> screenshot would produce another color value compared to both the original and in some cases (not tested as extensively, but enough to confirm some negligible inconsistency). I don't know what would happen if the process was repeated, so this is where's room for some research if someone else would find similar results to mine (just mind that not every frame nor every pixel was different). If there was some tool that could automatically compare differences between frames of the same number in two videos that would greatly facilitate the process. (Edit: there is also the option of a crc check between uncompressed and compressed-> decompressed video. Not tried actually.).

However, even if one would take my findings to be correct, this is no reason to panic in the case of VHS, since obviously the color of the capture is simply not 100% 'accurate' to begin with. It's already determined by the vcr, then further altered by the capture card, further tweaked by it's contrast and brightness settings, not to mention procamps that are meant to improve quality. Like I mentioned, the detail itself was far from lost in all of my simple tests and the color difference didn't constitute a decrease in colorspace. Unless one would want to recompress the video endlessly, I don't think there would be much to worry about, other than that analogue captures themselves are inherently not lossless (which may be for the best, I don't mind my JVC's DNR ridding my material of chroma noise).

On a related note, this long-term speculation on lossless codecs to me seems a bit redundant and theoretical to me, since in as little as ten years decompressed YUY2 videos should hopefully be nothing relative to available harddrive space (and that comes from a paranoid purist with tens of hours of captured homevideo material like myself). In the worst case, I'd just keep an old PC/OS that supports the codec of preference until that day comes.

[jagabo]

I've taken quite a few screenshots of huffyuv encodes (in .png) of various captured videos, and while all the detail was still preserved down to the pixel, upon closer inspection in Corel Photopaint the individual rgb values of the pixels in screenshots would apparently randomly deviate from the original by 1 or 2 (most often observed was an increase in the reds by a value of 1 out of 255).

You are getting losses from colorspace conversion, not the compression. If you start with YUY2 video and compress with HuffYUV it will be truly lossless (ie, the YUY2 video that comes out of HuffYUV will be exactly the same as the YUY2 that went in). If you start with RGB video and have HuffYUV set to convert toYUY2 you will get losses. If you start with RGB video and compress as RGB the compression will be lossless (but you won't get as good compression).

In short, with HuffYUV:
YUY2 -> compressed as YUY2 -> decompressed as YUY2 --> lossless
RGB -> compressed as YUY2 -> decompressed as YUY2 or RGB --> lossy
RGB -> compressed as RGB -> decompressed as RGB --> lossless

Of course, there may also be colorspace conversions in the software you are using to feed HuffYUV and in later processing. Different programs may use different algorithms: rec.601, PC.601, rec.709, PC.709, and variations (rounding errors, upsampling, downsampling) on how those algorithms are handled in different programs.

The conversions of RGB to YUV and YUV to RGB are not lossless. Hence a round trip conversion of RGB to YUV to RGB is not lossless. There is not a unique YUV value for every RGB value, and vice versa.

You need to analyze your workflow to determine where your losses are occurring. The problem is not in HuffYUV.

[jmac698]

I believe Mr. Smurf has a good point. I accept that I can't blindly trust any tool unless I test it myself. I know that you can get slightly different results in decoding mpeg2 or mp3's for example. Also you need at least 10 bits to store the results of yuv<>rgb conversions. In fact you could say that any conversion to RGB is lossy, because YUV can specify negative RGB colors. These undisplayable colors can be practically used in YUV though; if you boost brightness of dark scenes in YUV, you can see muddy colors in the shadows. Those colors would be cut off iin RGB. In YUV you can boost them from the negative range into the viewable range.

As for lag vs huffy vs ffv1, I find that ffv1 compresses better. UT compresses as well and faster than lag.
http://forum.doom9.org/showthread.php?p=1463278#post1463278

[Kereellis]

I've taken quite a few screenshots of huffyuv encodes (in .png) of various captured videos, and while all the detail was still preserved down to the pixel, upon closer inspection in Corel Photopaint the individual rgb values of the pixels in screenshots would apparently randomly deviate from the original by 1 or 2 (most often observed was an increase in the reds by a value of 1 out of 255).

You are getting losses from colorspace conversion, not the compression. If you start with YUY2 video and compress with HuffYUV it will be truly lossless (ie, the YUY2 video that comes out of HuffYUV will be exactly the same as the YUY2 that went in). If you start with RGB video and have HuffYUV set to convert toYUY2 you will get losses. If you start with RGB video and compress as RGB the compression will be lossless (but you won't get as good compression).

In short, with HuffYUV:
YUY2 -> compressed as YUY2 -> decompressed as YUY2 --> lossless
RGB -> compressed as YUY2 -> decompressed as YUY2 or RGB --> lossy
RGB -> compressed as RGB -> decompressed as RGB --> lossless

Of course, there may also be colorspace conversions in the software you are using to feed HuffYUV and in later processing. Different programs may use different algorithms: rec.601, PC.601, rec.709, PC.709, and variations (rounding errors, upsampling, downsampling) on how those algorithms are handled in different programs.

The conversions of RGB to YUV and YUV to RGB are not lossless. Hence a round trip conversion of RGB to YUV to RGB is not lossless. There is not a unique YUV value for every RGB value, and vice versa.

You need to analyze your workflow to determine where your losses are occurring. The problem is not in HuffYUV.

Yes, I realized that this would serve as the most satisfying explanation for what I've witnessed, but unless I'm overlooking a setting (or vdub doesn't serve the content properly) there may be something wrong with huffyuv. Whatever the case, there is a very realistic chance a lot of encoders have unknowingly been converting colorspaces while trusting that they were achieving a losless conversion.

For clarification: my raws are all in YUY2. I've always used fast recompress in vdub. Also, manual input/output selection for YUY2 in normal recompress mode makes no difference to this result. Huffyuv's options (such as force RGB) aren't checked. The codec is however set to convert to YUY2 in the case of RGB compression, but that shouldn't be relevant (that is to say, the only thing I can think of beyond that huffyuv isn't working properly, is that vdub still serves the content as RGB in spite of my settings. I'll try a newer version of the program and see what happens as a next step).

EDIT: Sadly, there was no difference in my findings for using the same compression method in vdub 1.9.11. Maybe someone could write me a simple avisynth script that does little else than compress a YUY2 file to huffyuv so I can narrow down the number of possible causes?

[jagabo]

I've tested huffyuv many times myself using binary file comparisons and AviSynth subtraction scripts. After several generations of huffyuv recompression, comparing all types of video, even random noise, I've never see a single bit difference as long as the video stays in YUY2 or RGB.

Of course, that isn't absolute proof that HuffYUV can never made an error. But I'm pretty confident in its abilities.