Converting DV to a more compressed new "Source".

[PuzZLeR]

Calling upon this forum for a little help again.

I’ve been working with a lot more DV lately, even experimenting with CineForm – fabulous editing formats. After a few recent threads about archiving, saving, and even compressing - HUGE - DV source, I felt maybe I could put a new twist to it.

Before I begin, I will say the best option for archiving long-term is indeed to keep the source. It doesn’t matter what you encode it to, DvD, iPod, DivX, whatever playback option fancies you, as long as you retain the “Master” so you can keep your options open for max quality again and again for the future.

And this is especially true for your most special stuff – family videos, personal projects, irreplaceable footage, etc.

So this thread isn’t about what’s the best solution. Thanks for understanding.

My question is: how would you compress this DV source for stuff that you are fully willing to sacrifice a bit of quality for a much, much smaller bitrate? I’m talking about storing this, and replacing it as the new “source”.

Again, I agree this isn’t the best solution my friends. But I’m talking about stuff that I don’t care for as much, but still want a suitable high quality source – accepting a slight quality hit for better archival management.

I’m thinking a high bitrate, or high keyframe MPEG-2, (and NO deinterlacing) although still big would still be much, much smaller than DV.

Assuming SD content for the moment, I’ve experimented with CCE using Q=10, Min=0, Max=9000, and in one (FAST) pass get a result visually identical to the DV source - no script necessary, it's "as is". For my less important stuff, I have retained this as the new “source” to encode from as I manage my archiving, but I know with better options I can do even better.

I know it’s still lossy, yes, but this is about one's choice to relieve the storage burdens of less important content and that’s what this thread is all about.

I’m just wondering what other people would do with CCE's Advanced options, or with another encoder, or can suggest.

Thanks for reading my friends. Your advice is always appreciated.

[JohnnyMalaria]

Honestly, given the extremely affordable cost of external hard drives, I'd just keep with the DV. $200 for a 1TB drive = approx 75 x 60min tapes = $2.60 per tape. That's cheaper than many high quality tapes!

Archiving shouldn't be a pain. Easy in, easy out and a faithful copy of the original. Personally, I don't want to fret over getting the right compromise of bit rate and compression artifacts or spend hours converting the material to get it into the archive.

[poisondeathray]

Actually 1TB drives are closer to ~$110.

A 1.5TB drive is $150 @ directcanada.ca . You can't beat that price/GB

CAD prices (since puzzler is from .ca as well), probably even cheaper in the USA

But for discussion purposes, it sounds like you found an alternative for your specific goals. Personally I find HCEnc better than CCE SP2, especially at lower bitrates. i.e. you can achieve visually identical results at a lower bitrate. In my testing across various genres and sources, you can usually shave 10-20% off on average by using HCenc instead. I suggest you do some testing to prove this to yourself.

If your 1st priority is compression, I think the only alternative is h264. I would use TempGaussMC_Beta1 to bob, and even then you would still get better compression than a MPEG2 strategy. Interlaced h264 is still work in progress and not up to par yet. This would take a long long long time. For editing, yes it's harder to edit at the moment, but almost everything has avs import plugins. A new one for CS4 was released recently. You need a fast system, but hey what else to do with all those cores? (unforunately TempGaussMC_Beta1 doesn't scale very well )

IMO it's not worth the hassle. cheap HD FTW!

[poisondeathray]

OOPS double post

[vhelp]

I'll spare you the FAT and be very brief..

It depends: no pixleation, mpeg errors, artifacts, and it could be considered near-lossless* and safe for archiving for later processing.

* When measureing color channels (RGB/YUV) for pic-A vs. pic-B, and numbers are found to be slightly off, this can still be considered near-lossless because there is no getting around most rounding errors, even in lossless, and in this case the perceived view does not indicate loss as does the examples of pixelation, etc., reveal in those cases.

However, the major issue (archiving to new format) has to do with the processing aspects. How much time does it take to process the video to such (near-lossless-ness) format. That's the deciding factor here. And I guess that would depends also on the relation with the HDD space vs. what is left vs. what you need to capture equip for (for next capture session, etc) vs. whatever else might be needed to factorialize in all this.

H264 sounds like a great example and advantage but at the moment, it has some minor drawbacks. For one, there is the issue (though only during playback) that reveals strange artifacts (you've seen an example of this from one of my posts elsewheres in another thread) that there are prob other phenominas not yet addressed, waiting to be realized. But, even still, none of those issues came/come up during the editing stages such as when importing via dgavcindex or other means of importing or parsing through a timeline. So there's still room for hope, yet, with this given codec/format, not to mention that it is fastly becoming the new standards if not already.

As far as buying 1TB drives. Well, that all depends on how you can chain them into a system. At the moment, I have'nt yet seen anything that allows you to buy a fat usb hdd, fill it up, and then buy another one, and then easily addit (chain it) to or ontop of, your current 1TB, making it now, drive E, and F, and G.. etc., unless I misundstand something there. I thought you could chain 127 (usb) devices like this, no matter what they were. But, I guess I'm still confused

But I'm with ya.. I want to do the same thing and have experimented a few times already.. using h264, but the artifact issue scared me off for a while..so until i can figure out the cause/fault, i'm not locking into that concept just yet.

Oh well, that's all I got at the moment.

-vhelp 5010

[edDV]

I grade the archives for likely future use. I also edit out bad takes from the original camcorder source (usually ~25-35%).

Grade A stuff I archive to DV-AVI on both tape and HDD.
Grade B stuff I archive to DVD 720x480i MPeg2 (high bitrate)
Grade C stuff I archive to DVD 352x480i MPeg2 (half bitrate)
Grade D stuff goes to h.264 or wmv

I'm experimenting with interlace h.264 and VC-1 for Grade B/C.

An example of Grade D might be a long lecture or conference procedings.

[PuzZLeR]

Hey guys, thanks for the response!

Reporting back to this thread to say I do agree with the fact that hard drives are dirt cheap today, and maybe I’m a bit fussy – therefore my Grade A (according to edDV’s post) has widened.

But I have TONS of video and DV is still huge (no, not huge – HUGE!). So I will archive in a lossy format for my lesser wares. The MPEG-2 encode doesn't take too long to discourage me anyway.

Yes Poison, HC Encoder is arguably comparible to CCE, and I have indeed seen some sources come out better with it but at such higher bitrates, it shouldn’t make a difference. I’m not compromising on the MPEG-2 encodes, just the DV. Then again, I did spend 2 grand on CCE last summer so I do want it to be productive… :P

As for x264 I will say that it is, IMO, not a good archival format. (It’s actually quite terrible.) Not only will it take forever to encode, not play anywhere currently outside of a PC and is useless for editing, it does not give any compression advantage at the higher bitrates – you are better off with MPEG-2.

And yes, it’s bad at handling interlaced content. As well, as Vhelp mentions, there are indeed artifacts – I see a smearing. This was very noticeable on the big screen, hence I dropped it (which was why I bought CCE - to go along with my new setup).

x264 is great for low-bitrate, somewhat blurry, video. In fact, it’s the best for this. But this concept is either outdated, or only for mobiles, not for professional encodes, or for archival content.

[PuzZLeR]

As I’ve mentioned, I’ve gotten excellent results with CCE. Now I’ve experimented even with Q=1, Min = 0, Max =9800 and get a stunning equivalent of the original. It’s still a big MPEG-2 file, but still much much smaller than the DV source. And yes it’s lossy, but not visually, and I can see no introduction of new artifacts. (Tell me if I need an eye checkup. )

And it's a VERY fast encode when it's straight from the DV source and no filters - and almost literally drop and go. No problem here.

Ok, here's my logic:

Most of us don’t keep the Acquisition Source (ex: DV) really to watch it, only retain it as a Master to encode from it to a lossy Target Format to actually watch it (such as DvD/DivX/Xvid/x264/WMV/etc).

Hence:

a) Acquisition Source -> Target Format
is lossy.

But,

b) Acquisition Source -> Median Source (ex: high bitrate MPEG-2) -> Target Format
shouldn’t be much more lossier if one uses higher bitrate requirements to the Target Format to compensate for the loss along the way to the final product.

I mean, it's lossy anyway either way right?

If b) is lossier, even with the "make up bitrate", then it should be very minor, and probably worth it.

For important stuff (subjective) this isn't the correct way. I know. But this is how I can at least see it for the least important stuff when I can save 80% source space. If anybody's got better ideas, I'd love to hear them.

[pijetro]

Man.....
I've been away from these forums for too long!!!!

Just a thought....
What if you were to recapture the footage to some sort of motion JPEG codec. I was thinking specifically the BlackMagic Intensity pro card....
I'm sure the visual qualities would meet your criteria.
Now, i can't vouch for datarate, but i'm sure it would be less than the current DV rate..

So, no matter how you swing it, you're going to need to spend time either, recapturing, converting, archiving etc....

You're in for many hours of funtime...

[lordsmurf]

Re-encode to I-frame-only MPEG-2 with a bitrate of about 15,000kpbs. That would suffice for excellent, though slightly more compressed, source. I've done this on select projects.

[vhelp]

regarding HDD as replacement for archive storage purposes..

I think a lot of people get some issues confused with something else, or I don't know. I mean, most of our video needs stem from daily captures, not: family videos, which may be only a few tapes or so. I believe that we are actually refereing to every day (dv) captures. This consumes fast (or as PuzZLeR puts it) HUGE files, not to mention, MANY. So, buying HDD's is not exactly the solution.

regarding compression/codec and HDD's storage solution..

But what may be (partially) a solution, is HDD's and optimum compression/codec format, for final storage.

DV offers 5:1 ratio, and if I recall reading recently, MJPEG can go as high as 8:1, but I haven't tested that and can not say pro/con wise for mjpeg.

But, h264 is still an option. Its a matter of fine-tuning it: finding the right combination of param setup for max compression w/out loss or near-lossless: ie, --qp vs. --crf, followed by other key (param) elements for a finished h264 video. fwiw: these days, when I encode (via x264 cli cmd-line encoder) I now encode to the raw .264 instead of the mp4 I was previously doing. I found that the mp4 causes more problems then worth when trying to bring them into a timeline for review/analizing: ie, dgavcindex. As it turns out, dgavc will open straight (or, raw) encoded .264 vids when useing x264 cli tool..been using ever since I discovered it and I no longer have to apply double-duty work: ie, creating .ts or new .264 vids from third-party tools.

Arriving at near-lossless would have to mean at least one absence, pixelation. I've said it many times. That is the single most obvious case to the eyes. After that, it comes down to matrix blocks: ie, 16x16 or 8x8 or 4x4, etc., and comparing such matrixes to one another: source vs. compressionized source.

And, as I type away this response, I'm currently encoding an .264 vid via x264 cli tool. I've been attempting all sorts of test encodes for this purpose (this discussion) as well as others. Its an never ending endeavor, though a hobby..case anyone doesn't realize..again, its a hobby. Oh, there goes my gui's "chime" souding that my encode is complete, hehe.. guess I'm not finished here..lots more, but I gotta run..

-vhelp 5033

[PuzZLeR]

I think a lot of people get some issues confused with something else, or I don't know. I mean, most of our video needs stem from daily captures, not: family videos, which may be only a few tapes or so. I believe that we are actually refereing to every day (dv) captures. This consumes fast (or as PuzZLeR puts it) HUGE files, not to mention, MANY. So, buying HDD's is not exactly the solution.

This is why I started this thread. I was concerned when posting that many may not have seen this point. Not everything falls under "family videos". Most are basic captures, etc. and much too awkward to store at 13GB/hr, which is still 5x less than raw uncompressed, but still HUGE.

Re-encode to I-frame-only MPEG-2 with a bitrate of about 15,000kpbs. That would suffice for excellent, though slightly more compressed, source. I've done this on select projects.

Indeed. As I mentioned in my first post, high keyframe encoding was a possibility. Now, with your post, I’m currently testing with all I frames (Advanced options in CCE).

I’m still using CCE’s One Pass VBR since I love the speed and the fact that the video will be smaller with lower complexity in some cases.

However, I seem to reach some form of equilibrium converging at around 12,000kbps to 13,000kbps, a bit lower than your 15,000kbps (which is good news). Anything more than this point seems to be overkill, anything less increases the rate of lossiness.

Using VBR one pass, Q=1, Min=0, Max=12,500 and all I frames for the “Median File”, I find that:

DV source -> Target Format

is virtually no less lossy than

DV source -> MPEG-2 Median File -> Target Format

Both lower bitrate Target Formats seem equivalent, even with the same Target Format encoder settings in both experiments. And if there’s a difference, it wouldn’t be by much. This "low delta" should suffice for me. I wish I could translate this to HC Encoder similar settings though for others to test/benefit from.

Comparisons were done with encoding to MPEG-2, but it was re-encoding to MPEG-4 afterwards with quantizers determining quality retention that quantified it for me, along with LOTS of closeup squinting at the monitor… :P, which leads me to my next point...

So, no matter how you swing it, you're going to need to spend time either, recapturing, converting, archiving etc....

You're in for many hours of funtime...

I've been attempting all sorts of test encodes for this purpose (this discussion) as well as others. Its an never ending endeavor, though a hobby..case anyone doesn't realize..again, its a hobby.

You guys have it downpacked. Yes. It’s time consuming. Yes, I’m still testing, testing, testing. In fact, I’m running several different tests right now confirming my original hypothesis. Really, if I didn’t love this hobby so much I wouldn’t have bothered…

[lordsmurf]

I almost wrote 12,000kbps in my post. I go back and forth myself, between 12 and 15.

[edDV]

If these are just SD TV captures what further editing are you expecting? Why not trim the DV and encode to traditional 4-6Mb/s DVD MPeg2 or ~3Mb/s MPeg4?

I record "clips" in DV format off TV using my Canopus ADVC-100 but then trim and encode those to MPeg2, xvid or wmv for archive. Most TV shows, documenataries or news shows get captured direct to MPeg2 or wmv.

DVR HD series just get captured as broadcast to MPeg2 ~10 to 19 Mb/s. Some of these are saved.

[vhelp]

* an experimental vhs upconvert to 1280x720p test encode via mpeg-2

I was going to go on a bit more, but thought I would save them for another time because I thought of an idea for you to try..

If you are up it:

You should post a few dv_original and mpeg_encoded demos so that the members following along can review it and even try there hand at encoded the dv_original ones, for comparison purposes. From this, you might discover some weakness in your process and could adjust it and possibly obtain better results.

And, I think I have a good example (because I was encoding a few scenes from it myself) for you to add into this upload. In one of the (dv and mpeg) demos, from the movie: "Contact" use the scene where Jody Foster is in the "pod" and the chair had just broken off -- its near the last 10 minutes of the movie -- From there, the scene gradually goes dark and then light again, and she begins to hover (no gravity i guess) and you see the pod spinning while she's in the center of it and slowly gliding like a bird. Anyway, its a very interesting scene. Also, the scene (before that one) where she is dropped through and begins her speedy journey through the wormholes. Another spectacular experience. And when you're finished, give homage to Carl Sagon (he's in the ending credits) another brilliant (space) scientist, imho, and I miss him and his special series: he has an interesting way of speaking

The reason I used the above "Contact" as an example was because I've seen this and captured it many of times (when I had satellite in 2001 or so) and the background of that sene always gave me a hard time, and at the time I didn't quite realize the reasons for the problems encoding it: but I do by now, and will let you figure it out if you care, for fun

And then pic other video content sources for the other tests..the more the merier.

To maximize the DV per minute/size for uploading:

Run it through winrar with these settings under winxp because it seems to compress the greatest vs. win98 -- at least in my experience:

[x] Create solid archive
[x] Use multimedia compression

1. If you find a better a compressor that compresses even small, then use it. My try 7z, but I'm not sure of its compressability nor optimum setting for videos.

2. don't use megaupload, nor rapidshare because these are the two major ones that most employers block.

3. make them eaiser to download

Try for a 1 minute (or more) section, unpresses for dv_original, give or take 100 to 200 MB in size should be enough for anyone with faster D/L access.

-vhelp 5034