DV Question!!!!!

[anonymous_whatever]

Hi,

I have a question regarding DV and working with it. I'll outline the issues below. Any and all suggestions welcome!

1. the video being used is miniDV. the DV video is easy to capture, but those file sizes of 13GB/hour are killer!

2. i can't work with video of that size. the comptuer can't handle it and the storage gets to be too much. what are the possible solutions?
- convert to intermediate codec like MJPEG?
- compress using xVID with every frame as a keyframe?
- what is the best method?

3. i'm very aware that working with uncompressed DV video is the best option, but this is not a viable solution here.

Anybody with a general direction of where to go with a problem like this, I would be glad to talk more technically with you.

[aedipuss]

buy an external hard drive.

[anonymous_whatever]

yeah..... not really the kind of suggestion i'm looking for. 100 hours of DV footage is 1TB. not intestested in doing that, especially when miniDV is a perfectly acceptable uncompressed storage format. not to mention that I stated in the original post that the computer can't handle working with uncompressed DV footage...

[poisondeathray]

yeah..... not really the kind of suggestion i'm looking for. 100 hours of DV footage is 1TB. not intestested in doing that, especially when miniDV is a perfectly acceptable uncompressed storage format. not to mention that I stated in the original post that the computer can't handle working with uncompressed DV footage...

DV is compressed. It uses DV compression which is relatively low compression. Uncompressed video would be 10x as large

If your computer can't handle DV, it won't be able to handle much else. It's pretty much the easiest format to edit

DV really is the best option. Get a HDD.

Less compressed formats are easier to edit (easier on the CPU to decode), but take more HDD space.

So is your problem HDD space or CPU power ? What is the bottleneck ?



You might try something like long GOP MPEG at 15Mb/s (which is about 1/2 of DV-AVI), but it's harder to edit because of temporal compression. I-frame MPEG2 at that bitrate won't look so good. You're better off with the original DV.

You can try MJPEG or I-frame Xvid - but they aren't much better. MJPEG is relatively poor compression and you will get artifacts at low bitrates (I'm assuming you don't want higher than the original, which defeats the purpose) , and Xvid gains most of it's compression from temporal (Long GOP). h.264 I-frame is another option that offers better quality/compression ratio, but is harder to decode, and not supported by all editors

You're obviously going to have to make some tradeoffs in either editing speed / smoothness, quality, and/or HDD space.

[turk690]

not to mention that I stated in the original post that the computer can't handle working with uncompressed DV footage...

Then get a computer that will. If you are serious enough, that is. You get a shoe that fits; you do not trim your feet, let alone enlarge them. And anyway, compressing DV into any form of MPEG (to reduce file size, play, edit) will ironically require a more powerful computer than if it was left in its original state.

[anonymous_whatever]

""DV is compressed. It uses DV compression which is relatively low compression. Uncompressed video would be 10x as large""

- yes, very true. for conversational purposes it may as well be uncompressed.


""So is your problem HDD space or CPU power ? What is the bottleneck ?""

- both. there is a lot of hard drive space to go around, but not enough to store large amounts of DV


""You might try something like long GOP MPEG at 15Mb/s (which is about 1/2 of DV-AVI), but it's harder to edit because of temporal compression. I-frame MPEG2 at that bitrate won't look so good. You're better off with the original DV.

You can try MJPEG or I-frame Xvid - but they aren't much better. MJPEG is relatively poor compression and you will get artifacts at low bitrates (I'm assuming you don't want higher than the original, which defeats the purpose) , and Xvid gains most of it's compression from temporal (Long GOP). h.264 I-frame is another option that offers better quality/compression ratio, but is harder to decode, and not supported by all editors""

- i'm looking for something with intraframe compression. whatever is best and can maintain quality.
- GOP MPEG at 15Mb/s sounds like a good start, at half size of DV

- as far as I've been understanding, anything with temporal compression is so tough to edit you may as well not bother. as a result, i'm looking for spatial compression, something with a good intraframe compression algorithm.

- would you say that h.264 intraframe compression would be the best option for editing? it's fine if it's not supported by a lot of decoders because this is just for editing purposes. i would be okay exporting to xvid when all is said and done, but i want a good encoder to give me some smaller, quality videos to work with while editing

[anonymous_whatever]

hope this doesn't sound odd, but i would love to engage in a conversation with you to pick your brain a bit through e-mail or msn. if not, on here is fine and great too!!

[anonymous_whatever]

not to mention that I stated in the original post that the computer can't handle working with uncompressed DV footage...

Then get a computer that will. If you are serious enough, that is. You get a shoe that fits; you do not trim your feet, let alone enlarge them. And anyway, compressing DV into any form of MPEG (to reduce file size, play, edit) will ironically require a more powerful computer than if it was left in its original state.

again, this is not an option. why not buy 3TB worth of hard disk space? why not buy a professional grade computer? why not hire a production team? because these aren't viable options for me. if buying a better computer was an option, i wouldn't have posted.

[poisondeathray]

- i'm looking for something with intraframe compression. whatever is best and can maintain quality.
- GOP MPEG at 15Mb/s sounds like a good start, at half size of DV

That's not intra; Long GOP, usually around a size of ~15 . You need that much temporal compression (P, B frames) in order to get the filesize down that low and maintain a decent level of quality . Harder to edit than DV-AVI, which is buttery smooth even on older computers

- as far as I've been understanding, anything with temporal compression is so tough to edit you may as well not bother. as a result, i'm looking for spatial compression, something with a good intraframe compression algorithm.

- would you say that h.264 intraframe compression would be the best option for editing? it's fine if it's not supported by a lot of decoders because this is just for editing purposes. i would be okay exporting to xvid when all is said and done, but i want a good encoder to give me some smaller, quality videos to work with while editing

That's the problem , spatial compression (intra, I-frame only) isn't very good in general - you require much higher bitrates for a certain level of quality, and the ones that are - they require much higher CPU for other features (e.g. CABAC and inloop deblocking for h.264), and will feel more sluggish to edit on old or slow computers . But if you start to disable all these other new fancy encoding features, then it starts to get as bad as all the old compression algorithms...

[anonymous_whatever]

how about this: what is the best spatial, intraframe compression encoder? hold on.. posted this after your post....

yes, i realize long GOP isn't intraframe but i'm looking for that type of file size reduction, even much more if possible.

so if i understand correctly, pretty much all temperal compression algorithms hard to edit with?

is there no intraframe, spatial compression that maintains decent image quality, comparable to say, the same results you would experience when exporting DV as xVid?

[poisondeathray]

so if i understand correctly, pretty much all temperal compression algorithms hard to edit with?

Yes, they are higher latency, because all the frames in a GOP have to be decoded. Very high compression formats might have a GOP size of 200-400 for example, and will feel sluggish even on modern computers (well HD, anyway, SD is ok)

is there no intraframe, spatial compression that maintains decent image quality, comparable to say, the same results you would experience when exporting DV as xVid?

At what bitrate? xvid looks fine at 100Mb/s . Also the content matters. A video of a white wall with no motion will require very little bitrate. An action movie with explosions will require more. This question is meaningless without context....

Also are you exporting interlaced xvid? or deinterlacing? If you're exporting progressive then there is a whole bunch of other processing factors which affect quality and compression (deinterlacing choices) . I'm assuming you're shooting interlaced (but some DV cameras formats have progressive modes)

[intracube]

anonymous; How old is your computer?

Mine is 5 years old and has no problem editing DV files.

It's unfortunate that hard disks prices are still inflated due to the asian floods, but an extra 1TB drive would really be a good solution.

One possibility is low-res proxy editing. Convert each DV files to a low quality MJPEG for editing in your NLE and also a high quality h264 video as the master . When you're happy with the edit, your NLE uses the h264 files to generate the full quality version.

[lordsmurf]

I have a computer that is 11 years old and handles DV fine.
I don't think the computer is the issue here.

[anonymous_whatever]

Well, decent, action-oriented xVid to maintain a decent filesize. Say, 1 hour per CD. I'm not expecting amazing quality, but watchable. I'm just looking for an intermediate, usable compression codec between DV and xVid.

This is also another point you make. The video is interlaced DV. Should I deinterlace or leave interlaced?

[anonymous_whatever]

so if i understand correctly, pretty much all temperal compression algorithms hard to edit with?

Yes, they are higher latency, because all the frames in a GOP have to be decoded. Very high compression formats might have a GOP size of 200-400 for example, and will feel sluggish even on modern computers (well HD, anyway, SD is ok)

is there no intraframe, spatial compression that maintains decent image quality, comparable to say, the same results you would experience when exporting DV as xVid?

At what bitrate? xvid looks fine at 100Mb/s . Also the content matters. A video of a white wall with no motion will require very little bitrate. An action movie with explosions will require more. This question is meaningless without context....

Also are you exporting interlaced xvid? or deinterlacing? If you're exporting progressive then there is a whole bunch of other processing factors which affect quality and compression (deinterlacing choices) . I'm assuming you're shooting interlaced (but some DV cameras formats have progressive modes)

Well, decent, action-oriented xVid to maintain a decent filesize. Say, 1 hour per CD. I'm not expecting amazing quality, but watchable. I'm just looking for an intermediate, usable compression codec between DV and xVid.

This is also another point you make. The video is interlaced DV. Should I deinterlace or leave interlaced?

[anonymous_whatever]

anonymous; How old is your computer?

Mine is 5 years old and has no problem editing DV files.

It's unfortunate that hard disks prices are still inflated due to the asian floods, but an extra 1TB drive would really be a good solution.

One possibility is low-res proxy editing. Convert each DV files to a low quality MJPEG for editing in your NLE and also a high quality h264 video as the master . When you're happy with the edit, your NLE uses the h264 files to generate the full quality version.

The computer is a Pentium IV, 2GB RAM. I seem to have solved the playback issue, but I just hate working with these large DV files for a variety of reasons.

Maybe I should look into NLE. How does the synchronization work with the MJPEGs and h264 files?

Also, are you suggesting h264 files because they have the highest quality compression next to DV?

[intracube]

Maybe I should look into NLE. How does the synchronization work with the MJPEGs and h264 files?

To clarify, NLE = non linear editor - a generic name for programs like Premiere, Vegas, AVID, FCP, etc.

I can't give any direct advice on the workflow for proxy editing as I've never had the need to use it. Someone else might be able to help you out with the specifics if you say what NLE you're using.

Also, are you suggesting h264 files because they have the highest quality compression next to DV?

h264 could preserve much of the quality of the original DV files, but take significantly less space. But as poisondeathray has already said, you might have problems editing the files directly. That's why you'd edit using low quality copies.

But I don't think this method will help you. You'd need to copy the DV files to your computer prior to converting them - so you'd need the space anyway. If you've got the space, then edit the DV files directly. If you haven't got the space, you'd have to transfer a few DV files at a time to your computer, convert each to h264 and MJPEG, delete the DV files to free up some space and repeat.

Expect this process to take some time...

This method is often used for people editing HD video - where processing power is the limitation, not storage.

[poisondeathray]

The video is interlaced DV. Should I deinterlace or leave interlaced?

Depends on your goal. General rule is leave things as they are. If your export goal was progressive for youtube/web, then it would be better off deinterlacing than leaving it to them. A DVD-video export would be better off interlaced

Also, are you suggesting h264 files because they have the highest quality compression next to DV?

Definitely has better compression, but most of it's compression gains are still from temporal, long GOP.

But in intra mode, it still has better compression than DV

Say, 1 hour per CD. I'm not expecting amazing quality, but watchable. I'm just looking for an intermediate, usable compression codec between DV and xVid.

Ok... I think everyone is on a different page or has very different perception of "quality". 1hour per 700MB ? That's about 1600kb/s. That's rediculous even for full fledged long GOP, extreme compression, difficult to edit compression. Even a low res, low quality SD proxy doesn't go that low.

Remember you incur generation loss when editing, so if you use a lossy intermedite, then export another lossy final format, you will lose quality each time. If you start with garbage to edit, it will look much worse after you're done. GIGO.

DV footage is notoriously noisy. Unless you've shot it with a stabilizer / tripod, very bright lighting, you probably have to do many things to preprocess it as well. Denosing will help filesizes



To put things into perspective, DV is often used as the proxy for difficult to edit HD footage, because it edits so smoothly

You're going to have to make some severe tradeoffs. Decide what you want more, quality, "editabilty", low bitrates. You can't have them all in your situation.

[poisondeathray]

You didn't mention what software you're using to edit. This can impact the choice you might make

I think a resonable alernative would be CBR long GOP MPEG2 at 8000-9000 kb/s.That would be about 1/3 the size of the original . P4 should be able to handle that, the GOP size is fairly low and constant ~12-18 frames. You should be able to retain decent quality if you've preprocessed it , or have exceptionally clean footage.

[anonymous_whatever]

The video is interlaced DV. Should I deinterlace or leave interlaced?

Depends on your goal. General rule is leave things as they are. If your export goal was progressive for youtube/web, then it would be better off deinterlacing than leaving it to them. A DVD-video export would be better off interlaced

Also, are you suggesting h264 files because they have the highest quality compression next to DV?

Definitely has better compression, but most of it's compression gains are still from temporal, long GOP.

But in intra mode, it still has better compression than DV

Say, 1 hour per CD. I'm not expecting amazing quality, but watchable. I'm just looking for an intermediate, usable compression codec between DV and xVid.

Ok... I think everyone is on a different page or has very different perception of "quality". 1hour per 700MB ? That's about 1600kb/s. That's rediculous even for full fledged long GOP, extreme compression, difficult to edit compression. Even a low res, low quality SD proxy doesn't go that low.

Remember you incur generation loss when editing, so if you use a lossy intermedite, then export another lossy final format, you will lose quality each time. If you start with garbage to edit, it will look much worse after you're done. GIGO.

DV footage is notoriously noisy. Unless you've shot it with a stabilizer / tripod, very bright lighting, you probably have to do many things to preprocess it as well. Denosing will help filesizes



To put things into perspective, DV is often used as the proxy for difficult to edit HD footage, because it edits so smoothly

You're going to have to make some severe tradeoffs. Decide what you want more, quality, "editabilty", low bitrates. You can't have them all in your situation.

ugh... yeah it seems so.

the way i see it right now the easiest solution is this: (please correct me anywhere you'd like)

i need something with smaller file sizes than DV. if there is some type of easy to edit encoding scheme, that's what i would be looking for.

what makes it easy to edit? every frame is a keyframe. this is why i would need a codec that is intraframe compression. what are the best options?

if i use long GOP MPEG compression, sure most of the quality will be retained and the file size wayyy down, but editing will be difficult and barely possible.

the way i see it, MJPEG or something of that sort would be the compromise. easy to edit, file sizes down and lots of quality retained. is this the case??

[poisondeathray]

the way i see it, MJPEG or something of that sort would be the compromise. easy to edit, file sizes down and lots of quality retained. is this the case??

Not really , MJPEG suffers from fairly low quality. You need fairly high bitrates to compensate, so file sizes up

But you might be ok with it. Some people are very picky, some less so. Try it out and see if it's acceptable, you can always adjust the quantizer to your choosing (lower quantizer = larger bitrates, higher filesizes, higher quality)

[aedipuss]

mjpeg quality sucks.

[anonymous_whatever]

the way i see it, MJPEG or something of that sort would be the compromise. easy to edit, file sizes down and lots of quality retained. is this the case??

Not really , MJPEG suffers from fairly low quality. You need fairly high bitrates to compensate, so file sizes up

But you might be ok with it. Some people are very picky, some less so. Try it out and see if it's acceptable, you can always adjust the quantizer to your choosing (lower quantizer = larger bitrates, higher filesizes, higher quality)

I'm really focusing on these intermediate codecs like MJPEG because they seem like a great solution. Are there any that work better??

Whatever the case, I realize editing in DV footage is the best option.

compressing using temporal compression = hard to edit

compressing using spatial, intraframe compression = easy to edit

true, no??

[poisondeathray]

the way i see it, MJPEG or something of that sort would be the compromise. easy to edit, file sizes down and lots of quality retained. is this the case??

Not really , MJPEG suffers from fairly low quality. You need fairly high bitrates to compensate, so file sizes up

But you might be ok with it. Some people are very picky, some less so. Try it out and see if it's acceptable, you can always adjust the quantizer to your choosing (lower quantizer = larger bitrates, higher filesizes, higher quality)

I'm really focusing on these intermediate codecs like MJPEG because they seem like a great solution. Are there any that work better??

Whatever the case, I realize editing in DV footage is the best option.

compressing using temporal compression = hard to edit

compressing using spatial, intraframe compression = easy to edit

true, no??

In general yes, but realize there are different types of intraframe compression. For example h.264 using all the bells and whistles will get significantly better quality than mjpeg even at smaller filesizes using intra mode - but is harder to edit, because of features like CABAC, inloop deblocking, are more CPU intensive

So in general, the higher the intraframe compression, the harder to edit as well

DV is relatively lowly compressed (larger filesizes), so it's easier to edit

Tradeoffs

[anonymous_whatever]

the way i see it, MJPEG or something of that sort would be the compromise. easy to edit, file sizes down and lots of quality retained. is this the case??

Not really , MJPEG suffers from fairly low quality. You need fairly high bitrates to compensate, so file sizes up

But you might be ok with it. Some people are very picky, some less so. Try it out and see if it's acceptable, you can always adjust the quantizer to your choosing (lower quantizer = larger bitrates, higher filesizes, higher quality)

I'm really focusing on these intermediate codecs like MJPEG because they seem like a great solution. Are there any that work better??

Whatever the case, I realize editing in DV footage is the best option.

compressing using temporal compression = hard to edit

compressing using spatial, intraframe compression = easy to edit

true, no??

In general yes, but realize there are different types of intraframe compression. For example h.264 using all the bells and whistles will get significantly better quality than mjpeg even at smaller filesizes using intra mode - but is harder to edit, because of features like CABAC, inloop deblocking, are more CPU intensive

So in general, the higher the intraframe compression, the harder to edit as well

DV is relatively lowly compressed (larger filesizes), so it's easier to edit

Tradeoffs

so besides being CPU intensive on the extra features, is h.264 intraframe compression my best bet for working with smaller file sizes? is h.264 generally considered to be best at intraframe compression?

the huge file sizes of DV is the only problem and i'm looking for a way to turn a 12GB 1 hour tape into something that's say.. 4GB or less while maintaing decent quality.. by decent I mean reasonable. I'm not expecting full-on DV quality here. I'm looking for the best solution to do this while keeping it easy to edit.

the way I've been looking at it, if I use any temperal compression scheme at all, it will be hard to edit an way less efficient than spatial, intraframe compression. if h.264 intraframe is best, that's what I'll go with.

[poisondeathray]

Yes, comrpession is better, but tou'll probably hate it with a single core p4, it will be 10x slower to encode, and editing won't be so smooth as DV . Try it out. (also not all editors will support it)

[anonymous_whatever]

Yes, comrpession is better, but tou'll probably hate it with a single core p4, it will be 10x slower to encode, and editing won't be so smooth as DV . Try it out. (also not all editors will support it)

Ok, if that's the best bet I'll give it a shot. Which encoder would you recommend? I've been looking at x.264 or AVC-Intra. any recommendation?

[Cornucopia]

Look, you've already heard the best recommendation: get larger hard drives and stick with the original DV files.

Nobody loads ALL their 100's of hours up all at once anyway. That's not reasonably reviewable, nor a good way to edit. Even if you were making a Hollywood movie you wouldn't need all that at once. Load up 5 or 6 hours (the parts that you need for whatever section you're going to create) and edit it down to ??20 minutes - 1 hour??, then unload those to DVD-R/Tape/OtherArchiveMedium and do it again with the next section. That's less than 100GB for ~7 hours worth - easily manageable with a single, fast, modern harddrive. And not too expensive either.

You seem to be doing all you can to avoid the simplest solution...?!

Scott

[anonymous_whatever]

Look, you've already heard the best recommendation: get larger hard drives and stick with the original DV files.

Nobody loads ALL their 100's of hours up all at once anyway. That's not reasonably reviewable, nor a good way to edit. Even if you were making a Hollywood movie you wouldn't need all that at once. Load up 5 or 6 hours (the parts that you need for whatever section you're going to create) and edit it down to ??20 minutes - 1 hour??, then unload those to DVD-R/Tape/OtherArchiveMedium and do it again with the next section. That's less than 100GB for ~7 hours worth - easily manageable with a single, fast, modern harddrive. And not too expensive either.

You seem to be doing all you can to avoid the simplest solution...?!

Scott

Look, first off, thanks for coming in at the latest point in discussion and throwing in your two cents. Not at all helpful or useful. Second, if this was feasable, I would do it. I'm fully capable of determining my own needs.

Third, I'm able to do simple calculations involving DV storage and processing. Thanks.

[_Al_]

Intermediate could be whatever, say mpeg1 320x240 if you use Sony Vegas. After you finish editing you just replace that long mpeg1 you imported into project at the beginning with original DV avi. Of course this is good only for editing, as soon as you start some color fixing it could be all off using this.
What editor are you using, is it a secret ?