Append 3 AVI (HuffYUV) files and encode them as 1 MP4/MKV in one step

[abolibibelot]

Hi,

I edited a family movie which is about 3 hours long with lots of added material (pictures and extra video excerpts and titles and whatnot, up to seven tracks are used at some points), using Magix Video Deluxe 17. At first I tried to export the whole thing in 1280x720 MP4 (using the internal encoder, from MainConcept if I'm not mistaken) : I let it encode while I was sleeping, when I woke up I saw that it had failed around a third (and the resulting file was unreadable of course). Then, considering I had size constraints (I want to send it on a DVD-R with other stuff, so it has to be around 3GB, whereas if the first encode attempt had finished it would have been at least double that, with the default settings), and that the x264 encoder is supposed to be more efficient than that of any NLE software, and that there is apparently as of yet no option to frame-serve from Magix softwares to external encoders (they're not supported by Debugmode FrameServer), I decided to export the whole movie in lossless HuffYUV, then encode it in H264 ; but it failed again : the software either crashed or showed an “out of memory” kind of message, even on a fresh Windows session, even after I massively increased the size of virtual memory (it did better with 4.5GB than the initial 2GB -- plus the 3.25GB of physical RAM, maximum under a 32-bit system -- but still failed a little past halfway, for some reason the software couldn't use more memory past a certain point ; it even became unstable when I added a 5GB pagefile on another partition). So I resorted to export it in three segments of about one hour, cleanly cutting at fade-to-black points, and it worked (I still had to close the damn software and re-open it between each export operation, otherwise it would still get clogged the second time around).

Bottom line : as the title says, I have three 1280x720 AVI (HuffYUV) files that I want to encode as one 3GB MP4/MKV with the best possible quality.

Now :
- How can I append those three files and encode them as one seamless MP4 or MKV file with a target file size in one step ? Handbrake doesn't seem to permit this ; Avidemux probably does (it can append files) but has its quirks and only supports Faac as AAC encoder ; then there's MeGUI which I'm not familiar with, which uses Avisynth scripting, and I know Avisynth can do about anything, and it supports other AAC encoders which are reportedly of better quality (like Nero's). Is MeGUI the right tool for the job, and what Avisynth command would tell it to seamlessly append those three files ? So far I can't find a way to do that within the GUI.
- What encoding settings should I use to get the best possible quality for the intended size of 3GB ? It's mostly people talking in a room and mostly steady shots, so little action, and the main video sources are AVCHD files from a compact digital camera (Panasonic ZS3).
- Is there a consensus about what is currently the best AAC encoder, among those available as plugins for third-party softwares ? Does it actually matter, or is the difference insignificant ?
- The edited movie mixes various audio sources with various sampling rate (48kHz, 44.1kHz, and even some 16kHz) ; I exported audio in PCM 48kHZ and plan on encoding in AAC at 48kHz as well : is it the best choice, or would I get a slightly better quality / more efficient encoding with a 44.1kHz rate ?

Thanks in advance.

[LMotlow]

Other problems aside, you might want to rethink your project.

1280x720 = not allowed for DVD
AVCHD = incompatible with DVD
x264/h264 = not allowed for DVD
DVD audio is 48KHz

Maybe you don't mean "DVD". Something else maybe?
PAL & NTSC DVD spec: https://www.videohelp.com/dvd#tech
BluRay & AVCHD spec: https://www.videohelp.com/hd#tech

[Cornucopia]

When you say "had to send it on a DVD"...what is your ultimate goal? Forget about codecs/containers for a second - who is going to use this and how are they going to play it (or what are they going to be playing it on)? That determines what your final spec needs to be, then you work your way backwards.

Scott

[_Al_]

Why one step? To join those avi's use VirtualDub, load one avi and append rest of them: video/direct stream copy, audio/direct stream copy and File/Save as Avi.

[abolibibelot]

1280x720 = not allowed for DVD
AVCHD = incompatible with DVD
x264/h264 = not allowed for DVD
DVD audio is 48KHz

I know all that. Sorry if that wasn't clear enough : I want to store the re-encoded MP4/MKV file on a DVD, not make a standard DVD video type of DVD. I mean DVD as in "DVD-R", a cheap recording medium that can store up to 4.3 gigabytes of anything digital. (It's a completely different project than my previous thread by the way, and this one is more urgent.)
AVCHD(-Lite) is one the video sources, not the intended final output.

When you say "had to send it on a DVD"...what is your ultimate goal? Forget about codecs/containers for a second - who is going to use this and how are they going to play it (or what are they going to be playing it on)? That determines what your final spec needs to be, then you work your way backwards.

I said I wanted to send it on a DVD because I'll have to send it by mail (as in post mail, “snail” mail, where you put something inside an enveloppe and it gets transported to a remote location by physical means) and the cheapest way to do that, as far as I know, is the good old DVD-R. So I'm pretty sure I have a good understanding of the WHATs involved here, I wrote this post hoping to get some help regarding the HOWs.

[newpball]

Sounds perfect for blu-ray or a USB stick!



http://www.amazon.com/gp/product/B00GSQ4DBM?psc=1&redirect=true&ref_=oh_aui_detailpage_o04_s00

HTL disks for under $1 per disc!

[abolibibelot]

Why one step? To join those avi's use VirtualDub, load one avi and append rest of them: video/direct stream copy, audio/direct stream copy and File/Save as Avi.

One step because : it's already annoying and space-consuming enough to have had to export a 3-hour 1280x720 movie in HuffYUV (total size : 355GB, no typo, it's 355 gigabytes -- actually I was expecting even more, close to 500GB), I'd be glad if I could now avoid the hassle of having to create a single 355GB file before I can start the actual encode (besides, I would have to spend a few more hours just to free enough space on that hard drive, as I already spent a few hours to free about 500GB, and it must be one hell of a stress to create an output file that big on the same hard drive where the input files are located, as I sure wouldn't have that kind of space available on another hard drive).

From what I've read, if I had used Sony Vegas or Adobe Premiere (and some others) I could have directly frame-served the edited movie to an external encoding software, so even that step of exporting in HuffYUV wouldn't have been necessary.

[newpball]

Epic, going through hours of lossless transcoding of HD material to eventually cram more than three hours of video onto a DVD with only an available space of 3GB.

It's like taking days to prepare a Michelin star worthy dish and then putting it in the food processor, adding ketchup and serving it onto a McDonald's tray!

[abolibibelot]

Sounds perfect for blu-ray or a USB stick!



http://www.amazon.com/gp/product/B00GSQ4DBM?psc=1&redirect=true&ref_=oh_aui_detailpage_o04_s00

HTL disks for under $1 per disc!

Well... it still doesn't reply to a single one of my (quite specific it seems to me) questions. I do not have a Blu-Ray Disc burner, only DVD burners. USB stick (or memory card) would be another option indeed, but I'd like to send this to several persons (~5), so it would be too expensive. And USB sticks formatted in standard FAT32 have a 4GB file size limit, so I'd still have to make a 2-pass encode to control the video file size.

(What is a HTL disk ?)

[abolibibelot]

Epic, going to hours of lossless transcoding of HD material to eventually cram it onto a DVD.

It's like preparing a Michelin star worthy dish and then putting it in the food processor, adding ketchup and serving it onto a McDonald's tray!

I guess it's just a bad, bad day... or week...
How else would you have done it, smartass ? Does it make more sense to do 3 generations of lossy encode ? (Actually it takes much less time to encode in a lossless format, only much more space.) Isn't it equally stupid to shoot movies with state-of-the-art technologies and high quality 35mm film (or high bitrate raw digital format nowadays) to have it "served" most of the time on lousy theaters with a poor quality visual and acoustic equipment, surrounded by popcorn-eating, cellphone-talking, notashit-giving sorry excuses for human beings, or as lousy streaming files meant to be viewed on tiny portable devices ? Isn't it equally stupid to upload high quality video content on YouTube, where it's going to be re-encoded at a very low bitrate anyway ? The point of this "stupidity" is to make the best of conditions which are not always ideal, rather than deciding that "since it's going to be crap at some point, might as well make it crap all the way" (because even your "Michelin star worthy dish" will go down as crap at some point, so why bother ?).

[blud7]

You can use ffmpeg with the concat command.

ffmpeg -i concat:vid1.avi|vid2.avi|vid3.avi -b:a 128k -b:v 2M output.mp4

The -b switches indicates the bit rate, which of course would determine your file size.
You can calculate file size based on the total length of the movie and the encoded bit rate.
ffmpeg would automatically use libx264 for video and probably libvo_aacenc for audio.

For two steps, you should be able to join the files using mkvmerge. Then encode as one file.
I'm not sure MeGUI should be your choice here as it would first try to multiplex the 3 files, then demux the audio and index the video.

[johns0]

Other problems aside, you might want to rethink your project.

1280x720 = not allowed for DVD
AVCHD = incompatible with DVD
x264/h264 = not allowed for DVD
DVD audio is 48KHz

Maybe you don't mean "DVD". Something else maybe?
PAL & NTSC DVD spec: https://www.videohelp.com/dvd#tech
BluRay & AVCHD spec: https://www.videohelp.com/hd#tech

All those specs are allowed for dvd as long as it's burned as avchd and if your blu-ray player supports it,it's not allowed for dvd players themselves.Just making sure other people who read this know the difference.

[blud7]

I don't think he means for DVD playback, just as a means of portable storage.

Scratch that, I just realized the intent of your post.

[poisondeathray]

How are the recipients going to play it? I'm assuming you know that it will only be accessible on a computer



To answer your question about appending in avisynth, it pretty simple

A = AVISource("video1.avi")
B = AVISource("video2.avi")
C = AVISource("video3.avi")

A ++ B ++ C



Why "3GB" ? DVD5 Media can hold more even if you factor in the audio. Did you consider DVD9 media? Whatever you choose, use a bitrate calculator to optimize what you are doing for bitrates

There is no consensus on "best AAC" encoder, except the consistenly better ones are Apple AAC (e.g QAAC), FDK-AAC, maybe Nero AAC (but it's lost favor because of a longer delay) . There is, however, consensus on clearly worse ones to avoid

For video, use a preset like --preset veryslow or --preset slower with additional b-frames, . The difference between extremely crazy / extremely slow settings like --preset placebo for 5-10x longer encoding time might actually yield 0.1% lower quality, or 0.1% better quality (i.e it's not worth it) .

[Cornucopia]

You STILL have not answered the simple question of your audience's capabilities and those accompanying target filespecs.

It is insanely simple, using HJSplit or similar app, to span multiple discs, so single disc capacity does NOT have to be your main concern here. Plus, those USB media CAN be formatted as NTFS, so you aren't limited by capacity there, either.

Are they playing on hardware players? Which one(s)? Software? Which apps/systems and codec capabilities? That should determine your target sizes, not temp delivery limitations.

500GB is NOT as huge as you make it out to be. I've currently got 7.2TB of video data stored on-HDD (3/4 of it personal/work stuff), and about to migrate to a 16TB RAID10 system. It is just the nature of the business that video files, and in particular High Quality ones, take up a whole lot of space, so you should be inured to and prepared for that.

Frameserving is a possibility, but your setup has to fully support it. I've yet to see an NLE that works flawlessly with Debugmode or similar FSes. There is something to be said for having intermediates so you can compartmentalize your workflow.

Scott

[Cornucopia]

Yes, as pdr mentioned, partial-length renders that are appended downstream in AVISynth are a good idea for memory-critical NLE apps. I would suggest that you remember to render a separate pass of audio-only at the full length (and then AudioDub() it in AVISynth). This avoids clicks/glitches at the join points. Have done this successfully many times.

Scott

[abolibibelot]

You can use ffmpeg with the concat command.

ffmpeg -i concat:vid1.avi|vid2.avi|vid3.avi -b:a 128k -b:v 2M output.mp4

The -b switches indicates the bit rate, which of course would determine your file size.
You can calculate file size based on the total length of the movie and the encoded bit rate.
ffmpeg would automatically use libx264 for video and probably libvo_aacenc for audio.

Thank you, that's the first concrete/useful reply I get in this thread.
Indeed FFMPEG would be a quite simple option, but as it is it would be a bit "rough", in that I can only control the bitrate ("-b:v 2M" means 2000kbps average bitrate, correct ? from what I calculated that's indeed approximately what I need). And I read that FFMPEG's internal AAC encoder was of poor quality compared to others.

For two steps, you should be able to join the files using mkvmerge. Then encode as one file.

What files would it join ? What I sure would like to avoid is joining first the three huge lossless AVI files into one ginormous 355GB file. If I really have to, I can resort to encoding separately the three AVI files, and then joining them (with MKVMerge or MP4Box), but I'm pretty sure there's a more "elegant" way (like the FFMPEG command above, but with more control over the quality/efficiency).

I'm not sure MeGUI should be your choice here as it would first try to multiplex the 3 files, then demux the audio and index the video.

What do you mean, "multiplex the 3 files", and why would it demux the audio and index the video ? As I asked in the first post, isn't there an Avisynth command that would tell MeGUI to treat those three files as one continuous file ?

[abolibibelot]

How are the recipients going to play it? I'm assuming you know that it will only be accessible on a computer

Another really useful post, thanks.

Well, that's another question, and a very good one. What you say isn't entirely true nowadays : many standalone devices can play such video files. One of the persons I wanna send this to has a LG BD550 Blu-Ray player (it's my brother and I offered it to him), so it would be very nice if the movie in question could be played by that. That thing is a little picky, it reads some MP4 or MKV fine, but "do not want" some others, and I haven't found a clear pattern so far (sometimes just converting MP4 to MKV or the other way around does the trick, sometimes not). My grandmother has a 2010 Thomson TV, possibly equipped with a USB port and support of MP4-type video files, it's a long shot though (about a week ago I asked my mother who lives nearby to tell me the exact model, I'm still waiting, and she's herself as technologically-illiterate as you can imagine) ; she doesn't have a DVD player or anything like that, no computer either (my mother has one but a desktop one, and she couldn't easily bring her own mother to her house) ; otherwise, maybe the retirement home nearby would have a TV room with recent enough equipment, or even computers (she lives alone in an apartment which depends of this retirement home where she takes her meals) ; failing that I'd have to buy her some cheap, simple to use device that could play the damn thing on her TV, if you've got an idea...

Anyway, what parameters should I be careful with to optimize compatibility with such devices ? H.264 profile & level maybe ? Others ? Should I choose MP4 or MKV as container ?

To answer your question about appending in avisynth, it pretty simple

A = AVISource("video1.avi")
B = AVISource("video2.avi")
C = AVISource("video3.avi")

A ++ B ++ C

Fine, that's what I was looking for (I have actually found similar commands in the meantime, it also works with : AVISource("video1.avi") ++ AVISource("video2.avi") ++ AVISource("video3.avi"), without the replacement letters, but I guess your command makes the script more easily readable, especially with long file paths). This indeed opens those three files as a single continuous one in MeGUI or VirtualDub. AviDemux fails though, I get the awfully spelled message : "AVS Proxy has detected an invlaid [sic] color space. Currently only the YV12 (YUV 4:2:0) color space is suppoted [sic]. Please try appending 'ConvertToYV12()' to your script!". I tried to open it with VLC : doesn't work ; Media Player Classic : audio plays at normal rate but video is way too slow, is it normal ? (Doesn't matter, just out of curiosity.)

So, if I load that script in MeGUI and run an encode, it should directly create a single MP4 or MKV file, right ? Can I be confident there won't be any synchronization issues ? (I want to be sure 'cause it's going to be a long process with my 2009 computer based on a Pentium Dual Core E5200 processor...)

Why "3GB" ? DVD5 Media can hold more even if you factor in the audio. Did you consider DVD9 media? Whatever you choose, use a bitrate calculator to optimize what you are doing for bitrates

As I said, I'd like to add some other files, totalling about 1.3GB (hence the 3GB intended file size to store it on a 4.34GB DVD-R). Sure, I could send two DVDs, or indeed use dual-layer DVDs (I have about 10 left), and have more room to encode at a higher bitrate but I'd still have a 4GB size constraint (even in DVD9, or it would require some trick that could compromize the readability on another computer or standalone player). Would you say that the compatibility of dual-layer / DVD9 medium is not an issue nowadays ? (As for the medium quality it would possibly be better : the last Verbatim DVD5 discs I bought seem very unreliable, that or my DVD drives are seriously worn out.)
From what I calculated, to get a 3GB file I should set an average video bitrate of about 2100kbps (2800MB x 1024 x 8 : 10980s, with the audio taking at most 200MB, which amounts to 200MB x 1024 x 8 : 10980s = 149kbps, more than enough for a good quality audio in AAC). It may not be much for 1280x720 video, but it's still better than YouTube quality for instance (verified two 720p files with MediaInfo, one is 1067kbps -- but in 4:3 letterbox --, the other 1686kbps), and as I said it's mostly steady shots of people talking in a room*, which should compress quite easily ; but it's also quite noisy (filmed in interior with a small sensor camera), not a conspicuous kind of noise but I do see some fluctuations in darker areas (screen capture), so perhaps a good denoise filter would significantly improve the encoding efficiency. What Avisynth command could I add to perform a slight noise reduction in order to enhance compressibility ?
How can I tell Avisynth to select only a small portion to run a few quality tests ? And is there a way to know before running a full encode what CRF-equivalent would a given 2-pass average bitrate correspond to, if that makes sense (as I know a CRF of about 25 generally still retains a good quality, beyond that it starts to dwindle), or any way to evaluate the resulting quality ?

(* My grandmother's living room. Last summer I drove 1500 kilometers in a little more than two days to bring my 39 years old brother, who has a mental disability vaguely apparented to “Rain-Man-type” autism, to see our 89 years old grandmother, whom he hadn't seen in 11 years, and of course once we were there she insisted to call our mother, who lives nearby, whom he hadn't seen since the exact same day in 2003, despite him telling his grandmother that he did not want to see his mother for he had decided he would “never see her again after all the wrong she did to me” for reasons such as not letting him watch some TV show when he was 12 or forcing him to do two digit divisions... Then she came, highly emotional, and he started talking to her and showing things on his computer as if nothing. Then I brought him to see the specialized instutution where he spent most of his childhood, the building were closed but he showed me the exact spot where he said goodbye to a girl of whom he has very fond memories, still talking about her more than 20 years later. That's what this movie is about. I began editing it as soon as I got back, but after a few weeks I began to feel overwhelmed by a sense of worthlessness, saturation and disgust, I felt as though I had done all that for almost nothing, I was expecting some huge change but it didn't come and I was exhausted, I couldn't possibly do more than that, and this clumsy video looked kinda ridiculous with that in mind ; I let it lie for months, among other things I equally neglected at that period ; only a few weeks ago did I finally muster the energy to get back to it and finish it, so now it's very important for me to get it over with.)

There is no consensus on "best AAC" encoder, except the consistenly better ones are Apple AAC (e.g QAAC), FDK-AAC, maybe Nero AAC (but it's lost favor because of a longer delay) . There is, however, consensus on clearly worse ones to avoid

From what I read, Faac and the one included in FFMPEG are indeed subpar. And I also read that Nero AAC was no longer developped. So QAAC is the same as Apple AAC ? I already have it in MeGUI so that would be the best choice. The default setting is "91", does it mean 91kbps average bitrate or is it an arbitrary value ? Since you talk about delay, there's a "No Delay" option, should I check it ? FDK-AAC is also supported but has to be installed separately, and I read it required about 200MB which is huge.

For video, use a preset like --preset veryslow or --preset slower with additional b-frames, . The difference between extremely crazy / extremely slow settings like --preset placebo for 5-10x longer encoding time might actually yield 0.1% lower quality, or 0.1% better quality (i.e it's not worth it) .

Would you say there's a similar time/efficiency ratio between "Slow", "Slower" and "Very slow" ? Again, my machine is not very recent and it's a three hours movie, a 10 or 15 hours encode would still be manageable, but beyond that "I would prefer not to".

[poisondeathray]

Well, that's another question, and a very good one. What you say isn't entirely true nowadays : many standalone devices can play such video files...
.
.

Anyway, what parameters should I be careful with to optimize compatibility with such devices ? H.264 profile & level maybe ? Others ? Should I choose MP4 or MKV as container ?

Yes, but that puts you into specific settings territory. You can't use some settings that will be beneficial for compression ratio.

As a general rule, currently most devices are limited to High@L4.1 . You can't use things like 10bit, you can't use huge numbers of b-frames or reference frames. You should use VBV settings for device compatiblity (specific to the device). SO if you knew the exact target, that would be more ideal. MP4 or MKV doesn't matter in terms of those devices unless you have a preference towards Apple devices - then MP4 is a better choice

Fine, that's what I was looking for (I have actually found similar commands in the meantime, it also works with : AVISource("video1.avi") ++ AVISource("video2.avi") ++ AVISource("video3.avi"), without the replacement letters, but I guess your command makes the script more easily readable, especially with long file paths). This indeed opens those three files as a single continuous one in MeGUI or VirtualDub. AviDemux fails though, I get the awfully spelled message : "AVS Proxy has detected an invlaid [sic] color space. Currently only the YV12 (YUV 4:2:0) color space is suppoted [sic]. Please try appending 'ConvertToYV12()' to your script!". I tried to open it with VLC : doesn't work ; Media Player Classic : audio plays at normal rate but video is way too slow, is it normal ? (Doesn't matter, just out of curiosity.)

So, if I load that script in MeGUI and run an encode, it should directly create a single MP4 or MKV file, right ? Can I be confident there won't be any synchronization issues ? (I want to be sure 'cause it's going to be a long process with my 2009 computer based on a Pentium Dual Core E5200 processor...)

The original huffyuv doesn't support YV12, probably it's YUY2 (4:2:2) . Which is a waste since your source files are 4:2:0 , but you probably don't have much a choice in your editor. You're up and downscaling the chroma, which is a lossy operation, and the files are unnecessarly larger. Most devices can only play the common 4:2:0 subsampling of h.264 - so that's what you want in the final product

Probably your computer is too slow to play huffyuv . Or less likely there was a problem with your export from your NLE. You can test it by doing a short test by encoding a segment, even resized smaller like 640x360 . For example, add Trim(0,999).Spline36Resize(640,360) to encode 1000 frames for a quick test . Or choose some other frames in the middle . If it's not in sync, then you need to provide additional information and more debugging

and as I said it's mostly steady shots of people talking in a room*, which should compress quite easily ; but it's also quite noisy (filmed in interior with a small sensor camera), not a conspicuous kind of noise but I do see some fluctuations in darker areas (screen capture), so perhaps a good denoise filter would significantly improve the encoding efficiency. What Avisynth command could I add to perform a slight noise reduction in order to enhance compressibility ?

Yes, denoising will help with compression efficiency. It depends on the noise characteristics. A screenshot doesn't show the temporal pattern of noise (fluctuations between frames) . A screenshot only shows spatial patterns. If only certain segments are noisy, you might want to apply the filters only to those segments (not only will it encode faster, it won't "degrade" the "good" segments). If you provide a video sample, people here will provide some suggestions. Quality denoising will significantly slow your process down

So QAAC is the same as Apple AAC ? I already have it in MeGUI so that would be the best choice. The default setting is "91", does it mean 91kbps average bitrate or is it an arbitrary value ? Since you talk about delay, there's a "No Delay" option, should I check it ? FDK-AAC is also supported but has to be installed separately, and I read it required about 200MB which is huge.

Yes, QAAC is the AppleAAC implementation. Q of 91 is the quality factor . The scale is 0-127 , higher yields larger filesizes, better quality. You probably don't want to use that mode, and want to use a set bitrate in your case, because you have fixed capacity limitations. I typically use no delay with QAAC, but the actual delay without it is not that much anyways , maybe 20ms . Nero's is closer to 40ms on average

Would you say there's a similar time/efficiency ratio between "Slow", "Slower" and "Very slow" ? Again, my machine is not very recent and it's a three hours movie, a 10 or 15 hours encode would still be manageable, but beyond that "I would prefer not to".

I would say there are diminishing returns. But in a lower bitrate scenario, small % efficiency gains are proportionately larger. It's going to take longer than 10-15 hours for sure on your computer if you throw in additional processing such as denoising. Video encoding is all about trade offs and the things you are willing to sacrifice

[Stears555]

You STILL have not answered the simple question of your audience's capabilities and those accompanying target filespecs.

It is insanely simple, using HJSplit or similar app, to span multiple discs, so single disc capacity does NOT have to be your main concern here. Plus, those USB media CAN be formatted as NTFS, so you aren't limited by capacity there, either.

Are they playing on hardware players? Which one(s)? Software? Which apps/systems and codec capabilities? That should determine your target sizes, not temp delivery limitations.

500GB is NOT as huge as you make it out to be. I've currently got 7.2TB of video data stored on-HDD (3/4 of it personal/work stuff), and about to migrate to a 16TB RAID10 system. It is just the nature of the business that video files, and in particular High Quality ones, take up a whole lot of space, so you should be inured to and prepared for that.

Frameserving is a possibility, but your setup has to fully support it. I've yet to see an NLE that works flawlessly with Debugmode or similar FSes. There is something to be said for having intermediates so you can compartmentalize your workflow.

Scott

After cutting and recoloring save it with lossless UT video (with the highest compression settings) in a traditional YV12 color space. IT can fit your HDD. (Even bluray films use the YV12 colors!!!)

[abolibibelot]

You STILL have not answered the simple question of your audience's capabilities and those accompanying target filespecs.

I think I gave all the relevant details in my reply to “poisondeathray” above. So : computer, and if possible LG Blu-Ray player & 2010 Thomson TV. Here is a thread about playability issues with that player -- apparently AVC profile "High@4.1" should work, not a higher level. But my brother complained about some files not playing which have a lower level than that.

It is insanely simple, using HJSplit or similar app, to span multiple discs, so single disc capacity does NOT have to be your main concern here. Plus, those USB media CAN be formatted as NTFS, so you aren't limited by capacity there, either.
Are they playing on hardware players? Which one(s)? Software? Which apps/systems and codec capabilities? That should determine your target sizes, not temp delivery limitations.

The audience as you say are mostly technologically-illiterate persons, so I'd like to keep it as simple as possible. And the spanning-over-multiple-discs solution would work for computer playback (I could even reconstitute the whole file myself while remote-controlling the computer with TeamViewer), not on standalone devices like the aforementioned BR player.
USB memory can be formatted in NTFS : I guess it can, just like you can store files bigger than 4GB on a DVD medium, but it's not standard, and could make it unreadable by a standalone device (the BR player supports a NTFS hard drive though so maybe it would work).

500GB is NOT as huge as you make it out to be. I've currently got 7.2TB of video data stored on-HDD (3/4 of it personal/work stuff), and about to migrate to a 16TB RAID10 system. It is just the nature of the business that video files, and in particular High Quality ones, take up a whole lot of space, so you should be inured to and prepared for that.

To me it's quite huge, as it's the first time I had to produce such big files (before that I had always directly encoded my edited movies -- filmed lectures and associative events mostly -- in MP4 with Magix Video Deluxe and its internal encoder using default constant quantizer settings, or in MPEG2 at a chosen bitrate so I would get ~5GB files). Of course if you're a seasoned video enthusiast or even a professional it must be quite usual, I get that. The point was (in my reply to “_Al_” who advised me to append the files first with VirtualDub), I had a hard time freeing that amount of space on one hard drive, and I'd have an even harder time freeing the extra space that would be required to losslessly append those files before encoding, plus as I said it must be quite a stressful operation for a hard drive to copy hundreds of GBs over itself.

Frameserving is a possibility, but your setup has to fully support it. I've yet to see an NLE that works flawlessly with Debugmode or similar FSes.

Right, that's what I investigated first, and after reading a few relevant discussions I came to the conclusion that it wasn't an option in my case. (And from what you say it wouldn't have worked either with another NLE software -- well, that's a consolation. I still want to try Sony Vegas, as it seems quite praised in the “enthusiast” range.)

There is something to be said for having intermediates so you can compartmentalize your workflow.

Could you elaborate on that a bit ?


Gabriel

[Stears555]

Why did you chose the backward huffyuv format? It had no bt709 mode, so called HD colors

[abolibibelot]

Yes, but that puts you into specific settings territory. You can't use some settings that will be beneficial for compression ratio. As a general rule, currently most devices are limited to High@L4.1 . You can't use things like 10bit, you can't use huge numbers of b-frames or reference frames. You should use VBV settings for device compatiblity (specific to the device). SO if you knew the exact target, that would be more ideal. MP4 or MKV doesn't matter in terms of those devices unless you have a preference towards Apple devices - then MP4 is a better choice.

Again, this BR player would be a specific target, but just looking at the specs sheet won't help, as there were some firmware updates, and in particular, when I bought it (used, 20 euros !), the seller told me that with the most recent firmware, which was upgraded automatically from what he said, it could no longer play MKV files ; I managed to downgrade to an older firmware which restored MKV readability, and apparently it's a discontinued model with no newer firmware updates, so I don't want to tamper with it anymore, I think it has impressive playing ability as it is for what it cost me.

The original huffyuv doesn't support YV12, probably it's YUY2 (4:2:2) . Which is a waste since your source files are 4:2:0 , but you probably don't have much a choice in your editor. You're up and downscaling the chroma, which is a lossy operation, and the files are unnecessarly larger. Most devices can only play the common 4:2:0 subsampling of h.264 - so that's what you want in the final product

The HuffYUV configuration dialog has this option which I did not check : “Convert RGB to YUY2 when compressing. If this option is checked, Huffyuv will convert RGB to YUY2 before compressing. This will improve compression significantly, but the compression will no longer be lossless. The loss is much smaller than MJPEG on even the highest quality setting.” I'm not familiar enough with the notions of color spaces to understand exactly what's going on here, but I guess those resulting files are in RGB (MediaInfo doesn't specify that characteristic, neither do VirtualDub or Avidemux). Is it good or bad ? In what color space should be the final encoded file ? (The x264vfw configuration window in VirtualDub proposes several options, keep original or convert to XXX, I haven't found the same in MeGUI's x264 config.)

Probably your computer is too slow to play huffyuv . Or less likely there was a problem with your export from your NLE. You can test it by doing a short test by encoding a segment, even resized smaller like 640x360 . For example, add Trim(0,999).Spline36Resize(640,360) to encode 1000 frames for a quick test . Or choose some other frames in the middle . If it's not in sync, then you need to provide additional information and more debugging

The individual files play fine in VLC Media Player and VirtualDub but lag in Avidemux 2.5.2/2.5.6 and play at normal speed but with no sound in 2.6.0. The AVS script plays fine in VirtualDub and is correctly identified with a duration of 3:03:00:00 (I went out of my way to have the duration rounded that way, my “OCD” side ! in fact the actual movie is exactly 3:00:00:00, then there's about 1min of black screen and a small “bonus” sequence). In MeGUI it plays slowly with no sound (may be normal), and the duration is displayed as 3:02:59:960 (one frame shy, may be normal).
Thanks for the “Trim” command, I'll make a few tests.
Is there a way to use QAAC or Nero AAC within VirtualDub ? That would be nice as otherwise it would do all I need and I'm more familiar with it than with MeGUI (which seems tremendously versatile but also convoluted and not “user-friendly”). But I just remembered VirtualDub only allows export in AVI, so it won't work...

Yes, denoising will help with compression efficiency. It depends on the noise characteristics. A screenshot doesn't show the temporal pattern of noise (fluctuations between frames) . A screenshot only shows spatial patterns. If only certain segments are noisy, you might want to apply the filters only to those segments (not only will it encode faster, it won't "degrade" the "good" segments). If you provide a video sample, people here will provide some suggestions. Quality denoising will significantly slow your process down

*Sigh* I guess everything good comes at a cost in the video world, as it does anywhere. Significantly by how much would you say, roughly, in % ? (Not so long ago I thought the actual encoding process was always the most demanding part of video conversion, then I tried high quality deinterlacing with QTGMC, and now learn that special filtering like denoising is also a chore.)
Of course a screenshot is only a vague indication. Here is a short sample, encoded with Avidemux using x264 CRF18 to keep the native grain/noise (my brother is showing our grandmother one of his favorite music band he saw on a magazine, and one particular singer whose eyes give him goosebumps, then talking about Mike Tyson whom he dreams about defeating since he saw him bite Evander Holyfield's ear in 1996, while the grandmother is trying to cook some rice... I added pictures and a funny montage based on the NES game "Punch out") :
https://www.mediafire.com/?2hljmwp7yyo295h (22.6MB)
I first tried with VirtualDub but had the following warning :

"Few frames probably would be lost. Ways to fix this:
- if you use VirtualDub or its fork than you can enable 'VirtualDub Hack' option
- you can enable 'File' output mode
- you can enable 'Zero Latency' option"

and the resulting AVI files would play poorly (with Vorbis : audio not recognized in VLC and completely scrambled in MPC, with MP3 : out of sync in VLC, fine sound but scrambled picture in MPC -- and the files are four times larger than the MP4 from Avidemux).

This is the same sample encoded in 2-pass mode with a target file size of 10MB (half that of the CRF18 one, resulting average bitrate of 1624kbps) :
https://www.mediafire.com/?l31r8d5j59lglxd (10.8MB)
The quality seems still very good to me. The only defect I notice is the slightly less fluid picture fade-in (if that's the correct term) at 00:05.

Yes, QAAC is the AppleAAC implementation. Q of 91 is the quality factor . The scale is 0-127 , higher yields larger filesizes, better quality. You probably don't want to use that mode, and want to use a set bitrate in your case, because you have fixed capacity limitations. I typically use no delay with QAAC, but the actual delay without it is not that much anyways , maybe 20ms . Nero's is closer to 40ms on average

Even with size constraints, I guess the average bitrate won't fall outside a relatively predictable range, and the variability doesn't affect the final size as much as for the video bitrate. But yes, if I set a bitrate for video it makes sense to do the same for audio. Does the 2-pass mode also optimize the distribution of bitrate for audio, though ?
What exactly is the delay in that sense and context ?

I would say there are diminishing returns. But in a lower bitrate scenario, small % efficiency gains are proportionately larger. It's going to take longer than 10-15 hours for sure on your computer if you throw in additional processing such as denoising. Video encoding is all about trade offs and the things you are willing to sacrifice

Well, let's say I'm willing to accept a 10-20% increase in encoding-time if the quality/efficiency gain is really worth it, not beyond. But still, I'd like to test a good Avisynth denoising filter, even if I don't end up using it for this task it might come in handy for another.

Thank you for your quick and thorough replies, I was kinda desperate after the first batch of feedback !

[abolibibelot]

Why did you chose the backward huffyuv format? It had no bt709 mode, so called HD colors

Because it was already installed and available within the NLE software, and because I didn't know better. Is HuffYUV really obsolete, and would another codec have made a difference ? Isn't lossless supposed to do just that, i.e. preserve the exact same quality as the original ? What is bt709 / HD colors, and does it matter considering the capture devices used for this ? (Mostly a Panasonic ZS3 compact digital camera filming in 1280x720 AVCHD-Lite, and a Canon SX20 bridge camera for a few sequences shot in 640x480 MOV.)
I can still do the export again with another codec if there is a good reason (it took “only” about three hours in HuffYUV), but I doubt there is in this case. Still, what is currently the most recommanded lossless video codec ? (I've been planning on converting my VHS tapes for a long time, it might be useful when I finally begin this intimidating task.)

[abolibibelot]

Yes, as pdr mentioned, partial-length renders that are appended downstream in AVISynth are a good idea for memory-critical NLE apps. I would suggest that you remember to render a separate pass of audio-only at the full length (and then AudioDub() it in AVISynth). This avoids clicks/glitches at the join points. Have done this successfully many times.

I learned that the hard way... (Are there other similar NLE softwares which behave better in that regard, with a more robust handling of memory for instance ? Another thing I don't like with MVD is that once a rendering process is started there is no way to pause or interrupt it, other than shutting down the process in the task manager.)

Thanks for this tip regarding the audio, but would it be useful in my case, considering I carefully selected the junctions at fade-to-black picture, fade-to-silence audio points ?

[Cornucopia]

@Stears555, probably because that's what he already had on his computer. I fully understand that. Going forward it does make sense to use something more compatible with newer systems and multiple platforms & architectures as well as using multiple colorspaces, so UT video or MagicYUV seem good choices.

@abolibibelot, the AVS script

A = AVISource("video1.avi")
B = AVISource("video2.avi")
C = AVISource("video3.avi")
Aud = WAVSource ("audio.wav")
Vid = A ++ B ++ C 
Mix = Audiodub (Vid, Aud)
Ready = ConvertToYV12(Mix)
Return(Ready)

...should get you a proper YV12 combined source with which to encode with MeGUI. I would use a standard hardware-based profile using MP4 container, AVC video & AAC audio. Start with ~CRF 18-22.

Suggestion: cut a clip of ~20 sec from your AVISynth script (using an extra line of TRIM() at the end, and save as a 2nd AVS) and do the encode (saving your MeGUI settings to a custom preset) and send those to the interested parties to have them test the file. If it doesn't work, you haven't wasted a long time. If it does work, you just do the full length encode.

3hours at avg ~2Mbps would be 2.7GB (not uncommon for 720p highly compressed encodes). Too big to email, but you could put it up on dropbox etc., or you could mail a SL data DVD, or a small, standard FAT-formatted USB.

Scott

[Cornucopia]

Regarding the audio, it's your call, but if it were me, yes I'd do it. Glitches in silence will be very low, but will still be glitches unless you are extremely lucky to be exactly at digital 0 for a good length of time (on both ends). Those clicks are of unusual character so are quite noticeable even at low levels.

Scott

[poisondeathray]

Again, this BR player would be a specific target, but just looking at the specs sheet won't help, as there were some firmware updates, and in particular, when I bought it (used, 20 euros !), the seller told me that with the most recent firmware, which was upgraded automatically from what he said, it could no longer play MKV files ; I managed to downgrade to an older firmware which restored MKV readability, and apparently it's a discontinued model with no newer firmware updates, so I don't want to tamper with it anymore, I think it has impressive playing ability as it is for what it cost me.

Yes, specific firmware versions can reduce compatibility. Some early Sony players actually won't play MKV off optical disc, but only from USB . So as Cornucopia suggested - you should run some short tests before you waste time jumping though hoops only to have it not work.

The HuffYUV configuration dialog has this option which I did not check : “Convert RGB to YUY2 when compressing. If this option is checked, Huffyuv will convert RGB to YUY2 before compressing. This will improve compression significantly, but the compression will no longer be lossless. The loss is much smaller than MJPEG on even the highest quality setting.” I'm not familiar enough with the notions of color spaces to understand exactly what's going on here, but I guess those resulting files are in RGB (MediaInfo doesn't specify that characteristic, neither do VirtualDub or Avidemux). Is it good or bad ? In what color space should be the final encoded file ? (The x264vfw configuration window in VirtualDub proposes several options, keep original or convert to XXX, I haven't found the same in MeGUI's x264 config.)

First check what type of huffyuv you have, and what AVISource is returning. Use info() and preview it and it will provide the information on colorspace e..g it will say RGB32 or YUY2 or something like that

e.g
AVISource("video1.avi")
Info()

Eitherway, as mentioned eariler, the FINAL file has to be 4:2:0 or YV12 to be playable on the BD player

Is there a way to use QAAC or Nero AAC within VirtualDub ? That would be nice as otherwise it would do all I need and I'm more familiar with it than with MeGUI (which seems tremendously versatile but also convoluted and not “user-friendly”). But I just remembered VirtualDub only allows export in AVI, so it won't work...

Yes there is, newer versions of vdub have an "external encoder feature" now. Basically you can frameserve out of vdub to other binaries like x264, or ffmpeg. So you can use other containers, like MKV, MP4. It's a bit difficult to set up for beginners at first, but there are guides and profiles you can download. Even though megui is a bit complicated too, Megui would actually be easier IMO for most people. There are easier GUI's to use, such as ripbot, simple x264 launcher etc..

Significantly by how much would you say, roughly, in % ?

It depends on the filters used, hardware situation etc... For your video, it's not that noisy. I would only use a mild temporal denoiser . Something like SMDegrain with a setting of 1, or MCTemporalDenoise(settings="low"). Other people might provide better suggestions. I don't have access to a dual core, but it will definitely slow it down a bit , maybe 30-50% it's a wild guess. Do some short tests. Heavy denoising might slow it down 300-400% easily. Just like crazy slow encoding settings might slow it down 10x, or really fast encoding settings might speed it up 5x

The quality/speed tradeoff is something you're going to have to decide for yourself. Run some short tests with varied settings. You're not going to be able to make the time deadline even with a "fast" filter on anything slower --preset veryslow on a dual core . You probably could afford to use default "medium" or "slow" settings with increased b-frames. Your average person won't be able to see the difference in motion. On multipass encodes, many people use a lossless intermediate to apply the filters first before the final encode - especially if slow bottlenecking filters are used. The reason is it's actually faster overall because you encouter the slow filters once instead of two or more times.

Does the 2-pass mode also optimize the distribution of bitrate for audio, though ?

It does, but it's not as important factor as for video. Video makes better use of temporal redundancies , so it's more important in video context

What exactly is the delay in that sense and context ?

When you have time, you can read about it, it's discussed in other threads in detail. But the short explanation is lossy audio always has a delay. It varies by type of audio (e.g. LC- AAC will have a different delay than HE-AAC, or AC3), and by encoder. Audio delay means it can go perceptibly out of sync - eg. person opens mouth to talk, but the audio comes slightly later. A large delay can compound (additively worsen) when video is re-encoded. This occurs frequently on Youtube for example. Threshold for humans is about ~40ms on average. Grandma might be 80ms. Some musicans might be able to perceive a +/- 20ms delay.


RE: Rec709/601 - first post the huffyuv info() requested above
In the end, it won't matter that much, and it's "fixable" in your script. The difference between Rec709 and Rec601 is a slight shift in colors. Most noticable on reds such as skin tones and greens such as grass.

[abolibibelot]

@Stears555, probably because that's what he already had on his computer. I fully understand that. Going forward it does make sense to use something more compatible with newer systems and multiple platforms & architectures as well as using multiple colorspaces, so UT video or MagicYUV seem good choices.

OK, noted. Is the compression ratio comparable with those ?

@abolibibelot, the AVS script

A = AVISource("video1.avi")
B = AVISource("video2.avi")
C = AVISource("video3.avi")
Aud = WAVSource ("audio.wav")
Vid = A ++ B ++ C 
Mix = Audiodub (Vid, Aud)
Ready = ConvertToYV12(Mix)
Return(Ready)

...should get you a proper YV12 combined source with which to encode with MeGUI. I would use a standard hardware-based profile using MP4 container, AVC video & AAC audio. Start with ~CRF 18-22.

Thanks for the improved script. Does it matter to convert the color space before encoding ? Does the "Aud" line automatically extract a full-length WAV file as you wrote earlier ? Does the "Audiodub" work flawlessly in your experience, with no sync issues ? What do "Ready", "Return", "Mix" mean exactly ?
Should I choose "Blu-Ray" as hardware-based profile ?
CRF mode won't produce a predictable file size, or I would have to make many tests to get the correct value, which defeats the purpose of saving time with a single pass encode.

Suggestion: cut a clip of ~20 sec from your AVISynth script (using an extra line of TRIM() at the end, and save as a 2nd AVS) and do the encode (saving your MeGUI settings to a custom preset) and send those to the interested parties to have them test the file. If it doesn't work, you haven't wasted a long time. If it does work, you just do the full length encode.

Very good idea indeed, I can send my brother a sample and ask him to test it on his external hard drive connected to the LG player. I can't do the same with my mother and that Thomson TV, though (she barely knows what a USB port looks like, as for my grandmother she was fascinated by my compact camera which films "in colors" !).

3hours at avg ~2Mbps would be 2.7GB (not uncommon for 720p highly compressed encodes). Too big to email, but you could put it up on dropbox etc., or you could mail a SL data DVD, or a small, standard FAT-formatted USB.

I've seen far worse regarding the bitrate (actually one of the extra files I plan on adding on the DVD is the movie “Rain Man”, YIFY encode, which is a meager 621MB for 2h13 in 1280x720, that is 613kbps average bitrate ! and, well, it's definitely watchable... at least way better looking than another version I had “found” earlier in 720x404 RV9 for 701MB...).
Again, sending it through the internet would work with my brother (I regularly have TeamViewer sessions with him, so I can do the technical stuff myself), but I have a slow uploading capacity and he has a relatively slow download capacity (even more so within a TeamViewer session), it would take me about 10 hours to upload, and 2 hours to download it from his computer. Couldn't do the same with the other intended recipients. So physical means of transfer seem preferable in this case. USB sticks would be way more expensive (5 euros at best for a 4GB one when a SL DVD is 0.25 € and a DL DVD about 1€). I might use DL DVDs after all, but there would still be a 4GB limit, hence my settling on a 2-pass encode in any case (not sure this is grammatically correct).

[abolibibelot]

First check what type of huffyuv you have, and what AVISource is returning. Use info() and preview it and it will provide the information on colorspace e..g it will say RGB32 or YUY2 or something like that
e.g
AVISource("video1.avi")
Info()
Eitherway, as mentioned eariler, the FINAL file has to be 4:2:0 or YV12 to be playable on the BD player

It says (opened with MPC) :
ColorSpace: RGB32
Width: 1280 pixels, Height: 720 pixels
Frames per second: 25.0000 (25/1)
FieldBased (Separated) Video: No
Parity: Bottom Field First
Video Pitch: 5120 bytes
Has Audio: YES
Audio Channels: 2
Sample Type: Integer 16 bit
Samples per second: 48000
Audio length: 527040000 samples. 03:03:00.000
CPU detected: x87 MMX ISSE SSE3 SSSE3

Yes there is, newer versions of vdub have an "external encoder feature" now. Basically you can frameserve out of vdub to other binaries like x264, or ffmpeg. So you can use other containers, like MKV, MP4. It's a bit difficult to set up for beginners at first, but there are guides and profiles you can download. Even though megui is a bit complicated too, Megui would actually be easier IMO for most people. There are easier GUI's to use, such as ripbot, simple x264 launcher etc..

But they won't be as complete and flexible I guess, with Avisynth and the like ? (I never tried those two tools.)

It depends on the filters used, hardware situation etc... For your video, it's not that noisy. I would only use a mild temporal denoiser . Something like SMDegrain with a setting of 1, or MCTemporalDenoise(settings="low"). Other people might provide better suggestions. I don't have access to a dual core, but it will definitely slow it down a bit , maybe 30-50% it's a wild guess. Do some short tests. Heavy denoising might slow it down 300-400% easily. Just like crazy slow encoding settings might slow it down 10x, or really fast encoding settings might speed it up 5x

No, it's not very noisy (there may be sequences where it's a bit more), but probably still enough to explain the quite high bitrate for the kind of static video it is. Or would you say that 3552kbps average is on the low to normal side of the spectrum for such a sample encoded at CRF=18 ? (According to Bitrate Viewer it goes from 1164kbps minimum to 6194kbps maximum.)

The quality/speed tradeoff is something you're going to have to decide for yourself. Run some short tests with varied settings. You're not going to be able to make the time deadline even with a "fast" filter on anything slower --preset veryslow on a dual core . You probably could afford to use default "medium" or "slow" settings with increased b-frames. Your average person won't be able to see the difference in motion.

By deadline you mean the "10-15 hours" I wrote above ? I don't have an actual deadline, it was just a rough estimate, if it has to be 20, so be it (it's still annoying as I sleep in the same room, well, I'll use earplugs...), but if as you just wrote above a given filter or setting slows down the process by 300-400% for a 5% improvement in visual quality or compression efficiency, then it's not worth it (maybe it would be in a professional context, but then I'd soon buy new hardware !).
As for "increased b-frames", you said earlier that to optimize compatibility with standalone devices I should avoid that. What are their specificities, in a nutshell ?

On multipass encodes, many people use a lossless intermediate to apply the filters first before the final encode - especially if slow bottlenecking filters are used. The reason is it's actually faster overall because you encouter the slow filters once instead of two or more times.

Why is that ?

RE: Rec709/601 - first post the huffyuv info() requested above
In the end, it won't matter that much, and it's "fixable" in your script. The difference between Rec709 and Rec601 is a slight shift in colors. Most noticable on reds such as skin tones and greens such as grass.

See above, but I haven't seen that kind of information (neither RecXXX nor 4:X:X). Perhaps "RGB32" is enough for you to know exactly what it is and it not ?
Edit : At first I tested an export in uncompressed RGB (damn that was huge ! it would have been close to 1TB for the whole thing), and I definitely noticed a shift in colors, then tested in HuffYUV and compared to the original source files, there was no shift at all. So that's the explanation, the uncompressed RGB was in the wrong color mode ?