Current recommended workflow for analog capture to H.264 'archive'?

[Killer3737]

As per the thread title, I'm looking for opinions and experiences in capturing and archiving analog sources in the present day.

I know the basics, I've been capturing and encoding analog stuff for a fair few years, but the output has always been to DVD format as a lossy 'archive' format (Yes I know anything lossy can't really be considered archive, but considering the sources I don't think keeping lossless digital copies makes much sense). However as the years pass by, DVD is seeming less and less relevant. MPEG-2 is not a particularly efficient codec in comparison to H.264 and you're restricted on many levels by the DVD spec, if you want a DVD legal stream. Plus the idea of burning DVDs altogether, I rarely do it any more with media players and the like, it seems a waste.

So, going forward I was thinking of instead encoding video to H.264 files, with no specific format, just something playable on computers and most modern hardware, USB players, built in HDTV software etc.

These are my specific questions that I'd like some thoughts on:

- Capture hardware in terms of VCR/TBC/cables etc I'm good on, but what about the A>D converter? Does it matter too much?

- Capture format. Bearing in mind the final target, what makes the most sense? Is DV-AVI still ok? What about lossless codecs like HuffYUV, or I've even read about capturing directly to H.264 lossless with x264vfw, and then compressing again with x264 to reasonable levels. What are the pros and cons of each? Which makes the most sense?

- Capture resolution/format. Since I won't be aiming for a DVD, should I still capture at 720x480/576 interlaced?. What about colour? I would probably be cropping the results to remove the garbage, so would end up with a random final res anyway. Or would it be a good idea to capture and encode Blu-Ray legal files, just in case I should ever want to put them to disc, or anyone else would?

- x264 encoding. I know this is a minefield of options and choices but as a general base, without getting into source specific options, what would you go with? At the moment I've been experimenting with some DV-AVI files I already have on my PC, which are of course interlaced BFF. I've been using QTGMC to deinterlace, applying a crop to remove the garbage and encoding the resulting file as 50p. Is that a good idea? I don't see any reason to really keep the, but is that the correct way about it? Is a 50fps, progressive file the 'correct' output from a PAL VHS, for example?

- Last question is really going back, but how do things differ with PC and Mac? I'm not really familiar with Mac, I've always worked on PC, but I may be capturing on a Mac in future. How does the workflow differ there? Would the recommended hardware capture change, or would DV-AVI from an ADVC-110 or whatever be ok? Obviously if so from there it'd be no different, encode that with x264.

- The above was mostly talking about VHS, would anything change with other formats? SVHS, Hi8, Video8 etc?

Thanks in advance.

[pippas]

I use a Canopus ADVC50 to capture as DV, then Virtualdub to de-interlace, crop, adjust colour , brightness, etc and then re-size (if necessary) so that the edited file has square pixels (saves worrying about whetheer AR flags are recognised or not).
Then export from Virtualdub using the vfw x.264 codec - usually with the default CRF settings.
The resulting avi is not very compatible with most external players, and still has a PCM audio which takes up a lot of space. So I save the audio from virtualdub as a .wav, and convert it to AAC using the freebie fre:ac converter utility.

Finally, I demux the video with MP4Box , and remux the .h264 video file with the new AAC audio.

I find the resulting mp4 file plays on all my devices (WDLive - Panasonic PVR - and TV USB slots).

Same sort of quality as mpeg2, but about half the size (or less!)

[Brad]

I don't think most USB media players like random vertical resolutions any more than Blu-ray does. Resolutions that aren't divisible by 16 are also argued to be worse for compression efficiency. Just mask the head-switch noise with black (Letterbox in Avisynth for example).

Encoding to the actual Blu-ray specs does limit how well you can compress the files, though.

Stick to standard 720x___ capture resolutions, but for VHS/Video8 you can lower the horizontal resolution when you encode if you do decide to violate BD spec.

If these are "archives", either don't deinterlace, or stick to a lossless/reversible bob deinterlace (not QTGMC's default mode). That's so that you're not stuck with today's deinterlacing capabilities in 15 years.

The pros and cons of lossless capture vs DV are argued all over the place on this forum. No need to do it again in this thread.

[newpball]

Archiving means capturing VHS interlaced and (near) lossless!

Anything else is just foolish because one day those VHS tapes won't work anymore and those who did not do it come here and ask on how to 'restore' their bad and lossy captures.

Sorry but I think there are no excuses, hard disk space is dirt cheap!

Just my opinion, other members on this forum will certainly differ with me on it and feel the 'pain' of having 'those frighteningly large files' on common terabyte drives!

[_Al_]

If these are "archives", either don't deinterlace, or stick to a lossless/reversible bob deinterlace (not QTGMC's default mode). That's so that you're not stuck with today's deinterlacing capabilities in 15 years.

I'm not sure what that means, QTGMC does bob-deinterlace by default.

My take on bob-deinterlace by QTGMC, sure it is a procedure when video is being reconstructed basically, original is lost. But it does really well. But if you encode interlace , very low CRF, one is still closer to the original and you can actually can see the quallity providing screening device deinterlacer is not a garbage.

Or for example you load that interlaced "archived" footage into NLE and it deinterlaces badly, resizes it badly, or mixes that video with new , true 50p video, not doing good job. Just to be concrete, load interlace SD footage into Sony Vegas , into progressive 50p project, export 50p, you might see things are not optimal, then to make some sort of high quallity CRF 16-17 intermediate mp4, (I know does not sound right), perhaps even with GOP 50 frames max. might be handy in 10 year future. To have archived it as interlaced (possibly in avi) - quallity is kept definitely better but some action is needed to prepare it for actual use then, and in 10-15 years, someone will grab it and look at it, what it is? Etc., etc., so there is no correct answer what is best, one has to make a decision , neither is correct for a certain scenario .... to keep it interlaced is better, but question is if someone will play with it later at all ...

[Brad]

If these are "archives", either don't deinterlace, or stick to a lossless/reversible bob deinterlace (not QTGMC's default mode). That's so that you're not stuck with today's deinterlacing capabilities in 15 years.

I'm not sure what that means, QTGMC does bob-deinterlace by default.

I have highlighted the relevant text. QTGMC alters the original fields by default, to reduce shimmer and such, but this can be disabled. A lossless bob keeps the original fields intact for future use, but is perhaps worse quality for here-and-now viewing.

[newpball]

A lossless bob keeps the original fields intact for future use

Correct!

[_Al_]

OK, but I'd rather keep regular captured interlaced video if interlaced would be a keeper,

again, just pointing out if that interlaced archived footage would be used later directly in some videoeditor loading it into some High Definition 50p project, then that original intelaced lossless footage loses its purpose, just laying it out here, NLE will resize badly, not sure if that is clear for everybody, op etc., that upon whatever new HD workflow QTGMC bob-deinterlace (or any better if there will be) should be done beforehand and resize or crop later in that videoeditor because resizing of progressive video is executed well in there ......

cropping should definitively not be done even for that interlaced archive video if archived as interlaced

[Killer3737]

Thanks for the replies so far. I was reluctant to use the archive word because I didn't mean it in the true sense of the word. Of course I know the only true archive would be either raw or lossless compressed, what I should have more specifically stated is a 'watching archive' for the here and now. Lossless AVI files are obviously still not practical to watch or share, so I was looking for thoughts on a 2015 archive of files for viewing, hopefully good for the next few years.

But for the 'true' archive copy, is HuffUV or Lagarith a good idea, bearing in mind support for those codecs may be non existent in 20-30 years? Or is the 'safest' way totally raw RGB, bearing in mind that means much more space required obviously?

Moving on, with a better explanation of what I'm looking for in terms of viewing today, does the advice change?

I'm wondering why you state not to crop? It's always tricky asking for video advice because a lot is opinion, there is not always a 'best', but I was told on another forum that cropping was better for efficient encoding, so in theory a random size cropped video with no black/garage would compress better than a 720x576 version with that. Is that true? Also a similar thought on re-sizing, I was again told it was better to keep it anamorphic and apply the correct SAR, but would a say 720x540 version really look/compress worse?

As for deinterlacing, some say no, some say yes (In regards to the compressed encode, obviously not on the lossless). I did test encode something with MBAFF in x264 and I wasn't too keen on the results, at least when played via USB on my TV. Motion didn't seem smooth, and I noticed some combing. When I deinterlaced with QTGMC to 50p it looked better to me, smooth and no combing artefacts.

I suppose these latter questions are better suited to the encoding forum actually. Getting back on the capturing side of things, are people in agreement then the 'best' practice is still to capture 720x480/576, interlaced? What about colour and chroma?

Also what about Mac? Is it easy to capture HuffYUV or Lagarith there?

[2Bdecided]

I should have more specifically stated is a 'watching archive' for the here and now.

What do you want to watch it on?

e.g.
On a PC, use some efficient square pixel bob-deinterlaced format like 768x576p50 mp4 is good.
For YouTube, upscale SD bob-deinterlaced content to 960x720p50 to get half-decent results.
Optical disc formats and players handle 720x576i25 MPEG-2 (and for BluRay, MPEG-4/AVC) well.

Years later, you might look back and think "Oh, I could have done the denoising/deinterlacing/whatever better", but you're more likely to think "glad I copied that stuff into a format that still works - shame I didn't edit out all the boring bits".

Cheers,
David.

[Killer3737]

Playback will be either on a PC or HDTV via in-built software, USB media player etc . Basically the target is a digital file that can be played as is in as much hardware as possible today. Not YT and not really Blu-Ray.

[_Al_]

How much of stuff do you have for capture? Couple of tapes you can go with lossless. If it is a lot of capture, then capture into DVavi, and that could be your archive. Then encode to square pixel to 50p with QTGMC, squeeze the size as much as you like for playback.

You seem to know what it takes to have true lossless archive, (capture interlace though, original is interlaced as well) , but you want someone else to tell you, encode it, it is alright. I'm not sure you'd get that answer.

About that cropping. 720x576 is anamorphic video, as soon you start to crop, you have to change resolution as well to keep correct aspect ratio and that for sure is not archived stuff anymore, that is why I said that. This subject could go onto another 10 pages, as it happened before. Roughly to lay it down: You can crop 8 pixels left and right to get 704 and then encode it using SAR like 12:11 (again another one page topic why 16:15 or why 12:11). If you do not use SAR then you have to change resolution to square pixel, like 768x576 etc. and then crop it. And we did not start to discuss those bad lines at the bottom after capture, that might be masked too by black strip or cut it off. You do not want to change vertical resolution by couple of pixels if you simply crop the bottom and resize to 576 again, not a good solution. But again , this would be hardly called archived footage then.

Half archive/ half playback medium would be using QTGMC, and encoding using low CRF and leave everything in video, grain, not changing resolution and not cropping, something like:

x264.exe --crf 16.5 --sar 16:15 --ref 6 --keyint 50 --tune grain --vbv-bufsize 25000 --vbv-maxrate 23000 --output "out.264"  "input.avs"

that would give you stream about 10Mbit/s, not resized, still anamorphic 50p video, you could play it with any device, but procedure and the whole archive thing would be highly irregular not optimal

[newpball]

'watching archive'

Does 'watching archive' mean "I do not want to bother making a good archive of my VHS sources" or does it simply mean a good delivery format?

Here is what I think you should do:

1 Archive the VHS source ASAP, near lossless without any cropping or processing.
2 Save this archive at least on two different media carriers and in two different locations.
3 Make as many copies from this archive as you like and play with it to your hearts content!

Seems to me your question is about stage three!

For a good delivery copy in stage 3 I would:

1. Deinterlace QTGMC with frame doubling
2. Crop the crap
3. Color correct
4. Only if necessary apply some mild processing, avoid idiotic sharpening and "grain" removal, we are dealing with low quality VHS, processing tends to remove any detail that is left. The only exception to this is cartoons.

[Killer3737]

Well I know what I could do and the results would look ok to me, I was really just wanting to hear what other people are currently doing, and why, the pros and cons and reasons for those choices.

So yeah, I'll capture and archive a 720x576/480 interlaced totally raw, untouched file, that's fine. It's the lossy encode for today and near future I'm really wondering what people do. As I say in the past few years I've been encoding DVDs and that doesn't really need much thought or opinion, you encode to DVD specs and mask any garbage with solid black. Encode with reasonably high VBR and recommended settings, job done. When you're only targeting a file, and with H.264, things are trickier.

Regarding cropping yeah, as I say this is the issue with getting different inputs, everyone has differing opinions. Which is good, I suppose, I'd like to hear different thoughts and ideas. So as recommended somewhere else, they would say to crop the 720x576 anamorphic input to, for example, 692x564, removing any black bars and garbage, still anamorphic. They then gave me a script to find the correct SAR, which is as follows:

avsfile = "encode.avs" # Encoding script
format = 0 # 1=NTSC, 0=PAL
wide = 0 # 1=Widescreen 16:9, 0=Full screen 4:3
#########################
ITU = (format==1?10:12)/11.0*(wide==1?4.0/3:1)
SARs = """"12:11","16:11","10:11","40:33","16:15","64:45" ,"8:9","32:27""""
ITUprof = "SDB "+(wide==1?"ANAMORPHIC ":"")+(format==1?"NTSC":"PAL")
i=import(avsfile).converttorgb
i
ab = round(height*(sqrt(45.0/44)-1))
a = spline36resize(round(width*ITU),height)
a = a.addborders(0,floor(ab/2.0),0,ceil(ab/2.0))
bb = width(a)-round(width*ITU/sqrt(45.0/44))
b = spline36resize(round(width*ITU/sqrt(45.0/44)),height+ab)
b = b.addborders(floor(bb/2.0),0,ceil(bb/2.0),0)
interleave(a,b)
scriptclip("""subtitle("Playback Resolution: "+\
string(round(width(i)*ITU*pow(44.0/45,current_frame%2)))+"x"+string(height(i))+\
"\nMeGUI Profile: "+ITUprof+(current_frame%2==1?" NON-ITU":"")+\
"\nx264 --sar "+eval("select(2*format+wide+current_frame%2*4,"+S ARs+")"),lsp=0)""")

I'm not 100% clear on how SAR works but with that script I just flick through the file, judge which SAR is correct and apply that on the encode, which may then give me a display dimension of 738x564. These files then play correctly on PC, but I guess some media players may not like them. As far as I see the benefit of cropping rather than masking is that the resulting video fills whichever device you're watching on, rather than creating a small windowframe effect with masking.

Is there anything inherently 'wrong' with that process in your opinion then?

[_Al_]

I use this table in batch script to get proper SAR, that might help you, try to multiply that SAR to horizontal resolution %w%: 704, 720, 1440 or 1920 and you shoud get always square pixel horizontal resolution:

if %ar%==4:3 if %h%==480 if %w%==720 set sar=8:9 
if %ar%==4:3 if %h%==576 if %w%==720 set sar=16:15
if %ar%==16:9 if %h%==480 if %w%==720 set sar=32:27
if %ar%==16:9 if %h%==576 if %w%==720 set sar=64:45
if %ar%==16:9 if %h%==720 if %w%==1280 set sar=1:1
if %ar%==16:9 if %h%==1080 if %w%==1440 set sar=4:3
if %ar%==16:9 if %h%==1080 if %w%==1920 set sar=1:1

if %ar%==4:3 if %h%==480 if %w%==704 set sar=10:11
if %ar%==4:3 if %h%==576 if %w%==704 set sar=12:11
if %ar%==16:9 if %h%==480 if %w%==704 set sar=40:33
if %ar%==16:9 if %h%==576 if %w%==704 set sar=16:11

[_Al_]

you encode to DVD specs and mask any garbage with solid black. Encode with reasonably high VBR and recommended settings, job done. When you're only targeting a file, and with H.264, things are trickier.

It is easier because you do not need to mask or keep 720x576 for encoding. You you can resize to square pixel. If that is a case and not an anamorphic video with SAR. So first you resize to square pixel, 768x576, anything that gives you 4:3 ratio. Then you decide to crop whatever out of picture. As long as you keep a decent mode, like 4, you should be fine. To keep those 576 lines intact is always a good idea (or crop afterwards). In PAL land you just upscale to768 a bit. In NTSC land it is not that easy because if you decide to keep 480 vertical resolution, you need to downscale to 640 for horizontal resolution. One could say that is not optimal as well. But in PAL land it is kind of easier.

[Brad]

But for the 'true' archive copy, is HuffUV or Lagarith a good idea, bearing in mind support for those codecs may be non existent in 20-30 years? Or is the 'safest' way totally raw RGB, bearing in mind that means much more space required obviously?

I figure you're pretty safe with anything that's a part of ffmpeg, but if you want the backing of a standards body you could use lossless x264. If you feel uncompressed is safer, you'd want to use YUY2, not RGB. Barring some strange catastrophe, I don't think the math required to convert from YCbCr to RGB is going anywhere.

Also a similar thought on re-sizing, I was again told it was better to keep it anamorphic and apply the correct SAR, but would a say 720x540 version really look/compress worse?

For square pixel, resize horizontally, not vertically. Then you keep the original discrete lines from the analog version (or the mo-comped versions of them in the case of QTGMC), and for NTSC you end up with fewer pixels to compress without actually losing anything besides noise. Unless your source is unusually high-resolution S-VHS/Hi8.

I did test encode something with MBAFF in x264 and I wasn't too keen on the results, at least when played via USB on my TV. Motion didn't seem smooth, and I noticed some combing. When I deinterlaced with QTGMC to 50p it looked better to me, smooth and no combing artefacts.

If these are the devices you're playing on, then sure, do whatever looks best for you.

[capture side] What about colour and chroma?

YUY2 is sufficient (and overkill).

These files then play correctly on PC, but I guess some media players may not like them.

Indeed.

[Killer3737]

Ok thanks for the input folks. You've helped a lot. I will probably try a few of the different options mentioned and see which I prefer, and which is most compatible with the hardware I have to test with.

Final quandary is audio, and again I guess it comes down to hardware compatibility. I know the MKV container allows LPCM or FLAC but support for both isn't always great on hardware (I know my Samsung HDTV USB software sometimes doesn't play LPCM cleanly and flat out doesn't support FLAC), so I guess high bitrate AAC may be the best choice. Unless anyone has other thoughts.

[pippas]

+1 for AAC audio. I find that .mp4 files with square pixels and AAC audio is the most compatible for all my players (PVR - WD Live -TV USB slots...)

[Killer3737]

In NTSC land it is not that easy because if you decide to keep 480 vertical resolution, you need to downscale to 640 for horizontal resolution. One could say that is not optimal as well. But in PAL land it is kind of easier.

Just to touch on this, since I'm in PAL land I shouldn't be encoding too many NTSC files but for reference, obviously the other option would be to keep the horizontal res and go to 720x540. Wouldn't that be a better idea, rather than dropping from 720 to 640?

[_Al_]

If you'd have some high quality DVavi , captured by camcorder, I'd choose 720x540. If you had captured VHS, 640x480 is fine.

[Brad]

I seriously doubt any consumer NTSC camera of any format ever managed to utilize full D-1 horizontal res. I'm not counting making a DV recording using a HDV camcorder (even then?).

[_Al_]

I used Sony VX2000, desperately tried to keep at least resolution it recorded, but basically you are right, ..., that DV avi usually is archive anyway, some space could be saved too using 640x480 mp4.

[Cornucopia]

I've already described why it is usually better practice to retain the original vertical (480) and resize the horizontal.

Do whatever you want, though - you're the one who's going to live with it.

Scott

[Brad]

^ Here is a post where Cornucopia discusses this in the context of encoding from a DVD source. I haven't been following that particular thread, personally.