Best format for archiving HD videos

[Xoanon]

I've created some HD video content that is, primarily, intended for online viewing only (it's a web series). However, it would be wise (in my experience) to archive the videos offline for future use (e.g. Blu-ray video creation). I am exporting from Premiere Pro CS5.5 using the YouTube Widescreen HD H.264 setting for the online videos (with some slight tweaks to the audio settings as needed). But what format is good for encoding the HD masters for archiving? I've got the audio masters rendered to 24-bit FLAC files but I am not sure of what format is best for the videos. Does anyone have any suggestions? When I was editing SD footage I exported the videos to Huffyuv or Lagarith but with HD footage the resultant file sizes were quite large.

Thank you for any help or advice you may have.

[KarMa]

Could just use a low CRF in x264 with one of the slower settings. I would not bother with being Bluray compatible unless you really need that.

[Xoanon]

At the moment, there is no original file; just the original timeline in Premiere.

[teodz1984]

Just Keep your Original Source Video as Is..
That is the essence of archiving...

[Xoanon]

I understand that, but there is no original source video at the moment. I literally edited a bunch of clips and stills in Premiere, and have exported an MP4 for YouTube. What is the best way to export my timeline as a whole new encode for archiving?

[_Al_]

Premiere complicates these sort of ordeals because it does not support quality encoding, so you might encode lossless first anyway or use debugmode frame server to encode using x264,

so might go with x264 encoder, lower CRF (as low as much bitrate you can afford to store), watch for latency during playback (for editing later) so you set short GOP and complexity (rather easier to decode), so perhaps
--crf 16 --preset medium --tune grain -keyint 50

or just use Premieres templates, something 50Mbit/s but as I said it is not that space effective because of lacking CRF settings within Premiere

[KarMa]

Could export a lossless version, and then encode that with x264.

[Xoanon]

yes, H.264 lower CRF (as low as much bitrate you can afford to store), watch for latency during playback (for editing later) so you set shor GOP and complexity (rather easier to decode), so perhaps
--crf 16 --preset medium --tune grain -keyint 50

How do I set those parameters in Premiere?

[KarMa]

Does Premiere even support x264?

Looks like it does but the plugin costs $300, to use a free codec. https://forums.adobe.com/thread/1026138

[Xoanon]

Could export a lossless version, and then encode that with x264.

Assuming I export, say, a Lagarith AVI file from Premiere and use Handbrake to encode an x264 master file, what settings do you propose I use? I understand it's a lossy codec but it's something.

[teodz1984]

http://lags.leetcode.net/codec.html

[JVRaines]

FFV1 from the FFmpeg suite will give you better lossless compression than Lagarith. FFV1/MKV has been adopted as an archival standard by a number of cultural institutions.

[Xoanon]

I am not familiar with command line applications. Is there another, simpler MKV-encoding software I can use? Or is FFV1 not as complex as it first appears?

[KarMa]

FFV1 from the FFmpeg suite will give you better lossless compression than Lagarith. FFV1/MKV has been adopted as an archival standard by a number of cultural institutions.

Kinda silly to go to the effort to use FFV1 with its extra encoding/decoding just to make an intermediary 5-10% smaller. I use FFV1 for saving lossless stuff myself, for stuff that will stay lossless.

[_Al_]

I'm sure if you go into Premieres export templets and codec choices there might be some intra-frame encoding format or other , maybe marked as MOV (for container), just watch for final bitrates. And those are usually set up for later encoding in mind and archiving/intermediate, so no need for special settings like in x264 command line where we have to make sure.
Is this only what you got there?

[Xoanon]

No MOV option, no.

[_Al_]

Not sure what advanced options for quicktime or H264 are, there might be option to set keyframes , if setting 1, you have sort of intraframe , but it makes more sense to set it 50 or 60 or something, also default complexity settings are perhaps moderate also, nothing crazy.
So you are left with bitrate, as was said, you do not have quality setting oprion (CRF) only 2pass VBR or CBR, your choice.
As for 2pass VBR, you can select some average bitrate that is appropriate for you , 40Mbit/s, max bitrate you select 50 or something.
Choosing CBR you get encoding time shorter, 1 pass only, but it is a constant bitrate stream, so pretty inefficient, and also you choose not only average but maximum bitrates at the same time.

[JVRaines]

FFV1 from the FFmpeg suite will give you better lossless compression than Lagarith. FFV1/MKV has been adopted as an archival standard by a number of cultural institutions.

Kinda silly to go to the effort to use FFV1 with its extra encoding/decoding just to make an intermediary 5-10% smaller. I use FFV1 for saving lossless stuff myself, for stuff that will stay lossless.

Intermediary? I thought we were talking about offline archiving.

[JVRaines]

I am not familiar with command line applications. Is there another, simpler MKV-encoding software I can use? Or is FFV1 not as complex as it first appears?

It's not that fearsome. FFmpeg is an executable program without a graphic interface. You invoke it as a command with all desired options and it produces an output file for you. Once you have the options tweaked the way you like, put the command into a batch file and forget about the details. In a batch file, you can set it up to work on a whole bunch of files at once using looping commands. Here is a basic FFmpeg command, with options, for making an archive with video and audio.

ffmpeg -i uncompressed.avi -c:v ffv1 -level 3 -coder 1 -g 1 -c:a copy archived.mkv

-i tells it what file to use for input
-c:v tell it what video codec to use
-level, -coder, and -g are recommended FFV1 settings
-c:a tells it what audio codec to use (in this case, just copy the audio and leave it uncompressed)
and the last parameter is the name of the output file to write. The extension tells FFmpeg that you want a Matroska format.

[KarMa]

Intermediary? I thought we were talking about offline archiving.

OP is not interested in archiving the original losslessly, as said in the first post.

[_Al_]

As I understood, no lossless, op says it is too much, those can be what, 150-200 Mb/s? So I gathered he aims for something like 50Mb/s or whatever he chooses.

[Xoanon]

I am not familiar with command line applications. Is there another, simpler MKV-encoding software I can use? Or is FFV1 not as complex as it first appears?

It's not that fearsome. FFmpeg is an executable program without a graphic interface. You invoke it as a command with all desired options and it produces an output file for you. Once you have the options tweaked the way you like, put the command into a batch file and forget about the details. In a batch file, you can set it up to work on a whole bunch of files at once using looping commands. Here is a basic FFmpeg command, with options, for making an archive with video and audio.

ffmpeg -i uncompressed.avi -c:v ffv1 -level 3 -coder 1 -g 1 -c:a copy archived.mkv

-i tells it what file to use for input
-c:v tell it what video codec to use
-level, -coder, and -g are recommended FFV1 settings
-c:a tells it what audio codec to use (in this case, just copy the audio and leave it uncompressed)
and the last parameter is the name of the output file to write. The extension tells FFmpeg that you want a Matroska format.

Thanks for your reply. So this will produce an HD .MKV file with good enough quality to encode to Blu-ray in future, if so desired?

[_Al_]

Yes because ffv1 is lossless,
but in the first post you excluded lossless codecs because of the size. Understand lossless codecs can differ here or there, but not that drastically so you cannot say this lossless is not ok (sizes are too large) but this lossless is many times better (size is considerably smaller). There is always a certain amount of "chaos" in video that you cannot compress anymore so to speak (having limited computing power for algorithm and decoding as well)

Some codecs are in kind of different category they are "almost" lossless, lossy, byt not that much, using different quality settings or bitrate where you just select bitrate or quality (that makes sense) and you get intermediate or archived file that gets filesize a bit smaller but it is still ok for the task.
Like Cineform or DNxHD, so you might want to check those.

[Xoanon]

I excluded lossless as I was thinking my only options were Lagarith or Huffyuv. But the DNxHD codec sounds like it would be perfect if it creates smaller sizes than the two aforementioned codecs. Any comment on what the best encoder settings to use are?

[smrpix]

With DNxHD your options are bit rate and bit depth.
https://en.wikipedia.org/wiki/List_of_Avid_DNxHD_resolutions

[SameSelf]

I am jumping in a little late in this thread, but one alternative to keeping large rendered versions of your PP timeline is to export an EDL or XML. That is the format used by pros. Unless you are doing some really fancy stuff in your timeline, the EDL/XML should have all the information needed to rebuild the timeline from the masters in not only PP but nearly every NLE/finishing program out there as well, should you ever ditch PP in the future. Easy to test.

Otherwise, the only way to really export anything out of PP without it screwing up your footage (think lossless round trips) is uncompressed UYVY (see my lossless workflows thread in my signature below). With that, you can easily transcode to a archivable timeline. My preference would be an edit-ready codec like ProRes or DNxHD/HR which you already seem clued in on. But keep in mind that the only reason to really archive a timeline is if you envision having to encode for other delivery formats e.g. DVD, Bluray, etc. and don't want to have to export more than once. IMO, it is just not worth archiving a timeline; that is what masters are for. It is much more efficient just exporting an EDL/XML and keeping that with your masters. Guaranteed whatever you did today, in six month or whenever you come back, there will be something you will want to change/tweak, and you will want to do that on a timeline built from the masters, not an archived timeline.

[Xoanon]

Thank you everyone, and thanks for your very detailed reply, SamSelf. I am not wishing to keep masters, just a video file of the final edit, so I think DNxHD may be the way forward once I take a look at the codec settings.

[Xoanon]

I installed the Avid codecs but do not seem them in my Premiere Pro export options. Any ideas, anyone?

[SameSelf]

I don't have the Avid codecs installed on my system, but IIRC, they will be found under Quicktime Format dropdown and the Video Codec dropdown. So you may need to install QT for Windows as well if you don't see QT listed.

[smrpix]

There are presets under quicktime and DNxHR/DNxHD MXf op1-a. You can also make custom presets in either .mov or .mxf