HD to MKV from 25p (?) upconvert. Best option?

[Tiernan]

WTTW in Chicago has been airing the newer series of Doctor Who (1 and 2 up till now) on their HD channel in 720p (59.94fps), specifically THE RUNAWAY BRIDE last weekend. My understanding is that it is filmed 25i and edited in 50p for broadcast in the UK, and as it's unlikely that I'll be able to get an answer from WTTW about the exact process by which they achieve this result, I thought I'd put it in MeGUI to see what the analyzer determined about the source. Eventually, it indicated a progressive source, and I set color correction (which it may not have needed). So, after a few hours processing (using the x264: DXVA-HD-HQ template), the end result (4.5GB .TS to 680MB MKV) was only OK, and certainly not to be confused for real HD (a fair amount of macroblocking, etc.)

So, the question is: since this was not filmed as HD, and never intended to be presented as HD in any form, should there be any possible settings (in MeGUI/AVS/etc.) that would allow the MKV conversion while maintaining as much of the existing HD quality and not compromise the quality any further?

[jagabo]

Use CRF encoding in x264. Pick the quality you want and encode. A ratefactor around 20 will probably give you what you want.

[Tiernan]

Thanks for the suggestion. I can't find that option in the x264 portion of MeGUI.

Another thought came to mind also. When I ran the analysis the first time, it didn't suggest decimation, or add that to the script it generated. With the original being 25p, would it make sense to leave it as 1280X720, and just decimate to that? I don't remember ever seeing such a thing, but this is kind of an odd exercise anyway...

[poisondeathray]

You still have to adjust the bitrate (for 2pass) or crf (for 1pass) in the settings. The template is just a starting point. If you didn't touch anything on the 1st attempt, it was probably 1000kb/s... which would explain your poor results. Push the config button beside the profile dropdown

If you are using a DXVA profile, this suggests you might be targetting a device for playback. If this is true, you should use a 2pass mode, as VBV doesn't work well with crf mode

I wouldn't rely any of the auto analyzers. They all make mistakes.

If there are dupes , it would make sense IMO to get rid of them , otherwise you are just wasting space. If each frame is unique, then obviously keep 50p

[Tiernan]

Thanks for the further follow-up. I kind of went with my gut instinct and added:

TDecimate(Mode=7,Rate=25)

to my existing script. About 12 hours later, it was finished, and it does look better, and given how relatively poor the upconvert from 25p/i appears to be, it seems nearly exactly the same as the existing .TS file. I've also realised that, it almost appears that RTD very specifically included very possible encoding complication in the first 30 seconds: slow deliberate movment, quick pans, static textures, crowds, dark blacks and browns, etc.

I still haven't found a reference to CRF, but I did verify that (per the template), it ran in "Automated 2-pass" mode (I dont see a reference to VBV, but I'm assuming it's a variable bitrate setting, and I didn't (initially, anyway) choose any options like that.) I do see, though, that the bitrate is set to 1000. Given the decimating, going to mp4, etc., I really don't know how to judge (and wouldn't have known to make any change to the setting) for what should have been a good bitrate to use.

As I say though, the resulting MKV does compare very favorably to the existing TS, and at this point, I'm disinclined to spend MUCH more time on the exercise, as I really don't have the time or inclination to spend QUITE so much effort for 1 episode, when I have to do the same for SO many other episodes yet. The last little complication that I may have to deal with is what adjustment I may need to make on the AC3 file. Having decimated the original TS file, I'm assuming that I'll have synch issues now. Should I have to adjust for this?

[jagabo]

Yeah, since the Doctor Who series was shot in SD there's not a lot to be gained by capping the upscaled HD and converting back to SD. The later HD episodes would be worth capping in HD though.

[edDV]

Hmm, imdb says the 2005- Dr Who series was mastered to 16:9 Digital Betacam presumably at 720x576i/25fps but maybe 24p. 2009- was mastered HD presumably 1080i but maybe 1080p/24p. This is a low budget TV series.
http://www.imdb.com/title/tt0436992/technical

So what you are capturing is 16:9 720x576i H stretched to 1024x576 then upscaled to 1280x720p/59.94 for broadcast. Certain models of DigiBeta decks can internally upscale to 720p or 1080i but more likely a national distribution would be externally processed.

[edDV]

Yeah, since the Doctor Who series was shot in SD there's not a lot to be gained by capping the upscaled HD and converting back to SD. The later HD episodes would be worth capping in HD though.

That depends how you capture it. If ATSC, it only comes in 720p. If by SD cable or sat, 1280x720p would be downscaled to 524x480i (or less) and bit rate starved to around 3.6Mb/s (ave), then H stretched to 720x480i.

I'd say cap the 720p upscale for now, then get the DVD later for native SD 720x576 or 480.

[jagabo]

Most of them are already out on DVD in the UK.

I've downloaded most of the episodes as MPEG 2 720x576i25, direct off digital broadcast in the UK. The encoding is interlaced but the frames are progressive. They run about 4500 kbps.

[edDV]

Most of them are already out on DVD in the UK.

I've downloaded most of the episodes as MPEG 2 720x576i25, direct off digital broadcast in the UK. The encoding is interlaced but the frames are progressive. They run about 4500 kbps.

An SD digital cap off BBC would beat an analog cap off US cable.

I just thought of another capture senario. Most PBS stations run a second digital channel in SD on a separate schedule. They would downscale the HD net feed to SD 704x480i and broadcast ~3.5-4.0Mb/s (ave) in digital MPeg2. This would look better than cable analog and can be digitally captured off ATSC or off the HD cable box IEEE-1394 port as an MPeg2-TS stream.

[jagabo]

Most of them are already out on DVD in the UK.

I've downloaded most of the episodes as MPEG 2 720x576i25, direct off digital broadcast in the UK. The encoding is interlaced but the frames are progressive. They run about 4500 kbps.

An SD digital cap off BBC would beat an analog cap off US cable.

The US broadcasts on the SciFi channel used the terrible partial field blending conversion technique too.

[Tiernan]

Thanks guys for the sudden influx of information..

Yes, this is WTTW Digital, the 2nd digital channel, though in this case, it is their HD channel, and yes, I am digitally capturing it of the HD cable box IEEE-1394 port as an MPEG-2-TS stream.

I already own the DVDs and got just about all the recordings from the UK in just about every form possible, but there was just something inherantly interesting/cool about capturing what may be the only us broadcast in (upconverted) HD. I was just curious about what they've done to create it, and whether or not it would be possible to get something vaguely good out of it. I'm pretty happy with the results so far, but as was already pointed out, the default bitrate with the "x264: DXVA-HD-HQ" template in MeGUI is probably too low, so that might be something I'll increase for future attempts. Luckily, subsequent airings are only ~42 minutes, and the package they purchased does not appear to include the "Confidential Cutdown" that the previous 2 seasons included.

(on a side note, I have also capture from other local independent stations that ARE running their second channel at SD 704.480i and as I'd like to be able to author some of this stuff to DVD, it's pretty annoying!)

[edDV]

...

(on a side note, I have also capture from other local independent stations that ARE running their second channel at SD 704.480i and as I'd like to be able to author some of this stuff to DVD, it's pretty annoying!)

All SD ATSC is 704x480 (mostly i but can be p). After demux, it can be authored to DVD directly.

BTW, wherever I said 720x576 above I should have said 704x576.

[poisondeathray]

1000kbps is very likely too low for any content on HD , it might be ok for your SD shows depending on complexity of the content.

jagabo was suggesting a CRF mode, instead of a 2pass mode (which would give you an exact average bitrate). You just enter a "quality level" and it will give return that. So instead of a set quantizer, the quantizer varies giving you more variablility according to source complexity. A complex source (eg. action movie, lots of noise & explosions) will end up with a bigger bitrate & filesize, but a simple source (eg. a slow moving documentary on painting) will take up less size. By arbitrary setting a 2pass bitrate value (e.g. 1000kbps), you might be starving those encodes that need it, or wasting bitrate on those that don't

Most people use a crf value around 18-22 (22 is lower quality, smaller filesize) , although crf values have been "rebalanced" several times in the past and are about to be again very shortly - so don't get "fixated" on a value

The problem with crf is that the VBV model doesn't work as well - this is why you don't see any DXVA presets with CRF. VBV settings are required for buffered playback and necessary for targets like devices to function properly. If you are using PC only as the target (not a device, like a media player, or bluray player), and are using software to decode (not like a graphic card using DXVA mode), then DXVA or VBV isn't necessary, and CRF mode might be a better choice (it's definitely faster, about the speed of the 2nd slower pass). If you need a set filesize, then you have to use 2pass (and a bitrate calculator)

[vhelp]

Those doctor who series should be able to restore back to 25p (or 24.975 or something) I successfully restore many (of the ones I did work on) without major problems, if any. In fact, the only one I could not restore back to was the very first "pilot" episode, it was a 2 hr long problem (although they split them) and it was retelecined for time expansion or compression. I did not review forth which direction it was, expansion or compression. But because of that, the restoration was practically impossible and I gave up on that. Beside, it was right at my earling stages of my PAL conversion education back then. Some day I'll take another crack at it though the master (now) is an EP recording.

I have seen a change in the commercial dvd releases. Most of the tv series (at least the ones that I have so far) seem to have gone through a change. Maybe not all releases, but I've noticed that on the dvd's I've been working on, the telecine is now consistant instead of "edited" or cut. There's more fading in/out. I suppose to help aid in the HD tv's dealings to some degree. Just image how friends ( member them ? ) were all "edited" real messy. The last dvd disc I did work on with them had the edits inside the telecine and you couldn't restore back properly. I wonder if they fixed these today. I have to pick up one to review. I love that show--it would be worth a looksees to see if can be clean restore, finally. Anyway..back to topic..

Here are a few script snipplets to play around with. These were from a few experimental "who" restores I was trying to improve quality on. The thing you migth want to consider is: where you start the script at inside the telecine pattern will provide the best results. So, play around with them and fine-tune them to your likings. Perhaps things have improved upon since I last worked in this series and someone could post an even better script snipplet. For now, at least give these a try and see how you fair. So, unless they changed something in the telecine, like time expansion or compression, these snips below should help get you started, I hope.

   separatefields()
   SelectEvery( 4, 0,3, 2,5)
   weave()  

   TDeint(mode=1)
   RePAL()

. . .

  LeakKernelBob(Order = 1) # 1=TFF, 2=BFF
  RePAL()

-vhelp 5169

[Tiernan]

Whew, even more responses. I'll need to start with the less technical response first...

vhelp, it appears that you tried working on episodes of the original series (which ran from '63-'89). Most of the pre-'70 episodes (like the pilot episode you referred to) were subject to junking by the BBC (a very long story), so what ended up being recovered/saved in the archives are very typically 16mm telecines of various types. They came back to the BBC in just about every possible form imaginable, with damage, cuts, splices, etc. which would very likely ruin any kind of consistant pattern needed for IVTC (never mind that the original telecining wasn't necessarily 100% correct to start with). If you check out the Restoration Team page at:

http://www.restoration-team.co.uk/

there's a fair amount of (sometimes) highly technical discussion about ALL of the BBC's Doctor Who releases (and others).

With them working directly from the archive, AND having access to equipment most of us would only dream about being able to use, they've restored the stories to amazing quality (sometimes, even better than original broadcast) and the resulting DVDs would be almost totally different than what would have been shown on PBS over here in the 80's and 90's.

Getting a bit more back on topic... the code you've posted - I'm guessing that it was specifically designed to attempt to compensate for the afore-mentioned inconsistancy. How much would it change if there was not a need to account for that? Additionally, I'm not familiar with the "RePAL" and "LeakKernalBob" plugins. Where do they fit in the picture?

Thanks for bringing in your experiences too, vhelp. This is how I'm going to learn more about this (sometimes) complicated process.

[Tiernan]

Poisondeathray,

Thanks for the further clarification. I've tended to use VBR when possible for other media projects in past, so I'm not quite sure why it didnt occur to me to use it with this one. I AM a bit curious why the DXVA template would use what seems to be such a low bitrate, but I was concerned about making any change to the template, for fear of making a change that might have some negative affect that I'd only be able to determine AFTER the encoding. Still, having read more about x264 and MeGUI, I do feel like I have a bit better understanding of what I'm doing, even if I'm still disinclined to make TOO many changes to the default.

So, you mention about the VBV model not working well. Should that prevent me from trying it? I still prefer to encode to something that I know will be DXVA-compliant, but from what I've read, adhering to Level 4.1 specs is really the most important part of the encode, and that AVC Level selection shouldn't be affected by any of the changes you've suggested, right? To further clarify, though it won't specifically be important on THIS encode, most of my MKV authoring is planned for playback on my HTPC (ATI HD2600 PRO-based), so the intention is that all HD playback be done on the card in DXVA, yes.

Given what we've discussed now, I'm just a bit more inclined (since I don't need a set filesize) to try a CRF encode (at 20 or 22) Then, I can do it in 1-pass, and if I'm lucky, perhaps get just as good a quality/result out of it.

[Tiernan]

Follow-up:
OK, I've done a couple of encodes (concert video from Palladia, and a local history show) with the suggested changes (specifically, using the DXVA preset, using CRF at 23/24, instead of the 2-pass at 1000) The results seem pretty good, and I've tested one of them, and MPC seems to indicate that it is still DXVA-compliant, so that is good news after all the effort. Still, with CRF generating such wildly varied filesizes, is there a better way to use MeGUI with the CRF options and have more control over the resulting filesizes?

[jagabo]

Still, with CRF generating such wildly varied filesizes, is there a better way to use MeGUI with the CRF options and have more control over the resulting filesizes?

No. You either pick a file size and live with whatever quality you get, or pick a quality and live with whatever file size you get.

[creamyhorror]

OK, I've done a couple of encodes (concert video from Palladia, and a local history show) with the suggested changes (specifically, using the DXVA preset, using CRF at 23/24, instead of the 2-pass at 1000) The results seem pretty good

CRF 23-24 trades off quality for smaller filesizes. You can try 21-22 for better quality on HD material (and 19-20.5 for SD material).

Still, with CRF generating such wildly varied filesizes, is there a better way to use MeGUI with the CRF options and have more control over the resulting filesizes?

In the first place, why do you want similar filesizes? The point of CRF is to allocate however many bits a video needs to achieve a certain quality. Controlling the filesize goes directly against that, and only makes sense if you need to fit episodes into a limited space (e.g. large videos on a DVD-R). Otherwise, CRF is the way to go.