TMPGEnc DVD Author 3 - Stopping it Re-encoding

[Incast]

I've recently moved from TMPGEnc DVD Author 1.6 to 3.

For me this move is essential as the new version gives my MPEG2 material the correct ITU-601 colorimetry flag, which no other software does to my knowledge.

However, preventing it from re-encoding is proving impossible. With the correct settings (I don't set a target size) I've managed to prevent it re-encoding up until the final few GOPs of each track. But once it gets here it seems to always insist upon re-encoding. Quite how much it does tends to be quite sporadic, ranging from one to three gops.

This is hardly earthshattering, but I don't want to re-encode what is compliant material. I have tested from an enormous array of sources, commercial DVDs, DVD recorder material, DVB recordings, with audio, without audio, closed GOP, open GOP.

Nothing I throw at it will get through the program completely untouched.

I've looked into the archive here and found a post by a TMPGEnc Tech arguing that this happens for incorrect timecodes in version 2. But I can't see how everything has incorrect timecodes.

Can anyone shed any light on what it's doing? I don't want to re-encode any source material unless it's justified.

[nTekka]

That sounds a bit strange, even commercial DVDs required you to re-encode? This may sound stupid to ask but are you sure you are in the correct DVD project template, ie. NTSC or PAL? If you are in PAL and importing NTSC, or vice versa then pretty much all NTSC imports will require re-encoding. If you can post a GSpot screen shot of a sample MPEG2 clip then we might be able to figure out the problem.

[Incast]

Yeah, this happens to literally everything. I've tried various PAL/NTSC sources - all in the right settings. I don't think a gspot clip is going to help, the more I look at it the more this is a problem/'feature' of the application.

It even repeats the procedure when I import a disc I just authored with DVD Author 3, which makes me incredibly suspicious as to how 'smart' this system truly is.

I've just tested in DVD Author 2 and exactly the same happens there, although it does it far more subtlely since there's no video preview when it re-encodes.

If anyone else with TMPGEnc DVD Author 3 wishes to test - it's very simple to spot. Author as usual, when you output it will show you a preview of the end of the video as it re-encodes. If you don't get a preview at the end, it isn't re-encoding.

[j4gg3rr]

GSpot is showing 1 closed GOP closing out the 3rd last GOP where the re-encoding starts.

EDIT:
Re-authoring the previous output has GSpot reporting even more seamingly random closed GOP's at 15 and 16 second marks of the 30 second test file.

EDIT 2:
Womble MPEG Video Wizard DVD exports the same 30 second test MPG to VOB without re-encoding any frames.

[Alex_ander]

I don't have this problem so far. To get it untouched I set chapters only at key frames highlighted in red. TDA3 is as fast as TDA1.6 (TDA2 was much slower) and gives ability to correct adjacent button links after re-arranging buttons in template.

[j4gg3rr]

My test was with no chapters or menu.
It changed the 3rd last 15 frame GOP into a 13 frame closed GOP
and changed the last 10 frame GOP into a 12 frame GOP.
So all 40 I P B frames after the closed GOP now correspond to different frames than originally encoded.

eg. frame 720 was a B frame is now the last I frame.

I would prefer my I and P frames not to be re-encoded B frames.

[manono]

For me this move is essential as the new version gives my MPEG2 material the correct ITU-601 colorimetry flag, which no other software does to my knowledge.

Care to explain that? Are you saying there's something "correct" about SMPTE 170M (ITU-R BT.601)? Since the vast majority of commercial DVD releases use ITU-R BT.709, and since the vast majority of the software encoders available to us and hardware encoders (as far as I know, based on the examination of DVDs) also output ITU-R BT.709, I'm not quite sure I understand. It seems to me that assigning that flag to something already encoded as ITU-R BT.709 can't be good. It shouldn't be blindly reassigning the colorimetry, but keeping it the same. The help file says it Displays the source registered YUV color space, and that's not true. It's displaying what it'll become after it gets through messing with it.

In my tests, it also encoded at least the last little bit of the video. And what's worse, if it was originally 23.976fps progressive with pulldown applied, it reencoded that little bit as already telecined interlaced 29.97fps, also bloating the size a little bit.

[Incast]

Thanks for your comments.

The Test
I've done some further investigations. I've compared the GOP structure of the end of an MPEG2 video in:
a. in untouched elementary form
b. authored with dvd author 3
c. authored with dvd author 3, demuxed with dgindex back to elementary, and then authored a second time with dvd author 3 (in this case if there is any logic behind the re-encoding the program should not perform any changes the second time, and the structure of c should match b)

Below is the GOP and frame structure that follows:
a.GOP 4340 (Open) BBIBBPBBPBBPBP GOP 4341 (Open) BBIBBPBBPBBPP[End of video] Total number of frames: 27

b. Re-encoded: GOP 4340 (Closed)IBBPBBPBBPBBP GOP 4341 (Open) BBIBBPBBPBBPBP[End of video] Total number of frames: 27

c. Re-encoded: GOP 4340 (Closed)IBBPBBPBBPBBP GOP 4341 (Open) BBIBBPBBPBBPBP[End of video] Total number of frames: 27

Conclusion:
In test c it re-encoded yet again, but used the exact same frame structure, which was a completely pointless procedure.

However, this test shows that it isn't simply arbitrary it's clearly re-encoding for a reason. But the system is hopelessly stupid and when it reaches the end of any video it will re-encode into a specific format no matter what.

So, it's clearly following a pattern. But why? What does this gain? I've never read anything in the specs about a GOP one or two away from the end needing to be closed.

It should also be noted that it will re-encode when you encode something entirely in closed GOP too.

[Incast]

Care to explain that? Are you saying there's something "correct" about SMPTE 170M (ITU-R BT.601)? Since the vast majority of commercial DVD releases use ITU-R BT.709, and since the vast majority of the software encoders available to us and hardware encoders (as far as I know, based on the examination of DVDs) also output ITU-R BT.709, I'm not quite sure I understand.

I have a large collection of DVB recordings, DVD recordings and commercial DVDs from Formula One racing.

All of the commercial/DVB material has no colorimetry flag. Both should default to BT.709 as a result. However, after questioning this at doom9 (http://forum.doom9.org/showthread.php?t=127907) it seems that all SD material should actually be flagged as BT.601.

As a result DVD Author 3 seemed like my answer since it patches everything with the 601 flag, and should ensure absolute compliancy.

[manono]

Hi-

But the Doom9 guys were talking about broadcast transmissions. I don't know anything about caps. I was referring to DVDs. Of the DVDs I see, maybe 90% are ITU-R BT.709. However, since I believe DGIndex defaults to ITU-R BT.709 in the absence of a flag, I suppose there could be some flagged as such which are really ITU-R BT.601. You must have some other tools available to show you the colorimetry, or lack thereof, where I only use DGIndex. If you use something else as well, please let me know, as you've piqued my interest, and I like to think I'm doing it right in my encoding with CCE. How do you know if the flag hasn't been set?

I'm still not sure that I understand the importance of using TDA for authoring if your source is ITU-R BT.601. Are you saying that other authoring apps are adding the ITU-R BT.709 flag wrongly, or are you saying that with no flag added by other authoring programs, the video may play with those slightly garish colors of that one test screen of ronnylov's?

Also, I was wondering why TDA insists on reencoding the very last part of any video stream. Those people aren't stupid, and there must be a reason. Perhaps it's to close the GOPs so that other tracks can be joined without the artifacts that come from joining open GOP material. The new TDA prides itself in being able to join clips seamlessly, and for that closed GOP is (usually) needed. I encode all mine with open GOP, and even though I wasn't joining anything in my tests, maybe TDA just does it as a matter of course, just in case.

[Incast]

Also, I was wondering why TDA insists on reencoding the very last part of any video stream. Those people aren't stupid, and there must be a reason. Perhaps it's to close the GOPs so that other tracks can be joined without the artifacts that come from joining open GOP material. The new TDA prides itself in being able to join clips seamlessly, and for that closed GOP is (usually) needed. I encode all mine with open GOP, and even though I wasn't joining anything in my tests, maybe TDA just does it as a matter of course, just in case.

That's an interesting theory, and has some merit. Whatever is triggering this is deemed valuable since it's been carried over from DVD Author 2 to DVD Author 3.

On the colorimetry issue: I've tested a number of film DVDs and none have the BT.709 flag. A lot register as BT.709 because there is no flag. The ones which do have flags are all in BT-470-2 (ie BT.601).

The easiest way to test this is using GSpot, open a vob file with it. The colorimetry information is near the top right. If all the boxes stay grey that means there is no colorimetry flag and in DGIndex it will default to BT.701. Any CCE outputs will be in this category.

If all the boxes are black but none are lit, that means it's unspecified. The specification says that - again- it should default to BT.701.

Broadly speaking SD MPEG2 tends to be in BT.601, HD MPEG2 tends to be in BT.701. These colorimetry flags are largely ignored by decoders, in fact I've yet to see them used. But DVD Author 3 has appealed to me, because I'd rather have the SD colorimetry flag in case it will be important in future years when HD becomes more mainstream.

[manono]

Hi-

On the colorimetry issue: I've tested a number of film DVDs and none have the BT.709 flag. A lot register as BT.709 because there is no flag. The ones which do have flags are all in BT-470-2 (ie BT.601).

After testing the 10 or so DVDs and reencoded DVDs on my computer, and checking them in both GSPot and DGIndex, I agree that the flagged ones are ITU-R BT.601 (except mine say SMPTE 170M and not BT.470-2). I have seen quite a few BT.470-2 ones, but in my experience the majority of the ITU-R BT.601 ones say SMPTE 170M in DGIndex and now GSpot.

However, the ones that register in DGIndex as ITU-R BT.709, even though they have no flag, really are ITU-R BT.709, I believe. These are retail DVDs now, not your caps, which, evidently, are some form of ITU-R BT.601. Why do I say that? Even GSpot says this when you hold your mouse over the I709 box:

ITU-R BT.709 (MPEG-2 Default)

MPEG-2 default. Perhaps the reason it's not flagged as ITU-R BT.709 is that it doesn't have to be. It's so common that it's the default. You, with your ITU-R BT.601 caps, might have a problem. I, with my DVDs, don't.

Broadly speaking SD MPEG2 tends to be in BT.601

A little too broad for me, since DVD is included in that category. Are you familiar with the ColorMatrix filter for AviSynth? Here are some quotations from the included doc:

ColorMatrix corrects the colors of MPEG-2 streams of dvds. More correctly, many MPEG-2 streams use slightly different coefficients (called Rec.709) for storing the color information than AviSynth's color conversion routines or the XviD/DivX decoders (called Rec.601) do, with the result that DivX/XviD clips or MPEG-2 clips encoded by TMPGEnc/QuEnc are displayed with slighty off colors (which looks like a small difference in brightness). This can be checked by opening the MPEG-2 stream directly in VDubMod.

This filter recalculates the YUV values (using the default mode = "Rec.709->Rec.601") assuming the coefficients which are used by AviSynth/VDub/DivX/XviD, with the consequence that your final encoding (MPEG-2 or MPEG-4) is displayed correctly. However, you can also use hints = true instead or specifying the d2v-file d2v = filename which does the correction automatically if needed. See Options for more information.

In case you captured something or you have a XviD/DivX (both are encoded Rec.601 coefficients), and you want to encode it to mpeg-2 using CCE (which assumes Rec.709 coefficients), you should use the following script (progressive material)

ColorMatrix(clip, mode="Rec.601->Rec.709")
.
.
.
There are several ways to convert a YUV stream to RGB. The most well known one, uses Rec.601 coefficients. It is for example used in the color conversion routines of AviSynth, VirtualDub and XviD/DivX. When playing back a XviD/DivX the stream is converted to RGB using Rec.601 coefficients. The main issue is that sometimes other coefficients are used for the YUV to RGB conversion (the other two are Rec.709 coefficients and FCC coefficients). A problem arises if a stream is encoded using one set of coefficients (Rec.709 coefficients for many dvd streams for example), and somewhere in the reencoding-processing-playback chain a different set of coefficients is assumed (Rec.601 coefficients for the XviD/DivX decoder or FCC coefficients for TMPGEnc/QuEnc or Rec.709 coefficients for CCE). You will get a slightly color distortion, which looks like a change in brightness (it's not really a change in brightness, the colors are just slightly off).

How do you know what set of coefficients are using when encoding a MPEG-2 stream? Sometimes the coefficients are stored in the header of the MPEG-2 file (the "matrix coefficients" field in the "sequence display extension"). Newer versions of GSpot will be able to read this part of the header, but also DGDecode (with Mpeg2source(info=1)) can be used to view them. If this extension field is not present in the header of the MPEG-2 file, the specs say we are supposed to use the default Rec.709 coefficients (emphasis mine) (0.2126, 0.7152, 0.0722).

I may be interpreting this wrongly, but this tells me several things. One is that if the information isn't in the header, it's almost certainly really ITU-R BT.709 (DVDs now, and not your caps). CCE, while not placing anything in the header, is almost certainly outputting for ITU-R BT.709. If you're reencoding your ITU-R BT.601 caps, you can easily convert to ITU-R BT.709 with a filter (the ColorMatrix filter or others) and end your worry about the problem. If you're not reencoding, and your caps are using ITU-R BT.601 but without being flagged as such, then maybe TDA is useful to you after all (unless or until there's a way to flag it outside of TDA).

The TMPGEnc encoder accepts RGB24 and outputs ITU-R BT.601, but it still doesn't make sense to me that the authoring app is flagging everything it sees, rightly or wrongly, as ITU-R BT.601. That seems to imply that people using encoders other than Tsunami's shouldn't author with TDA. And TMPGEnc's is the only software encoder with which I'm familiar that outputs ITU-R BT.601. Most accept YUY2 and output ITU-R BT.709. The HCEnc accepts YV12 and also outputs ITU-R BT.709. Your thread caught my attention, but this subject is far from my speciality, and TDA isn't my authoring app of choice. I could easily be wrong about some of this, and I'm half expecting jagabo to step in any time now and correct me.

[Incast]

MPEG-2 default. Perhaps the reason it's not flagged as ITU-R BT.709 is that it doesn't have to be. It's so common that it's the default. You, with your ITU-R BT.601 caps, might have a problem. I, with my DVDs, don't.

Or this part of the MPEG2 specification is widely ignored. Your whole argument relies on real world decoders reading the MPEG2 specification to the letter, which is what all the specialist video programs do.

It should be noted that most decoders will default to 709 for HD and 601 for SD, presently they pay little regard to this MPEG2 flag. For example ATI graphics cards decode in BT.601 for SD, and BT.709 for HD, the same is true of FFDshow (in default mode), according to the Doom9 Colormatrix thread.

All SD content is either are lacking a flag or is labelled as BT.601. If you add the DVB cap to this from Doom9 which has no flag, but is BT.601, that appears plenty of evidence to me. SD material appears almost exclusively in BT.601. The question is whether the studio bothers with a colorimetry flag to explicitly label it.

It should strike you as highly dubious that neither of us have come across SD material which actually has an explicit BT.709 flag.

I don't see you agreeing with me on this, so I suggest we leave at that. I'd rather we focused on TMPGEnc DVD Author re-encoding, I'd really like to get to the bottom of what's going on here.