TDA1.5 Dropping Frames on HC encoded M2V?

[gadgetguy]

After reading glowing comments about HCenc I thought I'd give it a try. I was unable to "see" a difference in the video produced from my old standby QuEnc and that by HC, but HC seemed to be doing a better job with the bitrate allocation and the encoding time was faster, so I decided to switch. However when I used TDA1.5 to author, I noticed that the chapter points were not where I set them to be. I went back to the source files and compared what was displayed in TDA and what VirtualDub-MPG2 showed. TDA showed 79 fewer total frames than VDub. On a frame by frame comparison they matched until frame 1808, where the frame types no longer matched up. TDA showed it as a B frame and VDub correctly showed it as a P frame. Close examination shows they are not displaying the same frame (the numbers match but the picture is different) and at that point it looks like TDA dropped a frame. I continued frame by frame for a while but I ran out of patience and didn't find another "drop" point, but I assume it must have happened another 78 times to account for the difference in framecount.

If I set the chapter points as near to the VDub frame number as possible and ignore what I see in the display, I can get very close to where I want them to be but since chapter points can only be on I frames and the I frames don't line up it's only "close". It does appear as though the final authored video contains all of the frames (same running time as VDub shows) and I don't experience synch issues.

Can anyone else duplicate my findings? I never had this problem with QuEnc, and the only difference is that with QuEnc I encode to an mpg (video and audio muxed) and of course HCenc encodes to a M2V (video only), audio encoded separately (I still use QuEnc for audio). At this point the time savings in encoding are wasted setting chapter points so I'm thinking I'll just go back to QuEnc.

[gadgetguy]

I'm still having this issue although the frame number that I find the change is different for each file. Has anyone else noticed chapter points not being in the correct location when authoring a HC encoded file?

[FulciLives]

I have been using HCenc here and there (even though I am now a long time CCE user). I've been using it because I want to write some guides and figured if I can do that with a freeware MPEG encoder then all the better since anyone can follow the guide and not need this costly MPEG encoder or that costly MPEG encoder etc.

In my testing of HCenc I have discovered (confirmed by others) that 23.976fps encoded with 3:2 pulldown is not being done correctly. It seems to have something to do with the way the 3:2 pulldown is being done.

So as a work-a-round I still use the built-in 3:2 option but then run the resulting M2V through DGPulldown to "redo" the 3:2 pulldown (i.e., the 23.976fps ---> 29.970fps pulldown).

So far that seems to be working A-OK for me although come to think of it since that appears to be needed then I guess the 3:2 option in HCenc isn't even needed which means it should just output progressive 23.976fps which can then be run through DGPulldown.

However it doesn't end there ... I have discovered another oddity that has to do more with TMPGEnc DVD Author 2.0 (I just recently upgraded) than with HCenc.

Like I said this has nothing to do with HCenc as I've tested with CCE and the same thing happened ...

When doing PAL to NTSC the DGPulldown method (720x576 25fps resized to 720x480 but encoded at 25fps then DGPulldown for 25fps ---> 29.970fps pulldown) I have noticed that TMPGEnc DVD Author 2.0 will inexplicably "ruin" the last few frames of the video. By "ruin" I mean that some interlacing can be seen! This appears to affect the last couple of seconds.

I never noticed it with the old TMPGEnc DVD Author but all my PAL to NTSC stuff have been full blown movies where the last few frames are usually the credits and it would be easy not to spot it there especially since the last second or two tend to be just black or a still image etc.

I was doing a PAL to NTSC "shootout" with deadrats (talk about a loser) and the short clip I used ended with "action" and when I authored it with TMPGEnc DVD Author 2.0 ... well ... that is when I discovered the interlaced frames at the end of the clip.

As a work-a-round I have been adding 4 seconds (to be safe) of soundless black video to the end of my PAL to NTSC conversions (easy with AviSynth) since I like to use TMPGEnc DVD Author 2.0 and I figure who cares if there are some interlacing artifacts in the last couple of seconds when it is just plain black video for a "padding" if you will ... you know?

However this points to a "problem" or "issue" with the way TMPGEnc DVD Author 2.0 is handling certain types of videos.

I should point out that the sample video that was doing this oddness in TMPGEnc DVD Author did so when the clip was encoded with HCenc or with CCE so that is why I saw it is not a HCenc issue. Also the files (be it the CCE file or the HCenc file) have no such issues when authored with Muxman or DVDauthorGUI so that rules out DGPulldown being a problem. Again the files are fine until you auther with TMPGEnc DVD Author 2.0

In your particular situation I am guessing it was a 23.976fps source with 3:2 pulldown and that running the M2V through DGPulldown to redo the 3:2 pulldown is all you need to do.

But I also wanted to bring up the issue with TMPGEnc DVD Author so as to let you know what else might be at play here.

Please update this with more information.

- John "FulciLives" Coleman

[gadgetguy]

The source was DV-AVI 29.97 fps 720x480 Interlaced video captured using Pass-Through on my Sony DCR-TRV230. No De-interlacing, inverse telecine, nor pulldown was involved. The same avisynth script was used to feed to HCenc and QuEnc using similar settings. I can accept that the problem may be in TDA and how it "reads" the files, but I don't understand why it would have a problem with the HC encoded file, but not the QuEnc encoded file.

The HCenc encoding parameters were:

C:
CD Program Files\Quenc
Rem Audio
Quenc -i "G:\Video\Video.avs" -o "E:\Video.ac3" -b 6587 -maxbitrate 9000 -dc 10 -priority 3 -2 -hq -vbr -noscene -notrell -cgop -interlaced -noextreme -gopsize 12 -maxbframes 2 -nocmatrix -aspectratio 4:3 -mpeg2mux novideo -audiocodec AC3 -audiobitrate 384 -bff -auto -close
cd..
cd HC
Rem Video
HCenc_019 -i "G:\Video\Video.avs" -o "E:\Video.m2v" -ini "G:\Video\Video.ini"

With ini file

*LOGFILE E:\Video\HC.log
*BITRATE 6587
*MAXBITRATE 9000
*PROFILE best
*ASPECT 4:3
*GOP 12 2
*DC_PREC 10
*INTERLACED
*BFF
*NOSCD
*CLOSEDGOPS
*MATRIX mpeg
*CHAPTER 1
8932

The QuEnc encoding parameters were:

C:
CD Program Files\Quenc
Quenc -i "G:\Video\Video.avs" -o "E:\Video.mpg" -b 6587 -maxbitrate 9000 -dc 10 -priority 3 -2 -mpeg2mux -hq -vbr -noscene -notrell -cgop -interlaced -noextreme -gopsize 12 -maxbframes 2 -nocmatrix -aspectratio 4:3 -mpeg2mux dvd -audiocodec AC3 -audiobitrate 384 -bff -auto -close

[FulciLives]

Well I use the HCenc GUI when I use it but your setting seem to be OK to me. To be honest though I've only used HCenc so far with progressive DVD Video rips or progressive MPEG-4 AVI clips.

Well OK looking at the settings again I see you used 12-2 for the GOP settings. For 29.970fps NTSC you should use 15-2 instead but I doubt that is causing the issue you are having.

I don't understand some of the other settings like I have no idea what -noextreme means. Also the muxing stuff and the AC-3 audio stuff makes no sense to me either since HCenc doesn't do audio so ... ???

I also see -auto in there and I know the GUI has an auto GOP setting that doesn't work as I would like it. I use the manual setting and again for 29.970fps I use 15-2 whereas for 23.976fps I use 12-2

I've heard that trying to force "I" frames in the encode for pre-selected chapter points is a bad thing. For instance manono pointed out that Criterion does this and it has caused issues with interlacing frames around chapter points in otherwise progressive video.

My suggestion? Try using the GUI instead (along with your AviSynth AVS file) instead of that command line ini stuff and don't pre-set any chapters. At the very least try it again without any pre-set chapters.

If you still think something is going wrong after importing into TDA then I suggest ... if nothing more than a test ... to author with something else like MuxMan or DVDautherGUI or DVD-Lab etc. and see if the problem is still there or not.

- John "FulciLives" Coleman

[Alex_ander]

What do MPEG editors say about HC output at analyzing frames to be re-encoded?

[FulciLives]

What do MPEG editors say about HC output at analyzing frames to be re-encoded?

What ???

- John "FulciLives" Coleman

[gadgetguy]

I prefer the 12-2 GOP sizing and it's easily within spec. The -noextreme is a QuEnc setting which for the HC batch file has no effect because I'm only encoding the Audio there, because as you said HC doesn't do audio.

My suggestion? Try using the GUI instead (along with your AviSynth AVS file) instead of that command line ini stuff and don't pre-set any chapters. At the very least try it again without any pre-set chapters

That would defeat two-thirds of the reason for switching to HCenc.
I have tried without forcing a Chapter Point but the problem still occurs. I originally used the GUI but had switched to the INI method by the time I noticed the problem, I did go back and check the file that I had encoded using the GUI method and the problem was there, and no chapter point had been designated. The file looks fine in Vdub-MPeg2. Only TDA sees it differently.

Edit: I can't think about this any more tonight, I have to go to bed. Maybe it will make more sense tomorrow.

[FulciLives]

I have tried without forcing a Chapter Point but the problem still occurs. I originally used the GUI but had switched to the INI method by the time I noticed the problem, I did go back and check the file that I had encoded using the GUI method and the problem was there, and no chapter point had been designated. The file looks fine in Vdub-MPeg2. Only TDA sees it differently.

I'm sorry but I am slightly "lost" or confused if you will.

The output from HCenc is screwed up as soon as it comes out of HCenc or not until you import it into TDA?

I should point out that the person behind HCenc lives in PAL land and has admitted to ... how to say ... not being the best at NTSC ... hence the wrong NTSC GOP settings when in autoGOP mode and the incorrect 3:2 pulldown issue (both of which are to be fixed at some point).

I thought those were the only issues (with very easy work-a-rounds) but now this ... damn I don't know what to think now.

I have found HCenc to be as good as CCE and perhaps even better when the AVG bitrate is a bit on the "low" side.

So I find this all very troubling (i.e., the issues you are having).

Anyways I bring up the programmer's lack of NTSC knowledge only to suggest that perhaps the program is doing something wrong with 29.970fps NTSC but like I said 23.976fps seems OK only the 3-2 pulldown is "broken" but 23.976fps is progressive. Maybe the programming is missing something that is needed for true interlaced 29.970fps but ... odd that there is no problem until imported into TDA (again if I am understanding you correctly on that point).

I don't have a digital cam of any sort ... how about uploading a DV AVI file from a camcorder so I have some true 29.970fps interlaced NTSC to work with? You can post a single file up to 100MB for free on rapidshare.com

http://www.rapidshare.com/

That way it gives me something to work with and do some tests myself. I realize 100MB (or just under) of a DV AVI file will probably only be a few seconds long or so but it's better than nothing.

- John "FulciLives" Coleman

[Alex_ander]

What do MPEG editors say about HC output at analyzing frames to be re-encoded?

What ???

- John "FulciLives" Coleman

When you use an MPEG editor (like TMPG, Womble) for recreating an MPEG2, you have pre-information on what parts/ranges it will correct by re-encoding. TMPG is the most sensitive to problems in an MPEG, Womble is the most tolerant. VideoReDo shows info in the end. If there's something wrong with HC, there's a chance to get new info that way or to fix the problem if just small parts are shown to be re-encoded.

[gadgetguy]

I tried to make it as clear as I could in my first post, but I'll try again.
The output from HCenc seems to be fine. I can open the m2v in Vdub-mpeg2 and a frame by frame analysis shows it to look exactly as I expect.

It is only when I import the file into TDA and am setting chapter points that there is a problem. It appears that there are frames missing when scrolling to decide where to set the chapter points. If I step through frame by frame in TDA and compare frame by frame in Vdub-mpeg2, eventually they do not match. My first test I had to look through over 8000 frames to locate the "missing" frame, but subsequent tests have had varied amounts.

There does not seem to be any pattern to which frames go "missing" and it doesn't happen alot (79 frames missing from a file with 78368 total frames), but it's enough to throw things off. A 100MB DV-AVI sample would be too small to be of any use here because of the infrequency of the occurance. Once a frame "goes missing" TDA does not report the same frame number or type (I, P, B) as Vdub-mpeg2 for the same frame (compared visually).

The final authored DVD has all of the frames, identical to the source m2v but the chapter points are in the wrong place because of the apparent missing frames while selecting the chapter location. In other words, what I see on the screen when selecting the chapter point is not where the chapter point is actually set.

I started to try using a different authoring program (guiforDVDAuthor), but TDA is the only one I've ever used. I haven't had time to get past the learning curve to be able to test results.

As I said in my first post I have found a workaround by ignoring what I see on the screen. It's not exact, but so far has been close enough to work. I'm perfectly willing to concede that the problem may be in TDA, but HCenc is the new ingredient in a process that has been working for me for a couple years, so it's my prime suspect right now.

[FulciLives]

@gadgetguy

OK I think I understand now what it is you are saying.

Very odd behavior.

I normally set auto chapters with TMPGEnc DVD Author but I'll see if the same thing happens to me when I try to set them manually.

Kinda busy now (at work) and with the Holiday and all ... may be a few days until I can test.

- John "FulciLives" Coleman

[gadgetguy]

... Very odd behavior.

Indeed.

... Kinda busy now (at work) and with the Holiday and all ... may be a few days until I can test.

- John "FulciLives" Coleman

I understand. I'm having trouble finding time to work on this myself. Thanks for sticking with me here.

[manono]

My guess is that it's a framerate issue, specifically 30->29.97fps. Maybe TDA is expecting to see 30fps video, perhaps based on some TDA setting of yours.

The reason I suggest this is:

79 frames missing from a file with 78368 total frames

30/29.97=1.001
78368/1.001=78289.7
78368-78289.7=78.3

The final authored DVD has all of the frames, identical to the source m2v but the chapter points are in the wrong place because of the apparent missing frames while selecting the chapter location.

I don't use TDA, so I don't know what it's doing, but I don't believe this to be an HC issue at all. Have you tried muxing using Muxman and a Celltimes.txt file for the chapters (it takes them by 29.97fps framecount)? I'll bet you won't "lose" any frames doing that, and won't have the chapters any farther off than the nearest I-Frame. You could make a Muxman authored DVD of it, and then use it to replace the TDA authored video in VobBlanker, thus keeping the menu and whatever else you created in TDA, while at the same time getting the better chapter placement. Kind of a lousy workaround, but it's just as an experiment.

[gadgetguy]

My guess is that it's a framerate issue, specifically 30->29.97fps. Maybe TDA is expecting to see 30fps video, perhaps based on some TDA setting of yours.

The reason I suggest this is:

79 frames missing from a file with 78368 total frames

30/29.97=1.001
78368/1.001=78289.7
78368-78289.7=78.3

You may be on to something here and I'll have to try some more tests with this in mind to see if I can prove or disprove a connection. The only confusing part is that I would think it would be more consistant about where the "missing" frames are located. But the math is certainly compelling and worth investigating.

The final authored DVD has all of the frames, identical to the source m2v but the chapter points are in the wrong place because of the apparent missing frames while selecting the chapter location.

I don't use TDA, so I don't know what it's doing, but I don't believe this to be an HC issue at all. Have you tried muxing using Muxman and a Celltimes.txt file for the chapters (it takes them by 29.97fps framecount)? I'll bet you won't "lose" any frames doing that, and won't have the chapters any farther off than the nearest I-Frame. You could make a Muxman authored DVD of it, and then use it to replace the TDA authored video in VobBlanker, thus keeping the menu and whatever else you created in TDA, while at the same time getting the better chapter placement. Kind of a lousy workaround, but it's just as an experiment.

I've never used Muxman, but that workaround doesn't really sound any different than the workaround I'm using now, but a lot more complicated. I know I can specify chapter points and have them authored by TDA to the nearest I frame as long as I ignore what I see on the screen, which is what the celltimes.txt file would be doing using Muxman, right? The issue is what I see on the screen when searching for points to set chapters.

I'm beginning to believe it is a TDA issue and not an HCenc issue, but I have to figure out why it has only raised it's ugly head after switching to HC.

[manono]

Hi-

I know I can specify chapter points and have them authored by TDA to the nearest I frame as long as I ignore what I see on the screen, which is what the celltimes.txt file would be doing using Muxman, right?

As I understand it, what's happening to you is that the chapter points are OK at the beginning, but get farther and farther off the more into the video you are, and might be off by more than 2.5 seconds or so (2.5 seconds=75 frames) by the end. When choosing the chapter points in TMPGEnc, unless you compensate for what you're being shown, you're doomed to have the wrong chapter points (for some reason not understood yet). However, if you open the AviSynth script or the M2V in VDubMod, you'll choose the chapter points at the frame numbers you want. You stick them into the celltimes.txt for use in Muxman. The free version won't create any menus for you (the Pro version does, I think), but it should give you the chapters where you want them.

I don't know if you're married to TDA or not, but there are other authoring apps around. Or you might bring up the issue with Tsunami tech support. If this is a sometime problem with TDA, I don't think this'll be the first time they've heard of it.

And you are sure the script is feeding 29.97fps into the HCEnc, aren't you? You can confirm by opening the script in VDubMod and going File->File Information.

[gadgetguy]

It's not a question of being married to TDA, it's about my previously working method became broken when I switched to HCenc.

I went back and analyzed an old QuEnc encoded mpg and discovered that it was not encoded as I specified or expected. It has a GOP length of 10 even though I specified 12. I did an experiment and specified 12 again to verify, and it encoded 10 again. I repeated the experiment and specified a GOP length of 15 and the resulting encode has a GOP length of 13. I then specified a GOP of 17 expecting to get a GOP length of 15, but ended up with 16. I pulled all of these into TDA. In comparing what was in TDA to the source MPGs the first thing I saw was that in TDA all of the files showed fewer frames than the originals. Comparing the ends of the files revealed that TDA discards the final GOP. And in at least one case, QuEnc did not adhere to the (wrong) GOP length. The one that I specified 15 had 13 frame GOPs until the last GOP which had 14 frames. In the other 2 files the final GOP was shorter than the rest.
In all three cases the difference in total frame numbers between TDA and the source MPG equals the length of the last GOP. So even though the QuEnc encoded files are not being encoded with the correct GOP length, they align properly and the chapter points go where designated.

I'm going to do more experimenting with different GOP lengths in HCenc and see what results I get.

[FulciLives]

In comparing what was in TDA to the source MPGs the first thing I saw was that in TDA all of the files showed fewer frames than the originals. Comparing the ends of the files revealed that TDA discards the final GOP.

Could this be why my progressive PAL to NTSC MPEG-2 had interlaced frames in the last second or so ... because TDA did something weird to the last GOP?

I've never noticed it before though ... and I've never had chapter issues like you said ... weird man ... very weird.

- John "FulciLives" Coleman

[gadgetguy]

Could this be why my progressive PAL to NTSC MPEG-2 had interlaced frames in the last second or so ... because TDA did something weird to the last GOP?

I'm not sure, but in my case the frames are only dropped when viewing to set chapter points. The final authored VOBs include all of the frames of the original.

I've never noticed it before though ... and I've never had chapter issues like you said ... weird man ... very weird.

- John "FulciLives" Coleman

It gets weirder. I encoded three test files with HCenc using the same AVS that I used to make the QuEnc test files. One with a Gop length of 12 and 1 forced chapter point, one with a GOP length of 12 with no forced chapter point, the third with a GOP length of 15 and no forced chapter point. The HCenc encoded files are 1 frame longer than the QuEnc encoded files and according to the Trim command in the AVS, the HCenc files are the correct length.
The last GOP in TDA matches the last GOP in the m2v, but the total frames shows 12 frames fewer than the original m2v in all three files, so it appears GOP length or forced chapter points have no effect. I've discovered that when scrolling through frame by frame, if I go backwards and forwards over several frames the frame type changes. For example if a frame shows as an "I" frame, if I scroll forward and backward over several frames and come back to that same frame, it now shows as a "B" frame. So the problem seems to be with the seek function and it would appear that QuEnc encoded files seek fine and HCenc encoded files don't.

I'm not sure what this means yet, but I'm too tired to think about it any more tonight. I'm going to bed.

[gadgetguy]

I tried pulling the HC encoded files into Ulead VS6 and it shows the same "lost frame" behavior as in TDA although it doesn't show frame types so I can't compare that way. It just reports fewer frames than there really are, and interestingly, a different figure than TDA reports. The QuEnc encoded files report the right amount of frames in VS6.
At this point I'm not sure what the problem actually is. Does anyone know what would cause this behavior?

[gadgetguy]

Good news. Problem resolved with HCenc v0.20.