Converting ABC TV HD Programs to DVD

[jccruit]

I am having a problem converting ABC TV HD programs to DVD. Have used both ConvertXtoDVD and SVCD2DVD with both .ts and .mpg input. The resulting output skips noticibly although output from SVCD2DVD seems to skip a little less. Anyone else having this problem and if so have you found a program or procedure that will correct the problem? I do not have problems converting programs from other networks.

[manono]

Hi-

A friend of mine was also having trouble with the Hi-Def ABC broadcasts and sent me some samples to go over. It turns out that they're changing the "base" framerate by dropping frames. They probably do this to shorten the movie or show to fit into a given timeslot, or to fit in more advertisements. So, what should have a base framerate of 23.976fps (the samples I saw were shot on film) becomes 25fps or so, varying with the show. This means that if encoded for 23.976fps, one frame or so is dropped every second, and during certain kinds of movement this jerky playback will become quite obvious.

Now, I don't use ConvertXToDVD or SVCD2DVD, and I don't know how they're going from 59.94fps to 29.97fps. But if they're just dropping every other frame, they're still going to get tripped up, but not as frequently, because even if doing that and encoding for 29.97fps, there are regular instances where unique frames follow each other (because an adjacent frame was removed), and if one of them is dropped, that'll result in a jerk or stutter.

The best way to fix this is to find out the pattern, which repeats every couple of hundred frames, remove the duplicate frames, encode the video with the dupe frames removed, and then use DGPulldown for the correct framerate->29.97fps. For this you'll need AviSynth. I realize this isn't what you wanted to hear, But it's either that or put up with jerky stuttering playback (if you want them on DVD). I suppose you could toss out half the fields pretty easily and reinterlace it to get smooth playback (also using AviSynth), but that's not the way I'd do it.

If you posted a sample of 1000 frames or so from the source, of a section with steady movement, there are probably some people around here that might want to have a look.

[jccruit]

Thank you for the info manono. I'm afraid I do not have a sample at hand because I got frustrated and deleted the file. Will capture another program and see about uploading.

[edDV]

ABC/ESPN and FOX* use 720p at 59.94 frames per second.

Film is not sent with telecine as is done for 480i or 1080i. Instead full frames are repeated in a 3 then 2 pattern to build 23.976 fps up to 59.94 fps. If you linearly decimate every other frame the result will be jerky. You need to respect the 3-2 pattern. Then you can extract the exact 23.976 progressive frame sequence.

see
https://forum.videohelp.com/topic313258.html


* also National Geographic HD, A&E HD and some local PBS stations. These need to be format tested with GSpot.

[manono]

Film is not sent with telecine as is done for 480i or 1080i. Instead full frames are repeated in a 3 then 2 pattern to build 23.976 fps up to 59.94 fps. If you linearly decimate every other frame the result will be jerky. You need to respect the 3-2 pattern. Then you can extract the exact 23.976 progressive frame sequence.

I was involved in that thread to which you linked which is now 9 months old. A lot has changed. jccruit has noticed, and I'm confirming, that it's no longer 23.976fps. At least some ABC shows aren't. They're more like 25fps because frames are being removed before broadcast. The effect is to make the show play more quickly. Where before you could remove every other frame to get something that played reasonably smoothly at 29.97fps because at least no unique frames were being removed (yes, many dupes were being kept), by doing that now you remove a lot of unique frames, making the output play much more jerky. Missing frames plays much worse than does duplicate frames. There is no longer a 3 2 3 2 pattern. If I remember correctly, it sometimes goes 3 2 3 2 3 2 3 2 2 and sometimes 3 2 3 2 3 2 3 2 1 1. Something like that. I forget exactly, because it wasn't my cap. Those are the "mini-cycles". With another sample I could figure it out again. The "grand-cycle" was over 200 frames before it repeated. If you do a standard IVTC on it to 23.976fps, you'll get very jerky playback. If you remove every other frame, you'll get less jerky, but still noticeable, stuttering playback.

As I said, I don't use the 2 programs jccruit mentioned, but if they're removing every other frame as an easy way to go from 59.94fps to 29.97fps, the result won't be good, not because of the dupe frames still remaining, but because of the unique frames being removed. I could go back to my friend to get the names of the shows where this happens, what the cycle is like, and the script we came up with to deal with it, if you'd like to know, edDV, in case you'd like to check it out yourself.

[edDV]

Film is not sent with telecine as is done for 480i or 1080i. Instead full frames are repeated in a 3 then 2 pattern to build 23.976 fps up to 59.94 fps. If you linearly decimate every other frame the result will be jerky. You need to respect the 3-2 pattern. Then you can extract the exact 23.976 progressive frame sequence.

I was involved in that thread to which you linked which is now 9 months old. A lot has changed. jccruit has noticed, and I'm confirming, that it's no longer 23.976fps. At least some ABC shows aren't. They're more like 25fps because frames are being removed before broadcast. The effect is to make the show play more quickly. Where before you could remove every other frame to get something that played reasonably smoothly at 29.97fps ...

Interesting. My experience was with last year's "Lost" and "Boston Legal". The 32323232 pattern is the textbook answer to film on 720p/59.94 but maybe they are playing games with the pattern to frustrate recording at full quality. Maybe I'm just paranoid.

Local stations often butcher movies and film based series to fit time slots and/or expand commercial minutes. At least in the past, major networks sent clean video.

[Soopafresh]

Interesting indeed. I think DGIndex 1.49 has resolved this, although I haven't tested it.


http://forum.doom9.org/showthread.php?p=957879

[vhelp]

Hi guys. I've been busy away from my usual (video/imaging) things, and
spent most of it in OTA headackes, too.

So in many of my drulling dealings with this source medium, I found things out
that seems to be becoming more common as OTA takes the new position in
this digital age. Now I've been trying to find the standards of consistancy in
this area, but there just doesn't seem to be any definative lock, unfortunately.
fwiw, I guess some things will never change in this field.

Some of the things I was able to notice regulary were:

A - More inconsistant Telecine patterns, as if breaking away from tradition.
B - Poor signal strength or OTA interuption, causing MPEG artifacts/broken telecines, etc.
C - Time De/Expansion for the given broadcast time slot

You see, I thought that with HD, things were going to change for the better.
I am sorry to say, that that *ain't* gonna happen. In fact, I would say that
it is worse (dependability'wise) than Standard Analog sources, though most of
it can be blamed on signal strength behavior.

So, for reasons written below, we can't depend on Clean and Consistant Telecine
processes, etc. for this new (HD) format, ever.

For C-- I'm sure there is a made-up terminology/phrase for it, but it escapes me.

We learned that this is where they re-telecine the video program to fit within a
certain time slot. Sometimes, a (film) movie source is a little LESS or GREATER
than, say, 2:00 hrs. So, they either Speed up or Slow down the video, though
just enought for the time slot. This is probably done using the sources original
24p or else its already-based-telecine and its fields manipulated through Field-Blends
and other mysterious [s:3ab9009c37]nonsense[/s:3ab9009c37] technique or stratigies.

I would theorize (cause I'm not in the field) that somewhere out there, is an original
(new) HD medium already run through this process and is shared with all the entities
that come into possesion of such video programs.

So in the end, given C--, it is possible that when these video programs are run
through this process for time expansion (or whatever you want to officially call it)
there may be the possibility of not being able to restore (by locking onto the sources'
original telecine pattern [decimating and/or de-blending, etc]) and undoing the process
there-by restoring to 24p video.. pfew! Or, there is a sligght chance that even afer
run though this process, that the video source is Edited again.

Anything that is put through a process should be (in theory at least) a way to do
the reverse. If not, then the source is tainted in some way or another.

I've had a few of these, captured (recorded) through my Pinnacle PCTV HD pro stick
usb ota tuner card. I mostly get the ABC station, too.

But, when I get stuck with this (IVTC) problem, if the source is Time Expanstioned,
(after determining that it was) then its either too elaborate or else I have to assume
that the recording was marred on account of intermittant signal or interuption or some
other interfence that breaks up the video's transmision. (when decoded, the video may
no longer have certain MPEG attributes (ie, flags, etc) and the decoder may not decode
the source accurately or properly) I find this a lot in my OTA recordings when the signal
is weak or is showing signs thereof. Sometimes, the BFF or TFF flags get mangled, mid-stream.
And that can Fale or Confuse an IVTC process.

I really think we need to let people know that signal strength is another factor to
cause problems in our HD video processes. And especially for IVTC processes, this is
an area easily effected but not realized. And knowing this, could help more of us
fiddle less in this area. It could save us a lot of greaf, among other things. But right
now, I feel that the majority of ivtc (failed) processes applied in HD video sources is
to blamed mostly on this signal strength issue. You see, even when the signal is at
90% or 100%, it doesn't mean that the video stream will be perfect. There will no
doubt we tiny glitches seapping through, and consiquentially effecting our ivtc processes,
and even in these small (unnoticed) amounts, can confused and confound us to no
avail in our ivtc processes. I've seen where I could sware I had an Time Expansion
source, and counted forever all those assumed patterns, but never really locking
onto something cleanly. So my theory goes with signal strength interuption, which
can be anything. I would also theories that if the video source could be determined
accurately that it is of time expansion nature, then there should be a way to come up
with a reverse of this with perfect accuracy. Again, theory of course. But if you can't
fine a pattern easily (and it should be) then you have a sick video source.

In the end, after all else has (or seems to have) failed, we have to make a comprimise
and go with a workable solution. That usually intails keeping the source all Interlace
or de-interlace it while keeping it 29.970 fps etc. I find myself going this later route
in some of my source dealings, thought with pretty good results. I'm not ready for
59.94 fps transfer just yet. DVD is more than fine for OTA recordings I make on my
misly home computer. If I really wanted absolute quality, I would keep the recording
as is and not mess with IVTC or other conversion tactics when it all boils down to
Transfer vs. Conversion, in terms of HD dvd over Standard DVD. I'm saying that
the time is not yet there -- at least not for me

Well, that's all I have to say.. I'm too tired.

-vhelp 4389

[vhelp]

fwiw, I just realized, that I still have a bunch of ABC 's "LOST" on my HDD.
I did OTA record them when my Hauppauge HVR-1600 was working. Now,
its dead, and I could care less. My Pinnacle is (IMHO) far better than the
HVR-1600 is, minus the QAM support Anyway. I still have a bunch of
LOST episodes in HD on my HDD if anyone wants to use that as a gauge
or other factor. I haven't gotten around to finding the time to do anything
with them just yet. I might just dump them onto a bunch of DVD's as
archives. I don't know.

LOST-- I do recall this show being a headacke. It does not have a consistant
telecine pattern.

Boston Legal-- The last time I deal with this show, it had consistant teleince
pattern to lock onto. I had no trouble with it.

Here's an idea. Those of us who have HD tuner cards, record some HD
contents tomorrow, and we all compare notes, in terms of telecine consistancy,
etc. I think we all can learn from this and help each other out. If we can
determine the consistancy of various programs, (to work accurately) then we
would have a guage to go by and move further with success.

Also, another fwiw, when I want to attempt an IVTC in the hd conversion,
I use this AVIsynth script to do the process, and it always works perfect
unless the signal glitched un-noticed.

--> separateFields().selectEvery( 4, 0,3 ).selectEvery( 5, 0,1,2,3 ).weave()

-vhelp 4390

[Soopafresh]

I usually do:

SelectEven().Tdecimate()

[manono]

http://forum.doom9.org/showthread.php?p=957879

I remember that thread. 1080i stuff is harder than the ABC 720p stuff, because you're working with fields. But it's the same idea, stretching or shrinking the length of the program. And the solution is to do a lot of frame (or field) counting to figure the cycle and how many dupes have to be removed in that cycle, so you can figure the actual framerate. For AVI, you just encode to that framerate, and for DVD you use DGPulldown afterwards to go from the true framerate to 29.97fps. I'll see if I can get a sample.

And, yes, one of the keys was to not fix the D2V, as things were unfixable if we did. I have no idea why.

[manono]

I've got a couple of samples here:

http://www.mediafire.com/?by5nykmzcrn

It's 45.4 MB and from The Knights Of Prosperity.

http://www.mediafire.com/?4iecwu9mjgc

67 MB. My friend forgot which program this came from.

He mentioned new ABC shows specifically as having the problem; Masters Of Science Fiction, The Nine, and Knights Of Prosperity. He says he thinks, but doesn't know for sure yet, that when the big shows return, such as Lost and Grey's Anatomy, they'll be doing the same thing. Open them in something where you can advance a frame at a time (VDub(Mod) is good). Start counting 1 2 3 1 2 1 2 3 1 2, and see how long it is before the cadence breaks. Then try and figure the pattern. And once you've done that, try and figure how to return it to the original framerate.

...but maybe they are playing games with the pattern to frustrate recording at full quality. Maybe I'm just paranoid.

You give them too much credit. It's all about the money. Without knowing the full length of these shows, I figured that by speeding them up they've cut a full minute from the running time. That's worth what - a hundred thou or so of increased ad revenue?

[edDV]

Well for 720p/59.94 the "text book" 32323232 frame sequence needs to be separated as an issue from speed up or capture frustration cadence manipulations. I'm sure adaptive software could be developed to detect the frame changes and restore the original 23.976 film frames.

It was interesting in Soopafresh's link
http://forum.doom9.org/showthread.php?p=957879

that they needed to find 30sec of continuous motion (i.e. the dog's tail wagging) to figure out the cadence sequence by manual technique.

[manono]

I'm sure adaptive software could be developed to detect the frame changes and restore the original 23.976 film frames.

Are you talking about restoring the removed duplicate frames to also restore the 3 2 3 2 pattern? Offhand, I have no idea how to do that, and that might be a subject worth pursuing.

As for needing 30seconds of motion, yes, to do all the framecounting necessary, first to find the pattern and then to confirm what you've found, a long sequence of steady movement is necessary.

[edDV]

I'm sure adaptive software could be developed to detect the frame changes and restore the original 23.976 film frames.

Are you talking about restoring the removed duplicate frames to also restore the 3 2 3 2 pattern? Offhand, I have no idea how to do that, and that might be a subject worth pursuing.

I'd spec the project as adaptive cadence detection that changes at edits, odd speed changes, commercials or program changes. The goal would be 24p with reasonably quick reaction to cadence change with secondary strategy to bridge transitions (i.e. smooth transition jerkiness while calculating a strategy).

Real time this would take massive frame delay but offline software doesn't have that constraint.

[Soopafresh]

23.984fps using manono's example "The Knights Of Prosperity", with selecteven() and tdecimate(). Almost, but no cigar.