M2TS to MPEG-2 1080i

[dellsam34]

I cannot make a commend or a script I'm just not capable of doing, I'm just trying what you guys provide, So I can use dgpulldown to make a sample file, That's about all what I can do.
Pandy your script is easy to use but I couldn't get it record above 22Mbps, If you have another mod that can process 25Mbps and up I will be more than happy to try it, Also I noticed with ffmpeg commend line processing speed is way faster it's like 1.5X.
I will make a sample with the avsforum guy commend after doing the pulldown with dgpulldown and report back if any problems arise.
Oh, One more thing, how do I get all this to work under windows XP, I got tired of converting on Win7 and reboot to XP to do the D-VHS recording.

[dellsam34]

Ok, dgpulldown is out of questions, too many conversions involved, Any other tools?

[poisondeathray]

I cannot make a commend or a script I'm just not capable of doing, I'm just trying what you guys provide, So I can use dgpulldown to make a sample file, That's about all what I can do.
Pandy your script is easy to use but I couldn't get it record above 22Mbps, If you have another mod that can process 25Mbps and up I will be more than happy to try it, Also I noticed with ffmpeg commend line processing speed is way faster it's like 1.5X.
I will make a sample with the avsforum guy commend after doing the pulldown with dgpulldown and report back if any problems arise.
Oh, One more thing, how do I get all this to work under windows XP, I got tired of converting on Win7 and reboot to XP to do the D-VHS recording.

the simple ffmpeg command line posted earlier , if you mean this one

ffmpeg -i Terminator.m2ts -c:v mpeg2video -r 30/1.001 -b:v 22000000 -bufsize 9781248 -maxrate 23000000 -acodec ac3 -muxrate 25000000 terminator.ts

Doesn't have post processing like pandy's script . The more processing, the slower it will be.

If you're using dgpulldown, it has to be encoded progressively at 23.976 , not 29.97

Ok, dgpulldown is out of questions, too many conversions involved, Any other tools?

dgpulldown doesn't re-encode, or did you mean just too many steps, and stuff to do ? Or use another encoder that can insert soft pulldown flags on the fly eg. hcenc

Note that encoding 25% fewer frames also will be faster. E.g if normal movie with hard telecine is 250,000 frames, soft version is only 200,000 frames .

The quality isn't going to be 25% better , long gop encoding doesn't work like that; duplicates don't "cost" very much .

If it's too much trouble, just carry on with what you're doing. Most people would actually do a multipass encode too . With mpeg2 it's typically better. But you have to decide on what types of tradeoffs you're willing to make.

[dellsam34]

For the dgpulldown, yes I mean too many steps, That's why I initially was looking for a one step GUI program but I know now that doesn't exist for what I'm trying to accomplish, Although VideoRedo S5 is full featured program only if I know how to use it.

So will this soft pulldown converts to interlaced or it keeps the stream progressive? I'm still confused about if D-VHS can handle progressive 1080 or just interlaced. If it can handle 30p why it didn't work with native frame rate 24p?

[pandy]

For the dgpulldown, yes I mean too many steps, That's why I initially was looking for a one step GUI program but I know now that doesn't exist for what I'm trying to accomplish, Although VideoRedo S5 is full featured program only if I know how to use it.

So will this soft pulldown converts to interlaced or it keeps the stream progressive? I'm still confused about if D-VHS can handle progressive 1080 or just interlaced. If it can handle 30p why it didn't work with native frame rate 24p?

Soft telecine a.k.a. pulldown is setting special bit in MPEG-2 header - "repeat last field" - so MPEG-2 decoder interpret such MPEG-2 syntax as physical repeating last field located in buffer. Thus no additional data are required to output field.
Well, i', confused also, one you says everything is perfect except bitrate and in another message you wrote that everything was frozen.
As I have no access to D-VHS deck you must be responsible for systematic approach to tests.

i can try to create script that will do soft telecine with dgpulldown and combine both ES to target TS but such script will assume that your source is progressive, 23.976 - ffmpeg doesn't provide (i'm not aware of such functionality) conditional conversion (depends on progressive vs interlace, 23.976 vs 29.97 etc). You need to use proper script - this is your responsibility.
And yes, MPEG-2 encoding i use is slightly longer than simple MPEG-2 encoding - it is more quality than speed focused - ffmpeg for long time didn't offer high quality MPEG-2 encoder, this was changes but also to use new functionality different than very crude command line is required - if you are interested how those two commandlines differ you should google for it. I provided reference used by me to create MPEG-2 encoding script.

[dellsam34]

I also mentioned that I don't have a tool to analyse the samples if they are progressive or interlaced, Mediainfo always shows progressive. That's why I asked if someone can analyse the sample terminatortele2 and tel me if it is progressive or interlaced,
In my understanding anything 23.xx, 24, 29.xx, 30 is progressive and anything 59.xx, 60 is interlaced in D-VHS standards. If that's the case then why leaving the frame rate untouched didn't work and telecine it to 29.xx worked, Could it be D-VHS accepts only 29.xxp and not 23.xxp? Isn't that confusing?

Here is what media info shows me about that 25Mbps file, It doesn't even show 29.97fps, Unless the guy gave me the wrong commend:

General
ID                                       : 0 (0x0)
Format                                   : MPEG-TS
File size                                : 206 MiB
Duration                                 : 1 min 8 s
Overall bit rate mode                    : Constant
Overall bit rate                         : 25.0 Mb/s

Video
ID                                       : 49 (0x31)
Menu ID                                  : 1 (0x1)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@High
Format settings, BVOP                    : No
Format settings, Matrix                  : Default
Format settings, GOP                     : N=12
Codec ID                                 : 2
Duration                                 : 1 min 7 s
Bit rate mode                            : Variable
Bit rate                                 : 23.3 Mb/s
Maximum bit rate                         : 23.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Scan order                               : 2:3 Pulldown
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.469
Time code of first frame                 : 00:00:00:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 188 MiB (92%)

Audio
ID                                       : 52 (0x34)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : 129
Duration                                 : 1 min 7 s
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel layout                           : L R C LFE Ls Rs
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Delay relative to video                  : -21 ms
Stream size                              : 3.63 MiB (2%)
Language                                 : English
Service kind                             : Complete Main

[dellsam34]

Ok, I read about 3:2 pulldown a little bit and now I think I have a clear picture about frame rate conversion, So when 24 or 30 frames are stored in a video stream it doesn't mean anything, The video player (software or hardware) can decide to scan them as entire frames (progressive) or as fields (interlaced), Hence mediainfo showing progressive has no bearing on how the video is going to be played back later.
So a 3:2 pulldown makes 4 frames (24fps) into 10 fields and then stored as 5 frames (30fps) for later playback, Then based on the player capability those 30fps could be read as 30 frames a second (30p) or as 60 fields a second (60i).

So now the commend I used it does hard pulldown, which is physically adding 2 fields into the original four frames, And you guys suggested that I do soft pulldown which it just picks two fields from the stream and vertually repeat them in the order of the pulldown, Therefore no extra encoding/decoding of extra fields/frames.

So the D-VHS deck cannot process 24fps because there is no such 48i or 48 fields per second for 1080x1920, I'm pretty sure it can do 24p for 720x1280

[pandy]

Ok, I read about 3:2 pulldown a little bit and now I think I have a clear picture about frame rate conversion, So when 24 or 30 frames are stored in a video stream it doesn't mean anything, The video player (software or hardware) can decide to scan them as entire frames (progressive) or as fields (interlaced), Hence mediainfo showing progressive has no bearing on how the video is going to be played back later.

nope, not that simple - decoder follow bitstream syntax however if telecine detected then decoder may be guided to skip/ignore repeated fields.
Mediainfo is clever to show real (before telecine) framerate.

So a 3:2 pulldown makes 4 frames (24fps) into 10 fields and then stored as 5 frames (30fps) for later playback, Then based on the player capability those 30fps could be read as 30 frames a second (30p) or as 60 fields a second (60i).

more or less but yes

So now the commend I used it does hard pulldown, which is physically adding 2 fields into the original four frames, And you guys suggested that I do soft pulldown which it just picks two fields from the stream and vertually repeat them in the order of the pulldown, Therefore no extra encoding/decoding of extra fields/frames.

Yes

So the D-VHS deck cannot process 24fps because there is no such 48i or 48 fields per second for 1080x1920, I'm pretty sure it can do 24p for 720x1280

Not necessarily - MPEG-2 decoder may work fine as it can increase field/frame rate internally and also HDMI is not restricted - generally restrictions are used to reduce cost of product. For example MPEG-2 max bitrate for MP@HL is 80Mbps - far from real life used but i bet as a part of conformance testing any D-VHS deck decoder is capable to do 80Mbps MPEG-2.

From my perspective you provided two contradictory statements - work OK vs not work at all - so i'm confused if your D-VHS deck is capable or not decode 23.976 fps MPEG-2 video.

[dellsam34]

From my perspective you provided two contradictory statements - work OK vs not work at all - so i'm confused if your D-VHS deck is capable or not decode 23.976 fps MPEG-2 video.

I tried his first commend which leaves the frame rate intact, only converts to ts and AC-3 that didn't work, the same commend he used after doing soft pulldown externally that worked, and I used the second commend that does hard pulldown and that worked, So I guess the deck has to see either real 29.97 (hard pulldown) or fake 29.97 (soft pulldown) or it won't work. Offcourse we are talking about 1080x1920, I have no interest in testing 720.

[DB83]

Now I realise I am ignored in this topic but curiosity has come over me again.

I read a little about D-VHS (a format that never really took on over this side of the pond). The wiki article mentions that, apart from video recording, the US models (maybe not all of them) had a bitstream recording mode which essentially means a bit for bit transfer of the source to the tape. Effectively an archive.

If this is possible then could that be the answer to the 23.976 or 24p issue with playback to a digital tv/monitor. Just a thought.

[drmpeg]

Well, it looks like I still have an account here. I haven't posted here since 2007. Doh! BTW, I'm the guy that dellsam34 has been communicating with on AVSForum.

The command:

ffmpeg -i Terminator.m2ts -c:v mpeg2video -r 30/1.001 -b:v 22000000 -bufsize 9781248 -maxrate 23000000 -acodec ac3 -muxrate 25000000 terminator.ts

doesn't do hard or soft telecine. It just adjusts the video PTS/DTS timestamps to repeat a frame every 4 frames. Also, it creates a 29.97p video bitstream.

I agree with pandy that it's not optimized for quality. I intentionally kept it simple to solve the first order problem, getting proper playback.

[drmpeg]

Here's the PTS/DTS sequence that ffmpeg generates.

Video PTS(P) = 6219087, DTS(P) = 6216084, 6006, 3003
Video PTS(P) = 6222090, DTS(P) = 6219087, 3003, 3003
Video PTS(P) = 6225093, DTS(P) = 6222090, 3003, 3003
Video PTS(P) = 6231099, DTS(P) = 6225093, 3003, 6006
Video PTS(P) = 6234102, DTS(P) = 6231099, 6006, 3003
Video PTS(P) = 6237105, DTS(P) = 6234102, 3003, 3003
Video PTS(P) = 6240108, DTS(P) = 6237105, 3003, 3003
Video PTS(P) = 6246114, DTS(P) = 6240108, 3003, 6006
Video PTS(P) = 6249117, DTS(P) = 6246114, 6006, 3003
Video PTS(P) = 6252120, DTS(P) = 6249117, 3003, 3003
Video PTS(P) = 6255123, DTS(P) = 6252120, 3003, 3003
Video PTS(P) = 6261129, DTS(P) = 6255123, 3003, 6006

As you can see, every 4th frame is repeated.

[dellsam34]

Welcome back to the forum drmpeg, good to have you here.
So when it adds a frame does it flag the stream for progressive or interlaced? That's what I've been trying to find out. Pandy is very knowledgeable but he doesn't have access to a D-VHS system that's why this kept dragging for pages.

[dellsam34]

Now I realise I am ignored in this topic but curiosity has come over me again.

I read a little about D-VHS (a format that never really took on over this side of the pond). The wiki article mentions that, apart from video recording, the US models (maybe not all of them) had a bitstream recording mode which essentially means a bit for bit transfer of the source to the tape. Effectively an archive.

If this is possible then could that be the answer to the 23.976 or 24p issue with playback to a digital tv/monitor. Just a thought.

That's what we are trying to find out, What that source file is going to be before it gets archived to tape while it can be decoded by the deck's mpeg board, Pretty much any digital tape format, transfer from tape or to tape is lossless thru firewire. D-VHS can archive scrambled files but it won't be able to read them, it needs an external decoder.
Add to all that, Tools like capDVHS and DVHStool won't archive a file if it is not compliant.

[drmpeg]

The stream is flagged as progressive. Here's the dump:

 Found Sequence Start Code at byte position 0.
     435 Sequence_Header_Code                          
    1920 Horizontal_Size_Value                         
    1080 Vertical_Size_Value                           
       3 Aspect_Ratio_Information                      
       4 Frame_Rate_Code                               
   57500 Bit_Rate_Value                                
       1 Marker_Bit                                    
     597 Vbv_Buffer_Size_Value                         
       0 Constrained_Parameters_Flag                   
       0 Load_Intra_Quantiser_Matrix                   
       0 Load_Non_Intra_Quantiser_Matrix               
* --------  Sequence Start Code at 0        ---------
* Picture size: 1920x1080   Frame rate: 29.97   Bitrate: 23.00 Mb/s
* VBV size: 9552 Kb   Intra QM: default   Non Intra QM: default
     437 Extension_Start_Code                          
       1 Extension_Start_Code_Identifier               
      68 Profile_And_Level_Indication                  
       1 Progressive_Sequence                          <----- progressive = 1
       1 Chroma_Format                                 
       0 Horizontal_Size_Extension                     
       0 Vertical_Size_Extension                       
       0 Bit_Rate_Extension                            
       1 Marker_Bit                                    
       0 Vbv_Buffer_Size_Extension                     
       0 Low_Delay                                     
       0 Frame_Rate_Extension_N                        
       0 Frame_Rate_Extension_D                        
* Progressive sequence.
* Main Profile, High Level MPEG-2 bitstream.
     437 Extension_Start_Code                          
       2 Extension_Start_Code_Identifier               
       0 Video_Format                                  
       1 Colour_Description                            
       1 Colour_Primaries                              
       1 Transfer_Characteristics                      
       1 Matrix_Coefficients                           
    1920 Display_Horizontal_Size                       
       1 Marker_Bit                                    
    1080 Display_Vertical_Size                         
     440 Group_Start_Code                              
    4096 Time_Code                                     
       1 Closed_Gop                                    
       0 Broken_Link                                   
* --------       GOP Start Code at 34       ---------
* Closed GOP at 00:00:00.00  drop flag: 0
     256 Picture_Start_Code                            
       0 Temporal_Reference                            
       1 Picture_Coding_Type                           
   65535 Vbv_Delay                                     
       0 Extra_Bit_Picture                             
* Using VBR VBV model.
* --------          Intra frame at 42       ---------
* Temporal Reference:  0       VBV: 9781248 (+ 0) bits
     437 Extension_Start_Code                          
       8 Extension_Start_Code_Identifier               
      15 Forward_Horizontal_F_Code                     
      15 Forward_Vertical_F_Code                       
      15 Backward_Horizontal_F_Code                    
      15 Backward_Vertical_F_Code                      
       2 Intra_Dc_Precision                            
       3 Picture_Structure                             
       0 Top_Field_First                               
       1 Frame_Pred_Frame_Dct                          
       0 Concealment_Motion_Vectors                    
       0 Q_Scale_Type                                  
       0 Intra_Vlc_Format                              
       0 Alternate_Scan                                
       0 Repeat_First_Field                            
       1 Chroma_420_Type                               
       1 Progressive_Frame                             
       0 Composite_Display_Flag                        
* Intra DC: 10 bits            Quant scale: linear
* Scan: zig-zag                Intra VLC table: 0
* Top field first: 0           Repeat first field: 0
 Found Picture Start Code at byte position 60938.

[drmpeg]

I'm working on a higher quality ffmpeg command. Here's what I have so far.

ffmpeg -i Terminator.m2ts -c:v mpeg2video -r 30/1.001 -b:v 22000000 -bufsize 9781248 -maxrate 23000000 -bf 2 -sc_threshold -30000 -seq_disp_ext 1 -color_primaries 1 -color_trc 1 -colorspace 1 -color_range 1 -dc 10 -acodec ac3 -muxrate 25000000 terminator.ts

This is what I used for the bitstream dump I just posted above.

[poisondeathray]

Interesting that it's only timestamps

Does it playback ok motion wise ?

Normally the 2:3:3:2 advanced , or even the "regular" 2:3 judder, would be preferred to that 1 in 5 repeat frame pause. Or does it have some sort of mechanism to get around that like ignoring those timestamps ?

On some software players, it looks like they are ignored , so you get the same 23.976 , and 2:3 judder on a 60Hz display

[dellsam34]

The last two samples drmpeg made for me played fine, I have to admit that I'm using my 1080x1920 laptop monitor and streaming from firewire using VLC (don't know what VLC is doing to the stream), I should probably use the HDMI port of the deck to hook up to my TV, But I assume my TV would handle the video just fine.

[pandy]

Yeep, seem this is third way to perform telecine - looks like ffmpeg was able to workaround increase frame rate without encoding... i don't see interlace artefacts and video is progressive and 29.97 at the same time. Still unclear if ffmpeg approach do not add (waste) bits i.e. hard telecine but without CPU encoding as data are reused explicitly from already encoded frames. It would be anyway better to use dgpulldown and soft telecine as it can be recognized by many decoders automatically and they produce at the output 23.976fps video.

This is what i use to test "repeat last field" flag a.k.a. soft telecine:

@set name=%~1
@ffprobe -hide_banner -v 32 -stats -y -i "%name%" -select_streams v:0 -print_format compact -show_entries "frame=pkt_pts_time,repeat_pict" > "%~n1_.txt"

I'm working on a higher quality ffmpeg command. Here's what I have so far.

ffmpeg -i Terminator.m2ts -c:v mpeg2video -r 30/1.001 -b:v 22000000 -bufsize 9781248 -maxrate 23000000 -bf 2 -sc_threshold -30000 -seq_disp_ext 1 -color_primaries 1 -color_trc 1 -colorspace 1 -color_range 1 -dc 10 -acodec ac3 -muxrate 25000000 terminator.ts

This is what I used for the bitstream dump I just posted above.

VBV buffer size question - is there any specification for D-VHS on this ? I was unable to find anything except provided in my script sources.
Seem largest VBV buffer was provided by ANSI/SCTE (VBV=488) where you stating VBV=597 - this is outcome of empirical D-VHS o something else (like commercial D-VHS tapes use this value)?

[drmpeg]

VBV size of 9781248 bits is the maximum for MPEG-2 Main Profile @ High Level. D-VHS decoders have plenty of memory, so there's no need to limit VBV size.

[pandy]

VBV size of 9781248 bits is the maximum for MPEG-2 Main Profile @ High Level. D-VHS decoders have plenty of memory, so there's no need to limit VBV size.

If you say so... my impression was that MPEG-2 decoder RAM size for buffer was around 4MiB max (D-VHS is before 2000 and this time memory is very expensive - typical PC is equipped in 32 - 64MiB, typical graphic card has around 4 - 8MiB, HDD's are around 10GB).

[drmpeg]

Second generation D-VHS decks (which is what dellsam34 has) were based on the LSI Logic Domino chip, which was a two core SPARC based SoC. The memory size was 128 MiB. The first second generation deck was the HM-DH40000, released in 2003.

The internal code name for the SoC was E5, which stood for "encoder generation number five". Subsequent SoC's in the family were numbered E5.1, E5.2, etc. The operating system was VxWorks. I still have my E5.2 development system from when I developed the MPEG-2 encoder in the Motorola DCT-6412 P3 cable box.

[dellsam34]

Wonder what you could do with that now?

[drmpeg]

The development boards were pretty intense. Since they were also used for silicon bring-up and testing, all of the SoC interfaces had to be populated. Here's the board with an IEEE1394 CF card reader as a filesystem.

As new chips were designed, the old development boards were just scraped. So taking one home was not a problem.

[dellsam34]

That's quite a kit, What are those connectors on the back with red white and black moldings ?
Were you involved in any of the nextcomwireless or 169time projects?

[drmpeg]

Starting with the red RCA connectors.

1) 8 channels of analog audio output (to support 7.1)

2) digital audio output

3) 4 channels of analog audio input (two stereo pairs)

4) digital audio input

5) component input

6) composite input

7) RF tuner

8) HD-SDI input and output (BNC connectors)

9) component output

10) two composite outputs

11) component output

Block diagram of the SoC.



The 169time stuff was already shipping when I started the D-VHS project. I remember helping someone with a tool to fix 169time streams, but I can't remember if he was involved with nextcomwireless.

[drmpeg]

I spent a few hours today testing some quality settings. Here's my new command line:

ffmpeg -i Terminator.m2ts -c:v mpeg2video -r 30/1.001 -b:v 22000000 -bufsize 9781248 -maxrate 23000000 -bf 2 -sc_threshold -30000 -seq_disp_ext 1 -color_primaries 1 -color_trc 1 -colorspace 1 -dc 10 -me_range 960 -inter_matrix 16,17,18,19,20,21,22,23,17,18,19,20,21,22,23,24,18,19,20,21,22,23,24,25,19,20,21,22,23,24,26,27,20,21,22,23,25,26,27,28,21,22,23,24,26,27,28,30,22,23,24,26,27,28,30,31,23,24,25,27,28,30,31,33 -acodec ac3 -muxrate 25000000 terminator.ts

The Non_Intra_Quantiser_Matrix makes a big difference. This is the standard non-flat matrix that I came up with when I was working at C-Cube Microsystems back in 1994. I stole it from an MPEG-1 bitstream that was encoded with the Philips Video CD encoder.

[dellsam34]

I will give it a try and report back, Maybe we can crank up the muxrate little a bit if it is that efficient, what do you think?
I will do an extra test for DTS, I think the only way to accomplish this is down convert the DTS HD and demux with tsmuxer and mux back the DTS track to the ffmpeg processed file.

[pandy]

To encode DTS by ffmpeg you should use something like this "-c:a dca -ac 6 -b:a 1536k" but DTS is waste of bits. Also it depends on source - instead re-encoding perhaps copy DTS core is sufficient thus commandline is different, something like: "ffmpeg -core_only 1 -i filename -c:a:0 copy -bsf:a:0 dca_core"

[dellsam34]

You mean by DTS core the non HD version of DTS? And yes I'm aware that DTS takes more bandwidth, I'm just testing the deck capabilities.

Any way I demuxed the DTS track and muxed it to the ffmpeg processed file using tsmuxer and the audio sync is way off, about 4 seconds off between video and audio. both video and audio files have the exact same duration before muxing and the resulting file is few microseconds shorter which doesn't make sense.