TS format questions

[kyriako41]

Hello,
I have a settop box type "Imperial HD 3k" for the reception of cable TV.
This device can record and play back TV transmissions. I have recorded a short sequence. You can download it here:
http://home.arcor.de/guentermeinel/cx70/TS_Test.zip
The file is \TS_Test\Imperial\REC_00.ts.
Now I would like to adapt a different video so that it can be replayed on the "Imperial".
A sample would be \TS_Test\TerratecRemux\Terratec.ts

To be able to experiment, I need to learn more about the TS file format.
When I load REC_00.ts into "TSreaderLite", I get the following image:
http://home.arcor.de/guentermeinel/cx70/REC_00.PNG

My questions:
1. there are a number of PIDs. Where in the TS file are the definitions for these PIDs(which is Video, Audio, Teletext etc.)?
2. where in the stream can I find the data corresponding to "720x576i", "25fps", "16:9" etc.

Any hint would be appreciated.
Thank you.

kyriako41.

[DB83]

Check the manual for your set-top box about playing external files.

If it is not mentioned then you can not do this.

And you certainly can not copy any file to the HDD of your set-top box even if you could encode the .ts file correctly.

[netmask56]

Use MediaInfo which will reveal the following - also if you use ProjectX you can demux all the individual elements.

General
ID                                       : 1 (0x1)
Complete name                            : C:\Users\Netmask\Desktop\REC_00.TS
Format                                   : MPEG-TS
File size                                : 7.62 MiB
Duration                                 : 9s 170ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 6 947 Kbps

Video
ID                                       : 210 (0xD2)
Menu ID                                  : 28007 (0x6D67)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Custom
Format settings, GOP                     : M=3, N=12
Codec ID                                 : 2
Duration                                 : 9s 40ms
Bit rate mode                            : Variable
Bit rate                                 : 5 504 Kbps
Maximum bit rate                         : 15.0 Mbps
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 16:9
Active Format Description                : Letterbox 16:9 image
Frame rate                               : 25.000 fps
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.531
Stream size                              : 5.93 MiB (78%)

Audio #1
ID                                       : 220 (0xDC)
Menu ID                                  : 28007 (0x6D67)
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 3
Duration                                 : 9s 144ms
Bit rate mode                            : Constant
Bit rate                                 : 256 Kbps
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 KHz
Compression mode                         : Lossy
Delay relative to video                  : -249ms
Stream size                              : 286 KiB (4%)
Language                                 : German

Audio #2
ID                                       : 221 (0xDD)
Menu ID                                  : 28007 (0x6D67)
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 3
Duration                                 : 9s 168ms
Bit rate mode                            : Constant
Bit rate                                 : 192 Kbps
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 KHz
Compression mode                         : Lossy
Delay relative to video                  : -249ms
Stream size                              : 215 KiB (3%)

Audio #3
ID                                       : 222 (0xDE)
Menu ID                                  : 28007 (0x6D67)
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 3
Duration                                 : 9s 144ms
Bit rate mode                            : Constant
Bit rate                                 : 192 Kbps
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 KHz
Compression mode                         : Lossy
Delay relative to video                  : -209ms
Stream size                              : 214 KiB (3%)
Language                                 : Multiple languages

Audio #4
ID                                       : 225 (0xE1)
Menu ID                                  : 28007 (0x6D67)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 6
Duration                                 : 9s 56ms
Bit rate mode                            : Constant
Bit rate                                 : 448 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Delay relative to video                  : -172ms
Stream size                              : 495 KiB (6%)
Language                                 : German

Text #1
ID                                       : 230 (0xE6)-100
Menu ID                                  : 28007 (0x6D67)
Format                                   : Teletext
Language                                 : German

Text #2
ID                                       : 231 (0xE7)
Menu ID                                  : 28007 (0x6D67)
Format                                   : DVB Subtitle
Codec ID                                 : 6
Duration                                 : 5s 40ms
Delay relative to video                  : 6s 0ms
Language                                 : German

Menu
ID                                       : 200 (0xC8)
Menu ID                                  : 28007 (0x6D67)
Duration                                 : 9s 170ms
List                                     : 210 (0xD2) (MPEG Video) / 220 (0xDC) (MPEG Audio, German) / 221 (0xDD) (MPEG Audio) / 222 (0xDE) (MPEG Audio, Multiple languages) / 225 (0xE1) (AC-3, German) / 230 (0xE6) () / 231 (0xE7) (DVB Subtitle, German) / 950 (0x3B6) ()
Language                                 :  / German /  / Multiple languages / German /  / German

[kyriako41]

Hello netmask56,
thank you for your hint.
As it seems, the PIDs are defined in the packet with PID=0xc8.
The content looks like this:
0000:47 40 C8 10 00 02 B0 AB 6D 67 C5 00 00 E0 D2 F0 G@......mg......
0010:00 02 E0 D2 F0 08 52 01 01 02 03 1A 48 5F 03 E0 ......R.....H_..
0020:DC F0 09 52 01 02 0A 04 64 65 75 00 03 E0 DD F0 ...R....deu.....
0030:10 52 01 04 0A 04 6D 69 73 00 7F 05 06 87 64 65 .R....mis.....de
0040:75 03 E0 DE F0 09 52 01 09 0A 04 6D 75 6C 00 06 u.....R....mul..
0050:E0 E1 F0 0E 52 01 05 0A 04 64 65 75 00 6A 03 C0 ....R....deu.j..
0060:42 08 06 E0 E6 F0 29 52 01 03 56 05 64 65 75 09 B.....)R..V.deu.
0070:00 45 1D 01 18 E7 E8 E9 EA EB EC ED EE EF F4 F5 .E..............
0080:F6 C7 C8 C9 CA CB CC CD CE CF D4 D5 D6 04 01 F0 ................
0090:06 E0 E7 F0 0D 52 01 06 59 08 64 65 75 20 00 01 .....R..Y.deu ..
00A0:00 01 05 E3 B6 F0 08 52 01 08 6F 03 00 10 E0 6B .......R..o....k
00B0:87 A9 33 FF FF FF FF FF FF FF FF FF ..3.............

Now, what is the meaning of these bytes?
For example, where is determined that the video-PID is 0xd2?

Regards,
kyriako41.

[Selur]

you might want to read http://en.wikipedia.org/wiki/MPEG_transport_stream and google a bit for how transportstream headers look like,.. (overview in german: http://www.fr-an.de/projects/01/03_03_02.htm)

[netmask56]

Using ProjectX to demux reveals the following in the log file.

[22:52:13.885] Monday, August 26, 2013  8:52:13 AM EST
[22:52:13.885] ProjectX 0.90.04.00.b29 (21.02.2009)
[22:52:13.934] 
[22:52:13.935] -> working with collection 0
[22:52:13.935]  
[22:52:13.935] -> save normal log file
[22:52:13.935] -> log 'packets out of sequence' / bit errors
[22:52:13.936] -> log 'missing startcodes'
[22:52:13.936] -> log 'PES header found in ES'
[22:52:13.936] -> log WSS
[22:52:13.937] -> log VPS
[22:52:13.937] -> log RDS
[22:52:13.937] -> log max. 500 warnings/errors
[22:52:13.937] -> write all video data
[22:52:13.938] -> write all other data
[22:52:13.938] -> patch c.d.flagged infos of pictures
[22:52:13.938] -> add sequence end code
[22:52:13.938] -> set resolution in SDE 
[22:52:13.938] -> PVA: strictly specs. for audio streams
[22:52:13.938] -> VOB: determine diff. Cell timelines
[22:52:13.939] -> TS: ignore scrambled packets
[22:52:13.939] -> TS: enhanced search for open packets
[22:52:13.939] -> TS: join file segments (of Dreambox®)
[22:52:13.939] -> TS: generate PMT stream dependent
[22:52:13.940] -> get only enclosed PES/TS packets
[22:52:13.940] -> concatenate different recordings
[22:52:13.940] -> ensure 1st PES-packet start with video
[22:52:13.940] -> generate PCR/SCR from PTS
[22:52:13.940]  
[22:52:13.942] -> write output files to: 'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00'
[22:52:13.942] 
[22:52:13.942] -> main I/O-Buffersize in bytes 4096000 bytes
[22:52:13.948] 
[22:52:13.948] FileSegments:
[22:52:13.948] * (0) C:\Users\Netmask\Desktop\REC_00.TS
[22:52:13.949] second. Files:
[22:52:13.949] * ---
[22:52:13.950] 
[22:52:13.950] +> Input File 0:  'C:\Users\Netmask\Desktop\REC_00.TS' (7,989,248 bytes)
[22:52:13.951] -> Filetype is TS (generic PES Container)
[22:52:13.951] -> demux
[22:52:13.964] -> Service ID 0x6D67
[22:52:13.965] -> PMT 0xC8 refers to these usable streams:
[22:52:13.966] Video:
[22:52:13.966] PID: 0xD2(#1)
[22:52:13.967] Audio:
[22:52:13.967] PID: 0xDC(#2)(deu)
PID: 0xDD(#4)(mis)
PID: 0xDE(#9)(mul)
PID: 0xE1(#5)(deu)(AC-3)[PD]
[22:52:13.967] Teletext:
[22:52:13.967] PID: 0xE6(#3)(deu_i100 )
[22:52:13.967] Subpict.:
[22:52:13.968] PID: 0xE7(#6)(deu_0x20_p1_a1 )
[22:52:13.968] 
[22:52:13.970] --> PID 0x1FFF => stuffing packet -> ignored
[22:52:13.971] !> PID 0x0 (PAT) (564 #4) -> ignored
[22:52:13.971] !> PID 0xC8 (PMT) (752 #5) -> ignored
[22:52:13.975] ok> PID 0xE6 has PES-ID 0xBD (private stream 1) (TTX)  (2068 #12) 
[22:52:13.977] !> PID 0x12 (EIT) (2256 #13) -> ignored
[22:52:13.979] ok> PID 0xDD has PES-ID 0xC0 (MPEG Audio) (14664 #79) 
[22:52:13.982] ok> PID 0xDC has PES-ID 0xC0 (MPEG Audio) (31020 #166) 
[22:52:13.984] ok> PID 0xD2 has PES-ID 0xE0 (MPEG Video) (39104 #209) 
[22:52:13.996] ok> PID 0xDE has PES-ID 0xC0 (MPEG Audio) (49068 #262) 
[22:52:14.000] ok> PID 0xE1 has PES-ID 0xBD (private stream 1) (103212 #550) 
[22:52:14.018] !> PID 0x11 (BAT) (437664 #2329) -> ignored
[22:52:14.035] -> video basics: 720*576 @ 25fps @ 0.7031 (16:9) @ 15000000bps, vbvBuffer 112
[22:52:14.036] -> starting export of video data @ GOP# 0
[22:52:14.036] !> dropping useless B-Frames @ GOP# 0 / new Timecode 00:00:00.000
[22:52:14.141] ok> PID 0xE7 has PES-ID 0xBD (private stream 1) (SubID 0x20) (2364664 #12579) 
[22:52:14.339] packs: 32973 100% 7989248
[22:52:14.340] 
[22:52:14.341] ++> Mpg Video: PID 0x00D2 / PesID 0xE0 / SubID 0x00 :
[22:52:14.341] -> Video: fr-ct-1p-cg-og-dg -> 214-1-0-18-0-0
[22:52:14.347] -> Video length: 214 frames @ 00:00:08.560
[22:52:14.347] -> GOP summary: min. 20, max. 24 fields; contains interlaced frames
[22:52:14.348] -> avg. nom. bitrate 3867337bps (min/max: 2963200/4491200)
[22:52:14.348] -> set first sequenceheader bitrate to 4491200bps
[22:52:14.348] ---> new File: C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.mpv
[22:52:14.350] 
[22:52:14.350] ++> Teletext: PID 0x00E6 / PesID 0xBD / SubID 0x10 :
[22:52:14.364] 
[22:52:14.364] ++> Mpg Audio: PID 0x00DD / PesID 0xC0 / SubID 0x00 :
[22:52:14.406] -> check CRC of AC-3 / MPEG-Audio L1,2
[22:52:14.407] -> remove CRC in MPEG-Audio L1,2
[22:52:14.407] -> fill gaps with prev. frame
[22:52:14.407] -> add frames
[22:52:14.410] -> Audio PTS: first packet 10:49:27.434, last packet 10:49:36.434
[22:52:14.411] -> Video PTS: start 1.GOP 10:49:27.884, end last GOP 10:49:36.444
[22:52:14.412] -> adjusting audio at video-timeline
[22:52:14.416] -> src_audio: MPEG-1, Layer2, 48000Hz, stereo, 192kbps, CRC @ 00:00:00.000
[22:52:14.514] audio frames: wri-pre-skip-ins-add 356-0-0-0-0 @ 00:00:08.544 done...
[22:52:14.516] ---> new File: 'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.mpa'
[22:52:14.520] 
[22:52:14.520] ++> Mpg Audio: PID 0x00DC / PesID 0xC0 / SubID 0x00 :
[22:52:14.522] -> check CRC of AC-3 / MPEG-Audio L1,2
[22:52:14.522] -> remove CRC in MPEG-Audio L1,2
[22:52:14.522] -> fill gaps with prev. frame
[22:52:14.522] -> add frames
[22:52:14.524] -> Audio PTS: first packet 10:49:27.434, last packet 10:49:36.434
[22:52:14.525] -> Video PTS: start 1.GOP 10:49:27.884, end last GOP 10:49:36.444
[22:52:14.526] -> adjusting audio at video-timeline
[22:52:14.530] -> src_audio: MPEG-1, Layer2, 48000Hz, stereo, 256kbps, CRC @ 00:00:00.000
[22:52:14.618] audio frames: wri-pre-skip-ins-add 356-0-0-0-0 @ 00:00:08.544 done...
[22:52:14.620] ---> new File: 'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00-02.mpa'
[22:52:14.662] 
[22:52:14.663] ++> Mpg Audio: PID 0x00DE / PesID 0xC0 / SubID 0x00 :
[22:52:14.665] -> check CRC of AC-3 / MPEG-Audio L1,2
[22:52:14.665] -> remove CRC in MPEG-Audio L1,2
[22:52:14.666] -> fill gaps with prev. frame
[22:52:14.666] -> add frames
[22:52:14.666] -> Audio PTS: first packet 10:49:27.474, last packet 10:49:36.474
[22:52:14.674] -> Video PTS: start 1.GOP 10:49:27.884, end last GOP 10:49:36.444
[22:52:14.675] -> adjusting audio at video-timeline
[22:52:14.677] -> src_audio: MPEG-1, Layer2, 48000Hz, stereo, 192kbps, CRC @ 00:00:00.000
[22:52:14.768] audio frames: wri-pre-skip-ins-add 357-0-0-0-0 @ 00:00:08.568 done...
[22:52:14.769] ---> new File: 'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00-03.mpa'
[22:52:14.804] 
[22:52:14.804] ++> AC3/DTS Audio: PID 0x00E1 / PesID 0xBD / SubID 0x00 :
[22:52:14.806] -> check CRC of AC-3 / MPEG-Audio L1,2
[22:52:14.806] -> remove CRC in MPEG-Audio L1,2
[22:52:14.807] -> fill gaps with prev. frame
[22:52:14.807] -> add frames
[22:52:14.808] -> Audio PTS: first packet 10:49:27.512, last packet 10:49:36.312
[22:52:14.812] -> Video PTS: start 1.GOP 10:49:27.884, end last GOP 10:49:36.444
[22:52:14.813] -> adjusting audio at video-timeline
[22:52:14.817] -> src_audio: AC-3, CM, 2/0(2.0), dn -23dB, 48000Hz, 448kbps @ 00:00:00.000
[22:52:14.894] audio frames: wri-pre-skip-ins-add 267-0-0-0-0 @ 00:00:08.544 done...
[22:52:14.897] ---> new File: 'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.ac3'
[22:52:14.930] 
[22:52:14.930] ++> Subpicture: PID 0x00E7 / PesID 0xBD / SubID 0x20 :
[22:52:14.932] Multicolor OFF / switches 00000000
[22:52:14.932] -> selected DVB subpicture color model: (0) 4 colors ; fixed to page id: 
[22:52:14.933] -> export format: sup
[22:52:14.933] -> temp. file: REC_00.sp (6023 bytes)
[22:52:14.938] -> Subpicture PTS: first packet 10:49:33.684, last packet 10:49:38.484
[22:52:14.939] -> Video PTS: start 1.GOP 10:49:27.884, end last GOP 10:49:36.444
[22:52:14.939] -> adjusting subpicture at video-timeline
[22:52:14.948] -> source is DVB Subtitle...
[22:52:14.991] 1 subpictures written...
[22:52:14.999] ---> new File: C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.sup
[22:52:15.000] 
[22:52:15.000] summary of created media files:
[22:52:15.001] .Video (m2v):	214 Frames	00:00:08.560		'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.mpv'
[22:52:15.001] Audio 00 (mp2):	356 Frames	00:00:08.544	0-0-0-0	'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.mpa'
[22:52:15.001] Audio 01 (mp2):	356 Frames	00:00:08.544	0-0-0-0	'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00-02.mpa'
[22:52:15.001] Audio 02 (mp2):	357 Frames	00:00:08.568	0-0-0-0	'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00-03.mpa'
[22:52:15.002] Audio 03 (ac3):	267 Frames	00:00:08.544	0-0-0-0	'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.ac3'
[22:52:15.002] SubPicture 00:	1 subpictures		'C:\Users\Netmask\Desktop\2013-06-29_18.24.44_REC_00\REC_00.sup'
[22:52:15.002] => 5,316,647 bytes written...
[22:52:15.002] -> we have 5 warnings/errors.

[kyriako41]

Hello,
I found an answer to part of my questions, here:
http://en.wikipedia.org/wiki/Program-specific_information
http://cmm.khu.ac.kr/korean/files/02.mpeg2ts1_es_pes_ps_ts_psi.pdf

The results for my PMT example are:
1. Video: PID->0x13: D2, Stream Type->0x11:02
2. Audio: PID->0x20: DC, Stream Type->0x1E:03

Now I would like to remove some of the streams from the PMT.
If I do that, I have to recalculate the CRC.
In the above documents, I found the following:
Start of the byte-block for calculation: 8
End " : 0xAE, just in front of the CRC32

I tried to calculate the CRC32 of the existing example, but did not get the reqired result, which should be 0x6B87A933.
Any idea how to do it correctly?
The sample PMT binary file together with some code is here:
http://home.arcor.de/guentermeinel/cx70/TS_CRC.zip

Cheers,
kyriako41.

p.s. I found a solution for the crc32 calculation, here:
http://www.lammertbies.nl/forum/viewtopic.php?t=1398
http://www.codeforge.com/read/62241/crc32.c__html

and the starting position for the calculation is 5.

[WishMe]

ok, do not know much about ts, first ts stream has pat and pmt, and other table and packet, you first analysis pat/pmt, then you get
pid and number of pid,
then you analysis width, height, and other codec like things, tsremuxer is open source, it has all the
details, maybe you can analysis the code

[kyriako41]

Hello WishMe,
my question was: where do programs like tsremuxer get the information from(which packet, which bytes)?

Cheers,
kyriako41.

[WishMe]

ok, first you should download TsRemuxSrc0.20, (just now i searched, but no download link, tryhttp://www.megaupload.com/?d=6G5LPRI4, if still can not download source code, i will give you a copy).


open the TsRemux project and open file TsToPes.cs, line 1594, TsRemux.TsPesFile.GetInitialValues
then analysis,


from pat, then pmt, ...
ParseElementaryStreams method analysis width/height, and other codec params,


i am not expert in ts, hope this can help you

[El Heggunte]

ok, first you should download TsRemuxSrc0.20, (just now i searched, but no download link, try www.megaupload.com/?d=6G5LPRI4,

You are kidding alright. megaupload is dead, Jim.

if still can not download source code, i will give you a copy).

https://github.com/antiochus/tsremux

OR... just see the attachment below

[kyriako41]

Hello El Hegunte,
thank you for the code. I hope I can decipher the content.
As mentioned earlier in the thread, I now understand at least PAT and PMT.
Meanwhile, I have solved my problem, namely: making my settop-box play back external TS-files.
I found out that the box inserted two "private" packets(PID=0x1FFF)in front of each PAT, containing information on video- and audio PIDs.
Now I took these two packets from an original settop-box-file, put in the PIDs of the external file, and inserted them accordingly into that file.
Astonishing enough: it worked.

Thank you all for your support.
kyriako41.

[pandy]

http://www.tek.com/support/faqs/what-packet-identifier-pid-used-mpeg

PID 0x1FFF = stuffing packets.

[kyriako41]

Hello pandy,
by definition, the content of these "Null packets" is ignored by the player.
The settop-box makes use of this fact by writing "private" data into them.

A different item would interest me:
when I start TsRemux(or VLC) and want to open a file, an opening dialog would show up.
As soon as I mark a file, the playing time(besides other data) is shown. This seems to be a Windows(7) function.
TS-files are associated with the Windows Media Player. This is obviously the program that provides the data (together with the starting image).
Where are these data to be found?

kyriako41.

[pandy]

Hello pandy,
by definition, the content of these "Null packets" is ignored by the player.
The settop-box makes use of this fact by writing "private" data into them.

A different item would interest me:
when I start TsRemux(or VLC) and want to open a file, an opening dialog would show up.
As soon as I mark a file, the playing time(besides other data) is shown. This seems to be a Windows(7) function.
TS-files are associated with the Windows Media Player. This is obviously the program that provides the data (together with the starting image).
Where are these data to be found?

kyriako41.

This is tricky part - they can be everywhere and they can be not explicit i.e. it can be based on some data inside stream (like TOT/TDT; PTS/DTS ; PCR for example) it can be based on calculation (if TS is CBR type then size/bitrate will give you time) or it can be even more tricky (like ES length checking - GOP time stamps, bitrate vs size).
Computer have free access to TS file and TS is designed to be simple and fast thus i can believe that time length can be easily calculated by modern CPU almost in real time. But there is no standardized way to insert/embed time length explicitly inside TS. However it can be some TS variant that is used by file players (m2ts for example) they can have additional information that cover also this aspect (some players use regular TS but they providing second - index file to easier navigation on stream, sometimes there is third file that carry short description event - time etc thus there is no need to recover such info from TS).

[kyriako41]

Hello pandy,
very good explanation. Thank you.
I hope, I'll find out at least how TsRemux does it by studying the code.

kyriako41.