MPEG-2 TS PAT header decoding problem

[jeromedroz]

Hi,
I'm trying to develop a basic MPEG-2 Transport Stream decoder in order to extract metadatas about PSI packets. I have a bunch of books and documents (but not the standard itself) describing the stream layout; I also have some consistent TS files for testing and a TS analyser software (Manzanita's MP2TSA) to validate my assertions. The documents I use sometimes disagree, but I was able to filter the correct information up to this point. Now I have the following problem while trying to decode a PAT section: there's an extra byte in the PAT header, undocumented in all documents I've used so far. Below is the beginning of a TS Packet containing a PAT section (the remaining bytes of the packet have a value of FF):

47 40 00 10 00 00 B0 0D 00 01 C1 00 00 00 02 E1 E0 2F 39 8D 8D  .....
|TS Header |   PAT Header                 | ? |  Program   |  CRC32      | Stuffing bytes

According to all the docs I have, the PAT header's length should be equal to 8; but it ended with a mismatch when comparing my results to the ones given by MP2TSA (which is an accurate tool). Adjusting the header's length value to 9 "solves" the problem... but I don't like to use intuitive reasoning! The question is:

Does someone know the exact layout of a PAT header?

Another problem, leading to the same question:

I'm confused about the section_number and last_section_number field values of the PAT header (respectively 0xC1 and 0x00 in the sample above). I found the following description of these fields in the book "Video Demystified, A Handbook for the Digital Engineer", 4th edition (Keith Jack, Newnes, 2005, p.675):

Section_number
[...] The section_number of the first section in the PAT must be 0x00. It is incremented by 1 with each additional section in the PAT.

Last_section_number
This 8-bit binary number specifies the number of the last section (that is, the section with the highest section_number) of the complete PAT.

This is obviously wrong, but I can't figure out why... any ideas?

Thanks for reading

Jerome

[jeromedroz]

The problem is solved...

According to the ITU H.222.0 (05/06) Recommendation (http://www.itu.int/rec/T-REC-H.222.0-200605-I/en), section 2.4.4.2:

[...] When at least one section begins in a given Transport Stream packet, then the payload_unit_start_indicator (refer to 2.4.3.2) shall be set to '1' and the first byte of the payload of that Transport Stream packet shall contain the pointer. When no section begins in a given Transport Stream packet, then the payload_unit_start_indicator shall be set to '0' and no pointer shall be sent in the payload of that packet.

The content of the PAT packet is correctly interpreted this way:

47 40 00 10 00 00 B0 0D 00 01 C1 00 00 00 02 E1 E0 2F 39 8D 8D  .....
|TS Header | * | PAT Header                   |  Program   |  CRC32      | Stuffing bytes
* = pointer_field

The payload_unit_start is effectively set to 1, so the first byte of the payload data refers to a pointer_field which indicates the number of bytes until the first byte of the section. This makes perfect sense now.

Thanks to myself
:-)

[racer-x]

Congratulations! Well done........

I know nothing about programing, but I'm interested in what you're doing to retrieve metadata from TS files. Maybe you can write a program that can correctly retrieve the metadata (shutter speed, aperture, and gain) from m2t files recorded by HDV cameras. They are after all Transport Stream Mpeg-2..........

[Farish]

Thanks a lot man. I was facing the same problem. good for me that you figured it out. i had missed that in the specifications... actually initially i didnt need to go into the pat packets; just get pid and some other stuff and so i skipped the bits in between...

[Farish]

If there is just one PAT section in the TS, then should the section number and last section number both contain "00"?

[Farish]

i got the answer. yes they should

[zizou]

hi Every body,

when we demultiplex a flux MPEG2-TS we flow this method : After acquiring the TS stream, you must filter packets whose PID is 0x0000. These TS packets containing the PAT.
From the table PAT, the PID is determined and the numbers of programs. We will now find the PID of each program, the TS packets that have this PID contain sections corresponding to the PMT table for this program. In this PMT table is found for each program: elementary streams, their types and their PID. Also in the tables PMT we find the PID_PCR, which is usually the PID of one elementary stream, specifically the flow PID video.

My problem is how we put all this in header specially that the section of PID(in header) is just 13bits !!!!!?????

So please please please i want to know how we put all this in the header

[Farish]

All this information is not in the header. the header is only 4 bytes. The PID field (13 bits) only contains the PID number which gives information about the type of packet u are currently looking at. the other information about PAT etc. is after the header. please see the MPEG-2 TS specifications. i dont have access to it right now otherwise i would have pointed out where to look.

[zizou]

thank a lot man,


can you post any documents that specifies the location of the SI / PSI tables on the TS packets

[Farish]

welcome. all the information is in the MPEG-2 TS specifications. i know its not easy to understand by reading just once. u will have to read it closely and then u will understand how u can access all the information. i dont have the specifications any more and i dont know if they are available for free or not...

[zizou]

ok i see , i just found some documentations in ETSI (that defined the standard DVB-T) http://broadcasting.ru/pdf-standard-specifications/internet%20protocol/a079.pdf so that what you are talking about .

I'm just starting my project on extracting information from an MPEG2-TS and implements a program (VHDL) on FPGA, so I try to understand the digital structure of the TS to manipulate and extract each program

[Farish]

No this isnt it. I am talking about the official standard. It is the MPEG-2 Systems standard:

ISO/IEC standard (13818-1)

I think it might not be available free...

[zizou]

http://mumudvb.braice.net/mumudrupal/sites/default/files/iso13818-1.pdf

[Farish]

yes exactly. good u found it! have fun on the weekend

[zizou]

thank you so much and check your PM ^^

[kyriako41]

Hello jeromedroz,
do you have an idea (or code) how to calculate the CRC32 of the PAT?
I experimented with a similar theme without success, here:
https://forum.videohelp.com/threads/358357-TS-format-questions

Cheers,
kyriako41.

[anhngock42]

I don't understand PAT of Mpeg-2 ts. and how find PMT?

[kyriako41]

Hello anhngock42,
I have written a simple TS-Viewer. There is no program help because it was meant only for my personal use.
Download it here:
http://home.arcor.de/guentermeinel/cx70/TS_Test.zip
Read "readme.txt" for more hints.
Just start TS-View.exe and load a .ts file into it. It will show the first 16 packets.

The ID of a packet is found in the lower five bits of byte one plus byte two, like this:
ID=(TS_Buffer[1]&0x1F)*0x100 + TS_Buffer[2];
The ID of PAT is always 0, example:
0000:47 40 00 10 00 00 B0 0D 00 01 01 00 00 01 02 E1 G@..............
0010:00 8F A5 26 CF FF FF FF FF FF FF FF FF FF FF FF ...&............
....
The ID of PMT is found at position 0xF + 0x10 of the PAT-packet:
ID_PMT=(TS_Buffer[0xF]&0x1F)*0x100 + TS_Buffer[0x10];
In this case it would be 0x100. PMT-Example:
0100:47 41 00 10 00 02 B0 23 01 02 C1 00 00 E0 E0 F0 GA.....#........
0110:00 02 E0 E0 F0 03 52 01 01 03 E0 C0 F0 09 52 01 ......R.......R.
0120:02 0A 04 64 65 75 01 5B A7 88 58 FF FF FF FF FF ...deu.[..X.....
0130:FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................

By the way: the adresses like 0100: do not represent the true adress in the buffer, but indicate the packet-count.
If the "pointer" TS_Buffer[4] is not zero, the data is placed with the offset of this pointer.

Cheers,
kyriako.

[vipulpithiya]

Hello,
I am developing PSI SI software. I am new to this technology. By reading some material, I have made some of the PSI packets and sending through UDP. I am converting it into Hexadecimal and sending them throough UDP. I am using wireshark and TS Reader to check whether packets are properly created. In wireshark it shows the table name in info column i.e. PAT, BAT etc. But in TS reader nothing comes up. What i am missing ? I have tried many ways but no way it comes in TSReader. Is there any special that need to be used ? Is there any other part that I need to work before sending the packets and to make them view in TS Reader ?

Please Help !!

Thanks and Regards,

PACKET

47 60 00 10 00 00 f0 35 00 01 e3 00 01 00 00 00 10 01 01 01 b1 01 02 01 b2 01 03 01 b3 01 04 01 b4 01 06 1f ff 01 07 01 b7 01 0a 01 ba 01 0b 01 bb 01 18 10 30 01 1a 10 31 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff



[Attachment 34574 - Click to enlarge]

[vipulpithiya]

Hello,


How PSI packets should be sent through network, that means which protocol should be used ? We tried UDP but it is not working.

udpclinet.connect(IPAddress.parse("10.15.17.124",8 000)
udpclinet.send(bytes,bytes.Length)

[pandy]

http://www.videolan.org/developers/bitstream.html
http://www.videolan.org/developers/libdvbpsi.html
https://www.wireshark.org/

[vipulpithiya]

Hello Sir,

We are working on sending UDP packets to PSI SI. We are developing PSI SI generator.
But we are stuck on CheckSum CRC32 - we are not able to find the check sum. I tried on few code from the internet . It comes up with some checksum but that checksum doesnt match with wireshark check sum .

We have wireshark dump of the PSISI packets working with correct checksum .

Can anyoone help me in calculating the checksum for the PSI SI ?


Regards,
Bimal