020501 - what to call this effect and how to avoid it?

[hamidi2]

Hi
I'm going to share a video with you. The video is erroneous. It comes from an environment with CCTV cameras and a native software which collects and records them. The developers need to know why the videos get erroneous and how can they change their code to prevent it. They decode the frames of cameras by ffmpeg libraries (libav-utils, libav-codec, etc.) under linux. The OS is Ubuntu 12.04 and they can't upgrade it because of a driver for their DVR cards which is discontinued and works only with this OS kernel.
The problem has not been existing before and once a change in something which they don't know what it has been caused the problem to occur.
Let me know what this effect should be called (What's the common name [word] for it in profession)?
Then let me know how can they avoid it and what has caused it? What has been wrong in their recording or gathering and collecting frames?
A note about the video is that when I play it by VLC and I change the "Hardware-accelerated decoding" option under Input/Codecs, the effect changes!
What's the problem and how can they resolve it?
Thank you

[Subtitles]

The nearest technical term for this would be multicolor vertical lines on the video or multicolor vertical streaks on the video.
I get this sometimes in winter with my satellite dish receiver because of the rain or clouds.
It is a common problem also with CCTV cameras, as you can see from this link
https://community.arlo.com/t5/Arlo/Pixelated-thumbnail-Live-stream-has-green-screen/m-p/1771028
Click on Go to Solution. I hope there is something useful there for you.

[hamidi2]

Thank you, I'll read it.

[jagabo]

Those are decoding errors from corrupt data in high compression inter-frame codecs. Exactly what you get depends on how the decoder deals with corrupt data. Hence the differences between hardware and software decoding in VLC. The errors can be caused by a bug in the encoder, transmission losses, or using a decoder that doesn't fully support the compression codec settings.

Error scan with ffmpeg:

ffmpeg.exe -v error -i wer.mp4 -f null - >wer.mp4.error.txt 2>&1

result:

[h264 @ 0000013bb25e6600] left block unavailable for requested intra4x4 mode -1
[h264 @ 0000013bb25e6600] error while decoding MB 0 11, bytestream 69031
[h264 @ 0000013bb274aec0] left block unavailable for requested intra4x4 mode -1
[h264 @ 0000013bb274aec0] error while decoding MB 0 11, bytestream 69031
[h264 @ 0000013bb266e500] cabac decode of qscale diff failed at 34 35
[h264 @ 0000013bb266e500] error while decoding MB 34 35, bytestream 18907
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 146 >= 146
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 149 >= 149
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 156 >= 156
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 168 >= 168
[h264 @ 0000013bb266e500] error while decoding MB 68 44, bytestream -11
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 181 >= 181
[h264 @ 0000013bb2f7bf80] left block unavailable for requested intra4x4 mode -1
[h264 @ 0000013bb2f7bf80] error while decoding MB 0 35, bytestream 16819
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 201 >= 201
[h264 @ 0000013bb2882180] error while decoding MB 47 6, bytestream -9
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 220 >= 220
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 242 >= 242
[h264 @ 0000013bb2881980] cabac decode of qscale diff failed at 77 41
[h264 @ 0000013bb2881980] error while decoding MB 77 41, bytestream 3291
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 251 >= 251
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 260 >= 260
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 263 >= 263
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 293 >= 293
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 300 >= 300
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 320 >= 320
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 323 >= 323
[h264 @ 0000013bb2881980] error while decoding MB 33 26, bytestream -5
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 331 >= 331
[h264 @ 0000013bb2868700] left block unavailable for requested intra4x4 mode -1
[h264 @ 0000013bb2868700] error while decoding MB 0 17, bytestream 74608
[h264 @ 0000013bb294b880] error while decoding MB 55 8, bytestream -7
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 351 >= 351
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 356 >= 356
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 362 >= 362
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 371 >= 371
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 382 >= 382
[h264 @ 0000013bb274b2c0] error while decoding MB 68 38, bytestream -7
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 389 >= 389
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 432 >= 432
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 451 >= 451
[h264 @ 0000013bb274aec0] error while decoding MB 44 43, bytestream -5
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 469 >= 469
[null @ 0000013bb288eac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 481 >= 481

[hamidi2]

Thank you. Please let me know how can I guide the team to resolve the problem, by this clue. What has been incorrect in their code? Did they alter the dts of frames provided by the camera? Did they give the frames to ffmpeg in incorrect order? In another words, please let me know the next step. I'm not so expert in ffmpeg.

[jagabo]

It comes from an environment with CCTV cameras and a native software which collects and records them. The developers need to know why the videos get erroneous and how can they change their code to prevent it. They decode the frames of cameras by ffmpeg libraries (libav-utils, libav-codec, etc.) under linux.

So the video you uploaded is directly from the CCTV cameras? Or after some processing by ffmpeg?

Please let me know how can I guide the team to resolve the problem, by this clue.

If the sample is directly from the camera you either have transmission errors (WiFi?) or something wrong with the "native software". If it's poor WiFi signals better hardware may help. Or see if there is an option for more error resilient protocols. For example, TCP instead of UDP.

[hamidi2]

1. No, it's after some processing by ffmpeg. Even the program itself may have altered the frames (like pts or dts) before giving them for decoding to ffmpeg.
2. The rtsp link of the cameras when played directly by ffplay in the server (the program is down) we see the artifact too. But, when played by vlc there's no artifact.
3. We prefer udp, because we need live frames and tcp sometimes causes delays.
4. There's no WiFi. It's just lan.

[jagabo]

1. No, it's after some processing by ffmpeg. Even the program itself may have altered the frames (like pts or dts) before giving them for decoding to ffmpeg.

I'm going to guess ffmpeg is being used in "copy" mode, not reencoding. If reencoding there would be a corrupt picture but no stream errors.

2. The rtsp link of the cameras when played directly by ffplay in the server (the program is down) we see the artifact too. But, when played by vlc there's no artifact.

This implies that the video stream is corrupt before it gets to ffplay. It may be that VLC is using TCP rather than UDP at the transport layer. Or VLC is ignoring corrupt packets leaving no, or much less severe, corruption of the video. Keep in mind that once the picture becomes corrupt it will likely remain corrupt until the next I frame. Does the frequency of the errors vary with the time of day (using that as a proxy for network congestion)?

3. We prefer udp, because we need live frames and tcp sometimes causes delays.

Of course. But if you can, try using TCP instead as an experiment. If there is much less corruption you can assume UDP transmission errors are at the heart of the problem.

4. There's no WiFi. It's just lan.

Even a LAN can be overloaded and drop/corrupt UDP packets.

[hamidi2]

- I think the bottleneck is the overloaded LAN and the bandwidth required by the number of cameras and their stream volumes exceeds and overloads the network which can't handle it. Without changing anything in our software, once they said that the images has got erroneously. Maybe they've added new cameras, although they didn't confess about it.
- What should be changed for converting udp into tcp? In camera settings or ffmpeg in our software?
- You said that maybe vlc is ignoring corrupt packets. This means that it can detect errors in frames. So the question is that how can I detect errors in frames too?
- Yeah, ffmpeg is just used to decode and no re-encoding occurs. We don't need it. Besides, we can't do it, since we don't know how to do it by using GPU and the server load increases if we want to do it by using CPU.
- If I can detect a corruption in a frame, and drop it (don't give to ffmpeg for decoding), I should ignore next frames until the next I frame, because it causes corrupted images, even if the frames are not corrupted, right? If this is the case, and there're only two key frames in a second, I get non-smooth motion and lose many frames.
- The frequency of errors doesn't differ with the time of day. I couldn't figure out from your text that how could this help.
Thank you

[hamidi2]

I tried "-rtsp_transport tcp" option with ffplay and it resolved the issue. Now I like to know:
- How can I get correct result without this option too?
- Programmatically, how can I set this option?

[jagabo]

I tried "-rtsp_transport tcp" option with ffplay and it resolved the issue.

Good. Now you know for certain what the problem is.

- How can I get correct result without this option too?

You would have to detect missing and corrupt packets and refrain from sending them to the h.264 decoder. This will probably require a deep understanding of h.264 compression as you can't just discard UDP packets -- you have to analyze the data and discard at the h.264 level. And it won't completely eliminate the problem, just reduce it.

- Programmatically, how can I set this option?

There should be some way to specify the transport layer when you open the rstp connection.

Some other possible workarounds:

1) Specify shorter GOPs for the h.264 encoder (if possible). That won't eliminate the problem but will give you shorter corrupt segments. Unfortunately, shorter GOPs also means more data to transmit (leading to more packet loss on the network) or lower image quality at the same data rate.

2) Use a codec that can compress the data even more -- like h.265. That may reduce network congestion enough to eliminate (or greatly reduce) the lost pack problem. Of course, it requires cameras (or whatever device is compressing the video) that support the codec.

3) Specify an all i-frame codec instead of h.264, like MJPEG. That will limit errors to single frames. But again, that means more data (intra-frame codecs doesn't compress as much as inter-frame codecs) or lower image quality. And it requires support for the codec in the cameras.

4) Switch to a faster network infrastructure. For example, TX1000 instead of TX100.

[hamidi2]

Yeah, I know. But I can't figure out why the top side portion of images never corrupt. TCP is just a workaround, because it causes packets load and delay between live and images increase continually. ffmpeg itself is the source of realizing packets and it may know whether a packet is corrupted or not, and if it's corrupted it may prevent decoding it. There's a note, even if a packet is not decoded because of corruptness, the next non-key frame packet may not be decoded too, because it depends to it.
Let's try another solution, is there a way to pull packets from cameras instead of the camera streaming all of its packet to the network? If it's possible, we may still use TCP protocol without losing "live" images and the delay won't bother us. We use rtsp://... link to get packets from cameras. Is it possible to get the next packet on demand? If so, the camera may decide what kind of packet (I-Frame, B-Frame or P-Frame) should generate based on the last packet sent to us. It's ideal.

[hamidi2]

Now, another problem. The program is compiled and linked with ffmpeg v2.8.7 and this version of ffmpeg has not rtsp.h or RTSPState structure. Is there a way to change rtsp mode from UDP to TCP?

[jagabo]

Now, another problem. The program is compiled and linked with ffmpeg v2.8.7 and this version of ffmpeg has not rtsp.h or RTSPState structure. Is there a way to change rtsp mode from UDP to TCP?

I don't know. Can you change to a full(er) build of ffmpeg?

[hamidi2]

Oh, I think I misunderstood something. I downloaded the latest sources of ffmpeg, not libav! It seems that they differ. I think the rtsp.h and rtsp.c are header file and source file of ffmpeg, not libav which I may use from within my program. If so, I should look into ffmpeg sources and see what will change if tcp is specified at command line of ffplay. This may give me a clue if I investigate in the change of ffplay's behavior with libav, and this is not so easy. I should struggle with the sources. Do you have any clue?
And about your question, even the latest sources of libav don't include rtsp.h or rtsp.c. It seems that they belong to ffmpeg and ffplay, not libav itself, although when you install ffmpeg, you'll have some libav include and library files too. The installer installs them for ffmpeg and ffplay because they need them, but they're different from libav, I think.

[jagabo]

Well, you're outside my range of knowledge now.