mp4 and multiple video streams

[hagai_sela]

Hi,
I am writing a recording feature for a video client. I have a one-on-one voice and video call, and I am supposed to save the 2 video and 2 audio streams to a .mp4 file.
When the file is played I want to be able to see and hear both video and audio streams concurrently. I have a directshow based decoder for the audio and video so the player has to be directshow based.
I have a few questions:
1. I want to use an existing splitter and mux filters, and my client can't use GPL or LGPL. Can I use Haali? Does it support playback of 2 concurrent video streams?
2. Can I use windows media player for the playback? If so, How will I see the 2 video streams?

Thanks,
Hagai.

[poisondeathray]

1) no

2) not possible, unless you combine the streams into 1 actual stream (e.g. split screen). Same with audio, unless you use divide into separate channels (not streams)

[hagai_sela]

1. OK... Anything else I can use? Can I use avi or another container format instead of mp4?
2. I have no problem with split screen. Does that mean that I have to decode the 2 streams and re-encode them to one frame or is there a more intelligent way?

[poisondeathray]

Nothing that I know of supports multiple concurrent stream playback. This question gets asked quite frequently

I don't know about licensing issues for existing software, you might as at doom9 or doom10 forums, they have people well versed in that area

2 - yes, it generally means re-encoding them if you want it permanent. You would have probably have to develop custom playback software to do this any other way without re-encoding

You can play a script in avisynth that can do this without re-encoding, but most people won't know how to do this, certainly not something you would want to do for a client

e.g.

a= ffvideosource("video1.mp4")
b= ffvideosource("video2.mp4")
stackhorizontal(a,b)

[hagai_sela]

OK, so I guess I'll have to write the player myself.
The .mp4 file format can host 2 streams that will be played concurrently, right?

[poisondeathray]

I think it can host >2 streams, but no existing splitter will allow you to play them concurrently. Only 1 output pin at a time, same with any other container format. I'm guessing you would probably have to write the splitter as well, or intergrate something that plays 2 separate videos in your video player solution (like avisynth)

Why do you need a single container for 2 streams ?

[hagai_sela]

I looked at another thread here, and there seem to be 2 video output pins in graphedit (they may not play concurrently though).

https://forum.videohelp.com/threads/319455-How-to-Demux-avi-with-multiple-video-streams

I wanted one container for both streams so I won't have to write a player, but that's impossible apparently.
I guess I can have more than 1 file, but I need to sync all 4 streams. is that possible?

[poisondeathray]

But if this is a recorder , all you have to do is record the screen (I'm assuming your chat software already displays split screen) . The video & audio is already in sync and recorded properly

I'm imagining something like a conference call with 2 parties on 2 separate computers, maybe your setup is different? Or , why do you need 4 separate streams in 1 container ?

[hagai_sela]

I am not sure I understand what you mean. Do you mean capture the frames from the screen? Doesn't that mean that the client will have to encode them twice? (once for recording and once for sending to the other side).
The setup you are talking about is correct, I am probably missing something here.
I don't need the 4 streams in one container if I have to write a player. I just need them to be synced.

[poisondeathray]

I don't know about your setup. What I imagined was a video conference call. There might be 8 parties for example. The split screen would have 8 "boxes" of people talking, all remotely from 8 different locations. There is only 1 audio track, 1 video track = 1 recording. If you have 12 people, it's still 1 recording. Maybe your setup is different

[hagai_sela]

It's not a conferencing software, it's only one on one. There are 2 audio tracks and 2 video tracks.
I looked at the other thread again. It seems that one of the directshow graphs there is for concurrent playing... Am I wrong?

[poisondeathray]

I think both streams are playing in that graph. This is a 3d video, with L & R streams .

But if you try with mp4, it's not possible with current splitters. You have to manually select 1 stream

What's the difference between 1 on 1 vs. 12 or 8 or 100000? It's still a one on one video conference. If you have 1 on 1, it will be 2 boxes instead of 12. Most video conference software have recording features

[hagai_sela]

OK, I think I may not be using the right terms here. What do you mean by "stream"?
I currently have 4 rtp sessions in the client. outgoing video, outgoing audio, incoming video, incoming audio. I need to record the payload from all of them to a file / files and be able to play them.
I played the file from the other thread in graphedit, it showed 2 video windows and played an audio track. That is basically what I need (plus one audio track).

[poisondeathray]

But this is what video conferencing software does. Incoming/outgoing video & audio from "x" number of sessions is displayed and recording in a grid. So if you have 2 people, you have 2 boxes. If you have 8 people, 8 boxes. Audio is mixed in realtime. So whatever transpires in that session is recorded in 1 video.

Do you need separate streams? Are you doing something that requires editing or access to separate streams?

Maybe I am misunderstanding you ?


You can pm "jagabo" about the graphstudio/ graphedit graph, he's usually around helping people often

[hagai_sela]

I am sorry, but I really don't know how to better explain myself. The number of displayed boxes is not relevant since I want to record the payload from the RTP packets and not from whatever is displayed on the screen.
Again, what is in the other thread's graph is almost what I need. an avi / mp4 with 2 video "streams" and 2 audio "streams" (for lack of a better term).

[poisondeathray]

You would have to program an audio mixer for the graph. It's not possible to play 2 audio streams. PM jagabo

I still think video conferencing software is exactly what you need. It records video and mixes audio in realtime. Don't fixate on the number of boxes, it's only an illustrative example.

[hagai_sela]

I still don't understand what you mean. I am writing a feature for a video client, how can I use a video conferencing software?

[poisondeathray]

I guess why code it when solution already exists?

By "video client" do you mean software or a person? ie. is the video client a "client" or person or company that is hiring you , or is it a piece of software

If you want to write from scratch, have a look at open source code like vmukti

[hagai_sela]

I mean software. The software already exists and I just need to add the recording feature. I don't need to write it from scratch.

[poisondeathray]

I see, but problem for playback is mixing audio. 2 separate audio have to be mixed into 1

[hagai_sela]

I thought of using 2 directshow audio renderers. I changed the graph I used before so it uses a tee filter to split the audio stream and then sends them to the 2 audio renderers. It seems to work.

[poisondeathray]

So everything works then? You did this with AVI , correct ? I don't know of any MP4 splitter that allows access to >1 video or audio stream simultaneously

[hagai_sela]

Yes, I did it with avi. Does it matter which video codec I am using? (I need to use h.264).

[poisondeathray]

Use whatever works

But beware that h264 can have problems in AVI, b-frame decoder lag and green frames

[hagai_sela]

Yeah, that's what I was afraid of. Since the decoder is using almost every h.264 feature I don't think I'll be able to use it.