Video support

[Hollymingyi]

dear author,
could u help to telll me how to pull video meida from sip server and push video media to sip server? or do u have the plan to support video with sip module?

[Sean-Der]

Hi @Hollymingyi

would you mind explaining what you are doing more?

what SIP client are you using?

Do you need the SIP server to mix video, or can it be delivered as separate media sections?

[Hollymingyi]

hi sean,
what i want is that when i call into livekit room via freeswitch , i can communicate with livekit endpoint with video and audio together ,but now only audio support. Then i read the livekit/sip source code, i found that sdp message was maked like below：
[]*sdp.MediaDescription{
{
MediaName: sdp.MediaName{
Media: "audio",
Port: sdp.RangedPort{Value: rtpListenerPort},
Protos: []string{"RTP", "AVP"},
Formats: []string{"0", "101"},
},
Attributes: []sdp.Attribute{
{Key: "rtpmap", Value: "0 PCMU/8000"},
{Key: "rtpmap", Value: "101 telephone-event/8000"},
{Key: "fmtp", Value: "101 0-16"},
{Key: "ptime", Value: "20"},
{Key: "maxptime", Value: "150"},
{Key: "sendrecv"},
},
},
}
l
it means only audio support。

after all ,i want sip server to mix video into likekit room ,and separate media back to freeswitch.

[Sean-Der]

You are trying to connect Freeswitch + LiveKit?

Why are you running both servers? Would it be possible to only run one? What features are missing from either?

It would probably be more effective to use egress. That has composition and has more flexibility. If you can describe exactly what you are trying to build we would love to help!

[Hollymingyi]

ah ,if use ingress and egress to connect freeswitch and livekit , for example ,rtmp protocol ,the latency is more than 9s,its not suitable for our applications.    
right now , we can use sip softphone to connect livekit/sip and communicate with audio good.
but my hope is that we can use sip softphone to connect livekit/sip ,then we can communicate each other with audio and video. so if livekit/sip support video communication will be helpful for us.

[Hollymingyi]

[dennwc]

The most important detail is missing from the diagram - who is making SIP video call? How it produces and consumes video streams?

Sean is right to point out that passing m=video won't be enough. If device that sends SIP request can only handle one incoming video stream, LiveKit server would need to compose videos from multiple participants from the room. Thus, you would still need Egress for that room. Or active speaker detection at least (to select one stream).

[Hollymingyi]

hi dennwc，
yes,you r right. only m=video is not enough,LiveKit server would need to compose videos from multiple participants from the room,that's what i desire to do. As your suggestion，if we use egress to handle our problem, how should i to get egress and push to sip server, i would be happy to have a try， anyway thanks

[dennwc]

OK, got it. To emphasize, we don't have any plans to implement video for SIP.

However, if this turns out to be simple enough (e.g. pick one participant/egress, stream from it), then I might experiment with it when I have some free time.

There might be additional constraints here. For audio, we convert Ulaw <-> Opus in SIP server. If there's similar mismatch between video formats in SIP and LK, that will be a blocker. We definitely won't convert video in SIP server.

[Hollymingyi]

ah ,thanks anyway, our application wanna have video communication between sip and livekit, so i would prefer to get egress media from lk and then push it back to sip server (the sip invite have sdp,) via mediaconn. could u help to give me a plan or workflow to coding them?

[davidzhao]

Closing as we do not plan to support video with SIP.

[Hollymingyi]

hi david，
thanks anyway, when i publish h264 video into livekit ,the video is very fuzzy ,and the log like below:

2023-12-22T14:45:02.275+0800 INFO livekit.pub.sfu buffer/rtpstats_receiver.go:306 received sender report, out-of-order, resetting{"room": "91233445", "roomID": "RM_QCzP286EekLz", "participant": "Phone 89650050", "pID": "PA_L7PCC44n6mYa", "remote": false, "trackID": "TR_VCeqyecBzUPLHs", "relayed": false, "mime": "video/h264", "layer": 0, "last": "ntp: 2023-12-22 06:45:01.273902416 +0000 UTC, rtp: 1954390734, extRtp: 1954390734, at: 2023-12-22 14:45:01.274211609 +0800 CST m=+2591.952357079", "current": "ntp: 2023-12-22 06:45:02.274554252 +0000 UTC, rtp: 1954390045, extRtp: 1954390045, at: 2023-12-22 14:45:02.275287018 +0800 CST m=+2592.953432502", "count": 300}

is it something wrong?