Sending h264 frames

[HigerSkill]

I'm new to WebRTC and video encoding. I'm trying to encode images in cv::Mat to h264 and send them via WebRTC to the frontend, similar to the media-sender example. I'm facing a problem where, although I encode the image and set up the h264 packetizer, when I send the video, there are no errors, but no video appears on the frontend.

        rtc::Description::Video media("video", rtc::Description::Direction::SendOnly);
        media.addH264Codec(102);
        constexpr rtc::SSRC ssrc = 42;
        media.addSSRC(ssrc, "video");
        video_track = peer->addTrack(media);
        auto rtpConfig = std::make_shared<rtc::RtpPacketizationConfig>(ssrc, "video", 102,
                                                                          rtc::H264RtpPacketizer::defaultClockRate);
        auto packetizer = std::make_shared<rtc::H264RtpPacketizer>(rtc::NalUnit::Separator::LongStartSequence, rtpConfig);
        auto srReporter = std::make_shared<rtc::RtcpSrReporter>(rtpConfig);
        packetizer->addToChain(srReporter);
        auto nackResponder = std::make_shared<rtc::RtcpNackResponder>();
        packetizer->addToChain(nackResponder);
        video_track->setMediaHandler(packetizer);

Why might my RTP packets be ignored?

I found related problem in discussions #699
I have the same problem, but there no answers

[vvalentinich]

Do you encode to h264 properly (you can write raw h264 frames to a file for testing)? Do you observe suspicious records in libdatachannel's log? If a browser is used on receivers side, do you see something wrong from webrtc's details (Chrome - chrome://webrtc-internals; Edge - edge://webrtc-internals; Firefox - about:webrtc)?

[HigerSkill]

I see that traffic is passing and packets are being sent, but I don't see where to look for bitrate and video parameters

Which timestamps should I check? Do I need to do something with the timestamp when encoding?

[vvalentinich]

I meant things like rtpConfig->timestamp (from streamer example). As for pictures, I would look carefully at "Stats graphs for inbound-rtp (kind=video...".

[HigerSkill]

I didn't change anything in the example, I just output the rtpConfig->timestamp, for some reason it's incorrect.

Timestamp: 3437318314
Timestamp: 3321433375
Timestamp: 3437319274
Timestamp: 3437320234
Timestamp: 3321436375
Timestamp: 3437321194

I manually set rtpConfig->startTimestamp to the current time, it didn't help either.

For the original frames the timestamps are also strange, but they work:

Timestamp: 4143417708
Timestamp: 4143418668
Timestamp: 2680257504
Timestamp: 4143419628
Timestamp: 4143420588
Timestamp: 2680260504
Timestamp: 4143421548
Timestamp: 2680263504
Timestamp: 4143422508
Timestamp: 4143423468

[paullouisageneau]

The timestamps look fine: the difference between two audio timestamps is 960 (20ms at 48 KHz), and the difference between video timestamps is 3000 (around 31.25ms at 96KHz).

As you can see on the Chrome graph, it receives packets but no valid frames. This looks like an encoding issue. How do you bundle NAL units together into frames? It has to be consistent with the packetizer separator setting.

[HigerSkill]

@paullouisageneau @vvalentinich Thanks for the help, the problem was indeed in NalUnit::Separator and it was necessary to correctly assemble the frame