Unable to Write Streams to File in WebRtc Pipeline and Aim to broadcast to other Peers connected

[kinsleykajiva]

Hello,

This is Question is based on the demo provided !

I'm currently facing an issue with my Webrtc pipeline where I'm unable to save the incoming streams to files. I have provided the relevant code snippets and a description of the problem. Below is the information you need to understand the issue :

public class WebRTCSendRecv {

// skipping some existing code in this class for brevity
private static final String PIPELINE_DESCRIPTION
            = "videotestsrc is-live=true pattern=ball ! videoconvert ! queue ! vp8enc deadline=1 ! rtpvp8pay"
            + " ! queue ! application/x-rtp,media=video,encoding-name=VP8,payload=97 ! webrtcbin. "
            + "audiotestsrc is-live=true wave=sine ! audioconvert ! audioresample ! queue ! opusenc ! rtpopuspay"
            + " ! queue ! application/x-rtp,media=audio,encoding-name=OPUS,payload=96 ! webrtcbin. "
            + "webrtcbin name=webrtcbin bundle-policy=max-bundle stun-server=stun://stun.l.google.com:19302 ";

// skipped other bunch of code

 private final PAD_ADDED onIncomingStream = (element, pad) -> {
        System.out.println("Receiving incoming stream...");
        LOG.info(() -> "Receiving stream! Element: " + element.getName() + " Pad: " + pad.getName());
        if (pad.getDirection() != PadDirection.SRC) {
            return;
        }
        DecodeBin decodeBin = new DecodeBin("decodebin_" + pad.getName());
        decodeBin.connect(onDecodedStream);
        pipe.add(decodeBin);
        decodeBin.syncStateWithParent();
        pad.link(decodeBin.getStaticPad("sink"));
    };
   private final PAD_ADDED onDecodedStream = (element, pad) -> {
        System.out.println("Receiving decoded stream...");
        if (!pad.hasCurrentCaps()) {
            LOG.info("Pad has no current Caps - ignoring");
            return;
        }
        Caps caps = pad.getCurrentCaps();
        LOG.info(() -> "Received decoded stream with caps: " + caps.toString());

        if (caps.isAlwaysCompatible(Caps.fromString("video/x-raw"))) {
            Element q = ElementFactory.make("queue", "videoqueue");
            Element conv = ElementFactory.make("videoconvert", "videoconvert");
            Element filesink = ElementFactory.make("filesink", "videosink");
           // filesink.set("location", "video_output.mp4");
            filesink.set("location", "C:\\\\Users\\\\Kinsl\\\\IdeaProjects\\\\gst1-java-examples\\\\audio_output.mp4");
            pipe.addMany(q, conv, filesink);
            q.syncStateWithParent();
            conv.syncStateWithParent();
            filesink.syncStateWithParent();
            pad.link(q.getStaticPad("sink"));
            q.link(conv);
            conv.link(filesink);
        } else if (caps.isAlwaysCompatible(Caps.fromString("audio/x-raw"))) {
            Element q = ElementFactory.make("queue", "audioqueue");
            Element conv = ElementFactory.make("audioconvert", "audioconvert");
            Element resample = ElementFactory.make("audioresample", "audioresample");
            Element filesink = ElementFactory.make("filesink", "audiosink");
            filesink.set("location", "C:\\\\Users\\\\Kinsl\\\\IdeaProjects\\\\gst1-java-examples\\\\audio_output.wav");
           // filesink.set("location", "audio_output.wav");
            pipe.addMany(q, conv, resample, filesink);
            q.syncStateWithParent();
            conv.syncStateWithParent();
            resample.syncStateWithParent();
            filesink.syncStateWithParent();
            pad.link(q.getStaticPad("sink"));
            q.link(conv);
            conv.link(resample);
            resample.link(filesink);
        }
    };

In the logs I am getting this

Receiving incoming stream...
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$new$21
INFO: Receiving stream! Element: webrtcbin Pad: src_1
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$setupPipeLogging$7
INFO: Pipe state changed from PAUSED to PAUSED
Receiving decoded stream...
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$new$19
INFO: Received decoded stream with caps: audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)2, channel-mask=(bitmask)0x0000000000000003
Receiving incoming stream...
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$new$21
INFO: Receiving stream! Element: webrtcbin Pad: src_0
Receiving decoded stream...
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$new$19
INFO: Received decoded stream with caps: video/x-raw(memory:D3D11Memory), format=(string)NV12, width=(int)640, height=(int)480, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$setupPipeLogging$7
INFO: Pipe state changed from PAUSED to PAUSED
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$setupPipeLogging$7
INFO: Pipe state changed from PAUSED to PLAYING
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$setupPipeLogging$5
SEVERE: Error from source : BaseSrc: [nicesrc0], with code : 1, and message : Internal data stream error.
Jun 01, 2023 11:05:01 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv lambda$setupPipeLogging$5
SEVERE: Error from source : Element: [queue4], with code : 1, and message : Internal data stream error.
Jun 01, 2023 11:05:38 PM org.freedesktop.gstreamer.examples.WebRTCSendRecv$1 onClose
INFO: WebSocket onClose : 1011 : keepalive ping timeout

Currently, I am able to create the audio_output.wav file successfully. However, the file appears to be corrupted after the session ends. My main issue is not with the web app abruptly ending the session with white or sine noise. Additionally, I am unable to write to the disk for both the video and audio files. I kindly request your assistance in resolving these issues.

Additional Information:

The audio_output.wav file seems to be corrupted after the session ends. My primary concern is the inability to write both the video and audio files to disk.

As a newcomer to GStreamer, I am unsure if the defined pipeline is necessary solely for receiving a stream from a peer connection that is broadcasting the stream. I attempted to run the pipeline without the defined string, but it appears that it doesn't trigger the WebRTC code, so I have kept it.

Furthermore, I would appreciate some guidance on how to share a broadcast with other peers who wish to stream or receive the broadcast from a live host streaming to the server.

Please provide your assistance in resolving these issues and any additional tips or guidance you can offer.