Extrating a single image frame from BC Media

[entropin]

Hi! I have now added Tenserflow and face recognition to my Neolink project, but right now am connecting to the camera, initiating the full video stream then using GStreamer to export a jpeg from the stream that I use for the face detection.

But am just wanna check if anyone sees a way to optimize, lets say I just capture the first BC Media pack and then disconnect Can I use some trick to use this data to create an image frame? Do I need a Gstreamer for this? Alternatively can feed GStreamer just a frame/byte array? :D GStreamer and I is still not the best of friends :D

[QuantumEntangledAndy]

I think perhaps connect and then pull the first BCMedia packet of the I frame that should be all you need.

[QuantumEntangledAndy]

How do you get the image into tensor flow? It might already be able to handle a raw h264 frame as is.

[entropin]

hehe, that's probably smarter.. Am using https://github.com/image-rs/image to convert it generic pixel values, but am sure I can get this from the war data as well, I might not need a jpeg. Will look into that.

[QuantumEntangledAndy]

I'm not an expert on h264 internals and the specification document is like 750 pages long. So might be best in the long run to put it through gstreamer as a stream just so it's done right.

I'm not sure how your setup is working is it in rust? Might want to look at both the appsrc code we use for the video stream and the appsink code we use talk. The way I see it you need appsrc for the frames from the camera and appsink for the decoded frames send off to tensor flow.

[entropin]

Cool, so I got everything working. I just have one question regarding reading the appsink. We use a callback for this that is called on every "new_sample" What unit is this? Can I control the buffer size am waiting for here? Like we can control the buffer input to the appsrc? "gstreamer::Buffer::with_size()"

I don't need to understand this now, and am not sure if the question even makes sense but am thinking if I like to implement video filters down the line, a way of balancing the size of the input and output buffers would be nice. but am not sure, working with realtime streams hearts my head :D

[QuantumEntangledAndy]

I am not actually sure 90% of that code for appsink just comes from this example. Maybe check out the docs there and the other examples