Help needed with decoding H264 into frames

[Valkhes]

Hello, I'm asking here because I can't figure what I'm supposed to do with the library.

I'm trying to change a code I have in python using PyAV (which is a Python binding of FFMPEG) to Go, using go-astiav, as it seems to be one of the only maintained binding of FFMPEG in go.

I'm trying to retrieve H264 stream from a socket sended by a scrcpy-server (I know that some people have been using the lib here for the same reason). I read these one :

#45
#13

But I don't exactly understand how I need to use go-astiav. I'm quite a beginner with H264 and codec, so this is a bit hard for me to understand.

In python, what I'm doing is :

#Receive from the socket byte stream
raw_h264 = self.video_socket.recv(0x10000)

#Split up a byte stream into list of Packet
all_packets = self.codec.parse(raw_h264)

packets_index = 0
decoded_frame = None

while not decoded_frame:
    #Iterating over Packet to find a frame
    decoded_frame = codec.decode(all_packets[packets_index])
    packets_index += 1

if decoded_frame:
    #do something with frame decoded

In go, what I've tried to do for the moment :

  fmt.Println("----> Creating stream")
  s := &stream{inputStream: nil}

  fmt.Println("----> Find IDH264 decoder")
  #Find decoder for H264
  if s.decCodec = astiav.FindDecoder(astiav.CodecIDH264); s.decCodec == nil {
    fmt.Println("main: codec is nil")
    return
  }

  // Alloc codec context
  if s.decCodecContext = astiav.AllocCodecContext(s.decCodec); s.decCodecContext == nil {
    fmt.Println("main: codec context is nil")
    return
  }
  defer s.decCodecContext.Free()

  // Open codec context
  if err := s.decCodecContext.Open(s.decCodec, nil); err != nil {
    fmt.Println("main: opening codec context failed: %w", err)
    return
  }

  // Alloc packet
  pkt := astiav.AllocPacket()
  defer pkt.Free()

  // Alloc frame
  f := astiav.AllocFrame()
  defer f.Free()

  //get bytes stream from socket

  buf := make([]byte, 0x10000)
  _, err = videoSocket.Read(buf)
  if err != nil {
    fmt.Println("unable to read from socket")
  }
  fmt.Println("Got H264 byte stream data")

  //Transform into Packet
  if err := pkt.FromData(buf); err != nil {
    fmt.Println(fmt.Errorf("main: packet header load failed: %w", err))
    return
  }
  fmt.Println("pkt size is : %v", pkt.Size())

  //What to do know ? How to decode ?

At this step, I'm pretty sure I managed to parse the byte stream to a Packet list (but I'm not really sure, I would need confirmation), but I don't understand what I'm supposed to do after if I want to find a Frame by decoding the packets. The naming are different from PyAV here, so I'm a bit lost. I was expecting to find a decode() function from the codec, as it knows it's H264 packets, but it seems it's not the case.

Can everybody help me about this ? I'm sure I'm not that far...

[Valkhes]

I saw this code :

// Loop through packets
	for {
		// Read frame
		if err := inputFormatContext.ReadFrame(pkt); err != nil {
			if errors.Is(err, astiav.ErrEof) {
				break
			}
			log.Fatal(fmt.Errorf("main: reading frame failed: %w", err))
		}

		// Get stream
		s, ok := streams[pkt.StreamIndex()]
		if !ok {
			continue
		}

		// Send packet
		if err := s.decCodecContext.SendPacket(pkt); err != nil {
			log.Fatal(fmt.Errorf("main: sending packet failed: %w", err))
		}

		// Loop
		for {
			// Receive frame
			if err := s.decCodecContext.ReceiveFrame(f); err != nil {
				if errors.Is(err, astiav.ErrEof) || errors.Is(err, astiav.ErrEagain) {
					break
				}
				log.Fatal(fmt.Errorf("main: receiving frame failed: %w", err))
			}

			// Do something with decoded frame
			log.Printf("new frame: stream %d - pts: %d", pkt.StreamIndex(), f.Pts())
		}
	}

Which seems to be something I need to use, it really looks like it's iterating over the packets to find a frame.
I don't understand inputFormatContext here, how to replace it, as it seems it's not what I need.

// Alloc input format context
	inputFormatContext := astiav.AllocFormatContext()
	if inputFormatContext == nil {
		log.Fatal(errors.New("main: input format context is nil"))
	}
	defer inputFormatContext.Free()

	// Open input
	if err := inputFormatContext.OpenInput(*input, nil, nil); err != nil {
		log.Fatal(fmt.Errorf("main: opening input failed: %w", err))
	}
	defer inputFormatContext.CloseInput()

	// Find stream info
	if err := inputFormatContext.FindStreamInfo(nil); err != nil {
		log.Fatal(fmt.Errorf("main: finding stream info failed: %w", err))
	}

The inputFormatContext is taking an input which seems to be a string, so this is not really what I need I suppose.

[asticode]

@Valkhes first off, thanks for using go-astiav ❤️

The first thing I'd recommend (maybe you've done it already) is checking out the examples especially the Demuxing/Decoding one that seems to fit your needs.

Before digging deeper in your problem, could you tell me whether you can receive the socket stream only via videoSocket.Read(buf)?

[asticode]

I think I'll need to create an example for this part as well 🤔

In fact you need to use both .GuessImageFormat() to guess the image format and then provide the returned image to .ToImage() to actually copy the content. That would be better than creating your own image.

image, err := .Data().GuessImageFormat()
err = f.Data().ToImage(image)

[Valkhes]

Ah yes, tested it, that was it

Indeed that was implicit, GuessImageFormat is here only to define the image size/format and create it with the good ratio, and you need to fill it using ToImage after, it make sense.
Not sure if an exemple is necessary, it surely make sense for a lot of people, I'm just quite a beginner in this field so I still need to understand exactly what I need to do each time.

[asticode]

FYI I've added an example on how to process frame data in the demuxing/decoding example 👍

[Valkhes]

Hello !

Everything is working like a charm with the exemple you gave, I managed to finish my decoding pipeline with everything we discussed.

I wanted to thank you again for everything!

I'm closing the discussion 👍

[asticode]

❤️