feat: add AVSynchronizer #1168
feat: add AVSynchronizer #1168
Conversation
|
| ) | ||
|
|
||
| async for video_frame, audio_frame in video_generator: | ||
| av_sync.push(video_frame) |
There was a problem hiding this comment.
| av_sync.push(video_frame) | |
| await av_sync.push(video_frame) |
There was a problem hiding this comment.
for docs, maybe also clarify that it could be pushed from different tasks?
There was a problem hiding this comment.
Yeah, I'll create an example to test and demonstrate the use case.
There was a problem hiding this comment.
this example is awesome! it'll be super useful. It might be useful to include an example file too.. I'll see about moving some of these examples to its own organized repo in livekit-examples org
There was a problem hiding this comment.
Yes, or I was thinking of another example of reading video and audio from the client and sending it back, or an audio wave with a graph so we can easily see the synchronization while not needing an additional mp4 file.
| ) | ||
| self._capture_video_task = asyncio.create_task(self._capture_video()) | ||
|
|
||
| async def push(self, frame: Union[rtc.VideoFrame, rtc.AudioFrame]) -> None: |
There was a problem hiding this comment.
Should this wait to have at least one AudioFrame and one VideoFrame to start the capture? So we synchronize the start of the stream as well
There was a problem hiding this comment.
I think no, the agent can stream only video frames if there is no audio at the beginning. The user only need to make sure when there is a audio for playing, the first audio frame is pushed at the same time with the corresponding video frame.
There was a problem hiding this comment.
we need to make the queue_size smaller enough for that case, e.g. 100ms or even less.
|
|
||
| async def push(self, frame: Union[rtc.VideoFrame, rtc.AudioFrame]) -> None: | ||
| if isinstance(frame, rtc.AudioFrame): | ||
| # TODO: test if frame duration is too long |
There was a problem hiding this comment.
Yes, I think we need to keep track of the pushed audio.
|
Any idea on how we could handle interruptions with it? |
First if queue_size_ms is small, the whole process is synced and the pushing speed is controlled by the playing. so for interruption we just need to stop the audio pushing like the current pipeline agent. I can also add a |
|
@longcw I'm also wondering, perhaps it'd be better for the core sync utils to go into our Python SDK? https://github.com/livekit/python-sdks/ since it doesn't require agents here. |
I agree. I'll finish this with some more testing and documents, then maybe create a pr for the sdk? |
|
@davidzhao The PR for the python sdk livekit/python-sdks#324. I think we can close this one after that. |
No description provided.