Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve API ergonomics, updated examples and readme #91

Merged
merged 7 commits into from
Nov 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 56 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,72 @@

[![pypi-v](https://img.shields.io/pypi/v/livekit.svg)](https://pypi.org/project/livekit/)

# 📹🎙️🐍 Python Client SDK for LiveKit
# 📹🎙️🐍 Python SDK for LiveKit

The Livekit Python Client provides a convenient interface for integrating Livekit's real-time video and audio capabilities into your Python applications. With this library, developers can easily leverage Livekit's WebRTC functionalities, allowing them to focus on building their AI models or other application logic without worrying about the complexities of WebRTC.
<!--BEGIN_DESCRIPTION-->

Official LiveKit documentation: https://docs.livekit.io/
The LiveKit Python SDK provides a convenient interface for integrating LiveKit's real-time video and audio capabilities into your Python applications. With it, developers can easily leverage LiveKit's WebRTC functionalities, allowing them to focus on building their AI models or other application logic without worrying about the complexities of WebRTC.

## Installation
<!--END_DESCRIPTION-->

This repo contains two packages

- [livekit](https://pypi.org/project/livekit/): Real-time SDK for connecting to LiveKit as a participant
- [livekit-api](https://pypi.org/project/livekit-api/): Access token generation and server APIs

## Using Server API

RTC Client:
```shell
$ pip install livekit
$ pip install livekit-api
```

### Generating an access token

```python
from livekit import api
import os

token = api.AccessToken(os.getenv('LIVEKIT_API_KEY'), os.getenv('LIVEKIT_API_SECRET')) \
.with_identity("python-bot") \
.with_name("Python Bot") \
.with_grants(api.VideoGrants(
room_join=True,
room="my-room",
)).to_jwt()
```

API / Server SDK:
### Creating a room

RoomService uses asyncio and aiohttp to make API calls. It needs to be used with an event loop.

```python
from livekit import api
import asyncio

async def main():
room_service = api.RoomService(
'http://localhost:7880',
'devkey',
'secret',
)
room_info = await room_service.create_room(
api.room.CreateRoomRequest(name="my-room"),
)
print(room_info)
results = await room_service.list_rooms(api.room.ListRoomsRequest())
print(results)
await room_service.aclose()

asyncio.get_event_loop().run_until_complete(main())
```

## Using Real-time SDK

```shell
$ pip install livekit-api
$ pip install livekit
```

## Connecting to a room
### Connecting to a room

```python
from livekit import rtc
Expand Down Expand Up @@ -64,21 +111,6 @@ async def main():
print("track publication: %s", publication.sid)
```

## Create a new access token

```python
from livekit import api

token = api.AccessToken("API_KEY", "SECRET_KEY")
token = AccessToken()
jwt = (
token.with_identity("user1")
.with_name("user1")
.with_grants(VideoGrants(room_join=True, room="room1"))
.to_jwt()
)
```

## Examples

- [Facelandmark](https://github.com/livekit/client-sdk-python/tree/main/examples/face_landmark): Use mediapipe to detect face landmarks (eyes, nose ...)
Expand Down
20 changes: 16 additions & 4 deletions examples/basic_room.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
import logging
from signal import SIGINT, SIGTERM
from typing import Union
import os

from livekit import rtc
from livekit import api, rtc

URL = "ws://localhost:7880"
TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE5MDY2MTMyODgsImlzcyI6IkFQSVRzRWZpZFpqclFvWSIsIm5hbWUiOiJuYXRpdmUiLCJuYmYiOjE2NzI2MTMyODgsInN1YiI6Im5hdGl2ZSIsInZpZGVvIjp7InJvb20iOiJ0ZXN0Iiwicm9vbUFkbWluIjp0cnVlLCJyb29tQ3JlYXRlIjp0cnVlLCJyb29tSm9pbiI6dHJ1ZSwicm9vbUxpc3QiOnRydWV9fQ.uSNIangMRu8jZD5mnRYoCHjcsQWCrJXgHCs0aNIgBFY" # noqa
# ensure LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET are set


async def main(room: rtc.Room) -> None:
Expand Down Expand Up @@ -127,7 +127,19 @@ def on_reconnecting() -> None:
def on_reconnected() -> None:
logging.info("reconnected")

await room.connect(URL, TOKEN)
token = (
api.AccessToken()
.with_identity("python-bot")
.with_name("Python Bot")
.with_grants(
api.VideoGrants(
room_join=True,
room="my-room",
)
)
.to_jwt()
)
await room.connect(os.getenv("LIVEKIT_URL"), token)
logging.info("connected to room %s", room.name)
logging.info("participants: %s", room.participants)

Expand Down
59 changes: 34 additions & 25 deletions examples/face_landmark/face_landmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,9 @@
from mediapipe import solutions
from mediapipe.framework.formats import landmark_pb2

from livekit import rtc
from livekit import api, rtc

URL = "ws://localhost:7880"
TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE5MDY2MTMyODgsImlzcyI6IkFQSVRzRWZpZFpqclFvWSIsIm5hbWUiOiJuYXRpdmUiLCJuYmYiOjE2NzI2MTMyODgsInN1YiI6Im5hdGl2ZSIsInZpZGVvIjp7InJvb20iOiJ0ZXN0Iiwicm9vbUFkbWluIjp0cnVlLCJyb29tQ3JlYXRlIjp0cnVlLCJyb29tSm9pbiI6dHJ1ZSwicm9vbUxpc3QiOnRydWV9fQ.uSNIangMRu8jZD5mnRYoCHjcsQWCrJXgHCs0aNIgBFY" # noqa
# ensure LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET are set

tasks = set()

Expand All @@ -30,10 +29,41 @@
running_mode=VisionRunningMode.VIDEO,
)

# from https://github.com/googlesamples/mediapipe/blob/main/examples/face_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Face_Landmarker.ipynb

async def main(room: rtc.Room) -> None:
video_stream = None

@room.on("track_subscribed")
def on_track_subscribed(track: rtc.Track, *_):
if track.kind == rtc.TrackKind.KIND_VIDEO:
nonlocal video_stream
if video_stream is not None:
# only process the first stream received
return

print("subscribed to track: " + track.name)
video_stream = rtc.VideoStream(track)
task = asyncio.create_task(frame_loop(video_stream))
tasks.add(task)
task.add_done_callback(tasks.remove)

token = (
api.AccessToken()
.with_identity("python-bot")
.with_name("Python Bot")
.with_grants(
api.VideoGrants(
room_join=True,
room="my-room",
)
)
)
await room.connect(os.getenv("LIVEKIT_URL"), token.to_jwt())
print("connected to room: " + room.name)


def draw_landmarks_on_image(rgb_image, detection_result):
# from https://github.com/googlesamples/mediapipe/blob/main/examples/face_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Face_Landmarker.ipynb
face_landmarks_list = detection_result.face_landmarks

# Loop through the detected faces to visualize.
Expand Down Expand Up @@ -111,27 +141,6 @@ async def frame_loop(video_stream: rtc.VideoStream) -> None:
cv2.destroyAllWindows()


async def main(room: rtc.Room) -> None:
video_stream = None

@room.on("track_subscribed")
def on_track_subscribed(track: rtc.Track, *_):
if track.kind == rtc.TrackKind.KIND_VIDEO:
nonlocal video_stream
if video_stream is not None:
# only process the first stream received
return

print("subscribed to track: " + track.name)
video_stream = rtc.VideoStream(track)
task = asyncio.create_task(frame_loop(video_stream))
tasks.add(task)
task.add_done_callback(tasks.remove)

await room.connect(URL, TOKEN)
print("connected to room: " + room.name)


if __name__ == "__main__":
logging.basicConfig(
level=logging.INFO,
Expand Down
62 changes: 38 additions & 24 deletions examples/publish_hue.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,51 @@
import asyncio
import colorsys
import logging
import os
from signal import SIGINT, SIGTERM

import numpy as np
from livekit import rtc

URL = "ws://localhost:7880"
TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE5MDY2MTMyODgsImlzcyI6IkFQSVRzRWZpZFpqclFvWSIsIm5hbWUiOiJuYXRpdmUiLCJuYmYiOjE2NzI2MTMyODgsInN1YiI6Im5hdGl2ZSIsInZpZGVvIjp7InJvb20iOiJ0ZXN0Iiwicm9vbUFkbWluIjp0cnVlLCJyb29tQ3JlYXRlIjp0cnVlLCJyb29tSm9pbiI6dHJ1ZSwicm9vbUxpc3QiOnRydWV9fQ.uSNIangMRu8jZD5mnRYoCHjcsQWCrJXgHCs0aNIgBFY" # noqa
from livekit import api, rtc

WIDTH, HEIGHT = 1280, 720


# ensure LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET are set


async def main(room: rtc.Room):
token = (
api.AccessToken()
.with_identity("python-publisher")
.with_name("Python Publisher")
.with_grants(
api.VideoGrants(
room_join=True,
room="my-room",
)
)
.to_jwt()
)
url = os.getenv("LIVEKIT_URL")
logging.info("connecting to %s", url)
try:
await room.connect(url, token)
logging.info("connected to room %s", room.name)
except rtc.ConnectError as e:
logging.error("failed to connect to the room: %s", e)
return

# publish a track
source = rtc.VideoSource(WIDTH, HEIGHT)
track = rtc.LocalVideoTrack.create_video_track("hue", source)
options = rtc.TrackPublishOptions()
options.source = rtc.TrackSource.SOURCE_CAMERA
publication = await room.local_participant.publish_track(track, options)
logging.info("published track %s", publication.sid)

asyncio.ensure_future(draw_color_cycle(source))


async def draw_color_cycle(source: rtc.VideoSource):
argb_frame = rtc.ArgbFrame.create(rtc.VideoFormatType.FORMAT_ARGB, WIDTH, HEIGHT)
arr = np.frombuffer(argb_frame.data, dtype=np.uint8)
Expand Down Expand Up @@ -42,26 +76,6 @@ async def draw_color_cycle(source: rtc.VideoSource):
await asyncio.sleep(1 / 30 - code_duration)


async def main(room: rtc.Room):
logging.info("connecting to %s", URL)
try:
await room.connect(URL, TOKEN)
logging.info("connected to room %s", room.name)
except rtc.ConnectError as e:
logging.error("failed to connect to the room: %s", e)
return

# publish a track
source = rtc.VideoSource(WIDTH, HEIGHT)
track = rtc.LocalVideoTrack.create_video_track("hue", source)
options = rtc.TrackPublishOptions()
options.source = rtc.TrackSource.SOURCE_CAMERA
publication = await room.local_participant.publish_track(track, options)
logging.info("published track %s", publication.sid)

asyncio.ensure_future(draw_color_cycle(source))


if __name__ == "__main__":
logging.basicConfig(
level=logging.INFO,
Expand Down
56 changes: 35 additions & 21 deletions examples/publish_wave.py
Original file line number Diff line number Diff line change
@@ -1,42 +1,41 @@
import asyncio
import logging
from signal import SIGINT, SIGTERM
import os

import numpy as np
from livekit import rtc

URL = "ws://localhost:7880"
TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE5MDY2MTMyODgsImlzcyI6IkFQSVRzRWZpZFpqclFvWSIsIm5hbWUiOiJuYXRpdmUiLCJuYmYiOjE2NzI2MTMyODgsInN1YiI6Im5hdGl2ZSIsInZpZGVvIjp7InJvb20iOiJ0ZXN0Iiwicm9vbUFkbWluIjp0cnVlLCJyb29tQ3JlYXRlIjp0cnVlLCJyb29tSm9pbiI6dHJ1ZSwicm9vbUxpc3QiOnRydWV9fQ.uSNIangMRu8jZD5mnRYoCHjcsQWCrJXgHCs0aNIgBFY" # noqa
from livekit import rtc, api

SAMPLE_RATE = 48000
NUM_CHANNELS = 1


async def publish_frames(source: rtc.AudioSource, frequency: int):
amplitude = 32767 # for 16-bit audio
samples_per_channel = 480 # 10ms at 48kHz
time = np.arange(samples_per_channel) / SAMPLE_RATE
total_samples = 0
audio_frame = rtc.AudioFrame.create(SAMPLE_RATE, NUM_CHANNELS, samples_per_channel)
audio_data = np.frombuffer(audio_frame.data, dtype=np.int16)
while True:
time = (total_samples + np.arange(samples_per_channel)) / SAMPLE_RATE
sine_wave = (amplitude * np.sin(2 * np.pi * frequency * time)).astype(np.int16)
np.copyto(audio_data, sine_wave)
await source.capture_frame(audio_frame)
total_samples += samples_per_channel
# ensure LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET are set


async def main(room: rtc.Room) -> None:
@room.on("participant_disconnected")
def on_participant_disconnect(participant: rtc.Participant, *_):
logging.info("participant disconnected: %s", participant.identity)

logging.info("connecting to %s", URL)
token = (
api.AccessToken()
.with_identity("python-publisher")
.with_name("Python Publisher")
.with_grants(
api.VideoGrants(
room_join=True,
room="my-room",
)
)
.to_jwt()
)
url = os.getenv("LIVEKIT_URL")

logging.info("connecting to %s", url)
try:
await room.connect(
URL,
TOKEN,
url,
token,
options=rtc.RoomOptions(
auto_subscribe=True,
),
Expand All @@ -57,6 +56,21 @@ def on_participant_disconnect(participant: rtc.Participant, *_):
asyncio.ensure_future(publish_frames(source, 440))


async def publish_frames(source: rtc.AudioSource, frequency: int):
amplitude = 32767 # for 16-bit audio
samples_per_channel = 480 # 10ms at 48kHz
time = np.arange(samples_per_channel) / SAMPLE_RATE
total_samples = 0
audio_frame = rtc.AudioFrame.create(SAMPLE_RATE, NUM_CHANNELS, samples_per_channel)
audio_data = np.frombuffer(audio_frame.data, dtype=np.int16)
while True:
time = (total_samples + np.arange(samples_per_channel)) / SAMPLE_RATE
sine_wave = (amplitude * np.sin(2 * np.pi * frequency * time)).astype(np.int16)
np.copyto(audio_data, sine_wave)
await source.capture_frame(audio_frame)
total_samples += samples_per_channel


if __name__ == "__main__":
logging.basicConfig(
level=logging.INFO,
Expand Down
Loading
Loading