MediaPipe Unity Plugin

This is a Unity (2020.3.8f1) Plugin to use MediaPipe (0.8.6).

Prerequisites

To use this plugin, you need to build native libraries for the target platforms (Desktop/UnityEditor, Android, iOS). If you'd like to build them on your machine, below commands/tools/libraries are required (not required if you use Docker).

Python >= 3.9.0
Bazel >= 3.7.2, (< 4.0.0 for iOS)
GCC/G++ >= 8.0.0 (Linux, macOS), < 11.0
NuGet

Platforms

Example Graphs

	Android	iOS	Linux (GPU)	Linux (CPU)	macOS	Windows
Face Detection	✅	✅	✅	✅	✅	✅
Face Mesh	✅	✅	✅	✅	✅	✅
Iris	✅	✅	✅	✅	✅	✅
Hands	✅	✅	✅	✅	✅	✅
Pose	✅	✅	✅	✅	✅	✅
Holistic (with iris)	✅	✅	✅	✅	✅	✅
Hair Segmentation	✅	✅	✅	✅	✅	✅
Object Detection	✅	✅	✅	✅	✅	✅
Box Tracking	✅	✅	✅	✅	✅	✅
Instant Motion Tracking	✅	✅	✅	✅	✅	✅
Objectron	✅	✅	✅	✅	✅	✅
KNIFT

Installation Guide

Run commands at the project root if not specified otherwise.
Also note that you need to build native libraries for Desktop CPU or GPU to run this plugin on UnityEditor.

As long as Docker is available on your environment, Docker is always preferable.
Docker is only used to build native libraries, not at runtime, so there is no impact on performance.

Docker for Linux
Linux
Docker for Windows
Windows
macOS

Docker for Linux

Build a Docker image

docker build -t mediapipe_unity:latest . -f docker/linux/x86_64/Dockerfile

# Above command may fail depending on glibc version installed to your host machine.
# cf. https://serverfault.com/questions/1052963/pacman-doesnt-work-in-docker-image
#
# In that case, apply a patch.
#
#   git apply docker/linux/x86_64/glibc.patch

# You can specify MIRROR_COUNTRY to increase download speed
docker build --build-arg RANKMIRROS=true --build-arg MIRROR_COUNTRY=FR,GB -t mediapipe_unity:latest . -f docker/linux/x86_64/Dockerfile

Run a Docker container

# Run with `Packages` directory mounted to the container
docker run \
    --mount type=bind,src=$PWD/Packages,dst=/home/mediapipe/Packages \
    --mount type=bind,src=$PWD/Assets,dst=/home/mediapipe/Assets \
    -it mediapipe_unity:latest

Run build command inside the container

# Build native libraries for Desktop CPU.
# Note that you need to specify `--opencv=cmake`, because OpenCV is not installed to the container.
python build.py build --desktop cpu --opencv=cmake -v

# Build native libraries for Desktop GPU and Android
python build.py build --desktop gpu --android arm64 --opencv=cmake -v

If the command finishes successfully, required files will be installed to your host machine.

Linux

(Optional, Deprecated) Install OpenCV and FFmpeg You can skip this if you plan to build OpenCV with MediaPipe (see Build Command for more details).

By default, it is assumed that OpenCV 3 is installed under /usr (e.g. /usr/lib/libopencv_core.so).
If your version or path is different, please edit third_party/opencv_linux.BUILD and WORKSPACE.

For example, if you use ArchLinux and opencv3-opt, OpenCV 3 is installed under /opt/opencv3.
In this case, your WORKSPACE will look like this.

new_local_repository(
    name = "linux_opencv",
    build_file = "@//third_party:opencv_linux.BUILD",
    path = "/opt/opencv3",
)

On the other hand, if you use Ubuntu, probably OpenCV's shared libraries is installed under /usr/lib/x86_64-linux-gnu/.
In that case, your opencv_linux.BUILD would be like this.

cc_library(
    name = "opencv",
    srcs = glob(
        [
            "lib/x86_64-linux-gnu/libopencv_core.so",
            "lib/x86_64-linux-gnu/libopencv_calib3d.so",
            "lib/x86_64-linux-gnu/libopencv_features2d.so",
            "lib/x86_64-linux-gnu/libopencv_highgui.so",
            "lib/x86_64-linux-gnu/libopencv_imgcodecs.so",
            "lib/x86_64-linux-gnu/libopencv_imgproc.so",
            "lib/x86_64-linux-gnu/libopencv_video.so",
            "lib/x86_64-linux-gnu/libopencv_videoio.so",
        ],
    ),
    ...
)

Install Bazelisk and NuGet, and ensure you can run them
```
bazel --version
nuget
```

Install numpy

pip install numpy --user

# or
# pip3 install numpy --user

(Optional) Install Android SDK and Android NDK, and set environment variables

# bash
export ANDROID_HOME=/path/to/SDK
# ATTENTION!: Currently Bazel does not support NDK r22, so use NDK r21 instead.
export ANDROID_NDK_HOME=/path/to/ndk/21.4.7075529

Run build command

Docker for Windows

Desktop/UnityEditor

Switch to windows containers

Note that Hyper-V backend is required (that is, Windows 10 Home is not supported).
Build a Docker image
```
docker build -t mediapipe_unity:windows . -f docker/windows/x86_64/Dockerfile
```
This process will hang when MSYS2 is being installed.
If this issue occurs, remove C:\ProgramData\Docker\tmp\hcs*\Files\$Recycle.Bin\ manually (hcs* is random name).
cf. docker/for-win#8910

Run a Docker container

Rem Run with `Packages` directory mounted to the container
Rem Specify `--cpus` and `--memory` options according to your machine.
docker run --cpus=16 --memory=12288m ^
    --mount type=bind,src=%CD%\Packages,dst=C:\mediapipe\Packages ^
    --mount type=bind,src=%CD%\Assets,dst=C:\mediapipe\Assets ^
    -it mediapipe_unity:windows

Run build command inside the container

python build.py build --desktop cpu --opencv=cmake -v

If the command finishes successfully, required files will be installed to your host machine.

Android

Switch to Linux containers

Note that you cannot build native libraries for Desktop with Linux containers.

Build a Docker image

Rem You may skip applying a patch depending on your machine settings.
Rem See https://serverfault.com/questions/1052963/pacman-doesnt-work-in-docker-image for more details.
git apply docker/linux/x86_64/pacman.patch

docker build -t mediapipe_unity:linux . -f docker/linux/x86_64/Dockerfile

Run a Docker container

Rem Run with `Packages` directory mounted to the container
Rem Specify `--cpus` and `--memory` options according to your machine.
docker run --cpus=16 --memory=8192m ^
    --mount type=bind,src=%CD%\Packages,dst=/home/mediapipe/Packages ^
    --mount type=bind,src=%CD%\Assets,dst=C:\mediapipe\Assets ^
    -it mediapipe_unity:linux

Run build command inside the container

python build.py build --android arm64 -v

If the command finishes successfully, required files will be installed to your host machine.

Windows

Desktop/UnityEditor

Follow mediapipe's installation guide and install MSYS2, Python, Visual C++ Build Tools 2019, WinSDK and Bazel (step1 ~ step6).
(Optional) Install Opencv

By default, it is assumed that OpenCV 3.4.10 is installed under C:\opencv.
If your version or path is different, please edit third_party/opencv_windows.BUILD and WORKSPACE.
Install NuGet, and ensure you can run them
```
bazel --version
nuget
```
Install numpy
```
pip install numpy --user
```
Set PYTHON_BIN_PATH
```
set PYTHON_BIN_PATH=C:\path\to\python.exe
```
When the path includes space characters (e.g. C:\Program Files\Python39\python.exe), it's reported that build command will fail.
In that case, install python to another directory as a workaround (it's unnecessary to set the path to %PATH%, but don't forget to install numpy for the new Python).

Run build command

python build.py build --desktop cpu --opencv=cmake -v
Rem or if you'd like to use local OpenCV
python build.py build --desktop cpu --include_opencv_libs -v

Android

You cannot build native libraries for Android on Windows 10, so use Docker for Windows instead.

macOS

Install Homebrew
(Optional) Install OpenCV 3 and FFmpeg You can skip this if you plan to build OpenCV with MediaPipe (see Build Command for more details). Note that you need to install Xcode in this case.
```
brew install opencv@3
brew uninstall --ignore-dependencies glog
```

Install Python

brew install python

# Python version must be >= 3.9.0
sudo ln -s -f /usr/local/bin/python3.9 /usr/local/bin/python
pip3 install --user six numpy

Install Bazelisk and NuGet

brew install bazelisk
# Note that you need to specify bazel version if you'd like to build for iOS
# See https://github.com/bazelbuild/bazelisk for more details.
#
#  e.g. export USE_BAZEL_VERSION=3.7.2

brew install nuget

(Optional) Install Xcode

(Optional) Install Android SDK and Android NDK, and set environment variables

# bash
export ANDROID_HOME=/path/to/SDK
# ATTENTION!: Currently Bazel does not support NDK r22, so use NDK r21 instead.
export ANDROID_NDK_HOME=/path/to/ndk/21.4.7075529

Run build command

Build command

# Required files (native libraries, model files, C# scripts) will be built and installed.

# Build for Desktop with GPU enabled.
python build.py build --desktop gpu -v

# If you've not installed OpenCV locally, you need to build OpenCV from sources for Desktop.
python build.py build --desktop gpu --opencv=cmake -v

# Build for Desktop with GPU disabled, and copy OpenCV shared libraries to `Packages`.
python build.py build --desktop cpu --include_opencv_libs -v

# Build for Desktop, Android, and iOS
python build.py build --desktop cpu --android arm64 --ios arm64 -v

Run python build.py --help and python build.py build --help for more details.

Run example scenes

UnityEditor

Select Mediapipe/Samples/Scenes/Start Scene and play.

Desktop

If you've built native libraries for CPU (i.e. --desktop cpu), select CPU for inference mode from the Inspector Window.

Android, iOS

Make sure that you select GPU for inference mode before building the app, because CPU inference mode is not supported currently.

FAQ

DllNotFoundException: mediapipe_c

This error can occur for a variety of reasons, so it is necessary to isolate the cause.

1. Native libraries are not built yet

If native libraries (libmediapipe_c.{so,dylib,dll} / mediapipe_android.aar / MediaPipeUnity.Framework) are not built yet, this error can occur because Unity cannot load them.

If they don't exist under Packages/com.github.homuler.mediapipe/Runtime/Plugins, run build command first, and make sure that this command finishes successfully.

2. Native libraries are incompatible with your machine

Libraries built on Linux machines won't work on your Windows machine. If you'd like to run the plugin on Windows, you need to build libmediapipe_c.dll on Windows.

3. Dependent libraries are not linked

This error typically also occurs when OpenCV is incorrectly configured. See opencv_linux.BUILD / opencv_windows.BUILD and check if the path is correct (if not, edit the BUILD file).
You can also build and link OpenCV statically with --opencv=cmake option instead.

Tips:
In this case, when you check on Load on startup and click Apply button, error logs like the following will be output.

Plugins: Couldn't open Packages/com.github.homuler.mediapipe/Runtime/Plugins/libmediapipe_c.so, error:  Packages/com.github.homuler.mediapipe/Runtime/Plugins/libmediapipe_c.so: undefined symbol: _ZN2cv8fastFreeEPv

4. Dependent libraries do not exist

When you build an app and copy to another machine, you need to bundle dependent libraries with it.

For example, when you build libmediapipe_c.so with --opencv=local, OpenCV is dynamically linked to libmediapipe_c.so.
To use this on another machine, OpenCV must be installed to the machine too.
In this case, the recommended way is to build libmediapipe_c.so with --opencv=cmake.

If you are unsure of the cause, try checking the dependent libraries using ldd command, etc.

5. Dependent libraries are not loaded

On Windows, when you're using local OpenCV (i.e. --opencv=local), opencv_3410world.dll must be loaded before libmediapipe_c.so.
Unfortunately, currently no easy way to do this is known.
As a workaround, try loading both opencv_3410world.dll and libmediapipe_c.dll on startup.

DllNotFoundException will be thrown even after restarting UnityEditor, but you can ignore it safely if everything else is going well.
If you still cannot run sample scenes, the cause is probably something else.

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000

If you encounter an error like below and you use OpenGL Core as the Unity's graphics APIs, please try Vulkan.

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000_mediapipe/mediapipe/gpu/gl_context_egl.cc:261)

How to debug?

When debugging, you may want to read the MediaPipe log. If you set true to Glog.logtostderr before calling Glog.Initialize, MediaPipe will output logs to standard error, so you can check them from Editor.log or Player.log.

You can set various Glog flags as well. See https://github.com/google/glog#setting-flags for available options.

void OnEnable() {
    Glog.logtosdderr = true;
    Glog.Initialize("MediaPipeUnityPlugin");
}

void OnDisable() {
    Glog.Shutdown();
}

TODO

Prepare API Documents
Implement cross-platform APIs to send images to MediaPipe
use CVPixelBuffer on iOS
KNIFT

LICENSE

MIT

Note that some files are distributed under other licenses.

MediaPipe (Apache Licence 2.0)
FontAwesome (LICENSE)

Name		Name	Last commit message	Last commit date
Latest commit History 491 Commits
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
Assets		Assets
Packages		Packages
ProjectSettings		ProjectSettings
docker		docker
mediapipe_api		mediapipe_api
third_party		third_party
.bazelrc		.bazelrc
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
WORKSPACE		WORKSPACE
build.py		build.py
packages.config		packages.config

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

MediaPipe Unity Plugin

Prerequisites

Platforms

Example Graphs

Installation Guide

Docker for Linux

Linux

Docker for Windows

Desktop/UnityEditor

Android

Windows

Desktop/UnityEditor

Android

macOS

Build command

Run example scenes

UnityEditor

Desktop

Android, iOS

FAQ

DllNotFoundException: mediapipe_c

1. Native libraries are not built yet

2. Native libraries are incompatible with your machine

3. Dependent libraries are not linked

4. Dependent libraries do not exist

5. Dependent libraries are not loaded

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000

How to debug?

TODO

LICENSE

About

Releases

Packages

Languages

License

longgood/MediaPipeUnityPlugin

Folders and files

Latest commit

History

Repository files navigation

MediaPipe Unity Plugin

Prerequisites

Platforms

Example Graphs

Installation Guide

Docker for Linux

Linux

Docker for Windows

Desktop/UnityEditor

Android

Windows

Desktop/UnityEditor

Android

macOS

Build command

Run example scenes

UnityEditor

Desktop

Android, iOS

FAQ

DllNotFoundException: mediapipe_c

1. Native libraries are not built yet

2. Native libraries are incompatible with your machine

3. Dependent libraries are not linked

4. Dependent libraries do not exist

5. Dependent libraries are not loaded

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000

How to debug?

TODO

LICENSE

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages