Skip to content

Latest commit

 

History

History
420 lines (311 loc) · 19.5 KB

README.md

File metadata and controls

420 lines (311 loc) · 19.5 KB
Raw

EdgeAI AmebaPro2

教材網址：https://github.com/rkuo2000/EdgeAI-AmebaPro2

LLM服務器程式範例： AmebaPro2_server (.py)
Arduino 程式範例： Arduino/AMB82-mini (.ino)

自強基金會 WiFi

SSID: TCFSTWIFI.ALL
Pass: 035623116

1. AI 介紹

AI 簡介

AI 硬體介紹

2. 開發板介紹

RTL8735B晶片簡介

32-bit Arm v8M, up to 500MHz, 768KB ROM, 512KB RAM, 16MB Flash (MCM embedded DDR2/DDR3L up to 128MB)
802.11 a/b/g/n WiFi 2.4GHz/5GHz, BLE 5.1, NN Engine 0.4 TOPS, Crypto Engine, Audio Codec, ...

HUB 8735 Ultra

https://github.com/ideashatch/HUB-8735

AMB82-mini

Ameba Arduino

3. Arduino IDE使用介紹

Arduino IDE 2.3.2 下載 & 安裝

偏好設定 (Preferences)

Hub8735 ultra
https://raw.githubusercontent.com/ideashatch/HUB-8735/main/amebapro2_arduino/Arduino_package/ideasHatch.json

AMB82-mini
https://github.com/ambiot/ambpro2_arduino/raw/main/Arduino_package/package_realtek_amebapro2_index.json

Ameba AIoT 影片

選定開發板 AMB82-MINI

Tools > Board Manager > AMB82 package > 4.0.7

Getting Started

首先將AMB82-mini板子用MicroUSB線 連接至電腦的USB port

Arduino examples 範例練習

  • 01.Basics> Blink

  • 02.Digitial> GPIO> Button

修改程式碼：

const int buttonPin = 1;  // the number of the pushbutton pin
const int ledPin = LED_BUILTIN;    // the number of the LED pin

AMB82-Mini 程式範例

  1. 瀏覽器打開 EdgeAI-AmebaPro2, 點[Code]並選 [Download ZIP]
  2. 解壓縮.zip, 並將 Arduino/AMB82-mini 複製到 Documents/Arduino底下

4. 感測器範例練習

Serial-monitor baud rate = 115200

紅外線測距模組

VL53L0X v2
VL53L1X v2

Datasheet: VL53L0X - Time-of-Flight ranging sensor

Sketch: IR_VL53L0X

慣性感測模組

MPU6050

慣性元件介紹

Sketch: IMU MPU6050-DMP6v12

5.彩色顯示器 TFTLCD

AMB82-mini + ILI9341 TFT-LCD

SPI - LCD Screen ILI9341 TFT

Interface signal names:

  • MOSI: Standard SPI Pin
  • MISO: Standard SPI Pin
  • SLK: Standard SPI Pin
  • CS: Standard SPI Pin
  • RESET: Used to reboot LCD.
  • D/C: Data/Command. When it is at LOW, the signal transmitted are commands, otherwise the data transmitted are data.
  • LED (or BL): Adapt the screen backlight. Can be controlled by PWM or connected to VCC for 100% backlight.
  • VCC: Connected to 3V or 5V, depends on its spec.
  • GND: Connected to GND.

AMB82 MINI and QVGA TFT LCD Wiring Diagram:

Sketch: Camera_TFTLCD

Exmples/AmebaSPI:

  • LCD_Screen_ILI9341_TFT : LCD Draw Tests
  • Camera_2_Lcd : Camera output , then Jpeg Decoder to TFT-LCD
  • Camera_2_Lcd_JPEGDEC : Camera output, saved to SDcard, then Jpeg Decoder to read to TFT-LCD

6. 影像串流範例練習

影像串流

Sketch: RTSP_VideoOnly

動作偵測 (motion detection)

Examples: AmebaMultimedia / MotionDetection / LoopPostProcessing.ino

  • 修改ssid, passwd, 後燒錄到AMB82-mini,
  • 按reset後程式即開始運行, 用serial-monitor 查看顯示串流網址
  • 啟動手機或電腦上之VLC player, 設定RTSP串流網址

Motion Detection Google Line Notify

Examples: AmebaMultimedia / MotionDetection / MotionDetectGoogleLineNotify

7. 音頻應用

Audio & Mic

音頻環回測試

Examples: AmebaMultimedia / Audio /LoopbackTest

MP3 播放

Skatch: MP3_Player

  • store a song.mp3 onto SDcard

安裝 Hub8735 ultra

  • cd ~/.arduino15/packages/ideasHatch/hardware/AmebaPro2/4.0.13-Release/
  • copy variants/common_libs/MP3_Player.a to ~/.arduino15/packages/realtek/hardware/AmebaPro2/4.0.7/variants/common_libs
  • edit platform.txt (~/.arduino15/packages/realtek/hardware/AmebaPro2/4.0.7/platform.txt) to add "{com_lib.path}/MP3_Player.a" into Ameba archive list (compiler.ameba.ar.list)
  • copy libraries/MP3_Player to ~/.arduino15/packages/realtek/hardware/AmebaPro2/4.0.7/libraries

Upload & Reset to run

音頻串流範例

Examples: AmebaMultimedia / Audio / RTSPAudioStream

RTSP Audio Stream

MP4錄音範例

Examples: AmebaMultimedia / RecordMP4 / AudioOnly

Multimedia - MP4 Recording

音頻分類範例

Examples: AmebaNN / AudioClassification

YAMNet

8. 人臉辨識與識別

人臉辨識與識別介紹

人臉檢測範例

Examples: AmebaNN / RTSPFaceDetection

人臉識別範例

Examples: AmebaNN / RTSPFaceRecognition

Serial_monitor: REG=RKUO

  • Enter the command REG=Name to give the targeted face a name.
  • Enter the command DEL=Name to delete a certain registered face. For example, DEL=SAM
  • Enter the command BACKUP to save a copy of registered faces to flash.
  • If a backup exists, enter the command RESTORE to load registered faces from flash.
  • Enter the command RESET to forget all previously registered faces.

9. 影像分類 (Image Classification)

影像分類範例

RTSP_ImageClassification.ino

Garbage模型訓練與檔案轉換

模型訓練： kaggle.com/rkuo2000/garbage-cnn

required in kaggle for AmebaPro2

  1. pip install tensorflow==2.14.1
  2. model.save('garbage_cnn.h5', include_optimizer=False)

模型轉換：AMB82 AI Model Conversion

  1. Download garbage_cnn.h5 from kaggle.com/rkuo2000/garbage-cnn Output
  2. Compress garbage_cnn.h5 to garbage_cnn.zip
  3. Go to Amebapro2 AI convert model, fill up your E-mail
  4. Upload garbage_cnn.zip
  5. Upload one (.jpg) test picture (EX. glass100.jpg from Garbage dataset)
  6. Email will be sent to you for the link of network_binary.nb

程式範例：RTSP_GarbageClassification.ino

  1. click the recieved Email link to download network_binary.nb
  2. create NN_MDL folder in SDcard, save network_binary.nb under NN_MDL folder, and rename it to imgclassification.nb
  3. plugin SDcard back to AMB82-MINI
  4. modify Sketch RTSP_GarbageClassification.ino
    1. modify SSID and PASSWD
    2. modify imgclass.modelSelect (change DEFAULT_IMGCLASS to CUSTOMIZED_IMGCLASS)
  5. burn code into board AMB82-MINI, and run it with VLC player streaming

10. 物件偵測 (Object Detection)

Public Dataset

Roboflow

物件檢測介紹

Kaggle範例:

Pothole模型訓練與檔案轉換

模型訓練： kaggle.com/rkuo2000/yolov7-pothole

  1. repro https://github.com/WongKinYiu/yolov7
  2. create pothole.yaml %%writefile data/pothole.yaml
train: ./Datasets/pothole/train/images
val:  ./Datasets/pothole/valid/images
test: ./Datasets/pothole/test/images

# Classes
nc: 1  # number of classes
names: ['pothole']  # class names
  1. YOLOv7-Tiny Fixed Resolution Training
!sed -i "s/nc: 80/nc: 1/" cfg/training/yolov7-tiny.yaml
!sed -i "s/IDetect/Detect/" cfg/training/yolov7-tiny.yaml

模型轉換：AMB82 AI Model Conversion

  1. Download best.pt from kaggle.com/rkuo2000/yolov7-pothole
  2. Compress best.pt to best.zip
  3. Go to Amebapro2 AI convert model, fill up your E-mail
  4. Upload best.zip
  5. Upload one (.jpg) test picture (EX. pothole_test.jpg from Pothole dataset)
  6. Email will be sent to you for the link of network_binary.nb

程式範例：RTSP_YOLOv7_Pothole_Detection.ino

  1. click the recieved Email link to download network_binary.nb
  2. create NN_MDL folder in SDcard, save network_binary.nb under NN_MDL folder, and rename it to yolov7_tiny.nb
  3. plugin SDcard back to AMB82-MINI
  4. modify Sketch RTSP_YOLOv7_Pothole_Detection.ino
    1. modify SSID and PASSWD
    2. modify ObjDet.modelSelect(OBJECT_DETECTION, CUSTOMIZED_YOLOV7TINY, NA_MODEL, NA_MODEL);
  5. burn code into board AMB82-MINI, and run it with VLC player streaming

AMB82 Mini 物件偵測範例

RTSP_ObjectDetectionLoop

AMB82 Mini SD卡加載模型範例

RTPS_ObjectDetection_AudioClassification.ino

Online NN Conversion Tool (客製化模型轉換工具)

11. 大型語言模型範例 (LLM)

語音辨識範例

ffmpeg.exe is needed for Windows to run Whisper!

語音交談範例

ffmpeg.exe is needed for Windows to run Whisper!

Download ffmpeg-master-latest-win64-gpl.zip, extract & put ffmpeg.exe into where you run Whisper server.

AmebaPro2 Whisper LLM_server

AmebaPro2_Whisper_Gemini_server

RecordMP4_HTTP_Post_Audio.ino

  • 修改server IP位址 in RecordMP4_HTTP_Post_Audio.ino server IP位址, then 燒錄到 AMB82-MINI
  • reset AMB82-MINI 來啟動, 按鍵兩秒後即可錄音詢問 LLM/Gemini

12. 視覺語言模型 (VLM)

影像+語音交談範例

12. AmebaPro2 應用專題實作

Vaccum Robot

(trash classification, IR collision detection)

RoboCar

(voice-control, BLE remote-control, line-notify, object-detection)

AI Door-Bell

Portable Chatbot with a local LLM/VLM models

Portable ChatGPT with a local LLM

Portable Gemini with Google cloud LLM

step 1. running AmebaPro2_Gemini_server.py Gemini-1.5-flash

Step 2. running HTTP_Post_ImageText_TFTLCD.ino
press button for 2 seconds, when Blue-LED is on, it starts capturing image and send to Gemini

Step 3. AmebaPro2_Gemini_server will pass the image & text to Gemini, and reply the recieved result.text
(AmebaPro2 Gemini server will add , please answer briefly. to the prompt)