Applications

Also, see the list of current ESP examples.

Accelerometer / Gyroscope / IMU

Detecting individual gestures:

• free-air swipe / navigation gestures
• sports gestures: tennis strokes, swinging a baseball or golf club, etc.
• punching, kicking, etc.
• foot-taps
• flying / wing-flapping

Detecting pose / posture:

• correct grip on a bat, golf club, bow, rifle, etc.
• balance training (childhood development, stroke patients, arthritis patients, sports training, etc.)
• detecting bad sitting posture (e.g. while sitting at a desk)
• correct yoga posture
• eye-tracking (e.g. see EyeWriter project)

Extracting qualities of gestures:

• range of motion (e.g. for physical therapy patients)
• speed of motion (e.g. if someone is lifting a barbell too quickly or slowly)
• breathing speed (e.g. based on chest expansion / contraction)

Detecting the occurrence of physical activities:

• walking
• running
• dancing
• cycling
• boxing / martial arts
• knitting

The BMI160 IMU has built-in step detection, no-motion detection, tap detection, and free-fall detection. See datasheet

Walk Detection

Agata Brajdic and Robert Harle. 2013. Walk detection and step counting on unconstrained smartphones. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing (UbiComp '13). ACM, New York, NY, USA, 225-234. DOI=http://dx.doi.org/10.1145/2493432.2493449

Free Fall Detection

• http://cache.freescale.com/files/sensors/doc/app_note/AN3151.pdf

Fall Detection

• http://www.analog.com/library/analogdialogue/archives/43-07/fall_detector.html

Jump Detection

Activity Recognition

Gesture Recognition (using DTW), Wii-like

• Exemplar
• MAGIC
• wiigee

Microphone

General Applications

• pitch/inflection tracking on vocal patterns (a popular project I’ve seen a few times here is training English speakers not to reflexively go up in pitch at the end of sentences so everything sounds like a question)
• detecting water flow
• speaker identification
• basic robust volume detection
• basic robust speech detection (i.e. detecting whether or not someone is talking)
• beat detection
• pitch / musical note detection

Pipeline / feature considerations:

• SVM on raw FFT data can work well for single-pitch (or dominant frequency) sounds (e.g. bell, ringing a glass).
• FFTFeature provided by GRT is super noisy. It picks the highest magnitude frequencies, but with noise, the highest magnitude frequencies change randomly / frequently. Can we filter out background? Need a way to get dominant frequencies and their magnitudes together (parallel feature extraction and composition).
• Zero-crossing can also work well. PCA too.
• PCA is a CoreAlgorithm, it's not clear if we can use it as a feature extraction module.

Audio localization (directionality) using two (or more?) microphones.

Geometric sensing using audio:

Valkyrie Savage, Andrew Head, Björn Hartmann, Dan B. Goldman, Gautham Mysore, and Wilmot Li. 2015. Lamello: Passive Acoustic Sensing for Tangible Input Components. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1277-1280. DOI=http://dx.doi.org/10.1145/2702123.2702207

Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2161-2170. DOI=http://dx.doi.org/10.1145/2702123.2702414

Chris Harrison: scratch detection, touching phone w/ fingertip vs. knuckle

Frequency Detection

• http://www.instructables.com/id/Arduino-Frequency-Detection/
• http://www.instructables.com/id/Arduino-Pitch-Detection-Algorithm-AMDF/

Beat detection / spectrum analysis in music.

• Discussion: http://stackoverflow.com/questions/79445/beats-per-minute-from-real-time-audio-input/81462#81462
• FFT-based Algorithm: http://archive.gamedev.net/archive/reference/programming/features/beatdetection/
• Test Data Set: http://www.music-ir.org/mirex/wiki/2006:Audio_Beat_Tracking
• Arduino-based approach: http://dpeckett.com/beat-detection-on-the-arduino

Speech-Onset detection

• http://www.mirlab.org/conference_papers/International_Conference/Eurospeech%201997/pdf/tab/a0199.pdf)
• https://www.reddit.com/r/arduino/comments/23qk5r/electret_microphone_suitable_for_speech_detection/

Gender Identification

• http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1221721
• AUTOMATIC GENDER IDENTIFICATION OPTIMISED FOR LANGUAGE INDEPENDENCE
• http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=543213
• Automatic Gender Classification Using the Mel Frequency Cepstrum of Neutral and Whispered Speech: a Comparative Study - has a good summary of previous work.

Speaker Identification

Gunshot Detection

• http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5620932
• http://dl.acm.org/citation.cfm?id=2736143

Ring-Tone Detection

Irina Diaconita, Andreas Reinhardt, Delphine Christin, and Christoph Rensing. 2014. Bleep bleep!: determining smartphone locations by opportunistically recording notification sounds. In Proceedings of the 11th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MOBIQUITOUS '14). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 110-119. DOI=http://dx.doi.org/10.4108/icst.mobiquitous.2014.258035

Optical / Pressure / Capacitive Sensing

Object Recognition:

• detecting presence / movement of people using distance sensors, floor pressure sensors, PIR sensors, etc.
• identifying objects or people based on sensor arrays (e.g. capacitive, pressure, magnetic)

Liquid recognition (from Interactive Device Design)

• http://sean-c.com/drinke.html

Or, more generally, recognizing objects using a color sensor

Hand gesture recognition (Artem's paper)

Grasp recognition: Brandon T. Taylor and V. Michael Bove, Jr.. 2009. Graspables: grasp-recognition as a user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 917-926. DOI=http://dx.doi.org/10.1145/1518701.1518842

Detecting who's sitting in a chair (fab class project) Detecting posture.

Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 483-492. DOI=http://dx.doi.org/10.1145/2207676.2207743

• Touche for Arduino
• https://github.com/Illutron/AdvancedTouchSensing
• Techniques in Swept Frequency Capacitive Sensing: An Open Source Approach
• SweepingCapSense library for Arduino

Bio-Signals

• distinguishing signal from noise for things like EEG, ECG, EMG, GSR

Sensor Calibration / Filtering

• dealing with changing / drifting baselines (e.g. ambient light levels, baseline capacitance)
• detecting intentional interactions w/ noisy/unreliable sensors (e.g. handmade pressure or stretch sensors)
• filtering out glitches in sensor readings (e.g. inhuman motions in Kinect data)
• debouncing (e.g. filtering out rapid successive changes in sensor readings)
• understanding and use of mean and standard deviation (not just fixed thresholds)