Asynchronous Hibike (#633)

* Use profiling and event loop libs

* [HIBIKE] C extension (#622)

* Add hibike_packet as submodule

* Enable use of hibike_packet

Detects whether the extension is installed by trying to import it.

* Update hibike_packet

* Remove process_buffer

* [HIBIKE] fixing Disconnect and register_sensor syntax errors

* Final preparation for asyncio merge (#631)

* Do not crash on invalid COBS data

* Rename hibike_process_async to hibike_process

Remove old hibike_process and replace it with async version. API
compatibility is maintained, so Runtime will not be changed.

* Prevent hibike_tester hang after termination

hibike_tester would hang after terminating hibike_process,
because one of its threads was still running.
We tell the thread to shut down after the process is done
instead of running forever.

* Remove stub file

* Remove send_transport

This function is identical to send, so it doesn't make sense
to keep it around.

* Replace virtual device with async version

The asynchronous version uses less CPU and has saner defaults.

* Remove virtual device dependency on hibike_process

Some parts of the virtual device perform similar functions
to parts of hibike_process, but it is better that the two
implementations be allowed to evolve independently.

* Update tests for async; add read/write tests

Async tests need to deal with the event loop. In addition,
we test that hibike is actually reading and writing to devices
using virtual devices.

* Remove outdated portions of README, update others

* Add explanation for read/write retries

* Add test for nonexistent device errors

As it turns out, we were not sending error messages when a
nonexistent device was accessed; a test should ensure this behavior
stays in.

* Update developer documentation

* Fix lint errors

* [RUNTIME] changed kill process timeout from one second to three seconds

* Start async hibike rewrite

* [HIBIKE] Full implementation of async smart sensor protocol (#523)

Now, SmartSensorProtocol automagically registers itself with
Hibike when it connects.

* Fix bugs related to virtual devices

Essentially, we exclude existing serial ports from our
scan for new ones, but this didn't extend to virtual
devices, leading them to be added multiple times.

The other bug was that "connection_lost()" could
get called before a device was identified, triggering
a key error when we tried to take it out of the
devices map. This is now checked for.

Add function to create heartbeat requests

* Add async virtual devices

In addition, async virtual devices send heartbeat requests too, although
they don't do anything with the responses.

* Don't block event loop

* Don't block event loop on state queue
* Port process tests to async

* Use aiofiles for nonblocking IO

* Allow profiling measurements

* Use an external cobs library

* Memoize a few hot functions

* [HIBIKE] Pause reading of hibike messages when max size exceeded

* [HIBIKE] implemented backpressure on device side

* Exclude new name of hibike_packet from linting

* Unify runtime and hibike pipfiles

* Bump required python version to 3.7

* Bump runtime version

.gitignore

@@ -396,3 +396,6 @@ hibike/virtual_devices.txt
 
 # Webstorm project settings
 .idea
+
+# Visual Studio Code config
+.vscode

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "hibike/hibike_packet"]
+	path = hibike/hibike_packet_extension
+	url = https://github.com/pioneers/hibike_packet.git

hibike/DEVELOPERS.md

@@ -29,20 +29,20 @@ Look at hibike/devices/ExampleDevice for an example implementation of a hibike d
 
 Device specific firmware must:
 
- - Implement the arduino setup() and loop() functions, which must call hibike_setup() and hibike_loop() correspondingly
- - Implement device_write() and device_data_update(), which are called when the BBB wants to either write to the device or get a data update from the device. 
+ - Implement the arduino `setup()` and `loop()` functions, which must call `hibike_setup()` and `hibike_loop()` correspondingly
+ - Implement `device_write()` and `device_read()`, which are called when the BBB wants to write to or read from the device, respectively.
    - Both functions are given with a buffer containing data and a param index. Because params can have different data types and therefore values with different sizes, these functions must return the number of bytes they read from/wrote to their buffer.
    - Both functions are given the size of the buffer to avoid overflow. They must return 0 if their normal operation would overflow the buffer.
  - Conform to their device type definition described in hibikeDevices.json
  - Have the uid in their .h file filled out correctly
    - The uid references the device type id, which by convention is defined in an enum in hibike/lib/hibike/devices.h
- - Call hibike_loop() at a higher frequency than both
+ - Call `hibike_loop()` at a higher frequency than both
    - The expected frequency of the BBB sending packets to the device
    - The expected subscription frequency (The BBB will subscribe to device data updates at some frequency)
 
 ### Using libraries in device firmware
 
-For standard arduino libraries like <Servo.h> or <SPI.h>:
+For standard arduino libraries like `<Servo.h>` or `<SPI.h>`:
 
  - go to hibike/Makefile and update the line that looks starts with `ARDUINO_LIBS :=`
 
@@ -51,4 +51,4 @@ For external libraries:
  - add the library folder to hibike/lib
    - see the libraries already there
  - go to hibike/Makefile and update the line that looks starts with `SKETCH_LIBS =`
-
+

hibike/Makefile

@@ -55,7 +55,7 @@ AVRDUDE_CONF		 = /opt/arduino-1.8.1/hardware/tools/avr/etc/avrdude.conf
 CFLAGS_STD        = -std=gnu11
 
 ### CXXFLAGS_STD
-CXXFLAGS_STD      = -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -fno-devirtualize -fno-use-cxa-atexit
+CXXFLAGS_STD      = -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -fno-devirtualize -fno-use-cxa-atexit -fno-strict-aliasing
 
 ### CPPFLAGS
 ### Flags you might want to set for debugging purpose. Comment to stop.

hibike/Pipfile

@@ -1,20 +1,23 @@
 [[source]]
-
 url = "https://pypi.python.org/simple"
 verify_ssl = true
 
 
 [requires]
-
-python_version = "3.6"
+python_version = "3.7"
 
 
 [packages]
-
+protobuf = "==3.2.0"
 pyserial = "==3.2.1"
 flask = "*"
+aioprocessing = "*"
+pyserial-asyncio = "*"
+aiofiles = "*"
+uvloop = "*"
+cobs = "*"
 
 
 [dev-packages]
-
-pylint = "==1.7.2"
+pylint = "==1.8.1"
+yappi = "*"

hibike/README.md

@@ -1,62 +1,7 @@
-# hibike 2.0!
+# Hibike
 Hibike is a lightweight communications protocol designed for the passing of sensor data for the
 PiE Robotics Kit, a.k.a Frank or "Kit Minimum" to some.
 
-#### This branch contains documentation and implementation for the 2016-2017 version of the hibike protocol, which should feature iterative improvements over last year's protocol
-
-#### These suggestions should make their way into protocol documentation first, then implemented in code. The basic read/write functions need to be implemented in Python, and everything needs to be implemented in Arduino, before merging this branch back to develop.
-
-## Suggested Protocol Changes (Please make suggestions)
-
-1. The checksum should be changed to something more robust
-    - Janet suggested using UDP's checksum
-2. COBS encoding should be implemented cleaner
-    - It should be documented and part of the protocol itself, not a wrapper around it
-    - There should not be a redundant length field
-    - Only the first 2 bytes of a packet should not be cobs encoded
-        - 0 byte and packet length
-3. Data Update and Device Update/Status should be unified
-    - Huge advantage: The BBB polling, writing, and subscribing can have identical responses from SD, and runtime can treat them the same
-    - Protocol can abstract a device as only key value pairs
-        - Current implementaion has key value pairs and one custom "data update" struct per device
-        - Custom "data update struct" is nice because it is the exact size we need
-        - Only Key Value pairs means 32 bits per digital IO pin...
-        - Does ease of abstraction and implementaion justify larger packet length?
-        - Is packet length significant anyways?
-            - 32 bits at 115200 baud is .2 milliseconds?
-            - Someone should test actual throughput
-                - Especially how fast BBB can actually receive data
-                - Even when doing blocking reads byte by byte in python?
-                - While runtime and student code are running?
-                - With 20+ devices?
-        - Should the size of values be unique to reduce packet length?
-            - Harder to implement
-            - Both devices need to know the size of each value
-                - But they needed to know the types anyways so maybe this is ok
-    - SubRequest can specify which keys it wants to receive
-        - Each DataUpdate will have to also encode this information to be stateless
-        - What if we subscribe to more than the max payload size?
-            - Just respond with as many as you can fit?
-            - Error Packet?
-        - A uint16_t bitmask could work? 16 keys is plenty for a device
-4. Unused packets should be removed from the protocol
-    - DescriptionRequest and Response are redundant
-        - maintaing one config file is better for production
-    - Do we have a use for the error packet yet?
-        - Maybe when SD receives write requests for a read only key?
-        - Maybe when a checksum fails/unexpected 0 byte is seen?
-            - Can there be infinite loops of back and forth error packets?
-            - Maybe only SD can send errors, and they'll only be used for logging statistics
-5. Hot-Plugging behaviour should be optimized and well-defined
-    - Current status quo (rebooting BBB to add devices) is unacceptable
-    - Reenmerate devices every x seconds and also when runtime requests it
-        - Or runtime can just request it every x seconds
-    - Hibike can notify runtime when a device disconnects/connects
-    - Student code accessing disconnected devices should raise a well-defined exception in *student code*
-    - If a SD disconnects, will BBB find out until it tries reenumerating it?
-        - If so, should BBB even bother reenumerating SDs it hasn't detected as disconnected?
-
-
 
 ## Section 0: A Quick Introduction
 
@@ -233,9 +178,6 @@ Device Type Enumeration:
 
 Note: These assignments are totally random as of now. We need to figure
       out exactly what devices we are supporting.
-Note: As of now, Grizzlies are not supported by Hibike (pyGrizzly should
-      be used instead) But they should be in the near future, to preserve
-      the idea of treating every peripheral as a SmartDevice.
 
 Error ID Enumeration:
 
@@ -356,7 +298,7 @@ Note: These assignments are also fairly random and may not all even be
 8. Heart Beat Response: Sent in response to a Heart Beat Request
     - This message pathway is a two way street, both BBB and SD can receive requests and send responses to the other
     - Should only be sent upon receiving a Heart Beat Request
-    - Payload is currently unused, but can be used for future functionality in keeping track of individual heartbeat requests and responses (for latency purposes)
+    - The payload is used for flow control; 0 indicates that packets should be sent at full speed, and 100 indicates as slow as possible.
 
     Payload format: