Add decoder, re align encodings with real apps, refactor encoders to …

…ease future customizations

components/ble_adv_controller/CUSTOM.md

@@ -18,36 +18,153 @@ There are a few basics to know about android applications exchanging messages wi
   - if the same message was already processed, discard it
   - else process it
 * **Pairing** consists in having the phone (or controller) and the device agreeing on an identifier to be sent in all messages and that would indicate the device is the effective target of the message. This is done by sending a pairing message including an identifier generated by the app/controller to the device. When the device receives this message with an identifier it does not know it will just ... ignore it ... EXCEPT if it has been restarted (power off/on) less than 5 seconds ago, in this case it will accept the new identifier. This is the choosen way to have a specific device accepting a new identifer, and not all the devices in the roomm...
-* Pairing with several controllers is possible, at least a remote and a phone, but it seems the phone apps and our apps may have to share the same identifier, which for now is not supported as the ID is generated differently.
-* When the device is delivered to the end-user, it will no more change, meaning that the messages it receives to be controlled will have to be always the same. This particularly means that if an application is able to control it at a given time, any new version of this application in the future (new android version, bug fix, ...) will have to support the send of those exact same messages.
-
+* Pairing with several controllers is possible, at least a remote and a phone, but it seems the phone apps and our apps may have to share the same identifier.
+* When the device is delivered to the end-user, it will no more change, meaning that the messages it receives to be controlled will have to be always the same. This particularly means that if an application is able to control it at a given time, any new version of this application in the future (new android version, bug fix, ...) will have to support the send of those exact same messages, and then the last version of the app will generate different variants corresponding to previous encoding at a given time.
 
 ## The messages
 The BLE Advertising messages are composed of different parts:
-* The BLE Advertising Parameters (esp_ble_adv_params_t) which are always the same and part of the BLE standard
-* The BLE Advertising data (usually 31 bytes) composed of:
-  * A Header (esp_ble_adv_data_t) also part of the BLE standard
-  * A **Manufacturer Data** section, usually composed of 26 bytes
-
-The goal of this component is to convert each entity action into this Manufacturer Data section (encoding) and emit it. Still this is not so simple as there are several applications using this methodology, and for each application different ways of encoding the data that evovled in the last years.
+* The BLE Advertising Header and stack which are always the same and part of the BLE standard, the behaviour can be customized vi 'esp_ble_adv_params_t'
+* The BLE Advertising data (max 31 bytes) composed of repeated data sections:
+  * 1 byte for the length of the section (length of the data + length of the type)
+  * 1 byte for the type of the section
+  * the data of the section
+
+  Example:
+    * raw data: 02.01.02.1B.03.F9.08.49.13.F0.69.25.4E.31.51.BA.32.08.0A.24.CB.3B.7C.71.DC.8B.B8.97.08.D0.4C
+    * section 1:
+      * 02.01.02 
+      * => Length: 2, type: 01 (AD_FLAG), data: 02
+    * section 2:
+      * 1B.03.F9.08.49.13.F0.69.25.4E.31.51.BA.32.08.0A.24.CB.3B.7C.71.DC.8B.B8.97.08.D0.4C
+      * => Length: 1B (27), type: 03 (UUIDs), data: F9.08.49.13.F0.69.25.4E.31.51.BA.32.08.0A.24.CB.3B.7C.71.DC.8B.B8.97.08.D0.4C (26 uint)
+
+The goal of this component is to convert each entity action into this raw data and emit it. Still this is not so simple as there are several applications using this methodology, and for each application different ways of encoding the data that evovled over the years.
 
 Supported Applications:
 * **Zhi jia**, includes several encoding variant: v0 (MSC16) / v1 (MSC26) / v2 (MSC26A)
-* **FanLamp Pro** / **LampSmart Pro**, includes several encoding variant: v1a / v1b / v2 / v3
+* **FanLamp Pro**, includes variants v1 / v2 / v3
+* **LampSmart Pro**, includes variants: v1 / v3
+* Other (probably remotes), found in other repositories. Similar to what FanLamp Pro and Lamp Smart Pro generate but with small differences, variants v1a / v1b / v2 / v3.
 
-To build the Manufacturer Data section corresponding to a command, the encoding is done as follow:
-* Convert the command and its parameters into a base 26 bytes structure containing among other:
-  * A **type**: a 2 bytes code. No one found out yet the use of it, seems to be always 0x0100 but can be set to anything it seems
-  * An **index** / group_index: a 1 byte code allowing to specify a sub-identifier. Quite useless for this component as it is able to specify different ids with forced_id / controller id.
+To build the Data section corresponding to a command, the encoding is done as follow:
+* Convert the command and its parameters into a base structure containing among other:
   * A **command id**: a code based on one byte identifying a command. Different in between applications but usually common between variants of a same app.
   * Command **arguments**: 0 to 4 bytes containing parameters of the command (fan speed value, light brigthness, ...)
 * Add parameters from the controller part:
+  * A **type**: a 2 bytes code. No one found out yet the use of it, seems to be always 0x0100 but can be set to anything it seems
+  * An **index** / group_index: a 1 byte code allowing to specify a sub-identifier. 
   * An **identifier**: a 2 or 4 bytes code generated and exchanged during pairing, identifying a device in a 'unique' way
   * A **transaction count**: a 1 byte code increased by 1 on each transaction. Allows the device to identify if a message was already read and processed, and not re-process it (guess)
 * Signing: compute an id based on a hard coded key allowing the device to be sure the message is coming only from the allowed app and would not be an interference from another message from another app (or a way to try to prevent smart people to reproduce the message...)
 * CRC computing: to be sure the message is complete and not corrupted
 * Whitening: to avoid the message to be mostly zeros
 
+# Capturing Advertising messages
+It can be usefull to capture the messages sent by a phone app or a remote already paired with the device you want to control, in order to extract some info such as the `identifier` for instance.
+
+This can be done quite 'easily' using the [ESPHome BLE Tracker](https://esphome.io/components/esp32_ble_tracker.html) component, and this config (you need to have at least one ble_adv_controller defined):
+
+```
+esp32_ble_tracker:
+  scan_parameters:
+    interval: 15ms
+    window: 15ms
+  on_ble_advertise:
+    then:
+      - lambda: 'ble_adv_static_handler->capture(x, true);'
+
+ble_adv_controller:
+  - id: my_controller
+    encoding: fanlamp_pro
+```
+
+This will generate DEBUG logs such as those ones each time a raw advertising message is received:
+```
+[17:37:52][D][ble_adv_handler:297]: raw - 02.01.02.03.03.27.18.15.16.27.18.A8.01.51.3F.91.A2.00.E2.DC.38.AD.F0.64.03.07.00.00.00 (29)
+```
+
+It TRIES to capture EVERYTHING, meaning:
+* if you have existing Bluetooth devices doing BLE Advertising, you will also capture the logs of those devices...
+* it tries to capture as much as it can, but it can miss some of the messages, I would say it captures 75% of the messages
+
+Moreover, the phone app or the remotes are generating several advertising messages for a same command issued, for example the ***FanLamp Pro app is generating 6 distinct raw message for each action*** (2 commands for each variant with different AD Flag section...)
+
+For each message captured, it tries to decode it with each encoder available, and if one matches it produces the following:
+```
+[16:08:56][D][ble_adv_handler:268]: raw - 02.01.01.1B.03.F0.08.30.80.B8.F7.E1.27.DB.F4.95.C1.65.7D.A4.9F.67.F6.B6.30.34.8B.53.2B.38.A2 (31)
+[16:08:56][I][lampsmart_pro - v3:233]: Decoded OK - tx: 131, cmd: '0x11', Args: [0,0,0,0]
+[16:08:56][I][ble_adv_handler:243]: config: 
+[16:08:56]ble_adv_controller:
+[16:08:56]  - id: my_controller_id
+[16:08:56]    encoding: lampsmart_pro
+[16:08:56]    variant: v3
+[16:08:56]    forced_id: 0xB4555A3F
+[16:08:56][D][lampsmart_pro - v3:103]: UUID: '0xB4555A3F', index: 0, tx: 131, cmd: '0x11', args: [0,0,0,0]
+[16:08:56][D][ble_adv_handler:254]: enc - 02.01.19.1B.03.F0.08.30.80.B8.F7.E1.27.DB.F4.95.C1.65.7D.A4.9F.67.F6.B6.30.34.8B.53.2B.38.A2 (31)
+[16:08:56][I][ble_adv_handler:256]: Decoded / Re-encoded with NO DIFF
+```
+
+The config loggued gives you what iall the info you would need to setup a copy of the remote / phone app you listen!
+
+STILL if you listen to your phone app, you will end up with more or less 6 configs (and 3 removing the dupe, one for each variant), so you will have to find the relevant one as the controlled device probably listen to only ONE of those variants...
+
+# Raw injection service
+If you captured a raw advertising message emitted by a phone app or a remote, just define a dummy controller and you can re inject the message as such with the following HA service:
+```
+esphome: <device_name>_inject_raw_<ble_adv_controller_id>
+```
+Just put the raw hexa string in the `raw` parameters, accepted formats (leading 0x, trailing length, spaces and dots are removed automatically):
+```
+0201021B03F9084913F069254E3151BA32080A24CB3B7C71DC8BB89708D04C
+0x0201021B03F9084913F069254E3151BA32080A24CB3B7C71DC8BB89708D04C
+02 01 02 1B 03 F9 08 49 13 F0 69 25 4E 31 51 BA 32 08 0A 24 CB 3B 7C 71 DC 8B B8 97 08 D0 4C
+02.01.02.1B.03.F9.08.49.13.F0.69.25.4E.31.51.BA.32.08.0A.24.CB.3B.7C.71.DC.8B.B8.97.08.D0.4C
+02.01.02.1B.03.F9.08.49.13.F0.69.25.4E.31.51.BA.32.08.0A.24.CB.3B.7C.71.DC.8B.B8.97.08.D0.4C (31)
+```
+This will lead to the component re emitting the raw command as such, which can be useful if none of the encoding / variant is working for you, as you can still be able to build a template light in Home Assistant directly if you managed to capture the ON and OFF command, such as this one:
+```
+light:
+  - platform: template
+    lights:
+      my_raw_light:
+        friendly_name: "My Raw Light"
+        turn_on:
+          service: esphome.my_device_inject_raw_my_controller
+          data:
+            raw: 02.01.02.1B.16.F0.08.10.00.DC.36.2F.22.9A.A0.0F.BE.FC.F9.68.C1.28.0C.1D.AD.09.DA.19.A9.35.23
+        turn_off:
+          service: esphome.my_device_inject_raw_my_controller
+          data:
+            raw: 02.01.02.1B.16.F0.08.10.00.DF.59.DC.4B.A4.38.7A.C8.A8.B0.6D.F8.3F.FD.B7.A9.FC.7C.4A.C0.AA.7C
+```
+The controller parameters forced_id / index / encoding / variant are ignored, but the `duration` and `max_duration` are available. The messages are also put in the sequencing queue and benefit from the centralized advertising of the component.
+
+# Raw decoding Service
+If you captured a raw advertising message emitted by a phone app or a remote, you can try to have it decoded by the existing decoders available in the app, using the service:
+```
+esphome: <device_name>_raw_decode
+```
+Just put the raw hexa string in the `raw` parameters, see previous section for the formats.
+You will see the result of the decoding in the logs of the application, as such:
+```
+[16:08:56][D][ble_adv_handler:268]: raw - 02.01.01.1B.03.F0.08.30.80.B8.F7.E1.27.DB.F4.95.C1.65.7D.A4.9F.67.F6.B6.30.34.8B.53.2B.38.A2 (31)
+[16:08:56][I][lampsmart_pro - v3:233]: Decoded OK - tx: 131, cmd: '0x11', Args: [0,0,0,0]
+[16:08:56][I][ble_adv_handler:243]: config: 
+[16:08:56]ble_adv_controller:
+[16:08:56]  - id: my_controller_id
+[16:08:56]    encoding: lampsmart_pro
+[16:08:56]    variant: v3
+[16:08:56]    forced_id: 0xB4555A3F
+[16:08:56][D][lampsmart_pro - v3:103]: UUID: '0xB4555A3F', index: 0, tx: 131, cmd: '0x11', args: [0,0,0,0]
+[16:08:56][D][ble_adv_handler:254]: enc - 02.01.19.1B.03.F0.08.30.80.B8.F7.E1.27.DB.F4.95.C1.65.7D.A4.9F.67.F6.B6.30.34.8B.53.2B.38.A2 (31)
+[16:08:56][I][ble_adv_handler:256]: Decoded / Re-encoded with NO DIFF
+```
+* raw: the hexa string injected
+* Decoded OK: the message was decoded by an encoder, here 'lampsmart_pro - v3'. Action Parameters are loggued. 
+* config: the config to setup for your controller to duplicate the source of the message. The defined controller will use the same identifier and will then be able to control the same device without any need to pair!
+* enc: the hexa string as it would be re-encoded by the encoder from the parameters extracted for the controller and the Action parameters.
+* the result of the comparison between what was injected and what was re encoded, to be sure the encoder would work OK! This comparison ignores the irrelevant differences in AD_Flag section (02.01.01 / 02.01.19).
+
 # Custom Command Service
 if you are using 'api' component to communicate with HA, for each ble_adv_controller a HA service is available:
 * name of the service:
@@ -70,7 +187,7 @@ For info here are the "known" commands already extracted from code and their cor
   * uses ZhiJia encoding, variant v2: encoding MSC26A. 
   * Only arg0 used for known commands, BUT arg1 and arg2 available in message structure
 * **FanLamp v1**: 
-  * for FanLamp Pro / SmartLamp Pro, with variant v1a and v1b using the same data structure.
+  * for FanLamp Pro / SmartLamp Pro, with variant v1, and Other for variant v1a and v1b using the same data structure.
   * arg0 and arg1 used for known commands, but arg2 available (and used by pair command btw...)
 * **FanLamp v2**: 
   * for FanLamp Pro / SmartLamp Pro, with variant v2 and v3 (v3 is v2 plus a signing step)
@@ -160,7 +277,7 @@ Example custom commands:
   * custom command parameters: {cmd: 222, arg0: 0, arg1: 0, arg2: 0, arg3: 0}
 
 
-### FanLamp v1 (v1a and v1b)
+### FanLamp v1 (and v1a / v1b)
 * Source code [HERE](https://gist.github.com/NicoIIT/527f21f7bbbd766b9844d5efbef86959)
 * Commands are calling function 'getMessage' with 3, 4 or 5 parameters most of the time:
 ```