wont connect to flask-socketio

[itsjesseyo]

i'm making a python/flask example because i thought you might like to add it to your examples, and because python/flask is my goto stuff, but messages, strings or json, won't seem to work. while the esp starts to boot the server registers the connection but the serial port output says the connection failed, then, on each attempt from the esp8226 to send some data, i get bad syntax, bad format errors serverside.

67.182.247.83 - - [07/Mar/2016 22:24:32] code 400, message Bad request version ('\x07\x00\x00\x00\x06\x01\x00')
67.182.247.83 - - [07/Mar/2016 22:24:32] "??8451
nJQVTQB__kFTVGZCgAEUY@POB]ZF^J_DFAXUQ
" 400 -
(15773) accepted ('67.182.247.83', 37155)
67.182.247.83 - - [07/Mar/2016 22:24:42] code 400, message Bad request syntax ('\x81\xdc1685\x05\x04c\x17CS[PX@Q[ViKP_EWGnCHQPB]\x17\x1dM\x1aFTXKZC\x14\x02\x17VFK\x17\x1d\x14L\S\x1a\x0f\x00\x05\r\x04\t\x04\x0c\x04\x03\x06\x14\x17UWLT\x13\x0cc\x01\t\x18\x0f\x00\x07\x06\x00\x04\x1d\x04\x16\x06\x01\x04\x08\x06\tkEh\x81\xdc3855\x07')
67.182.247.83 - - [07/Mar/2016 22:24:42] "??1685cCS[PX@Q[ViKP_EWGnCHQPB]MFTXKZCVFKL\S

you can see the example i'm building here: https://github.com/luxnovalabs/water_sensor

and you are welcome to try to connect your exp8266 at 107.170.232.185:5000
107.170.232.185:5000/home will give you a simple webpage that i've been using to ping the server and monitor the messages.

[washo4evr]

could you post your esp code too please?
I can connect from my PC but nothing happens when I click on ping server

[itsjesseyo]

I just use the hello esp example you gave, except i swap out the host for my server address, put the port to 5000, and then I change either the

client.send("atime", "message", "Time please?");
or the
client.sendJSON("JSON", JSON);

to

client.send("receiving_sensor_update", "message", "Time please?");
or the
client.sendJSON("receiving_sensor_update", JSON);

i've even tried renaming the serverside channel/message to 'atime' or 'JSON' and left the hello time example as is, save the changes for connecting to the server.

[washo4evr]

did you update in app.js and index.html the port too?
on your github, it is still app.listen(1234); and not 5000

[itsjesseyo]

i've updated the server app to more closely mimick your app.js example. i have a 'connection', 'atime' and 'JSON' message reciever, and i'update the website to mimick these events and receive callbacks. new members of the server will be broadcast to all members. here's the exact arduino code, after the wifi login details, which are very correct:

Hello_Time_ESP82663.zip

the behavior I'm getting today is that some sort of reset is getting called (from serial):

Connecting to clear_net
.......
WiFi connected
IP address:
10.0.1.15

Exception (0):
epc1=0x40212b37 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

ctx: sys
sp: 3ffffda0 end: 3fffffb0 offset: 01a0

stack>>>
3fffff40: 40212a1d 00000000 3ffea7cc 3ffea7d8
3fffff50: 3ffea7d8 0000005e 00000000 0000001c
3fffff60: 00000002 4022468d 40207fab 3ffec698
3fffff70: 3ffea7cc 3fffdcc0 3ffe8ca8 3ffe8ca8
3fffff80: 00000080 3ffec698 00000000 3fffdcb0
3fffff90: 402078d3 3fffdab0 00000000 402044ab
3fffffa0: 3ffe8ca8 40000f49 3fffdab0 40000f49
<<<stack<<<

ets Jan 8 2013,rst cause:4, boot mode:(3,6)

wdt reset
load 0x4010f000, len 1264, room 16
tail 0
chksum 0x42
csum 0x42
~ld

Connecting to clear_net
.......
WiFi connected
IP address:
10.0.1.15

Exception (0):
epc1=0x40207f40 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

ctx: sys
sp: 3ffffde0 end: 3fffffb0 offset: 01a0

stack>>>
3fffff80: 00000080 3ffec6c0 00000000 3fffdcb0
3fffff90: 402078d3 3fffdab0 00000000 402044ab
3fffffa0: 3ffe8d30 40000f49 3fffdab0 40000f49
<<<stack<<<

ets Jan 8 2013,rst cause:4, boot mode:(3,6)

wdt reset
load 0x4010f000, len 1264, room 16
tail 0
chksum 0x42
csum 0x42
~ld

thanks for spending your time doing this. it's a super useful project.

[washo4evr]

Hey,
I uploaded your code on a ESP here and can somehow communicate with your server
but Im not being upgraded to socket.
there is something weird about the reply Im getting : Connected. SID=nterval":25000,"ping

It should contain the SID that node/socket.IO uses to talk with the client.

Im not getting any execptions though

[washo4evr]

I also get that

head>
title>Error response</title>
/head>
body>
h1>Error response
p>Error code 400.
p>Message: Bad request syntax ('\x81\xa66554\x02\x07n\x16WA\YS\x17\x19O\x14XPGETRQx14\x0f\x17_XP\x14FYPUEP').<p>Error code explanation: 400 = Bad request syntax or unsupported method. /body>

[washo4evr]

when connecting to your server, im not getting the
HTTP/1.1 101 Switching Protocols
the connection from the ESP never gets upgraded to socket

[itsjesseyo]

I'm going to reach out to the flask-socketio guy. and start trying to to see if i can solve it with a config option, or by looking in existing issues.

for reference, I'm currently using:
socketio = SocketIO(app, allow_upgrades=True, cookie=None)

I wish I could be more helpful. While this prgoresses, would you point me in the directions of docs you used to implement socketio so I can start catching up?

[washo4evr]

I did use some docs to make it work, but mostly, I had to use wireshark, especially at the end.
Before the version 1.x, it just worked
The difficult part was supporting the new version
I remember that website was useful 👍 http://enterprisewebbook.com/ch8_websockets.html

let me know when you hear back from the flask guy

[itsjesseyo]

adding some feedback just so you can see some effort on my end. I've added a config change to the server that gives me step by step logging details. what i'm seeing is the the server receives the ping messages and sends a pong, and the nodemcu eventually just resets, then starts over, trying to upgrade again.

I'm going to go through the various modules i have, remove passwords on my wifi, strip out everything and start from the beginning, so i'll probably go dark for a bit.

i really, really want this to work. sockets are cool, socketio is better, and i want a definitive starter kit built for all my projects - so i'm dedicated to this.

[washo4evr]

I would suggest trying another module
the ESP should not reset (at least, my code doesnt reset it)

I could have a server running here and you trying to connect to it from your ESP
let me know if you'd want that

[itsjesseyo]

it's not getting upgraded to server because the sid is not getting set correctly.

i've downloaded and installed the latest arduinojson from github, but i've also tried using the one that ships with arduino, and the latest registered update - all of which are 5.1.1 . I'll rollback a few versions, but what version are you using?

the bitshifting errors i'd like more info on as my old school skills are out of date, but (in case you dont know python), the key/value paris in dictionaries are not guaranteed to be in any particular order when accessed. theoretically, and in practice, at anytime you access the dictionary (like when converting to a json string) the pairs could move around. I think in javascript this is speced the same way but they always stay in the same place in practice. so, if you are bit shifting or doing other manipulation expecting some sort of order, that'd explain why the python isnt working (in addition to maybe the guy just created the data in a different order than you were expecting).

hopefully this gets us in the correct direction for the flask-socketio fix

[washo4evr]

Hey,
I just uploaded a SID fix
I realized that when connecting to your server last time
on my server SID is always 20 characters so I had designed a loop with i<20
now It will adapt to whether it is 20 or 32 (like in your case)
let me know if that fix it

[washo4evr]

try the BETA - SocketIOClient.cpp

[itsjesseyo]

dude. almost. sid was fine, but the upgrade failed because the server did not detect a ping packet

[washo4evr]

SID : Check !
that's good right ;)

you can add client.heartbeat(0); to "ping" the server
0 is for ping
1 is for pong
(the line is commented in the INO file)

[itsjesseyo]

didn't help. tried uncommenting it, then adding it in various stages of the websocket upgrade phase. contacted socketio dev for his upgrade implementation variation details. i expect there are subtle differences every step of the way. stuff that perhaps a desktop accounts for, or can ignore, but cause death on the microcontroller.

[washo4evr]

something is weird with the request I think
have you tried connecting to a regular node.js server with socket.io installed on it
this is how I do it here
at least, that could clear the ESP8266. it might be the implementation in Python

[itsjesseyo]

pretty sure it is the python end of things. the node server works fine, and i go back to it every once and while and make sure it still is ok. Are you able to find official specs on the handshake? the arduino prints a content length of 117 char long, and this website i found where a guy had to do packet sniffing says its 90 long :
http://showmetheco.de/articles/2011/8/socket-io-for-backend-developers.html

[washo4evr]

i usually have something around 95 here
117 was a first
it is also because your SID is 32 char long while mine is only 20
I have been using wireshark for at lest 50% of the development for that library.
sockets are tricky

[itsjesseyo]

yeah. they is.

so, i've spent the day with wireshark, and using httplib in python trying to catch up. the spec definitely wants a ping packet with the word "probe" sent in the long polling process before it upgrades.

https://github.com/socketio/engine.io-protocol#transport-upgrading

and the flask-implementation is strict to it : miguelgrinberg/python-socketio#3

i'm attempting to find the correct place to hack this in to your code for a test, and what i see is that line 169 : sendHandshake is the first step in upgrading via long polling, and you immediately go into waiting for a response then immediately try to upgrade to websockets. in the middle should be a ping message with probe in it, but since we aren't asynchronous, and we need to finish processing the SID and all that, i'm wondering what your thoughts are on the best way to hack a test. i suppose i could fire off a message, and not wait for a response, but will incoming responses mess with the buffer, or interrupt things?

EDIT:
the long polling ping:probe packet is supposed to be jsonp.

[washo4evr]

hey
after a while looking at wireshark with firefox and chrome connecting locally to my node server, I tried adding the ping:probe
It instantly stopped working : I could no longer connect from the ESP (firefox and chrome were fine)
the weirdest thing is

after the 101 switching protocol is received, FF/chrome do a HTTP POST and then, atfer succesful code 200, probe
with the ESP, I need to send 52 else it fails
but that 52 is never sent from a browser

the http post, by the way is identical to the 1st get (transport = polling //not websocket)

[itsjesseyo]

interesting. the flask guy says thats part of the spec so its staying, but he also said you can connect directly via websocket, so i guess i'll try that.