Wireless stepper control?

[Jaknil]

I am toying with the idea of giving each motor a wifi module (esp8266, 2$) and using a local wifi UDP clock signal from the controller module to sync up the onboard clock on each board.

The modules could then each start and run the same gcode in paralell, which is sent over regular TCP with big buffers and thus should stay in sync through the job. (Fancier solutions for this part can be added later.)

This would be in a peer to peer WLAN to reduce latency. If we can get the clocks synced to 1 ms we’ll only introduce 0,06mm errors when we feed 60mm/s. I wonder if this level of sync is realistic over local wifi, without using a router?

Note that I don’t need 1ms ping, just a somewhat consistent one so that I can measure it and sync the clocks once.

What do you think? It’s not something we desperately need but it would be fun to make work.

[jonnor]

You mean to sync the CPU clocks? Or some 'tick rate' in the path planner (lower frequency, maybe 10-100kHz). For CPU clocks the standard to do that over network with high precision is PTP, Precision Time Protocol. IEEE-1588.

[jonnor]

If you are willing to have a single shared data line, one could distribute a 32kHz RTC oscillator style clock to all the modules and use that for sync. Using such a signal to adjust CPU clock is very common, but unsure if ESP have it out of the box (STM32 does).

[hwalseng]

Interesting idea. Have you been talking to Leo? He had som thoughts about motors with wireless signal. In which use cases do you find a solution like this would be beneficial over having signal cables drawn along with the power cables, since we'll still need the power cables? (Still waiting for a world with wireless power as envisioned by Tesla)

[Jaknil]

Jon: I was hoping for full wireless, it would be so neat :)
I was thinking to sync the clocks so that I can have them start at the same moment. To me it sounds less vulnerable and easier to implement than a ticker. I see that the ESP32 has built in hardware support for PTP, if we can find examples on how to use it perhaps that could be easier than setting up our own solution? I have not researched it lots yet, but it might not be 100% developed yet.

Haakon: I'll drag him into this discussion! Good tip.

[jonnor]

Starting at the same time is relatively easy, I think?. Could for instance send "start in" 10, 9, 8...1 packets on UDP (with say 1ms intervals) and estimate the start time from that.
Maintaining sync over time might be the harder thing. A good clock crystal has drift over time of maybe 20ppm. That is 72 ms over a 1 hour milling job - well outside of the desired 1ms.
Drift is highly temperature dependent, so if all devices have similar temperature - drift between the devices may be actually smaller.

It seems that PTP is only specified for Ethernet, so maybe not as relevant as I thought. But it seems that there are mesh network solutions for ESP (based on their peer-to-peer 2.4Ghz ESPNow protocol), that implement clock sync. Like https://gitlab.com/painlessMesh/painlessMesh

[Jaknil]

In which use cases do you find a solution like this would be beneficial over having signal cables drawn along with the power cables, since we'll still need the power cables?

For several reasons! Provided it works every time. It will make the "scary" electronics part as easy as plugging in power. No wiring to mess up or cables to cut. For example you can build huge machines with one motor on each wall, or a machine with 20 motors.

[jonnor]

The motivation of more flexible/easy-to-build machines is very inline with the background for the Gestalt nodes and Modular Machines That Make projects that James Coleman, Ilan Moyer and Nadya (and others) worked on. They used "smart nodes" connected by a RS485 serial bus, but might have had some thoughts on wireless as well. http://archive.monograph.io/james/m-mtm

[Jaknil]

Maintaining sync over time might be the harder thing. A good clock crystal has drift over time of maybe 20ppm. That is 72 ms over a 1 hour milling job - well outside of the desired 1ms.

Wow! That was way worse than I thought! I guess it would have to re-sync every 10s then. Which might be messier since then other things are supposed to happen at the same time and all motors and the VFD will be adding noise. I was hoping for a "silent moment" to sync in before everything kicks off. Perhaps clock drift compensation is worth looking at, if we make our own sync solution.

It seems that PTP is only specified for Ethernet, so maybe not as relevant as I thought. But it seems that there are mesh network solutions for ESP (based on their peer-to-peer 2.4Ghz ESPNow protocol), that implement clock sync. Like https://gitlab.com/painlessMesh/painlessMesh

Super high precision timing for wireless is definitely coming for IoT sensors and industry control. Paper about microsecond clock sync with drift compensation

PainlessMesh looks worth testing, they seem to get within <10 ms offset which is good but not great.
Documentation for class MeshTime

[jonnor]

Temperature compensation is a key thing for low-drift clocks (and quite standard). This application notes describes a method that should get under -+5ppm for the typical room temperature. If one can between two-devices achieve a ppm of under 1, then 1 ms sync should be doable over 15 minute timespans. Could maybe then chop up jobs into smaller sub-job (with goto safe Z etc at end), and start them (in sync) every 5 minutes or so.
https://www.st.com/content/ccc/resource/technical/document/application_note/55/e2/3d/2a/87/ab/4d/e1/CD00232494.pdf/files/CD00232494.pdf/jcr:content/translations/en.CD00232494.pdf

[Jaknil]

Could maybe then chop up jobs into smaller sub-job (with goto safe Z etc at end), and start them (in sync) every 5 minutes or so.

We could do this split by inserting "M0" gcode commands into main gcode stream after JOG lines when 5 min has passed. This causes the controller to hold and wait for a "~" Feed resume command. Then we re-sync clocks and kick all of again. Possibly so fast that the user does not notice.

[jonnor]

To measure how much sync the setup is able to actually achieve, one can (during development/test) have IO lines going from each stepper controller to a microcontroller that can measure the timing differences (using interrupts). One can then inject gcode commands that should be synced into the stream (maybe coolant on/off? or spindle/on off in laser mode), and measure the times between when each node executing this command. This could be done by the "master ESP", or a separate device.

[Jaknil]

First crude test results are in!

Test question:

Can we use a UDP message broadcast as a timing pulse using "default" Arduino Core wifi libraries for ESP8266?

Success criteria:

Less than 1 millisecond random lag would be great.

Test setup

• ESP8266 is running a wireless access point (Soft AP) with a password
• PC running windows 10 and packet sender with default school firewall active and several other programs running.
• Code
• The PC is connected to the ESPs "wifi hotspot" and sends UDP messages every two seconds to the ESP8266s IP address.
  -The ESP8266 measures the time between received UDP messages and the difference of the last two time intervals is presented as "random lag".

Results

The "random lag" is more than 1 ms and sometimes much more.

Conclusion of test

We can not use UDP as a straight clock pulse in this way. Possibly we can use it to get an average time.

Improvement ideas

• Test painlessmesh network mode and its clock sync function
• Average the time between pulses and use that to keep a local clock running in sync.
• "Cleaner" beacon, use another ESP8266 that only sends the UDP messages and nothing else
• Drop the wifi encryption as it adds unnecessary processor load.
• Test if it makes a difference if the sender is the server instead of the receiver.
• Test with a router as wifi server instead of peer to peer between the units
• Find a way to use the hardware TSF they brag the board comes with.

Next test

We should next test painlessMesh.

Potential problems

• Will we be able to run minimum 8kHz (preferred 16kHz) nice and even pulses to the steppers while listening to real time wifi timing pulses with one 80Mhz processor core?

Repo update

Added a readme and a folder to the module development area of the repo

[jonnor]

Great to have data! PCs with an OS does multitasking, which generally introduces a lot of variability. I think that one must go ESP->ESP to get anywhere close to OK.

BTW, are you checking that your UDP packages are received in order? Might want to add a sequence number in them, an integer that counts upwards by one for each package sent

[cmonr]

Random lurker of the repo chiming in.

The 2.4GHz spectrum is awfuly congested, and I would be weary of thinking that UDP packets could be used as a reliable clocking source. Even with UDP, there are enough layers in between the packet and RF transmission that jitter could still occur.

Btw, what happens if a motor misses a clock signal? How could that be debugged?

I would suggest a different approach: add RTCs to each driver in addition to a WiFi module, suse NTP to sync all devices, and then send command to each module to control them.

[hwalseng]

@jonnor
Do you think "Time-of-Flight (TOF) measurements with normal Wi-Fi packets" could somehow
be utilitized to deal with the synchronization issue of wireless stepper signals?

The ESP32-S2 supports ToF over wifi and when reading a little about ToF over wifi I find that accuracy/granularity is mentioned to be in the order of nanoseconds.

I have to admit that this is beyond my theoretical knowledge of this field (at this moment) but I figured I'd ask for an expert opinion in case I'm on to something :-)

[hermanschmit]

Sorry to be late to this thread. I'm working on using ESPNOW synchronization for this. (I'm basically porting something like PTP 1588 to ESPNOW). The nice thing about ESPNOW is that it is point-to-point, so you won't have noise from a wifi router. I have had accuracy measurements of <1ms on my scope. I do see the frequencies meander, so it does seem like re-synching is important.

https://github.com/hermanschmit/espnow_sync