NAT busting when server is behind NAT

[keithw]

We want to be able to connect to a server behind a NAT, even after the client roams.

We could use techniques like http://samy.pl/pwnat/ (per Nelson Elhage), except we'd need to keep doing them to pick up any client IP address changes.

[devurandom]

See also RFC 5128. libnice (ICE) might be an alternative to pwnat.

[lann]

Assuming this is only intended to apply to the UDP portion of the connection, the various ICE methods are unnecessary, and even the pwnat technique is overkill. The client can bind a UDP socket and send the port number to the server over SSH, which will send a dummy UDP packet to the client before returning its own UDP port number to accomplish the hole-punching.

Edit: pwnat's predecessor chownat [http://samy.pl/chownat/] does exactly this

[devurandom]

That is what I suggested in IRC previously. It was rejected.

The issue is that we have no communication channel available to send the IP/port number through. The SSH connection is just established once and for a short time, to start the mosh-server, then it is closed immediately. When the client roams and thus his IP changes, the server will not know about the new IP unless we notify it somehow. Sending him a packet is not an option, because the firewall will block it. Opening a new SSH connection is not an option, because that would require entering the password again.

[lann]

Ah, good point. Still, it would be a good incremental improvement if a more robust solution isn't going to be implemented soon (no roaming behind a NAT vs no nothin' behind a NAT).

[joneskoo]

I think it'd be worse to have a situation where you can start mosh but not reconnect to it. You'd get hanging sessions you cannot connect to any other way than ssh'ing in and pkill mosh-server. (Which is the same as now if your client kernel panics?)

[kmcallister]

Which is the same as now if your client kernel panics?

Right, the server will hang forever if the client dies without contacting the server. This could happen because the client gets SIGKILL, or the client's machine reboots unexpectedly, or the packets notifying the server of shutdown are lost on each of several attempts.

[inducer]

I acknowledge that client roaming is a thorny issue. That said, given how simple UDP hole punching on initial set-up would be to implement, I don't quite understand mosh's current stance of "Your server is behind NAT? Well, that's tough--you know what, we'll just make sure you can't benefit from mosh at all without painful manual setup, just to remind you that there's a thorny issue left to resolve."

[inducer]

Another suggestion: I imagine the typical set-up for server-behind NAT would be for somebody to have an ssh ProxyCommand that takes care of the onward connection from the publicly visible end of the NAT. mosh doesn't support this use case well, it seems. In particular, it tries to resolve the name passed to ssh as a hostname, when that hostname is likely an internal one, or even only valid as an ssh config name. Instead, it should optionally use a service like http://checkip.dyndns.com/ to find the NAT's public IP, report that via the initial SSH connection and then use that as the address to connect to. Alternatively, there should at be least a way to tell mosh what that public host name is.

[kmcallister]

I imagine the typical set-up for server-behind NAT would be for somebody to have an ssh ProxyCommand

No, I think the typical setup is just a forwarded TCP port.

[keithw]

inducer: Thanks for your feedback. I would say that mosh is a young project, and the fact that we haven't yet accommodated a use case doesn't mean we're trying to be difficult!

However, generally speaking the philosophy with Mosh has been that things should work predictably and reliably. I don't like the idea that the user could start a session -- thinking it's roamable -- but then it would just hang when they actually tried to roam. That's like the worst kind of program, one that breaks only when you actually try to use it. :-)

Given that we've advertised Mosh as roamable, implementing a NAT-busting technique that doesn't roam worries me. I don't want to mislead users and have them cursing us after they try to roam.

[inducer]

I do see what you're saying. But I get a feeling that a 'proper' solution will be a longer-term project, and in the meantime, getting 90% of the functionality in place (and perhaps warning about the difficult 10% not being in place) still has value--at least IMO.

I also know code speaks louder than me bitching in a bug discussion, but I thought it might be helpful if I tell you about a use case that I tried and that didn't work out using mosh's current state. Perhaps the ProxyCommand+NAT handling should be split off into a different bug, although it is related.

[kmcallister]

Perhaps the ProxyCommand+NAT handling should be split off into a different bug

I think so too. We do want to hear about this and keep track of the issue.

[inducer]

See issue #285.

[lann]

Given that we've advertised Mosh as roamable, implementing a NAT-busting technique that doesn't roam worries me. I don't want to mislead users and have them cursing us after they try to roam.

@keithw what if simple NAT-busting was enabled with a flag? Maybe even something verbose like --nat-traversal-danger-roaming-will-break :P

I would be happy to work on implementing this - mosh looks perfect for me (heavy console work over sometimes-laggy connections) other than this missing feature.

[deutrino]

Would it be in any way possible without a major architectural change to have a single "hi I roamed" port where mosh-server listens for (cryptographically identifiable) packets from clients to re-establish their particular connection?

[antonpiatek]

I am very much in favour of a nat busting flag. I know roaming has no nice solutions, but even a flag to let me get into a natted system once would be rather nice

[lilydjwg]

Hi, I want to access my natted system too. I think mosh-client has to use a specific port when both side being restricted cone nats. Of course, roaming can't be done, but at least I can directly access the system now. I've made a dirty patch for mosh-client. And here's my little script for hole punching.

[chrishuan9]

I always wondered why I couldn't get mosh-server to work on my home-workstation. It's behind a NAT (I configured port-forwarding of the mosh-UDP Port range 60000:61000) but it didn't help. The server is reachable via dyndns (ip changes maybe once or twice a year depending how long the cable-modem goes down: less than 15 min --> no IP change). Would be nice if you could state in the FAQ that the mosh-server doesn't work behind a NAT (couldn't find it).

[eatnumber1]

Two years since this bug has been opened. Let me propose a non-optimal solution: support SOCKSv5. Here's why:

I wanted to expose all the hosts on my internal nat's ssh ports to the internet. I could have done the standard method of forwarding non-standard ports over my router manually for each host I wanted to expose, but that doesn't scale well. Instead, I chose to set up a socks server which accepts connections from the internet and allows proxied connections to the ssh port of my internal machines. I then could write some simple ssh configs on my laptop to transparently do this proxying and it Just Works™.

Now this same principle could be applied for mosh. I could configure my proxy server to allow udp packets to be forwarded to my internal network on ports 60000-610000, and have mosh use this proxy server to forward packets to the machine it wants to talk to. This should even work for the roaming case!

It's got some drawbacks of course in that you have to set up a proxy server on the firewall, and you are exposing UDP on quite a few ports (60000-610000) to the internet, but nobody seems to have any better ideas.

I'd be willing to contribute some time towards making this happen if I got the ok for adding SOCKSv5 support to mosh. 😄

[akanouras]

If mosh-server is behind a Full-cone NAT, it supporting STUN (and making sure to refresh the mapping every once in a while) would be enough to solve this reliably.

For other types of NAT (except for Carrier Grade NAT), there are three main standards-based solutions:

1. UPnP/NAT-PMP/PCP/etc.
2. ICE, with the client using a new SSH connection every time it roams to coordinate the negotiation. STUN can also help in this scenario, unreliably however.
3. SOCKSv5/manual port forwarding (both require administrator intervention however)

For Carrier Grade NAT and heavily firewalled NATs, assuming they allow outbound connections, an external TURN server to relay packets between client and server would be needed, if an SSH connection had already been achieved somehow. TURN support would be useful for ICE as well.

IMHO, implementing just STUN for the mosh-server would not be much effort using a library, would relieve a lot of people, and would be a good first step towards implementing the other methods as well.

[lachesis]

I'd love to see a way to make mosh work behind NAT with no setup. That'd let me use it in a lot of places I currently cannot.

[Manouchehri]

Just thought I'd add my two cents in here. If you want to use mosh behind a NAT now, I would suggest "manually" punching holes yourself. There's already scripts that can run as non-root to get the job done, for example:

https://gist.github.com/somic/224795

If we do end up adding punching into mosh, it would have to be extremely explicit that you are removing the ability to roam; i.e., have a "--disable-roaming" and a "--udp-punch" flag that both have to be defined.

[Neo-Desktop]

Is there an accepted way of nat-hole-punching?

[aep]

yo, i'm working on this, sort of.

[rkjnsn]

I would love to see STUN support for my use case. I have SSH access to several machines (some via ProxyCommand) behind networks friendly to STUN, and I would like to be able to use Mosh with them. A lot of the discussion here seems to concern roaming. It seems there are two cases here.

In the first, the fixed (not currently roaming) machine may have a NAT with a loose firewall allowing replies from any IP once the internal source IP/port to external IP/port mapping has been established by the first packet. In this case, roaming will just work, as the fixed machine could still receive packets from the roaming machine and update accordingly.

In the second case, the firewall only allows directly replies from the same IP address. In this case, there would need to be some external signalling method to update the IP address. In this case, I would be happy to re-enter my ssh password (or not if using public key authentication) to re-establish communication, as it would still be much better than losing the session, even if not 100% seamless.

[aep]

oh. i'm no longer working on adding that to mosh btw, it's more complicated than a patch. Instead built a new thing that establishes ssh though nats and firewalls https://devguard.io

[Izzette]

Sorry to necrobump, but it seems like although this is a thorny issue, there do exist a variety of potential "full" solutions. Either UPnP or Samy Kamkar's NAT penetration solution would be able to solve this problem. While UPnP (in general) certainly is full of drawbacks and pwnat is a total hack: so is NAT. The only real solution is IPv6 (which appears may need a little tweaking anyways, see: #81 & #855). But unfortunately adaption is moving slowly, and "acceptance" by the network administrator, development communities ever more so. In order to make mosh a useful alternative for interactive SSH sessions, it really needs to be able to traverse NAT gateways. Although obviously it comes down to development time, the mosh team (@andersk, @cgull, @eminence, @keithw) officially accepting and endorsing one of the solutions, and throwing it in the mosh-future milestone seems like an appropriate start.

[lann]

It has been suggested that pwnat is probably no longer viable on modern NATs: https://stackoverflow.com/questions/22985793/is-pwnat-still-working

[sergei-mironov]

Hi. I've managed to prepare a proof-of-concept solution for this problem. I described it in this StackOverflow answer https://stackoverflow.com/a/57948167/1133157

Currently it requires client and server wrapper scripts and also a custom udp-relay tool. Probably the scripts could be incorporated into usual Mosh running procedure. As for relay, we need to use something more secure in real life (stun/turn?).

See also a related blog post http://blog.fraggod.net/2017/06/02/upgrading-ssh-to-mosh-with-udp-hole-punching-to-connect-to-a-host-behind-nat.html on which my solution is based.

[sergei-mironov]

Hi. I've managed to prepare a proof-of-concept solution for this problem. I described it in this StackOverflow answer https://stackoverflow.com/a/57948167/1133157

Currently it requires client and server wrapper scripts and also a custom udp-relay tool. Probably the scripts could be incorporated into usual Mosh running procedure. As for relay, we need to use something more secure in real life (stun/turn?).

See also a related blog post http://blog.fraggod.net/2017/06/02/upgrading-ssh-to-mosh-with-udp-hole-punching-to-connect-to-a-host-behind-nat.html on which my solution is based.

After using this method for a while, I found a usability problem: currently mosh doesn't seem to support keep-alive packets from server to client. So if client disconnects, server's NAT closes hole soon and its not possible to re-create it because udp port is already bound by mosh-server. So I see no option other than patching mosh in order to fix it.

[Izzette]

I think the move would probably be to patch mosh anyways, rather than relying on wrapper scripts. But a working PoC, wrapper or not, indicates to me some substantial movement towards a possible production solution.

[normanr]

WebRTC's unordered DataChannels would probably be the right thing to do these days. It also means it'll be easier to get web clients working because they'll have native WebRTC support.

[Izzette]

WebRTC would be nice because it should have better ability to traverse firewalls as this is a normal use(abuse)case. However, WebRTC requires STUN/TURN and is itself a complex protocol.

[normanr]

I have a PoC working (also why I filed #1091). I don't think I can share the source code (yet), but if you contact me directly I can probably give you something to test with.

[normanr]

Actually there's a much easier test: does https://github.com/maxmcd/webtty#readme work for you? It doesn't use mosh, but it uses the same WebRTC libraries that my mosh-via-WebRTC PoC uses.

[zond]

Here's another 5c from the peanut gallery: Restarting the hole punching after the client has roamed is simple enough if the user uses ssh-agent, then another password entry is unnecessary.

[Zibri]

WebRTC and STUN/TURN should be a top priority for this project.