Skip to content
This repository has been archived by the owner on Sep 15, 2022. It is now read-only.

Support for Deepnote #76

Open
ochen1 opened this issue Nov 3, 2020 · 23 comments
Open

Support for Deepnote #76

ochen1 opened this issue Nov 3, 2020 · 23 comments

Comments

@ochen1
Copy link

ochen1 commented Nov 3, 2020

Is your feature request related to a problem? Please describe.
Deepnote just launched a few days ago, and it looks like they have 24/7 instances. I feel many people would love to be able to SSH & VNC into their 24/7 Deepnote servers.

Describe the solution you'd like
I tried to run remocolab on Deepnote, but nvida-smi not being present caused some errors.

Following the StackOverflow answer here, I was able to at least get the python-apt module working.

Would support for Deepnote be possible?
@demotomohiro
Copy link
Owner

I tried to run remocolab on deepnote but I cannot login to ssh.

I ran following code on the notebook cell:

!sudo apt update && sudo apt upgrade -q -y
!sudo apt install -q -y -o=Dpkg::Use-Pty=0 python3-apt
!pip3 install git+https://github.com/demotomohiro/remocolab.git@add-deepnote

with open("remocolab_run.py", "w") as f:
  f.write("""
import sys, os, tempfile
sys.path.append("/usr/lib/python3/dist-packages")
sys.path.append("/opt/venv/lib/python3.7/site-packages")

with open("/etc/ssh/sshd_config", "a") as f:
  f.write("\\n\\nListenAddress localhost:58387")

import remocolab
os.chdir(tempfile.mkdtemp(prefix = "remocolab"))
#remocolab.setupSSHD(check_gpu_available = False, tunnel = "argotunnel")
remocolab.setupVNC(check_gpu_available = False, tunnel = "argotunnel")

Then ran following command on deepnote's terminal:

sudo python3 remocolab_run.py

sshd worked on the server and setupVNC ran without errors but when I tried to connect the server, I got following error:

Connection closed by UNKNOWN port 65535

Note:

  • apt update need to be executed before installing python3-apt.
  • Unlike Google colab, deepnote runs cells as non-root user. So I need to use sudo when I install packages.
  • When command line program wait for user input, deepnote cells cannot take user input and wait for forever.
    • That means I have to run remocolab from deepnote's terminal
  • Files in home directory are kept after restarting machine. So I change working directory to temp directory before running setupVNC.
  • sshd doesn't run in default config. It need to listen port other than 22.

It seems deepnote shutdown the instance if I didn't use it for long time like google colab.
I run python3 -c "import time;time.sleep(1111111111)" in deepnote terminal and closed the tab.
When I open deepnote again after few minutes, that command was running.
But when I reopen the tab after about 1 hour, that command was not running.

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020

Hi! Any updates on Deepnote support?

I noticed a new branch being created, which fortuately fixes the nvidia-smi FileNotFound exception.

I found out how to install remocolab and its dependencies onto Deepnote, but ngrok is giving me

"Connection refused" (on ssh client when trying to connect)

and

"There are no tunnels currently online in your account." (on ngrok dashboard)

(at the time of testing (now))

argotunnel sorta works, but not really.

INFO[2020-12-23T01:07:37Z] Route propagating, it may take up to 1 minute for your new route to become functional
ERROR[2020-12-23T01:07:49Z] Cannot connect to remote: dial tcp 127.0.0.1:58387: connect: connection refused
ERROR[2020-12-23T01:07:49Z] CF-RAY: 605e30156af01bf4-SEA Proxying to ingress 0 error: websocket: bad handshake

It gets me closer than ever before to a connection, but at the last minute the above error message appears on the server.

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020 •
edited
Loading

Hello! It's odd, but it appears my findings are inconsistent with your notes!

Notes:
(by "Deepnote" I may mean "Deepnote's default docker instance")

  • Deepnote provides a handy Terminal, which I prefer using over the notebook
    • The terminal allows spawning new interactive Python REPL processes
    • Programs like htop work and are great for debug
  • The default version of Python on Deepnote is Python 3.7.9
    A Python 3.7.9 on Deepnote
    • Installing any other version of Python is very complicated (touch docker, get into a dependency hell with apt)
  • Deepnote does not have apt cache updated by default, need to run apt update before anything
  • python(3)-apt does not install successfully on Deepnote (pip and apt both fail installation)
    • pip fails due to ModuleNotFound: DistUtilsExtra error
      image
      • If the notebook cell is not running as root (as per your speculations), then pip would essentially be installing to (3) different places, which might cause confusion and/or errors: /root/.local/lib/python3.7/ (does not exist on a fresh new instance), /home/jovyan/.local/lib/python3.7/ (does not exist on a fresh new instance), and /usr/lib/python3.7/
    • the package manager apt succeeds when installing python-apt
      • apt still can't be imported, raising a ModuleNotFound exception at import
      • apt_pkg can't be imported either
      • the import names above were tested (ie. raised a ModuleNotFound exception) in the folowing situations:
        1. restarting the notebook (NOT the instance), in a notebook cell
        2. using a new instance of the Python 3 REPL
    • It appears usage of the python-apt module is still possible though, this StackOverflow question has some useful answers. Also see my (semi-)working code block below.
  • Deepnote runs cells as the user root. The terminal is also running as root.
    whoami root
    whoami root
  • Deepnote has no support for stdin, nor support for ipywidgets (yet) (so must use terminal to run remocolab.py (for now))
  • The content inside Deepnote's default ~/work directory is persistent across runtimes / sessions / instances / reboots
  • The $HOME for user root in the bash shell is /home/jovyan, but this is inconsistent with /etc/passwd
    • (my speculations are that) this may be due to some line overriding the $HOME variable in .bashrc or something similar
      echo HOME z/home/jovyan
      cat /etc/passwd /root

Here is the code block I used for testing:

# Install remocolab
%cd ~/
!git clone -b add-deepnote https://github.com/demotomohiro/remocolab.git
%cd remocolab/
!apt -qq update
!apt -qq install -y python3-dev python3-setuptools python3-distutils python3-distutils-extra python3-apt --reinstall
!mv /usr/lib/python3/dist-packages/apt_pkg.cpython-37m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.so
!mv /usr/lib/python3/dist-packages/apt_inst.cpython-37m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_inst.so
!echo "pyngrok" >> requirements.txt
!pip -q install -r requirements.txt
!cp -r /usr/lib/python3/dist-packages/* ~/remocolab/
!cd /tmp/ && wget "https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip" && unzip ngrok-stable-linux-amd64.zip && rm ngrok-stable-linux-amd64.zip && mv /tmp/ngrok /usr/bin && chmod +x /usr/bin/ngrok
# !cd ~/remocolab/ && wget "https://github.com/cloudflare/cloudflared/releases/download/2020.11.11/cloudflared-linux-amd64" && mv ./cloudflared-linux-amd64 ./cloudflared && chmod +x ./cloudflared

# Now run the following command in a terminal (so you can access stdin):
#!cd ~/remocolab/ && python -c "import remocolab; remocolab.setupSSHD()"
#!cd ~/remocolab/ && python -c "import remocolab; remocolab.setupSSHD(tunnel='argotunnel')"

Files in home directory are kept after restarting machine. So I change working directory to temp directory before running setupVNC.

I'm not so sure about this. From my experiments, anything inside the ~/work directory is persistent storage.

It appears that this is the case because ~/work/ is a directory symlink to -> /work/
ls -l home work -> root work

/work is mounted as a remote filesystem, as seen here:
df

This makes sense, because the remote persistent filesystem caps free accounts to 5GB of storage. However, the local usable filesystem is much larger, at 100GB. Anything inside the local filesystem will, however, be purged during a hardware restart or instance stop. Note that this means every Python module installed must be reinstalled after a hardware restart (unless a virtualenv is used inside persistent storage), because the directory /usr/share/python3/ is not persistent.

It also appears Deepnote gives 750 hours of free uptime per month, meaning 100% uptime of a machine, so you can have a persistent instance! (wouldn't bet on it though, the instance might get powered off)

I've tested this as my friend is now hosting a Discord bot on Deepnote, without issue (no, this is not against the ToSI don't think)

The bot is still online, even after over 8 hours. 😃

Dashboard displays online status
dashboard still displays online status

Session running on standard runtime long after the tab was closed
session running on standard runtime long after the tab was closed

Cell still running
cell running on standard runtime long after the tab was closed

Bot is online.
and the bot is still online

Bot is responding.
and well and responding to events

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020

This is the error message I am getting with cloudflared:
cloudflared error message screenshot

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020

Is this expected? I am getting a <defunct> with ps aux | grep sshd 🤔

defunct screenshot

@rayanfer32
Copy link

rayanfer32 commented Dec 23, 2020 via email

@rayanfer32
Copy link

rayanfer32 commented Dec 23, 2020 via email

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020 •
edited
Loading

Nope, nvm, I was just being stupid lol.

For some reason running remocolab failed to start sshd, so I had to invoke it manually via /usr/sbin/sshd.

Also, remocolab did not automatically add port configuration to /etc/ssh/sshd_config for some reason. I added it manually using terminal + vim.

Apparently, the error I posted before (connection refused; bad handshake) was because cloudflared could not find the open port on localhost.

Now I am getting this error:

Connection closed by UNKNOWN port 65535

Looks like @demotomohiro also got that error!

@rayanfer32
Copy link

rayanfer32 commented Dec 23, 2020 via email

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020

@rayanfer32 Are you running on Deepnote? I'm not sure if this issue is related to what you are trying to do.

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020

It looks like Deepnote is blocking port 22! Even root is unable to start a server on port 22.

Trying to use sshd with port 22 causes <defuct> in ps aux and trying to run SimpleHttpServer or http.server on port 22 causes a PermissionError (even as root). Please see attatched screenshot below:
PermissionError: [Errno 13] Permission denied

@rayanfer32
Copy link

rayanfer32 commented Dec 23, 2020 via email

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020

This issue is for adding support for Deepnote #76

It is note related to Google Colab or anything happening there.

If you are experiencing an issue, feel free to open another new issue for it. Just make sure to look for duplicates 🤗

@rayanfer32
Copy link

rayanfer32 commented Dec 23, 2020 via email

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020 •
edited
Loading

This is odd: sshd is started on port 8022 without error.
sshd

ngrok is forwarding to localhost (which 0.0.0.0 covers) port 8022.
ngrok

Yet I am unable to connect to ssh:
ssh

EDIT:
I think I've found some mistakes in the above screenshots:

  1. Logging in as a non-existent user: the user colab does not exist on Deepnote. What am I doing!
    It exists now, as it was created automatically by remocolab.

  2. PermitRootLogin needs to be enabled in sshd_config in order to login as root.

But its still not working. (Connection closed)

@rayanfer32
Copy link

rayanfer32 commented Dec 23, 2020 via email

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020 •
edited
Loading

After some more debugs, I got the following error message (found an error at last)!

When running sshd on debug mode with -d flag, I get the following error, and then the program terminates:

root@deepnote:~/work $ /usr/sbin/sshd -d
debug1: sshd version OpenSSH_7.9, OpenSSL 1.1.1d  10 Sep 2019
debug1: private host key #0: ssh-rsa SHA256:BJ2D+KSTupbYM+91QLoOauGSt5yEcQD1yP5PhOSjBsM
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:xUMDrs7kqf6YG+JfxJjSt357LK8sII5HEtF13AjhXvg
debug1: private host key #2: ssh-ed25519 SHA256:UJKQ3LFUbesZpTpgW99f3OQ2bzOb6nfrADuJ4meGccg
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: Set /proc/self/oom_score_adj from 925 to -1000
debug1: Bind to port 9558 on 0.0.0.0.
Server listening on 0.0.0.0 port 9558.
debug1: Bind to port 9558 on ::.
Server listening on :: port 9558.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 47388 on 127.0.0.1 port 9558
debug1: Client protocol version 2.0; client software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
chroot("/run/sshd"): Operation not permitted [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 2072
debug1: audit_event: unhandled event 12

(9558 was a random arbitrary port I chose)

EDIT: You can use sshd -T to check the current sshd server config and ssh -v -v -v -v -o UserKnownHostsFile=/dev/null for client.

@ochen1
Copy link
Author

ochen1 commented Dec 23, 2020 •
edited
Loading

At last! I got a connection (through ngrok)!!! (without using remocolab)

ssh connection

@rayanfer32
Copy link

At last! I got a connection (through ngrok)!!! (without using remocolab)

ssh connection

Woah thats great man... Can you plz provide your script for the setup. Thanks

@rayanfer32
Copy link

Deepnote allow users to use custom docker images... more over here:
https://docs.deepnote.com/environment/custom-environments

This means we can load docker with xfce4 and cloudflared preinstalled

@ochen1
Copy link
Author

ochen1 commented Dec 24, 2020

This means we can load docker with xfce4 and cloudflared preinstalled

The last time I checked, the only docker base image supported by Deepnote the is the default image, provided by Deepnote. 😭

This means you will still have to take time to install the dependencies. RIP. 🙁

It might be, however, to create a Dockerfile to install the dependencies for us, which may or may not reduce the startup time.

Either way, Deepnote offers 24/7 machines, whose instances never have to be powered off (unlike the 12h limit of Google Colab), so installation is only necessary once, as long as you keep your "hardware" running, for example with the following code.

import time
while True:
    time.sleep(10000)

@ochen1
Copy link
Author

ochen1 commented Dec 24, 2020 •
edited
Loading

cloudflared

ngrok is not banned (has not been banned yet) on Deepnote, so we can continue to use it. ¯\_(ツ)_/¯

@ochen1
Copy link
Author

ochen1 commented Feb 11, 2021

Update: Deepnote added support for custom Docker images

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@demotomohiro @rayanfer32 @ochen1