Skip to content

Commit

Permalink
Remove old files and remove old documentation for copying files aroun…
Browse files Browse the repository at this point in the history
…d cluster. (ray-project#274)
  • Loading branch information
robertnishihara authored and pcmoritz committed Feb 13, 2017
1 parent cb7f6ca commit 3934d5f
Show file tree
Hide file tree
Showing 3 changed files with 0 additions and 1,131 deletions.
64 changes: 0 additions & 64 deletions doc/using-ray-on-a-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,67 +96,3 @@ If the web UI doesn't work, it's possible that the web UI processes were never
started (check `ps aux | grep polymer` and `ps aux | grep ray_ui.py`). If they
were started, see if you can fetch the web UI from within the head node (try
`wget http://localhost:8080` on the head node).

### Copying Application Files to Other Nodes (Experimental)

If you're running an application that imports Python files that are present
locally but not on the other machines in the cluster, you may first want to copy
those files to the other machines. One way to do that is through Ray (this is
experimental). Suppose you're directory structure looks

```
application_files/
__init__.py
example.py
```

And suppose `example.py` defines the following functions.

```python
import ray

def example_helper(x):
return x

@ray.remote
def example_function(x):
return example_helper(x)
```

If you simply run

```python
from application_files import example
```

An error message will be printed like the following. This indicates that one of
the workers was unable to register the remote function.

```
Traceback (most recent call last):
File "/home/ubuntu/ray/lib/python/ray/worker.py", line 813, in fetch_and_register_remote_function
function = pickling.loads(serialized_function)
ImportError: No module named 'application_files'
```

To make this work, you need to copy your application files to all of the nodes.
The following command will do that through Ray, and will add the files to Python
path of each worker. This functionality is experimental. You may be able to do
something like the following.

```python
import ray

ray.init(redis_address="<redis-address>")

ray.experimental.copy_directory("application_files/")

# Now the import should work.
from application_files import example
```

Now you should be able to run the following command.

```python
ray.get([example.example_function.remote(0) for _ in range(1000)])
```
Empty file removed python/webui/.gitkeep
Empty file.
Loading

0 comments on commit 3934d5f

Please sign in to comment.