Research python options for geoda cluster

[sjsrey]

IPython http://ipython.org/ipython-doc/stable/parallel/parallel_intro.html

mpi4py http://mpi4py.scipy.org
Note that mpi4py is already installed with Anaconda

Other https://wiki.python.org/moin/ParallelProcessing

[sjsrey]

Trying to replicate the mpi example on ipython leads to

[srey1@magic-3-1-4 ~]$ ipython
Python 2.7.5 |Anaconda 1.7.0 (64-bit)| (default, Jun 28 2013, 22:10:09)
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import numpy as np

In [2]: from IPython.parallel import Client

In [3]: c = Client(profile='mpi')

In [4]: view = c[:]

In [5]: view.
view.abort              view.class_traits       view.history            view.outstanding        view.results            view.sync_imports       view.update
view.activate           view.clear              view.imap               view.parallel           view.run                view.targets            view.wait
view.apply              view.client             view.importer           view.pull               view.scatter            view.temp_flags
view.apply_async        view.execute            view.map                view.purge_results      view.set_flags          view.track
view.apply_sync         view.gather             view.map_async          view.push               view.shutdown           view.trait_metadata
view.block              view.get                view.map_sync           view.queue_status       view.skip_doctest       view.trait_names
view.class_trait_names  view.get_result         view.on_trait_change    view.remote             view.spin               view.traits

In [5]: view.activate()

In [6]: view.run('psum.py')
Out[6]: <AsyncResult: execute>

In [7]: view.scatter('a', np.arange(16, dtype='float'))
Out[7]: <AsyncResult: scatter>

In [8]: view['a']
Out[8]:
[array([ 0.,  1.,  2.,  3.]),
 array([ 4.,  5.,  6.,  7.]),
 array([  8.,   9.,  10.,  11.]),
 array([ 12.,  13.,  14.,  15.])]

In [9]: %px = totalsum = psum(a)
Out[9]: <AsyncResult: execute>

In [10]: view['totalsum']
[0:apply]:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)<string> in <module>()
/home/srey1/anaconda/lib/python2.7/site-packages/IPython/parallel/util.py in _pull(keys)
    243         return map(lambda key: eval(key, globals()), keys)
    244     else:
--> 245         return eval(keys, globals())
    246
    247 @interactive
<string> in <module>()
NameError: name 'totalsum' is not defined

[1:apply]:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)<string> in <module>()
/home/srey1/anaconda/lib/python2.7/site-packages/IPython/parallel/util.py in _pull(keys)
    243         return map(lambda key: eval(key, globals()), keys)
    244     else:
--> 245         return eval(keys, globals())
    246
    247 @interactive
<string> in <module>()
NameError: name 'totalsum' is not defined

[2:apply]:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)<string> in <module>()
/home/srey1/anaconda/lib/python2.7/site-packages/IPython/parallel/util.py in _pull(keys)
    243         return map(lambda key: eval(key, globals()), keys)
    244     else:
--> 245         return eval(keys, globals())
    246
    247 @interactive
<string> in <module>()
NameError: name 'totalsum' is not defined

[3:apply]:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)<string> in <module>()
/home/srey1/anaconda/lib/python2.7/site-packages/IPython/parallel/util.py in _pull(keys)
    243         return map(lambda key: eval(key, globals()), keys)
    244     else:
--> 245         return eval(keys, globals())
    246
    247 @interactive
<string> in <module>()
NameError: name 'totalsum' is not defined

[jlaura]

Was the psum.py file created?
What happens if the function is defined in the interactive prompt? Is it not scattered?

Also In[9] should read %px totalsum = psum(a)

I'll play with the example via iPython as well. The mpi_test1.py file is working as expected using mpi4py.

[sjsrey]

it was, i forgot to commit. there now

s.

Sergio (Serge) Rey
Professor, School of Geographical Sciences and Urban Planning
GeoDa Center for Geospatial Analysis and Computation
Arizona State University
http://geoplan.asu.edu/rey

Editor, International Regional Science Review
http://irx.sagepub.com

On Sep 25, 2013, at 4:45 PM, jlaura [email protected] wrote:

Was the psum.py file created?

What happens if the function is defined in the interactive prompt? Is it not scattered?

Also In[9] should read %px totalsum = psum(a)

I'll play with the example via iPython as well. The mpi_test1.py file is working as expected using mpi4py.

—
Reply to this email directly or view it on GitHub.

[jlaura]

Working as long as I am in the same directory as the psum script. Even the absolute path to that file (In [6]) does not seem to function properly.

In [2]: from IPython.parallel import Client

In [3]: c = Client(profile='mpi')

In [4]: view = c[:]

In [5]: view.activate()

In [6]: view.run('psum.py')
Out[6]: <AsyncResult: execute>

In [7]: import numpy as np

In [8]: view.scatter('a', np.arange(16, dtype='float'))
Out[8]: <AsyncResult: scatter>

In [9]: view['a']
Out[9]:
[array([ 0., 1., 2., 3.]),
array([ 4., 5., 6., 7.]),
array([ 8., 9., 10., 11.]),
array([ 12., 13., 14., 15.])]

In [10]: %px totalsum = psum(a)
Out[10]: <AsyncResult: execute>

In [11]: view['totalsum']
Out[11]: [array(120.0), array(120.0), array(120.0), array(120.0)]