Parallelization (deepdish.parallel)

See the Parallelization chapter for a tutorial.

deepdish.parallel.rank()

Returns MPI rank. If the MPI backend is not used, it will always return 0.

deepdish.parallel.imap_unordered(f, params)

This can return the elements in any particular order. This has a lower memory footprint than the ordered version and will be more responsive in terms of printing the results. For instance, if you run the ordered version, and the first batch is particularly slow, you won’t see any feedback for a long time.

deepdish.parallel.imap(f, params)

Analogous to itertools.imap (Python 2) and map (Python 3), but run in parallel.

deepdish.parallel.starmap_unordered(f, params)

Similar to imap_unordered, but it will unpack the parameters. That is, it will call f(*p), for each p in params.

deepdish.parallel.starmap(f, params)

Analogous to itertools.starmap, but run in parallel.

deepdish.parallel.main(name=None)

Main function.

Example use:

>>> if gv.parallel.main(__name__):
...     res = gv.parallel.imap_unordered(f, params)