Working with Big Data inPython | 251
>>> del fpo
>>> newfp = np.memmap(filename, dtype=’int’, mode=’r’,
shape=(10,3))
>>> newfp
memmap([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26],
[27, 28, 29]])
9.3.2 Parallel Computing Using
mp4pi
Library
Message passing interface (MPI) is a standard messaging system which is leveraged for parallel
computing. Right from its inception, it has been a leading standard for message-passing libraries.
It was originally implemented using popular scientic programming languages, like C, C++,
Fortran, etc. MPI for Python provides a platform for message passing. MPI for Python supports
convenient, pickle-based communication of generic Python object at a fast, near C-speed. It can
communicate with any built-in or user-dened Python object leveraging the pickle module of
Python.
Python programs that use MPI commands must be executed using an MPI interpreter, which
is provided with the command mpirun or mpiexec. For a program to run using 4 processors
(which is the number of cores in the workstation or laptop), it looks like the following.
$ mpiexec -n 4 python mpi_script.py
A sample script to do a simple communication of numpy array is given below. For more complex
programming examples and other details, refer to the MPI for Python website https://mpi4py.
readthedocs.io/en/stable/index.html.
from mpi4py import MPI
import numpy
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
# passing MPI datatypes explicitly
if rank == 0:
data = numpy.arange(1000, dtype=’i’)
M09 Big Data Simplified XXXX 01.indd 251 5/10/2019 10:22:59 AM
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.