© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022
A. DanialPython for MATLAB Developmenthttps://doi.org/10.1007/978-1-4842-7223-7_6

6. Call Python Functions from MATLAB

Albert Danial1  
(1)
Redondo Beach, CA, USA
 

The MathWorks introduced the py module for MATLAB in 2014. This module provides a direct binary interface between MATLAB and Python, opening vast new possibilities for both languages. In addition to making it possible to call any Python function, MATLAB’s py module allows one to create native Python objects within MATLAB. Such variables retain their full complement of Python methods—for example, a native Python string created in MATLAB can call all its 40 methods, from .capitalize() to .zfill().

The py module has improved with each new release of MATLAB. MATLAB code examples in this book use MATLAB 2020b, so if you’re working with an older version of MATLAB, some of the examples may not work.

This chapter just introduces the basic mechanics of using the py within MATLAB. Appendix A contains a list of “recipes” that show complete MATLAB programs that call Python to solve problems the core MATLAB product can’t solve on its own.

6.1 Configure MATLAB to Recognize Python

The first step to using py in MATLAB is setting up a Python virtual environment with shared libraries that are compatible with MATLAB. The matpy environment described in Section 2.​5.​1 has been verified to work on Linux Ubuntu 20.04, Windows 10, and macOS 11.6, all with MATLAB 2020b.

Next, activate this environment, and invoke the matlab command from it:

console:
> conda activate matpy
> matlab

Within MATLAB, check if it already recognizes Python, and if it doesn’t, update the configuration to point to your Python installation. The pyenv command will tell if MATLAB is already configured for interaction with Python.

MATLAB (2020b):
>> pyenv
ans =
   PythonEnvironment with properties:
          Version: "3.8"
       Executable: "/usr/local/anaconda3/2020.07/bin/python"
          Library: "/usr/local/anaconda3/2020.07/lib/libpython3.8.so"
             Home: "/usr/local/anaconda3/2020.07"
           Status: NotLoaded
    ExecutionMode: InProcess

Note in particular the last entry, “ExecutionMode: InProcess.” A troubleshooting option we’ll see in the next section is to change this to “OutOfProcess.”

If the pyenv command comes up empty, invoke it a second time telling it the path to your Python executable, for example:

MATLAB (2020b):
>> pyenv('Version','/usr/local/anaconda3/2020.07/bin/python')

On Windows, find the path to the Python interpreter by opening an Anaconda Prompt and entering where python.exe:

Windows:
(base) C:Usersal> where python.exe
C:ProgramDataAnaconda3python.exe
C:UsersalAppDataLocalMicrosoftWindowsAppspython3.exe

Specify the path to the Anaconda version rather than Microsoft’s! Microsoft’s version doesn’t come with NumPy, SciPy, or many of the additional modules you installed into your matpy virtual environment, so if you use it, most of the examples will fail.

Tip

MATLAB on Windows may complain with

Error using pyversion

Path argument does not specify a valid executable.

when setting the Python executable path with pyenv or pyversion. One solution to this error is to reinstall Anaconda, this time selecting “Install for all users.”

On Linux and macOS, the comparable command is which python3, done from a terminal which is configured for Anaconda Python.

Your MATLAB version will support a subset of available Python versions; 2020b supports Python 2.7, 3.6, 3.7, and 3.8. Both MATLAB and Python must be on compatible architectures, that is, both 64 bits or both 32 bits.

6.2 Does It Work?

There are many ways a MATLAB/Python hybrid environment can fail, most of which trace back to conflicting shared libraries (or DLLs in the Windows world). Both MATLAB and Anaconda’s Python distribution use the Intel Math Kernel Library (MKL), Intel’s OpenMP library, and Qt 5 libraries, among others—but the library versions aren’t necessarily at the same level. It is entirely possible for MATLAB or Python to crash because one called an incompatible library function from the other’s installation. The MathWorks’ ability to get these two large, complex applications to coexist peacefully most of the time is a considerable feat of software engineering.

Before proceeding, verify that MATLAB can load Python modules without difficulty. Let’s jump straight to a challenging command:

MATLAB (2020b):
>> py.importlib.import_module('numpy')
At least four different things can happen:
  • The successful case shows a long list of NumPy module attributes. If you see this, continue to the next section.

  • Python Error: ImportError.

  • Unable to resolve the name numpy.

  • Segmentation violation detected.

The last three errors can be caused by a Python installation that is not version consistent with, or not visible to, MATLAB; the Python installation does not have NumPy installed; or you are not in the matpy conda environment and are hitting shared library conflicts between MATLAB and Python.

Possible workarounds:

1) Change “ExecutionMode” reported by pyenv from the default “InProcess” to “OutOfProcess” with

MATLAB (2020b):
>> pyenv("ExecutionMode","OutOfProcess")

2) [Linux only] Set the Python dynamic library load flag to 10:

MATLAB (2020b):
>> py.sys.setdlopenflags(int32(10));

I set this in my startup.m file on my Linux computer.

3) [Linux or macOS] If the terminal from which you configured matpy and then launched MATLAB shows an error about OpenMP library conflicts, try setting the environment variable KMP_DUPLICATE_LIB_OK to TRUE. This can be done in Python’s view of the environment variables within MATLAB with

MATLAB (2020b):
>> os = py.importlib.import_module('os');
>> os.environ.update(pyargs('KMP_DUPLICATE_LIB_OK','TRUE'))

I include the preceding two lines in my startup.m file on my macOS computer.

See Section 6.4 for details on creating and modifying startup.m.

If the module import still fails, it is likely lower-level libraries installed to your matpy environment won’t work with your version of MATLAB. There’s no clear method to determining the right versions though.

6.3 Importing (and Reloading) Python Modules

As mentioned in Section 3.​14, any Python source file may be imported as a Python module. Functions and classes in a module file can only be accessed if the file is in the Python search path, sys.path (see Section 6.6 on how to add directories to the Python search path within MATLAB), or exists in the current directory. Once on the search path, functions and classes in the Python module can be accessed in MATLAB in two ways: directly through py or as an alias created with py.importlib.import_module().

As an example, say we have a Python file (a.k.a. a module), integrate.py, containing three functions that perform numeric integration: rectangle(), trapezoid(), and simpsons(). We can call the trapezoid() function using either of these methods:

  • Directly through py

>> area = py.integrate.trapezoid(Fn, a, b, delta)
  • Through a module import

>> Int = py.importlib.import_module('integrate')
>> area = Int.trapezoid(Fn, a, b, delta)
Code changes to integrate.py are not seen in MATLAB if the changes are made after the module has already been imported. We need to explicitly reload the module for code updates to be seen:
>> py.importlib.reload(Int);

The reload method only works for modules brought in with py.importlib.import_module(). Functions accessed through py.modulename.functionname() cannot be reloaded; the only option there is to restart MATLAB.

6.4 Configure startup.m for Python Work

MATLAB will run the commands in startup.m at the start of each session if it finds such a file in the userpath directory. On my Linux computer, this is

MATLAB:
>> userpath
   '/home/al/Documents/MATLAB'

The location can be changed by calling userpath with the desired location, for example, userpath('/home/al/mfiles').

userpath is also the first directory in path, the collection of directories MATLAB searches to find m-files and compiled extensions. This directory is a perfect location for m-files containing utilities you use often.

To optimize your Python-within-MATLAB experience, make these customizations:
  • Linux: Add to your startup.m file the line py.sys.setdlopenflags(int32(10));

This will help deconflict shared library loads common to MATLAB and Python and reduce occurrences of MATLAB crashes.
  • Windows: Figure out the location of the Python executable associated with the matpy virtual environment and then add to your startup.m file a line that calls pyversion() with this path. This can be done in two steps. First, open an Anaconda Prompt, then

(base) C:> conda activate matpy
(matpy) C:> where python.exe
You may see multiple paths; choose the one containing matpy. Then, in startup.m, add, for example
matpy_ path = 'C:/ProgramData/Anaconda3/envs/matpy/python.exe'; pyversion(matpy_path);
fprintf('Configured Python to %sn', matpy_ path);
  • Add to your userpath directory a copy of, or a symbolic link to, py2mat.m, mentioned in Section 6.5.7 and listed in Appendix D. You will call this function frequently from MATLAB code that calls Python functions.

    Tip A platform-independent way to edit your startup.m file is with this MATLAB command:

    >> edit(fullfile(userpath,'startup.m'))

    (If the file does not exist, you’ll be prompted on whether or not to create it; click Yes.)

My Linux-based startup.m looks like this:

MATLAB 2020b:
pyenv('Version','/usr/local/anaconda3/2020.07/envs/matpy/bin/python');
py.sys.setdlopenflags(int32(10)); % prevents crashes w/numpy
addpath '/home/al/book/code/matlab_py'
fprintf('Ran code in /home/al/Documents/MATLAB/startup.m ')

On macOS, my startup.m is

MATLAB 2020b:
fprintf('Running /Users/albert/Documents/MATLAB/startup.m' )
os = py.importlib.import_module('os');
os.environ.update(pyargs('KMP_DUPLICATE_LIB_OK','TRUE'))

6.5 Create Python Variables and Call Python Functions in MATLAB

We saw in Chapter 4 that Python and MATLAB data containers—lists, cell arrays, structs, dictionaries, numeric arrays—are similar but do not have a direct one-to-one mapping. Here, we’ll give explicit examples that show how to create native Python data containers within MATLAB, how to pass MATLAB data to Python functions, and how to convert native Python variables back to native MATLAB variables.

For now, all Python files mentioned in the following should be in the MATLAB working directory. Later in Section 6.6, we’ll show how to update the Python search path within MATLAB to allow MATLAB to find your Python code in other locations.

6.5.1 Scalars

Native Python doubles, integers, and strings can be created in MATLAB with the functions py.float(), py.int(), and py.str(). Despite their names, py.float() and py.int() create a 64-bit Python floating point value and a 64-bit Python integer.

MATLAB 2020b:
>> x = py.float(4.5)
x =
  Python float with properties:
    imag: 0
    real: 4.5000
    4.5
>> i = py.int(-22)
i =
  Python int with properties:
    denominator: [1x1 py.int]
           imag: [1x1 py.int]
      numerator: [1x1 py.int]
           real: [1x1 py.int]
    -22
>> s = py.str("native Python string")
s =
  Python str with no properties.
    native Python string

py.complex() is not implemented in MATLAB 2020b, but Section 6.5.4 shows py.numpy.complex() can be used. In either case, this is superfluous as complex numbers—and doubles—transfer natively between MATLAB and Python.

Explicit conversion to Python scalars is unnecessary when calling Python functions in MATLAB; these conversions happen automatically. In the following, we create a 32-bit integer, a double, a complex number, and a string in MATLAB and pass them directly to a Python function. The Python function returns modified values back to MATLAB. Within Python, return n, x, c, s means “return a tuple with four values.” On the receiving end, MATLAB sees this tuple as a cell array—variable RV, for “return value”—having four items.

Note that while a 32-bit integer came in, default Python integers are 64 bits, so we retrieve the value back on the MATLAB side through the variable’s .64bit attribute. Other integer types such as unsigned 16 bits are covered in Section 6.5.4. Of the other three return values, only strings have to be converted back to native MATLAB types; doubles and complex scalars pass seamlessly through the MATLAB/Python divide.

MATLAB 2020b:

Python:

>> n = int32(10);

>> x = 3.1415;

>> c = 5.5 - 6.6i;

>> s = "in matlab";

>> RV = py.xfer.modify_scalars(n,x,c,s);

>> pn = RV{1}.int64;

>> px = RV{2}; % double: 1-to-1

>> pc = RV{3}; % complex: 1-to-1

>> ps = string(RV{4});

>> pn,px,pc,ps

pn =

  int64

   15

px =

   15.7075

pc =

  27.5000 -33.0000i

ps =

    "IN MATLAB"

# file: xfer.py

def modify_scalars(n,x,c,s):

    n += 5

    x *= 5

    c *= 5

    s = s.upper()

    return n,x,c,s

6.5.2 Lists and Cell Arrays

In Section 4.​3, we showed that Python lists are analogous to MATLAB cell arrays. A native Python list is created by passing a cell array to py.list(). The list can be modified with any of the Python list methods:

MATLAB 2020b:
>> L = py.list({42, "alpha", 2j})
L =
  Python list with no properties.
    [42.0, 'alpha', 2j]
>> L.pop()
ans =
   0.0000 + 2.0000i
>> L
L =
  Python list with no properties.
    [42.0, 'alpha']
>> L.append(6.022e+23)
>> L
L =
  Python list with no properties.
    [42.0, 'alpha', 6.022e+23]
As with scalars, MATLAB will automatically convert a cell array to a native Python variable when calling a Python function. The Python function, however, receives the MATLAB cell array as a Python tuple (Section 4.​4). The tuple must be converted to a list (or dictionary, or array if possible) if its elements are to be modified. Finally, when a Python function returns a list to MATLAB, the underlying cell array can be accessed through the returned object’s .cell attribute:

MATLAB 2020b:

Python:

>> ca = {'a', int32(20), -9.8}

>> ca =

  1x3 cell array

      {'a'}    {[20]}    {[-9.8000]}

>> RV = py.xfer.modify_list(ca);

>> CA = RV.cell

CA =

  1x4 cell array

   {1x7 py.str} {[-9.8000]}

     {1x1 py.int} {1x1 py.str}

# from file xfer.py

def modify_list(ca):

    # ca comes in as a tuple

    L = list(ca)

    L.append('flipped')

    L.reverse()

    return L

Aside from –9.8, all elements of the CA cell array are native Python objects—not useful at all for MATLAB work; further conversion is needed. We present a generic Python-to-MATLAB data converter function, py2mat(), in Section 6.5.7. There, we’ll revisit this example and show how all elements of CA can be made MATLAB native.

6.5.3 Tuples

Python tuples (Section 4.​4) can be created in MATLAB by passing a MATLAB array to the py.tuple() function. These are only needed when calling Python functions that explicitly expect a tuple argument—and in many cases, that requirement may be satisfied by a MATLAB array. As an example, recall NumPy’s function np.ones() takes a tuple describing the matrix dimensions as its first argument:

Python:
In : np.ones( (3,5) )
Out:
array([[1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.]])

To make the same matrix in MATLAB, we can explicitly pass in a tuple like so:

MATLAB:
>> py.numpy.ones(py.tuple([int64(3),int64(5)]))
  Python ndarray:
     1    1    1    1    1
     1    1    1    1    1
     1    1    1    1    1

(Recall also that NumPy only accepts integer arguments for matrix dimensions. Since MATLAB’s default numeric constant is a double, we have to first explicitly cast 3 and 5 to be 64-bit integers.)

The interface to np.ones() is rather forgiving though, and an array instead of an explicit tuple also works:

MATLAB:
>> py.numpy.ones( [int64(3),int64(5)] )
  Python ndarray:
     1    1    1    1    1
     1    1    1    1    1
     1    1    1    1    1

6.5.4 Numeric Arrays

Unlike scalars and cell arrays, MATLAB numeric arrays cannot be passed to Python functions directly. They must first be converted to NumPy arrays (Section 11.​1) within MATLAB with py.numpy.array().

Similarly, NumPy arrays (“ndarrays”) returned by Python also require conversion to native MATLAB variables before they can be used for computation. While the py2mat() mentioned in the previous section can perform this conversion, here we cover the details of how such conversions are done.

Python ndarray-to-MATLAB array conversion depends on the array type: Python double-precision arrays are converted to MATLAB with double(), unsigned 8-bit integers are converted with uint8(), and so on. The following example uses single-precision floats. The first MATLAB command creates a 2 x 2 matrix; the second creates an equivalent NumPy array from the MATLAB array and then passes it to the Python modify_arr() function in the third command; the final command converts the NumPy array returned by modify_arr() back to a MATLAB array:

MATLAB 2020b:

Python:

>> A = single([2.2 3.3; 4.4 -1])

A =

  2x2 single matrix

    2.2000    3.3000

    4.4000   -1.0000

>> py_A = py.numpy.array(A)

py_A =

   Python ndarray:

     2.2000    3.3000

     4.4000   -1.0000

>> RV = py.xfer.modify_arr(py_A)

RV =

  Python ndarray:

    2.4200    3.6300

    4.8400   -1.1000

>> result = single(RV)

result =

   2x2 single matrix

     2.4200    3.6300

     4.8400   -1.1000

# from file xfer.py

def modify_arr(A):

    return A * 1.1

MATLAB has ten type functions which accept NumPy arrays: single, double, int8, int16, int32, int64, uint8, uint16, uint32, uint64. Absent are single- and double-precision complex conversion functions; as of MATLAB 2020b, the py.numpy.array() function does not accept complex arrays. Complex arrays must be split into their real and imaginary parts to cross the language divide.

Python-native scalar constants with explicit numeric types can also be created in MATLAB using NumPy functions described in Section 11.​1.​3, for example:

MATLAB 2020b:
>> a = py.numpy.uint32(4)
  Python uint32:
   4
>> b = py.numpy.float16(44.55)
  Python float16 with properties:
       base: [1x1 py.NoneType]
       data: [1x1 py.memoryview]
      dtype: [1x1 py.numpy.dtype]
      flags: [1x1 py.numpy.flagsobj]
       flat: [1x1 py.numpy.flatiter]
       imag: [1x1 py.numpy.float16]
   itemsize: [1x1 py.int]
     nbytes: [1x1 py.int]
       ndim: [1x1 py.int]
       real: [1x1 py.numpy.float16]
      shape: [1x0 py.tuple]
       size: [1x1 py.int]
    strides: [1x0 py.tuple]
   44.56
>> c = py.numpy.complex(-3,4)
  -3.0000 + 4.0000i

6.5.5 Dictionaries and Structs

MATLAB’s native key/value data type, container.Map , seems like the ideal counterpart to Python dictionaries, but maps are not supported:

MATLAB 2020b:
>> m = containers.Map({'x','y'},[3.4 5])
m =
  Map with properties:
        Count: 2
      KeyType: char
    ValueType: double
>> py.dict(m)
Error using py.dict
Conversion of MATLAB 'containers.Map' to Python is not supported.

Instead, Python a dictionary is associated with a MATLAB struct. A MATLAB struct passed to a Python function is seen by Python as a dictionary. Changes done to the dict in Python are seen by MATLAB as changes to the struct.

The other direction is less flexible. A dictionary returned by a Python function can only be converted to a MATLAB structured variable (with MATLAB’s struct() function) if the dictionary keys are valid MATLAB variable names. This means the keys must be strings that begin with a letter followed by one or more letters, digits, and/or underscores.

MATLAB 2020b:

Python:

>> x.a = 1;

>> x.b = -1;

>> x.c = 0.5;

>> x

x =

  struct with fields :

    a: 1

    b: -1

    c: 0.5000

>> pyx = py.xfer.modify_dict(x)

pyx =

  Python dict with no properties.

    {'a': 1.0, 'b': -1.0,

     'c': 0.5, 'new': 6.1}

>> xnew = struct(pyx)

xnew =

  struct with fields:

      a: 1

      b: -1

      c: 0.5000

    new: 6.1000

# from file xfer.py

def modify_dict(d):

    d['new'] = 6.1

    return d

Methods to create, inspect, and update Python dictionaries within MATLAB are demonstrated as follows:

Create an empty dictionary:

MATLAB 2020b:
>> a = py.dict()
a =
  Python dict with no properties.
    {}

Create a dictionary with initial values:

MATLAB 2020b:
>> a = py.dict(pyargs('x', 3.4, 'y', 5))
a =
  Python dict with no properties.
    {'x': 3.4, 'y': 5.0}

A limitation is that the odd numbered arguments to pyargs(), meaning the dictionary keys, may only be strings. Python dictionaries can, of course, have integer keys, among many other types, but adding such keys to a Python dictionary within MATLAB violates the rule that keys be valid MATLAB variables. The integer 7 may be a dictionary key in Python, but is clearly not an allowable variable name in MATLAB.

Show all keys:

MATLAB 2020b:
>> py.list(a.keys())
ans =
  Python list with no properties.
    ['x', 'z']

Iterate over keys:

MATLAB 2020b:
>> for x = cell(py.list(a.keys()))
     string(x)
   end
ans =
    "x"
ans =
    "y"

Retrieve a value with a key:

MATLAB 2020b:
>> value = a.get('x')
value =
    3.4000

Add or change a key/value pair:

MATLAB 2020b:
>> a.update(pyargs('z', -10))
>> a
a =
  Python dict with no properties.
    {'x': 3.4, 'y': 5.0, 'z': -10.0}

Remove a key/value pair:

MATLAB 2020b:
>> a.pop('y')
ans =
     5
>> a
a =
  Python dict with no properties.
    {'x': 3.4, 'z': -10.0}

6.5.6 Keyword Arguments

Keyword arguments (Section 3.​8.​3) can be passed to Python functions in MATLAB with pyargs(). A function call that looks like this
Fn(a, B="blue", C=1.0)
in Python, would look like this in MATLAB:
Fn(a, pyargs('B', "blue", 'C', 1.0))

The following example calls the NumPy np.linspace() function with keyword arguments specifying the number of values and data type to return. Both Python and MATLAB versions are shown:

Python:
In : np.linspace(4, 5, num=6, dtype=np.float32)
Out: array([4. , 4.2, 4.4, 4.6, 4.8, 5. ], dtype=float32)
MATLAB:
>> py.numpy.linspace(4, 5, pyargs('num', int64(6), 'dtype', py.numpy.float32))
  Python ndarray :
    4.0000   4.2000   4.4000   4.6000   4.8000   5.0000

6.5.7 Python-to-MATLAB and MATLAB-to-Python Variable Converters

Appendix D has listings for py2mat() and mat2py(), MATLAB functions which convert, where possible, a Python variable within MATLAB into a native MATLAB variable and vice versa. Such conversions may be necessary to pass MATLAB data to Python functions and then convert data returned from Python functions back to native MATLAB variables. These functions can also be found in this book’s Github repository, https://github.com/Apress/python-for-matlab-development .

6.5.8 Traversing Generators

MATLAB cannot traverse generators using a simple for loop. Instead, they have to explicitly call Python’s next() function in a while loop. This Python generator, for example

Python:
#!/usr/bin/env python3
# file: code/matlab_py/gen_demo.py
def gen():
   for i in [10, 20, 30, 40, 50]:
      yield i

can be traversed in MATLAB with

MATLAB:
% code/matlab_py/traverse_gen.m
G = py.importlib.import_module('gen_demo');
fn = G.gen();
while true
    try
        i = py.next(fn);
        fprintf('got i=%d ', i);
    catch EO
        if strcmp(EO.message, "Python Error: StopIteration")
            break
        else
            fprintf(EO.message)
        end
    end
end

Python’s next() raises a StopIteration exception if it is called after the last item has been returned. The MATLAB code checks for this exception to leave the while loop cleanly. If any other error is caught, its message is printed.

A side note: Python’s range() function is not a generator. Instead, it is a “lazily evaluated” function and, as such, can be traversed with a for loop in MATLAB. The catch is that range() has list-like behavior which translates to cell-like behavior in MATLAB. This means the iteration variable i’s value must be dereferenced as i{1}:

MATLAB:
for i = py.range(int64(5))
    fprintf('i = %d ', int64(i{1}))
end
% output:
i = 0
i = 1
i = 2
i = 3
i = 4

6.5.9 Traversing zip()

A zipped Python list can be traversed in MATLAB with a conventional for loop if the call to py.zip() is explicitly expanded with py.list(). Using x as the iteration variable, the iteration item pairs are indexed as x{1}{1} and x{1}{2}:

MATLAB 2020b:
>> for x = py.list(py.zip({1,2,3},{'a','b','c'}))
          fprintf('i=%2d S=%s ', x{1}{1}, x{1}{2})
   end
i= 1  S=a
i= 2  S=b
i= 3  S=c

6.6 Modifying the Python Search Path Within MATLAB

We saw in Section 3.​14.​3 that the Python list sys.path contains the module search path, the collection of directories Python scans to find code. The Python search path can be augmented in Python code with sys.path.append() just as MATLAB’s path can be expanded with addpath. Extending Python’s search path within MATLAB is less obvious. Although Python lists in MATLAB can use the .append() method, lists accessed as module parameters such as sys.path cannot. In other words, the sys module has among its many member functions and parameters the list sys.path , but one cannot use sys.path.append() from MATLAB.

MATLAB (2020b):
>> a = py.list([1,2])
a =
  Python list with no properties.
    [1.0, 2.0]
>> a.append(3)
>> a
a =
  Python list with no properties.
    [1.0, 2.0, 3.0]
>> import py.sys.path
>> py.sys.path.append('/my/new/directory')
Unable to resolve the name py.sys.path.append .

6.6.1 Extending sys.path with an Alias

An interesting twist on .append()’s inability to update lists accessed as module parameters is that .append() works fine on aliases to such lists. Recall from Section 4.​8 that a list assignment in Python such as newlist = oldlist merely creates an alias to oldlist; both newlist and oldlist point to the same data, and a change to either of them is reflected in both.

We can take advantage of this behavior to expand sys.path by using .append() on an alias to sys.path:

MATLAB (2020b):
>> py.sys.path
>> py.sys.path
  Python list with no properties.
    ['/path/dir/one', '/path/dir/two', '']
>> sys_path_alias = py.sys.path
>> sys_path_alias.append('/another/directory')
>> py.sys.path
    ['/path/dir/one', '/path/dir/two', '', '/another/directory']

6.6.2 Extending sys.path with insert()

MATLAB’s insert() function can also modify Python lists. (Aside: insert() is a curious function. It has no built-in help, at least in MATLAB 2020b, and online documentation shows it as a function for adding data to SQLite, MongoDB, and other databases. More revealing is that its signature matches the Python .insert() method for lists—including requiring an explicit integer as the index argument. The obvious conclusion is that insert() is a direct interface to Python’s list .insert() method.)

The following examples show how to add a directory to the beginning of Python’s sys.path list and append a directory to the end of it:

MATLAB (2020b):
>> insert(py.sys.path, int64(0), '/new/directory/start/of/sys/path')
>> nDirs = length(py.sys.path);
>> insert(py.sys.path, int64(nDirs), '/new/directory/end/of/sys/path')

6.6.3 Extending sys.path with append()

There’s an alternate way to achieve the same result by using Python’s append() function from the Python list module. Unfortunately, MATLAB does not offer the ability to cherry-pick individual functions from a module like Python’s from list import append, so we import everything from list. Afterward, we’ll have access to append() which we call to update py.sys.path:

MATLAB (2020b):
>> py.sys.path
  Python list with no properties.
        ['', '/usr/local/anaconda3/2020.07/lib/python38.zip',
        '/usr/local/anaconda3/2020.07/lib/python3.8',
        '/usr/local/anaconda3/2020.07/lib/python3.8/lib-dynload',
        '/usr/local/anaconda3/2020.07/lib/python3.8/site-packages']
>> import py.list.*
>> import py.sys.path
>> append(py.sys.path, '/my/new/directory')
>> py.sys.path
  Python list with no properties.
        ['', '/usr/local/anaconda3/2020.07/lib/python38.zip',
        '/usr/local/anaconda3/2020.07/lib/python3.8',
        '/usr/local/anaconda3/2020.07/lib/python3.8/lib-dynload',
        '/usr/local/anaconda3/2020.07/lib/python3.8/site-packages',
        '/my/new/directory']

One might think the append() function can be called directly from py.list, but that, too, fails:

MATLAB (2020b):
>> py.list.append(py.sys.path, '/my/new/directory')
The class py.list has no Constant property or Static method named 'append'.

Something to be aware of is that the import py.list.* command in MATLAB brought 12 new functions into our namespace:

Python:
In :list.<tab>
append()  copy()   extend()  insert()  pop()     reverse()
clear()   count()  index()   mro()     remove()  sort()

Of the 12, all but mro() and pop() are existing MATLAB commands. Fortunately, MATLAB handles function overloading well. When given MATLAB-native variables, MATLAB invokes its own sort(), append(), and other functions. When these functions are given Python variables, MATLAB calls the Python versions instead.

6.7 Python Bridge Modules

Calls to Python functions from MATLAB can fail for a couple of reasons: there could be dynamic library load collisions that cause MATLAB to crash, or the Python objects visible from MATLAB have attributes or methods that cannot be seen in MATLAB.

The workaround for both cases is to write bridge modules that wrap problematic interfaces with ones MATLAB can handle. An example is the object returned by the weighted least squares function WLS() from the Python statsmodels module; MATLAB cannot access this object’s .predict() method (Section 11.​8). To get around this, we need a small Python module with a function that takes inputs from MATLAB, calls WLS() and the solution’s .predict() method, then returns the answer in a dictionary that MATLAB can access.

Many examples of bridge modules appear in the recipes.

6.8 Debugging Python Code Called by MATLAB

Troubleshooting hybrid MATLAB/Python code can be a challenge because the MATLAB debugger won’t step into Python code. The only recourse to see what’s going on inside Python code is to instrument the code with print statements. Even that doesn’t always work though; sometimes, prints in Python do not appear in MATLAB’s console. If that happens with your code, you’ll need to write debug statements to a file instead. Here’s an example of that:

Python:
with open('/path/to/log_file.txt', 'a') as LOG:
   LOG.write(f'in function abc(), XYZ = {XYZ} ')

The file is opened in append mode with 'a' to accumulate repeated writes. Note also that we have to use the file handle’s .write() method rather than the print() function.

Logging can be automated somewhat with decorators or Python’s own trace module, but both of these methods have drawbacks over tailored print statements. Decorators wrap entire functions and so cannot provide details on the inner workings of functions they act on. The trace module has the opposite problem in that it reports on every line that runs—which can be a prodigious amount of output to wade through—but has no mechanism to show values of variables along the way.

Nonetheless, trace can provide useful insight when a Python function called from MATLAB goes completely off the rails. The following code shows how trace is used to follow execution of functions that compute terms of the Mandelbrot sequence (we’ll see this example again in Chapter 14):

Python:
 1   # code/matlab_py/trace_MB.py
 2   import trace
 3   import numpy as np
 4
 5   tracer = trace.Trace(count=0)
 6
 7   def nIter(c, imax):
 8     z = complex(0, 0)
 9     for i in range(imax):
10       z = z*z + c
11       if abs(z) > 2:
12          break
13     return np.uint8(i)
14
15   def MB(Re, Im, imax):
16     nR = len(Im)
17     nC = len(Re)
18     img = np.zeros((nR, nC),
19             dtype=np.uint8)
20     for i in range(nR):
21       for j in range(nC):
22         c = complex(Re[j],Im[i])
23         img[i,j] = nIter(c,imax)
24     return img
25
26   def do_work(N, imax):
27       nR, nC = N, N
28       Re = np.linspace(-0.7440, -0.7433, nC)
29       Im = np.linspace( 0.1315, 0.1322, nR)
30       img = MB(Re, Im, imax)
31       return img
32
33   def trace_do_work(N, imax):
34       img = tracer.runfunc(do_work, N, imax)
35       return img

Lines 7–31 are the unmodified functions. Tracing is enabled by importing the trace module (line 2) and calling the trace_do_work() function (lines 33–35) which is a wrapper to the regular entry point function, do_work(). First, we’ll do a regular, untraced run by calling do_work() directly:

MATLAB:
>> Im = @py.importlib.import_module;
>> MB = Im('trace_MB')
>> MB.do_work(int64(5), int64(255))
    65   60   60   61   64
    83  203  158   64   65
   254  131  156  205   89
    37   41   42   69  129
    35   37   38   39   40

Now we’ll call the wrapped function, trace_do_work(). It produces a huge amount of output, so before invoking it, enable MATLAB’s diary mode to save STDOUT to a file. Only a small portion of the thousands of lines of output is shown here:

MATLAB:
>> diary
>> MB.trace_do_work(int64(5), int64(255))
MB.trace_do_work(int64(5), int64(255))
 --- modulename: pyio, funcname: do_work
MB.py(28): nR, nC = N, N
MB.py(29): Re = np.linspace(-0.7440, -0.7433, nC)
 --- modulename: function_base, funcname: _linspace_dispatcher
function_base.py(20):     return (start, stop)
 --- modulename: function_base, funcname: linspace
function_base.py(113):    num = operator.index(num)
            :
MB.py(10): for i in range(imax):
MB.py(11):   z = z*z + c
MB.py(12):   if abs(z) > 2:
MB.py(10): for i in range(imax):
MB.py(11):   z = z*z + c
MB.py(12):   if abs(z) > 2:
            :
MB.py(10): for i in range(imax):
MB.py(11):   z = z*z + c
MB.py(12):   if abs(z) > 2:
MB.py(13):     break
MB.py(14): return np.uint8(i)
MB.py(22):   for j in range(nC):
MB.py(21): for i in range(nR):
MB.py(25): return img
MB.py(32):   return img
            :
    65   60   60   61   64
    83  203  158   64   65
   254  131  156  205   89
    37   41   42   69  129
   35    37   38   39   40
>> diary off

6.9 Summary of Steps to Calling Python Code from MATLAB

Calling Python functions from MATLAB takes more effort than calling native MATLAB functions. These steps summarize the process:
  1. 1.

    Implement a pure Python testbed of the code fragment you want to call from MATLAB if you’re calling many functions or manipulating several native Python variables within MATLAB. The pure Python solution will serve as a reference that can provide intermediate values and greatly simplify troubleshooting.

     
  2. 2.

    Start MATLAB from the command line in a virtual environment configured properly for your versions of MATLAB and Python (Section 2.​5.​1).

     
  3. 3.

    Augment the Python search path to include the directory containing your code (Section 6.6) if it isn’t visible from your work directory.

     
  4. 4.

    Import the Python modules you want to call (Section 6.3).

     
  5. 5.

    Convert, where necessary, input arguments to the Python function from MATLAB-native variables to Python-native variables (Section 6.5). Scalars and strings are usually exempt from this, except for cases where the Python function expects an integer. Numeric scalars in MATLAB default to being double-precision floating-point type, so these need to be converted to integers with int32() or int64(). The mat2py() function (Appendix D) can simplify such conversions.

     
  6. 6.

    Call the desired Python function(s).

     
  7. 7.

    Convert objects returned from Python functions to MATLAB-native variables, either by calling built-in MATLAB functions such as struct() or by using the generic py2mat() function (Appendix D).

     
  8. 8.

    If the returned object does not include methods or attributes you need, write a bridge module with functions that explicitly call methods or fetch attributes that aren’t exposed to MATLAB.

     

The call-Python-from-MATLAB recipes that appear in subsequent chapters employ either a subset of these steps for simple cases or use all for the harder ones.

6.10 Call MATLAB from Python

The MathWorks also makes it possible to call MATLAB from Python. This is done with a Python module, matlab.engine, that users can optionally add to a given Python installation.

6.10.1 Install matlab.engine

To install the MATLAB Engine API for Python, first determine the Python environment where you want to add this module by listing your available environments. On Windows, using an Anaconda Prompt, it might look like this:

Windows:
(base) C:Usersal> conda env list
# conda environments:
#
base        * C:ProgramDataAnaconda3
matpy         C:ProgramDataAnaconda3envsmatpy

Next, start MATLAB and explicitly tell it the environment where you want to install the module by giving pyversion() the path to the Python executable for the desired environment. For the example of the matpy virtual environment listed earlier, that would be

MATLAB:
>> pyversion('C:ProgramDataAnaconda3envsmatpypython.exe')

Then install the module from MATLAB with

MATLAB:
>> cd (fullfile(matlabroot,'extern','engines','python'))
>> system('python setup.py install')

The two MATLAB commands earlier are operating system independent and work on Windows, macOS, and Linux.

6.10.2 Call Functions in a New MATLAB Session

The Python interface to MATLAB through its Engine API differs from MATLAB’s py module because matlab.engine needs to either start a MATLAB session or connect to an existing one. In contrast, the py interface from MATLAB to Python works by calling Python library functions; no running Python session is needed. Here, I’ll cover starting a new MATLAB session; connecting to an existing session will come in the following section.

A new MATLAB session can be created by calling start_matlab(). This function returns a handle to an engine instance from which all other MATLAB functions may be called:

Python:
In : import numpy as np
In : import matlab.engine
In : eng = matlab.engine.start_matlab()
In : eng.magic(3)
Out: matlab.double([[8.0,1.0,6.0],[3.0,5.0,7.0],[4.0,9.0,2.0]])
In : np.array(eng.magic(3))
Out:
array([[8., 1., 6.],
       [3., 5., 7.],
       [4., 9., 2.]])

Just as Python functions called from MATLAB return native Python variables, MATLAB functions called from Python return native MATLAB variables. Containers such as numeric arrays are easily converted to NumPy arrays simply by calling np.array() as done earlier.

6.10.3 Call Functions in an Existing MATLAB Session

If you already have a running MATLAB session on the same computer, you can save yourself the considerable session start overhead by connecting directly to the existing session. To do so, you’ll first need to make your existing session shareable: MATLAB:
>> matlab.engine.shareEngine

Next, in Python, check if the engine can see any MATLAB sessions with find_matlab(). If exactly one engine is found, Python can attach to it by calling connect_matlab() without arguments. If multiple sessions are found, supply the name of the session of interest as an argument to connect_matlab():

Python:
In : import matlab.engine
In : matlab.engine.find_matlab()
Out: ('MATLAB_3000634',)
In : eng = matlab.engine.connect_matlab()
In : np.array(eng.magic(6)).astype(np.int)
Out:
array([[35,  1,  6, 26, 19, 24],
       [ 3, 32,  7, 21, 23, 25],
       [31,  9,  2, 22, 27, 20],
       [ 8, 28, 33, 17, 10, 15],
       [30,  5, 34, 12, 14, 16],
       [ 4, 36, 29, 13, 18, 11]])
Variables in the MATLAB session can be seen in a connected Python session through the dictionary eng.workspace:

MATLAB:

Python:

>> z = [ pi exp(1) ];

In : import matlab.engine

In : eng = matlab.engine.connect_matlab()

In : eng.workspace['z']

Out: matlab.double([[3.141592653589793,

                     2.7182818284590455]])

Unfortunately, env.workspace lacks true dictionary functionality. As of MATLAB 2020b, one cannot call len(env.workspace) to count the number of known variables, iterate through variables with for var in env.workspace:, or access variable names through env.workspace.keys().

The env.workspace interface is bidirectional. Values created in Python will appear in the MATLAB workspace:

Python:
In : import matlab.engine
In : eng = matlab.engine.connect_matlab()
In : eng.workspace['a_dict'] = { 'Mo' : 'Monday', 'Tu' : 'Tuesday' }
MATLAB:
>> a_dict
  struct with fields:
    Mo: 'Monday'
    Tu: 'Tuesday'

6.11 Other Mechanisms for MATLAB/Python Interaction

Older versions of MATLAB without the py module have less convenient vehicles for data exchange with Python. These include reading and writing .mat, JSON, XML, or plain text files or sending and receiving data over a network connection. MATLAB invocations of Python functions have to be done through system calls (Section 9.​2) to stand-alone Python programs that implement the functions.

6.11.1 System Calls and File I/O

Calling Python functions directly from MATLAB is convenient and efficient, but is not the only way for the two languages to interact. Older versions of MATLAB without the py module obviously can’t use this method, and even newer versions of MATLAB may crash on calling Python functions that load conflicting shared libraries.

In these cases, the more primitive method of employing system calls and exchanging data via files can be employed. In Section 9.​2, we’ll run other executables from MATLAB with its system() function. This simple example shows how MATLAB would run the Python program compute_stuff.py that exists in the MATLAB working directory:

MATLAB:
>> [status,result] = system('./compute_stuff.py');

result contains the combined lines of STDOUT and STDERR generated by compute_stuff.py, while the integer status has the program’s exit code—zero indicates success, while any non-zero value means the program either was unable to run or ended with an error.

Small amounts of data may be passed from MATLAB to Python via the command line, or environment variables, or the returned result. As the data volume or parsing complexity grows, file or network-based data exchange becomes more attractive. The most MATLAB-friendly file format is, of course, the .mat file; reading and writing these from Python is covered in Section 7.​14.

6.11.1.1 JSON

The .mat file is just one of several formats that MATLAB and Python have in common. Others include HDF5, NetCDF4, XML, and JSON. The first two were covered in Sections 7.​9 and 7.​10. XML, while infinitely flexible, is commensurately tedious because one must laboriously define all entries. In contrast, JSON—JavaScript Object Notation—functions in MATLAB and Python automatically serialize and deserialize complex data structures to and from text strings. These strings can then either be written to and read from text files or sent and received over a network connection. Still, both XML and JSON are text formats and therefore become unwieldy for large numeric arrays.

6.11.1.2 MATLAB to Python via JSON

The MATLAB function jsonencode() serializes a variable into a character array. We’ll use it to write b to a file, then read the file in Python and use the load() function from Python’s json module to convert the text back into a structured variable:

MATLAB (step 1):

Python (step 2):

>> b{1}.pos = [.5; -.3];

>> b{1}.vel = [2.7,3.8];

>> b{2}{1}.u = [0; 3];

>> b{2}{1}.v = { 'a '; 'bc' };

>> b{2}{2} = 22;

>> jsonencode(b)

    '[{"pos":[0.5,-0.3],

     "vel":[2.7,3.8]},[{"u"

     :[0,3],"v":["a ","bc"

     ]},22]]'

>> fh = fopen('b.json', 'w');

>> fprintf(fh, jsonencode(b));

>> fclose(fh);

In : import json

In : fh = open('b.json')

In : b = json.load(fh)

In : fh.close()

In : b

Out:

[ {'pos': [0.5, -0.3],

   'vel': [2.7, 3.8]},

  [{'u': [0, 3],

    'v': ['a ', 'bc']},

   22]]

6.11.1.3 Python to MATLAB via JSON

Going in the reverse direction works equally well: in Python, we can either first create a string serialization of a Python variable with json.dumps() and then write that string to a file or combine both steps with a call to json.dump(). On the MATLAB side, we’ll read the file into a string and then pass the string to jsondecode() to deserialize the string into a variable:

Python (step 1):

MATLAB (step 2):

In : b

Out:

[ {'pos': [0.5, -0.3],

   'vel': [2.7, 3.8]},

[{'u': [0, 3],

    'v': ['a ', 'bc']},

   22]]

In : import json

In : json.dump('b.json', b)

>> fh = fopen('b.json');

>> L = fgetl(fh);

>> fclose(fh);

>> b{1}.pos

    0.5000

   -0.3000

>> b{1}.vel

    2.7000

    3.8000

>> b{2}{1}.u

     0

     3

>> b{2}{1}.v

    {'a '}

    {'bc'}

>> b{2}{2}

    22

6.11.2 TCP/IP Exchange

MATLAB/Python network communication is described in Section 7.​17.​3. Data exchange over a network takes more coding effort than file-based exchange but offers higher I/O performance and does not consume file storage space. Also, the communicating programs must run simultaneously which complicates start-up and shutdown.

The code examples in Sections 7.​17.​3.​1 through 7.​17.​3.​4 sent integers and arrays of double-precision floats in native binary formats. Packing and unpacking each variable to and from binary buffers is tedious though. Another option is to serialize one or more variables into a commonly understood format—JSON, for example—and exchange that.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.226.170.187