Short Description

Enables Python scripts that are run inside Icy to communicate with other Python instances outside Icy. This allows the access to CPython-only libraries such as Numpy.

Team: Bio Image Analysis
Institution: Institut Pasteur
Website: http://icy.bioimageanalysis.org

Documentation

This plugin provides execnet, a Python module that allows several python instances to communicate. Among its several purposes, it is useful in Icy because it offers a way to use powerful libraries that are only available with CPython and not with Jython, the latter being the heart of the Python scriping engine in Icy. For example, you can use execnet to send images opened in Icy to a script that runs computations with powerful libraries such as Numpy, Scipy or Pandas.

Requirements

You need a working CPython installation outside of Icy. No particular setup is required in this external Python, apart from installing the libraries you plan to use, such as Numpy or Pandas.

Icy will do some invisible work to setup the remote interpreter. It uses execnet under-the-hood, which is capable of bootstrapping itself, and later on it will send some additional modules to setup standard streams (so that you can print in the remote and see the output in Icy) and provide functions to exchange data.

Basic example

The following piece of code (source on GitHub) can be executed from the Script Editor:

Basic IcyExecnetGateway example

On line 1, we import the IcyExecnetGateway class, which will do the hard work of launching a remote interpreter.

Lines 4 and 7 show how to launch a remote interpreter and execute some code in it. In this example, we print some text in the remote interpreter, and that text is automatically sent back to Icy and printed in the console of the Script Editor, as shown in the screenshot below:

Transferring data between Icy and the remote interpreter

Send a pure-Python object to the remote, from Icy:

  • On Icy side: use gateway.send(object),
  • On the remote side: use channel.receive(). channel is a variable automatically available in the remote to exchange data with its parent.

Send a pure-Python object to Icy, from the remote:

  • On the remote side: use channel.send(object).
  • On Icy side: use gateway.receive().

Transfer an image (of type IcyBufferedImage) to the remote, from Icy:

  • On Icy side: convert first the image to a pure-python object using icyexecnetgateway.pack_image(image) and send that object.
  • On the remote side: use channel.receive() to receive the image object, import the numpyexecnet module, and use numpyexecnet.unpack_image(packed_image). The result will be a 2D Numpy array.

Transfer a 2D Numpy array to Icy, from the remote:

  • On the remote side: import the numpyexecnet module, use numpyexecnet.pack_image(array) to convert the Numpy array to a pure-python object, and send that object.
  • On Icy side: use gateway.receive() to receive the array object, and icyexecnetgateway.unpack_image(array) to convert it to an IcyBufferedImage.

Limitations:

  • It is not currently possible to send a Sequence as-is to the remote interpreter. Instead, send each image of the sequence to the remote.
  • Likewise, it is not possible to send Java objects. If you try, you will get a execnet.DataFormatError, saying that the object cannot be serialized. Be sure to convert them to Python objects (for example a Java ArrayList has to be converted to a Python list).

Example (Icy to Numpy/Matplotlib)

In this example, we send an image (first image, first channel, first z, of the currently active sequence) to the remote interpreter. It is received as a Numpy array, and displayed on the screen with the Matplotlib library (source on GitHub).

From Icy to Matplotlib

Example (Numpy to Icy)

In this example, we generate a random 2D array using Numpy ‘rand’ function. We display this array in matplotlib, and we also sendit to Icy. It is received as an IcyBufferedImage. A sequence is made from this image, and displayed (source on GitHub).

From Numpy to Icy

Launching a specific interpreter:

By default, when calling IcyExecnetGateway(), the interpreter that will be launched is the ‘python’ executable that is found on the system path. In some situations, it is desirable to specify another executable (a specific python installation in a virtual environment, or a locally-installed version instead of the system Python in MacOS X). You can specify the desired python executable located at /path/to/python by using: IcyExecnetGateway(python_path=”/path/to/python”).

It is also possible to call an interpreter on a different machine, using execnet specifications. For example, you can call a Python 2.4 interpreter through ssh on a host called wyvern using IcyExecnetGateway(gateway_spec="ssh=wyvern//python=python2.4"). See http://codespeak.net/execnet/basics.html for more details about the specifications.

Coming soon on this page:

  • Examples of fancy things to do with Numpy, Scipy, scikit.image, Pandas, Fenics, Matplotlib, etc.

 

Resources needing this

One review on “Jython execnet for Icy

Leave a Review

Leave a review
Cancel review
View full changelog
Close changelog

Changelog

  • Version 1.0.2.0 • Released on: 2020-07-22 18:15:00
    Download
    Description:

    Updated following "Jython For Icy" plugin update (Stephane)

  • Version 1.0.1.0 • Released on: 2020-02-20 17:30:00
    Download
    Description:

    Reverted to previous to fix Aleix issue (Stephane)

  • Version 1.0.0.0 • Released on: 2020-01-09 13:00:00
    Download
    Description:

    Updated following "Jython For Icy" plugin update (Stephane)

  • Version 0.10.0.0 • Released on: 2013-12-09 17:00:27
    Download
    Description:

    Update to development version 1.2.0dev1 plus a lock fix. This fixes the use of execnet as a gateway pool for multiprocess computation.

  • Version 0.9.0.0 • Released on: 2013-11-27 14:58:58
    Download
    Description:

    - Make Numpy array writable.

  • Version 0.8.0.0 • Released on: 2013-11-26 16:20:39
    Download
    Description:

    - Send binary data arrays from Icy to Numpy and back. This dramatically improves the speed of the data transfer between Icy and an external Python interpreter (was taking several seconds for a 512x512 image, now instantaneous). This also enables the transfer of images of any data types without conversion (with some restrictions when sending numpy arrays to Icy), whereas the transfer was limited to double before.

    - Fix the deadlock caused by the handling of errors on the local side inside the 'with'-block.

  • Version 0.7.0.0 • Released on: 2013-11-21 09:50:33
    Download
    Description:

    - Fix missing file (execnet/gateway_base.py)

    - Please note that since JythonForIcy version 0.5, it is not necessary anymore to manually import the 'with' statement.

  • Version 0.6.0.0 • Released on: 2013-11-18 12:34:20
    Download
    Description:

    Update for the new extraction mechanism provided by the Python Extractor plugin.

  • Version 0.5.0.0 • Released on: 2013-11-05 14:17:12
    Download
    Description:

    Update for the new annotation system of Jython For Icy 0.4.

  • Version 0.4.0.0 • Released on: 2013-06-25 16:52:07
    Download
    Description:

    Fixes for previous update (typo in apipkg.py, locations of Icy execnet helper scripts).

  • Version 0.2.0.1 • Released on: 2013-06-25 15:50:15
    Download
    Description:

    Extract python library files to disk instead of loading from the jars. This improves the readability of backtraces. It also improves the compatibility with python libraries like execnet that play with the source files.

    Move back to execnet code from upstream (current git), since local modifications are no longer needed.

    Provide new functions pack_image and unpack_image to exchange images between Jython and CPython/Numpy. They replace the methods send_image and receive_image.

  • Version 0.2.0.0 • Released on: 2013-03-18 16:51:21
    Download
    Description:

    Now provides a class named IcyExecnetGateway. It setups a fully-functioning interpreter with properly redirected standard streams, provides helper methods to exchange images with Icy and is designed to be used via the 'with' statement so that resources are properly cleanup at exit.

  • Version 0.0.1.0 • Released on: 2013-03-05 09:53:14
    Download