[Python] Chapel-Python interopability - export records (or interface with Python dataclasses)

I have the following use case. My main program is written in Python, and I want to accelerate some parts of it. The Python program makes use of dataclasses (~structures) to store parameters, arrays, etc. Consider the following main Python program ; which uses the Point structure.

# 0/main.py

from dataclasses import dataclass

@dataclass
class Point:
x: float


def addOne(p: Point) -> Point:
return Point(p.x + 1.0)


p1 = Point(1.0)
print(p1)
p2 = addOne(p1)
print(p2)


(0) $ python main.py
Point(x=1.0)
Point(x=2.0)


Now I want to accelerate the addOne function. My starting point is to have Chapel define the Point structure, and have Python import it.

# 1/main.py
from libexample import Point, makePoint, printPoint, addOne

p1 = Point(1.0) # or makePoint(1.0)
printPoint(p1)
p2 = addOne(p1)
printPoint(p2)


// 1/libexample.chpl
// -- types used by exported functions need to be exported
// -- cannot export records, so we use extern
// export record Point {
// var x: real(64);
// }


extern {
typedef struct {
double x;
} Point;
}

// -- altenatively
// require "point.h";
// --
// $ cat point.h
// #ifndef POINT_H
// #define POINT_H
// typedef struct {
// double x;
// } Point;
// #endif


extern record Point {
var x: real(64);
}


export proc makePoint(x: real): Point {
var p: Point = new Point(x);
return p;
}

export proc printPoint(in p: Point) {
writeln("Point(", p.x, ")");
}

export proc addOne(in p: Point): Point {
return new Point(p.x + 1.0);
}


A Chapel test program works fine:

// 1/main.chpl
use libexample;


var p1 = makePoint(1.0);
printPoint(p1);
var p2 = addOne(p1);
printPoint(p2);


(1) $ chpl main.chpl
(1) $ ./main
Point(1.0)
Point(2.0)


But compiling as a shared library does not:

(1) $ chpl --library --library-python libexample.chpl

Error compiling Cython file:
------------------------------------------------------------
...
from libc.stdint cimport *
from chplrt cimport *

cdef extern from "libexample.h":
void chpl__init_libexample(int64_t _ln, int32_t _fn);
Point makePoint(double x);
^
------------------------------------------------------------
./chpl_libexample.pxd:6:1: 'Point' is not a type identifier

# ... and more of such errors


As I understand, exporting records and classes is not yet implemented: https://github.com/chapel-lang/chapel/issues/9326.

A working (but tedious and unmaintainable) workaround would be to write Python wrappers to unpack everything into basic types to call into Chapel, and repack the returned value(s)

# 2/main.py
from dataclasses import dataclass


@dataclass
class Point:
x: float


def addOne(p: Point):
from libexample import addOne as addOne_chpl
return Point(addOne_chpl(p.x))


p1 = Point(1.0)
print(p1)
p2 = addOne(p1)
print(p2)


// 2/libexample.chpl
export proc addOne(p_x: real(64)): real(64) {
return p_x + 1.0;
}


Then compiling and running:

(2) $ chpl --library --library-python libexample.chpl
(2) $ python main.py
Point(x=1.0)
Point(x=2.0)


Obviously doing this is undesirable and error-prone, especially when dealing with nested structures.

I have also looked into the Python module and saw it allows defining custom types (https://github.com/chapel-lang/chap...y/packages/Python/correctness/customType.chpl).
As far as I understand, you can call Python code from Chapel (with possibly custom structures), but I want to do the opposite: call Chapel code from Python with custom structures. The structures may be defined first either in Python or in Chapel (though defining them in Chapel first seems to be more reasonable)

Is there currently a "good" way to do this ? Ideally, the structure would be defined only once, without having to rely on custom-written TypeConverters and such (which in practice duplicate the code). Thank you for your help !

Continue reading...