HPy (Posts by antocuni)

hpy 0.0.2: First public release

antocuni — Thu, 15 Jul 2021 15:00:00 GMT

HPy 0.0.2 is out! This is the first version which is officially released and made available on PyPI.

The major highlight of this release is that it is supported by three different Python implementations: CPython, PyPy and GraalPython.

What is HPy?

HPy provides a new API for extending Python in C. In other words, you use #include <hpy.h> instead of #include <Python.h>. For more info, look at the official documentation.

Installation

HPy 0.0.2 only supports Linux systems, and it's only tested on x86_64. Windows support is already present on master, and it will be included in the next release.

For CPython, you need to install it manually, using pip:

$ pip install hpy==0.0.2

Note

Currently, we provide only the sdist (i.e., the .tar.gz, no binary wheels). See also issue #223, contributions are welcome :).

PyPy and GraalPython ships their own version of HPy, so no installation is necessary. HPy 0.0.2 will be included in the next release of both, i.e. PyPy 7.3.6 (expected in October 2021) and GraalPython 21.2.0 (expected on 2021-07-20). In the meantime, you can download a nightly build:

PyPy 3.7 nightly builds, for example revision 3bf99c09018b
GraalPython: download the latest graalpython-dev package from this page

To double check the version of HPy which is shipped with those, you can either use pip or hpy.universal.get_version():

$ pypy -m pip show hpy
Name: hpy
Version: 0.0.2
Summary: A better C API for Python
Home-page: https://hpyproject.org
Author: The HPy team
Author-email: hpy-dev@python.org
License: MIT
...

$ pypy -c 'import hpy.universal; print(hpy.universal.get_version()[0])'
0.0.2

API

At the moment HPy supports only a small fraction of the full API offered by the old Python/C API, but it is enough to write non-trivial extensions, and the documentation is scarce. public_api.h, which is used to autogenerate parts of the HPy code, is a reliable list of all the supported functions.

Warning

The HPy API is still considered in alpha status and it's subject to change between versions. In fact, the current master is already incompatible with hpy-0.0.2 because of PR #182, which renamed all occurences of HPyContext into HPyContext*.

Examples

The best way to get a glimpse of how to use HPy is to look at examples:

the HPy repository contains a "proof of concept" package. Make sure to checkout the branch release/0.0.2.
ultrajson-hpy is a port of the popular ultrajson package. Make sure to checkout the hpy-0.0.2 branch.
piconumpy contains a very tiny implementation of an array-like class. Make sure to checkout the hpy-0.0.2 branch.

#hpy IRC moves to Libera.Chat

antocuni — Sat, 29 May 2021 10:00:00 GMT

Following the example of many other FOSS projects, the HPy team has decided to move its official #hpy IRC channel from Freenode to Libera.Chat: irc.libera.chat/hpy

The core devs will no longer be present on the Freenode channel, so we recommend to join the new channel as soon as possible.

wikimedia.org has a nice guide on how to setup your client to migrate from Freenode to Libera.Chat.

HPy @ Python Language Summit

antocuni — Wed, 12 May 2021 10:00:00 GMT

Yesterday I had the privilege to give a talk about HPy (sildes) at the Python Language Summit 2021.

The organizers of the summit will soon publish a full report about the event (edit: now available here), but for the HPy-specific part, we got generally good feedback. Someone has a few concerns that if CPython is to change the API, HPy might not be going far enough. Others said that Python shouldn't wait for the "perfect" API if HPy can be the "good" one that helps it evolve.

Everyone was open to have HPy-compatible wheels on PyPI, once the HPy Universal ABI stays relatively stable. Many people suggested that we should really write a PEP to propose HPy as a "semi-official" API for Python.

An interesting question was about which are the VM optimizations which are compatible with the HPy API. The following is a non-exhaustive list of things which are known to work because they already used by PyPy and/or GraalPython:

JIT compiler
moving/compacting GCs
storage strategies
maps (also known as "hidden classes")

One notable missing optimization from the list above is tagged pointers. Currently there is no implementation which uses tagged pointers and supports HPy. However, we don't think there is any fundamental design issue in HPy which would prevent it: if you turn tagged pointers into "tagged handles", things should just work out of the box.

Hello, HPy

antocuni — Mon, 29 Mar 2021 10:00:00 GMT

Welcome to the shiny new HPy blog!

HPy has been around for a while now. The initial discussion started during EuroPython 2019, in the good old times when we could still go to conferences and have real-life meetings. Since then, HPy progressed a lot from the point of view of the actual code, but we have been a bit too silent w.r.t. communicating what we are doing to the external world and to the broader Python community. Hopefully, now that this blog is online we will do a better job at periodically communicating the status of HPy, so make sure to subscribe to the RSS feed.

What is HPy?

Quoting the frontpage of our website:

HPy provides a new API for extending Python in C. In other words, you use #include <hpy.h> instead of #include <Python.h>.

The official Python/C API is specific to the current implementation of CPython: it exposes many internal details which makes it hard:

to implement it for other Python implementations (e.g. PyPy, GraalPython, Jython, IronPython, etc.)
to experiment with new things inside CPython itself: e.g. using a GC instead of refcounting, or to remove the GIL.
to correctly check things like refcount handling: the external API gets mixed in with implementation details that should be hidden.

Over the years, it has become evident that emulating the Python/C API in an efficient way is challenging, if not impossible. The main goal of HPy is expose a C API which is possible to implement in an efficient way on a number of very diverse Python implementations.

There are several advantages to writing your C extension in HPy:

it runs much faster on PyPy, and at native speed on CPython
it is possible to compile a single binary which runs unmodified on all supported Python implementations and versions
it is simpler and more manageable than the Python/C API
it provides an improved debugging experience: in "debug mode", HPy actively checks for many common mistakes such as reference leaks and invalid usage of objects after they have been deleted. It is possible to turn the "debug mode" on at startup time, without needing to recompile Python or the extension itself

See also the official docs for a more in-depth overview.

Show me an example

This is a "normal" Python/C extension:

2021/03/hello-hpy/hello_old.c (Source)

#include <Python.h>

static PyObject* add(PyObject* self, PyObject* args) {
    long a, b;
    if (!PyArg_ParseTuple(args, "ll", &a, &b))
        return NULL;
    return PyLong_FromLong(a+b);
}

static PyMethodDef HelloMethods[] = {
    {"add", (PyCFunction)add, METH_VARARGS, "add two integers"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef moduledef = {
    PyModuleDef_HEAD_INIT,
    "hello_old",
    "hello example using the old Python/C API",
    -1,
    HelloMethods,
};

PyMODINIT_FUNC
PyInit_hello_old(void) {
    return PyModule_Create(&moduledef);
}

We can compile it with a simple setup.py:

from setuptools import setup, Extension
setup(
    name="hello",
    ext_modules = [
        Extension('hello_old', ['hello_old.c']),
    ],
)

$ python setup.py build_ext --inplace
...
$ python
>>> import hello_old
>>> hello_old.add(10, 20)
30

Now, let's start to uncover the HPy equivalent, piece by piece:

#include <hpy.h>

HPyDef_METH(add, "add", add_impl, HPyFunc_VARARGS,
            .doc = "add two integers");

static HPy add_impl(HPyContext ctx, HPy self, HPy *args, HPy_ssize_t nargs) {
    long a, b;
    if (!HPyArg_Parse(ctx, NULL, args, nargs, "ll", &a, &b))
        return HPy_NULL;
    return HPyLong_FromLong(ctx, a+b);
}

There are a bunch of things which are different from the usual C-extension module:

the former PyObject * is now HPy, which we call "a handle". Handles are similar to PyObject *, but are completely opaque: for more information, see the official docs.
There is an additional parameter, HPyContext ctx. One of the problems of the old API is that often it implicitly relies on the existence of a per-thread or per-subinterpreter local state. HPyContext makes this state explicit. This makes the whole API more regular and makes it possible to develop new interesting features such as the Universal ABI and the Debug mode.
HPy introduces the concept of HPyDefs. HPyDef_METH is a macro which generates the definition of an HPyDef static constant named add, which represents the definition of a Python method implemented by the C function add_impl. In this specific example HPyDef_METH contains more or less the same informations as the old PyMethodDef, but HPyDef are more general. For example, when defining custom types you can use things like HPyDef_SLOT, HPyDef_GETSET, etc.
Notice that we no longer need the cast to (PyCFunction). One of the biggest advantages of HPyDef_METH is that since it's a macro, it can automatically generate a forward declaration for add_impl, with the correct signature. This means that if you use the wrong number and/or type of parameters, you get a nice compile-time error instead of an obscure crash at runtime.
The signature corresponding to HPyFunc_VARARGS is slighly different than the old METH_VARARGS: we pass positional arguments as a C array instead of a Python tuple. This means that it is possible to call the function without having to allocate a Python tuple, and for example the PyPy implementation of HPy takes advantage of that. This is very similar to CPython's VectorCall protocol.

Note

In this post, we are using a slightly old version of HPy. If you try with a newer version you should use HPyContext *ctx instead of HPyContext ctx. See also Issue #150 and PR #182.

Let's continue our tour of hello_new.c:

static HPyDef *hello_defines[] = {
    &add,
    NULL
};

static HPyModuleDef moduledef = {
    HPyModuleDef_HEAD_INIT,
    .m_name = "hello_new",
    .m_doc = "hello example using the new HPy API",
    .m_size = -1,
    .defines = hello_defines,
};

HPy_MODINIT(hello_new)
static HPy init_hello_new_impl(HPyContext ctx) {
    return HPyModule_Create(ctx, &moduledef);
}

This is pretty similar to the old code. The biggest change is that instead of declaring an array of PyMethodDef, we create an array of HPyDef as discussed above.

Finally, we need to modify setup.py. Compiling an HPy extension is as easy as adding setup_requires=['hpy.devel'] and use hpy_ext_modules:

2021/03/hello-hpy/setup.py (Source)

from setuptools import setup, Extension

setup(
    name="hello",
    ext_modules=[
        Extension("hello_old", ["hello_old.c"]),
    ],
    hpy_ext_modules=[
        Extension("hello_new", ["hello_new.c"]),
    ],
    setup_requires=["hpy.devel"],
)

Compiling HPy extensions

In this demo, we will show how to setup an environment to try HPy and compile extensions on both CPython and PyPy.

At the moment HPy is still in its early stages and the API is still subject to change, so we have not done any official release yet. For the same reason, if you want to use HPy on PyPy or GraalPython, you need to manually ensure to install a version which is supported. This is just temporary, and this kind of things will be sorted out automatically once we start to roll out official releases.

So, we need to install HPy from the github repo. Moreover, the HPy implementations inside PyPy and GraalPython are lagging behind a little, so we will install a slightly old revision:

HPy revision eb07982
nightly build of the PyPy hpy branch: a2f7c80062e8 for linux64
PyPy nighly builds: main page and hpy branch
GraalPython nightly build: Linux or macOS
source code of this example

The first step is to create a venv for CPython and install hpy:

$ python3 -m venv tryhpy
$ . tryhpy/bin/activate
$ pip install wheel
$ pip install git+git://github.com/hpyproject/hpy.git@eb07982

To install a nightly build of PyPy it is enough to unpack the tarball and run -m ensurepip. We can check what is the HPy version supported by PyPy by calling hpy.universal.get_version():

$ curl -O http://buildbot.pypy.org/nightly/hpy/pypy-c-jit-101860-a2f7c80062e8-linux64.tar.bz2
$ tar xf pypy-c-jit-101860-a2f7c80062e8-linux64.tar.bz2
$ ./pypy-c-jit-101860-a2f7c80062e8-linux64/bin/pypy -m ensurepip

$ ./pypy-c-jit-101860-a2f7c80062e8-linux64/bin/pypy
>>>> import hpy.universal
>>>> hpy.universal.get_version()
('0.1.dev959+geb07982', 'eb07982')

For GraalPython, just unpack the tarball and create a venv:

$ curl -LO https://github.com/graalvm/graalvm-ce-dev-builds/releases/download/21.1.0-dev-20210330_0726/graalpython-dev-linux-amd64.tar.gz
$ tar xzf graalpython-dev-linux-amd64.tar.gz
$ graalpython-21.1.0-dev-linux-amd64/bin/graalpython -m venv hpy-venv
$ hpy-venv/bin/graalpython
>>> import hpy.universal
>>> hpy.universal.get_version()
>>> ('0.1.dev950+g98f448a', '98f448a')

Now that our enviroment is ready, we can compile and try our extensions:

$ cd /path/to/example/
$ . /path/to/tryhpy/bin/activate    # activate the venv
$ python setup.py build_ext --inplace
[...]

$ ls -1 *.so
hello_new.cpython-38-x86_64-linux-gnu.so
hello_old.cpython-38-x86_64-linux-gnu.so

$ python
>>> import hello_old, hello_new
>>> hello_old.add(10, 20)
30
>>> hello_new.add(30, 40)
70
>>>

It worked! One important thing to note is the filename of hello_new: .cpython-38-x86_64-linux-gnu.so is the standard filename for CPython 3.8 extension modules. This happens because by default hpy_ext_modules targets the CPython ABI. As such, from the point of view of CPython hello_new is indistinguishable from hello_old. It also means that HPy is required only to compile it but not to import it later. Finally, we expect the performance to be the very same as the extensions using the old API.

However, we can also explicitly ask HPy to produce an "universal binary", which targets the HPy Universal ABI: as the name implies, universal binaries can be imported by CPython, but also by alternative implementations such as PyPy. We can build universal binaries by passing --hpy-abi=universal to setup.py:

$ # clean the previous build
$ rm -rf build/ *.so

$ python setup.py --hpy-abi=universal build_ext --inplace
$ ls -1 *.so
hello_new.hpy.so
hello_old.cpython-38-x86_64-linux-gnu.so

Note the filename: hello_old is still a CPython-specific extension, but hello_new.hpy.so is an universal binary. Once compiled, you can import it as usual:

$ python
>>> import hello_old, hello_new
>>> hello_old.add(10, 20)
30
>>> hello_new.add(30, 40)
70
>>> hello_new.__file__
'/.../hello-hpy/hello_new.hpy.so'

Note

At the moment of writing, because of Issue #191 if you try to print the repr of hello_new, you see something like this (note the .py extension):

>>> hello_new
<module 'hello_new' from '/.../hello-hpy/hello_new.py'>

Note that on its own, CPython does not know how to import .hpy.so files. The magic is done by the hello_new.py, which is automatically generated by setup.py:

$ cat hello_new.py
[...]
def __bootstrap__():
    [...]
    from hpy.universal import load_from_spec
    ext_filepath = pkg_resources.resource_filename(__name__, 'hello_new.hpy.so')
    m = load_from_spec(Spec('hello_new', ext_filepath))
    [...]
    sys.modules[__name__] = m

__bootstrap__()

Finally, we can try to import our shiny new universal binary on PyPy:

$ /path/to/pypy-c-jit-101860-a2f7c80062e8-linux64/bin/pypy
>>>> import hello_new
>>>> hello_new.add(10, 20)
30
>>>> hello_new.__file__
'/.../hello-hpy/hello_new.hpy.so'
>>>>

Similarly, it also just works on GraalPython:

$ /path/to/graalpython/hpy-venv/bin/graalpython
>>> import hello_new
>>> hello_new.add(10, 20)
30

That's all you need to get started with HPy. What we presented today is just the basics, of course: in the next posts we will dig more into the technical details, and show more interesting features than just a hello world.

Stay tuned!

(edited on 2021-03-31 to include GraalPython)