HPy

hpy 0.9.0: Fourth public release

fangerer — Thu, 12 Oct 2023 19:00:00 GMT

I'm excited to announce that HPy 0.9 was recently released! The fourth official HPy release comes with many new features and was again made available on PyPI.

This is the first release that also provides binary wheels, so installation is now fast and smooth as you expect it.

The major highlights of the release are support for subclassing of built-in types, metaclasses, the trace mode, multi-phase module initialization, and the hybrid ABI. Please checkout the full release notes here on GitHub: https://github.com/hpyproject/hpy/releases/tag/0.9.0

Also, we invested a lot of effort to improve official documentation.

Great news too is that this release was used to make a first (partially-migrated) port of NumPy, we call it NumPy/HPy, run some examples.

What is HPy?

HPy provides a new API for extending Python in C. In other words, you use #include <hpy.h> instead of #include <Python.h>. For more info, look at the official documentation.

Installation

HPy 0.9 is best tested on Linux and Mac OS X systems but there is also initial support for Windows (both x86_64). For CPython, you need to install it manually, using pip:

$ pip install hpy==0.9.0

PyPy and GraalPy usually already come with intrinsic HPy support, so no installation is necessary. HPy 0.9 is already included in GraalPy 23.1. Unfortunately, due to lack of resources, development is still in progress for PyPy but should be finished soon.

Also checkout PyPy's nightly builds to follow the progress:

PyPy nightly builds

To check the version of HPy which is shipped with those, you can either use pip or hpy.universal.get_version().

Major Features

Support subclasses of built-in types

It is now possible to create pure HPy types that inherit from built-in types like type or float. This was already possible before but in a very limited way, i.e., by setting HPyType_Spec.basicsize to 0. In this case, the type implicitly inherited the basic size of the supertype but that also means that you cannot have a custom C struct. It is now possible inherit from a built-in type AND have a custom C struct. For further reference, see HPyType_Spec.builtin_shape and HPyType_BuiltinShape.

Support for metaclasses

HPy now supports creating types with metaclasses. This can be done by passing type specification parameter with kind HPyType_SpecParam_Metaclass when calling HPyType_FromSpec.

HPy Hybrid ABI

In addition to CPython ABI and HPy Universal ABI, we now introduced the Hybrid ABI. The major difference is that whenever you use a legacy API like HPy_AsPyObject or HPy_FromPyObject, the prdouced binary will then be specific to one interpreter. This was necessary to ensure that universal binaries are really portable and can be used on any HPy-capable interpreter.

Trace Mode

Similar to the Debug Mode, HPy now provides the Trace Mode that can be enabled at runtime and helps analyzing API usage and identifying performance issues.

Multi-phase Module Initialization

HPy now support multi-phase module initialization which is an important feature in particular needed for two important use cases: (1) module state support (which is planned to be introduced in the next major release), and (2) subinterpreters. We decided to drop support for single-phase module initialization since this makes the API cleaner and easier to use.

HPy Calling Protocol

This was a big missing piece and is now eventually available. It enables slot HPy_tp_call, which can now be used in the HPy type specification. We decided to use a calling convention similar to CPython's vectorcall calling convention. This is: the arguments are passed in a C array and the keyword argument names are provided as a Python tuple. Before this release, the only way to create a callable type was to set the special method __call__. However, this has several disadvantages. In particlar, poor performance on CPython (and maybe other implementations) and it was not possible to have specialized call function implementations per object (see HPy_SetCallFunction)

HPy meetup and CPython core dev sprint in Brno (Oct 7-10, 2023)

fangerer — Tue, 10 Oct 2023 10:00:00 GMT

We just ended our rather informal HPy meetup in Brno (Czech Republic) which started on Saturday (October 7th) and lasted until Tuesday (October 10th).

The main topics of the HPy meetup were:

The CPython Core Dev Sprint
Cython/HPy

The CPython Core Dev Sprint

Petr Viktorin, one of the CPython core developers, invited us to partially attend the CPython core developer sprint. As a part of the sprint, he organized a C API Summit Monday where we were invited and happily attended this event.

Besides of lots of discussions, there were also very interesting presentations about the C API:

1. C API Work Group Summary

Irit presented the current status of the C API Work Group GitHub organisation. The idea of this GitHub org was to gather curent problems of the C API and also to collect ideas for the (re-)volution of the API. Irit thankfully went over all issues and discussions to structure and summarize them. She identified nine categories for problems:

API Evolution and Maintenance
API Specification and Abstraction
Object Reference Management
Type Definition and Object Creation
Error Handling
API Tiers and Stability Guarantees
The C Language
Implementation Flaws
Missing Functionality

She also mentioned strengths of the C API which the people contributing to the repository agree on:

Heap types
APIs taking a C string to look up based on Python strings
Limited API and Stable ABI

Now, the idea is to write an Informational PEP and then to create:

A process for discussions about replacement APIs
Guidelines for evolution of the C API.

The PEP draft is already available.

2. Petr's Vision for the C API

Petr presented his understanding of how the C API should look in future. The slides for his talk are public. He made seven suggestions:

Split API from ABI so you can compile the API that it would use different ABI variants.
Consistency: The API should have a consistent behavior. E.g. APIs should not return borrowed references and the exception behavior should be clear by the naming convention.
Versioning: Essentially, a C extension should include the ABI version it was compiled with.
Errors: All API functions must be able to signal an error that can be handled.
Concrete types: The API should provide dedicated C types for certain built-in objects to have type safety already at the C language level.
Blueprint structures: Creating types and objects should rather be done by using specifications than by allocating empty objects and filling them with content. Just think of PyType_Spec or PyModuleDef.
Context argument: Every API function should have a context argument.

Except for points 4 and 5, these goals are very close to HPy's. We, the HPy team, have to say that this sounds very good and we would be happy to see that in CPython.

3. HPy's Lessons Learned

We were also asked to give a talk and to concentrate in the talk on two things:

How do handles affect the API, and
What is the reason for the HPy context to contain the function table.

We didn't advertise HPy in our talk but we talked about why opaque handles are not only useful but sometimes necessary and how the API would change compared to having only PyObject *. We further showed that having the function table in the context gives us a lot of flexibility and finally enables us to switch on the debug mode essentially at run time (when an extension is loaded).

We are not sure if we managed to convince everyone that those concepts are useful. The biggest problem is that introducing handles and a context argument will radically change the API and require changes in many libraries (see also C API Workgroup - API Revolution #2 and C API Workgroup - API Revolution #7).

4. Python C API: Past, Present, and Future

Victor did a thorough presentation of the past, present, and (possible) future of the Python C API. He pointed out some past and present problems and gave some impression of its complexity. Victor also suggests to make the stable ABI be the default. Overall, he sees a positive trend in how the C API evolves.

Discussions

We had a lot of good discussions and probably the most important one was about the PEP 703 (noGIL) implementation and the new version of ABI (abi4) that is planned to be compatible with both noGIL and GIL builds of CPython. In order to achieve that, Sam, the author of PEP 703, plans to make PyObject * opaque (so it will narrow to the semantics of a handle) and also, the Py_INCREF will, in some cases, use an indirection over a function pointer to actually call the incref logic. This is, again, a big step forward for alternative implementations.

Cython/HPy

Besides the main focus on the C API Summit at the CPython Core Dev Sprint, we were also hacking a bit on Cython/HPy and made the doctest system work. The problem there was that the doctest module is binding functions to self in order to get a method but HPy did not yet provide the necessary slot HPy_tp_descr_get for that. So, we've implemented that and will include it in the next (minor) release.

We hoped that we would do more hacking on HPy itself but the CPython core dev sprint was worth to spend our time too.

HPy Sprint Status Update and Feedback Session

fangerer — Mon, 26 Sep 2022 10:00:00 GMT

As we announced in this blog post, we are currently having our joint PyPy/HPy/GraalPy sprint in Düsseldorf.

On Thursday (September 22nd, 2022), we did the HPy sprint report. Most notably, we also had several CPython core developers in the call.

The sprint report was streamed on Twitch and the recording is available here:

HPy Sprint Call with CPython Core Developers

One of the main goals in the sprint was to write a PEP to officially propose HPy as an Python C extension API. We started to do so but before we actually submit this PEP, we wanted to have some feedback. So, we invited the Python core developers to our call and some of them joined.

We gave a short overview of HPy since not everyone might know it yet. We talked about the goals and non-goals of HPy and basically asked if Python core developers would be willing to participate in HPy design discussions. We discussed PSF endorsement and the conclusion was that we need to talk to the PSF.

We had some very good discussions and we also got some valuable feedback. Our next steps are now to incorporate this feedback and prepare the PEP to continue discussions there and gather more feedback.

We thank the Python core developers for participating in the call and for the offered support.

Call with NumPy Core Developer Sebastian Berg

Since we are working on the migration of NumPy to HPy, we had a call with Sebastian Berg on Tuesday (September 20th, 2022).

We discussed some problems we discovered during our migration work and were discussing how to solve it.

These were the main discussion points:

NumPy's structured data type allows to have objects in NumPy arrays which problematic for HPy. This is mainly because doesn't allow to use the API during deallocation of an object. Hence, there is no way to determine the location of those objects by fetching the data type (which requires an API call). We discussed to mirror the structure in pure C-level data structures instead.
HPy requires to migration all static types to heap types and we discovered that static type behave a bit differently compared to heap types in the case of multiple inheritance. We discussed if that is a bug that CPython needs to solve or if that is something that NumPy should change. So far, it looks rather to be a CPython problem since heap types are pretty new.
We further discussed how to handle NumPy's C API when migrating to HPy. We agreed that we will probably need to provide a separate NumPy HPy C API as well which will maybe removed at some point in future.

Main Technical Achievements of the Sprint

We have added support for stack traces when running in the debug mode on Mac OS X (PR 346).
We are up to finish our incremental porting example (PR 246)
HPy does now also correctly support object members in HPy types (PR 347).

HPy on GraalPy and Matplotlib/HPy

fangerer — Thu, 08 Sep 2022 15:30:00 GMT

Recently, the GraalVM Python team started a series of blog posts on Medium about HPy on GraalPy and about the migration of Matplotlib to HPy . The second blog post is in particular interesting since it not only describes the migration process but also shows performance numbers. The source code is also publicly available.

Dusseldorf PyPy/HPy/other sprint Sept 19-23, 2022

fangerer — Fri, 29 Jul 2022 10:00:00 GMT

I'm happy to announce that we will finally have another PyPy/HPy/other sprint in Düsseldorf, Germany from September 19-23, 2022.

The sprint will be located at Heinrich Heine University Düsseldorf. Many thanks to Carl Friedrich for organizing this.

We will again follow an open format, i.e., everyone that is interesting in HPy, PyPy, or some related topic is welcome to join the sprint.

See also the announcement on the PyPy blog.

Goals and topics of the sprint

From the HPy point of view, there is no strict agenda. The main goal of the sprint is to make progress on HPy in any way you can think of. This includes but is not limited to:

Review, discuss, maybe merge open GitHub PRs.
Hacking on HPy: implement new features, fix bugs, etc.
Come up with ideas for blog posts or even write blog posts.
Migrate packages to HPy or just determine candidates for migration.
Document the HPy API.
Create small documented and tested porting examples and/or a small porting guide.

Accommodation

We do not organize an accommodation. A hotel close the campus is the HK-Hotel Düsseldorf but we rather recommend to book something in the city center since we will usually end up there in the evenings. The convenient parts of town are something like Altstadt, Stadtmitte, Friedrichstadt, Carlstadt, or Unterbilk.

Registration

There is no formal registration nor any fee for participation. Just to know who we should expect to show up, please follow the instructions on the PyPy blog.

hpy 0.0.4: Third public release

fangerer — Thu, 02 Jun 2022 09:15:00 GMT

HPy 0.0.4 is out! The third official HPy release comes with many new features and was again made available on PyPI.

Major highlights of the release are a bunch of new API functions (e.g. HPyErr_ExceptionMatches, HPyErr_WarnEx, HPy_Contains, and more), Python 3.10 support, better support for native fields (HPyField) and global variables (HPyGlobal), new debug mode features (detect invalid raw data pointer usage, detect invalid closing of argument handles, detect return of invalid handles).

Great news too is that we are now able to provide two more non-trivial projects that have been (partially) migrated to HPy. This is, Kiwisolver and Matplotlib.

What is HPy?

HPy provides a new API for extending Python in C. In other words, you use #include <hpy.h> instead of #include <Python.h>. For more info, look at the official documentation.

Installation

HPy 0.0.4 is best tested on Linux systems but there is also initial support for Windows (both x86_64). For CPython, you need to install it manually, using pip:

$ pip install hpy==0.0.4

PyPy and GraalPython already come with intrinsic HPy support, so no installation is necessary. HPy 0.0.4 will be included in the next release of both. In the meantime, you can download a nightly or dev build:

To check the version of HPy which is shipped with those, you can either use pip or hpy.universal.get_version():

$ pypy -m pip show hpy
Name: hpy
Version: 0.0.4
...

$ graalpython -m pip show hpy
Name: hpy
Version: 0.0.4
...

$ pypy -c 'import hpy.universal; print(hpy.universal.get_version()[0])'
0.0.4

$ graalpython -c 'import hpy.universal; print(hpy.universal.get_version()[0])'
0.0.4

API

We are constantly working on the HPy API and keep adding functions that are missing. We've added following API functions to the new release:

HPyErr_SetFromErrnoWithFilename, HPyErr_SetFromErrnoWithFilenameObjects
HPyErr_ExceptionMatches
HPyErr_WarnEx
HPyErr_WriteUnraisable
HPy_Contains
HPyLong_AsVoidPtr
HPyLong_AsDouble
HPyUnicode_AsASCIIString, HPyUnicode_DecodeASCII
HPyUnicode_AsLatin1String, HPyUnicode_DecodeLatin1
HPyUnicode_DecodeFSDefault, HPyUnicode_DecodeFSDefaultAndSize
HPyUnicode_ReadChar

For an overview of the current API, please refer to the public API declaration in public_api.h, which is used to autogenerate parts of the HPy code and is a reliable list of all the supported functions. Also have a look at additional helpers in inline_helpers.h.

Warning

The HPy API is still considered in alpha status and it's subject to change between versions.

Debug Mode

We again improved HPy's debug mode and added following new features:

Enable Debug Mode via Environment Variable

The debug mode can now be enabled using environment variable HPY_DEBUG. It is possible to enable the debug mode for all HPy extensions or it is also possible to enable it just for certain extensions by enumerating them.

Example:

$ # enable debug mode for all HPy extensions
$ HPY_DEBUG=1 python3 my_application.py

$ # enable debug mode just for ujson_hpy and piconumpy_hpy
$ HPY_DEBUG=ujson_hpy,piconumpy_hpy python3 my_application.py

Detect Invalid Use of Raw Data Pointers

Some API functions return a raw data pointer from an object. For example:

const char* HPyUnicode_AsUTF8AndSize(HPyContext *ctx, HPy h, HPy_ssize_t *size)

returns a raw data pointer to the UTF8 representation of a Python unicode object. HPy doesn't expose the internal representation of the unicode object, so the Python implementation may use an arbitrary internal representation. This means that the UTF8 representation is just temporarily created for this API call and so the raw data must be released at some point. The contract here is that the raw data pointer is valid as long as the corresponding handle is valid.

Example:

#include <string.h>

static const char *s_hello_world = "Hello, World!";

static const char * foo(HPyContext *ctx)
{
    HPy h_unicode = HPyUnicode_FromString(ctx, s_hello_word);
    HPy_ssize_t size;
    const char *res = HPyUnicode_AsUTF8AndSize(ctx, h_unicode, &size);

    /* closing 'h_unicode' is, of course, correct */
    HPy_Close(ctx, h_unicode);

    /* raw data pointer 'res' may have become invalid when closing
       'h_unicode' */
    return res;
}

static int bar(HPyContext *ctx)
{
    const char *s = foo(ctx);

    /* accessing 's' will cause a fatal error in debug mode (on supported
    systems) */
    return strcmp(s, s_hello_world) == 0;
}

It is easy to forget about this resriction and if the raw data pointer is used after the handle was closed, it may point to garbage. If the debug mode is enabled, it will make the underlying memory inaccessible and every access to the pointer will then cause a crash of the application. This is currently only implemented for Linux systems. We use a different strategy on other systems and fill the pointer with some marker bytes that make it easy to detect.

Detect Incorrect Closing of Argument Handles

HPy functions that are called from Python receive handles that are owned by the caller. This means that those handles must not be closed by the callee but it is, of course, possible to erroneously call HPy_Close on them. For example:

HPyDef_METH(foo, "foo", foo_impl, HPyFunc_O, .doc="closing argument")
static HPy foo_impl(HPyContext *ctx, HPy self, HPy arg)
{
    // error: 'arg' is owned by the caller
    HPy_Close(ctx, arg);
    return HPy_Dup(ctx, ctx->h_None);
}

Detect Invalid Handles Returned from Function

A common problem when returning handles is that the author may easily forget to create a new handle. The debug mode now detects situations like the following:

HPyDef_METH(foo, "foo", foo_impl, HPyFunc_NOARGS, .doc="returns arg w/o dupping it")
static HPy foo_impl(HPyContext *ctx, HPy self)
{
    // should be: return HPy_Dup(ctx, self);
    return self;
}

Examples

Besides the known examples, this is HPy's "proof of concept" package, ultrajson-hpy, piconumpy, we are excited to present two new packages we have migrated to HPy:

Kiwi is an efficient C++ implementation of the Cassowary constraint solving algorithm.
Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. Since Matplotlib also has a dependency to NumPy, the migration is not fully finished but luckily, HPy provides the legacy compatibility API such that we can still call legacy C API functions from HPy.

We are still cleaning these ports up and will write another blog post about the ports and open them for discussion with the project owners.

hpy 0.0.3: Second public release

fangerer — Fri, 22 Oct 2021 15:00:00 GMT

HPy 0.0.3 is out! This is already HPy's second officially released version and was again made available on PyPI.

The major highlights of this release are new API functions, new debug mode features, and Windows support.

What is HPy?

HPy provides a new API for extending Python in C. In other words, you use #include <hpy.h> instead of #include <Python.h>. For more info, look at the official documentation.

Installation

HPy 0.0.3 is best tested on Linux systems but there is also initial support for Windows (both x86_64). For CPython, you need to install it manually, using pip:

$ pip install hpy==0.0.3

Note

Currently, we provide only the sdist (i.e., the .tar.gz, no binary wheels). But we plan to ship binary wheels starting with the next release.

PyPy and GraalPython already come with intrinsic HPy support, so no installation is necessary. HPy 0.0.3 is included in the current releases of both, i.e. PyPy 7.3.6 (released on 2021-10-17) and GraalPython 21.3.0 (released on 2021-10-19).

To double check the version of HPy which is shipped with those, you can either use pip or hpy.universal.get_version():

$ pypy -m pip show hpy
Name: hpy
Version: 0.0.3
...

$ graalpython -m pip show hpy
Name: hpy
Version: 0.0.3
...

$ pypy -c 'import hpy.universal; print(hpy.universal.get_version()[0])'
0.0.3

$ graalpython -c 'import hpy.universal; print(hpy.universal.get_version()[0])'
0.0.3

API

HPy is still under heavy development and does not yet provide as many API functions as Python's C API, but more and more pieces are being added. In this release, we've added following functions:

HPy_Is
HPyBytes_FromStringAndSize
HPyErr_NewException
HPyErr_NewExceptionWithDoc
HPyUnicode_AsUTF8AndSize
HPyUnicode_DecodeFSDefault
HPyImport_ImportModule

All of the above functions except of HPy_Is have equivalents in the C API. You just need to remove the leading H from the name.

Function HPy_Is was added to be able to do object identity comparisons. In the C API, you would just do a pointer comparison like this:

PyObject *object_a = /* ... */;
PyObject *object_b = /* ... */;
return object_a == object_b;

In HPy, direct comparisons of handles are forbidden by design, because distinct handles can point to the same object. To compare objects by identity, you need to use HPy_Is:

HPy object_a = /* ... */;
HPy object_b = /* ... */;
return HPy_Is(ctx, object_a, object_b);

We've also extended functions HPyArg_Parse and HPyArg_ParseKeywords and added support for format specifier s. The new specifier converts a Python unicode object into a C string. For example:

const char *result;
if (!HPyArg_Parse(ctx, NULL, args, nargs, "s", &result)) {
    return HPy_NULL;
}
printf("result = %s\n", result);

Warning

The HPy API is still considered in alpha status and it's subject to change between versions.

Debug Mode

One of the great features of HPy is the debug mode. It was already introduced a while ago (and therefore already included in release 0.0.2) but it got a new feature that can help C extension developers a lot. HPy's debug mode is now able to track closed handles. Previously, it was not guaranteed that accesses to closed handles will cause a fatal error. This is because it's possible that another object is associated with the closed handle. Then the program would just operate on the wrong object. The new feature is able to prevent that.

Examples

The best way to get a glimpse of how to use HPy is to look at examples:

the HPy repository contains a "proof of concept" package. Make sure to checkout the branch release/0.0.3.
ultrajson-hpy is a port of the popular ultrajson package. Make sure to checkout the hpy-0.0.3 branch.
piconumpy contains a very tiny implementation of an array-like class. Make sure to checkout the hpy-0.0.3 branch.

hpy 0.0.2: First public release

antocuni — Thu, 15 Jul 2021 15:00:00 GMT

HPy 0.0.2 is out! This is the first version which is officially released and made available on PyPI.

The major highlight of this release is that it is supported by three different Python implementations: CPython, PyPy and GraalPython.

What is HPy?

HPy provides a new API for extending Python in C. In other words, you use #include <hpy.h> instead of #include <Python.h>. For more info, look at the official documentation.

Installation

HPy 0.0.2 only supports Linux systems, and it's only tested on x86_64. Windows support is already present on master, and it will be included in the next release.

For CPython, you need to install it manually, using pip:

$ pip install hpy==0.0.2

Note

Currently, we provide only the sdist (i.e., the .tar.gz, no binary wheels). See also issue #223, contributions are welcome :).

PyPy and GraalPython ships their own version of HPy, so no installation is necessary. HPy 0.0.2 will be included in the next release of both, i.e. PyPy 7.3.6 (expected in October 2021) and GraalPython 21.2.0 (expected on 2021-07-20). In the meantime, you can download a nightly build:

PyPy 3.7 nightly builds, for example revision 3bf99c09018b
GraalPython: download the latest graalpython-dev package from this page

To double check the version of HPy which is shipped with those, you can either use pip or hpy.universal.get_version():

$ pypy -m pip show hpy
Name: hpy
Version: 0.0.2
Summary: A better C API for Python
Home-page: https://hpyproject.org
Author: The HPy team
Author-email: hpy-dev@python.org
License: MIT
...

$ pypy -c 'import hpy.universal; print(hpy.universal.get_version()[0])'
0.0.2

API

At the moment HPy supports only a small fraction of the full API offered by the old Python/C API, but it is enough to write non-trivial extensions, and the documentation is scarce. public_api.h, which is used to autogenerate parts of the HPy code, is a reliable list of all the supported functions.

Warning

The HPy API is still considered in alpha status and it's subject to change between versions. In fact, the current master is already incompatible with hpy-0.0.2 because of PR #182, which renamed all occurences of HPyContext into HPyContext*.

Examples

The best way to get a glimpse of how to use HPy is to look at examples:

the HPy repository contains a "proof of concept" package. Make sure to checkout the branch release/0.0.2.
ultrajson-hpy is a port of the popular ultrajson package. Make sure to checkout the hpy-0.0.2 branch.
piconumpy contains a very tiny implementation of an array-like class. Make sure to checkout the hpy-0.0.2 branch.

#hpy IRC moves to Libera.Chat

antocuni — Sat, 29 May 2021 10:00:00 GMT

Following the example of many other FOSS projects, the HPy team has decided to move its official #hpy IRC channel from Freenode to Libera.Chat: irc.libera.chat/hpy

The core devs will no longer be present on the Freenode channel, so we recommend to join the new channel as soon as possible.

wikimedia.org has a nice guide on how to setup your client to migrate from Freenode to Libera.Chat.

HPy @ Python Language Summit

antocuni — Wed, 12 May 2021 10:00:00 GMT

Yesterday I had the privilege to give a talk about HPy (sildes) at the Python Language Summit 2021.

The organizers of the summit will soon publish a full report about the event (edit: now available here), but for the HPy-specific part, we got generally good feedback. Someone has a few concerns that if CPython is to change the API, HPy might not be going far enough. Others said that Python shouldn't wait for the "perfect" API if HPy can be the "good" one that helps it evolve.

Everyone was open to have HPy-compatible wheels on PyPI, once the HPy Universal ABI stays relatively stable. Many people suggested that we should really write a PEP to propose HPy as a "semi-official" API for Python.

An interesting question was about which are the VM optimizations which are compatible with the HPy API. The following is a non-exhaustive list of things which are known to work because they already used by PyPy and/or GraalPython:

JIT compiler
moving/compacting GCs
storage strategies
maps (also known as "hidden classes")

One notable missing optimization from the list above is tagged pointers. Currently there is no implementation which uses tagged pointers and supports HPy. However, we don't think there is any fundamental design issue in HPy which would prevent it: if you turn tagged pointers into "tagged handles", things should just work out of the box.