PyPy (Posts about CPython)

Leysin 2020 Sprint Report

hodgestar — Tue, 17 Mar 2020 21:57:00 GMT

At the end of February ten of us gathered in Leysin, Switzerland to work on
a variety of topics including HPy, PyPy Python 3.7 support and the PyPy
migration to Heptapod.

We had a fun and productive week. The snow was beautiful. There was skiing
and lunch at the top of Berneuse, cooking together, some late nights at
the pub next door, some even later nights coding, and of course the
obligatory cheese fondue outing.

There were a few of us participating in a PyPy sprint for the first time
and a few familiar faces who had attended many sprints. Many different
projects were represented including PyPy, HPy, GraalPython,
Heptapod, and rust-cpython. The atmosphere was relaxed and welcoming, so if
you're thinking of attending the next one -- please do!

Topics worked on:

HPy

HPy is a new project to design and implement a better API for extending
Python in C. If you're unfamiliar with it you can read more about it at
HPy.

A lot of attention was devoted to the Big HPy Design Discussion which
took up two full mornings. So much was decided that this will likely
get its own detailed write-up, but bigger topics included:

the HPy GetAttr, SetAttr, GetItem and SetItem methods,
HPy_FromVoidP and HPy_AsVoidP for passing HPy handles to C functions
that pass void* pointers to callbacks,
avoiding having va_args as part of the ABI,
exception handling,
support for creating custom types.

Quite a few things got worked on too:

implemented support for writing methods that take keyword arguments with
HPy_METH_KEYWORDS,
implemented HPy_GetAttr, HPy_SetAttr, HPy_GetItem, and HPy_SetItem,
started implementing support for adding custom types,
started implementing dumping JSON objects in ultrajson-hpy,
refactored the PyPy GIL to improve the interaction between HPy and
PyPy's cpyext,
experimented with adding HPy support to rust-cpython.

And there was some discussion of the next steps of the HPy initiative
including writing documentation, setting up websites and funding, and
possibly organising another HPy gathering later in the year.

PyPy

Georges gave a presentation on the Heptapod topic and branch workflows
and showed everyone how to use hg-evolve.
Work was done on improving the PyPy CI buildbot post the move to
heptapod, including a light-weight pre-merge CI and restricting
when the full CI is run to only branch commits.
A lot of work was done improving the -D tests.

Miscellaneous

Armin demoed VRSketch and NaN Industries in VR, including an implementation
of the Game of Life within NaN Industries!
Skiing!

Aftermath

Immediately after the sprint large parts of Europe and the world were
hit by the COVID-19 epidemic. It was good to spend time together before
travelling ceased to be a sensible idea and many gatherings were cancelled.

Keep safe out there everyone.

The HPy & PyPy Team & Friends

In joke for those who attended the sprint: Please don't replace this blog post
with its Swedish translation (or indeed a translation to any other language :).

Using CPython extension modules with PyPy natively, or: PyPy can load .pyd files with CPyExt!

Alexander Schremmer — Fri, 09 Apr 2010 22:56:00 GMT

PyPy is now able to load and run CPython extension modules (i.e. .pyd and .so files) natively by using the new CPyExt subsystem. Unlike the solution presented in another blog post (where extension modules like numpy etc. were run on CPython and proxied through TCP), this solution does not require a running CPython anymore. We do not achieve full binary compatiblity yet (like Ironclad), but recompiling the extension is generally enough.

The only prerequisite is that the necessary functions of the C API of CPython are already implemented in PyPy. If you are a user or an author of a module and miss certain functions in PyPy, we invite you to implement them. Up until now, a lot of people (including a lot of new committers) have stepped up and implemented a few functions to get their favorite module running. See the end of this post for a list of names.

Regarding speed, we tried the following: even though there is a bit of overhead when running these modules, we could run the regular expression engine of CPython (_sre.so) and execute the spambayes benchmark of the Unladen Swallow benchmark suite (cf. speed.pypy.org) and experience a speedup: It became two times faster on pypy-c than with the built-in regular expression engine of PyPy. From Amdahl's Law it follows that the _sre.so must run several times faster than the built-in engine.

Currently pursued modules include PIL and others. Distutils support is nearly ready. If you would like to participate or want information on how to use this new feature, come and join our IRC channel #pypy on freenode.

Amaury Forgeot d'Arc and Alexander Schremmer

Further CPyExt Contributors:

Alex Gaynor
Benjamin Peterson
Jean-Paul Calderone
Maciej Fijalkowski
Jan de Mooij
Lucian Branescu Mihaila
Andreas Stührk
Zooko Wilcox-O Hearn

Using CPython extension modules with PyPy, or: PyQt on PyPy

Alexander Schremmer — Mon, 30 Nov 2009 11:19:00 GMT

If you have ever wanted to use CPython extension modules on PyPy, we want to announce that there is a solution that should be compatible to quite a bit of the available modules. It is neither new nor written by us, but works nevertheless great with PyPy.

The trick is to use RPyC, a transparent, symmetric remote procedure call library written in Python. The idea is to start a CPython process that hosts the PyQt libraries and connect to it via TCP to send RPC commands to it.

I tried to run PyQt applications using it on PyPy and could get quite a bit of the functionality of these working. Remaining problems include regular segfaults of CPython because of PyQt-induced memory corruption and bugs because classes like StandardButtons behave incorrectly when it comes to arithmetical operations.

Changes to RPyC needed to be done to support remote unbound __init__ methods, shallow call by value for list and dict types (PyQt4 methods want real lists and dicts as parameters), and callbacks to methods (all remote method objects are wrapped into small lambda functions to ease the call for PyQt4).

If you want to try RPyC to run the PyQt application of your choice, you just need to follow these steps. Please report your experience here in the blog comments or on our mailing list.

Download RPyC from the RPyC download page.

Download this patch and apply it to RPyC by running patch -p1 < rpyc-3.0.7-pyqt4-compat.patch in the RPyC directory.

Install RPyc by running python setup.py install as root.

Run the file rpyc/servers/classic_server.py using CPython.

Execute your PyQt application on PyPy.

PyPy will automatically connect to CPython and use its PyQt libraries.

Note that this scheme works with nearly every extension library. Look at pypy/lib/sip.py on how to add new libraries (you need to create such a file for every proxied extension module).

Have fun with PyQt

Alexander Schremmer