Burgled-Batteries: A Python-Lisp Bridge

Hi, I'm Pixie. And I'm here to talk about embedding the CPython interpreter into other things, specifically Common Lisp implementations.

So, a while back I got this idea for a project. I happen to really like Common Lisp, so I thought "Oh, I'll do this in Lisp". One of the requirements for this project is to parse RSS and Atom feeds. If you want to parse RSS and Atom feeds, you use Feedparser. It's the feed parsing library. And it's written in Python.

I have a project. I have a library I'd like to use. So I have a number of options on how to proceed:

Write my project in Python

Should an entire project's language be dictated by one library? Probably not. Maybe there's a Ruby or Perl library I'd like to use.

Write a data shim

If all you're really after is some sort of data, you can massage it into some sort of ad-hoc output format (JSON, XML, protocol buffers, sexps, what-have-you) and read it in on the other side. Not a lot of work, but limited usefulness.

Port it

Don't really have to make any API decisions, don't have to do a lot of research into making things work (just copy whatever the other guy did). Won't be bug-for-bug compatible, might have to port dependencies as well, and when I'm done I only have that one library.

Use the Python library from Lisp

I don't have to write the library, I don't have to maintain the library, I don't even have to look under the hood. Ideal!

If you're on a deadline, or have a boss breathing down your neck, you go with the expedient solution and write a quick shim and move on. But if you're playing around for fun, you get the luxury of aiming for the ideal solution.

Naturally, my first instinct is to look for something which already does what I want. And it turns out mixing Python and Lisp is a fairly common idea.

The first thing I look into is compiling Python. There's this Python compiler, written in Common Lisp, called CLPython. So it's perfect: I don't have to write it, and it's incredibly well integrated into the rest of Lisp.

But it does have some problems:

Heavy

Had to upgrade to a better computer to get it to run at all. (The upgrade was long overdue, but it was originally an issue.)

Writes .lisp files alongside the .py files

Not so great if you're trying to use distro-provided Python libraries. Have to copy them somewhere writable. Not a blocker, but definitely an annoyance.

Incomplete

A number of Python's built-in libraries aren't available yet. "re" exists but is all marked "TODO", "os" doesn't exist at all, etc. These things are requirements for feedparser.

No support for C-level libraries

No numpy, etc. Not a big deal for me.

If I can't use CLPython to compile Python, what about embedding the CPython interpreter into Lisp? An option so popular there were three different libraries already!

Python-on-Lisp

The original embed cpython undertaking, by a very ambitious Lisp newbie. It was a stand-up attempt, and still has some features I don't yet, but it's not really maintained and has undergone a fair amount of bitrot.

pythononlisp-ex

Takes PoL as a base and shimmies data between the Lisp and Python worlds as JSON. Which breaks as soon as you hit something that can't be converted to JSON which amounts to...a lot of things. To be honest, I'm not really sure what the point of embedding the CPython interpreter is if you're just going to use JSON as the transfer format: if JSON works, you don't need to add in the extra complication of embedding CPython.

Pyffi

NIH version of PoL. Fewer features, but the code is a lot easier to follow.

None got very far before being (apparently) abandoned. I think the latest version of Python any of them reference is either 2.4 or 2.5, so they're getting pretty old at this point.

So what does an enterprising programmer do when they have too many options? They add another!

I tried to avoid that. I started off by patching the existing libraries. I started by patching pythononlisp-ex, then realized I was pushing it towards pyffi, so switched to patching pyffi. Eventually, there wasn't really anything left of the original so I gave it a new name: burgled-batteries.

As I begin work on burgled-batteries, I have a few goals in mind. Obviously, I want the ability to use Python libraries from Common Lisp. Users should not have to deal with C-level objects (no pointers or refcnts!). Along that same line, I want deep integration. Using a Python library should feel as much like using a native Lisp library as possible.

To pull this off, I'm really going to need in-depth knowledge of both Python and Common Lisp. Translating between the two in a way that feels natural to both sides I'll need to be aware of the idioms and naming conventions and so forth that each one has.

And of course, because all of this runs through something written in C, C knowledge is going to be very helpful for understanding what's going on.

What I've actually got is Lisp knowledge of arguable depth; I do okay, but there are definitely lots of people way smarter and more capable than I am in the lisp community. And I know enough C to be dangerous—there are good odds my C code would have vulnerabilities—fortunately, really only need to be able to read it, and CPython's source is blissfully straightforward.

Oh, and I've heard of Python.

So I'm not terribly well qualified to tackle this project, but why let a little thing like that stop me? Onward!

[Can pretty much skip right over this slide: the next few slides deal with each point individually]

If you're planning on embedding CPython, look through the documentation because it covers pretty much everything. Much of b-b was generated by copying function signatures from the online documentation and massaging them into lisp with a little elisp function.

For instance, in the high-level API docs you'll find this function PyRun_String. That gets turned into this lisp form, which is just a slightly different form meaning the same thing. Then you read the documentation for the function and realize "Oh, this start parameter isn't just any old integer, it actually takes one of three constants indicating whether the string should be treated as an expression, a statement, or multiple statements", so you note it a little more specifically. Likewise, you note that the global and local bindings are always going to be Python dictionaries, so you get more specific there, too.

Repeat that process for several hundred functions or so and hey, you've got the CPython API available. Of course, if you're writing C code and compiling against the CPython header files, you don't have to worry about any of that.

While the CPython API is really well documented, it does seem to be pretty well targeted towards being embedded within C programs.

Maybe this is just me, but it really feels like there's an implicit assumption of being compiled against the C header files. Not just in terms of constants that have to be grovelled—that's fairly normal—but some of the API functions are actually preprocessor macros. In at least one case, an API function changed from being an exported function in Python 2.5 to being a preprocessor macro in 2.6 and beyond. And pretty much all of the unicode functions are documented as having one name, but exported as another name.

Python 2.5 (exported function)

PyAPI_FUNC(PyObject *) PyImport_ImportModuleEx(
  char *name, PyObject *globals, PyObject *locals, PyObject *fromlist);

Python 2.6+ (preprocessor macro)

#define PyImport_ImportModuleEx(n, g, l, f) \
        PyImport_ImportModuleLevel(n, g, l, f, -1)

Lisp (handle both)

(defpyfun "PyImport_ImportModuleEx"      module!
  ((name :string) (globals dict) (locals dict) (fromlist list))
  (:implementation (import.import-module-level* name globals locals fromlist -1)))

Docs say

PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size)

.h file says

#ifndef Py_UNICODE_WIDE
⋮
# define PyUnicode_FromUnicode PyUnicodeUCS2_FromUnicode
⋮
#else
⋮
# define PyUnicode_FromUnicode PyUnicodeUCS4_FromUnicode
⋮
#endif

Lisp (detect, carry on)

(defpyfun* unicode.from-unicode
    (("PyUnicodeUCS2_FromUnicode" unicode! ((u ucs2-string) (size ssize-t)))
     ("PyUnicodeUCS4_FromUnicode" unicode! ((u ucs4-string) (size ssize-t)))))

CPython uses reference counting to keep track of an object's liveness. It's a legitimate and widely used strategy, relatively easy to implement. But it's also a PITA if you're not used to having to deal with it.

The CPython API complicates reference counting somewhat: depending on where you are in the C API, references can be new, borrowed, stolen, or copied. Functions return new and borrowed references. A new reference means you now own it, and are responsible for decrementing the reference count when you're done. A borrowed reference means you get to look, but if you want to keep the object you'll have to increment the reference count. Stolen and copied references on the other hand are passed to functions as arguments. Stolen references mean you are no longer responsible for decrementing the reference count. If a function copies the reference, you're still responsible for your reference to the object.

In C, that's not so bad. You have to understand what every function is doing with your reference, but which behavior each function exhibits is chosen in a way that generally saves you lines of code. So there's less manual reference-count adjustment than there might otherwise be at the C level.

But for me, it definitely adds complication. I try to ensure my users never have to deal with reference counts, by dealing with them automatically. But all those possible side-effects on the reference counts from each function mean I have to deal with a tiny explosion of possibilities.

And, as I'm sure you're all aware, the more possibilities there are, the harder it is to check them all and their interactions with each other. In spite of a lot of thinking, I'm still not confident I've gotten all the refcnt bugs worked out on my end.

In CPython, Most C API functions can return errors. Sometimes the value indicating an error is also a legitimate value. You can differentiate between the two using PyErr_Occurred(). Since I handle all the error checking centrally, I just call PyErr_Occurred() after every C call which might error, not bothering with comparing return values. (It was easier that way.)

To get really specific for bit, the single-most important function in the CPython API to be familiar with is probably Py_Initialize. Because it takes care of various initializations for the CPython interpreter, and most of the other API functions will crash or segfault if you have not called it. It might only appear in one place in your codebase, but it's vital to successful embedding. If you forget to call it, you might end up spending hours on end trying to figure out why your very simple code is crashing for no apparent reason. That's no fun.

Py_IncRef and DecRef, on the other hand, you'll use a lot, and they're very important to get right. One too many increments, you leak memory. One too many decrements and it's crashyville. So check, double-check, and peer-review 'em, 'cause you gotta get 'em right.

Depending on what you're doing, you might not ever actually use PyRun_String, but it's good for rudimentary interactive debugging, to check at the most basic level if what's happening is what you were expecting to happen.

And PyErr_Occurred I just mentioned on the last slide.

Of course, there are lots more functions you'll be needing, so check the docs. As I've mentioned before, the docs are amazing. There are tutorials if you want something a little more guided than a function listing. If you're interested in embedding CPython, read the docs. I don't think I can say that enough.

Trying to make Python libraries feel "native" within Lisp means I end up translating Python objects into Lisp objects. I do that by building on top of the very wonderful Common Lisp library CFFI, which provides lots of awesome machinery for doing type translation. I use that machinery to not only translate types, but handle error checking, and provide automagical handling of reference counts. A huge part of what burgled-batteries does is only possible because CFFI makes it tractable.

The API functions are noted through use of the defpyfun macro, which is a big, hairy, organically grown macro, which expands into two CFFI defcfun macros: one which provides the type translation machinery, and one which does not, the latter being useful for the low-level code which implements the type-translation bits.

But not everything in Python can undergo a direct 1:1 translation into Common Lisp.

Python's Boolean type is a subclass of Integer (0 is false), and multiple objects evaluate to false (zero, the empty string, and the None singleton). Lisp only has a single false value: NIL. So if data is going from Python to Lisp and back to Python, there'd be information lossage if all false values were mapped to NIL, because I wouldn't necessarily know what to map NIL back into. On the other hand, if something is false in Python having it evaluate to true on the Lisp side could be rather awkward.

Python uses Tuples like Lisp uses lists and Lists like Lisp uses arrays. So that creates a problem: Do I translate into an object whose type has the same name, or an object more closely matching the spirit of usage? I copied CLPython, so tuples are lists and lists are arrays.

Lisp's condition system is very different from Python exceptions (and even more different from checking return values in C). Lisp conditions are basically a superset of exceptions, so it's easy enough to map exceptions into conditions, but it's not possible to provide the niceties a lisper might expect, like good restarts, because by the time an exception gets to my code the stack has already been unwound.

The implementations of dynamic languages tend to use signal traps for things, and expect floating point traps to be set a certain way—two dynamic languages in the same program are liable to conflict. (I haven't run into this yet. I will.)

In Python function caller decides whether to pass arguments positionally or by keyword, in Lisp function writer decides which arguments are positional or keyword-based. As far as I know, introspection isn't going to tell me which arguments of a function are normally keyword-based, so I'll probably end up with functions having more keyword arguments than they would in Python.

If you start up Python at the command prompt, you find yourself at a little interactive prompt in which you can type Python. If you give it a statement, it runs the statement. If you give it an expression, it returns the value of the expression. That's nice. I'd like to be able to do that with my RUN function.

CPython doesn't provide such an option in the API. There's no "run as if this were from the interactive prompt" function. The closest thing it has, PyRun_String, requires knowing whether your code is an "expression", "statement", or "file" (multiple statements). You have to know what you're running and it affects the return value (if you tell it "this is a statement" it won't return a value, but if you tell it "this is an expression", it will).

Maybe there's a better way to do this, but I get around the limitation by compiling the snippet: first, trying to compile the string as an expression. If that doesn't work, try compiling it as a statement. Evaluate it whichever way it compiled.

Not complicated, but something to keep in mind if you're building a REPL into a game or something. Though if you're doing a REPL, there's also PyRun_InteractiveLoop which might work for that purpose.

What I have now is a library which provides the tools to manually wrap an existing Python library. That is, it's a foreign-function interface where the foreign language is Python.

As I progress, I'd like to add some things that Python-on-Lisp has that I'm missing: namely stream redirection and callbacks. Being able to get at Python streams from Lisp, and Lisp streams from Python, would definitely be handy, as would the ability to run Lisp from Python.

Another nice-to-have would be the ability to map any given Python object into a Lisp object, rather than having to define a translation for each type. Along that same line of thought, it'd be nice to have importing a Python library be as easy as a single line of code specifying which library to wrap. Python provides a lot of introspective capabilities, they should be made use of!

The really big goal is to get to a point where I can throw it all out. I'm a fan of ideal solutions. So even though I've put a fair amount of work into burgled-batteries so far, and expect to put in lots more as time goes by, I'd definitely like to see CLPython get to the point where it obviates the need for burgled-batteries. Now that I have it running on my machine, I'm hopeful I'll be able to help get it there.

(defpytype "PyInt"
  (:type integer)
  (:to   (value type) (int.from-long* value))
  (:from (value type) (int.as-long value)))

(define-foreign-type foreign-python-int-type (foreign-python-type) nil)
(defpyvar "&PyInt_Type" +int.type+)
(defun int.check (o) (object.type-check o +int.type+))
(defun int.check-exact (o) (%object.type-check-exact o +int.type+))
(define-parse-method int
    (&rest options)
  (let ((reference-type
          (or (find :borrowed options) (find :new options) :new))
        (argument-type
          (or (find :stolen options) (find :copied options) :copied)))
    (make-instance 'foreign-python-int-type :actual-type :pointer :borrowedp
                   (ecase reference-type (:new nil) (:borrowed t)) :stolenp
                   (ecase argument-type (:stolen t) (:copied nil)))))
(define-parse-method int!
    (&rest options)
  (parse-type `(can-error (int ,@options))))
(defmethod translate-to-foreign (value (type foreign-python-int-type))
  (int.from-long* value))
(defmethod translate-from-foreign (value (type foreign-python-int-type))
  (int.as-long value))
(defmethod foreign-is-convertable-to-type-p
    (value (type foreign-python-int-type))
  (int.check value))
(defmethod lisp-is-convertable-to-foreign-p
    (value (type foreign-python-int-type))
  (declare (ignorable value)
           (ignore type))
  (typep value 'integer))
(register-python-type 'int (find-type-parser 'int))

(defpyfun "PyRun_String" object!
    ((str :string) (start parser-context) (globals dict) (locals dict)))

(defcfun ("PyRun_String" run.string)
    object!
  (str :string)
  (start parser-context)
  (globals dict)
  (locals dict))

(defcfun ("PyRun_String" run.string*)
    (can-error :pointer)
  (str :string)
  (start parser-context)
  (globals dict)
  (locals dict))

(defun run.string (str start globals locals)
  (multiple-value-bind (#:g997 #:param1005)
      (translate-to-foreign str #<cffi::foreign-string-type :utf-8>)
    (unwind-protect
         (multiple-value-bind (#:g998 #:param1004)
             (translate-to-foreign start #<cffi::foreign-enum parser-context>)
           (unwind-protect
                (multiple-value-bind (#:g999 #:param1003)
                    (translate-to-foreign globals #<foreign-python-dict-type>)
                  (unwind-protect
                       (multiple-value-bind (#:g1000 #:param1002)
                           (translate-to-foreign locals #<foreign-python-dict-type>)
                         (unwind-protect
                              (let ((#:value1001 (cffi-sys:%foreign-funcall "PyRun_String"
                                                                            (:pointer #:g997 :int #:g998 :pointer #:g999 :pointer #:g1000 :pointer)
                                                                            :convention :cdecl
                                                                            :library :default)))
                                (if (%error-occurred-p)
                                    (raise-python-exception)
                                    (translate-from-foreign #:value1001 #<foreign-python-object-type>)))
                           (free-translated-object #:g1000 #<foreign-python-dict-type> #:param1002)))
                    (free-translated-object #:g999 #<foreign-python-dict-type> #:param1003)))
             (free-translated-object #:g998 #<cffi::foreign-enum parser-context> #:param1004)))
      (free-translated-object #:g997 #<cffi::foreign-string-type :utf-8> #:param1005))))

(defun run.string* (str start globals locals)
  (multiple-value-bind (#:g1006 #:param1014)
      (translate-to-foreign str #<cffi::foreign-string-type :utf-8>)
    (unwind-protect
         (multiple-value-bind (#:g1007 #:param1013)
             (translate-to-foreign start #<cffi::foreign-enum parser-context>)
           (unwind-protect
                (multiple-value-bind (#:g1008 #:param1012)
                    (translate-to-foreign globals #<foreign-python-dict-type>)
                  (unwind-protect
                       (multiple-value-bind (#:g1009 #:param1011)
                           (translate-to-foreign locals #<foreign-python-dict-type>)
                         (unwind-protect
                              (let ((#:value1010 (cffi-sys:%foreign-funcall "PyRun_String"
                                                                            (:pointer #:g1006 :int #:g1007 :pointer #:g1008 :pointer #:g1009 :pointer)
                                                                            :convention :cdecl
                                                                            :library :default)))
                                (if (%error-occurred-p)
                                    (raise-python-exception)
                                    #:value1010))
                           (free-translated-object #:g1009 #<foreign-python-dict-type> #:param1011)))
                    (free-translated-object #:g1008 #<foreign-python-dict-type> #:param1012)))
             (free-translated-object #:g1007 #<cffi::foreign-enum parser-context> #:param1013)))
      (free-translated-object #:g1006 #<cffi::foreign-string-type :utf-8> #:param1014))))