pytessel 0.2.0

A Python interpreter written in Tessel. It reads Python source, parses it, and runs it by walking the syntax tree: no part of CPython is used.

tessel build . -o pytessel          # makes ./pytessel (under 3 MB, nothing else needed)

./pytessel examples/mandelbrot.py   # run a program
./pytessel -c "print(sum(range(101)))"
./pytessel                          # the interactive prompt

>>> def fib(n):
...     return n if n < 2 else fib(n - 1) + fib(n - 2)
...
>>> [fib(i) for i in range(10)]
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

What it runs

PyTessel follows Python 3. The programs in tests/ print exactly what CPython prints for them, error messages included.

• Values: int, float, bool, str, None, list, tuple, dict, set, range, bytes, bytearray, slice, Ellipsis, NotImplemented, with their operators, slices, and methods (str.split, list.sort, dict.setdefault, set.union, bytes.hex, …).
• Bytes: b"…" and rb"…" literals, str.encode and bytes.decode (utf-8, ascii, latin-1; errors= strict, ignore, replace), int.to_bytes and int.from_bytes, files opened with "rb", "wb", "ab".
• Statements: if/elif/else, while and for (with else, break, continue), assignment with unpacking (a, *rest = xs), augmented assignment, del, assert, pass, global, nonlocal, with, try/except/else/finally, raise, and match with every kind of pattern (match and case stay ordinary names elsewhere).
• Functions: defaults, *args, **kwargs, keyword arguments, keyword-only parameters, closures, lambda, decorators, recursion.
• Names live where Python says: a name a function assigns anywhere is its own everywhere in it (reading it too early is an UnboundLocalError), comprehensions have their own variables, except … as e forgets e after its block.
• Generators: a function with yield runs up to its next yield each time it's asked for a value, so one that never ends (while True: yield …) is fine; send, throw, close, yield from and generator expressions. map, filter, zip, enumerate, reversed and iter give a value at a time too, and so do your own classes with __iter__ and __next__.
• Classes: inheritance (several bases too), super(), properties with setters and deleters, staticmethod, classmethod, and the special methods: __init__, __new__, __repr__, __str__, __eq__, __lt__, __add__ and the other operators (with NotImplemented and the reflected ones, __radd__, …), __len__, __getitem__ (given a real slice for x[1:2]), __contains__, __iter__, __next__, __call__, __bool__, __hash__, __enter__, __exit__, __getattr__, __getattribute__, __setattr__, __delattr__, __dir__, __init_subclass__, __class_getitem__, __index__.
• Descriptors, __slots__, metaclasses: your own classes with __get__, __set__, __delete__, __set_name__; __slots__; class C(metaclass=M) with __new__, __init__, __call__ and the metaclass's methods on the class; type(name, bases, namespace); abc (ABC, ABCMeta, abstractmethod).
• Expressions: comprehensions (list, dict, set, generator), f-strings with format specs (f"{price:>10,.2f}", f"{x=}"), % formatting, str.format, chained comparisons, x if c else y, :=.
• Exceptions: the built-in ones (ValueError, KeyError, …), your own classes, tracebacks for the ones nothing catches.
• Built-in functions: print, len, range, sorted, enumerate, zip, map, filter, sum, min, max, isinstance, open, input, eval, exec, and the rest of the usual ones.
• Modules: math, random, time, json, sys, os, os.path, string, and the ones under The standard library: datetime, re, collections, itertools, functools, statistics, operator, copy, textwrap, bisect, heapq, enum, abc; and import of other .py files next to the program.

What it doesn't

• Integers are 64-bit. Python's grow without limit; PyTessel raises OverflowError where a result doesn't fit (2 ** 100).
• yield inside a larger expression. yield is understood where it's nearly always written: as a statement (yield v, yield from g) or as the whole right side of an assignment or a return (x = yield v, x += yield v, x = yield from g, return (yield v)). Anywhere else (print((yield)), if (yield v):, [(yield a), (yield b)]) it raises a SyntaxError when the generator gets there.
• Nothing is ever freed. PyTessel has no garbage collector: a program that makes millions of lists or objects uses more and more memory. And a generator that's dropped half way doesn't run its finally blocks, as Python does when it frees one: only close() does, or leaving (break, return) a for loop that called the generator function itself (for x in lines(): … break).
• Not there: async, complex numbers, classes that inherit from a built-in type (class D(dict), list, tuple, str, int: a TypeError says so), and the rest of the standard library (decimal, fractions, pathlib, csv, …).
• Smaller things that differ:
  • A comparison gives True or False even when __eq__ or __lt__ gives something else back.
  • Error messages name a method plainly (__init__() missing …), where Python 3.10 and later say Widget.__init__().
  • A class body's names are looked for as it runs, in the class, then outwards: Python skips the functions around a class for a name the class itself assigns.
  • A metaclass has no __prepare__; a class with __slots__ has no slot objects of its own (P.x is an AttributeError).
  • Dictionary keys: an object whose __hash__ is that of a tuple (or, with __index__, of a whole number) is the same key as that value; two objects with the same __hash__ are one key even if their __eq__ says they differ.
  • exec(source, namespace) copies the dictionary's entries in and what the code sets back out: a function made there doesn't see what's put in the dictionary afterwards.
  • bytes have no % formatting, translate or maketrans; the codecs are utf-8, ascii and latin-1; memoryview isn't there.
• Speed: about a fifth of CPython's on loops and calls.

The standard library

These modules come with PyTessel: import finds them with no file, in a sandbox too. They do what CPython's do for the parts listed, error messages included. (They show themselves as Python 3.9 does where versions differ: OrderedDict([('a', 1)]).)

| enum | Enum, IntEnum, Flag (IntFlag is the same), auto, unique, EnumType; Color.RED, Color(1), Color["RED"], for c in Color, __members__, aliases, methods and properties on members | An IntEnum's members aren't ints to isinstance (they compare, hash and do arithmetic as their values). StrEnum, _ignore_, the functional way (Enum("Color", "RED GREEN")), __new__ on members. | | abc | ABC, ABCMeta, abstractmethod (alone and under property, classmethod, staticmethod) | register, abstractproperty. Any class with an abstract method left can't be instantiated, whatever its metaclass. |

Most of them are written in Python, in lib/: PyTessel carries their text inside itself (stdlib.tsl) and runs it the first time a module is imported. re's engine is in Tessel (regex.tsl). After changing a file of lib/:

tools/make-stdlib.sh         # writes stdlib.tsl again (needs python3)
tessel build . -o pytessel

A .py file next to the program with the name of one of these modules is imported instead of it, as in Python.

Inside another program

PyTessel is a published Tessel library (Tessel 0.8 or later). Add it to a program and import it; embed.tsl is the way in:

tessel add pytessel

import pytessel

var py = pytessel.newSession()                 // in a sandbox, by default
let r = pytessel.run(py, "def double(x):\n    return x * 2\n")
let c = pytessel.callFunction(py, "double", [.int(i: 21)])   // c.value is .int(i: 42)

• run runs a text in the session (what it defines stays) and gives back what it printed and, if it failed, the error, its line and the traceback. callFunction calls a function the Python defined, with HostValues (none, bool, int, float, str, list). functions lists them with their parameters.
• The sandbox (on unless newSession(sandbox: false)): no open, no input, no os, no time.sleep, no importing files from disk (the modules of the standard library aren't files: they're there).
• budget: on run and callFunction limits how many statements and calls may run; past it the Python gets a TimeoutError it can't catch its way out of. A regular expression that goes on and on is stopped by it too. Importing a module of the standard library costs nothing.
• addModule(session, name, source) gives the Python a module as text for import name.
• The host: a struct of the program that conforms to pytessel.Host, set with setHost, answers __host__("name", …) calls made in Python; hostOf(session) as? YourHost gives it back with what it changed.
• A Session is a value: copy it, run something in the copy, throw the copy away.

How it's made

Generators. The evaluator runs a statement by calling itself for the statements inside it, and a call can't be left half way and gone back into later. So a generator's body runs another way. The parser marks every statement that has a yield somewhere inside it. A generator keeps its own variables and a stack of frames, one for each block it's inside of (the function's body, a loop's, a try's, a with's), each with how far it has got, the loop's iterator, or what was on its way out when a finally began. A statement with no yield in it is run whole by the evaluator, as anywhere else; a marked one is opened into a frame of its own; and at the yield itself the generator stops, and gives the value. The next next or send goes on from the frames as they were left: what's sent becomes the yield's value, and throw and close raise an exception there instead.

About 18,800 lines of Tessel, and 5,100 of Python in lib/. Every .tsl file in the folder is part of the program; tessel.toml says which version of Tessel builds it.

Tests

tessel build . -o pytessel
tests/run.sh             # each tests/*.py, compared with what CPython prints
tests/run.sh --update    # write the expected output again (needs python3)

Licence

MIT: see LICENSE.