Skip to content
paveluvPublic

About

Scheme Low Level

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

150 Commits

Folders and files

Repository files navigation

SchemeLL

LLVM for Chez Scheme — and LLVM IR as s-expressions.

SchemeLL gives Scheme the whole LLVM toolchain with no glue code: full FFI bindings over stock libLLVM.so, an in-memory ORC JIT that hands back ordinary Scheme procedures, ahead-of-time object emission, and sll ("Scheme's Low Level") — a complete s-expression dialect of LLVM IR. Programs are plain data, so quasiquote is the macro layer and Scheme is the metaprogramming language.

SchemeLL is built to be a lowering target: if you're writing a compiler in Scheme, sll is the data structure you lower to — then JIT it, optimize it, or emit native objects, all from the same representation.

Clients that prepare modules themselves can query (llvm jit)'s target-triple and data-layout, or (llvm target)'s machine-triple and machine-data-layout, before building instructions. Supply these as sll triple and datalayout headers so builder alignments and optimization see the target from the start. Both layout queries accept an optional list of non-integral address spaces to add to the target's policy. sll:build releases partial modules and builders on failure; a successful module still belongs to its caller. Woof clients normally use Woof's owned execution API.

The project's Scheme code, including the samples below, uses Schematter canonical form: (...) for single-line lists and [...] for multiline lists.

(import (chezscheme) (prefix (sll) sll:))

[define
 fact
 [sll:procedure
  '[[define
     i64
     (@fact (i64 %n))
     [label
      %entry
      (= %base (icmp slt i64 %n 2))
      (br i1 %base (label %one) (label %rec))]
     (label %one (ret i64 1))
     [label
      %rec
      (= %n1 (sub i64 %n 1))
      (= %f (call i64 (@fact (i64 %n1))))
      (= %r (mul i64 %n %f))
      (ret i64 %r)]]]
  "fact"]]

(fact 20)                       ; => 2432902008176640000, running as native code

That's the whole program: one import, and IR-as-data becomes a callable native procedure (in about 20 ms, ORC JIT included). The syntax is textual LLVM IR transliterated — commas dropped, parens added — so anything you know about IR carries over directly.

Programs are data

Because an sll program is a list, generating code is just building lists. Here is a fully unrolled x^n, specialized at run time:

[define
 (power-prog n)
 `[[define
    i64
    (@pow (i64 %x))
    [label
     %entry
     ,@[let
        loop
        ((i 1) (prev '%x) (acc '()))
        [if
         (>= i n)
         (reverse (cons `(ret i64 ,prev) acc))
         [let
          ((next (sll:name '%p i)))
          (loop (+ i 1) next (cons `(= ,next (mul i64 ,prev %x)) acc))]]]]]]]

((sll:procedure (power-prog 11) "pow") 2) ; => 2048

(sll:name assembles %/@ names from symbol, string, and integer pieces — the generator's replacement for string->symbol+format.)

The same trick scales to real problems: platform-specific code becomes a Scheme function returning the platform-specific forms (examples/aot/hello-portable.ss cross-compiles one source into both x86-64 and AArch64 Linux objects this way).

A 194-byte executable, no toolchain

tools/sllc.ss compiles .sll files using the LLVM C API alone, and for self-contained programs it even writes the final static executable itself (a built-in minimal ELF64 emitter; no compiler, assembler, or linker anywhere).

A .sll file is the inverted format: its top level is sll data, and Scheme is escaped into it. The whole file is one quasiquote body — ,expr and ,@expr evaluate at compile time, top-level (scheme ...) forms hold the definitions and imports they use, and plain data is the degenerate case. This is examples/aot/hello-metaprog.sll, a self-contained file that picks the host's kernel ABI and builds its inline asm structurally, at compile time:

[scheme
 (import (prefix (sll asm) asm:))
 [define-values
  (instr nr-reg ret-reg arg-regs clobbers sys-write sys-exit)
  [case
   (machine-type)
   ((a6le ta6le) (values "syscall" 'rax 'rax '(rdi rsi rdx) '(rcx r11) 1 231))
   ((arm64le tarm64le) (values "svc #0" 'x8 'x0 '(x0 x1 x2) '() 64 94))]]
 (define (syscall nr . args) ...)] ; a few lines of generator

[define
 void
 (@_start)
 [label
  %entry
  ...                           ; plain sll, verbatim
  ,@(syscall sys-write '(i64 1) '(ptr %buf) '(i64 17))
  ,@(syscall sys-exit)
  (unreachable)]]
$ scheme --libdirs . --script tools/sllc.ss --opt O2 --exe examples/aot/hello-metaprog.sll
wrote executable examples/aot/hello-metaprog (194 bytes, entry #x400078)
$ ./examples/aot/hello-metaprog
Hello, SchemeLL!

194 bytes, talking to the kernel directly, from constraint strings no human spelled. sllc also emits relocatable objects and assembly (including cross-target: an x86 host emits genuine AArch64 objects), JIT-runs @main with --run, and prints your program as textual LLVM IR with --render-llvm-ir — using SchemeLL's own pure-Scheme renderer, no LLVM in the path.

Verified against LLVM itself

sll is not a toy subset. Its coverage was driven by round-tripping LLVM's own regression corpus — all 36,488 .ll files — through parse → sll:unbuild → sll:build → print and demanding byte-identical canonical output:

  • 98.9% of every file LLVM's parser accepts round-trips exactly (35,324 of 35,709; the remainder sit in named, documented buckets, every one a C-API limitation — the campaign's definition of 100% was utilizing the C API to full potential, and it got there).
  • Zero unexplained failures: no mismatches, no crashes, across the entire corpus.
  • IR construction runs at roughly 150 µs per module (the corpus harness builds ~28,000 modules in ~4 seconds, benchmarked on every run).
  • The exclusions ledger (project/not-modeled.md) maintains the invariant implemented ∪ documented = LLVM IR — checked by tests that parse the LLVM C headers as the oracle.

Everything that can appear in IR is expressible: all 67 opcodes, EH funclets, atomics with syncscopes, operand bundles, statepoints, inline asm (all dialects), constant expressions, metadata operands, scalable vectors, ifuncs, aliases, unnamed struct types, cross-function blockaddress, bit-exact NaN payloads, module asm, datalayout/triple, and more.

The stack

layer what it is
(llvm raw) the C API verbatim — (prefix (llvm raw) LLVM) reconstructs exact C names
(llvm ir) safe construction: contexts/modules/builders as records with ownership tracking — use-after-free raises a Scheme condition instead of segfaulting
(llvm jit) ORC LLJIT: foreign signatures derived from LLVM types automatically; JIT'd code resolves process symbols (call @cos or @puts by declaring them); refuses modules targeting a foreign platform
(llvm target) objects and assembly, to disk or bytevector; any backend in your libLLVM (X86, AArch64, ARM, RISCV, WebAssembly on stock Debian)
(llvm object) bounded ELF64 little-endian relocatable-object sections, symbols and explicit-addend relocations, owned as Scheme data
(llvm disassemble) instruction boundaries and text through LLVM's C API; malformed or truncated instructions raise
(llvm datalayout) the datalayout string as structured data: parse/unparse, byte-identical round-trips (validated against every layout in LLVM's test corpus), unknown components pass through verbatim
(sll) the s-expression dialect: build, jit, procedure, dump, and unbuild (modules back into sll data)
(sll render) sll->ll: textual LLVM IR from sll data in pure Scheme
(sll asm) structured inline asm: named operands, computed $N numbering, assembled constraint strings
— the freestanding kernel ABI ((abi ...)) moved to the Woof repo (2026-08-27): SchemeLL is a pure LLVM wrapper; machine and kernel knowledge live with the systems language
tools/sllc.ss the .sll compiler: .o / .s / --run / --exe / IR printing

For code admission, (jit:make codegen inspector) calls inspector with a bytevector copy of each actual ORC object before linking. Return normally to accept it or raise to refuse materialization. Editing the copy cannot change the emitted code. The callback and its captured values remain alive until JIT disposal; it must not transfer continuations or reenter that JIT. The ordinary zero/one-argument constructors install no inspector. AOT callers can inspect the bytevector from target:emit-object-bytevector before writing or linking it. These APIs expose final code; they do not themselves certify its stack or memory behavior.

jit:function-at-address applies a named function's captured scalar ABI to an alternate native entry address. This is a trusted adapter API: the caller owns that entry and guarantees its ABI. The returned procedure retains the JIT and checks its lifetime on every call, just like jit:function. jit:check-live! lets an adapter check that lifetime before accessing JIT-owned globals; it performs no native call.

object:read-elf64 returns sections, symbols and relocation records with checked bounds. It refuses other object formats, extended section indexes, compressed sections and implicit-addend relocations. No relocation is applied. disassemble:instructions returns (offset byte-count assembly-text) rows and requires a decoder in the loaded LLVM build. Target-specific admission policy belongs to the client.

The stack is written in Scheme. There is no C to compile: everything talks to stock libLLVM through Chez's FFI.

For procedures returned by jit:function or sll:procedure, i1 arguments and results use the integers 0 and 1, not Scheme booleans. An i1 result is masked to its low bit because the native ABI may leave the upper bits unspecified. For example, truncating the byte 2 to i1 returns 0.

Installation instructions

SchemeLL needs Chez Scheme 10 and one qualified LLVM release, 19.1.7 (the default), 20.1.8, or 16.0.6, installed as a shared library with the C API. Nothing else: no C compiler, no Python. The library is located automatically in the layouts below; anything else is pointed at explicitly. LLVM 16 is the release to pick when the bitcode must be read by a consumer that predates opaque pointers (Apple's Metal compiler): it is the last one that can write typed pointers. Everything else works alike on all three; the C API that arrived after 16 is refused by capability name there (see capabilities).

Debian / Ubuntu

apt install chezscheme llvm-19-dev      # or llvm-20-dev, or both

llvm-N-dev installs /usr/lib/llvm-N/lib/libLLVM-N.so plus the headers the coverage oracle checks, and /usr/lib/llvm-N is searched first. Debian's Chez binary is scheme (chez-scheme on some releases; the Makefile tries both). llvm-16-dev comes from apt.llvm.org on current releases.

macOS (Homebrew)

brew install chezscheme llvm@19         # or llvm@20 / llvm@16, or all three

Apple's own toolchain (Xcode, Command Line Tools) ships no LLVM C API library, so Homebrew's keg-only llvm@N is the source of LLVM. Its /opt/homebrew/opt/llvm@N/lib/libLLVM-N.dylib (Apple Silicon; /usr/local/opt/llvm@N on Intel) is found without any configuration: no PATH, LDFLAGS or DYLD_* changes. Homebrew's Chez binary is chez, which the Makefile finds. MacPorts' /opt/local/libexec/llvm-N is searched too.

FreeBSD

pkg install chez-scheme llvm19          # or llvm20

The port installs under /usr/local/llvm19, which is searched. The Chez binary is chez-scheme.

Any other layout

Point SchemeLL at the installation explicitly, either from Scheme before importing the bindings, through the pure selection API:

(import (prefix (llvm selection) llvm:))
[llvm:select!
 [llvm:make-selection
  '[(major-version . 20            )
    (prefix        . "/opt/llvm-20")]]]

or, for the hosted commands (make test, the tools and the examples), on the command line: make test LLVMFLAGS="--llvm 20 --llvm-prefix /opt/llvm-20". Neither library reads the environment: with no selection, the first installed of 19, 20 and 16 is used, and a program that needs a feature states it ((llvm:require! 'typed-pointers)) and gets the release that has it. A prefix holds lib/libLLVM-N.<so|dylib|dll> and, for the coverage tests, include/; a shared-object entry names the library file directly. SchemeLL reads the loaded library's version with LLVMGetVersion and refuses any other release.

Submodules. Schematter, the formatter, is pinned as a submodule: run git submodule update --init --recursive, or clone with --recurse-submodules. make CHEZ=... overrides the Chez binary.

Getting started

$ make build       # compile the libraries to .so (later runs start ~5x faster)
$ make test        # the first installed of LLVM 19, 20, 16
$ make test-llvm20 # a named release (LLVMFLAGS="--llvm 20")
$ make test-llvm16
$ make test-version-cache  # needs all three releases installed
$ make examples    # smoke-runs 38 Scheme scripts and the CLI checks
$ make reference   # (optional) fetch LLVM's test corpus for `make corpus`
$ scheme --libdirs . --script examples/sll/01-add.ss
2 + 40 = 42

make format formats all tracked Scheme sources with Schematter, including .sll, and make check-format checks them. Enable the pre-commit formatting hook with git config core.hooksPath project/hooks. The hook also formats Scheme code blocks in staged Markdown files.

Then read examples/README.md — 38 standalone Scheme scripts in three buckets: sll scripting (21), the binding layers (10), and AOT objects & executables (7), plus the AOT .sll input files.

Design documents live in project/: the sll grammar and its rationale (sll-design.md, built nanopass-friendly: prefix-only forms, no mid-form keywords), the coverage methodology and final standing (coverage-plan.md), and the exclusions ledger (not-modeled.md).

Tested platforms

Each row records the latest SchemeLL test run for an environment; update its row when you retest it. The linked historical records also cover the LLVM 16 regression corpus; an ordinary suite rerun does not repeat that corpus campaign.

Date OS CPU Chez LLVM Result
2026-09-12 FreeBSD 15.0-RELEASE, amd64 (freebsd01) AMD EPYC 9374F; 4 exposed CPUs 10.4.0 (chez-scheme, ta6fb) 16.0.6, 19.1.7, 20.1.8 (/usr/local/llvm16, /usr/local/llvm19, /usr/local/llvm20) 332/332 checks on 16, 362/362 on 19, 372/372 on 20, plus 15 selection checks each; source and compiled libraries; 38 example scripts each (four skip on 16), factorial and the 194-byte FreeBSD ELF executable; version cache 20/19/16/20; all 31,622 LLVM 16 corpus files examined, zero unexplained failures; two Makefile portability fixes
2026-09-19 Debian GNU/Linux 13.7 (trixie), Linux 6.12.107+deb13-amd64, x86_64 AMD Ryzen Threadripper PRO 9965WX 24-Cores; 48 logical CPUs 10.0.0 (scheme, ta6le; Debian 10.0.0+dfsg-5) 16.0.6, 19.1.7, 20.1.8 (/usr/lib/llvm-16, /usr/lib/llvm-19, /usr/lib/llvm-20) Upstream f122438 plus the flags-example correction and ARM-alias fixture fix: 358/358 checks on 16, 389/389 on 19, 399/399 on 20, plus 28 selection checks each; compiled libraries and all 38 example scripts each (four capability skips on 16); factorial and the 194-byte --exe; version cache 20/19/16/20. The LLVM regression corpus was not rerun.
2026-09-12 macOS 15.7.9 (Darwin 24.6), arm64 Apple M3 Pro 10.4.1 (Homebrew, chez) 16.0.6, 19.1.7, 20.1.8 (Homebrew llvm@16, llvm@19, llvm@20) 301/301 checks on 16, 332/332 on 19, 342/342 on 20; 37 examples on each (--exe skipped; four skip on 16 by capability); version cache 20/19/16/20

Notes:

  • --exe writes x86-64 ELF executables for the Linux and FreeBSD process ABIs only; make examples skips that step on other hosts. Objects, assembly and the JIT are exercised by the recorded Linux and macOS runs; this table does not qualify untested hosts.
  • The Linux runs used GNU Make 4.4.1, native target X86, object format elf, and LLVM triple x86_64-pc-linux-gnu. The selected libraries and their C headers reported the exact releases above. The September 12 Linux corpus record retains that campaign's environment and raw results. That run verifies 28,864 LLVM 16 corpus cases through native building, text rendering or documented builder-folding checks; 596 parser rejections and 2,162 explicitly unsupported cases remain accounted for. It does not repeat the earlier LLVM 19/20 corpus campaigns. The September 19 suite and example checks above update the Linux qualification without replacing those historical corpus results or claiming a new ARM/FreeBSD run.
  • On Apple Silicon LLVMGetHostCPUFeatures returns an empty string (the CPU name carries the features), Mach-O section names are spelled segment,section, the stack-map section is __LLVM_STACKMAPS,__llvm_stackmaps, and the JIT's stack-map keeper needs LLVM's \01 no-mangle spelling of __LLVM_StackMaps. The library and its tests handle all of these; see project/HANDOFF.md.
  • The recorded macOS environment uses GNU Make 3.81, which lacks != assignment but supports $(shell ...). The latter is GNU-specific, so the Makefile detects Chez in the recipe shell to also accommodate BSD make.

About

Scheme Low Level

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages