Development

Install from source

Requires Python ≥3.10 and Rust ≥1.98. In your Python environment:

git clone https://github.com/AnswerDotAI/basedpl.git
cd basedpl
pip install .

For a standalone executable without Python, run cargo install --path .. Cargo installs it in its bin directory, normally ~/.cargo/bin; put that directory on your PATH.

Commands

cargo test
cargo run -- -e '2×3+4'
cargo fastfmt
maturin develop
pytest -q
ship-rs-build

Rebuild with maturin develop after Rust changes before checking the installed extension. Cargo tests alone do not update the editable Python installation. Use cargo fastfmt, not cargo fmt.

Use lowercase j in complex literals throughout tests and examples, including adapted reference cases. Reserve uppercase J for explicit input-alias tests. Keep archived upstream source unchanged.

Write literal matrices in array notation, [10 20 30 ⋄ 40 50 60], not as a reshape, 2 3⍴10 20 30 40 50 60. Keep ⍴ where the example is about reshape.

The development profile uses optimization level 1 without LTO. Tests inherit these settings. Debug information, assertions, overflow checks and incremental compilation remain enabled.

CI uses the development profile for Rust and Python tests. Distribution wheels use the dist profile: optimization level 2, no LTO, 16 codegen runs, no incremental compilation and stripped symbols.

Documentation

Documentation lives in nbs/. index.ipynb generates both the homepage and README.md. The glyph reference and the remaining guides are Quarto Markdown. Pages that document system functions are APL notebooks with ]help cells, as are tutorials where saved output helps. sidebar.yml lists pages explicitly; glyph pages are reached through glyphs.qmd and search.

nbs/apl.xml defines APL syntax highlighting for Quarto, and nbs/styles.css colours its operator and argument classes. editors/vim/ has matching vim rules.

magics.ipynb exports basedpl.notebooks; dyalog.ipynb exports basedpl.dyalog; j.ipynb exports basedpl.j. Edit those notebooks, then run nbdev-export. The other Python modules remain hand-written. Cargo owns the package version.

xml.ipynb, plot.ipynb and the index contain executed SVG examples. Update them with bapl-nb --save; do not run a full nbdev-docs build for routine edits.

bapl-nb nbs/                          # execute APL notebooks without changing files
bapl-nb nbs/getting-started.ipynb --save
bapl-nb nbs/index.ipynb --save        # update homepage examples
nbdev-readme                        # regenerate README.md
nbdev-test nbs/ --save                # execute Python notebooks
nbdev-test nbs/dyalog.ipynb --flags dyalog
nbdev-proc-nbs
cd _proc
quarto preview

Install development and documentation tools with pip install -e '.[dev]'. Rendering uses saved outputs. bapl-nb --save updates output only; execution counts are unchanged. The Rust documentation tests run APL examples in .qmd files. Set BASEDPL_PAGE to part of a page path to run only matching pages. pytest runs their Python examples. Use nbdev-test for Python notebook examples. Edit the homepage in nbs/index.ipynb, then regenerate the README.

Structure

Parser and evaluator

syntax.rs: byte-spanned lexer and structural parser, character, string and numeric literals, definition kind/full span, and structural guards and default arguments. The parser collects nodes and separators first. The enclosing delimiter then gives the separators their meaning. At the top level and in braces, a line break or ⋄ ends a statement. Parentheses group: a line break inside them is a space, and ; is an error. With ⋄ they write rows: (4 ⋄ 4 5) is [[4] [4 5]]. encloses decides from the text alone when parentheses make a scalar: round a literal, a strand, or glyphs separated by spaces. The evaluator builds the scalar, and a vector of functions for several glyphs. In brackets, spaces separate items, ; separates items that contain spaces, and ⋄ separates major cells. ; and ⋄ can't share one pair of brackets. Brackets round one item make a one-item vector, and a trailing ; is an error. Brackets are a record when any item is key:value: a : with only values before it. Literals separated only by spaces merge into one vector literal before runs form. A run is a sequence of nodes with no spaces between them. Assignment, pipes and guards also end runs. The run before ← stays flat, so assignment can take its target from the end of it: the arrays applied to each other there, as in v[2], and any function of a modified assignment. A parenthesised group counts as an array only when its expression gives one. A selection writes into the first array of the applied run at its end, as mt in (3↑mt[i])←…. parse returns complete syntax, incomplete input, or a structural error without evaluating expressions.
eval.rs: persistent session, explicit right-to-left category-reduction stack, and shared primitive/operator/dfn/train calls. Structural resolution consumes one item. A run resolves to one value by the same binder. One category table selects binding actions, priority and waiting states. An array next to an array applies: the binder turns v i into a call to ⌷ with ⊂i as its left argument. Application binds more loosely than a left argument and more tightly than a call, and it groups from the right. A run that ends in a dyadic operator binds with the item to its right before any other reduction, as if no space came between them. Everything before a function with no argument joins its train: Term::Train holds the raw functions and arrays, and Function::train builds them from the right. A function and the item before it make a fork, and an array item is a constant. An array directly before what is built binds with ⊸. A function left over at the start makes an Atop. So 32+1.8× is (32⊸+)⍤(1.8⊸×), and y 2× x is an error because 2⊸× is bound. Grammatical reduction returns application requests to the evaluator; it never calls functions itself. APL and Python share operand normalization and validation in Function::new. Reduced entities rebind against their right context when their category changes. Functions have immutable shared nodes; unfinished trains and bound left arguments exist only in the binder. No per-glyph arithmetic precedence. Output and the final result are separate from errors.
selection.rs: temporary labels for selective assignment. The binder marks their data flow and permits only selection functions; masks still read real user bindings. Option<SelectionKind> distinguishes ordinary evaluation, whole-item selection and element selection. An array with no nested items labels its items 1…n, and each label is its item's offset. Other arrays store a path for each label. Nested labels retain their storage and paths. Prototype labels preserve empty-cell structure.
error.rs: retained source text, byte spans, inspectable error kinds, and readable Unicode-width diagnostics with separate call-site context. Tabs use four-column stops; other control characters are escaped.
execution.rs: evaluation deadlines, a thread-safe interrupt flag and an optional poll hook that can request cancellation. check only counts down on most calls. Every 64th call reads the clock, tests the flag and the deadline, and calls the poll if 10 ms have passed since it last ran. Loops call check on every item. Context lends primitive code the current execution control and source span. Cancellation unwinds through ordinary errors but bypasses APL guards.

Values and arrays

array.rs: ordinary Value atoms and immutable shared arrays; checked construction, recursive prototypes, axis offsets, direct/mapped result frames, cell descriptors, padded cell assembly and row-based display. Gather copies items from source arrays into a new array. Its buffer takes the widest compact kind among the nonempty sources and widens for wider items. A mixed source, or an item that no compact kind holds, makes it mixed. Its walk reads items at the sums of one offset from each table, cycle repeats a source, and scatter replaces items at offsets. Assembly copies whole cells when their shapes agree. Gather::items and Gather::add build an array one item at a time, taking the kind of the first items. Layout::collect and the scalar element path build through them. Value::mixed builds mixed storage directly, as JSON objects do. Items borrows an array's items in their storage type. Its integers and nonnegative_integers read counts, positions and axes without a Value per item. Results print as source that reads back. Simple vectors and vectors of strings print as strands. A vector of two or more items, all nonempty vectors other than strings, prints as rows, (4 ⋄ 4 5). Other vectors print in brackets, including a one-item vector, as [5]. A rank-0 array holding a literal, a glyph or a vector of glyphs prints in parentheses, as (5) and (+ -). Other rank-0 arrays print as ⊂x. A matrix row with one item prints in its own brackets. Higher ranks use array notation, and one major cell ends with ⋄. An empty array other than ⍬ and "" prints as its shape reshaping its prototype, 0 3⍴0, and an empty record prints as ⍬:⍬. Inside brackets, an item with a space or a : between its runs gets parentheses. An array with named axes prints as its keyed shape reshaping the array without names, ["k":2]⍴3 12. An array of rank 2 or more with keys prints as a key list for each axis applied to the array without keys, ["a" 1;"x" "y"]:[1 2 ⋄ 3 4], where a position stands for a missing key. Strings print in double quotes and characters in single quotes. Arrays retain function handles and cache lexical dependencies without owning lexical frames or Python objects.
number.rs: Integer/Exact/Float/Complex values, normalization, checked arithmetic, promotion, comparison and structural conversion. i64 and BigRational share one exact domain. Checked integer overflow falls back to BigRational; integral rational results return to i64 when they fit. Representation is private. Complex values use num_complex::Complex64, normalizing exactly zero imaginary parts to Float. The real and int modules hold the real and integer cases of the scalar functions. Number and the compact kernels both use them, so the two paths give the same results.
scalar.rs: compact kernels for the scalar primitives. Each dispatch matches a primitive once and hands the loop its element kernel, so each loop is compiled for its function. Loops cover plain calls, folds, scans and inner products.
search.rs: search and classification of cells for ⍳ ∊ ~ ∪ ∩, monadic ∪ ≠ = and Key. Two cells match when array_match says they do. A search for up to 16 needles among compact integers or floats scans the items once for each needle. Any other search for one needle compares pairs directly. Searches with fewer than 8 haystack cells or 256 pairs, and classifications of fewer than 16 cells, compare pairs directly. Larger ones build an index. Integer vectors use a table indexed from their least value when their range is at most twice their length, and a map from value to first position otherwise. Exact data is hashed: exact numbers, characters and arrays of these without keys or functions. array_match confirms each hash candidate. Reals are hashed with tolerance. Each goes into a bucket of 256 neighbouring float_keys, and each of its matches lies in its own bucket or the next one on either side. A search indexes only the first copy of each value. Classification takes the earliest representative in those three buckets that matches, so tolerant matches don't chain. Integers join the reals only below 2^43, where tolerance never makes two different integers equal. Other data compares pairs directly. The rank-0 cells of an array are its items, so vectors aren't split into cells. sort_rows grades a table with an LSD radix sort on u64 keys, one byte at a time, and skips any byte that is the same in every row. ⍋ and ⍒ use it for floats, integers and characters in up to 16 columns. float_key orders floats as total_cmp does, and rows that compare equal keep their order of position. The radix sort and Key's grouping share one stable counting pass, scatter.
agreement.rs: positional broadcasting and key union through one output layout and two index maps. Scalar functions, Each, rank frames, native mathematical functions and explicit scalar axes use this path. Missing entries use prototype fill. Compact numeric kernels read the maps directly. Equal-shape and repeated-block mappings avoid coordinate calculations and expanded input copies. Contracted-axis maps require equal key sets and retain the first axis's order.
keyed.rs: axis-label construction, lookup and named updates. Each immutable Keys holds an optional name for each position, and a name-to-position hash. Present names are unique. align matches named positions by name and unnamed positions in order. Agreement, match, catenate and reordering all use it. A Keys with no names at all is dropped, so the axis is unkeyed. Fills and positions from an unkeyed part have no name. Records, JSON objects, CSV headers and system-function options need a name for every entry. Layout in array.rs owns dimensions and keys together. Its axis selection, concatenation and replacement operations describe structural results. Frames and cells carry layouts into assembly, which retains cell-axis labels shared by all result cells. Dot access is a binder rewrite to Pick in eval.rs. Plain assignment extends each missing name along a path, outermost first. An axis with no keys gains them: the new position has a key, and the existing positions have none. A new vector entry that a longer path descends into starts as an empty record. Other new entries take the prototype, which the assignment replaces. Extension keeps mixed storage mixed. The design is in meta/axiskey.md.
primitive.rs: one row per primitive and per operator, and array-level implementations. A primitive's row holds its glyph and names, and for each form its natural ranks, whether it is pervasive, whether it has its own meaning with ⍠, and a dyad's reduction identity. A monad that extends applies itself to each cell of a larger argument. An operator's row holds its glyph, names and the operand kinds it takes. Lexing, operand checks, axis dispatch, reduction identities and names all read these rows. Take, drop, reverse, rotate, transpose, replicate, expand, windows, partitions and selection by axes read their items through remap, which takes one offset table for each level of the result. Catenate and Key copy whole blocks. A selection writes through Targets. Flat offsets scatter into a copy of the array's items through Gather, without a Value per item for compact storage. Paths reach nested items. Replicate, products and assignment retain their own agreement rules. Allocation caps are separate from numeric-to-integer conversion.
number_theory.rs: segmented prime enumeration, Miller–Rabin primality and Brent/Pollard–rho factorisation; exact integer results and unit-cell assembly.
polynomial.rs: coefficient/factored/exponent-table forms, Horner evaluation, companion-matrix roots through faer, and analytic polynomial gradients/VJPs.

System functions and output

system.rs: case-insensitive •name table for constant arrays and native functions. System functions use ordinary function nodes and application. primitive.rs contains single-character primitives.
regex.rs: •r compiles a Rust regex into a keyed vector of native functions. Functions share an Arc<Regex> through system::Call; dot access, composition and Python use ordinary function values. Positions count Unicode characters.
distribution.rs: statrs-backed probability distributions, plus closed-form logistic. Constructors return keyed native functions sharing one distribution. Sampling checks shape/allocation limits and cancellation; numeric methods preserve layouts through pervasion. Discrete draws and quantiles are exact integers.
Generators: •rand seed returns a record of roll and deal sharing one Xoshiro256PlusPlus behind a mutex. The generator algorithm is fixed. Range sampling and the distribution samplers come from rand and statrs, and new releases of either can change the draws. distributions.apl pins one sequence to detect that. sample accepts that record on its left and finds the stream through its roll function. Roll and deal take the generator as a parameter. Plain ? passes the thread-local generator.
csv.rs: •csv import and •tocsv export, keyed options, per-column numeric inference and lossless compact storage. The csv crate handles records and quoting.
data.rs: shared keyed-option decoding, numeric fill, the import rule that CSV columns and JSON arrays share (imported: floats when any number other than a fill is a float and every number converts exactly, and each item's own kind otherwise), •vfi numeric-field parsing and UTF-8/binary file I/O (•nget/•nput). Binary vectors use exact integers in 0…255. Writes validate before opening and use exclusive creation unless overwrite is explicit. System functions take options on the left. Options::new reads a keyed vector or a plain-text shorthand. The monadic form uses the defaults.
json.rs: •json parses and •tojson serializes, with keyed objects, exact integers and explicit null fill. Arrays take the shared import rule, and objects keep mixed storage so each field keeps its kind. The tagged worker protocol remains in protocol.rs.
xml.rs: XML element trees, serialization and •svg. Elements are keyed vectors with tag, attrs and children entries. •xml checks names and escapes & < > ". •svg adds a _mime_ field holding its renderer. •mime calls a keyed vector's _mime_ function through ordinary dispatch, with implicit echo disabled and cancellation still active. Implicit display falls back to text when a renderer fails. Interrupts and timeouts still propagate. Keys of the form _name_ are hooks that the language calls.
plot.rs: •plot returns a spec record holding data, the left settings and a native _mime_ renderer. The renderer validates the spec through data::Options, reads every default itself and draws SVG with plotters (SVG backend only, no font files). The data's structure supplies x positions, category labels and named series. All axes use one linear Coord with ticks from Axis::ticks. Log scales transform values before drawing. A vector or matrix of spec records draws a figure. Repeated specs span cells, and share unions axis ranges within column or row groups. Legends are opt-in. 'end' writes names beside each series' last point. Corners use plotters' boxed legend. The renderer draws data labels and end labels itself. place moves each one vertically clear of earlier labels.
display.rs: boxed-array diagrams, function trees and session display settings. Returned values and ⍕ remain independent of these settings. bundle converts a MIME-keyed vector to the bundle sent to frontends. Implicit display builds that record only for a value with a _mime_ renderer. Other values emit their text directly. with_renderer adds a native _mime_ renderer to a record. svg wraps SVG text as a bundle. •svg and •plot share both.

Frontends

cli.rs / main.rs: native command and REPL. The REPL evaluates the parsed result once, not during completeness checking.
inspection.rs: non-executing name/glyph inspection and cursor lookup. build.rs embeds the glyph pages in nbs/glyphs/ for glyph and syntax help. System functions take their help from BUILTINS in system.rs, which names a glyph page for a few of them. ]help output carries both plain text and Markdown. Session methods own lexical name listing, classification, source and erasure.
kernel.rs: native kernmini adapter. ThreadWorker runs the interpreter on its own thread, so async transport and interrupts remain responsive. Implicit display becomes Jupyter results, explicit output becomes stdout, and completion reuses the REPL glyph matcher plus user/system names. Inspection and whole-cell name?/name?? use the shared metadata path. The wheel installs its kernelspec from wheel/data/share/jupyter/kernels/basedpl/. History uses kernmini's default; subshells are not advertised.
j.rs: the J engine and J kernel, built with the python feature. Engine loads libj at run time, registers output and input callbacks through JSM, and runs profile.ijs. J sets its recursion limit from the stack of the thread that starts an engine. An engine therefore runs J only on that thread. The kernel starts its engine on a worker thread with a 64 MB stack. JSetM returns J's error flag, which an earlier failure leaves set. set clears the flag first with an empty sentence. The J kernel shares kernel::serve and kernel::next_count with the APL kernel. The wheel ships no J kernelspec. install_j_kernel writes one through kernmini's install_kernelspec.
editor.rs: Rustyline terminal adapter using the shared naming table. Only typed backtick names auto-expand; Tab is an explicit completion request. Bracketed paste/history/navigation cancel automatic expansion. Rustyline owns terminal modes, editing, in-memory history and the final newline on Ctrl-D. No history file or input rewriting in the interpreter/frontends.
symbols.rs: every glyph's canonical name, monadic name, dyadic name and extra completion aliases. Syntax glyphs have their rows here, and primitives and operators take theirs from primitive.rs. Used by editor.rs and exposed as basedpl.symbols for Python exports and notebook JavaScript completion. Also owns shortcut formatting from keyboard.json; Python symbol rows append this display suffix. Add or change names in those rows, not in individual consumers.
python/basedpl/keyboard.json: shared Alt-chord map, embedded by symbols.rs and packaged for editor adapters. Keys are US characters after Shift, before Alt. Chords insert literals even in strings/comments.
protocol.rs: JSON-lines encoding over ordinary Rust values and sessions. No protocol types enter evaluation or arrays.
worker.rs: sequential evaluations with a separate stdin reader for control messages. python/basedpl/worker.py owns process lifetime, deadlines and hard-kill fallback. No APL execution occurs on the reader thread.
python.rs: optional PyO3 boundary. _Array and _Function hold native values; _Session holds the evaluator and runs each request on the calling thread. Requests and replies carry shared native values and retained diagnostics, never Python objects. python/basedpl/__init__.py provides arrays, conversions, the apl workspace and errors. functions.py constructs functions and exports word names from symbols.rs. The optional ipython.py adapter supplies Function help/source and completion inside apl strings. _cli.py forwards arguments to the Rust CLI.

Libraries and tests

lib/*.apl: Dyalog dfns adapted from April and the Dyalog dfns workspace. Reference cases load these shared definitions with •load; case-specific setup stays in each test. See lib/README.md for usage and provenance.
reference.rs: independent fixture decoding and structural comparison, shared by Rust tests, the worker's case request and Python's private _check_reference. Every reference case gets a fresh session.
tests/core.rs: storage, ownership, parser diagnostics, API behaviour and cross-call recovery checks. Use equiv_in! { &mut session; code => expected_apl, ... } for session workflows and fails_in(&mut session, kind, &[code, ...]) for errors in the same session. Expected expressions run in fresh sessions. Keep Rust constructors for foundational and representation checks. Self-contained language cases belong in tests/reference/core.apl. tests/cli.rs exercises the actual native process. Documentation examples share one session per APL code block. Unannotated code supplies setup; ⍝ introduces an independently evaluated expectation.
tests/scaling.rs: how time and memory grow from 1,000 to 8,000 items, counted by a global allocator. The loops of updates and appends stay ignored until in-place updates exist.
python/basedpl/reference.py: Source importers, Dyalog expectation capture, scan/review/activation, and Corpus inventory access. bqn(src) runs BQN through links/BQN/bqn.js with Node and returns what it prints, for checking BQN comparisons. Corpus searches and patches tests/reference/inventory/*.jsonl from a kernel. Default views omit large expectations; request fields explicitly. scripts/reference.py is a thin scan/review/activation CLI. python/basedpl/apltests.py reads/writes .apl records and appends reviewed inventory cases. Its check_file and check_page run a reference file or a page through the installed extension, as the Rust tests do, without a cargo build.
tests/reference.rs and tests/reference/*.apl: bAsedPL semantic cases plus acceptance tests from ngn, April, APLcart and Dyalog documentation. .apl files determine active coverage. Use ⍝⍝ section headings and short case descriptions for non-obvious checks. Optional ⍝ ⎕: lines assert explicit output. core.apl retains exact Rust array equality, including numeric domains. Implementation gaps belong only in pending JSONL records, never passing error tests. See tests/reference/README.md for format and workflow.

Design

The language should keep becoming more concise to write, simpler to read and more consistent to understand. A design rule is a plan on trial, not a law. When applying one makes the language worse on any of these measures, stop and discuss the rule before going further. The signs include a test or example that must become longer or harder to read to keep its meaning, something that could be expressed before and no longer can, and a special case added so the rule still fits.

Values

Value is a number, character, function or shared ArrayData. Array storage is Integer(Vec<i64>), Float(Vec<f64>), Complex(Vec<Complex64>), Character(Vec<char>) or Mixed(Vec<Value>). A Value is 24 bytes. An exact rational is boxed, and a function is one pointer to a node that holds its depth, environment and late binding. An ArrayData is 48 bytes, because a Layout keeps its keys and names in one optional box. Atoms, scalars and singleton vectors remain distinct. Function equality is identity, and functions have no ordering. Values and shared function nodes are Send + Sync.

as_integers(), as_floats() and as_complex() expose borrowed slices. at() and elements() yield owned values without materializing the array. Shape, numeric-domain summary and lexical dependencies are cached in immutable Arc storage. Shape and tally return exact integers directly from dimensions.

Numeric storage has prototype 0, and character storage has a blank. Empty construction requires a prototype. Mixed storage holds its prototype. An empty mixed array stores it, and a nonempty one fills its first item on the first request and keeps the result. Builders take a Prototype, either a value or a function, and call the function only for an empty result. Prototype filling memoizes shared nested arrays and retains function handles.

Numbers

Numbers in compact storage share one kind. An operation's result widens from integer to float to complex. An array built from a float and exact integers therefore holds floats, and writing a float into integer storage converts the whole array. Items written in a literal list or in brackets keep each number's exactness. Among them, only floats widen, to complex, and exact numbers beside approximate ones stay mixed.

Mixed storage keeps each item's kind. It holds rationals, which have no compact form, and any array that also holds characters, nested arrays or functions. Rearranging, combining or writing into mixed storage keeps it mixed, so nothing scans for compaction. Computations build fresh storage from their results, so 1×x converts. Equality compares items, whatever storage holds them. Boxed display marks mixed storage with +.

CSV columns and JSON arrays become floats when a number other than a missing-value fill is a finite float, and every number converts exactly. Otherwise each number keeps its kind. JSON objects stay mixed.

Infinities never make exact numbers approximate. ⌊ and ⌈ return an exact argument unchanged beside ∞ or ¯∞, so 3ₓ⌊∞ is 3ₓ. When an array chooses its storage, ∞ and ¯∞ join exact integers only in mixed storage, so (⍳3ₓ),∞ is mixed. Kind::Infinity gives this rule. ArrayData.infinite records float storage whose items are all infinite. from_storage finds it once, when the array is built. Gather never rescans a source. Arithmetic with an infinity still gives floats. A missing-value fill of ∞ leaves an imported integer column exact, in mixed storage.

Monadic ⌊, ⌈ and × give exact integers, as comparisons do. A float beyond i64 floors to an exact big integer. An infinity stays a float. The float kernels in scalar.rs give integer storage through Monad::integer. They fall back to the element path for an infinity or a value beyond i64.

A float argument that must be an integer, such as an index, a count or a character offset, may be within comparison tolerance of one. near_integer in number.rs is the one test. ⌊, ⌈, real::integer, real::nonnegative_integer and Number::big_integer all use it.

Numeric comparisons use fixed relative tolerance 1e-14 when approximate, exact rational comparison otherwise. Each tolerance-sensitive operation must ship with independent inside/outside-tolerance cases: comparisons, membership/index-of, match, unique/grouping, and floor/ceiling. Reuse the rounded pair 0.3 and 0.1+0.2 across applicable operations, with an outside-tolerance control, zero/negative boundaries, and exact/mixed counterparts. Keep structural Rust assertions exact; do not use the language's comparison as the test oracle. The concrete acceptance matrix is in meta/PRD.md §9.4.1. The reference corpus deliberately retains unsupported cases; enable them as their requirements are met.

Numeric semantics are Rust-owned. Explicit Python conversions copy exact integers through PyO3 and non-real values through PyComplex::from_doubles. Fraction components transfer as native integers. NumPy Boolean, integer and real arrays cross the boundary as one int64 or float64 buffer copy. Python casts them first. _Array.numeric fills compact storage from the buffer. _Array.buffer copies compact storage out for np.frombuffer. uint64 values above the int64 range raise ValueError. Complex, string and object arrays convert per element. Python lists take the import rule that CSV columns and JSON arrays share, and dicts keep mixed storage, as JSON objects do. No Python numeric objects enter the core. JSON remains a separate process boundary: exact integers use arbitrary-sized JSON integers with serde_json's arbitrary_precision feature. Fractions retain tagged decimal-string components, complex values a tagged numeric pair. Reference-interpreter cases compare equal numeric values across exact/float domains because Dyalog has no corresponding explicit exact domain. Rust core tests assert the exact representation, prototypes and compact buffers separately.

Complex arithmetic extends the existing scalar/array/operator dispatch, not a second evaluator. The lexer shares one real-component scanner between ordinary and ajb literals. Equality uses magnitude-based tolerance separately from real ordering; counts still require exact integrality and zero imaginary part. Approximate prototypes/identities normalize to Float. Complex division scales its denominator and direction scales its input to avoid avoidable squared-magnitude overflow/underflow. Powers, logs, circle functions and factorial/binomial extend this numeric layer. Complex components/results remain finite. Complex storage can also hold a float that widened into it, including ∞. Such an item reads back as the float. Real Float values admit ±infinity but never NaN. Comparisons handle infinity before tolerance or exact-to-float conversion. The compact float path explicitly checks zero divisors. JSON uses signed infinity tags; Python uses floating infinities. Dyalog 20.0.53963.0 executions supply structured documentation-example expectations, except labelled bAsedPL policy differences.

The numeric policy is documented in README.md. Primitives raise LIMIT for any generated array of more than MAX_GENERATED_ELEMENTS (one million) elements. serde_json is a frontend dependency; PyO3 remains optional. JSON requests decode directly to String, one per line, with no object envelope or custom escaping. Responses remain structured objects. Keep JSON and Python conversions separate: they implement different external contracts.

Performance

Hash maps use foldhash, which is seeded randomly for each process. Scalar primitives on compact storage run through scalar.rs in plain calls, unseeded folds, scans and inner products. One match per primitive picks the element kernel outside the loop. A kernel that cannot give an element result sends the whole call down the Number path, so errors and exact or complex results are unchanged. ○ and monadic π read integer storage as floats and use the float kernels, as Number converts an integer before it computes. Other numeric folds bypass scalar-array allocation and interpreter calls. Float +.× is a faer matrix product. Homogeneous float sum/product reductions use Rust 1.98 algebraic operations, including axis reductions; grouping and bitwise reproducibility are not promised. All scans use successive left accumulation. Seeded and unseeded forms share lane traversal. Reduction starts from its whole seed on the right. Scan starts from its seed on the left, and a unit seed stands for its content, as in BQN. Each axis lane uses the same seed. Rank supplies separate seeds per cell. Generic reductions remain right-associated at every rank. Generic functions keep their call order and side effects. Exact arithmetic is unchanged. Finite-result checks remain at construction boundaries; division retains 0÷0=1. No fast-math flags, custom SIMD intrinsics, CPU-specific wheel flags, compensated summation or strict/fast modes are used.

Brackets, indexing and axes

Bracket lists are structural syntax. Each item contributes one element, including array and function values. Items evaluate left to right, as statements do. [a b]← destructures by the assignment rules below. First ↑, Pick and complete atomic ⌷ indices return stored values through the shared call-result path, including functions. Array indices supply result frames; partial coordinates retain trailing cell axes. Empty coordinates preserve the argument. Array consumers and cell assembly store function results as elements. Python can construct Array([plus, times]) and retrieve callables with first, pick, .py or .np. •tojson and the worker protocol omit keyed-vector entries that hold functions. They reject any other function.

Brackets round one item make a one-item vector. ⊂ encloses any value, and parentheses enclose a literal, a strand or glyphs separated by spaces. A list of counts as Power's operand gives the result for each count. A count of ∞ runs until the state matches the previous one, through the same ≡ call as ⍣≡. Power keeps every state of the until form when its operand is a one-item vector holding a function, as in f⍣[≡]. Python's .history(p) builds that operand for a predicate, and the counts (×p)×⍳1+|p for a count.

Session::members rewrites dot access before binding. After a value, x.name becomes "name"⊃x, and x.(I) or x.[I] becomes (I)⌷x or [I]⌷x. The rewritten node is a group. Assignment, adding keys and chaining reuse the ⊃ and ⌷ paths. Between functions, . stays inner product.

⍠ builds an axis node from its operand, a list of numbers or names. Python's f[A] builds the same node. A primitive with its own axis meaning handles the node in call_axes, and a fold takes the axis directly. Every other function goes through cell_axes. It moves the selected axes of each argument last, applies rank, then moves the result axes back. selection, behind ⌷ and Python indexing, reads an atomic ∞ part as a whole axis and ¯∞ as a whole axis reversed. It counts negative positions from the end. When the first part is the only one and its elements are arrays, it does choose indexing. Pick, choose, reach and Agenda share the same position rule.

Vector-axis reduction is (f/⍤,)⍠axes Y through the general axis rule, so it assembles like any function along axes. The selected axes move last in the order given, so axis order decides the ravel. A seed binds to the reduction before composition with Ravel, so the general rule doesn't split it along the axes. Single-axis reductions retain their direct traversal and numeric kernels. An axis node on a fold passes its axis to the fold.

Layout also carries optional axis names. Agreement pairs equal names first, then remaining axes positionally, leaving differently named axes separate. Its existing index maps handle the resulting permutations and broadcasts. Layout construction retains names until result assembly drops all occurrences of collisions. Public name attachment rejects duplicates. Axis selections resolve strings against argument layouts before numeric-axis dispatch. ⍴ returns the shape keyed by the axis names, with no key for an unnamed axis. Reshape takes the result names from the keys of its left argument, and keeps position keys only when the shape is unchanged. ArrayData shares its element buffer through a separate Arc<Storage>; layout changes copy metadata only.

Operators

Inverse dispatch carries an optional fixed argument: the pair's Boolean is true for a fixed left argument and false for a fixed right argument. Commute switches sides. Before, After and binding propagate that information. Inverse scans apply the operand's dyadic inverse to adjacent accumulators, using the seed for the first pair. They share forward scan's axis/seed validation. General forks, one-argument Before and arbitrary dfns require rules beyond this propagation and remain unsupported.

Agenda selector◶cases stores its operands in an ordinary composed-function node. Construction validates the nonempty function vector and constant selectors. Function selectors run once per call. Unit indices use the shared position conversion, counting from 0. The selected branch receives the original arguments through shared function dispatch.

Each and Outer call a primitive whose form is pervasive once, on the whole arguments. Its kernels then do the work. Rank calls a pervasive dyad the same way when both cell ranks are 0. Rank calls a monad directly when the monad extends to any rank and the cell rank is at least the monad's natural rank. Rank keeps its general path for atom arguments, because Rank gives them a unit result. Outer gives each argument its own axes, and its right argument maps as a repeated block, Mapping::Tiled. Empty Each with any other operand calls it once, replacing only empty arguments with their prototypes. Function::call_prototype scopes prototype mode around the ordinary call path. Compositions, dfns, dops and helpers inherit the mode. Pick uses structural prototype selection throughout that call. Other errors and explicit output remain observable. The caller's mode is restored on success and error.

Scope, dfns and assignment

Lexical frames are a stack with non-owning parent indices, separate from dynamic call/handler state. Nested functions see live lexical bindings, not snapshots. Plain dfn name assignment is local. Modified and selective array assignment update the nearest existing lexical binding. Each update retains its resolved owner and original array across modifier calls. The write does not look up the name again. Arrays may contain functions, including active local captures. Returns and outer updates reject functions or arrays whose lexical dependencies would not survive. This includes empty prototypes. Tail calls retain lexical dependencies in argument arrays as well as the called function. Dfns can return functions directly. Public export walks the shared array/function graph once and rejects active frame references. Operator derivation retains operand values without running the body or creating an invocation frame. Frames pop on success and error; no collector is needed. Execute uses the same lexical capture rule for newly created definitions. Python calls retained functions through its one workspace, apl. Rust callers supply the session when calling a retained function. Recheck the lifetime argument before adding namespaces, nonlocal function assignment or escaping lexical closures. Dyalog reference runs and the scope boundary are recorded in meta/PRD.md gate L.

Dfn return selection follows statement syntax, not display shyness. A final assignment returns its value silently; a non-assignment call returns immediately even when its result is silent. Empty bodies and exhausted guards return no value. Default ⍺ assignment skips its RHS when supplied and does not itself return a value. Each invocation shadows its caller's ⍺.

The binder returns either a value or a tail application at eligible return positions. Defined calls loop over tail applications and discard frames above the callee's highest lexical dependency. This dependency is cached with immutable function nodes, including function operands and train arms. Tests run 10,000 tail calls with one frame, or two when an outer lexical binding is retained. Installed handlers disable tail reuse. Non-tail evaluation and retained lexical frames have a 1,024-level limit. Tail call diagnostics retain the final tail site, not an unbounded history.

Catch-all and numbered guards checkpoint the installing frame's local binding map after the condition executes. Restoring it removes later introduced locals and restores previous local values, including modified assignments. Outer/global writes and output are not rolled back. This deliberately omits Dyalog's distinction between rebinding a local and modifying its existing binding. Assignments within the condition survive rollback. Guard handlers are popped before execution and unwind dynamically through ordinary calls. Cancellation and unsupported-feature errors are not caught. Ordinary and error guards may have an empty result; selecting one returns no value.

Flat binding/operator derivation and assignment chains use explicit vectors. A common evaluation-depth budget covers groups and all function representations; function construction separately limits graph depth, protecting recursive application and destruction. Neither limit is an execution-time sandbox. stacker extends the stack onto the heap at each call and binding level, so evaluation needs no large thread stack.

An assignment arrow drains its right-hand binding stack, assigns to the structural target suffix and resumes binding the prefix. Its RHS is never evaluated twice. Target recognition uses runtime categories at top level and dfn-local name rules inside definitions. Inside a dfn, a name directly after an array is classified by its value, as at top level, so a(f)←3 and (a)f←3 are modified assignments. Statement return selection distinguishes an assignment from an expression containing one. Modified destructuring calls its modifier left to right. Ordinary destructuring assigns right to left.

Evaluation API

eval_with and call_with install per-evaluation EvalOptions. echo defaults to true; false suppresses implicit display at top level and inside execute, without suppressing explicit output or display commands. An optional output sink receives explicit/display events as they occur instead of collecting Evaluation.output; ordinary callers retain the capture interface. Clone the InterruptHandle to another thread or supply a timeout. Checks run at binding/call boundaries and inside long interpreted/primitive loops. Tight bounded float kernels keep their existing slice paths; native-library calls and individual BigInt operations are not preempted. Python's apl uses cooperative cancellation only. The separate process Worker.eval allows a grace period before killing an unresponsive process. Killing loses the session; requests are never replayed.

Session::call resolves a function expression and invokes it with one or two existing arrays through the ordinary APL call path. call_function_with accepts a retained node. Neither binds temporary argument names. Evaluation.function exposes an unshy exportable function result; set_function checks and binds a function. PyO3 _Session.request accepts code or a native function, native arguments/bindings, timeout and echo. The process worker separately decodes bindings and args in protocol.rs; exact JSON integers must not pass through f64. The process protocol does not export native function handles.

Evaluation.output contains ordered Output { kind, data } events. data is a MIME bundle with text/plain; OutputSink receives the same events. Python Result.events exposes them and .output extracts text. _repr_mimebundle_() calls •mime through apl. The notebook runner saves display events as display_data and explicit output as streams. JSON value transfer carries array contents/axes. The worker omits keyed-vector entries that hold functions, such as _mime_ renderers. It rejects any other function. Rich output bundles cross that boundary separately.

Python

Python has one workspace, apl: a _Workspace created at import around the Rust evaluator. fn is apl.fn. Module-level builtin calls and the notebook magics use apl. IPython completion accepts apl or a method bound to it. Plain calls and 'explicit' calls request echo=False, and 'repl' calls request echo=True. Plain calls print explicit output and return an Array or an unshy Function; capturing calls return Result without printing. The capture mode is positional-only because keywords bind APL names. Errors follow the same print/capture rule. Keywords bind APL names, including native functions; apl.timeout sets a per-evaluation deadline. ]clear removes every name and restores the display settings the frontend started with. ]box reset restores those display settings. Array retains the native value losslessly. .py and .np make Pythonic copies; NumPy is optional and lazy. Python integers transfer through PyO3's BigInt support without decimal-string conversion. Array indexing uses the shared Rust selector behind ⌷, counting from 0. : takes a whole axis, and negative positions count from the end. Arithmetic and function construction use native nodes, never generated APL source.

The module and apl share builtin attribute lookup in functions.py. The registry comes from Rust glyph metadata and system-function names; module functions are constructed lazily through __getattr__. Glyphs, canonical names and aliases are ambivalent; operation names select valence and curry. Operation names win overlaps, and glyph/operation names win unbulleted system-name collisions. Identifier spellings normalize hyphens, Python keywords and Unicode NFKC. __all__ and __dir__ expose the registry without creating every function. .fn() functions are ambivalent. Function.__call__ passes keyword arguments as a keyed left argument, and its positional arguments then form a vector as the right argument (⍬ when there are none). Calls without keyword arguments keep f(⍵) and f(⍺, ⍵).

.fn() parses once and retains a late-bound expression. Function nodes cache whether they contain late-bound operands. At a call boundary, resolution rebuilds affected nodes with the calling session's current bindings; unaffected nodes remain shared. A memo preserves shared function graphs. Reduction identities, inverse recognition and primitive fast paths then see ordinary functions. Cyclic name resolution errors at the depth limit. The binder still resolves ordinary APL names at execution time. Python word names select direct-call valence and currying; operators use the underlying APL function.

Python operator properties build native nodes when operands are complete. _Pending retains unfinished constructions and fills missing operands innermost-first; f.power.each(n) builds (f⍣n)¨. No source generation or Python evaluation callback enters the interpreter.

printing.py renders native function parts as Python names and combinators. The PyO3 adapter exposes each node's immediate operands as shared handles. The printer propagates valence and groups Python expressions by precedence. Dfns, defined operators and late-bound source remain fn(...); printing performs no evaluation or name resolution.

The private _core._Session holds the evaluator in a mutex. Each request waits for the lock without holding the GIL, then evaluates on the calling thread with the GIL released. A second mutex holds the active request's interrupt handle, so interrupt() works from another thread. Each request owns a fresh cancellation flag. The request's poll hook takes the GIL at most every 10 ms to check Python signals. Ctrl-C sets the cancellation flag and raises KeyboardInterrupt with the captured output. Evaluation touches no Python objects. Arrays can be dropped on any thread. No unsafe Send implementation is used.

Workspace and builds

Prefer broad dependency ranges with a required lower bound, such as >=0.24.4, <1, rather than exact pins or Cargo's minor-constrained 0.x caret ranges. Resolve API changes when they arise; do not add compatibility layers preemptively.

The canonical version lives in Cargo.toml; Python uses dynamic = ["version"]. The crate produces an rlib, native executable, and optional basedpl._core extension. python enables PyO3; extension-module also enables PyO3's extension linking mode. Default Cargo builds have no Python dependency.

Rust 1.98 or later is required by the algebraic float methods. CI tests with stable Rust. Keep fastws-generated Cargo patches and .git/fastws-cargo-key under fastws control. Preserve the pyproject source/cache keys; do not commit the workspace-generated Cargo.lock or manually replace workspace configuration. meta/ is ignored planning material, never committed.

uv builds and maturin develop --release use the incremental release profile: LTO off, 16 codegen runs, this package incremental. Distributed wheels use dist: full LTO, one codegen run, incremental off, stripped output.

CI tests the native core/CLI/JSON process before installing the extension and running Python tests. Python tests cover boundary behavior and the real installed command, not a duplicate Rust semantic suite. A dist-profile CPython 3.13 wheel has also passed these checks in a clean temporary environment outside the workspace, with the source checkout and Rust absent from its import/command paths. The existing wheel/sdist and tagged publication flow remains unchanged.

tests/test_repl.py uses a real pseudo-terminal for symbol entry, ambiguity, bracketed paste, Ctrl-C recovery and Ctrl-D's final newline; piped process tests cannot exercise these paths. Rust editor tests cover matching and quoted/comment context. The editor's Enter callback records just the accepted replacement because Rustyline cannot combine replacement and submission in one command; the adapter applies it before history/evaluation and shows the glyph in its submission message. This does not reprocess an entire source string.

Do not repeat isolated wheel installs for routine feature changes. maturin develop plus the normal tests is the development default; reserve clean-install checks for packaging-sensitive changes or release preparation, using a small relevant subset of existing tests.

Release

Development tests and artifact checks precede release approval. Once Jeremy approves a release, confirm the clean tree and Cargo version, then use ship-release with no flags. For this maturin project fastship tags/pushes the current version, leaves publication to CI, then bumps Cargo and refreshes the editable installation. There is no changelog step. First publication requires Jeremy's PyPI trusted-publisher setup. Never commit, push, tag, or publish without approval.

FilesExpand file tree

DEV.md

Latest commit

History