|
| 1 | +# Extensible Waveform Library |
| 2 | +To generate the signal, streamer computes output values for a grid of sample clock time points. |
| 3 | +For each instruction, values are computed by evaluating the contained mathematical function |
| 4 | +(polynomial, sine, exponent, and so on) with specified parameters. |
| 5 | +We interchangeably call them "waveform functions", "waveforms", or simply "functions". |
| 6 | + |
| 7 | +The main challenge is that waveform library has to be written in Rust |
| 8 | +(since sample computation must be fast), but at the same time it has to be _extensible_. |
| 9 | + |
| 10 | +As project is evolving, more functions will be added. Moreover, users may need to introduce custom |
| 11 | +functions for their specific applications. There should be a clean and sustainable way of adding |
| 12 | +new waveforms without making any changes to the codebase core. |
| 13 | + |
| 14 | +This section describes the implemented library mechanism. |
| 15 | + |
| 16 | +## Waveforms as trait objects |
| 17 | +**Problem 1** - _treat diverse waveforms in a uniform way_. |
| 18 | + |
| 19 | +We represent each waveform by a struct containing corresponding function parameters like this: |
| 20 | +```Rust |
| 21 | +/// Linear function: |
| 22 | +/// `LinFn(t) = slope*t + offs` |
| 23 | +pub struct LinFn { |
| 24 | + slope: f64, |
| 25 | + offs: f64 |
| 26 | +} |
| 27 | +``` |
| 28 | +So technically, different waveforms have completely different types. But we need a way to threat them uniformly - |
| 29 | +to store them in the same collection (edit cache and compile cache), and to process them during compilation and streaming. |
| 30 | + |
| 31 | +A simple approach is to [use enums](https://doc.rust-lang.org/book/ch08-01-vectors.html#using-an-enum-to-store-multiple-types). |
| 32 | +However, every new waveform would require adding a new enum variant and, consequently, making explicit edits |
| 33 | +throughout the back-end core to add a corresponding match arm. |
| 34 | + |
| 35 | +A more natural and scalable approach is treating waveforms as [trait objects](https://doc.rust-lang.org/book/ch18-02-trait-objects.html) |
| 36 | +based on the common behavior we rely on - computing signal value array for a given time array. |
| 37 | +We formalize this behavior as the following trait: |
| 38 | +```Rust |
| 39 | +pub trait Calc<T> { |
| 40 | + fn calc(&self, t_arr: &[f64], res_arr: &mut [T]); |
| 41 | +} |
| 42 | +``` |
| 43 | +where `T` is the output sample type (`f64` for AO, `bool` for DO). |
| 44 | +Each waveform must implement this trait, essentially providing the mathematical expression for computing it: |
| 45 | +```Rust |
| 46 | +impl Calc<f64> for LinFn { |
| 47 | + fn calc(&self, t_arr: &[f64], res_arr: &mut[f64]) { |
| 48 | + for (res, &t) in res_arr.iter_mut().zip(t_arr.iter()) { |
| 49 | + *res = self.slope * t + self.offs |
| 50 | + } |
| 51 | + } |
| 52 | +} |
| 53 | +``` |
| 54 | +Practically, more assumptions are needed, so the actual trait in use is called |
| 55 | +`FnTraitSet<T>` and is derived from `Calc<T>` by requiring additional technical traits |
| 56 | +(see `nistreamer-base/src/fn_lib_base.rs` for details). |
| 57 | + |
| 58 | +That way all functions are treated uniformly as trait objects: |
| 59 | +```Rust |
| 60 | +Box<dyn FnTraitSet<T>> |
| 61 | +``` |
| 62 | +and expanding function library does not require any additional changes anywhere. |
| 63 | + |
| 64 | +## Waveform library classes |
| 65 | +**Problem 2** - _avoid explicit relay code at the Rust-Python interface_. |
| 66 | + |
| 67 | +Waveforms are defined in the Rust back-end, while user script makes pulse selections through the Python front-end - |
| 68 | +there has to be a relay mechanism between the two. And it has to allow for extensibility as well - |
| 69 | +no manual edits should be required as new waveforms are added to the library. |
| 70 | +Implemented mechanism is described below. |
| 71 | + |
| 72 | +Waveforms are bundled into two libraries - the built-in `std_fn_lib` ("standard function library") |
| 73 | +and an optional user-editable `usr_fn_lib`. Both are implemented in the same way, we focus on |
| 74 | +`std_fn_lib` in this section, details of `usr_fn_lib` are discussed later. |
| 75 | + |
| 76 | +Each library is defined as an empty Rust struct and is exposed as a Python class |
| 77 | +separately from the main `StreamerWrap` (see {ref}`this schematic <rust-py-interface>`): |
| 78 | +```Rust |
| 79 | +#[pyclass] |
| 80 | +pub struct StdFnLib {} |
| 81 | +``` |
| 82 | + |
| 83 | +For each waveform, `StdFnLib` implements and exposes a dedicated Python method named after this waveform: |
| 84 | +```Rust |
| 85 | +#[pymethods] |
| 86 | +impl StdFnLib { |
| 87 | + fn LinFn(&self, slope: f64, offs: f64) -> PyResult<FnBoxF64> { |
| 88 | + let fn_inst = LinFn::new(slope, offs); |
| 89 | + let fn_box = FnBoxF64 { inner: Box::new(fn_inst) }; |
| 90 | + Ok(fn_box) |
| 91 | + } |
| 92 | +} |
| 93 | +``` |
| 94 | +This method first creates a `LinFn` instance and then returns it as `Box<dyn FnTraitSet<f64>>` _into Python_. |
| 95 | +Here the returned type `FnBoxF64` is a thin wrapper: |
| 96 | +```Rust |
| 97 | +#[pyclass] |
| 98 | +pub struct FnBoxF64 { |
| 99 | + pub inner: Box<dyn FnTraitSet<f64>> |
| 100 | +} |
| 101 | +``` |
| 102 | +which is necessary because `#[pyclass]` cannot be attached directly to `Box<dyn FnTraitSet<f64>>` since PyO3 macros |
| 103 | +don't support type parameters. |
| 104 | + |
| 105 | +The waveform instance is then passed back into Rust to form a new instruction on a specific channel: |
| 106 | +```Rust |
| 107 | +#[pymethods] |
| 108 | +impl AOChan { |
| 109 | + pub fn add_instr(&mut self, func: FnBoxF64, t: f64, dur: f64) |
| 110 | +} |
| 111 | +``` |
| 112 | +(simplified for clarity, the actual methods are `StreamerWrap::ao/do_chan_add_instr` in `src/flat_wrap.rs`). |
| 113 | +A schematic of this process is shown in {ref}`this figure <rust-py-interface>`. |
| 114 | + |
| 115 | +For convenience, waveform library classes are instantiated in `py_api/nistreamer/__init__.py`: |
| 116 | +```Python |
| 117 | +from ._nistreamer import StdFnLib as _StdFnLib |
| 118 | +std_fn_lib = _StdFnLib() |
| 119 | +``` |
| 120 | + |
| 121 | +From the user script side, the whole process of creating and adding a waveform looks like this: |
| 122 | +```Python |
| 123 | +from nistreamer import std_fn_lib |
| 124 | + |
| 125 | +# ... streamer setup ... |
| 126 | + |
| 127 | +ao_chan.add_instr( |
| 128 | + func=std_fn_lib.LinFn(slope=1.0, offs=2.0), # <-- waveform instance is passed across Python here |
| 129 | + t=0.0, dur=1e-3 |
| 130 | +) |
| 131 | +``` |
| 132 | + |
| 133 | +Exposing waveform libraries as separate Python classes allows for the following key properties: |
| 134 | +* Streamer core only exposes a single entry point for all waveforms - the `add_instr` methods of `StreamerWrap`; |
| 135 | +* Waveform library `pyclass`es are exposed publicly (in contrast to `StreamerWrap`, which is hidden behind PyAPI), |
| 136 | + allowing direct use of the named Python methods that `PyO3` macros automatically generate for every waveform. |
| 137 | + |
| 138 | +As a result, there is no explicit relay code, so adding new waveforms to libraries does not require making |
| 139 | +any edits in the back-end core or PyAPI, as required for extensibility. |
| 140 | + |
| 141 | +## User-editable library |
| 142 | +**Problem 3** - _a place for custom waveforms_. |
| 143 | + |
| 144 | +It is always possible that a user needs a highly-specialized waveform which is not available |
| 145 | +in the built-in library. The user-editable `usr_fn_lib` is introduced for such cases |
| 146 | +to allow for adding custom waveforms locally. |
| 147 | + |
| 148 | +This library is contained in a separate crate `nistreamer-usrlib` which is an optional dependency |
| 149 | +behind `usrlib` feature: |
| 150 | +- by default, back-end compiles without it; |
| 151 | +- while compiling with `--features usrlib` flag includes it. |
| 152 | + |
| 153 | +(see {doc}`compilation instructions </getting_started/installation>` for details). |
| 154 | + |
| 155 | +Moreover, `nistreamer-usrlib` is split out into a separate `git`/`GitHub` [repository](https://github.com/NIStreamer/nistreamer-usrlib) |
| 156 | +to maximally decouple it from the rest of the codebase and simplify version control task for the users. |
| 157 | +Users are advised to create a fork of this GitHub repository. That way any necessary updates |
| 158 | +can be merged from the up-stream in a streamlined way. |
| 159 | + |
| 160 | +If compiled with `usrlib` feature, `usr_fn_lib` can be imported and used alongside with `std_fn_lib`: |
| 161 | +```Python |
| 162 | +from nistreamer import std_fn_lib, usr_fn_lib |
| 163 | + |
| 164 | +# ... streamer setup ... |
| 165 | + |
| 166 | +ao_chan.add_instr( |
| 167 | + func=std_fn_lib.LinFn(slope=1.0, offs=2.0), |
| 168 | + t=0.0, dur=1e-3 |
| 169 | +) |
| 170 | + |
| 171 | +ao_chan.add_instr( |
| 172 | + func=usr_fn_lib.GaussSine(t0=3e-3, sigma=0.5e-3, scale=1.5, freq=10e3), |
| 173 | + t=2e-3, dur=2e-3 |
| 174 | +) |
| 175 | +``` |
| 176 | + |
| 177 | +### Procedural macros |
| 178 | +Apart from being located in a separate crate and a separate repository, `usr_fn_lib` is implemented |
| 179 | +in the same way as `std_fn_lib`. The example below shows what a minimal version of `nistreamer-usrlib/src/lib.rs` |
| 180 | +would look like with just a single custom waveform `MyLinFn`: |
| 181 | +```Rust |
| 182 | +// Imports: |
| 183 | +use pyo3::prelude::*; |
| 184 | +use nistreamer_base::fn_lib_base::{Calc, FnBoxF64, FnBoxBool}; |
| 185 | + |
| 186 | +// `UsrFnLib` setup: |
| 187 | +#[pyclass] |
| 188 | +pub struct UsrFnLib {} |
| 189 | +#[pymethods] |
| 190 | +impl UsrFnLib { |
| 191 | + #[new] |
| 192 | + pub fn new() -> Self { |
| 193 | + Self {} |
| 194 | + } |
| 195 | +} |
| 196 | + |
| 197 | +// Custom waveform: |
| 198 | +#[derive(Clone, Debug)] |
| 199 | +pub struct MyLinFn { |
| 200 | + slope: f64, |
| 201 | + offs: f64, |
| 202 | +} |
| 203 | +impl MyLinFn { |
| 204 | + pub fn new(slope: f64, offs: f64) -> Self { |
| 205 | + Self { slope, offs } |
| 206 | + } |
| 207 | +} |
| 208 | +#[pymethods] |
| 209 | +impl UsrFnLib { |
| 210 | + /// Linear function: |
| 211 | + /// `MyLinFn(t) = slope*t + offs` |
| 212 | + #[allow(non_snake_case)] |
| 213 | + fn MyLinFn(&self, slope: f64, offs: f64) -> PyResult<FnBoxF64> { |
| 214 | + let fn_inst = MyLinFn::new(slope, offs); |
| 215 | + let fn_box = FnBoxF64 { inner: Box::new(fn_inst) }; |
| 216 | + Ok(fn_box) |
| 217 | + } |
| 218 | +} |
| 219 | +impl Calc<f64> for MyLinFn { /* implementation here */ } |
| 220 | +``` |
| 221 | +Then PyO3 macros generate `UsrFnLib` Python class with a method `MyLinFn`, |
| 222 | +with the doc-comment automatically converted into Python docstring: |
| 223 | +``` |
| 224 | +Signature: usr_fn_lib.MyLinFn(slope, offs) |
| 225 | +Docstring: |
| 226 | +Linear function: |
| 227 | + `MyLinFn(t) = slope*t + offs` |
| 228 | +Type: builtin_function_or_method |
| 229 | +``` |
| 230 | + |
| 231 | +This version would be completely acceptable for the built-in `std_fn_lib`, |
| 232 | +but is problematic for `usr_fn_lib`: |
| 233 | + |
| 234 | +- There are many internal details, that are not meant to be a part of public API but still have to be |
| 235 | + written explicitly (e.g. type names `UsrFnLib`, `FnBoxF64`, location of `fn_lib_base`, and so on). |
| 236 | + These details may change in the future, resulting in unnecessary breaking changes. |
| 237 | + |
| 238 | +- Users have to manually write the `#[pymethods] impl UsrFnLib { ... }` block for each new waveform, |
| 239 | + which may be tedious and intimidating (we don't assume Rust proficiency). |
| 240 | + |
| 241 | +To address these issues, we hide most of the code into procedural macros |
| 242 | +which reduce the above example down to the following: |
| 243 | +```Rust |
| 244 | +use nistreamer_base::usrlib_prelude::*; |
| 245 | +usrlib_boilerplate!(); |
| 246 | + |
| 247 | +/// Linear function: |
| 248 | +/// `MyLinFn(t) = slope*t + offs` |
| 249 | +#[usr_fn_f64] |
| 250 | +pub struct MyLinFn { |
| 251 | + slope: f64, |
| 252 | + offs: f64, |
| 253 | +} |
| 254 | +impl Calc<f64> for MyLinFn { /* implementation here */ } |
| 255 | +``` |
| 256 | +Here: |
| 257 | +- `usrlib_prelude` module bundles all necessary imports; |
| 258 | +- `usrlib_boilerplate!()` function-like macro expands to the `UsrFnLib` setup; |
| 259 | +- `usr_fn_f64` attribute-like macro automatically generates the `impl MyLinFn {pub fn new(...)}` |
| 260 | + and `#[pymethods] impl UsrFnLib { fn MyLinFn(...) }` blocks based on the struct contents |
| 261 | + it is attached to. |
| 262 | + |
| 263 | +So for each new waveform, users only have to write the definition of the struct |
| 264 | +and `Calc<T>` trait implementation. Then it is sufficient to attach one of the two macros - `usr_fn_f64` |
| 265 | +for analog or `usr_fn_bool` for digital waveforms - which auto-generate the rest of the code. |
| 266 | + |
| 267 | +We also utilize the optional `#[pyo3(signature = (...))]` attribute to specify default parameter |
| 268 | +values for the generated Python methods by passing it the token stream from `usr_fn_f64` argument: |
| 269 | +```Rust |
| 270 | +/// Linear function: |
| 271 | +/// `MyLinFn(t) = slope*t + offs` |
| 272 | +/// `offs` is optional and defaults to 0.0 |
| 273 | +#[usr_fn_f64(slope, offs=0.0)] |
| 274 | +pub struct MyLinFn { |
| 275 | + slope: f64, |
| 276 | + offs: f64, |
| 277 | +} |
| 278 | +``` |
| 279 | +results in a waveform method with default values: |
| 280 | +``` |
| 281 | +Signature: usr_fn_lib.MyLinFn(slope, offs=0.0) |
| 282 | +Docstring: |
| 283 | +My linear function: |
| 284 | + `MyLinFn(t) = slope*t + offs` |
| 285 | +`offs` is optional and defaults to `0.0` |
| 286 | +Type: builtin_function_or_method |
| 287 | +``` |
| 288 | + |
| 289 | +Although this is not strictly necessary, there are two similar macros - `std_fn_f64` and `std_fn_bool` - |
| 290 | +which are used for most waveforms in `std_fn_lib` and are based on the same logic as `usr_fn_f64`. |
| 291 | +All macros are defined in a separate sub-package `nistreamer-macros`. |
| 292 | + |
| 293 | +As a technical note, we rely on `multiple-pymethods` feature of `PyO3` to collect methods from all |
| 294 | +`#[pymethods] impl UsrFnLib` blocks together. |
0 commit comments