|
4 | 4 | [](https://pypi.org/project/ztensor/) |
5 | 5 | [](https://opensource.org/licenses/MIT) |
6 | 6 |
|
7 | | -Simple tensor serialization format |
8 | | - |
9 | | -## Key Features |
10 | | - |
11 | | -- **Simple Spec** — Minimalist [spec](SPEC.md) for easy parsing. |
12 | | -- **Zero-Copy Access** — Instant memory-mapping (mmap) with no extra RAM copying overhead. |
13 | | -- **Efficient Writes** — Supports streaming and append-only operations without rewriting files. |
14 | | -- **Future-Proof** — Decouples physical storage from logical representation for long-term compatibility. |
15 | | - |
16 | | -## Tools |
17 | | - |
18 | | -- **Rust Core** — High-performance, SIMD-aligned implementation. |
19 | | -- **Python API** — First-class bindings for **NumPy** and **PyTorch**. |
20 | | -- **Universal Converters** — CLI tools to convert **SafeTensors**, **GGUF**, **Pickle**, **NumPy (.npz)**, **ONNX**, and **HDF5** files. |
21 | | - |
22 | | -## Comparison |
23 | | - |
24 | | -| Feature | **zTensor** | SafeTensors | GGUF | Pickle | HDF5 | |
25 | | -| :--- | :---: | :---: | :---: | :---: | :---: | |
26 | | -| **Zero-Copy Read** | ✅ | ✅ | ✅ | ❌ | ⚠️ | |
27 | | -| **Safe (No Exec)** | ✅ | ✅ | ✅ | ❌ | ✅ | |
28 | | -| **Streaming / Append** | ✅ | ❌ | ❌ | ❌ | ✅ | |
29 | | -| **Sparse Tensors** | ✅ | ❌ | ❌ | ✅ | ✅ | |
30 | | -| **Compression** | ✅ | ❌ | ❌ | ❌ | ✅ | |
31 | | -| **Quantization** | ✅ | ✅ | ✅ | ✅ | ✅ | |
32 | | -| **Parser Complexity** | 🟢 Low | 🟢 Low | 🟡 Med | 🔴 High | 🔴 High | |
33 | | - |
34 | | -## Benchmark |
35 | | - |
36 | | - |
37 | | - |
38 | | -See [benchmark](benchmark/bench.py) for more details. |
| 7 | +A simple, safe tensor serialization format with zero-copy reads, streaming writes, and built-in compression. |
39 | 8 |
|
40 | 9 | ## Installation |
41 | 10 |
|
42 | | -### Python |
43 | | - |
44 | 11 | ```bash |
45 | | -pip install ztensor |
| 12 | +pip install ztensor # Python |
| 13 | +cargo add ztensor # Rust |
| 14 | +cargo install ztensor-cli # CLI |
46 | 15 | ``` |
47 | 16 |
|
48 | | -See the [Python docs](https://pie-project.github.io/ztensor/python) for the full API reference. |
49 | | - |
50 | | -### Rust |
51 | | - |
52 | | -```toml |
53 | | -[dependencies] |
54 | | -ztensor = "1.2" |
55 | | -``` |
56 | | - |
57 | | -See the [Rust docs](https://pie-project.github.io/ztensor/rust) or [docs.rs](https://docs.rs/ztensor) for the full API reference. |
58 | | - |
59 | | -### CLI |
60 | | - |
61 | | -```bash |
62 | | -cargo install ztensor-cli |
63 | | -``` |
64 | | - |
65 | | -See the [CLI docs](https://pie-project.github.io/ztensor/cli) for usage details. |
66 | | - |
67 | | -## Supported Data Types |
| 17 | +## Features |
68 | 18 |
|
69 | | -| Type | Description | |
70 | | -|------|-------------| |
71 | | -| `float32`, `float16`, `bfloat16`, `float64` | Floating point | |
72 | | -| `int8`, `int16`, `int32`, `int64` | Signed integers | |
73 | | -| `uint8`, `uint16`, `uint32`, `uint64` | Unsigned integers | |
74 | | -| `bool` | Boolean | |
| 19 | +- Zero-copy memory-mapped reads |
| 20 | +- Streaming, append-only writes |
| 21 | +- Zstd compression and checksums |
| 22 | +- Dense, sparse (CSR/COO), and quantized tensors |
| 23 | +- Convert from SafeTensors, GGUF, Pickle, NumPy, ONNX, and HDF5 |
75 | 24 |
|
76 | | -## File Format |
| 25 | +## Documentation |
77 | 26 |
|
78 | | -See [SPEC.md](SPEC.md) for the complete specification. |
| 27 | +- [Website](https://pie-project.github.io/ztensor/) — guides, API reference, benchmarks |
| 28 | +- [Spec](SPEC.md) — file format specification |
| 29 | +- [docs.rs](https://docs.rs/ztensor) — Rust API docs |
79 | 30 |
|
80 | 31 | ## License |
81 | 32 |
|
|
0 commit comments