Skip to content

Commit f98c7f6

Browse files
robinskilrobinskil
authored
update changelog (#243)
1 parent 7f41cdf commit f98c7f6

4 files changed

Lines changed: 446 additions & 44 deletions

File tree

‎docs/.vitepress/config.mts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -757,6 +757,19 @@ export default defineConfig({
757757
}
758758
],
759759
'/docs/changelog': [
760+
{
761+
text: 'Release Posts',
762+
items: [
763+
{
764+
text: 'What\'s new in 1.7.0',
765+
link: '/docs/changelog/release-1.7.0'
766+
},
767+
{
768+
text: 'What\'s new in 1.6.0',
769+
link: '/docs/changelog/release-1.6.0'
770+
},
771+
]
772+
},
760773
{
761774
text: 'Changelog',
762775
items: [

‎docs/docs/changelog/index.md‎

Lines changed: 158 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,174 @@
11
# Changelog
22

3-
## (Unreleased)
3+
> **Release posts:** [What's new since 1.7.0](/docs/changelog/release-1.7.0) · [What's new in 1.6.0](/docs/changelog/release-1.6.0)
44
5-
## (v1.7.0 - 2026-06-10)
5+
All notable changes to Beacon are documented here, newest first. Entries are
6+
grouped into **Added** (new features), **Changed** (behaviour or internal
7+
changes), and **Fixed** (bug fixes).
68

7-
- **Changed**: Managed SQL tables (`CREATE TABLE`) are now backed by [Apache Iceberg](https://iceberg.apache.org/) instead of the previous Parquet-manifest format, giving them an ACID, schema-tracked, snapshot-based storage layer. Their data and metadata live in Beacon's internal area of the configured storage (local or S3), alongside the datasets.
8-
- **Added**: Row mutations on managed tables. In addition to `INSERT`, managed tables now support `DELETE ... WHERE`, `UPDATE ... SET ... WHERE`, and `CREATE TABLE AS SELECT`. `DELETE` and `UPDATE` are copy-on-write.
9-
- **Added**: Schema evolution with `ALTER TABLE` on managed tables — `ADD COLUMN`, `DROP COLUMN`, `RENAME COLUMN`, and `ALTER COLUMN ... TYPE` (safe widening promotions). Existing rows keep reading correctly: added columns read `NULL` and renames preserve values. See [CREATE TABLE (Managed)](/docs/1.7.0/sql/managed-tables).
9+
## v1.7.0 — 2026-06-10
1010

11-
## (v1.6.1 - 2026-06-04)
11+
### Added
1212

13-
- **Added**: Support for the Atlas file format. Atlas is a directory-based array store — a single `atlas.json` registry describing one or more datasets — that Beacon discovers and queries automatically, just like Parquet or Zarr. Query a store with the `read_atlas()` table function or register it as an external table with `STORED AS ATLAS`. Atlas keeps per-dataset column statistics, so Beacon prunes whole datasets before reading any array data and only loads the projected arrays. This makes re-encoding large NetCDF or Zarr collections into a single Atlas collection the recommended way to speed up repeated spatial/temporal range queries. See [github.com/maris-development/atlas](https://github.com/maris-development/atlas).
14-
- **Added**: Materialized views. `CREATE MATERIALIZED VIEW` runs a query once and persists the result as Parquet, so repeated, aggregation-heavy queries read straight from the cached result instead of recomputing. Use the new `REFRESH` statement to recompute a view when its underlying data changes — a full recompute that atomically swaps in the new result, leaving the previous result intact if the refresh fails.
15-
- **Added**: Configurable base path via the `BEACON_BASE_PATH` environment variable. The HTTP API, OpenAPI document, and Swagger UI can now be served under a path prefix, making it easier to run Beacon behind a reverse proxy or on a shared subpath.
16-
- **Added**: Support for decompressing LZW-compressed, stripped GeoTIFFs, broadening the range of GeoTIFF/COG files Beacon can read.
17-
- **Changed**: Upgraded the query engine to DataFusion 53 and Arrow 58. The Arrow version is now configurable at build time to ease integration with downstream tooling.
18-
- **Fixed**: EDMO code extraction now uses the last set of parentheses in a SeaDataNet originator string, so institution names that themselves contain parentheses are mapped to the correct EDMO code.
13+
- **Row mutations on managed tables.** Alongside `INSERT`, managed tables now
14+
support `DELETE ... WHERE`, `UPDATE ... SET ... WHERE`, and `CREATE TABLE AS
15+
SELECT`. `DELETE` and `UPDATE` are copy-on-write.
16+
- **Schema evolution with `ALTER TABLE`** on managed tables — `ADD COLUMN`,
17+
`DROP COLUMN`, `RENAME COLUMN`, and `ALTER COLUMN ... TYPE` (safe widening
18+
promotions). Existing rows keep reading correctly: added columns read `NULL`
19+
and renames preserve values. See [CREATE TABLE (Managed)](/docs/1.7.0/sql/managed-tables).
20+
- **CF `calendar` support.** CF time-unit parsing now honours the optional CF
21+
`calendar` attribute, so non-Gregorian calendars are interpreted correctly.
22+
- **SeaDataNet L05 mappings.** New UDFs map SeaDataNet instrument L05 codes for
23+
salinity and temperature.
1924

20-
## (v1.6.0 - 2026-05-08)
25+
### Changed
2126

22-
- **Added**: Support for Flight SQL. In addition to the existing HTTP query endpoint, users can now query datasets using the Flight SQL protocol. This provides a more efficient and performant interface for querying datasets. It also allows Beacon to be used with a wider range of clients (JetBrains DataGrip, DBeaver) and tools that support Flight SQL, such as Apache Arrow Flight clients and BI tools.
23-
- **Added**: Support for SQL Tables. Possibility to create a SQL table, backed by Parquet files (local or in the cloud). This allows users to create custom tables that are not directly tied to existing datasets, and populate those tables with data from other files or bare insert statements.
24-
- **Added**: Support for SQL views. Create SQL views on top of datasets and expose those views via the API. This allows for more flexible data modeling and querying capabilities, as data managers can define custom views that combine and transform their underlying datasets in various ways.
25-
- **Added**: Support for Tiff files. Beacon can now read and query Tiff files, including geotiff and cloud-optimized geotiff formats. This allows users to work with raster data in addition to the existing support for tabular data formats.
26-
- **Added**: Support for ODV ASCII files. ODV ASCII files can be read directly as datasets in Beacon and queried via the API. ODV ASCII files can also be streamed via the S3 protocol, allowing for efficient access to large ODV datasets stored in S3-compatible object storage.
27-
- **Fixed**: Fixed a bug in the NetCDF reader, where attributes stored as a list of values were not being read correctly while containing a single value. This ensures that all attributes that are scalars can be read by Beacon.
28-
- **Added**: NetCDF reader supports streamed reading using chunks. This allows for more efficient reading of large NetCDF files, as data can be read in smaller chunks rather than loading the entire file into memory at once. This is especially beneficial for users working with large NetCDF datasets, as it can significantly reduce memory usage and improve performance.
29-
- **Added**: NetCDF reader supports pushing down filters to the reader for coordinate variables. This allows for more efficient querying of NetCDF datasets, as filters can be applied directly to the coordinate variables during reading, reducing the amount of data that needs to be read and processed in memory.
30-
- **Added**: Support for reading "ragged" NetCDF & Zarr datasets. Ragged datasets are a type of file that can contain variables with different lengths, allowing for more flexible data structures. This allows users to work with a wider range of datasets, including those that do not conform to the traditional rectangular structure.
27+
- **Managed tables are now backed by [Apache Iceberg](https://iceberg.apache.org/)**
28+
instead of the previous Parquet-manifest format, giving them an ACID,
29+
schema-tracked, snapshot-based storage layer. Data and metadata live in
30+
Beacon's internal area of the configured storage (local or S3), alongside the
31+
datasets.
32+
- **CF time parsing** for NetCDF and Zarr is centralized in one module, replacing
33+
the previous per-backend regex parsing.
34+
- **Global NetCDF attributes** are now surfaced with a leading dot (e.g.
35+
`.Conventions`) to cleanly distinguish them from variable attributes.
36+
- **Filesystem event watching** (`BEACON_ENABLE_FS_EVENTS`) now defaults to
37+
enabled, so new files in watched datasets are picked up automatically.
38+
- **Removed periodic table auto-sync** and its `BEACON_TABLE_SYNC_INTERVAL_SECS`
39+
setting — managed tables are now transactional and no longer need background
40+
refreshes.
3141

32-
## (v1.5.4 - 2026-01-05)
42+
## v1.6.1 — 2026-06-04
3343

34-
- **Added**: Support for SQL querying. Users can now query datasets using SQL syntax, in addition to the existing JSON query format. This provides a more familiar and powerful interface for users who are comfortable with SQL.
35-
- **Added**: Support for SQL querying for native datasets (eg. NetCDF, Parquet). Users can now use SQL to query native datasets directly, without needing to create a collection first. This allows for more flexible and ad-hoc querying of datasets.
36-
- **Fixed**: Fixed a bug in NetCDF output where the FillValue was not being set correctly. This ensures that missing values are properly represented in the NetCDF output.
37-
- **Fixed**: Updated querying documentation to include examples for SQL querying. This provides users with clear guidance on how to use the new SQL querying features.
38-
- **Fixed**: Update querying documentation to include examples for JSON querying. This ensures that the documentation is comprehensive and up-to-date with the latest features.
39-
- **Added**: Support for setting the "unit" field in the ODV output schema for columns. This allows users to specify the units of measurement for columns in the ODV output, improving the clarity and usability of the output data.
44+
### Added
4045

41-
## (v1.2.0 - 2025-09-01)
46+
- **Atlas file format.** Atlas is a directory-based array store — a single
47+
`atlas.json` registry describing one or more datasets — that Beacon discovers
48+
and queries automatically, just like Parquet or Zarr. Query a store with the
49+
`read_atlas()` table function or register it as an external table with `STORED
50+
AS ATLAS`. Atlas keeps per-dataset column statistics, so Beacon prunes whole
51+
datasets before reading any array data and only loads the projected arrays.
52+
This makes re-encoding large NetCDF or Zarr collections into a single Atlas
53+
collection the recommended way to speed up repeated spatial/temporal range
54+
queries. See [github.com/maris-development/atlas](https://github.com/maris-development/atlas).
55+
- **Materialized views.** `CREATE MATERIALIZED VIEW` runs a query once and
56+
persists the result as Parquet, so repeated, aggregation-heavy queries read
57+
straight from the cached result instead of recomputing. The new `REFRESH`
58+
statement does a full recompute and atomically swaps in the new result,
59+
leaving the previous result intact if the refresh fails.
60+
- **Configurable base path** via the `BEACON_BASE_PATH` environment variable. The
61+
HTTP API, OpenAPI document, and Swagger UI can be served under a path prefix,
62+
making it easier to run Beacon behind a reverse proxy or on a shared subpath.
63+
- **LZW-compressed, stripped GeoTIFFs** can now be decompressed, broadening the
64+
range of GeoTIFF/COG files Beacon can read.
65+
- **Zarr v3 support.** Zarr reading moved onto Beacon's shared n-dimensional
66+
array engine, adding Zarr v3 support and predicate pushdown for Zarr-backed
67+
datasets.
4268

43-
- **Added**: Support for S3. This all is abstracted via the Object Store Crate. All file sources are abstracted via the Object Store itself and support various backends. (Eg. MinIO, AWS S3, Cloudflare R2)
44-
- **Added**: Implemented support for ODV ASCII streaming via S3 protocol.
45-
- **Added**: Support NetCDF cloud support using #mode=bytes. This only works with http/https specified cloud storage endpoints.
46-
- **Fixed**: Update dependencies to work with the latest version of: Arrow, Datafusion, GeoParquet.
47-
- **Fixed**: Rewrite of file source reading to support S3.
48-
- **Fixed**: Added new table version such as: Preset Tables (Allows to create data collections with meta data descriptions.), Geo Spatial Tables (Allows to create geo-spatial data collections with meta data descriptions.).
69+
### Changed
4970

50-
## (v1.0.1 - 2025-05-05)
71+
- **Upgraded the query engine to DataFusion 53 and Arrow 58.** The Arrow version
72+
is now configurable at build time to ease integration with downstream tooling.
5173

52-
- **Added**: Support for SQL Querying. https://github.com/maris-development/beacon/issues/6
53-
- **Added**: Support for SQL Querying for native datasets (eg. NetCDF, Parquet). https://github.com/maris-development/beacon/issues/33
74+
### Fixed
5475

55-
- **Fixed**: Fixed a bug in NetCDF output where the FillValue was not being set correctly. https://github.com/maris-development/beacon/issues/45
56-
- **Fixed**: Updated querying documentation to include examples for SQL querying.
57-
- **Fixed**: Update querying documentation to include examples for JSON querying.
76+
- **EDMO code extraction** now uses the last set of parentheses in a SeaDataNet
77+
originator string, so institution names that themselves contain parentheses
78+
are mapped to the correct EDMO code.
5879

59-
- **Added**: Support for setting the "unit" field in the ODV output schema for columns. https://github.com/maris-development/beacon/issues/52
60-
- **Fixed**: Missing value flag in NetCDF output schema. https://github.com/maris-development/beacon/issues/45
80+
## v1.6.0 — 2026-05-08
81+
82+
### Added
83+
84+
- **Flight SQL.** In addition to the existing HTTP query endpoint, datasets can
85+
be queried over the Flight SQL protocol — a more efficient, Arrow-native
86+
interface that opens Beacon up to clients such as JetBrains DataGrip, DBeaver,
87+
and other Apache Arrow Flight and BI tools.
88+
- **SQL tables.** Create custom tables backed by Parquet files (local or in the
89+
cloud), populated from other files or bare `INSERT` statements — not tied to an
90+
existing dataset.
91+
- **SQL views.** Define views on top of datasets and expose them via the API,
92+
combining and transforming underlying datasets into purpose-built shapes.
93+
- **GeoTIFF support.** Read and query TIFF files, including GeoTIFF and
94+
Cloud-Optimized GeoTIFF (COG), via the new `read_tiff()` table function —
95+
adding raster data alongside Beacon's tabular formats.
96+
- **ODV ASCII support.** ODV ASCII files can be read directly as datasets and
97+
queried via the API, and streamed over the S3 protocol for efficient access to
98+
large ODV datasets in S3-compatible object storage.
99+
- **Ragged NetCDF & Zarr datasets.** Read datasets whose variables have differing
100+
lengths and don't conform to a rectangular structure.
101+
- **NetCDF chunked streaming.** Large NetCDF files are read in chunks rather than
102+
loaded wholesale into memory, reducing memory use and improving performance.
103+
- **NetCDF coordinate filter pushdown.** Filters on coordinate variables are
104+
applied during reading, so far less data is read and processed for bounded
105+
spatial/temporal queries.
106+
- **NetCDF statistics & partition pruning.** Per-column min/max statistics let
107+
Beacon prune whole files before reading, with new metadata table functions
108+
(`view_dataset_statistics`, `view_external_table_statistics`,
109+
`view_statistics_cache`) to inspect cached statistics.
110+
- **Merged tables.** A table type that references and combines data from other
111+
tables, with dependency tracking that prevents deleting a table a merged table
112+
still relies on.
113+
114+
### Fixed
115+
116+
- **NetCDF scalar attributes.** Attributes stored as a single-element list were
117+
not read correctly; all scalar attributes are now read properly.
118+
119+
## v1.5.4 — 2026-01-05
120+
121+
### Added
122+
123+
- **SQL querying.** Query datasets using SQL syntax in addition to the existing
124+
JSON query format.
125+
- **SQL querying for native datasets** (e.g. NetCDF, Parquet) directly, without
126+
first creating a collection.
127+
- **ODV output units.** Set the `unit` field per column in the ODV output schema,
128+
improving the clarity of the output data.
129+
130+
### Fixed
131+
132+
- **NetCDF FillValue** is now set correctly on output, so missing values are
133+
properly represented.
134+
135+
### Docs
136+
137+
- Added querying examples for both SQL and JSON query formats.
138+
139+
## v1.2.0 — 2025-09-01
140+
141+
### Added
142+
143+
- **S3 / object storage.** All file sources are abstracted via the Object Store
144+
crate, supporting backends such as MinIO, AWS S3, and Cloudflare R2.
145+
- **ODV ASCII streaming over the S3 protocol.**
146+
- **NetCDF cloud reading** using `#mode=bytes` (http/https cloud storage
147+
endpoints only).
148+
- **New table types:** Preset Tables (data collections with metadata
149+
descriptions) and Geo-Spatial Tables (geo-spatial collections with metadata
150+
descriptions).
151+
152+
### Changed
153+
154+
- Rewrote file-source reading to support S3.
155+
- Updated dependencies for the latest Arrow, DataFusion, and GeoParquet.
156+
157+
## v1.0.1 — 2025-05-05
158+
159+
### Added
160+
161+
- **SQL querying** ([#6](https://github.com/maris-development/beacon/issues/6))
162+
and **SQL querying for native datasets** such as NetCDF and Parquet
163+
([#33](https://github.com/maris-development/beacon/issues/33)).
164+
- **ODV output units.** Set the `unit` field per column in the ODV output schema
165+
([#52](https://github.com/maris-development/beacon/issues/52)).
166+
167+
### Fixed
168+
169+
- **NetCDF FillValue / missing-value flag** is now set correctly on output
170+
([#45](https://github.com/maris-development/beacon/issues/45)).
171+
172+
### Docs
173+
174+
- Added querying examples for both SQL and JSON query formats.

0 commit comments

Comments
 (0)