lantern.catalogue.BasCatalogue
The BAS Catalogue underpins the BAS Data Catalogue as the only supported Catalogue and part of the MAGIC Spatial Data Infrastructure (SDI).
Tip
See the Access docs for how to access the BAS Catalogue.
At a high level it consists of these components:
- a BAS multi-store Repository to read and write records in:
- an experimental editor, Zap ⚡️ to create Records manually
- automated systems maintaining their own Records
- sub-catalogues for:
- testing and live environments (used for previewing content and general access respectively)
- trusted and untrusted content, per environment
- for untrusted (public) content, a Site hosted in AWS S3 for public consumption
- for trusted (internal) content, a Site hosted in the BAS Operations Data Store 🛡, restricted to the Admin role
Note
Trusted content and Publishing is used for items containing decrypted Administration Metadata which are considered sensitive.
See these additional diagrams for:
- how these components are used for Creating and Updating Records
- how these components are used for Publishing content
The BAS Catalogue Repository:
- reads and writes Records as Record Revisions in GitLab:
- supports branches and merge requests, with methods to open, list and merge changesets of Records
- always reads the latest revision of each Record for a given branch
- always merges changesets into the default branch
- writes Records to Algolia in a global search indexing:
- using a single, global, all Records index tracking the default GitLab branch
- coordinates stores to ensure consistency wherever possible:
- by updating records in Algolia when a GitLab branch is merged into the default branch
- does not support renaming or removing Records
The trusted Export Metadata flag can be used to indicate where static site content
is considered sensitive, and MUST be only be available to a restricted audience. This is termed trusted publishing.
Where set, Outputs MAY include Administrative Metadata for example.
lantern
A Python application defined within the src/lantern package consisting of:
lantern.catalogues
Catalogues are the core component of this project, responsible for:
- managing Records
- transforming these into a static website for discovery
- Checking the static website's content and downloads linked from records
A minimum Catalogue consists of:
- a Store to manage and access Records created by Editors
- a Site generator
- an Exporter to publish the generated site to a hosting service, such as AWS S3
- a Checker to verify generated site content and downloads linked from records
Important
Only the BAS Catalogue is officially supported by this project.
lantern.repositories
Repositories abstract managing Records in one or more Stores within larger Catalogues.
lantern.site.Site
Sites are static websites built from a set of Records and other content as the output of a Catalogue.
They generate content, checks and/or cache invalidation keys for content from Outputs using a Store to access records.
Tip
A static site is used over a dynamic site for its robustness and ease of hosting, such as via AWS S3.
See the Outputs docs for information about the content within a site.
See the Static site docs for information about the site structure, templates, styles, scripts, etc.
See the Checks docs for information about the checks generated for a site.
Note
Some outputs MAY require additional properties, populated by Export Metadata where possible, or dict of extra values passed to a Site.
lantern.stores
Stores create, update, read and delete Records in local or remote systems, such as GitLab. They are used in Repositories in larger Catalogues.
They provide access to Records used to build a Site and may add or update Records for use in future builds.
See the Stores docs for more information.
lantern.outputs
Outputs create different parts of a Site, such as Item pages, general resources such as CSS files and monitoring / API discovery endpoints. Outputs also create Checks for their content.
See the Outputs docs for more information.
lantern.exporters
Exporters save files to local or remote systems, such as an S3 bucket.
They are used in Sites to store and/or publish content generated by Outputs.
See the Exporters docs for more information.
Editors create and update Record configurations. They may be interactive (for use by humans to author bespoke Records), or automated systems, managing sets of Records for a particular purpose. All editors are treated equally.
In either case, Record configurations are loaded into the Catalogue and then persisted via a Store.
See the Usage docs for information about creating and updating Records.
The BAS GitLab instance is used as a remote Records Store and change tracking tool.
GitLab is used for:
- consistency with other projects (including this project)
- support for merge requests for simple record review workflows via a hosted user interface
- being able to manage provisioning, permissions and git operations programmatically
See the GitLab Store for details on how Records are stored in GitLab.
See the Infrastructure docs for more information about the GitLab project used.
Amazon Web Services (AWS) Simple Storage System (S3) static hosting and a CloudFront Distribution are used to host untrusted Site content.
AWS is used for:
- its high availability, global infrastructure and low cost
- its support for object level server redirects and content metadata
- the ability to manage provisioning, access control and content programmatically
See the S3 Exporter for details on how information is stored in S3.
See the Static Site docs for more information on what is stored in S3.
Note
AWS CloudFront caches content in the live site environment by default. The BAS Catalogue automatically invalidates content as needed when publishing.
Algolia is used as a remote Records Store specifically for enabling Site Search functionality.
Algolia is used for:
- its focus on search, speed and generous free tier
- frontend library to implement a search interface
- ease of managing the backend index
See the Algolia Item Model for details on what is indexed for each Record.
See the Algolia Store for details on how Records are stored in Algolia.
ArcGIS Online is used in Records as a key data access system for spatial services. It also underpins the Embedded Maps Service used for Item Extent Visualisations functionality.
ArcGIS Online is used for:
- consistency with other projects and to align with the BAS Spatial Data Infrastructure
- its stability, performance and features
See the Infrastructure docs for more information about the underlying infrastructure used.
Note
Managed services, external to BAS on-premise infrastructure, are generally preferred where practical and available for stability, performance and agility as the project grows.