Skip to content

monitoring.md

Latest commit

 

History

History
154 lines (101 loc) · 6.62 KB

monitoring.md

File metadata and controls

154 lines (101 loc) · 6.62 KB
Raw

Lantern - Monitoring

Monitoring configuration

These options from the app lantern.Config class are used to configure application logging:

  • ENABLE_FEATURE_SENTRY: if true, enables backend Error Monitoring via Sentry
  • SENTRY_ENVIRONMENT: the Sentry Environment name
  • SENTRY_DSN: Sentry backend Data Source Name (DSN) for error logging
  • CHECKS_TRUSTED_USERNAME: credential for trusted publishing based Checks
  • CHECKS_TRUSTED_PASSWORD: credential for trusted publishing based Checks

See the Config docs for how to set these config options.

Health monitoring

Heartbeat

A static text file is available within a Site as a basic, binary, health monitoring endpoint at: /static/txt/heartbeat.txt.

This endpoint MAY be used by monitoring tools and/or load balancers to determine if the static site is available.

A non-200 status code MUST be considered as the static site being unavailable.

Health check endpoint

A static JSON health check is available within a Site, structured as per the Draft API Health Check specification.

It is available at /-/health and built by the Site Health Output.

This endpoint MAY be used by monitoring tools and/or load balancers to determine:

  • readiness: to determine if the site can process requests, expected to always be true
  • liveness: to determine if the site is running or not, expected to always be true

Tip

The health check may be 'live' but not 'ready', during deployments or maintenance for example.

Health check checks

  • site:records: The number of records contained in the static site
  • search:records: The number of records contained in the search index

Warning

Health checks are evaluated when the Static Site is built.

Sentry

Error monitoring

Errors in the Python backend (where enabled) and frontend Static Site (always) are logged to Sentry for aggregation and alerting.

Alerts are sent via email and to the #dev channel in the MAGIC Teams workspace.

Uptime checks

A Sentry Uptime Check checks the BAS Map Catalogue collection returns a 2xx response in the production environment every 5 minutes, automatically following the Item Alias redirect.

User feedback

Sentry's User Feedback feature is used to collect user feedback via a custom widget.

Warning

Sentry user feedback is only retained for 90 days. An anonymous version can be copied to GitLab if needed.

Plausible

Plausible is used for recording web analytics in the BAS Catalogue static site.

Site checks

Checks can be run from a Catalogue to verify the contents of a Site and downloads referenced in Records via making URL requests to expected resources.

Note

Check results are not considered sensitive and are publicly accessible, including for trusted publishing content.

Checks verify:

  • a subset of site resources (CSS, images, etc.)
  • guides, legal policies and other global, non-record related pages
  • the site 404 page and that the error handler is working (by requesting a page known not to exist)
  • Record pages (JSON, XML, HTML)
  • Item pages (HTML) and Aliases redirects
  • Item pages in Trusted Publishing
  • DOI redirects if set within records
  • distribution options set within records (with special handling for ArcGIS layers and services)

Warning

Checks are not run for:

  • BAS SAN and SharePoint distribution options within records
  • DOI redirects where not using the live site environment (as these only work in production)

Checks are intended to verify a whole site, including all records within a store, systematically on a regular basis. Ad-hoc checks for specific records CAN be run as part of record publishing workflows.

Checks are generated in a pending state for site content by Outputs alongside their content, via the Site class.

Tip

Checks for Record contents are generated by the RecordChecks class and included in the checks for the Record ISO XML Output. This attachment is somewhat arbitrary.

Sets of pending checks are executed in parallel using the lantern.checks.Checker class, then processed into JSON Data and a Report via the Checks Output.

Checks are executed using lantern.checks.CheckRunner classes, with methods for different types of check. For example, Item Alias checks verify the expected redirect location is returned, and that this location exists.

Note

Checks for some distribution options in Records are not implemented, and generated as skipped rather than pending. Skipped checks are not executed but are included in compiled data and reports.

Site checks data

Executed checks are compiled into a JSON data file for machine use and optional additional processing.

Available at: /-/checks/data.json

Tip

See the OpenAPI definition for the schema used for this JSON data.

Site checks report

Compiled results from executed checks are available in an HTML report for manual review.

Available at: /-/checks/index.html

Scheduled checks

The live site Environment is automatically verified via a cron job running on the BAS central workstations:

  • scope: all items
  • frequency: Wednesdays at 12:15 (local time)
  • results: data.bas.ac.uk/-/checks/
  • logs: /users/geoweb/cron_logs/lantern/lantern-checks-*.log
  • log retention: 90 days (enforced monthly)
  • deployment: via Ansible