README.Rmd

---
title: "CEFI"
output: github_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

[NOAA's Physical Science Laboratory (PSL)](https://psl.noaa.gov/) [Changing Ecosystems and Fisheries Initiative Portal](https://psl.noaa.gov/cefi_portal/) serves historical and forecast data useful in ecological studies. Data is served using a [THREDDS](https://psl.noaa.gov/thredds/catalog/Projects/CEFI/regional_mom6/cefi_portal/catalog.html) catalog, but PSL also makes [tabular variable catalogs](https://psl.noaa.gov/cefi_portal/#data_access) available, too.

### Variable Catalogs ("catalogs")

This package leverages the variable catalog listings to access ointers to data sets.  This system works well, but we are reliant on the catalog matching the data products offered.  Catalogs are organized by region ("nwa", "nep", *etc*) and by experiment type ("hindcast", "seasonal_forecast", "seasonal_reforecast", *etc*).

### THREDDS 

If you wish to use a THREDDS catalog service try the [cefitds R package](https://github.com/BigelowLab/cefitds).  Keep in mind that the THREDDS catalog is used for searching for data (programmatically), but once you have found the data, you will want to bring your findings back here to use this package to access the data.

### OPeNDAP/NETCDF

Ultimately, you will use the search tools here to access a URL to an OPeNDAP resource (NETCDF).  There are many tools for accessing OPeNDAP/NETCDF data, but for this work we have chosen [tidync](https://CRAN.R-project.org/package=tidync), but [ncdf4](https://CRAN.R-project.org/package=ncdf4) and others could work as well.


# Requirements

 + [R v4.1+](https://www.r-project.org/)
 + [rlang](https://CRAN.R-project.org/package=rlang)
 + [dplyr](https://CRAN.R-project.org/package=dplyr)
 + [sf](https://CRAN.R-project.org/package=sf)
 + [stars](https://CRAN.R-project.org/package=stars)
 + [jsonlite](https://CRAN.R-project.org/package=jsonlite)
 + [tidync](https://CRAN.R-project.org/package=tidync)

# Installation

```
remotes::install_github("BigelowLab/cefi")
```

# Usage

Load the libraries needed.
```{r}
suppressPackageStartupMessages({
  library(rnaturalearth)
  library(cefi)
  library(tidync)
  library(stars)
  library(dplyr)
})
```

## Catalogs

CEFI offers THREDDS catalogs which are great for mining programmatically, but they also provide simple tabular catalogs: hindcast, seasonal_reforecast, seasonal_forecast.

### Hindcast
```{r}
uri = catalog_uri(region = "northwest_atlantic", xcast = "hindcast")
hindcast = read_catalog(uri) |>
  dplyr::glimpse()
```

One thing to note is that as of Spring 2025 two forms of each grid are provided: "raw" and "regrid".

```{r}
count(hindcast, cefi_grid_type)
```


```{r}
uri = catalog_uri(region = "northwest_atlantic", xcast = "seasonal_reforecast")
refcst = read_catalog(uri) |>
  dplyr::glimpse()
```


```{r}
uri = catalog_uri(region = "northwest_atlantic", xcast = "seasonal_forecast")
fcst = read_catalog(uri) |>
  dplyr::glimpse()
```

Assuming these catalogs will remain and stay up-to-date, we can leverage them in lieu of coding up software to navigate the THREDDS catalogs.

# Getting data

To get data select one row from either catalog, and open that resource which we see in R as a [tidync](https://docs.ropensci.org/tidync/) object useful for navigating and extracting netcdf files.

### Historical data

Let's start with historical data. 

```{r}
nc = hindcast |>
  dplyr::filter(cefi_variable == "btm_o2", 
                cefi_grid_type == "regrid",
                cefi_output_frequency == "daily") |>
  cefi_open()
nc
```

Time requires some background understanding of the CEFI architecture coupled with the `tidync` approach to navigating the NetCDF object.  To ease that for the user we have created a function called `cefi_time()` which is a wrapper around `tidync::hyper_transforms()`, but adds a column called `time_` which can be either 'Date' of 'POSIXct' class.  

```{r}
cefi_time(nc)
```


Next we filter the array with `cefi_filter()`, so that we can collect a subset of the data. This function is a wrapper around the `tidync::hyper_filter()` function; we wrap because the time coordinate in the NetCDF files is not in user-friendly format.  Note that it is important that `time` comes before any other filtering element.

```{r}
nc = cefi_filter(nc, 
                 time = as.Date(c("1993-05-16", "1993-05-20")), 
                 lon = lon > 285 & lon < 300,
                 lat = lat > 40 & lat < 50)
```

It is important to understand that the actual data is not loaded into R (yet), think of this a prefiltering step.  To actually get the data we can call `cefi_var`.  You can explore more about prefiltering at the [tidync website](https://docs.ropensci.org/tidync/).

Now we can load the data into R, using `cefi_var()`.

```{r}
s = cefi_var(nc)
s
```

And finally we can plot the result.

```{r}
coast = rnaturalearth::ne_coastline(scale = "medium", returnclass = "sf") |>
  sf::st_geometry() |>
  sf::st_crop(sf::st_bbox(s)) 

plot_coast = function(){
  plot(sf::st_geometry(coast), add = TRUE, col = "darkorange")
}

plot(s, hook = plot_coast, key.pos = NULL)
```


### Forecast data

Now let's look at forecast data.  Multiple versions are hosted, we are interested in the most recent.

```{r}
nc = fcst |>
  dplyr::filter(cefi_variable == "tob",
                cefi_grid_type == "regrid") |>
  dplyr::slice_tail(n=1)|>
  cefi_open()
nc
```

Here time is measured in `lead` time (in months) relative to an initalization date.  But note that the actually time varying dimension is called `lead` even though we address it in the more familiar `time`. Once again we can use `cefi_time()` to expose the time-varying dimension relative to the starting date.

```{r}
tc = cefi_time(nc)
tc
```

Also note that we have ensemble 'member' is another dimension; it refers to the ensemble identifying member number.  These we can collapse, as shown below, into a single layer for each time step.

```{r}
nc = cefi_filter(nc, 
                 time = tc$time_[c(1,4)], 
                 lon = lon > -75 & lon < -40,
                 lat = lat > 40 & lat < 50)
```                 

Unlike the historical runs, the forecast results include one or more ensemble member results for each time period. These are identified as `member` which you might think of as replicates.  

Here we show retrieving all of the members.

```{r with_members}
s = cefi_var(nc, collapse_fun = NULL)
s
```

To see the ensemble results for one date, we can slice-and-dice using indexing.In this case, there are two variables, `tob` (temperature of bottom) and `tob_anom` temperature of bottom anomaly.  Below we show only `tob`.

```{r}
times = st_get_dimension_values(s, "time")
plot(s['tob'], hook = plot_coast)
```


Requesting all members for a given time is possible as shown above, but a more common use case is to extract a summary (mean or median) or a measure of variability (standard deviation or variance).

```{r cefi_var}
s = cefi_var(nc, collapse_fun = mean)
s
```

Note that the dimensionality is reduced because we computed the mean of the ensembles at each time. 


```{r collaped_mean_plot}
plot(s['tob'], hook = plot_coast, main = "Mean Temp of Bottom")
```


You may found it easier to work with a raster based on [-180,180] longitudes. We provide a function to "shift" the raster to that range.

```{r shift}
s = shift_stars(s)
```



### Granularity

Here we highlight the granularity of the data source by zooming in on the New England coastline. You can see that large embayments such as Penobscot and Narragansett bays are not included. 

```{r granularity}
plot(dplyr::slice(s['tob'], "time",  1),
     xlim = c(-72 , -63),
     ylim = c(39, 45),
     reset = FALSE,
     axes = TRUE)
plot(coast, add = TRUE, col = "orange")
```