datacaged

Leia em Português

CAGED microdata straight into your 'DuckDB', with a complete pipeline ready for analysis.

datacaged is an R package that automates the entire workflow with CAGED (Cadastro Geral de Empregados e Desempregados — Brazil's General Register of Employed and Unemployed Workers) microdata. Data is downloaded from a HuggingFace repository via HTTPS, parsed with streaming decompression, and stored in a local 'DuckDB' database for fast, scalable queries with dplyr or SQL.

Key features

• Automated download from HuggingFace (HTTPS — no FTP required)

• Parallel downloads (workers = 3 by default)

• Stream decompression of .7z files — no temporary disk extraction

• Efficient storage in a local 'DuckDB' database

• Full pipeline in a single function call

• Integration with dplyr for data analysis

• Compatible with Windows, macOS and Linux

• Support for multiple datasets:

  • Novo CAGED (2020+)
  • Legacy CAGED (2007–2019)
  • CAGED Adjustments (historical corrections)

Installation

Stable version from CRAN:

install.packages("datacaged")

Development version from GitHub:

# install.packages("pak")
pak::pak("gecomt/datacaged")

Platform compatibility

Feature	Windows	macOS	Linux
Download (HTTPS)	Yes	Yes	Yes
Parallel downloads	Yes	Yes	Yes
Novo CAGED (2020+, LZMA)	Yes	Yes	Yes
Legacy CAGED (pre-2020, PPMd)	Requires 7-Zip	Requires 7-Zip	Requires 7-Zip
DuckDB	Yes	Yes	Yes

Legacy CAGED (pre-2020) uses PPMd compression which requires 7-Zip. Novo CAGED (2020+) works on all platforms without any extra software.

Installing 7-Zip (only needed for pre-2020 data)

Windows — install the 7-Zip installer (standard installation to C:\Program Files\7-Zip is automatically detected).

macOS

brew install 7-zip

Linux (Debian/Ubuntu)

sudo apt install 7zip

Linux (Fedora/RHEL)

sudo dnf install 7zip

Quick start

library(datacaged)

# Full pipeline — downloads, parses and writes to DuckDB
caged_load(
  years   = 2022:2023,
  months  = seq_len(12L),
  db_path = file.path(tempdir(), "caged.duckdb")
)

# Connect and query
con <- caged_connect(file.path(tempdir(), "caged.duckdb"))

library(dplyr)

tbl(con, "caged_mov") |>
  filter(uf == 35, competenciamov >= 202201L) |>
  group_by(competenciamov) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  collect()

DBI::dbDisconnect(con, shutdown = TRUE)

Performance

# Parallel downloads (default: 3 workers)
caged_download(years = 2023, months = 1:12, workers = 3)

# Sequential (for slow connections or debugging)
caged_download(years = 2023, months = 1:12, workers = 1)

# Set globally
options(datacaged.workers = 4)

Parallel downloads process MOV, FOR and EXC files simultaneously per month — approximately 3× faster than sequential for Novo CAGED.

Pipeline

HuggingFace (HTTPS)
      ↓
  caged_download()   ← parallel, with local cache
      ↓
  caged_parse()      ← stream decompression via archive_read()
      ↓
  caged_to_duckdb()  ← bulk insert, deduplication by period
      ↓
  caged_connect()    ← dplyr / SQL / DBI

Main functions

Function	Description
caged_load()	Full pipeline (download → parse → DuckDB)
caged_adjustments_load()	Adjustments pipeline
caged_download()	Download .7z files with cache
caged_download_layouts()	Download official MTE layouts (from HuggingFace)
caged_parse()	Parse a single .7z file
caged_parse_batch()	Batch parse
caged_to_duckdb()	Write to DuckDB
caged_connect()	Open DBI connection
caged_info()	Database statistics
caged_status()	Check HuggingFace connectivity
caged_hf_files()	List available competencies
caged_update()	Download only new competencies
caged_to_parquet()	Export tables to Parquet files

Message language

Console messages (errors, warnings and progress) are written in English and translated to Portuguese. With R set to Portuguese they appear in Portuguese automatically; to switch within a session:

Sys.setLanguage("pt_BR")  # Portuguese
Sys.setLanguage("en")     # English

Database tables

Table	Content	Period
caged_mov	Movements	2020+
caged_for	Late declarations	2020+
caged_exc	Exclusions	2020+
caged_antigo	Historical data	2007–2019
caged_ajustes	Retroactive adjustments	2002–2019

Requirements

• R ≥ 4.2.0
• 7-Zip (optional — only required for CAGED pre-2020 with PPMd compression)

Use cases

• Labour market analysis
• Economic indicators
• Academic research
• Formal employment monitoring
• Econometric modelling

Licence

MIT © Alexsandro Prado

Contact

Author: Alexsandro Prado
Email: alexsandro.prado@ufersa.edu.br
GitHub: https://github.com/gecomt/datacaged