English | Português do Brasil

English:

Welcome to [Static Site Generator Benchmarks (SSGBerk)]

What is Static Site Generator (SSG)?

A static web page (sometimes called a flat page/stationary page) is a web page that is delivered to the user exactly as stored, in contrast to dynamic web pages which are generated by a web application.

What SSGBerk is?

A project Forked from TechEmpower/FrameworkBenchmarks that provides representative performance measures across a wide field of static site generators. Since the build time between the different SSGs can vary, we are trying to understand the behavior in different situations. The project presently includes generators on many languages including Go, Python, Java, Ruby, PHP, JavaScript and others.

If you're new to the project, welcome! Please feel free to ask questions here. We encourage new generators and contributors to ask questions. We're here to help!

If you find yourself in a directory or file that you're not sure what the purpose is, checkout our file structure in our documenation, which will briefly explain the use of relevant directories and files.

Quick Start Guide

To get started developing you'll need to install docker or see our Quick Start Guide using vagrant

1. Clone SSGBERK.

    $ git clone --recurse-submodules https://github.com/ssgberk/benchmark-tool.git
   

2. Run a test.

    $ ./ssgberk --test hugo -nf 10
   

   Or run a named suite (smoke, standard, stress, legacy-2019, P, M, G, GG; see Methodology; defined in toolset/benchmark/suites.json). --suite cannot be combined with -nf, -cs or -mr:

    $ ./ssgberk --suite standard
   

Explanation of the ./ssgberk script

The run script is pretty wordy, but each and every flag is required. If you are using windows, either adapt the docker command at the end of the ./ssgberk shell script (replacing ${SCRIPT_ROOT} with /c/path/to/benchmark-tool), or use vagrant.

The command looks like this: docker run -i [-t] --rm --network ssgberk -v /var/run/docker.sock:/var/run/docker.sock -v [FWROOT]:/FrameworkBenchmarks ssgberk/toolset [ARGS]

• -i tells docker to run this in 'interactive' mode; -t (simulate a TTY, so that ctrl+c is propagated) is added by the launcher only when stdout is a terminal.
• --rm tells docker to remove the container as soon as the toolset finishes running, meaning there aren't hundreds of stopped containers lying around.
• --network=ssgberk tells the container to join the 'ssgberk' Docker virtual network
• The first -v specifies which Docker socket path to mount as a volume in the running container. This allows docker commands run inside this container to use the host container's docker to create/run/stop/remove containers.
• The second -v mounts the tools-benchmark source directory (at /FrameworkBenchmarks inside the container) as a volume to share with the container so that rebuilding the toolset image is unnecessary and any changes you make on the host system are available in the running toolset container.
• ssgberk/toolset is the name of toolset container to run

A note on Windows:

• Docker expects Linux-style paths. If you cloned on your C:\ drive, then [ABS PATH TO THIS DIR] would be /c/benchmark-tool.
• Docker for Windows understands /var/run/docker.sock even though that is not a valid path on Windows. Docker Toolbox may not - use at your own risk.

Quick Start Guide (Vagrant)

Get started developing quickly by utilizing vagrant with SSGBERK. Git, Virtualbox and vagrant are required.

1. Clone SSGBERK.

    $ git clone --recurse-submodules https://github.com/ssgberk/benchmark-tool.git
   

2. Change directories

    $ cd tools-benchmark/deployment/vagrant
   

3. Build the vagrant virtual machine

    $ vagrant up
   

4. Run a test

    $ vagrant ssh
    $ ssgberk --test hugo -nf 10
   

Add a New Test

Either on your computer, or once you open an SSH connection to your vagrant box, start the new test initialization wizard.

    vagrant@SSGBERK-all:~/benchmark-tool$ ./ssgberk --new

This will walk you through the entire process of creating a new test to include in the suite.

Measurement notes

• Bundler-dominated JS generators: "app-like" JavaScript generators (Gatsby, Next.js, Astro, VitePress) with few files are dominated by their bundler time. That is their real behavior and is reported as-is, without adjustment.
• Docker Desktop on macOS is noisy: local numbers are for validation only. Publish numbers from native Linux.
• Large scenarios (e.g. -nf 1000000) generate content slowly in the bash build.sh (loop + sponge). This is kept in the minimal port and tracked as a future improvement.

Results

Each run writes to results/<timestamp>/:

• results.json: the raw data (hyperfine mean, stddev, median, min, max and all run times per generator).
• summary.csv: one row per generator with columns framework,language,numberOfFiles,contentSize,minRuns,mean,stddev,median,min,max,status. status is ok, failed or excluded (numbers are empty for the last two). Rows are sorted by mean ascending, then failed rows.
• summary.md: the same table in Markdown (seconds with 3 decimals, — when stddev is null, e.g. a single run), a header with run name, environment, start/completion time and git commit, and a list of failed generators. The table is also printed at the end of the run.

Both summary files are regenerated by --parse.

After a --suite run, results/<timestamp>/ also holds suite.json (cells, generator order, outcome counts), suite-summary.csv/suite-summary.md (all cells in one table) and scaling.csv (scaling exponent per generator). Each cell has its own results.json and summaries under <profile>/nf<nf>-cs<cs>/.

Quality pass

--quality audits the output of the last timed build of each generator, without timing it (spec docs/specs/009-output-quality):

    $ ./ssgberk --test mdbook -nf 50 -cs 5 --quality

With --suite, the pass runs only in the cell nf50-cs5 (suite P, cell 0). The toolset's tests need Python 3.12 (the toolset image and CI use it). It builds the ssgberk/quality image from quality/ once. That image pins Lighthouse, Playwright's Chromium, axe-core, html-validate and lychee. The image runs with no network and with the site copied in, and it audits the index, the first post and 404.html:

• Lighthouse, mobile and desktop presets, median of 3 runs;
• SEO signals (title, lang, description, viewport, canonical, Open Graph, Twitter card, JSON-LD, robots, hreflang, headings, alt, plus robots.txt, sitemap.xml, feed and favicon);
• HTML validity, axe-core violations and broken internal links;
• weight on disk, gzip and brotli;
• JS bytes, and whether the post works without JS;
• files outside the reference page set.

Results go to results/<ts>/quality/<generator>/quality.json, to quality in results.json, to the "Qualidade" section of summary.md and to quality-summary.csv. They have no combined score and no ranking. A failing check is a result and never fails the run. For generators that do not use the SF 005 index layout yet, the audited post is the first file matching output_glob (postSource: "glob" in quality.json).

Content size

-cs is the content size per post, in KB:

-cs	KB per post	Paragraphs per post
0.500 (default)	0.5	1
5	5	10
50	50	100
500	500	1000
1000	1000	2000
5000	5000	10000
10000	10000	20000
100000	100000	200000

Any other value is rejected.

Methodology

Full design: docs/specs/008-benchmark-methodology/ (spec.md, plan.md, tasks.md).

Suites

A suite is a fixed list of cells (number of files nf x content size cs), run in one go with a fixed number of timed runs per cell. They live in toolset/benchmark/suites.json; changing a suite bumps its version, and rankings never mix versions.

    $ ./ssgberk --suite standard --profile core

--suite cannot be combined with -nf, -cs or -mr; --test and --exclude still apply. --cell <index> (0-based, in the order of the table below) runs only that cell of the suite, with the suite's runs, cooldown and timeout; GitHub Actions uses it for one job per cell.

A run exits 1 when any selected generator fails (build or verification failure, nonconformant, timeout, oom, start failure); unsupported is not a failure. A suite exits 1 when any cell fails or the suite aborts. Usage errors (unknown suite, bad --cell, conflicting flags) exit 1, and an interrupted run exits non-zero.

Suite	Cells (nf x cs)	Runs	Cooldown	Timeout	Purpose
smoke	10 x 0.500	1	0 s	30 min	start-up and conformance check; not a speed measurement
standard	100 x 0.500, 100 x 500, 1000 x 0.500, 1000 x 500, 10000 x 0.500	5	15 s	2 h	the default comparison
stress	100000 x 0.500, 10000 x 500, 1000 x 1000, 100 x 5000	3	30 s	4 h	breaking points (timeout, oom); may take a day
legacy-2019	10, 100, 1000, 10000 x 0.500; 10 x 1000; 10 x 5000; 100000 x 0.500	10	15 s	2 h	the 2019 matrix, kept for comparison; unranked
P	50 x 5, 50 x 50	5	15 s	1 h	personal site (50 pages)
M	1000 x 5, 1000 x 50	5	15 s	2 h	corporate site (1000 pages)
G	10000 x 5, 10000 x 50	3	30 s	4 h	large portal (10000 pages)
GG	100000 x 5, 100000 x 50	1	60 s	5 h	massive portal (100000 pages); one run per cell, so no CV

In standard, the 100 x 0.500 cell is dominated by fixed overhead (runtime boot, bundler), 10000 x 0.500 by per-file cost, and the 500 KB cells by markdown parser throughput. Three file counts at 0.500 give the scaling exponent; at cs 500 standard has only two points, so it shows the two medians and no exponent.

Profiles

--profile core (default) benchmarks the Core feature set; --profile extended the Extended one (defined by ssg-frameworks spec 005). They are separate result series stored in separate directories and are never ranked together.

Extended needs ssg-frameworks spec 006: a generator supports it only when its benchmark_config.json declares config[0].profiles.extended. A generator without a profiles key supports only Core; with --profile extended it is not run and is listed as unsupported. Today every generator is Core-only, so an Extended run reports all of them as unsupported.

Resources

Every generator container gets the same limits, so results do not depend on the host size:

Flag	Default	Meaning
--cpus	4	CPUs per container
--memory	8g	memory limit per container; swap is disabled
--cpuset	auto	auto pins to the highest-numbered CPUs and leaves CPU 0 to the host (not applied if the host has no spare CPU); none disables pinning; or a list such as 2-5
--no-cache	off	build generator images without the Docker layer cache

Node generators all run with NODE_OPTIONS=--max-old-space-size=6144 (set in their images), one V8 heap cap inside the memory limit (spec 008, decision 6).

On Docker Desktop these are VM vCPUs, not physical cores. --cpus above the host's CPU count is a usage error (exit 1) before any image is built.

Protocol

• Sequential: one generator at a time. If another SSGBerk run is active, the run aborts unless --allow-concurrent is given. With it, a run that found another active run is marked concurrent (protocol.concurrent) and not ranked; protocol.allowConcurrent records the flag itself.
• Cold rebuild, warm OS cache: every timed run is a full rebuild with the generator's output and cache folders removed first, so generator caches are cold. An untimed verification build runs before the timed runs and loads the generator's binaries, node_modules and gems into the OS page cache, which is not dropped, so the page cache is warm. This verification build is the warm-up; hyperfine --warmup stays 0.
• Order: generators run in a deterministic rotation that shifts by cell, so thermal and background drift does not always hit the same generator. The order is recorded in suite.json.
• Cooldown: a pause between generators (see the table above).
• Noise: if the coefficient of variation (CV) of a generator exceeds 10%, it is re-run once with doubled runs (capped at 10); the last attempt is kept and all attempts are recorded. The result is flagged noisy only when the last attempt's CV is still above 10%. Fewer than 3 runs are never re-run and never get the flag.
• Conformance: until ssg-frameworks spec 006 Task 27 enables --require-conformance by default, the conformance check is not enforced and results carry conformance: unchecked. Pass --require-conformance to treat a run without a passing check as nonconformant.

Metrics

Per generator and cell (in results.json, schemaVersion 2, and the summaries):

• Wall time: median (the headline), with CV (stddev / mean) and min-max range as dispersion.
• CPU: user and system time, and CPU utilization (cores).
• Peak RSS: hyperfine's memory_usage_byte (largest single process).
• Input and output: files and bytes in and out, plus posts per second and input MB per second (from the median).
• Image build time, image id and generator version.
• status: ok, failed, nonconformant, timeout, oom or unsupported.
• Environment: CPU model, Docker version and kernel, host OS, resources and a 12-character fingerprint of machine plus resource configuration.

How to read the rankings

• Rank by median time within a group of identical suite, version, cell, profile, Extended feature set (for extended) and environment fingerprint.
• Ranks are statistical: if a generator's fastest run is not slower than the previous generator's slowest run, they share a rank, printed =n.
• Rows marked with a warning sign are noisy; treat them as indicative only.
• The scaling exponent b is the slope of log(median) over log(nf) at equal cs, from 3 or more points: b near 1 is linear, below 1 means fixed overhead dominates, above 1 is superlinear.
• Never quote one small cell as "the" result: at 100 files the bundler-based JavaScript generators mostly measure their bundler start-up.
• Merged GitHub Actions rounds (one runner per generator) rank only generators that share a runner fingerprint; others appear in separate tables, and generators without a fingerprint are not ranked.

Running a suite on GitHub Actions

    $ gh workflow run benchmark-round.yml -f suite=standard

suite accepts smoke, standard, stress, P, M, G or GG and runs one job per generator and cell (./ssgberk --suite <name> --cell <index>), so each job uses the suite's runs, cooldown and timeout from suites.json and records the suite object. The job timeout is the suite's timeout plus 30 minutes, or the 6 h hosted maximum for suites with 3 or more runs, where a noise re-run can follow the first attempt. A generator whose first attempt plus re-run exceeds 6 h (possible in G and stress) is cancelled by GitHub and is missing from the round. Leave suite empty to use number_of_files, content_size, min_runs and tests (space-separated generator names; empty runs all); such ad-hoc jobs use the default 2 h build timeout, no cooldown and a 6 h job limit. Hosted runners cannot guarantee CPU pinning; use native Linux for published numbers.

• The frameworks submodule must include ssg-frameworks spec 005 (-cs 5 and 50, SSGBERK_INPUT/SSGBERK_OUTPUT); with an older pin, P, M, G and GG fail with unknown content_size and input/output metrics stay empty.
• GG writes about 5 GB of Markdown plus the generated output inside the container; the ~14 GB disk of a hosted ubuntu-24.04 runner may run out.
• Each container gets --cpus 4 by default; a runner with fewer CPUs (e.g. a 2-vCPU private runner) fails every job with the --cpus usage error.

Validity threats

Id	Threat	Handling
T1	Docker Desktop (macOS/Windows) runs a VM with time-shared vCPUs and a virtual disk	fingerprint separates environments; caveat in summaries; reference rounds on native Linux
T2	Thermal throttling and turbo on laptops	rotated order, cooldown, CV re-run, noisy flag
T3	Filesystem caches favour later runs	untimed verification build warms the cache for every generator
T4	Overlay filesystem write cost for large outputs	identical for all generators; storage driver recorded
T5	Bundler-dominated JS generators at small N (Gatsby, Next.js, Astro, VitePress)	full matrix and scaling exponent; caveat at 100 files or fewer
T6	Core and Extended feature sets are not comparable	separate series; rank only within a profile
T7	Peak RSS is the largest process, not the sum over workers	multi-process generators are under-reported; stated in the summary caveat
T8	Background load or concurrent benchmarks	concurrency guard, --cpuset auto away from CPU 0
T9	Memory limit kills generators that hold all content in memory	reported as oom, same limit for everyone
T10	Image and dependency drift between rounds	image ids, base image digest and generator versions recorded
T11	Unavoidable generator-specific work (e.g. VitePress Shiki)	documented in the generator README and counted as its real cost

Resources

Official Documentation

Our official documentation can be found at GitHub Wiki.

Live Results

Results of continuous benchmarking runs are available in real time here (coming soon, after v1 release).

Contributing

The community has consistently helped in making these tests better, and we welcome any and all changes. Reviewing our contribution practices and guidelines will help to keep us all on the same page. The contribution guide can be found in the SSGBERK documentation.

Português do Brasil

Bem-vindo ao [Static Site Generator Benchmarks (SSGBerk)]

O que é um Static Site Generator (SSG)?

Uma página web estática (às vezes chamada de página flat ou stationary) é uma página entregue ao usuário exatamente como foi armazenada, ao contrário das páginas web dinâmicas, que são geradas por uma aplicação web.

O que é o SSGBerk?

Um projeto derivado (fork) do TechEmpower/FrameworkBenchmarks que fornece medições de desempenho representativas em um amplo conjunto de geradores de sites estáticos. Como o tempo de build varia bastante entre os diferentes SSGs, buscamos entender o comportamento em diferentes situações. Atualmente o projeto inclui geradores em várias linguagens, entre elas Go, Python, Java, Ruby, PHP, JavaScript e outras.

Se você é novo no projeto, seja bem-vindo! Fique à vontade para tirar dúvidas aqui. Incentivamos novos geradores e colaboradores a perguntar. Estamos aqui para ajudar!

Se você estiver em um diretório ou arquivo cuja finalidade não ficou clara, consulte a estrutura de arquivos na nossa documentação, que explica brevemente o uso dos principais diretórios e arquivos.

Guia de Início Rápido

Para começar a desenvolver, você precisa instalar o Docker ou seguir o nosso Guia de Início Rápido com Vagrant.

1. Clone o SSGBERK.

    $ git clone --recurse-submodules https://github.com/ssgberk/benchmark-tool.git
   

2. Execute um teste.

    $ ./ssgberk --test hugo -nf 10
   

Explicação do script ./ssgberk

O script de execução é bastante verboso, mas cada flag é obrigatória. Se você usa Windows, adapte o comando docker no final do script ./ssgberk (substituindo ${SCRIPT_ROOT} por /c/path/to/benchmark-tool) ou use Vagrant.

O comando tem esta forma: docker run -i [-t] --rm --network ssgberk -v /var/run/docker.sock:/var/run/docker.sock -v [FWROOT]:/FrameworkBenchmarks ssgberk/toolset [ARGS]

• -i diz ao Docker para executar em modo 'interativo'; -t (simula um TTY, para que o ctrl+c seja propagado) só é adicionado pelo launcher quando a saída padrão é um terminal.
• --rm diz ao Docker para remover o container assim que o toolset terminar de executar, evitando centenas de containers parados acumulados.
• --network=ssgberk diz ao container para entrar na rede virtual 'ssgberk' do Docker.
• O primeiro -v define qual caminho do socket do Docker será montado como volume no container em execução. Isso permite que os comandos docker executados dentro deste container usem o Docker do host para criar, executar, parar e remover containers.
• O segundo -v monta o diretório do código-fonte do tools-benchmark (em /FrameworkBenchmarks dentro do container) como volume compartilhado com o container. Assim, não é preciso reconstruir a imagem do toolset, e qualquer alteração feita no host fica disponível no container do toolset em execução.
• ssgberk/toolset é o nome do container do toolset a ser executado.

Uma nota sobre o Windows:

• O Docker espera caminhos no estilo Linux. Se você clonou o repositório no drive C:\, então [ABS PATH TO THIS DIR] seria /c/benchmark-tool.
• O Docker para Windows entende /var/run/docker.sock, mesmo que esse não seja um caminho válido no Windows. O Docker Toolbox pode não entender; use por sua conta e risco.

Guia de Início Rápido (Vagrant)

Comece a desenvolver rapidamente usando Vagrant com o SSGBERK. São necessários o Git, o Virtualbox e o Vagrant.

1. Clone o SSGBERK.

    $ git clone --recurse-submodules https://github.com/ssgberk/benchmark-tool.git
   

2. Entre no diretório

    $ cd tools-benchmark/deployment/vagrant
   

3. Crie a máquina virtual do Vagrant

    $ vagrant up
   

4. Execute um teste

    $ vagrant ssh
    $ ssgberk --test hugo -nf 10
   

Adicionar um Novo Teste

No seu computador, ou depois de abrir uma conexão SSH com a sua box do Vagrant, inicie o assistente de criação de novo teste.

    vagrant@SSGBERK-all:~/benchmark-tool$ ./ssgberk --new

Ele guia você por todo o processo de criação de um novo teste para incluir na suíte.

Notas de medição

• Geradores JS dominados pelo bundler: geradores JavaScript "app-like" (Gatsby, Next.js, Astro, VitePress) com poucos arquivos têm o tempo dominado pelo bundler. Esse é o comportamento real deles e é reportado como está, sem ajuste.
• O Docker Desktop no macOS é ruidoso: os números locais servem apenas para validação. Publique números de Linux nativo.
• Cenários grandes (por exemplo -nf 1000000) geram conteúdo lentamente no build.sh em bash (loop + sponge). Isso foi mantido no port mínimo e está registrado como melhoria futura.

Tamanho do conteúdo

-cs é o tamanho do conteúdo por post, em KB:

-cs	KB por post	Parágrafos por post
0.500 (padrão)	0,5	1
500	500	1000
1000	1000	2000
5000	5000	10000
10000	10000	20000
100000	100000	200000

Qualquer outro valor é rejeitado.

Resultados

Depende da feature 006 (resumo dos resultados); esta subseção espelha a "Results" da seção em inglês.

Ao final de uma execução, o diretório results/<timestamp>/ contém:

• results.json: os dados brutos de cada gerador;
• summary.csv: o resumo em CSV;
• summary.md: o mesmo resumo em Markdown.

O resumo tem uma linha por gerador, com média, desvio padrão, mediana, mínimo e máximo, todos em segundos.

Recursos

Documentação oficial

Nossa documentação oficial está disponível na Wiki do GitHub.

Resultados ao vivo

Os resultados das execuções contínuas de benchmark estarão disponíveis em tempo real aqui (em breve, após o lançamento da v1).

Contribuindo

A comunidade tem ajudado de forma consistente a melhorar estes testes, e aceitamos qualquer mudança. Conhecer nossas práticas e diretrizes de contribuição ajuda a manter todos na mesma página. O guia de contribuição pode ser encontrado na documentação do SSGBERK.