@riavzon/bot-detector is an express middleware that checks incoming requests through a two phase pipeline of 17 checkers across ip reputation, geolocation, tls fingerprinting, behavioral rate limiting, Tor analysis and more.
Each checker contributes a penalty score toward a configurable ban threshold. Requests that cross the threshold receive a 403 response, or are banned at the firewall level if configured.
@riavzon/bot-detector uses Shield-Base to fetch and compile its data sources into fast in memory databases. Checkers query these compiled databases synchronously, which allows the whole pipeline to make a decisions in milliseconds.
- Comes with 17 fully configurable server checkers
- Extensible, you can easily provide your own custom checkers via
CheckerRegistryand custom data sources. - Self optimized, uses collected visitor data to become smarter and faster over time. Instead of running the full pipeline for known offenders, it compiles your latest database rows into local mmdb files to instantly drop past threats and high risk visitors.
- Fast, around 1.2ms median latency for the full pipeline.
- Supports multiple storages and databases sql-lite/pg/mysql /redis/lru/memory
- Comes with a cli to manage data sources and generate custom threat databases.
- Supports cjs and fully typed
- Node.js 18 or later
- Express 5
- A supported database for visitor persistence
The fastest way to get started is with the create package. Run this in the root of your Express project:
npx @riavzon/bot-detector-createThis single command installs all dependencies, downloads and compiles every
threat intelligence feed, writes a fully annotated botDetectorConfig.ts with
all 17 checkers at their defaults, a mainBotDetector.ts ready-to-run Express
entry point, and creates the database tables, all without any manual steps. See the
@riavzon/bot-detector-create
package for details.
It defaults to better-sqlite3 as the database driver.
If you prefer to wire things up yourself:
npm install @riavzon/bot-detector express cookie-parser <data-base-driver>After installation, run bot-detector init to download its data sources and validate that mmdbctl is installed, if not it prompts you about it, and installs it automatically, it also ask you to provide an user agent that will be used to fetch BGP data from bgp.tools as they requires it before they allow you to use their data, more info at BGP.tools.
To skip the interactive setup you can download the mmdbctl dependency directly and provide the contact user agent with a flag:
npx @riavzon/bot-detector init
# OR
npx @riavzon/bot-detector init --contact=App - contact@example.com
The compiled databases are written to _data-sources/ inside the package directory, which include the following files:
├── asn.mmdb
├── banned.mmdb // generated on demand from your visitors history data
├── city.mmdb
├── country.mmdb
├── firehol_anonymous.mmdb
├── firehol_l1.mmdb
├── firehol_l2.mmdb
├── firehol_l3.mmdb
├── firehol_l4.mmdb
├── goodBots.mmdb
├── highRisk.mmdb // generated on demand from your visitors history data
├── proxy.mmdb
├── suffix.json
├── tor.mmdb
└── useragent-db
├── useragent.mdb
└── useragent.mdb-lock
More information about each database and its source can be found in Shield-Base readme.
These databases are read only, your can interact with them with getDataSources, for example:
import { getDataSources } from '@riavzon/bot-detector';
const ds = getDataSources();
// ip lookups — all return null if the ip is not in the database
ds.asnDataBase(ip); // BGP/ASN record: asn_id, asn_name, classification, hits
ds.cityDataBase(ip); // city-level geo: city, region, country, lat/lon, timezone
ds.countryDataBase(ip); // country-level geo: country, countryCode, isp, org, proxy, hosting
ds.torDataBase(ip); // Tor relay record: flags, exit_addresses, version, probabilities
ds.proxyDataBase(ip); // proxy record: type, sources that flagged this ip
ds.goodBotsDataBase(ip); // known good crawler record (Googlebot, Bingbot, etc.)
ds.fireholAnonDataBase(ip); // Firehol anonymous feed match
ds.fireholLvl1DataBase(ip); // Firehol threat level 1 (most severe)
ds.fireholLvl2DataBase(ip); // Firehol threat level 2
ds.fireholLvl3DataBase(ip); // Firehol threat level 3
ds.fireholLvl4DataBase(ip); // Firehol threat level 4
ds.bannedDataBase(ip); // your banned.mmdb, generated by `bot-detector generate`
ds.highRiskDataBase(ip); // your highRisk.mmdb, generated by `bot-detector generate`
// LMDB key value stores
ds.getUserAgentLmdb().get(uaString); // user agent pattern recorddefineConfiguration is async and must resolve before you attach detectBots to your routes. Call it exactly once at startup before app.listen.
import express from 'express';
import cookieParser from 'cookie-parser';
import { defineConfiguration, detectBots } from '@riavzon/bot-detector';
const app = express();
app.use(cookieParser());
await defineConfiguration({
store: {
main: { driver: 'mysql-pool', host: 'localhost', user: 'root', database: 'mydb' },
},
});
app.use(detectBots());
app.get('/', (req, res) => {
res.json({ banned: req.botDetection?.banned });
});Once your app has a defineConfiguration call wired up, run load-schema to create the database tables:
npx @riavzon/bot-detector load-schemadefineConfiguration accepts a configuration object. Every field has a default value, only store.main is required. The full schema with all defaults is defined in src/botDetector/types/configSchema.ts.
await defineConfiguration({
// Required: database connection for visitor persistence
store: {
main: { driver: 'mysql-pool', host: 'localhost', user: 'root', database: 'mydb' },
},
// Score required to ban a visitor (0–100). Default: 100
banScore: 100,
// Maximum score assignable per request (0–100). Default: 100
maxScore: 100,
// Points the reputation healer restores per clean request. Default: 10
restoredReputationPoints: 10,
// Score persistence strategy. See "Score modes" below. Default: false
setNewComputedScore: false,
// IPs that bypass all detection. Accepts IPv4, IPv6, or CIDR strings.
whiteList: ['127.0.0.1', '::1'],
// Recheck interval for returning visitors. Default: check every request
checksTimeRateControl: {
checkEveryRequest: false,
checkEvery: 1000 * 60 * 5, // ms
},
// Async write queue that persists visitor scores without blocking requests
batchQueue: {
flushIntervalMs: 5000,
maxBufferSize: 100,
maxRetries: 3,
},
// Cache driver for visitor state, behavioral data, and sessions.
// Defaults to memory when omitted.
storage: { driver: 'redis', host: 'localhost', port: 6379 },
// Whether to issue a UFW firewall ban in addition to a 403 response. Default: false
punishmentType: {
enableFireWallBan: false,
},
// Pino log level. Default: 'info'
logLevel: 'info',
// Individual checker configuration. All checkers are enabled by default.
// See "Checker reference" for available penalty options per checker.
checkers: {
enableBehaviorRateCheck: {
enable: true,
behavioral_window: 60_000, // window duration in ms
behavioral_threshold: 30, // max requests per window before penalty applies
penalties: 60,
},
honeypot: {
enable: true,
paths: ['/admin', '/.env', '/wp-login.php'],
},
enableGeoChecks: {
enable: true,
bannedCountries: ['KP', 'IR'], // ISO 3166-1 alpha-2 codes
},
// ...other checkers
},
// Controls custom MMDB generation from your visitor data. See `bot-detector generate`.
generator: {
scoreThreshold: 70, // minimum suspicious_activity_score to include in highRisk.mmdb
generateTypes: false, // generate typescript types
deleteAfterBuild: false, // delete source rows after compiling
mmdbctlPath: 'mmdbctl', // path to mmdbctl binary
},
});setNewComputedScore controls how the bot score is written to the database on each request.
false (default) snapshot then heal.: The detector writes the computed score once on the visitor's first request. The reputation healer then decrements it on each subsequent clean visit. The score only decreases until the cache expires and a new snapshot is taken.
true live snapshot.: The detector overwrites the stored score on every request, then the healer immediately decrements it. Use this when you want the database to always reflect the latest computed risk.
The store.main field accepts the following drivers:
| Driver | Value | Notes |
|---|---|---|
| MySQL (pool) | mysql-pool |
Peer dependency: mysql2 >=3 |
| PostgreSQL | postgresql |
Requires pg |
| SQLite | sqlite |
Requires better-sqlite3 |
| Cloudflare D1 | cloudflare-d1 |
Pass binding from the Worker environment |
| PlanetScale | planetscale |
Pass host, username, password |
// MySQL pool
{ driver: 'mysql-pool', host: 'localhost', user: 'root', password: 'secret', database: 'mydb' }
// PostgreSQL
{ driver: 'postgresql', connectionString: 'postgres://user:pass@localhost/mydb' }
// SQLite
{ driver: 'sqlite', name: './bot-detector.db' }The storage field configures where visitor state, behavioral rate data, and session records are stored between requests. When omitted, the package uses memory.
| Driver | Value | Notes |
|---|---|---|
| memory (default) | (omit storage) |
Single-process only |
| LRU cache | lru |
In-process LRU; configure max and ttl |
| Redis | redis |
Shared across instances; requires ioredis |
| Upstash Redis | upstash |
Serverless Redis via HTTP |
| Filesystem | fs |
Persistent local storage for development |
| Cloudflare KV (binding) | cloudflare-kv-binding |
Pass binding |
| Cloudflare KV (HTTP) | cloudflare-kv-http |
Pass accountId, namespaceId, apiToken |
| Cloudflare R2 | cloudflare-r2-binding |
Pass binding |
| Vercel | vercel |
Vercel Runtime Cache |
All 17 checkers are enabled with sensible defaults. To disable a checker, pass { enable: false } for its config key. To adjust penalties, pass { enable: true, penalties: { ... } } with the values you want to override.
| Checker | Config key | Phase | What it detects |
|---|---|---|---|
| ip validation | enableIpChecks |
cheap | Invalid or unresolvable client ip |
| Known good bots | enableGoodBotsChecks |
cheap | Legitimate crawlers (Googlebot, Bingbot, etc.) |
| Browser and device | enableBrowserAndDeviceChecks |
cheap | CLI/library user agent types, Internet Explorer, impossible browser and OS combinations |
| Locale consistency | localeMapsCheck |
cheap | Mismatch between Accept-Language header and geo locale |
| FireHOL threat feeds | enableKnownThreatsDetections |
cheap | IPs in FireHOL levels 1–4 and the anonymizer feed |
| ASN classification | enableAsnClassification |
cheap | Hosting and content ASNs with low route visibility |
| Tor node analysis | enableTorAnalysis |
cheap | Exit nodes, guard nodes, bad exits, and obsolete Tor versions |
| Timezone consistency | enableTimezoneConsistency |
cheap | Mismatch between declared timezone and geo timezone |
| Honeypot paths | honeypot |
cheap | Requests to configured trap URLs |
| Known bad IPs | enableKnownBadIpsCheck |
cheap | IPs in your custom highRisk.mmdb |
| Behavioral rate | enableBehaviorRateCheck |
heavy | Request count exceeding the configured threshold within the window |
| Proxy / ISP / cookie | enableProxyIspCookiesChecks |
heavy | Proxy and VPN detection, missing canary cookie, unknown ISP or org |
| user agent and headers | enableUaAndHeaderChecks |
heavy | Headless browsers, short user agents,tls fingerprint mismatch, header anomalies |
| Geo location | enableGeoChecks |
heavy | Missing geo fields, banned countries |
| Session coherence | enableSessionCoherence |
heavy | Referer mismatches and cross-site navigation inconsistencies |
| Velocity fingerprint | enableVelocityFingerprint |
heavy | Unnaturally consistent inter-request timing |
| Bad user agent list | knownBadUserAgents |
heavy | user agents matching the LMDB pattern library (critical → low severity) |
The pipeline runs in two phases to keep latency low.
Cheap phase runs on every request. All lookups are synchronous, in memory reads from MMDB or LMDB dbs. When the accumulated score reaches banScore during this phase, the middleware rejects the request and skips the heavy phase entirely.
Heavy phase runs only when the cheap-phase score stays below banScore. These checkers read from the visitor cache or perform async operations.
On every request that passes detection, the middleware populates req.botDetection:
req.botDetection: {
success: boolean,
banned: boolean,
time: string, // ISO timestamp
ipAddress: string
}You can add your own checkers to the pipeline without modifying any package files. Each checker is a class that implements IBotChecker and registers itself via CheckerRegistry.register(). The middleware picks it up automatically.
See CUSTOM.md for the full guide, which covers:
- The
IBotCheckerinterface and phase selection - All fields available on
ValidationContext(geo, Tor, ASN, parsed user agent, proxy, threat level, cookies, and more) - Typed custom context via
buildCustomContextfor full IntelliSense onctx.custom - Triggering an immediate ban via the
BAD_BOT_DETECTEDreason code - Writing async checkers with your own cache
For example, you may use a client side detection tools that collects data, and then can be send to your custom checker for analysis:
// types/clientSignals.ts
export interface ClientSignals {
hasWebDriver: boolean;
screenResolution: string | null;
touchPoints: number;
}
// server.ts
import { detectBots } from '@riavzon/bot-detector';
import type { ClientSignals } from './types/clientSignals.js';
app.use(
detectBots<ClientSignals>((req) => {
try {
return JSON.parse(req.headers['x-client-signals'] as string);
} catch {
return { hasWebDriver: false, screenResolution: null, touchPoints: 0 };
}
// or use ctx.req in ur checker directly
})
);
// checkers/clientSideChecker.ts
import { CheckerRegistry, getDataSources, getStorage } from '@riavzon/bot-detector';
import type { IBotChecker, ValidationContext, BotDetectorConfig, BanReasonCode } from '@riavzon/bot-detector';
import type { ClientSignals } from '../types/clientSignals.js';
class ClientSideChecker implements IBotChecker<BanReasonCode, ClientSignals> {
name = 'client-side-signals';
phase = 'cheap' as const;
isEnabled(_config: BotDetectorConfig) {
return true;
}
async run(ctx: ValidationContext<ClientSignals>, _config: BotDetectorConfig) {
const reasons: BanReasonCode[] = [];
let score = 0;
if (ctx.custom.hasWebDriver) {
reasons.push('BAD_BOT_DETECTED'); // immediate ban, no score needed
return { score, reasons };
}
const cached = await getStorage().getItem<number>(`client-signals:${ctx.ipAddress}`);
if (cached !== null) {
return { score: cached, reasons: cached > 0 ? (['BAD_BOT_DETECTED'] as BanReasonCode[]) : [] };
}
if (ctx.custom.screenResolution === null) score += 20;
if (ctx.tor.exit_addresses) score += 30;
// Combine stuff
if (ctx.tor.running && !ctx.touchPoints) {
score += 40;
}
await getStorage().setItem(`client-signals:${ctx.ipAddress}`, score, { ttl: 60 * 5 });
return { score, reasons };
}
}
CheckerRegistry.register(new ClientSideChecker());The package ships a cli with three subcommands.
Runs the installation wizard. Verifies that mmdbctl is installed (and installs it if not), prompts for a BGP.tools contact string, then compiles all data sources in parallel:
- BGP and ASN data
- City and geography databases
- Tor node lists
- Proxy and anonymizer lists
- Threat levels 1-4 and the anonymous feed
- Verified crawler ip ranges (Googlebot, Bingbot, Apple, Meta, etc.)
- user agent pattern (
useragent.mdb)
The compiled databases are written to _data-sources/ inside the package directory.
In non interactive environments, init skips silently if the databases already exist. If they do not exist, it prints a warning and exits without failing.
npx bot-detector initRedownloads and recompiles all data sources that the module uses, using the cached configuration. Requires init to have been run at least once.
npx bot-detector refreshRun this at least ones every 24h. More info Shield-Base readme
Reads your database and compiles two custom mmdb files:
banned.mmdb: built from all rows in thebannedtable with a non null ip addresshighRisk.mmdb: built fromvisitorsrows wheresuspicious_activity_score >= generator.scoreThreshold
Requires mmdbctl. If the path in generator.mmdbctlPath cannot be resolved, the command prompts to install it and exits with instructions.
npx bot-detector generateRun this periodically or after bulk ban operations.
Initializes the middleware. Opens all mmdb and lmdb databases, starts the batch write queue, and sets up the cache and database connection. Call it once before attaching detectBots to your app.
Returns an Express RequestHandler. Always call it as a factory, use detectBots(). The optional buildCustomContext function runs once per request before any checker executes and populates ctx.custom with typed data you define.
app.use(
detectBots<MyContext>((req) => ({
userId: req.user?.id ?? 'anonymous',
plan: req.user?.plan ?? 'free',
}))
);An Express Router that mounts detectBots() at /check and returns { results: req.botDetection, message: 'Fingerprint logged successfully' }.
import { ApiResponse } from '@riavzon/bot-detector';
app.use('/bot', ApiResponse); // POST /bot/checkReturns the initialized DataSources instance. Throws if called before defineConfiguration resolves.
Returns the initialized Storage instance. Throws if called before defineConfiguration resolves.
Returns the initialized BatchQueue instance used for deferred database writes. Throws if called before defineConfiguration resolves.
Programmatic equivalent of bot-detector generate. Compiles banned.mmdb and highRisk.mmdb from your database. If generator.deleteAfterBuild is true, source rows are deleted after each successful compile.
Issues a ufw firewall rule (sudo ufw insert 1 deny from <ip>) to block the ip at the OS level. Only runs when punishmentType.enableFireWallBan is true, returns immediately otherwise. Requires the Node.js process to have passwordless sudo access to ufw.
Parses a user agent string and returns a ParsedUAResult with browser name, version, OS, device type, vendor, and model.
Returns the full GeoResponse for any ip address using the mmdb databases. Useful for geo lookups outside the middleware context.
Updates the is_bot column in the visitors table for the given canary_id.
Upserts a row into the banned table with the visitor's canary cookie, ip address, country, user agent, ban reasons, and score.
Warms the database connection pool by running parallel SELECT 1 queries, then fires a dummy visitor query to prime the query plan cache. Call this after defineConfiguration resolves but before the server starts accepting traffic.
Updates the full fingerprint record in the visitors table for a given canary and visitor id pair. Returns { success: boolean, reason?: string }.
Registry for custom bot checker plugins. Use CheckerRegistry.register(checker) to add a checker that implements IBotChecker. Checkers are partitioned into cheap and heavy phases and filtered by your config at runtime.
Error subclasses thrown (or catchable) when a checker conclusively identifies a bad or good bot. Re-exported from helpers/exceptions for use in custom checkers and error-handling middleware.
A dedicated documentation site is coming soon.
Apache-2.0