Skip to content

Latest commit

 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Google Trends 429: a session-per-keyword client

Minimal, dependency-light clients (Python and Node) for pulling data from Google Trends' current widget protocol, built to survive the rate limiting that trips up most hand-rolled scrapers.

The two 429s

Two different 429s show up in this flow, and only one of them means "you are rate limited."

The first is on GET https://trends.google.com/trends/explore. This is a page load, not an API call. A fresh cookie jar gets a 429 here almost every time : and the response still sets the NID cookie, which is what the JSON endpoints check for. This 429 is the handshake, not a rejection: read the Set-Cookie header, keep the cookie, move on.

The second shows up on the real API surface, trends.google.com/trends/api/*. This is the actual limit: too many calls, too fast, from one identity (cookie jar + IP + TLS/header fingerprint together). This is the 429 worth reacting to : rotate to a fresh session and back off, rather than retrying the one that just got limited.

Why one keyword costs several requests

There's no single "give me the data for this keyword" endpoint. The flow is two steps:

  1. GET /trends/api/explore with the keyword, geo, and time range. This doesn't return data : it returns a list of widgets (TIMESERIES, RELATED_QUERIES, RELATED_TOPICS, GEO_MAP, ...), each with a short-lived token and its own request payload.
  2. One GET per widget, against that widget's own endpoint (/widgetdata/multiline for the time series, /widgetdata/relatedsearches for related queries and topics), passing that widget's token back.

So one keyword with time series and related queries enabled is a minimum of three HTTP calls (handshake, explore, one widget) before any retries; every extra widget you fetch is one more call.

The widget protocol, minimally

GET /trends/explore?q=trends&hl=en-US          -> 429 (expected; sets NID cookie)
GET /trends/api/explore?hl=en-US&tz=0&req=...  -> 200, )]}' + JSON: { widgets: [...] }
GET /trends/api/widgetdata/multiline?...&token=...  -> 200, )]}' + JSON: { default: { timelineData: [...] } }

Every JSON response is prefixed with )]}' (XSSI protection) : strip that before parsing. The req parameter on both the explore and widget calls is a JSON-encoded string inside the query string, not a JSON body: these are GET requests. A widget's token is single-use for that widget on that session; if the session rotates, you need a fresh explore call too.

Both clients here (python/trends_client.py, node/trends-client.mjs) give every keyword its own session from the start : never reusing one session across keywords : pace requests, and on a real 429 discard the session (cookie jar and all) and retry from a clean one with exponential backoff.

from trends_client import TrendsClient

client = TrendsClient()
rows = client.interest_over_time("python")
print(len(rows), "points")
import { TrendsClient } from './trends-client.mjs';

const client = new TrendsClient();
const rows = await client.interestOverTime('python');
console.log(rows.length, 'points');

Related queries: the empty-list reality

Pulling RELATED_QUERIES or RELATED_TOPICS from a cookie-only session (no signed-in account, no browsing history tied to the cookie) often returns an empty rankedList for a meaningful share of terms. This isn't a bug in a particular library : pat310/google-trends-api issue #174 documents the same relatedQueries emptiness independently. It's a valid 200 with an empty list, not a 429, and no amount of retrying or proxy rotation changes it.

pytrends, still the most commonly referenced Python wrapper for this API, is archived (confirmed via the GitHub API: archived: true, last pushed 2024-08-10) : its token protocol predates the current widget flow, which is part of why a small hand-rolled client against the current endpoints is worth having.

Tested

Live-run numbers, one keyword (python), worldwide, 12 months, interest over time only, no proxy, tested 2026-09-10 20:18 CDT:

  • Python (trends_client.py): 3 requests (1 handshake 429, explore 200, widget 200), 0 real 429s, 53 interest-over-time points returned.
  • Node (trends-client.mjs): 3 requests (1 handshake 429, explore 200, widget 200), 0 real 429s, 53 interest-over-time points returned.

Broader reliability numbers below are from our own actor's soak log (Central time), which runs this same session-per-keyword protocol at higher volume:

  • 2026-09-09, one session reused across a batch of calls (the older, higher-429 approach): 28 requests, 7 real 429s (plus 9 handshake 429s), all recovered by session rotation, 3 of 3 keywords returned data.
  • 2026-09-09, fresh session per keyword with ~500ms jittered pacing, run 1: 15 requests, 0 real 429s (3 handshake), 0 retries, 3 of 3 keywords returned data.
  • 2026-09-09, same configuration, run 2 (separate run): 15 requests, 0 real 429s (3 handshake), 0 retries, 3 of 3 keywords returned data.
  • 2026-09-09 19:05 CDT, four runs concurrently, 8 keywords each (32 total): 32 of 32 keywords returned data, 4 of 4 runs succeeded, 160 total requests (4 runs x 40), 0 real 429s (32 handshakes), 0 retries.

Limitations

  • No proxy rotation built in : these clients change cookie jar and TLS session on rotation but keep your IP. At volume, an IP-level rate limit will still bite; add a proxy layer if you need to run many keywords.
  • Google Trends values are relative (0-100 to that term's own peak), not absolute search volume.
  • Related queries/topics can change or disappear with no notice; don't build on a specific query showing up.

Hosted version

If you'd rather not run and maintain this yourself: I run a hosted version of this same session-per-keyword protocol on Apify, pay-per-result at $0.25 per 1,000 rows, no subscription : interest over time, interest by region, and related queries as flat CSV/JSON rows. https://apify.com/tallyrake/google-trends-scraper

License

MIT : see LICENSE.

About

Session-per-keyword Google Trends client that survives 429s; Python and Node

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages