Skip to content

Placeholder refresh (2026-11-20) and the fields its query versions return - #147

Merged
fa0311 merged 3 commits into
mainfrom
chore/placeholder-2026-11-20
Nov 20, 2026
Merged

fa0311 merged 3 commits into
mainfrom
chore/placeholder-2026-11-20

Conversation

@fa0311

@fa0311 fa0311 commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Three commits, each with its dist/. The first is the recording, the second is what the recorded query versions need, the third is independent of the queryIds and can be dropped on its own.

placeholder: queryIds and features recorded on 2026-11-20

Observed

  • Recorded with twitter-api-safe-dump and the sandbox account in Google Chrome: the dump's 37 pages plus 16 pages that are only viewed (own profile, community members / moderators / hashtags, Community Notes, /X/affiliates, /elonmusk/about, analytics, delegate, premium, a space). 704 GraphQL requests of 88 operations; of all API answers 1,936 were 200, seven 403 (UsersByRestIds, as in any browser) and nine 429 at the end of the session (ViewerBadgeCounts, CreatorStudioTabBarItemQuery, sent on every page load).
  • The same session was logged from the network side. Every one of the 88 catalog lines has a request on the wire with the same queryId, variables, features and fieldToggles, and no operation used two queryIds or two feature sets within the session.
  • tools/update_placeholder.py changed 42 entries:
    • 36 get a new queryId and post_ctas_fetch_enabled: true (was false): HomeTimeline, HomeLatestTimeline, TweetDetail, SearchTimeline, Likes, Bookmarks, BookmarkSearchTimeline, NotificationsTimeline, Retweeters, Followers, Following, BlueVerifiedFollowers, FollowersYouKnow, UserOriginalsTimeline, UserRepliesTimeline, UserRepostsTimeline, UserVideoTimeline, UserHighlightsTweets, UserSuperFollowTweets, ListLatestTweetsTimeline, ListMembers, ListSubscribers, ListSearchTimeline, ListsManagementPageTimeline, CombinedLists, CommunityTweetsTimeline, CommunityMediaTimeline, CommunityAboutTimeline, CommunityDiscoveryTimeline, GlobalCommunitiesPostSearchTimeline, GlobalCommunitiesLatestPostSearchTimeline, ExplorePage, ExploreSidebar, GenericTimelineById, ConnectTabTimeline, BirdwatchFetchGlobalTimeline.
    • TweetResultByRestId and AudioSpaceById keep their queryId and get the flag.
    • CommunitiesRankedTimeline and CommunitiesExploreTimeline get a new queryId, the flag, x_elections_post_context_enabled: true and responsive_web_made_on_x_label_enabled: false.
    • ViewerBadgeCounts and AboutAccountQuery get a new queryId only.
  • 34 more entries were sent and are not touched: 33 as the placeholder has them, and membersSliceTimeline_Query (below). 41 were not sent by any of the 53 pages and stay as they are: the 14 mutations, UserByRestId, UserTweets, UserTweetsAndReplies, UserMedia, Favoriters, BroadcastQuery, ListOwnerships, ListCreationRecommendedUsers, ListEditRecommendedUsers, DmAllSearchSlice, BirdwatchFetchContributorNotesSlice, CommunityHashtagsTimeline, affiliatesQuery, DelegatedAccountListQuery, useRelayDelegateDataPendingQuery, PremiumContentQuery, useSubscriptionProductDetailsQuery and the analytics / own-profile queries (useFetchAnalyticsQuery, QuickPromoteEligibility, TweetActivityQuery, isEligibleForAnalyticsUpsellQuery, useTotalAdCampaignsForUserQuery, UserPreferences, BlueVerifiedProfileEditCalloutQuery, useFetchProfileBlocks_profileExistsQuery, useFetchProfileSections_*).
  • Profile pages sent UserOriginalsTimeline, UserRepliesTimeline, UserRepostsTimeline and UserVideoTimeline (the first two for the own profile as well) and none of UserTweets, UserTweetsAndReplies, UserMedia; /X/affiliates sent UserBusinessProfileTeamTimeline, not affiliatesQuery.

Not verified

  • What the web client sends today for the 41 entries above. Eleven of them still carry post_ctas_fetch_enabled: false (CreateTweet, Favoriters, UserTweets, UserTweetsAndReplies, UserMedia, BroadcastQuery, ListOwnerships, ListCreationRecommendedUsers, ListEditRecommendedUsers, CommunityHashtagsTimeline, BirdwatchFetchContributorNotesSlice), while every request of this session that carries the flag sends true.
  • membersSliceTimeline_Query was sent four times without features; the entry keeps responsive_web_graphql_timeline_navigation_enabled, because the tool only overwrites what a request carries. ProfileSpotlightsQuery has features: {} and was sent without the parameter.
  • The HomeTimeline GET of the web client carries no seenTweetIds; the entry keeps the variable (variables are never overwritten).

spec: fields of the query versions recorded on 2026-11-20

Each field was compared on the same day and the same target, once with the queryId of main and once with the recorded one (single GETs).

field queryId of main recorded queryId
BadgeCounts.grok_bot_primary_unread_count, grok_bot_unread_conversations (integer) absent (1 GET) present in 79 recorded answers and 1 GET
AboutProfile.created_country_accurate (boolean) absent (elonmusk, X, NASA) true for elonmusk, X, SpaceX, Tesla; absent for NASA, which has no account_based_in either
AboutAccountVerificationInfo.is_identity_verified (boolean) absent (elonmusk, X, NASA) false for all five accounts
AdMetadataContainer.allowSpineClicks (boolean) absent on 7 ads of one HomeTimeline answer present on all 70 ads of the 10 recorded HomeTimeline / HomeLatestTimeline answers and on all 7 of 1 GET

All four are optional: answers of the previous versions are part of the stored set and have to keep validating. is_identity_verified and allowSpineClicks: false do not fail the strict comparison (it accepts false where the model has nothing); they were found by listing the keys the model does not have.

spec: shapes of other accounts, posts and communities in the recording

None of these depends on the queryId.

  • AboutProfile: account_based_in, affiliate_username and source are optional. With both queryIds NASA has none of the three and X has no affiliate_username; only elonmusk, the placeholder's sample, has all of them.
  • ProfileBio.with_held_scope (string, user): on one account of the People search for twitter, next to withheld_in_countries, with both queryIds. The answers stored on 2026-11-17 have nine bios with withheld_in_countries and none with this field.
  • Tweet.conversation_control_details ({mode}): on the sandbox account's own posts with a reply restriction, four posts, mode equal to legacy.conversation_control.policy (Community, Verified), with both queryIds. Two posts of other accounts with a restriction (ByInvitation, Co) do not have it. mode is a plain string: two values are not a closed set.
  • Community page (CommunitiesFetchOneQuery, queryId unchanged, 19 answers for 16 communities in the recording): question on 13 communities, primary_community_topic on 12 (the shared PrimaryCommunityTopic), a CommunityJoinActionUnavailable with message and reason on 6, no custom_banner_media on 1. The placeholder's community has none of these.

Not typed: one of the 16 communities has an admin whose affiliate label mentions an account; on this page the mention carries id, __typename and __isTimelineReferenceObject. One answer, and it needs a decision on the shared UserLabel* types.

Verified

  • One strict live run on the branch: the 130 calls of the nightly, each 200 and at 1.0, no error. It ran before is_identity_verified was added; its 123 saved answers are at 1.0 with the final client.
  • With the final client: 595 of the 596 recorded answers of operations the spec has (the one left is the label above), the 16 control GETs, 122 of the 123 answers of the strict run of 2026-11-17 on the previous queryIds (the one left is the space that no longer resolves), the 25 stored space answers. No key with false, [], {} or null is left that the model does not have.
  • tools/build.py and the three generators of test-build.

Not verified

  • The mutations (never called by the harness) and the operations without a spec path (CommunitiesRankedTimeline, CommunitiesExploreTimeline among the changed ones).
  • A second account: BadgeCounts and conversation_control_details come from the sandbox account alone, and whether another viewer gets conversation_control_details was not seen.
  • The wrappers' e2e suites.

🤖 Generated with Claude Code

fa0311 and others added 3 commits November 20, 2026 17:35
Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
BadgeCounts.grok_bot_primary_unread_count / grok_bot_unread_conversations, AboutProfile.created_country_accurate,
AboutAccountVerificationInfo.is_identity_verified, AdMetadataContainer.allowSpineClicks: absent with the previous
queryId, present with the recorded one, same day, same target.

Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
…g of 2026-11-20

AboutProfile without account_based_in / affiliate_username / source, ProfileBio.with_held_scope,
Tweet.conversation_control_details, and on the community page question, primary_community_topic, a join action
that is unavailable and a community without custom_banner_media. None of them depends on the queryId.

Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
@fa0311
fa0311 merged commit 0773fc8 into main Nov 20, 2026
1 check passed
@fa0311
fa0311 deleted the chore/placeholder-2026-11-20 branch November 20, 2026 08:47
alexander-schneider added a commit to alexander-schneider/twscrape that referenced this pull request Nov 21, 2026
…rch request shape

Adopted from other open-source X clients after the 2026-11-21 wave:

- Animation key matches Chromium (clamp, CubicBezier solver, printed matrix
  rounding, frame time rounded to tens): 1000/1000 browser-recorded vectors,
  was 559 (Lqm1/x-client-transaction-id#26, anatolykoptev/go-twitter vectors).
- Signing indices found in the hashed x-web build via static and
  `.default()` dynamic imports; a chunk needs at least two indices
  (vladkens/twscrape#380).
- curl-cffi API calls send fetch headers instead of page-load headers.
- A search 404 is quarantined only if a signed control read works.
- SearchTimeline sends the web client's variables and exact feature set
  (fa0311/twitter-openapi#147), no fieldToggles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alexander-schneider added a commit to alexander-schneider/twscrape that referenced this pull request Nov 21, 2026
…quarantine (v0.22.0) (#28)

* feat: account-bound proxies, browser fingerprint, GET search, search quarantine (v0.22.0)

X stopped serving search to every account of a production pool on
2026-11-21 (six of them since 2026-11-05). The accounts all left through one
rotating gateway, sent an iPhone user agent over an httpx TLS handshake,
searched with POST although the web client uses GET, and kept being asked
after the refusal: the 5-minute hold sent each about 400 more refused
searches in two hours.

- An account's own proxy now wins over the API proxy and TWS_PROXY.
- Optional curl-cffi transport (API(impersonate=...) / TWS_IMPERSONATE) sends
  requests with a browser TLS and HTTP/2 fingerprint; the browser profile owns
  the user agent.
- SearchTimeline goes as GET like the x.com web client.
- A refused search (HTTP 200 "(-1) Internal server error" without data, or
  HTTP 404 after fresh signing) locks the account for search for
  TWS_SEARCH_REFUSAL_LOCK_SECONDS (default 6 h), asks each account once and
  raises SearchRefusedError (a ServiceUnavailableError) when none is left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: give up waiting when every account is held beyond the wait

With every account quarantined from search for hours, each search waited
out the full wait_timeout (900 s in the worker) before NoAccountError.
get_for_queue_or_wait now gives up at once when the earliest lock ends after
the deadline and after the 15-minute in-use placeholder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: take an account freed while checking whether to give up waiting

An account in use can be released between the first look and the lock check.
The check then saw only the quarantined accounts and gave up although an
account was free; the worker's end-to-end test with two accounts hit this.
Ask for a free account once more before giving up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: satisfy pyright on new tests and pyotp's TOTP export

CI's pyright flagged the HTTP handler's log_message signature and an
optional account in the new tests, and pyotp no longer re-exports TOTP at
the package root (also red on main since the newer pyotp); use pyotp.totp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop header values curl-cffi types as optional

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: browser-exact transaction ids, x-web signing, fetch headers, search request shape

Adopted from other open-source X clients after the 2026-11-21 wave:

- Animation key matches Chromium (clamp, CubicBezier solver, printed matrix
  rounding, frame time rounded to tens): 1000/1000 browser-recorded vectors,
  was 559 (Lqm1/x-client-transaction-id#26, anatolykoptev/go-twitter vectors).
- Signing indices found in the hashed x-web build via static and
  `.default()` dynamic imports; a chunk needs at least two indices
  (vladkens/twscrape#380).
- curl-cffi API calls send fetch headers instead of page-load headers.
- A search 404 is quarantined only if a signed control read works.
- SearchTimeline sends the web client's variables and exact feature set
  (fa0311/twitter-openapi#147), no fieldToggles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: keep the browser captures out of the ignored data/ directories

The fork's .gitignore ignores every data/ directory, so the vectors and the
search capture never reached CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: cool a search account 30 minutes when even the control read fails

A signing failure cooled each account for 15 minutes, inside the 15-minute
in-use horizon of the account wait, so the worker's other searches waited up
to 15 minutes each for an account that would fail again. 30 minutes lets them
give up at once. Found by the worker's fallback-chain simulation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: name the search 404 plainly and state what the control read can tell

A live check with a search-limited account on 2026-11-21 showed X answering
UserByScreenName, TweetDetail and UserTweets with 200 for a random or missing
x-client-transaction-id, so the control read tells an account that still
reads from one that fails every read, not a broken signature. The 404 error
no longer says "Failed to get XClIdGen".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant