Repository navigation
Placeholder refresh (2026-11-20) and the fields its query versions return - #147
Merged
Merged
Conversation
Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
BadgeCounts.grok_bot_primary_unread_count / grok_bot_unread_conversations, AboutProfile.created_country_accurate, AboutAccountVerificationInfo.is_identity_verified, AdMetadataContainer.allowSpineClicks: absent with the previous queryId, present with the recorded one, same day, same target. Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
…g of 2026-11-20 AboutProfile without account_based_in / affiliate_username / source, ProfileBio.with_held_scope, Tweet.conversation_control_details, and on the community page question, primary_community_topic, a join action that is unavailable and a community without custom_banner_media. None of them depends on the queryId. Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
alexander-schneider
added a commit
to alexander-schneider/twscrape
that referenced
this pull request
Nov 21, 2026
…rch request shape Adopted from other open-source X clients after the 2026-11-21 wave: - Animation key matches Chromium (clamp, CubicBezier solver, printed matrix rounding, frame time rounded to tens): 1000/1000 browser-recorded vectors, was 559 (Lqm1/x-client-transaction-id#26, anatolykoptev/go-twitter vectors). - Signing indices found in the hashed x-web build via static and `.default()` dynamic imports; a chunk needs at least two indices (vladkens/twscrape#380). - curl-cffi API calls send fetch headers instead of page-load headers. - A search 404 is quarantined only if a signed control read works. - SearchTimeline sends the web client's variables and exact feature set (fa0311/twitter-openapi#147), no fieldToggles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alexander-schneider
added a commit
to alexander-schneider/twscrape
that referenced
this pull request
Nov 21, 2026
…quarantine (v0.22.0) (#28) * feat: account-bound proxies, browser fingerprint, GET search, search quarantine (v0.22.0) X stopped serving search to every account of a production pool on 2026-11-21 (six of them since 2026-11-05). The accounts all left through one rotating gateway, sent an iPhone user agent over an httpx TLS handshake, searched with POST although the web client uses GET, and kept being asked after the refusal: the 5-minute hold sent each about 400 more refused searches in two hours. - An account's own proxy now wins over the API proxy and TWS_PROXY. - Optional curl-cffi transport (API(impersonate=...) / TWS_IMPERSONATE) sends requests with a browser TLS and HTTP/2 fingerprint; the browser profile owns the user agent. - SearchTimeline goes as GET like the x.com web client. - A refused search (HTTP 200 "(-1) Internal server error" without data, or HTTP 404 after fresh signing) locks the account for search for TWS_SEARCH_REFUSAL_LOCK_SECONDS (default 6 h), asks each account once and raises SearchRefusedError (a ServiceUnavailableError) when none is left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: give up waiting when every account is held beyond the wait With every account quarantined from search for hours, each search waited out the full wait_timeout (900 s in the worker) before NoAccountError. get_for_queue_or_wait now gives up at once when the earliest lock ends after the deadline and after the 15-minute in-use placeholder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: take an account freed while checking whether to give up waiting An account in use can be released between the first look and the lock check. The check then saw only the quarantined accounts and gave up although an account was free; the worker's end-to-end test with two accounts hit this. Ask for a free account once more before giving up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: satisfy pyright on new tests and pyotp's TOTP export CI's pyright flagged the HTTP handler's log_message signature and an optional account in the new tests, and pyotp no longer re-exports TOTP at the package root (also red on main since the newer pyotp); use pyotp.totp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: drop header values curl-cffi types as optional Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: browser-exact transaction ids, x-web signing, fetch headers, search request shape Adopted from other open-source X clients after the 2026-11-21 wave: - Animation key matches Chromium (clamp, CubicBezier solver, printed matrix rounding, frame time rounded to tens): 1000/1000 browser-recorded vectors, was 559 (Lqm1/x-client-transaction-id#26, anatolykoptev/go-twitter vectors). - Signing indices found in the hashed x-web build via static and `.default()` dynamic imports; a chunk needs at least two indices (vladkens/twscrape#380). - curl-cffi API calls send fetch headers instead of page-load headers. - A search 404 is quarantined only if a signed control read works. - SearchTimeline sends the web client's variables and exact feature set (fa0311/twitter-openapi#147), no fieldToggles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: keep the browser captures out of the ignored data/ directories The fork's .gitignore ignores every data/ directory, so the vectors and the search capture never reached CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: cool a search account 30 minutes when even the control read fails A signing failure cooled each account for 15 minutes, inside the 15-minute in-use horizon of the account wait, so the worker's other searches waited up to 15 minutes each for an account that would fail again. 30 minutes lets them give up at once. Found by the worker's fallback-chain simulation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: name the search 404 plainly and state what the control read can tell A live check with a search-limited account on 2026-11-21 showed X answering UserByScreenName, TweetDetail and UserTweets with 200 for a random or missing x-client-transaction-id, so the control read tells an account that still reads from one that fails every read, not a broken signature. The 404 error no longer says "Failed to get XClIdGen". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, each with its
dist/. The first is the recording, the second is what the recorded query versions need, the third is independent of the queryIds and can be dropped on its own.placeholder: queryIds and features recorded on 2026-11-20
Observed
twitter-api-safe-dumpand the sandbox account in Google Chrome: the dump's 37 pages plus 16 pages that are only viewed (own profile, community members / moderators / hashtags, Community Notes,/X/affiliates,/elonmusk/about, analytics, delegate, premium, a space). 704 GraphQL requests of 88 operations; of all API answers 1,936 were 200, seven 403 (UsersByRestIds, as in any browser) and nine 429 at the end of the session (ViewerBadgeCounts,CreatorStudioTabBarItemQuery, sent on every page load).tools/update_placeholder.pychanged 42 entries:post_ctas_fetch_enabled: true(wasfalse):HomeTimeline,HomeLatestTimeline,TweetDetail,SearchTimeline,Likes,Bookmarks,BookmarkSearchTimeline,NotificationsTimeline,Retweeters,Followers,Following,BlueVerifiedFollowers,FollowersYouKnow,UserOriginalsTimeline,UserRepliesTimeline,UserRepostsTimeline,UserVideoTimeline,UserHighlightsTweets,UserSuperFollowTweets,ListLatestTweetsTimeline,ListMembers,ListSubscribers,ListSearchTimeline,ListsManagementPageTimeline,CombinedLists,CommunityTweetsTimeline,CommunityMediaTimeline,CommunityAboutTimeline,CommunityDiscoveryTimeline,GlobalCommunitiesPostSearchTimeline,GlobalCommunitiesLatestPostSearchTimeline,ExplorePage,ExploreSidebar,GenericTimelineById,ConnectTabTimeline,BirdwatchFetchGlobalTimeline.TweetResultByRestIdandAudioSpaceByIdkeep their queryId and get the flag.CommunitiesRankedTimelineandCommunitiesExploreTimelineget a new queryId, the flag,x_elections_post_context_enabled: trueandresponsive_web_made_on_x_label_enabled: false.ViewerBadgeCountsandAboutAccountQueryget a new queryId only.membersSliceTimeline_Query(below). 41 were not sent by any of the 53 pages and stay as they are: the 14 mutations,UserByRestId,UserTweets,UserTweetsAndReplies,UserMedia,Favoriters,BroadcastQuery,ListOwnerships,ListCreationRecommendedUsers,ListEditRecommendedUsers,DmAllSearchSlice,BirdwatchFetchContributorNotesSlice,CommunityHashtagsTimeline,affiliatesQuery,DelegatedAccountListQuery,useRelayDelegateDataPendingQuery,PremiumContentQuery,useSubscriptionProductDetailsQueryand the analytics / own-profile queries (useFetchAnalyticsQuery,QuickPromoteEligibility,TweetActivityQuery,isEligibleForAnalyticsUpsellQuery,useTotalAdCampaignsForUserQuery,UserPreferences,BlueVerifiedProfileEditCalloutQuery,useFetchProfileBlocks_profileExistsQuery,useFetchProfileSections_*).UserOriginalsTimeline,UserRepliesTimeline,UserRepostsTimelineandUserVideoTimeline(the first two for the own profile as well) and none ofUserTweets,UserTweetsAndReplies,UserMedia;/X/affiliatessentUserBusinessProfileTeamTimeline, notaffiliatesQuery.Not verified
post_ctas_fetch_enabled: false(CreateTweet,Favoriters,UserTweets,UserTweetsAndReplies,UserMedia,BroadcastQuery,ListOwnerships,ListCreationRecommendedUsers,ListEditRecommendedUsers,CommunityHashtagsTimeline,BirdwatchFetchContributorNotesSlice), while every request of this session that carries the flag sendstrue.membersSliceTimeline_Querywas sent four times withoutfeatures; the entry keepsresponsive_web_graphql_timeline_navigation_enabled, because the tool only overwrites what a request carries.ProfileSpotlightsQueryhasfeatures: {}and was sent without the parameter.HomeTimelineGET of the web client carries noseenTweetIds; the entry keeps the variable (variables are never overwritten).spec: fields of the query versions recorded on 2026-11-20
Each field was compared on the same day and the same target, once with the queryId of
mainand once with the recorded one (single GETs).mainBadgeCounts.grok_bot_primary_unread_count,grok_bot_unread_conversations(integer)AboutProfile.created_country_accurate(boolean)truefor elonmusk, X, SpaceX, Tesla; absent for NASA, which has noaccount_based_ineitherAboutAccountVerificationInfo.is_identity_verified(boolean)falsefor all five accountsAdMetadataContainer.allowSpineClicks(boolean)HomeTimelineanswerHomeTimeline/HomeLatestTimelineanswers and on all 7 of 1 GETAll four are optional: answers of the previous versions are part of the stored set and have to keep validating.
is_identity_verifiedandallowSpineClicks: falsedo not fail the strict comparison (it acceptsfalsewhere the model has nothing); they were found by listing the keys the model does not have.spec: shapes of other accounts, posts and communities in the recording
None of these depends on the queryId.
AboutProfile:account_based_in,affiliate_usernameandsourceare optional. With both queryIds NASA has none of the three and X has noaffiliate_username; only elonmusk, the placeholder's sample, has all of them.ProfileBio.with_held_scope(string,user): on one account of the People search fortwitter, next towithheld_in_countries, with both queryIds. The answers stored on 2026-11-17 have nine bios withwithheld_in_countriesand none with this field.Tweet.conversation_control_details({mode}): on the sandbox account's own posts with a reply restriction, four posts,modeequal tolegacy.conversation_control.policy(Community,Verified), with both queryIds. Two posts of other accounts with a restriction (ByInvitation,Co) do not have it.modeis a plain string: two values are not a closed set.CommunitiesFetchOneQuery, queryId unchanged, 19 answers for 16 communities in the recording):questionon 13 communities,primary_community_topicon 12 (the sharedPrimaryCommunityTopic), aCommunityJoinActionUnavailablewithmessageandreasonon 6, nocustom_banner_mediaon 1. The placeholder's community has none of these.Not typed: one of the 16 communities has an admin whose affiliate label mentions an account; on this page the mention carries
id,__typenameand__isTimelineReferenceObject. One answer, and it needs a decision on the sharedUserLabel*types.Verified
is_identity_verifiedwas added; its 123 saved answers are at 1.0 with the final client.false,[],{}ornullis left that the model does not have.tools/build.pyand the three generators oftest-build.Not verified
CommunitiesRankedTimeline,CommunitiesExploreTimelineamong the changed ones).BadgeCountsandconversation_control_detailscome from the sandbox account alone, and whether another viewer getsconversation_control_detailswas not seen.🤖 Generated with Claude Code