Error while requesting from Google Scholar (Error: Request failed with status code 429)

[amcc1996]

splurt throws the following error while running data command (see the full output here)

$ splurt data -p project.yaml
>> Error: Request failed with status code 429

Fetching from Scopus runs normally.

splurt fetch -p project.yaml

The contents from porject.yaml are derived from the example in the documentation:

sqlite: articles.db

fetch:
  query: 'data AND google'
  databases: [scopus]
  maximum: 1
  scopus: Here I inserted by SCOPUS_API_KEY

exclude:
  # SQL like WHERE using these columns: id, doi, title, year, authors, publication, 
  # type, origin, cites, abstract, link, bibtex
  criteria: ['cites = 0','year < 2000']

data:
  delay: 10
  cookie: "LM=1599585071:S=40CQ9rvXrNpRCV5f"

export:
  # id, doi, title, year, authors, publication, 
  # type, origin, cites, abstract, link, bibtex
  data: [title, doi, year, cites, abstract]
  # if the format is html, the data attribute is ignored
  format: csv # can be csv,json,table,html
  # if scihub is specified, available pdfs will be downloaded
  # requires the (doi attribute)
  scihub: 'https://sci-hub.se/' # current sci-hub valid url

The problem seems to be associated with the cookie attribute. I followed the documentation to get the cookie value using Mozilla Firefox:

1. Opened Google Scholar in a private window
2. Opened current cookies with Shift + F9
3. Copied the value of GSP cookie (which I noticed that changed from one attempt to another)

The documentation is very explanatory and easy to follow so I have repeated these steps several times but cannot understand what is happening. Any thought on why this is occurring?

Disclaimer: I am completely new to splurt, so this might not even be a true issue but simple misuse

[amcc1996]

Apparently, this has been noticed before. See #7.

[arestivo]

Thanks for the bug report.

Status code 429 is a rate limiting error: "Too many requests". But it also might just be that Google changed something. I'll check it out as soon as possible.

[Gropi]

Is there any solution to this?