Skip to content

Use less time/bandswitch intensive download method for Polis data #71

Description

@patcon

Right now, we by default download data via API, whether provided a report_id or not. But when we have a report ID, and the server is recent enough, we can just download the CSV export in one single HTTP request (instead of 1 request per user for scraping vote data). There are some edge-cases that we can handle, and we should do that thoughtfully behind the scenes.

Here's what should happen:

  • if we are given a report url, set the polis_instance_url, and try to fetch from CSV
    • is-meta is currently missing from csv exports. we should backfill statement data from the API, and throw a warning
  • if can't fetch from CSV, it's because the server is too old to support, so use the API (will take much longer) *
  • if conversation url provided, fetch via API (can't use report ID)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions