Skip to content

Fix/visitor activity 504#8

Open
dmosorast wants to merge 5 commits into
masterfrom
fix/visitor_activity_504
Open

Fix/visitor activity 504#8
dmosorast wants to merge 5 commits into
masterfrom
fix/visitor_activity_504

Conversation

@dmosorast

Copy link
Copy Markdown
Contributor

Description of change

Pardot's API will return a 504 on large requests with certain parameters for the visitorActivity endpoint. There's an inactive topic on the Salesforce community boards that suggests changing the request parameters could cause this to be more efficient. https://trailblazers.salesforce.com/answers?id=9063A000000DlRUQA0

In testing, sorting by updated_at descending proved to be the only consistent option, so this PR adds a stream type to handle that pattern (bookmarking based on tap-trello's implementation).

This PR also adds a method to translate existing "id-based" state to the current "updated_at" based state, using a request for the activity object whose ID is bookmarked.

Of note: The API seems to treat datetimes formatted as YYYY-mm-DD HH:MM:SS as the user's time zone, and datetimes formatted as YYYY-mm-DDTHH:MM:SSZ as UTC. A critical part of this PR is to never change the formatting of values so that these types can be mixed effectively for consistency.

Manual QA steps

  • Ran through cases of the state to confirm that:
    • The expected count of records is returned using this method
    • The pagination works and aligns using mixed datetime format (UTC vs user, in this case EDT)
    • The state translation works and successfully overrides a reasonable start_date (e.g., 2019-04-12T00:00:00.000000Z)

Risks

  • Medium, this is the replacement of an existing pattern, and testing will need to be performed end to end with a live connection to validate the resulting data.

Rollback steps

  • revert this branch, bump patch version, and release

Comment thread tap_pardot/streams.py
@chrisgonzalez0

Copy link
Copy Markdown

Will this be merged anytime soon? I am using Stitch to try to backfill some historical data from Pardot and this same issue seems to be occurring on the Stitch side. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants