TDL-21378 increase page_size for all streams

[vishalp-dev]

Description of change

• added new config param for configurable page_size prop
• added config.sample.json

Manual QA steps

Risks (None)

Rollback steps

• revert this branch

[sgandhi1311]

Why are we setting the stream.next_start while initializing the object?
I think we should capture this value from the API endpoint response within the pagination function.

[vishalp-dev]

i agree, and that is what its doing here(https://github.com/singer-io/tap-pipedrive/blob/master/tap_pipedrive/stream.py#L71-L82)

    def paginate(self, response):
        payload = response.json()

        if 'additional_data' in payload and 'pagination' in payload['additional_data']:
            logger.debug('Paginate: valid response')
            pagination = payload['additional_data']['pagination']
            if 'more_items_in_collection' in pagination:
                self.more_items_in_collection = pagination['more_items_in_collection']

                if 'next_start' in pagination:
                    self.start = pagination['next_start']

i don't think the attribute next_start is necessary, but i am not too confident of removing it in the context of this P.R

[luandy64]

Have you run this code?

To me it looks like self.streams doesn't exist, so I'm not sure how you are looping over it.

stream.limit = stream.next_start = page_size

This is also hard to read in my opinion.

stream.limit = page_size
stream.next_start = page_size

This makes more sense to me.

[luandy64]

Have you run this code?

To me it looks like self.streams doesn't exist, so I'm not sure how you are looping over it.

stream.limit = stream.next_start = page_size

This is also hard to read in my opinion.

stream.limit = page_size
stream.next_start = page_size

This makes more sense to me.

[luandy64]

Please at a newline at the end of this file

[vishalp-dev]

hi @luandy64 , i have tested this code. self.streams will refer to the class attribute streams declared on the class itself, it might not be a good practice to modify class attributes via instances as it can result in an inconsistent state but we are only creating one instance of this class.

does this need to be changed?

class PipedriveTap(object):
    streams = [
        CurrenciesStream(),
        ActivityTypesStream(),
        StagesStream(),
        ... 
]

    def __init__(self, config, state):
        self.config = self.get_default_config()
        ...

        page_size = int(self.config.get('page_size',500))
        for stream in self.streams:
            stream.limit = stream.next_start = page_size