variables.list() infinite-loops on workspaces with ≥100 variables (the /vars endpoint is not paginated)
Version
- Reproduced on
pytfe 0.1.5.
- Current
main is still affected (verified against the rewritten _Service._list; see "Does main fix it?" below).
- Python 3.12.
- Terraform Enterprise (self-hosted); same behavior expected on HCP Terraform.
What happens
client.variables.list(workspace_id) never terminates for any workspace that
has ≥100 variables. It issues an unbounded sequence of requests with an
ever-increasing page[number]:
GET /api/v2/workspaces/ws-XXXXXXXX/vars?page[number]=1901&page[size]=100
GET /api/v2/workspaces/ws-XXXXXXXX/vars?page[number]=1902&page[size]=100
GET /api/v2/workspaces/ws-XXXXXXXX/vars?page[number]=1903&page[size]=100
...
Each response also re-yields the same full set of variables, so a consumer
draining the iterator gets massive duplication in addition to never finishing.
Root cause
The workspace variables endpoint, GET /api/v2/workspaces/:workspace_id/vars,
is not paginated — it returns every variable for the workspace in a single
response, ignores the page[number] / page[size] query parameters, and
returns no meta.pagination block.
Variables.list (src/pytfe/resources/variable.py) drives it through the
generic _Service._list helper (src/pytfe/resources/_base.py), which
paginates by incrementing page[number] until a page returns fewer than
page[size] items. Because /vars ignores pagination and returns the full set
every time, that condition is never met once a workspace has ≥ page[size]
(default 100) variables, and the loop runs forever.
Reproduction
from pytfe import TFEClient, TFEConfig
client = TFEClient(TFEConfig(address="https://<your-tfe>", token="<token>"))
# Any workspace with >= 100 variables.
for v in client.variables.list("ws-XXXXXXXX"):
print(v.key) # never terminates; keys repeat every 100 items
Confirming the endpoint is unpaginated
Hitting /vars directly on an affected workspace (941 variables), requesting
two different pages:
page=1 len(data)=941 meta=null
page=2 len(data)=941 meta=null
The endpoint returns the full 941-row set on every page regardless of
page[number], and emits no pagination metadata (meta is null).
Does main fix it? No.
The _Service._list rewrite on main adds (1) an empty-page guard
(if not data: break) and (2) a meta.pagination-aware termination path,
falling back to the old len(data) < page_size heuristic only when no
pagination metadata is present. Neither helps this endpoint:
data is non-empty (941 rows) on every page → the empty-page guard never
fires.
meta is null → the meta.pagination branch is skipped.
- Control falls through to the same
len(data) < page_size heuristic
(941 < 100 is false) → page += 1 → the loop continues unbounded.
So a workspace with ≥100 variables still infinite-loops on current main.
Variables.list / list_all are unchanged and still route through _list.
Suggested fix
Fix this in Variables.list and Variables.list_all rather than with more
_list heuristics: issue a single, un-paginated request to
/api/v2/workspaces/:id/vars (and /all-vars) and return all rows from that
one response. This matches go-tfe, which does not paginate these endpoints.
A purely heuristic fix in _list cannot reliably cover this case: the endpoint
returns a non-empty, full result set with no pagination metadata, which is
indistinguishable (by row count alone) from a genuine full page that has a
successor.
Workaround
Call the endpoint directly via the transport and read data once:
r = client._transport.request("GET", f"/api/v2/workspaces/{ws_id}/vars")
variables = r.json().get("data", [])
variables.list()infinite-loops on workspaces with ≥100 variables (the/varsendpoint is not paginated)Version
pytfe0.1.5.mainis still affected (verified against the rewritten_Service._list; see "Doesmainfix it?" below).What happens
client.variables.list(workspace_id)never terminates for any workspace thathas ≥100 variables. It issues an unbounded sequence of requests with an
ever-increasing
page[number]:Each response also re-yields the same full set of variables, so a consumer
draining the iterator gets massive duplication in addition to never finishing.
Root cause
The workspace variables endpoint,
GET /api/v2/workspaces/:workspace_id/vars,is not paginated — it returns every variable for the workspace in a single
response, ignores the
page[number]/page[size]query parameters, andreturns no
meta.paginationblock.Variables.list(src/pytfe/resources/variable.py) drives it through thegeneric
_Service._listhelper (src/pytfe/resources/_base.py), whichpaginates by incrementing
page[number]until a page returns fewer thanpage[size]items. Because/varsignores pagination and returns the full setevery time, that condition is never met once a workspace has ≥
page[size](default 100) variables, and the loop runs forever.
Reproduction
Confirming the endpoint is unpaginated
Hitting
/varsdirectly on an affected workspace (941 variables), requestingtwo different pages:
The endpoint returns the full 941-row set on every page regardless of
page[number], and emits no pagination metadata (metaisnull).Does
mainfix it? No.The
_Service._listrewrite onmainadds (1) an empty-page guard(
if not data: break) and (2) ameta.pagination-aware termination path,falling back to the old
len(data) < page_sizeheuristic only when nopagination metadata is present. Neither helps this endpoint:
datais non-empty (941 rows) on every page → the empty-page guard neverfires.
metaisnull→ themeta.paginationbranch is skipped.len(data) < page_sizeheuristic(
941 < 100is false) →page += 1→ the loop continues unbounded.So a workspace with ≥100 variables still infinite-loops on current
main.Variables.list/list_allare unchanged and still route through_list.Suggested fix
Fix this in
Variables.listandVariables.list_allrather than with more_listheuristics: issue a single, un-paginated request to/api/v2/workspaces/:id/vars(and/all-vars) and return all rows from thatone response. This matches
go-tfe, which does not paginate these endpoints.A purely heuristic fix in
_listcannot reliably cover this case: the endpointreturns a non-empty, full result set with no pagination metadata, which is
indistinguishable (by row count alone) from a genuine full page that has a
successor.
Workaround
Call the endpoint directly via the transport and read
dataonce: