Skip to content

variables.list() infinite-loops on workspaces with ≥100 variables (the /vars endpoint is not paginated) #181

Description

@gadeleon

variables.list() infinite-loops on workspaces with ≥100 variables (the /vars endpoint is not paginated)

Version

  • Reproduced on pytfe 0.1.5.
  • Current main is still affected (verified against the rewritten _Service._list; see "Does main fix it?" below).
  • Python 3.12.
  • Terraform Enterprise (self-hosted); same behavior expected on HCP Terraform.

What happens

client.variables.list(workspace_id) never terminates for any workspace that
has ≥100 variables. It issues an unbounded sequence of requests with an
ever-increasing page[number]:

GET /api/v2/workspaces/ws-XXXXXXXX/vars?page[number]=1901&page[size]=100
GET /api/v2/workspaces/ws-XXXXXXXX/vars?page[number]=1902&page[size]=100
GET /api/v2/workspaces/ws-XXXXXXXX/vars?page[number]=1903&page[size]=100
...

Each response also re-yields the same full set of variables, so a consumer
draining the iterator gets massive duplication in addition to never finishing.

Root cause

The workspace variables endpoint, GET /api/v2/workspaces/:workspace_id/vars,
is not paginated — it returns every variable for the workspace in a single
response, ignores the page[number] / page[size] query parameters, and
returns no meta.pagination block.

Variables.list (src/pytfe/resources/variable.py) drives it through the
generic _Service._list helper (src/pytfe/resources/_base.py), which
paginates by incrementing page[number] until a page returns fewer than
page[size] items. Because /vars ignores pagination and returns the full set
every time, that condition is never met once a workspace has ≥ page[size]
(default 100) variables, and the loop runs forever.

Reproduction

from pytfe import TFEClient, TFEConfig

client = TFEClient(TFEConfig(address="https://<your-tfe>", token="<token>"))

# Any workspace with >= 100 variables.
for v in client.variables.list("ws-XXXXXXXX"):
    print(v.key)   # never terminates; keys repeat every 100 items

Confirming the endpoint is unpaginated

Hitting /vars directly on an affected workspace (941 variables), requesting
two different pages:

page=1  len(data)=941  meta=null
page=2  len(data)=941  meta=null

The endpoint returns the full 941-row set on every page regardless of
page[number], and emits no pagination metadata (meta is null).

Does main fix it? No.

The _Service._list rewrite on main adds (1) an empty-page guard
(if not data: break) and (2) a meta.pagination-aware termination path,
falling back to the old len(data) < page_size heuristic only when no
pagination metadata is present. Neither helps this endpoint:

  1. data is non-empty (941 rows) on every page → the empty-page guard never
    fires.
  2. meta is null → the meta.pagination branch is skipped.
  3. Control falls through to the same len(data) < page_size heuristic
    (941 < 100 is false) → page += 1 → the loop continues unbounded.

So a workspace with ≥100 variables still infinite-loops on current main.
Variables.list / list_all are unchanged and still route through _list.

Suggested fix

Fix this in Variables.list and Variables.list_all rather than with more
_list heuristics: issue a single, un-paginated request to
/api/v2/workspaces/:id/vars (and /all-vars) and return all rows from that
one response. This matches go-tfe, which does not paginate these endpoints.

A purely heuristic fix in _list cannot reliably cover this case: the endpoint
returns a non-empty, full result set with no pagination metadata, which is
indistinguishable (by row count alone) from a genuine full page that has a
successor.

Workaround

Call the endpoint directly via the transport and read data once:

r = client._transport.request("GET", f"/api/v2/workspaces/{ws_id}/vars")
variables = r.json().get("data", [])

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions