permafrost 🧊

Finds the S3 objects paying STANDARD rent while frozen solid.

Every data platform's buckets accumulate the same sediment: 2019's logs, exports nobody re-reads, the tmp/ prefix that outlived three reorganizations. All billed monthly at STANDARD rates, because lifecycle policies are the chore that loses to every sprint. permafrost reads the S3 Inventory reports your bucket already generates (no bucket scans, no credentials, no boto3), prices the frozen layers with the fine print included, and writes the lifecycle policy JSON that fixes it.

pip install -e .
permafrost plan --inventory path/to/manifest.json --out lifecycle-policy.json

The frozen layers  (move these)
  logs/2020                        -> DEEP_ARCHIVE after 365d
      3.7TB in 13,000 objects; saves $83.89/mo, one-time toll $650.00, breakeven 7.7 months
  logs/2019                        -> DEEP_ARCHIVE after 365d
      3.2TB in 11,000 objects; saves $71.15/mo, one-time toll $550.00, breakeven 7.7 months
  analytics/exports                -> STANDARD_IA after 90d
      1.7TB in 6,000 objects; saves $18.47/mo, one-time toll $60.00, breakeven 3.2 months

Left alone, and why  (the traps not fallen into)
  = tmp/cache: all 60,000 STANDARD objects are under 128KB: tiering would COST money (minimum-size billing)
  = models/current: median age 21d < 90d: churn risk beats the saving

=== The core sample ===
  recommendations : 3
  steady-state    : $173.51/mo  ($2,082/yr)
  one-time toll   : $1,260.00

That is real output from permafrost demo, which fabricates an aging 9.7TB bucket (93,500 objects across 6 strata) and finds exactly the planted story, pinned by a test.

The section most cost tools skip

Left alone, and why. A recommendation engine earns trust by showing the traps it did not fall into, not just the savings it found. The demo bucket plants two: a 60,000-object cache of ~14KB files that any naive calculator would happily tier at a loss (IA and Glacier bill a 128KB minimum per object), and a fresh, churning models prefix where minimum storage duration charges would eat the delta. permafrost names both and explains itself.

The fine print, priced in

Trap	How permafrost handles it
128KB minimum billing	Small-object bytes are split out per stratum and never move; mixed strata get a note with the count. The emitted rules carry ObjectSizeGreaterThan: 131072 so S3 enforces the floor even for objects that arrive later.
Transition tolls	Every recommendation shows the one-time request cost and months to breakeven. "Saves $40/mo after a $900 toll" is a different sentence than "saves $40/mo".
Minimum storage duration	Strata with median age under the class threshold stay put; churning data in IA is more expensive than leaving it alone.
Sub-dollar rules	Filtered. A lifecycle rule that saves forty cents is clutter, not FinOps.
Noncurrent versions	Excluded from every number (counting them as live data inflates reports); versioned cleanup is a separate, deliberate conversation.
Deletion	Never recommended. Colder storage only. Deleting data is a decision for humans with context, not a cost tool with a CSV.

Pricing knobs default to us-east-1 on-demand and print in every report; every number traces from dollars to GB to specific prefixes.

The plan is the product

permafrost plan ends where the work actually happens: an S3 Lifecycle configuration in the PutBucketLifecycleConfiguration shape, one scoped rule per prefix (bucket-wide rules fail big; scoped rules fail small), with provenance-carrying IDs for the engineer who finds the rule eleven months from now.

{
  "ID": "permafrost-logs-2019-DEEP_ARCHIVE",
  "Status": "Enabled",
  "Filter": { "And": { "Prefix": "logs/2019/", "ObjectSizeGreaterThan": 131072 } },
  "Transitions": [ { "Days": 365, "StorageClass": "DEEP_ARCHIVE" } ]
}

Reads what S3 actually delivers

S3 Inventory does not hand you one tidy CSV: it delivers a manifest.json whose fileSchema names the columns, next to headerless gzipped chunks. permafrost reads that directly (columns resolved by name, so order never matters, and field-count mismatches refuse to guess), plus bare headered CSVs for data that has been through somebody's notebook.

Performance

Measured by this repo's benchmark (make bench): single thread, 10 repeats per size, inventories in the real delivery format.

Objects	Read p50	Read p99	Analyze p50	Throughput
50,000	213ms	217ms	54ms	234k obj/s
200,000	880ms	1.22s	218ms	227k obj/s
800,000	3.61s	5.43s	0.95s	222k obj/s

Flat and linear: a 10-million-object inventory projects to ~45s single-threaded, and chunks are embarrassingly parallel if that ever matters. Results are committed (benchmark/results/) because the README should not claim what the repo cannot reproduce.

What this is not

• Not S3 Storage Lens. Lens shows dashboards; permafrost emits the policy JSON. Use Lens to watch the trend permafrost bends.
• Not Intelligent-Tiering. IT is the right answer for unpredictable access at a monitoring fee per object; permafrost is for the sediment whose access pattern you already know is "never".
• Not a deletion tool. It will make cold data cheap; making it gone is your call.

Development

make install   # editable install with dev extras
make test      # 13 tests, all fast, no infra
make lint      # ruff
make demo      # an aging bucket, drilled
make bench     # measure throughput on your machine

Easter egg

There is a permafrost mammoth command. Perfectly preserved.

MIT. Built by Prasannakumar Kasindala.