Skip to content

Commit 5fbf256

Browse files
authored
Merge pull request #172 from mikekosulin/split-retention
feat(retention): split email body & attachment retention
2 parents 575a5cb + 0db03f3 commit 5fbf256

11 files changed

Lines changed: 654 additions & 13 deletions

File tree

docs/docs/admin/platform-settings.md

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,9 @@ Keys prefixed with `app.` are reserved and cannot be modified.
5050
| `default_rate_limit_daily` | int | `1000` | Default daily send limit for new workspaces |
5151
| `max_batch_size` | int | `100` | Default max recipients per batch send |
5252
| `max_attachment_size_mb` | int | `10` | Default max attachment size in MB |
53-
| `retention_days` | int | `30` | Days to retain email logs |
53+
| `retention_days` | int | `30` | Days to retain email records (metadata). Upper bound for the content windows below |
54+
| `email_body_retention_days` | int | `retention_days` | Days to retain email body content (HTML/text). Capped at `retention_days` |
55+
| `email_attachment_retention_days` | int | `retention_days` | Days to retain attachments and raw inbound messages. Capped at `retention_days` |
5456
| `audit_log_retention_days` | int | `90` | Days to retain audit log entries |
5557
| `webhook_delivery_retention_days` | int | `30` | Days to retain webhook delivery history |
5658
| `global_bounce_threshold` | int | `5` | Platform-wide bounce rate threshold (percent) |
@@ -62,3 +64,55 @@ Keys prefixed with `app.` are reserved and cannot be modified.
6264
| `login_rate_limit_window_minutes` | int | `15` | Rate limit window duration in minutes |
6365
| `email_content_visibility` | bool | `false` | Show full email content (body/HTML) in logs and detail views |
6466
| `custom_headers_enabled` | bool | `false` | Allow workspaces to add custom email headers |
67+
68+
### Email retention layers
69+
70+
Email data has two very different cost profiles. The **record** — subject, sender,
71+
recipients, status, timestamps — is tiny and worth keeping around for a searchable
72+
history. The **content** — HTML/text bodies, attachments, and (for inbound mail) the raw
73+
`.eml` — is what actually fills the disk. Posta retains them on independent schedules, so
74+
you can keep a long, lightweight log while purging bulk content much sooner.
75+
76+
Three windows apply to every email, all measured from its creation time:
77+
78+
| Layer | Setting | What it removes | What survives |
79+
|-------|---------|-----------------|---------------|
80+
| Record | `retention_days` | the whole row — metadata **and** content | nothing |
81+
| Body | `email_body_retention_days` | HTML + text body | the record (metadata) |
82+
| Attachments | `email_attachment_retention_days` | attachments + their stored bytes | the record (metadata) |
83+
84+
The body and attachment layers **scrub** content in place: the row stays, so the email
85+
still appears in dashboard lists and detail views with its metadata intact — only the
86+
body/attachment fields come back empty. Only the record layer removes the row entirely.
87+
88+
#### Example
89+
90+
With `retention_days = 180`, `email_body_retention_days = 30`, and
91+
`email_attachment_retention_days = 30`:
92+
93+
- **Day 0–30** — full email: metadata, body, and attachments.
94+
- **Day 31–180** — metadata only: body and attachments are gone, but the email still
95+
shows up in the log with its subject, sender, and status.
96+
- **Day 181+** — the record itself is deleted.
97+
98+
#### Rules
99+
100+
- **Defaults & upgrades.** Both content windows default to the current value of
101+
`retention_days`, so a fresh install — and any upgrade of an existing one — behaves
102+
exactly as before (everything purged together) until an admin deliberately shortens a
103+
content window.
104+
- **Upper bound.** Content windows are **capped at `retention_days`**: a larger value has
105+
no effect, because the row and all its content are deleted first. The dashboard clamps
106+
the inputs to `retention_days`, and the cleanup job enforces the same cap server-side.
107+
- **Body and attachments are independent.** Either may be kept longer or shorter than the
108+
other. The one coupling is the inbound raw `.eml`, which contains *both*: it is purged
109+
at the **shorter** of the two windows, so it can never preserve content past either
110+
type's own retention.
111+
- **Not the same as redaction.** [`email_content_visibility`](#available-settings) only
112+
*hides* body content from the dashboard and API responses — the data is still stored at
113+
rest. The retention windows above actually *delete* it from the database and blob
114+
storage. Use visibility for day-to-day privacy in the UI, and the content windows to
115+
reduce what is stored on disk.
116+
117+
The retention cleanup job runs once daily at 03:00 UTC, so content is purged within
118+
roughly 24 hours of crossing a window.

docs/docs/gdpr/data-deletion.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,13 @@ curl -X POST http://localhost:9000/api/v1/workspaces/current/gdpr/delete-email-l
135135

136136
Administrators can configure automatic data retention via [Platform Settings](/docs/admin/platform-settings):
137137

138-
- `email_retention_days` — Auto-delete email logs after N days
139-
- `webhook_retention_days` — Auto-delete webhook delivery history after N days
138+
- `retention_days` — Auto-delete email records after N days
139+
- `email_body_retention_days` — Scrub email body content after N days (record kept)
140+
- `email_attachment_retention_days` — Scrub attachments and raw inbound messages after N days (record kept)
141+
- `webhook_delivery_retention_days` — Auto-delete webhook delivery history after N days
140142
- `audit_log_retention_days` — Auto-delete audit log entries after N days
141143

142-
The retention cleanup job runs daily at 2:00 AM.
144+
The body/attachment windows let you purge heavy content early while keeping a lightweight
145+
log; see [Email retention layers](/docs/admin/platform-settings#email-retention-layers).
146+
147+
The retention cleanup job runs daily at 03:00 UTC.

internal/cron/jobs/retention.go

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ import (
3131
)
3232

3333
// RetentionCleanupJob purges old email logs, audit events, and webhook
34-
// delivery records based on platform retention settings.
34+
// delivery records based on platform retention settings. It also scrubs email
35+
// bodies and attachments off records that outlive their (shorter) content windows.
3536
type RetentionCleanupJob struct {
3637
emailRepo *repositories.EmailRepository
3738
eventRepo *repositories.EventRepository
@@ -153,6 +154,32 @@ func (j *RetentionCleanupJob) deleteRawKeys(keys []string) int {
153154
func (j *RetentionCleanupJob) Name() string { return "retention-cleanup" }
154155
func (j *RetentionCleanupJob) Schedule() string { return "0 3 * * *" } // daily at 03:00 UTC
155156

157+
// capDays caps a content window at the record window. A value of 0 means "keep
158+
// forever" on either side, so it is never treated as a cap or capped.
159+
func capDays(v, limit int) int {
160+
if limit <= 0 || v <= 0 {
161+
return v
162+
}
163+
if v > limit {
164+
return limit
165+
}
166+
return v
167+
}
168+
169+
// minDays returns the shorter of two windows, treating 0 ("forever") as no bound.
170+
func minDays(a, b int) int {
171+
if a <= 0 {
172+
return b
173+
}
174+
if b <= 0 {
175+
return a
176+
}
177+
if a < b {
178+
return a
179+
}
180+
return b
181+
}
182+
156183
func (j *RetentionCleanupJob) Run(_ context.Context, _ *asynq.Client) error {
157184
// Clean up email logs (and any blob-stored attachments).
158185
emailRetention := j.settings.RetentionDays()
@@ -201,6 +228,82 @@ func (j *RetentionCleanupJob) Run(_ context.Context, _ *asynq.Client) error {
201228
}
202229
}
203230

231+
// Content windows can never usefully exceed the record window — the row (and
232+
// all its content) is deleted first — so cap them at the email log retention.
233+
bodyRetention := capDays(j.settings.EmailBodyRetentionDays(), emailRetention)
234+
attachmentRetention := capDays(j.settings.EmailAttachmentRetentionDays(), emailRetention)
235+
236+
// Scrub attachments off records that outlive the attachment window (keeps the row).
237+
if attachmentRetention > 0 {
238+
before := time.Now().AddDate(0, 0, -attachmentRetention)
239+
240+
if j.blobStore != nil {
241+
if jsons, err := j.emailRepo.AttachmentsJSONOlderThan(before); err == nil {
242+
j.deleteOutboundAttachmentBlobs(jsons)
243+
} else {
244+
logger.Error("retention cleanup: failed to enumerate outbound attachments for scrub", "error", err)
245+
}
246+
}
247+
if scrubbed, err := j.emailRepo.ScrubAttachmentsOlderThan(before); err != nil {
248+
logger.Error("retention cleanup: failed to scrub outbound attachments", "error", err)
249+
} else if scrubbed > 0 {
250+
logger.Info("retention cleanup: scrubbed outbound attachments", "count", scrubbed, "older_than_days", attachmentRetention)
251+
}
252+
253+
if j.inboundEmailRepo != nil {
254+
if j.blobStore != nil {
255+
if jsons, _, err := j.inboundEmailRepo.InboundBlobKeysOlderThan(before); err == nil {
256+
j.deleteInboundAttachmentBlobs(jsons)
257+
} else {
258+
logger.Error("retention cleanup: failed to enumerate inbound attachments for scrub", "error", err)
259+
}
260+
}
261+
if scrubbed, err := j.inboundEmailRepo.ScrubAttachmentsOlderThan(before); err != nil {
262+
logger.Error("retention cleanup: failed to scrub inbound attachments", "error", err)
263+
} else if scrubbed > 0 {
264+
logger.Info("retention cleanup: scrubbed inbound attachments", "count", scrubbed, "older_than_days", attachmentRetention)
265+
}
266+
}
267+
}
268+
269+
// Scrub bodies off records that outlive the body window (keeps the row).
270+
if bodyRetention > 0 {
271+
before := time.Now().AddDate(0, 0, -bodyRetention)
272+
273+
if scrubbed, err := j.emailRepo.ScrubBodiesOlderThan(before); err != nil {
274+
logger.Error("retention cleanup: failed to scrub outbound bodies", "error", err)
275+
} else if scrubbed > 0 {
276+
logger.Info("retention cleanup: scrubbed outbound bodies", "count", scrubbed, "older_than_days", bodyRetention)
277+
}
278+
279+
if j.inboundEmailRepo != nil {
280+
if scrubbed, err := j.inboundEmailRepo.ScrubBodiesOlderThan(before); err != nil {
281+
logger.Error("retention cleanup: failed to scrub inbound bodies", "error", err)
282+
} else if scrubbed > 0 {
283+
logger.Info("retention cleanup: scrubbed inbound bodies", "count", scrubbed, "older_than_days", bodyRetention)
284+
}
285+
}
286+
}
287+
288+
// The raw .eml holds both body and attachments, so purge it at the shorter of
289+
// the two windows — it must never outlive either content type.
290+
if rawRetention := minDays(bodyRetention, attachmentRetention); rawRetention > 0 && j.inboundEmailRepo != nil {
291+
before := time.Now().AddDate(0, 0, -rawRetention)
292+
293+
if j.blobStore != nil {
294+
if _, rawKeys, err := j.inboundEmailRepo.InboundBlobKeysOlderThan(before); err == nil {
295+
j.deleteRawKeys(rawKeys)
296+
} else {
297+
logger.Error("retention cleanup: failed to enumerate inbound raw blobs for scrub", "error", err)
298+
}
299+
}
300+
if scrubbed, err := j.inboundEmailRepo.ScrubRawOlderThan(before); err != nil {
301+
logger.Error("retention cleanup: failed to scrub inbound raw messages", "error", err)
302+
} else if scrubbed > 0 {
303+
logger.Info("retention cleanup: scrubbed inbound raw messages", "count", scrubbed, "older_than_days", rawRetention)
304+
}
305+
}
306+
204307
// Clean up audit/event logs
205308
auditRetention := j.settings.AuditLogRetentionDays()
206309
if auditRetention > 0 {
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*
2+
* Copyright 2026 Jonas Kaninda
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*
16+
*/
17+
18+
package jobs
19+
20+
import "testing"
21+
22+
// capDays encodes "content never outlives the record": a positive window is
23+
// clamped to the record window; 0 ("forever") is never capped or used as a cap.
24+
func TestCapDays(t *testing.T) {
25+
cases := []struct {
26+
v, limit, want int
27+
}{
28+
{v: 30, limit: 180, want: 30}, // below the cap
29+
{v: 180, limit: 180, want: 180}, // at the cap
30+
{v: 999, limit: 180, want: 180}, // above the cap → clamped
31+
{v: 0, limit: 180, want: 0}, // v = forever → not capped
32+
{v: 30, limit: 0, want: 30}, // record = forever → no cap
33+
{v: 0, limit: 0, want: 0}, // both forever
34+
}
35+
for _, c := range cases {
36+
if got := capDays(c.v, c.limit); got != c.want {
37+
t.Errorf("capDays(%d, %d) = %d, want %d", c.v, c.limit, got, c.want)
38+
}
39+
}
40+
}
41+
42+
// minDays encodes "raw .eml never outlives either content window": the shorter
43+
// finite window wins, and 0 ("forever") means no bound on that side.
44+
func TestMinDays(t *testing.T) {
45+
cases := []struct {
46+
a, b, want int
47+
}{
48+
{a: 30, b: 90, want: 30}, // shorter first
49+
{a: 90, b: 30, want: 30}, // shorter second
50+
{a: 30, b: 30, want: 30}, // equal
51+
{a: 0, b: 30, want: 30}, // a forever → b bounds
52+
{a: 30, b: 0, want: 30}, // b forever → a bounds
53+
{a: 0, b: 0, want: 0}, // both forever
54+
}
55+
for _, c := range cases {
56+
if got := minDays(c.a, c.b); got != c.want {
57+
t.Errorf("minDays(%d, %d) = %d, want %d", c.a, c.b, got, c.want)
58+
}
59+
}
60+
}

internal/services/seeder/settings.go

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ import (
2525

2626
// SeedDefaultSettings creates default platform settings if they don't already exist.
2727
func SeedDefaultSettings(repo *repositories.SettingRepository) {
28+
// Content-retention windows default to the record retention so that an upgrade
29+
// never starts scrubbing content earlier than emails were already kept.
30+
retentionDays := "30"
31+
if s, err := repo.FindByKey("retention_days"); err == nil && s.Value != "" {
32+
retentionDays = s.Value
33+
}
34+
2835
defaults := []models.Setting{
2936
{Key: "registration_enabled", Value: "false", Type: "bool"},
3037
{Key: "require_email_verification", Value: "true", Type: "bool"},
@@ -34,6 +41,8 @@ func SeedDefaultSettings(repo *repositories.SettingRepository) {
3441
{Key: "max_batch_size", Value: "100", Type: "int"},
3542
{Key: "max_attachment_size_mb", Value: "10", Type: "int"},
3643
{Key: "retention_days", Value: "30", Type: "int"},
44+
{Key: "email_body_retention_days", Value: retentionDays, Type: "int"},
45+
{Key: "email_attachment_retention_days", Value: retentionDays, Type: "int"},
3746
{Key: "global_bounce_threshold", Value: "5", Type: "int"},
3847
{Key: "smtp_timeout_seconds", Value: "30", Type: "int"},
3948
{Key: "maintenance_mode", Value: "false", Type: "bool"},

internal/services/settings/provider.go

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,16 @@ func (p *Provider) MaxAttachmentSizeMB() int { return p.GetInt("max_attachmen
127127
func (p *Provider) GlobalBounceThreshold() int { return p.GetInt("global_bounce_threshold", 5) }
128128
func (p *Provider) SMTPTimeoutSeconds() int { return p.GetInt("smtp_timeout_seconds", 30) }
129129
func (p *Provider) RetentionDays() int { return p.GetInt("retention_days", 30) }
130-
func (p *Provider) AuditLogRetentionDays() int { return p.GetInt("audit_log_retention_days", 90) }
130+
131+
// EmailBodyRetentionDays / EmailAttachmentRetentionDays default to the record
132+
// retention so content is purged with the row unless an admin sets a shorter window.
133+
func (p *Provider) EmailBodyRetentionDays() int {
134+
return p.GetInt("email_body_retention_days", p.RetentionDays())
135+
}
136+
func (p *Provider) EmailAttachmentRetentionDays() int {
137+
return p.GetInt("email_attachment_retention_days", p.RetentionDays())
138+
}
139+
func (p *Provider) AuditLogRetentionDays() int { return p.GetInt("audit_log_retention_days", 90) }
131140
func (p *Provider) WebhookDeliveryRetentionDays() int {
132141
return p.GetInt("webhook_delivery_retention_days", 30)
133142
}

internal/storage/repositories/email_repo.go

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,3 +162,21 @@ func (r *EmailRepository) AttachmentsJSONOlderThan(before time.Time) ([]string,
162162
Pluck("attachments_json", &out).Error
163163
return out, err
164164
}
165+
166+
// ScrubBodiesOlderThan clears HTML/text bodies of emails older than the cutoff,
167+
// keeping the record. Returns the number of rows scrubbed.
168+
func (r *EmailRepository) ScrubBodiesOlderThan(before time.Time) (int64, error) {
169+
result := r.db.Model(&models.Email{}).
170+
Where("created_at < ? AND (html_body <> '' OR text_body <> '')", before).
171+
Updates(map[string]interface{}{"html_body": "", "text_body": ""})
172+
return result.RowsAffected, result.Error
173+
}
174+
175+
// ScrubAttachmentsOlderThan clears attachment metadata of emails older than the
176+
// cutoff, keeping the record. Blobs are deleted separately by the caller.
177+
func (r *EmailRepository) ScrubAttachmentsOlderThan(before time.Time) (int64, error) {
178+
result := r.db.Model(&models.Email{}).
179+
Where("created_at < ? AND attachments_json <> ''", before).
180+
Update("attachments_json", "")
181+
return result.RowsAffected, result.Error
182+
}

0 commit comments

Comments
 (0)