Skip to content

Commit fdec28f

Browse files
author
song.jie
committed
Revise README for clarity and structure
Updated the README to enhance clarity and organization. Key changes include a refined introduction, restructured sections, and the addition of new resources and guidelines for securing AI systems. Contributions and language options are now more prominently featured.
1 parent a355e81 commit fdec28f

6 files changed

Lines changed: 893 additions & 167 deletions

File tree

‎README.md‎

Lines changed: 106 additions & 167 deletions
Large diffs are not rendered by default.

‎docs/patterns/agent-safety.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# LLM / AI Agent Safety Pattern
2+
3+
> Guardrail pattern for multi-agent and tool-using LLM systems with
4+
> policy-as-code, execution sandboxes, and standardized security events.
5+
6+
This pattern focuses on **LLM-based agents** that can:
7+
8+
- Invoke external tools / APIs / SaaS
9+
- Call internal microservices or RPA workflows
10+
- Collaborate with other agents (planner / worker / reviewer)
11+
12+
Without guardrails, agent stacks become **privileged automation surfaces** that
13+
attackers can steer through prompt injection or malicious tool output.
14+
15+
---
16+
17+
## 1. Why Agent Safety Matters
18+
19+
Common failure modes:
20+
21+
- **Tool pivoting / living-off-the-agent**: model convinces tool layer to exfil
22+
secrets, spin up infrastructure, or break RBAC.
23+
- **Implicit privilege escalation**: agent inherits creds for every downstream
24+
system because the orchestrator uses a single service account.
25+
- **Prompt / memory poisoning**: earlier steps inject instructions that later
26+
agents or the shared memory blindly follow.
27+
- **Unbounded automation**: no circuit breaker when agents loop or trigger
28+
costly, destructive operations.
29+
30+
> **Treat agent actions the same way you treat API calls from humans.**
31+
32+
---
33+
34+
## 2. Reference Architecture
35+
36+
```text
37+
User / Trigger
38+
↓
39+
Agent Orchestrator (LangGraph / AutoGen / custom)
40+
↓ ↑
41+
Policy & PDP (OPA) Security Events (ASB)
42+
↓
43+
Tool / Skill Gateway → Sandboxed Connectors → Systems of Record
44+
```
45+
46+
Key ideas:
47+
48+
1. **Agent orchestrator never calls tools directly** — it goes through a
49+
**Tool Gateway** that:
50+
- Builds an `agent_action` security event (ASB Security Schema)
51+
- Evaluates policy (OPA/Rego) before dispatching
52+
- Enforces output schemas / rate limits
53+
2. **Tools run in isolated sandboxes** (container, Temporal activity, Cloud
54+
Function) with scoped credentials.
55+
3. **All actions & intermediate messages are logged** for audit / playback.
56+
57+
---
58+
59+
## 3. Layered Controls
60+
61+
### 3.1 Prompt & Memory Hygiene
62+
63+
- Split **system vs user vs tool** memory segments; never let tool output modify
64+
system instructions.
65+
- Run prompt-injection detectors on:
66+
- User queries
67+
- Retrieved context
68+
- Tool responses
69+
- Store high-risk chunks in `context.risk_signals` of security events for later
70+
policy decisions (e.g., escalate to human review).
71+
72+
### 3.2 Policy Gating for Tool Calls
73+
74+
Represent each tool invocation as an `agent_action` event:
75+
76+
```jsonc
77+
{
78+
"operation": {
79+
"category": "agent_action",
80+
"name": "invoke_tool",
81+
"stage": "pre"
82+
},
83+
"subject": {
84+
"agent": {
85+
"id": "workflow-planner",
86+
"session_id": "sess-123"
87+
},
88+
"user": {
89+
"id": "alice",
90+
"roles": ["support"]
91+
}
92+
},
93+
"resource": {
94+
"agent_action": {
95+
"tool": "jira.create_ticket",
96+
"args": {
97+
"project": "SEC",
98+
"summary": "Reset all admin passwords"
99+
}
100+
}
101+
},
102+
"context": {
103+
"trace_id": "trace-456",
104+
"risk_signals": {
105+
"prompt_injection_score": 0.87
106+
}
107+
}
108+
}
109+
```
110+
111+
Policy examples (Rego):
112+
113+
- Block `jira.create_ticket` unless user role is `support_manager`.
114+
- Require **human approval** when risk score > threshold.
115+
- Limit tool frequency per session (`count` across recent events).
116+
117+
### 3.3 Sandboxed Connectors
118+
119+
- Run each tool in its own container / function with **ephemeral credentials**.
120+
- Enforce **least privilege** per tool (scoped API tokens, short-lived STS).
121+
- Apply **network egress controls** so tool code cannot reach arbitrary hosts.
122+
123+
### 3.4 Output Validation & Circuit Breakers
124+
125+
- Validate tool responses against JSON schema; reject if missing mandatory fields
126+
or containing suspicious text (e.g., base64 blocks, system prompts).
127+
- Add **budget caps** (max tool calls, max cost, max duration).
128+
- Add **loop detection**: if the same intent repeats N times, escalate to human.
129+
130+
### 3.5 Observability & Forensics
131+
132+
- Send events to SIEM:
133+
- `agent_action` (pre & post decision)
134+
- `llm_completion` for agent-to-agent messages
135+
- `rag_search` when agents retrieve knowledge
136+
- Tag with `workflow`, `project`, `tenant` for root-cause analysis.
137+
138+
---
139+
140+
## 4. Implementation Tips
141+
142+
1. **Start with read-only tools**. Gradually add write-capable tools once policy
143+
+ audit trail are proven.
144+
2. **Bundle default policies** with each tool (e.g., CloudOps tool requires
145+
Env=dev). This avoids drift.
146+
3. **Simulate attacks**:
147+
- User prompt tries to trick agent into emailing secrets.
148+
- Tool output injects `Ignore all safety instructions`.
149+
- Long-running loop draining API quota.
150+
4. **Provide human-in-the-loop UI** for high-risk actions:
151+
- Show security event + model reasoning
152+
- Require explicit approval
153+
154+
---
155+
156+
## 5. Summary
157+
158+
- Treat agents as **untrusted orchestrators** that need guardrails at each tool
159+
boundary.
160+
- Use the **ASB Security Schema** to normalize agent telemetry, enabling OPA
161+
policies and SIEM analytics.
162+
- Combine **prompt hygiene, policy gating, sandboxed tools, and circuit
163+
breakers** to prevent privilege escalation and data exfiltration.
164+
165+
When implemented, this pattern turns agent stacks from opaque automation scripts
166+
into **auditable, enforceable workflows** suitable for enterprise adoption.
167+

‎docs/patterns/llm-gateway.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# LLM Gateway Guardrail Pattern
2+
3+
> Deploy an API gateway in front of LLM services to enforce authn/z, content
4+
> filtering, policy-as-code, and ASB-standard telemetry for every prompt +
5+
> completion.
6+
7+
This pattern applies whether you host an open-source model, call a SaaS API, or
8+
proxy many providers. The **gateway** becomes the control plane for safety,
9+
compliance, and cost management.
10+
11+
---
12+
13+
## 1. Why an LLM Gateway?
14+
15+
- Central place to enforce **authentication, quotas, budgets**.
16+
- Consistent **prompt/response filtering** regardless of provider.
17+
- Observability + audit trail (`prompt_submission`, `llm_completion` events).
18+
- Ability to route / failover across models without app changes.
19+
20+
Without a gateway, each app re-implements controls (inconsistently) and you
21+
cannot answer “Who sent this prompt?” or “Which response leaked data?”.
22+
23+
---
24+
25+
## 2. Reference Flow
26+
27+
```text
28+
Client / App
29+
|
30+
v
31+
LLM Gateway
32+
| ↘ Policy Decision (OPA) + ASB events
33+
v
34+
Provider Adapters (OpenAI, Anthropic, OSS hosters)
35+
```
36+
37+
---
38+
39+
## 3. Core Capabilities
40+
41+
### 3.1 Authentication & Authorization
42+
43+
- Support API keys, OAuth2 client credentials, or mTLS.
44+
- Map tokens to **tenants / projects / roles**.
45+
- Enforce RBAC/ABAC:
46+
- Which models a tenant can access
47+
- Max context length / cost per request
48+
49+
### 3.2 Prompt & Response Filtering
50+
51+
1. **Pre-processing**
52+
- Sanitize structured inputs (JSON schema validation).
53+
- Run detectors for PII, secrets, malware intent.
54+
- Tag results as `context.risk_signals` → OPA can escalate/deny.
55+
2. **Post-processing**
56+
- Inspect completions for policy violations (toxic, data leakage).
57+
- Mask or reject responses; attach `decision.effect`.
58+
59+
### 3.3 Routing & Failover
60+
61+
- Policy-driven routing:
62+
- `if tenant == finance -> use internal model`
63+
- `if request contains PHI -> route to HIPAA-compliant provider`
64+
- Weighted or fallback routing when providers throttle / fail.
65+
66+
### 3.4 Quotas & Cost Controls
67+
68+
- Track tokens / dollars per tenant, per time window.
69+
- Hard/soft limits with alerting.
70+
- Optionally inject optimization (cache previous completion, apply compression).
71+
72+
---
73+
74+
## 4. Telemetry (ASB Security Schema)
75+
76+
Emit structured events for each stage:
77+
78+
1. `prompt_submission` (pre-decision)
79+
80+
```jsonc
81+
{
82+
"operation": {"category": "prompt_submission", "stage": "pre"},
83+
"subject": {"user": {"id": "alice", "tenant": "acme"}},
84+
"resource": {
85+
"prompt": {
86+
"model": "gpt-4o",
87+
"input": {"messages": [{"role": "user", "content": "Show SSNs"}]},
88+
"metadata": {"app_id": "kb-bot"}
89+
}
90+
},
91+
"context": {"risk_signals": {"pii_detected": true}}
92+
}
93+
```
94+
95+
2. Policy engine returns allow/deny/mask.
96+
97+
3. `llm_completion` (post-decision) captures provider response, tokens, latency,
98+
and `decision`.
99+
100+
Benefits:
101+
102+
- Unified audit logs across providers.
103+
- SIEM dashboards for risky prompts/responses.
104+
- Forensics: link completions back to user + tenant + app.
105+
106+
---
107+
108+
## 5. Policy as Code Examples
109+
110+
### Deny requests with unapproved model IDs
111+
112+
```rego
113+
package asb.prompt
114+
115+
default allow = false
116+
117+
allow {
118+
input.operation.category == "prompt_submission"
119+
allowed_models := {"gpt-4o", "internal-qa-v2"}
120+
allowed_models[input.resource.prompt.model]
121+
}
122+
```
123+
124+
### Require human review for high-risk completions
125+
126+
```rego
127+
package asb.llm_completion
128+
129+
default decision = {"effect": "allow"}
130+
131+
decision = {"effect": "review", "reason": "pii"} {
132+
input.context.risk_signals.pii_detected == true
133+
}
134+
```
135+
136+
Gateway pipeline:
137+
138+
1. Build ASB event → send to OPA.
139+
2. Apply decision (block, mask, review).
140+
3. Log final event with decision metadata.
141+
142+
---
143+
144+
## 6. Implementation Tips
145+
146+
- **Modular adapters**: wrap each provider with a standard interface, enabling
147+
retries, tracing, and provider-specific auth.
148+
- **Caching**: optionally memoize deterministic prompts to cut cost—store cache
149+
keys + hits in events for transparency.
150+
- **Developer ergonomics**: provide SDKs/clients so product teams don’t bypass
151+
the gateway.
152+
- **Health monitoring**: track provider latency/error rate; auto-brownout when
153+
they breach SLOs.
154+
- **Secrets handling**: gateway stores provider credentials; clients never see
155+
them.
156+
157+
---
158+
159+
## 7. Summary
160+
161+
- An LLM gateway centralizes **safety, compliance, and routing** for all
162+
generative workloads.
163+
- Pair it with **ASB security events + OPA** to make every prompt auditable and
164+
enforceable.
165+
- Add value with **quota management, multi-provider routing, and filtering** so
166+
application teams focus on business logic, not duplicating guardrails.
167+

0 commit comments

Comments
 (0)