Motivation
Depends on #1 (top-level `connections:` section).
Once `connections:` exists, each connection `type` needs a well-defined schema so validators, UIs, and runtimes can agree on required/optional fields.
Proposal
Define a reference schema per backend, starting with the ones we already support or plan to support short-term:
`spark`
| field |
type |
required |
notes |
| url |
string |
✓ |
`sc://host:port/` |
| app_name |
string |
|
default: "teckel" |
| connect_timeout_seconds |
int |
|
default: 30 |
| request_timeout_seconds |
int |
|
default: 300 |
| auth_token |
string |
|
bearer token if the cluster requires it |
`postgres`
| field |
type |
required |
| url |
string |
✓ |
| user |
string |
|
| password |
string |
|
| schema |
string |
|
`s3`
| field |
type |
required |
| bucket |
string |
✓ |
| region |
string |
|
| access_key_id |
string |
|
| secret_access_key |
string |
|
| endpoint |
string |
|
`kafka`
| field |
type |
required |
| bootstrap_servers |
string |
✓ |
| security_protocol |
enum |
|
| sasl_mechanism |
enum |
|
| sasl_username |
string |
|
| sasl_password |
string |
|
Scope
- Document each schema in the spec.
- Encode as JSON Schema so validators can check `connections.*` entries at pipeline-load time.
- Mark fields that support `${VAR}` interpolation vs. literal-only.
Out of scope
- Runtime implementation (lives in teckel-api).
- UI editor (lives in teckel-ui).
Motivation
Depends on #1 (top-level `connections:` section).
Once `connections:` exists, each connection `type` needs a well-defined schema so validators, UIs, and runtimes can agree on required/optional fields.
Proposal
Define a reference schema per backend, starting with the ones we already support or plan to support short-term:
`spark`
`postgres`
`s3`
`kafka`
Scope
Out of scope