Skip to content

Define per-backend connection schemas (Spark, Postgres, S3, Kafka…) #2

Description

@rafafrdz

Motivation

Depends on #1 (top-level `connections:` section).

Once `connections:` exists, each connection `type` needs a well-defined schema so validators, UIs, and runtimes can agree on required/optional fields.

Proposal

Define a reference schema per backend, starting with the ones we already support or plan to support short-term:

`spark`

field type required notes
url string `sc://host:port/`
app_name string default: "teckel"
connect_timeout_seconds int default: 30
request_timeout_seconds int default: 300
auth_token string bearer token if the cluster requires it

`postgres`

field type required
url string
user string
password string
schema string

`s3`

field type required
bucket string
region string
access_key_id string
secret_access_key string
endpoint string

`kafka`

field type required
bootstrap_servers string
security_protocol enum
sasl_mechanism enum
sasl_username string
sasl_password string

Scope

  • Document each schema in the spec.
  • Encode as JSON Schema so validators can check `connections.*` entries at pipeline-load time.
  • Mark fields that support `${VAR}` interpolation vs. literal-only.

Out of scope

  • Runtime implementation (lives in teckel-api).
  • UI editor (lives in teckel-ui).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions