Skip to content

README.md

Latest commit

 

History

History
533 lines (390 loc) · 15.4 KB

README.md

File metadata and controls

533 lines (390 loc) · 15.4 KB

ytdl_tg_bot

A telegram bot for downloading audio and video

Telegram: @yv2t_bot

Features

  • Video download
  • Audio download
  • Playlist download
  • Language selection
  • Skip download param
  • Random media
  • Media crop
  • Stats
  • Exclude domains list

Self Install

This guide assumes Kubernetes, Helm, and a working image registry. If you are deploying from this repository instead of using already-published images, build and push dev images first.

1. Prerequisites

Required local tools:

  • kubectl
  • helm
  • just

Check local tools:

kubectl version --client
helm version
just --version

Cluster components:

  • cert-manager
  • CloudNativePG 1.26+; use 1.29+ for new installs
  • Barman Cloud CNPG-I Plugin installed in the same namespace as the CloudNativePG operator
  • Valkey operator (valkey-operator) for the durable download queue

Install cert-manager:

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.20.2 \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --wait

kubectl rollout status deployment/cert-manager -n cert-manager
kubectl rollout status deployment/cert-manager-webhook -n cert-manager
kubectl rollout status deployment/cert-manager-cainjector -n cert-manager

Install CloudNativePG:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.29/releases/cnpg-1.29.0.yaml

kubectl rollout status deployment/cnpg-controller-manager -n cnpg-system

Install the Barman Cloud CNPG-I Plugin:

kubectl get deployment -n cnpg-system cnpg-controller-manager \
  -o jsonpath="{.spec.template.spec.containers[*].image}{'\n'}"

kubectl apply -f \
  https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.12.0/manifest.yaml

kubectl rollout status deployment/barman-cloud -n cnpg-system

Install the Valkey operator (backs the durable download queue):

helm repo add valkey https://valkey.io/valkey-helm/
helm repo update

helm install valkey-operator valkey/valkey-operator \
  -n valkey-operator-system --create-namespace

kubectl get pods -n valkey-operator-system -l app.kubernetes.io/instance=valkey-operator

Final checks:

kubectl get crd certificates.cert-manager.io
kubectl get crd clusters.postgresql.cnpg.io
kubectl get crd objectstores.barmancloud.cnpg.io
kubectl get crd valkeyclusters.valkey.io
kubectl get deploy -n cert-manager
kubectl get deploy -n cnpg-system
kubectl get deploy -n valkey-operator-system

2. Namespace

export NAMESPACE=dev
kubectl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -

Use NAMESPACE=prod for production.

3. Required Secrets

Create these once. These commands are safe to re-run because they render YAML client-side and apply it.

kubectl -n "${NAMESPACE}" create secret generic telegram-bot-api \
  --from-literal=api_id='<telegram api id>' \
  --from-literal=hash='<telegram api hash>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n "${NAMESPACE}" create secret generic db \
  --from-literal=username='admin' \
  --from-literal=password='<db password>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n "${NAMESPACE}" create secret generic db-superuser \
  --from-literal=username='postgres' \
  --from-literal=password='<db superuser password>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n "${NAMESPACE}" create secret generic s3 \
  --from-literal=access-key-id='rustfsadmin' \
  --from-literal=secret-access-key='<at least 8 characters>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n "${NAMESPACE}" create secret generic valkey \
  --from-literal=password='<valkey password>' \
  --dry-run=client -o yaml | kubectl apply -f -

The s3 Secret is used by the internal RustFS backup store and CNPG backup plugin. The chart creates a single-node RustFS instance and bootstraps the backups bucket automatically.

4. Config Files

Create local config files if they do not exist:

cp -n configs/config.example.toml configs/config.toml
cp -n configs/downloader.example.toml configs/downloader.toml
cp -n configs/cookie_assignment.example.toml configs/cookie_assignment.toml

Edit these files before installing:

  • configs/config.toml
  • configs/downloader.toml
  • configs/cookie_assignment.toml

Required config checks:

  • configs/config.toml has the real Telegram bot token.
  • Database credentials in configs/config.toml match the db Secret.
  • Normal downloader token from configs/config.toml [download].node_token is listed in configs/downloader.toml [auth].node_tokens.
  • Cookie-manager token matches in configs/downloader.toml [auth].cookie_manager_token and configs/cookie_assignment.toml [download].cookie_manager_token.
  • configs/config.toml [redis].host matches the Valkey service name (valkey with the bundled operator; confirm with kubectl get svc -n "${NAMESPACE}" after the ValkeyCluster reconciles).
  • configs/config.toml [redis].user is admin and [redis].password matches the valkey Secret's password.
  • Default in-cluster service URLs are correct if all charts are installed into the same namespace.

5. Optional Cookies

Cookie files are optional. Put them here if you have them:

cookies/<domain>/<cookie-id>.txt

The sync helper also creates an empty cookie Secret when no cookie files exist.

6. Install

Install order matters:

just helm-install-infra "${NAMESPACE}"

just k8s-update-bot-config "${NAMESPACE}"
just helm-install-bot "${NAMESPACE}"

just k8s-migration "${NAMESPACE}"

just k8s-update-downloader-config "${NAMESPACE}"
just helm-install-downloader "${NAMESPACE}"

just k8s-update-cookie-assignment-config "${NAMESPACE}"
just k8s-sync-cookie-assignment-cookies "${NAMESPACE}"
just helm-install-cookie-assignment "${NAMESPACE}"

Notes:

  • infra creates the shared internal CA issuer.
  • bot creates PostgreSQL, RustFS, Telegram Bot API, yt-toolkit, and the bot Deployment.
  • k8s-migration runs DB migrations after PostgreSQL exists.
  • The bot pod can restart until PostgreSQL is ready and migrations have run.
  • downloader creates the headless downloader service and worker pods.
  • cookie-assignment distributes cookies to downloader nodes and is safe with an empty cookie inventory.

7. Adding Another Bot

Another bot can reuse the same downloader nodes without using the Telegram bot crate.

  • Create a new bot crate, for example bot_discord, and depend on downloader_client directly.
  • Create a separate chart for the new bot, usually by copying charts/bot only as a starting point and removing Telegram-specific resources.
  • Give the new bot its own config Secret and client TLS certificate.
  • Put the new bot's normal downloader token in its bot config, then add the same token to downloader config [auth].node_tokens.
  • Do not give the new bot cookie_manager_token; only cookie_assignment should have that token.
  • Keep cookie distribution in cookie_assignment; do not add cookie assignment logic to a bot.
  • PostgreSQL, RustFS, migrations, and upload cache are optional for another bot. A simple bot can use downloader nodes without any cache.
  • If another bot has a cache, design that cache for that messenger. Do not blindly reuse the Telegram file_id cache model.

8. Verify

kubectl get pods -n "${NAMESPACE}"
kubectl get certificates -n "${NAMESPACE}"
kubectl get clusters.postgresql.cnpg.io -n "${NAMESPACE}"
kubectl get backups.postgresql.cnpg.io -n "${NAMESPACE}"
kubectl get scheduledbackups.postgresql.cnpg.io -n "${NAMESPACE}"
kubectl get objectstores.barmancloud.cnpg.io -n "${NAMESPACE}"
kubectl get valkeyclusters.valkey.io -n "${NAMESPACE}"
kubectl get secret -n "${NAMESPACE}" telegram-bot-api db db-superuser s3 valkey bot-config downloader-config cookie-assignment-config cookie-assignment-cookies

Logs:

just k8s-logs-bot "${NAMESPACE}"
just k8s-logs-downloader "${NAMESPACE}"
just k8s-logs-cookie-assignment "${NAMESPACE}"

Manual backup check:

kubectl -n "${NAMESPACE}" delete backup postgres-manual-backup --ignore-not-found

cat > /tmp/postgres-manual-backup.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: postgres-manual-backup
spec:
  cluster:
    name: postgres
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io
EOF

kubectl -n "${NAMESPACE}" create -f /tmp/postgres-manual-backup.yaml
kubectl -n "${NAMESPACE}" get backup postgres-manual-backup -w

How Barman backup config is applied

This project does not require you to manually set Barman environment variables inside the PostgreSQL pod.

The bot chart applies backup configuration in three steps:

  1. The s3 Secret provides credentials:
    • access-key-id
    • secret-access-key
  2. charts/bot/templates/postgres-backup-object-store.yaml creates a barmancloud.cnpg.io/v1 ObjectStore using:
    • s3.endpointURL
    • s3.destinationPath
    • credentials from the s3 Secret
  3. charts/bot/templates/postgres-cluster.yaml attaches that object store to the postgres cluster through spec.plugins, and charts/bot/templates/postgres-scheduled-backup.yaml creates the scheduled backup.

Current defaults from charts/bot/values.yaml:

  • object store name: postgres-backup-store
  • destination path: s3://backups/
  • endpoint URL: http://s3:9000
  • backup schedule: 0 0 3 * * *

To inspect the applied backup config:

kubectl -n "${NAMESPACE}" get objectstores.barmancloud.cnpg.io postgres-backup-store -o yaml
kubectl -n "${NAMESPACE}" get cluster postgres -o yaml
kubectl -n "${NAMESPACE}" get scheduledbackup postgres-backup -o yaml
kubectl -n "${NAMESPACE}" get secret s3 -o yaml

If you change backup credentials or endpoint settings:

  1. Update the s3 Secret.
  2. Update Helm values if endpointURL, destinationPath, or object-store naming changed.
  3. Run:
just helm-upgrade-bot "${NAMESPACE}"

Use shell exports only for manual S3 checks from your machine or a debug container, for example with mc or aws CLI:

export AWS_ACCESS_KEY_ID='<value from s3 secret access-key-id>'
export AWS_SECRET_ACCESS_KEY='<value from s3 secret secret-access-key>'
export AWS_ENDPOINT_URL='http://s3:9000'

Those exports are not consumed by the PostgreSQL pod directly in this setup; they are only for your manual tooling.

How to use the backups

In this setup, backups are used for recovery, not for direct browsing from PostgreSQL.

The normal restore flow is:

  1. Keep the original postgres cluster untouched.
  2. Create a new cluster, for example postgres-restore.
  3. Bootstrap that new cluster from the backup object store.
  4. Connect to the restored cluster and inspect or export data from it.

Important:

  • Recovery is not in-place. Do not try to “restore over” the running postgres cluster.
  • The backup object store keeps base backups and WAL archives.
  • The restored cluster does not need to archive WALs back to the bucket unless you explicitly configure that.

Example restore manifest for this project:

cat > /tmp/postgres-restore.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-restore
spec:
  instances: 1
  storage:
    size: 3Gi
  superuserSecret:
    name: db-superuser
  bootstrap:
    recovery:
      source: origin
  externalClusters:
    - name: origin
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: postgres-backup-store
          serverName: postgres
EOF

kubectl apply -n "${NAMESPACE}" -f /tmp/postgres-restore.yaml
kubectl get clusters.postgresql.cnpg.io -n "${NAMESPACE}"
kubectl get pods -n "${NAMESPACE}" -l cnpg.io/cluster=postgres-restore -w

When the restored cluster is ready, connect to it:

export RESTORE_POD="$(kubectl get pod -n "${NAMESPACE}" \
  -l cnpg.io/cluster=postgres-restore,role=primary \
  -o jsonpath='{.items[0].metadata.name}')"

kubectl exec -it -n "${NAMESPACE}" "${RESTORE_POD}" -- psql -U postgres -d api

Then you can:

  • inspect tables
  • run queries
  • export data with pg_dump
  • compare restored data with the live cluster

Example pg_dump inside the restored pod:

kubectl exec -it -n "${NAMESPACE}" "${RESTORE_POD}" -- \
  pg_dump -U postgres -d api > /tmp/postgres-restore.sql

If you need point-in-time recovery instead of the latest state, add a recovery target:

bootstrap:
  recovery:
    source: origin
    recoveryTarget:
      targetTime: "2026-04-16T21:55:00Z"

If you no longer need the restored cluster:

kubectl delete cluster postgres-restore -n "${NAMESPACE}"

Restore from an external SQL dump

This is a different flow from Barman recovery.

Use this when you already have a dump file outside CNPG/Barman, for example from pg_dump.

Use a temporary cluster if possible. Do not import unknown dumps directly into the live postgres cluster.

Plain SQL dump:

kubectl exec -i -n "${NAMESPACE}" -c postgres "${RESTORE_POD}" -- \
  psql -U postgres -d api < /path/to/dump.sql

Gzipped SQL dump:

gunzip -c /path/to/dump.sql.gz | \
kubectl exec -i -n "${NAMESPACE}" -c postgres "${RESTORE_POD}" -- \
  psql -U postgres -d api

Custom-format dump created with pg_dump -Fc:

cat /path/to/dump.dump | \
kubectl exec -i -n "${NAMESPACE}" -c postgres "${RESTORE_POD}" -- \
  pg_restore -U postgres -d api --clean --if-exists

Notes:

  • plain .sql dumps are restored with psql
  • custom-format dumps are restored with pg_restore

Operations

Upgrade charts:

just helm-upgrade-infra "${NAMESPACE}"
just helm-upgrade-bot "${NAMESPACE}"
just helm-upgrade-downloader "${NAMESPACE}"
just helm-upgrade-cookie-assignment "${NAMESPACE}"

Refresh configs:

just k8s-update-bot-config "${NAMESPACE}"
just k8s-update-downloader-config "${NAMESPACE}"
just k8s-update-cookie-assignment-config "${NAMESPACE}"

Refresh cookies:

just k8s-sync-cookie-assignment-cookies "${NAMESPACE}"
just k8s-rollout-cookie-assignment "${NAMESPACE}"

Scale downloader nodes:

just scale-downloader "${NAMESPACE}" 3

Run migrations:

just k8s-migration "${NAMESPACE}"

Run a different migration command:

just k8s-migration "${NAMESPACE}" down

Dev Images

Use this when deploying images built from your local checkout.

just docker-push-dev-bot
just docker-push-dev-downloader
just docker-push-dev-cookie-assignment
just docker-push-dev-migration

Each script builds with Docker buildx, pushes to the registry, uses remote cache, and defaults to dev-<git-short-sha>.

  • Bot, downloader, and cookie-assignment scripts print the next helm upgrade command after a successful push.
  • The migration script builds from deployment/Dockerfile.migration.dev and prints the next just k8s-migration command with IMAGE_REPO, IMAGE_TAG, and PULL_POLICY=Always.

Useful overrides:

  • NAMESPACE
  • IMAGE_REPO
  • IMAGE_TAG
  • MIGRATION_COMMAND
  • CACHE_REF
  • PLATFORM
  • DOCKERFILE
  • CONTEXT_DIR
  • BUILDER_NAME

Example:

export NAMESPACE=dev
export IMAGE_TAG=dev-manual-1

IMAGE_TAG="${IMAGE_TAG}" just docker-push-dev-bot
IMAGE_TAG="${IMAGE_TAG}" just docker-push-dev-downloader
IMAGE_TAG="${IMAGE_TAG}" just docker-push-dev-cookie-assignment
IMAGE_TAG="${IMAGE_TAG}" just docker-push-dev-migration

If only one component changed, rebuild and upgrade only that component.