lake: expand TiDB Cloud Lake overview #23932
+66
−12
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -13,18 +13,71 @@ TiDB Cloud Lake supports ANSI SQL, semi-structured data processing, vector searc | |||||
| > | ||||||
| > TiDB Cloud Lake is currently in **public preview**. Feature availability and service limits might change as we continue to improve the product. | ||||||
|
|
||||||
| ## Why {{{ .lake }}}? {#why-lake} | ||||||
|
|
||||||
| {{{ .lake }}} brings analytics, data engineering, search, and AI workloads together in one cloud-native platform. It combines independently scalable compute, object-storage-based data management, SQL access, and built-in multimodal capabilities in a managed service. | ||||||
|
|
||||||
| ### Scale compute independently from storage | ||||||
|
|
||||||
| {{{ .lake }}} separates compute from storage. Data is stored in durable, cost-effective object storage, while warehouses provide independently managed compute resources. You can choose a warehouse size for each workload, resize it as demand changes, and use separate warehouses for data loading and query execution. This architecture lets you scale compute without moving or duplicating the underlying data. | ||||||
|
|
||||||
| For more information, see [TiDB Cloud Lake Architecture](/tidb-cloud-lake/guides/tidb-cloud-lake-architecture.md) and [Warehouses](/tidb-cloud-lake/guides/warehouse.md). | ||||||
|
|
||||||
| ### Keep costs transparent and predictable | ||||||
|
|
||||||
| The main cost components are straightforward: | ||||||
|
|
||||||
| - **Warehouse compute** is billed per second while a warehouse is running. By default, a warehouse automatically suspends after five minutes of inactivity. A suspended warehouse does not consume compute resources. | ||||||
|
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: rg -n -i 'AUTO_SUSPEND|auto.?suspend|suspend.*warehouse|warehouse.*suspend|inactive.*cluster|release.*cluster' tidb-cloud-lake
sed -n '20,36p' tidb-cloud-lake/lake-overview.mdRepository: pingcap/docs Length of output: 12397 🏁 Script executed: printf '%s\n' '--- create warehouse SQL reference ---'
sed -n '24,40p' tidb-cloud-lake/sql/create-warehouse.md
printf '%s\n' '--- warehouse guide ---'
sed -n '34,52p' tidb-cloud-lake/guides/warehouse.md
sed -n '128,150p' tidb-cloud-lake/guides/warehouse.md
printf '%s\n' '--- lake architecture ---'
fd -i '.*lake.*architecture.*' tidb-cloud-lake
for f in $(fd -i '.*lake.*architecture.*' tidb-cloud-lake); do
sed -n '1,180p' "$f"
done
printf '%s\n' '--- pricing and overview references ---'
sed -n '26,38p' tidb-cloud-lake/guides/pricing-billing.md
sed -n '26,33p' tidb-cloud-lake/lake-overview.mdRepository: pingcap/docs Length of output: 13392 🤖 get_repo_knowledge executed:
Length of output: 1584 Use the documented warehouse auto-suspend default. The warehouse reference defines the default Committable replacement-**Warehouse compute** is billed per second while a warehouse is running. By default, a warehouse automatically suspends after five minutes of inactivity. A suspended warehouse does not consume compute resources.
+**Warehouse compute** is billed per second while a warehouse is running. By default, a warehouse automatically suspends after ten minutes of inactivity. A suspended warehouse does not consume compute resources.📝 Committable suggestion
Suggested change
|
||||||
| - **Storage** is billed according to the amount of data stored in object storage. | ||||||
| - **Cloud services** are billed according to API request usage. | ||||||
| - **Service hosting** for data integration services is billed per second while they are running. | ||||||
|
|
||||||
| Administrators can set a monthly spending limit. You can monitor usage and billing history. For details, see [TiDB Cloud Lake Pricing & Billing](/tidb-cloud-lake/guides/pricing-billing.md) and [Managing Costs](/tidb-cloud-lake/guides/manage-costs.md). | ||||||
|
|
||||||
| ### Run high-performance analytics at scale | ||||||
|
|
||||||
| {{{ .lake }}} is designed for analytical workloads such as interactive dashboards, ad hoc exploration, large-scale scans, and concurrent queries. Its optimizer uses techniques such as predicate pushdown, join reordering, and scan pruning to reduce unnecessary work. | ||||||
|
|
||||||
| For workload-specific optimization, you can use cluster keys to organize related data into adjacent storage blocks, materialized views to persist recurring query results, and specialized indexes and result caching to accelerate common access patterns. The appropriate strategy depends on your query filters, data distribution, and update patterns. | ||||||
|
|
||||||
| For more information, see [Performance Optimization](/tidb-cloud-lake/guides/performance-optimization.md), [Cluster Key](/tidb-cloud-lake/guides/cluster-key-performance.md), and [Materialized View](/tidb-cloud-lake/sql/materialized-view.md). | ||||||
|
|
||||||
| ### Build data pipelines in the platform | ||||||
|
|
||||||
| {{{ .lake }}} provides built-in primitives for data ingestion and incremental processing: | ||||||
|
|
||||||
| - [Stages](/tidb-cloud-lake/guides/stage-overview.md) provide managed locations and interfaces for loading, querying, and unloading files. | ||||||
| - [Streams](/tidb-cloud-lake/guides/track-and-transform-data-via-streams.md) capture table changes for incremental processing. | ||||||
| - [Tasks](/tidb-cloud-lake/guides/automate-data-loading-with-tasks.md) run SQL on a schedule or when a stream contains new rows. | ||||||
| - [Data integration](/tidb-cloud-lake/guides/data-integration-overview.md) provides a visual interface for importing or continuously synchronizing data from supported external systems. | ||||||
|
|
||||||
| Together, these capabilities support batch ingestion, change data capture (CDC), incremental ETL, scheduled transformations, and downstream analytical tables without requiring every pipeline to rely on separate orchestration tooling. | ||||||
|
|
||||||
| ### Replicate TiDB Cloud data continuously | ||||||
|
|
||||||
| TiDB Cloud Data Pipeline replicates full and incremental data from a TiDB Cloud instance to {{{ .lake }}} without requiring a third-party ETL tool. It first exports a full snapshot of the selected tables, and then continuously replicates row changes, including inserts, updates, and deletes, so that analytical data stays current. | ||||||
|
|
||||||
| You can use this integration to: | ||||||
|
|
||||||
| - Replicate operational TiDB data to an analytical warehouse. | ||||||
| - Keep dashboards and reports up to date. | ||||||
| - Make continuously updated data available to analytics, search, vector, and AI workloads in {{{ .lake }}}. | ||||||
|
|
||||||
| Data Pipeline uses TiCDC for incremental replication and an external stage to transfer data between TiDB Cloud and {{{ .lake }}}. Availability and restrictions depend on your TiDB Cloud plan. For details, see [Data Pipeline to TiDB Cloud Lake](/tidb-cloud/data-pipeline/data-pipeline-overview.md). | ||||||
|
|
||||||
| ### Work with structured and semi-structured data | ||||||
|
|
||||||
| You can use SQL to store, query, clean, and transform relational and semi-structured data in the same platform. The `VARIANT` data type preserves nested **JSON** structures, while JSON path expressions let you access nested fields, virtual columns accelerate frequently queried paths, and inverted indexes support text search. | ||||||
|
|
||||||
| This is useful for application events, logs, user activity data, API payloads, and AI agent traces, which often have evolving schemas and can be costly to flatten up front. | ||||||
|
|
||||||
| To protect sensitive data, you can apply row access policies (an experimental feature) to filter rows at query time and masking policies to redact column values, including selected keys in `VARIANT` data. For more information, see [JSON & Search](/tidb-cloud-lake/guides/json-search.md), [Row Access Policy](/tidb-cloud-lake/guides/row-access-policy.md), and [Masking Policy](/tidb-cloud-lake/guides/masking-policy.md). | ||||||
|
Member
Author
There was a problem hiding this comment. @awxxxxxx Since row access policy is still experimental, please help confirm whether we should keep it here or remove it and adjust wording? |
||||||
|
|
||||||
| ### Combine analytics, search, and AI workloads | ||||||
|
|
||||||
| Full-text search, [vector search](/tidb-cloud-lake/guides/vector-search-guide.md), [geospatial analysis](/tidb-cloud-lake/guides/geo-analytics.md), and SQL analytics run on the same data platform. You can use [full-text indexes](/tidb-cloud-lake/guides/full-text-index.md) (inverted indexes) for keyword-oriented retrieval, vector indexes for semantic similarity search, and SQL predicates and joins to combine retrieval results with structured business data. | ||||||
|
|
||||||
| This unified approach supports use cases such as product and business analytics, application-event analysis, search and recommendations, retrieval-augmented generation (RAG), and AI-agent trace analysis without requiring a separate data copy for each workload. For an end-to-end example, see [Multimodal Data Analytics](/tidb-cloud-lake/guides/multimodal-data-analytics.md). | ||||||
|
|
||||||
| ## Get Started | ||||||
|
|
||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@awxxxxxx PTAL