Skip to content

Commit c441423

Browse files
committed
add more info
1 parent b4baaaf commit c441423

3 files changed

Lines changed: 9 additions & 5 deletions

File tree

docs/ingestion/data-formats.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -752,8 +752,8 @@ FROM "wikiticker"
752752
This query returns:
753753

754754
| `kafka.header.env` | `kafka.key` | `kafka.partition` | `kafka.offset` | `kafka.timestamp` | `kafka.topic` |
755-
|--------------------|-------------|-------------------|----------------|-------------------|---------------|
756-
| `development` | `wiki-edit` | `0` | `12345` | `1680795276351` | `wiki-edits` |
755+
|--------------------|-----------|-------------------|----------------|---------------|---------------|
756+
| `development` |`wiki-edit`|`0`|`12345`| `1680795276351`| `wiki-edits` |
757757

758758
### Kinesis
759759

docs/ingestion/ingestion-spec.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -188,9 +188,13 @@ Treat `__time` as a millisecond timestamp: the number of milliseconds since Jan
188188
The `dimensionsSpec` is located in `dataSchema``dimensionsSpec` and is responsible for
189189
configuring [dimensions](./schema-model.md#dimensions).
190190

191-
You can either manually specify the dimensions or take advantage of schema auto-discovery where you allow Druid to infer all or some of the schema for your data. This means that you don't have to explicitly specify your dimensions and their type.
191+
You can either manually specify the dimensions or take advantage of type-aware schema auto-discovery where you allow Druid to infer all or some of the schema for your data. This means that you don't have to explicitly specify your dimensions and their type.
192192

193-
To use schema auto-discovery, set `useSchemaDiscovery` to `true`.
193+
:::caution
194+
When using type-aware schema auto-discovery, Druid discovers the type for all dimensions unless you explicitly specify dimensions for Druid to discover or list dimensions in the exclusion list. This helps you control storage costs by preventing Druid from ingesting dimensions unintentionally.
195+
:::
196+
197+
To use type-aware schema auto-discovery, set `useSchemaDiscovery` to `true`.
194198

195199
Alternatively, you can use the string-based schemaless ingestion where any discovered dimensions are treated as strings. To do so, leave `useSchemaDiscovery` set to `false` (default). Then, set the dimensions list to empty or set the `includeAllDimensions` property to `true`.
196200

docs/ingestion/schema-design.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -254,7 +254,7 @@ You can have Druid infer the schema and types for your data partially or fully b
254254
Before you use type-aware schema discovery, keep the following in mind:
255255

256256
- There maybe impact on downstream BI tools depending on how they handle ARRAY-typed columns.
257-
- You may ingest more columns than you intend if you don't maintain an exclusion list. For example, if you use type-aware schema discovery and the Kafka input format, you'll ingest optional dimensions like the Kafka offset and partition unless you add them to the exclusion list.
257+
- You may ingest more columns than you intend if you don't maintain an exclusion list. For example, if you use type-aware schema discovery and the Kafka input format, you'll ingest optional dimensions like the Kafka offset and partition unless you add them to the exclusion list or explicitly list the columns you want Druid to discover the type for.
258258

259259
When performing type-aware schema discovery, Druid can discover all the columns of your input data (that are not present in
260260
the exclusion list). Druid automatically chooses the most appropriate native Druid type among `STRING`, `LONG`,

0 commit comments

Comments
 (0)