InfluxDB 3 Enterprise Helm Chart: log level filtering per-node/component

[jstirnaman]

Problems

The customer wants to specify log-level filters for different Enterprise nodes/components--for example: debug for only the ingester node

InfluxDB 3 Enterprise chart supports one global filter

Reported by @mdevy-influxdata

logs:
  logFilter: debug

The documented logs.logFilter value correctly sets a global filter, but the chart has no supported per-component logFilter field. extraEnv is also global, so it cannot override only the ingester. A user would need to patch rendered StatefulSets outside normal chart configuration.

The chart writes LOG_FILTER to a shared ConfigMap. The ingester, querier, compactor, and processor pods all load that ConfigMap. Setting logs.logFilter therefore changes the log level for every component.

Note: The clustered chart exposes component-scoped filters, but the Enterprise chart distributes one LOG_FILTER value to every component pod.

• InfluxDB 3 Clustered supports component-specific filters.

  components:
    ingester:
      log:
        filters:
          - debug

The chart passes components.*.log.filters to the AppInstance resource.

README defect

The README’s logs.filter example is a documentation defect that could cause user error. The Enterprise README shows logs.filter, but the chart reads logs.logFilter. logs.filter has no effect.

[tomklapka]

ingester.extraEnv, and the same key on querier, compactor and processingEngine, is per component (chart 0.5.0 and later), and a container's own env wins over the shared ConfigMap. So this works on the current chart:

logs:
  logFilter: info
ingester:
  extraEnv:
    - name: INFLUXDB3_LOG_FILTER
      value: debug

Use INFLUXDB3_LOG_FILTER. A LOG_FILTER entry next to a global logs.logFilter loses: the server logs "both LOG_FILTER and INFLUXDB3_LOG_FILTER are set; using INFLUXDB3_LOG_FILTER" and the ingester stays at info. From 3.10 on, a debug filter also keeps influxdb3_wal, iox_query, datafusion and a few others at info; add INFLUXDB3_DISABLE_LOG_FILTER_NOISE_REDUCTION with value "true" as another extraEnv entry to lift that, and expect it to be loud.

Checked on kind with 3.11.2. The README's logs.filter example is wrong, fixed in #844.

[jstirnaman]

Feedback from Sales Engineering team is that extraEnv is a kind of workaround that isn't as elegant as our competitors'.

I'll gather more details.

[tomklapka]

Agreed - the extraEnv route was meant as a workaround the customer can use today, on the chart they already run. We can do this properly with a per-component logs.logFilter next to the global one:

logs:
  logFilter: info
ingester:
  logs:
    logFilter: debug

The component value wins over the global one, and the same key works for querier, compactor and processingEngine. A filter already set through extraEnv keeps working after the upgrade. Is the filter all the customer needs per component, or also format or destination?

[jstirnaman]

Is the filter all the customer needs per component, or also format or destination?

format and destination would be extra--we can ignore for now if they require considerable effort to add

[tomklapka]

Done in #848: ingester.logs.logFilter and the same block on querier, compactor and processingEngine, with logFormat and logDestination alongside since they cost nothing extra. The extraEnv workaround keeps working after the upgrade.