Headers conflict with Elastic-Api-Version on serverless

[kaisecheng]

Describe the bug

After upgrading the elasticsearch gem to v9, Logstash serverless tests started failing.

Elasticsearch::Client in v9.x sets compatible-with=9 in Content-Type and Accept headers via set_content_type!. When a client also sets the Elastic-Api-Version header (required for Elasticsearch Serverless), serverless rejects the request with HTTP 400:

The request includes both the [Elastic-Api-Version] header and a [compatible-with] parameter,
but it is not valid to include both of these in a request

These two versioning mechanisms serve different deployment models and are mutually exclusive on the server side:

• compatible-with=N: REST API compatibility for self-managed/hosted major version upgrades
• Elastic-Api-Version: 2023-10-31: date-based API versioning for serverless

The v9 client injects compatible-with=9 at initialization time (set_content_type!) unless the caller explicitly sets Content-Type and Accept headers. Callers that only set Elastic-Api-Version (as the Logstash plugins do for serverless) get the conflict.

Expected behavior

set_content_type! should not add compatible-with headers when Elastic-Api-Version is already present in transport_options[:headers].

Your Environment

• Ruby Version: JRuby 10.0.3.0 (Ruby 3.4.5)
• Elasticsearch client version: 9.4.1 / 9.4.2
• elastic-transport version: 8.5.2
• Elasticsearch: Serverless

Additional context

This affects Logstash plugins that connect to Elasticsearch Serverless:

• logstash-input-elasticsearch (v5.3.0)
• logstash-filter-elasticsearch (v4.4.0)

Both plugins recently upgraded their elasticsearch gem dependency from < 9 to < 10 (input PR#252, filter PR#213), which resolved to elasticsearch 9.x. Their serverless integration tests now fail: https://buildkite.com/elastic/logstash-serverless-integration-testing/builds/1151

The root cause is in Elasticsearch::Client#set_content_type! (elasticsearch.rb):

def set_content_type!(arguments)
  headers = {}
  user_headers = arguments&.[](:transport_options)&.[](:headers)
  unless user_headers&.keys&.detect { |h| h =~ /content-?_?type/i }
    headers['content-type'] = 'application/vnd.elasticsearch+json; compatible-with=9'
  end
  unless user_headers&.keys&.detect { |h| h =~ /accept/i }
    headers['accept'] = 'application/vnd.elasticsearch+json; compatible-with=9'
  end
  set_header(headers, arguments) unless headers.empty?
end

[picandocodigo]

@kaisecheng Patch versions 9.4.3 and 9.3.4 are out. Thanks!