Fix connection recycling with stalled HTTP 1.1 streams

[sbSteveK]

The HttpClientConnectionManager using manual_window_management = true was able to release a connection back to the pool while a HTTP 1.1 stream was still in progress. e.g. the stream window filled to zero and the user never called update_window. The connection manager would be able to vend the same connection for a new request but because HTTP 1.1 processes streams in FIFO order, the new stream's response callback would never fire and be permanently blocked waiting for the stalled stream to complete. This caused requests to hang indefinitely after a previous request on the same connection was abandoned with an unconsumed response body.

This can be seen as a user error but we should handle it.

A num_streams_in_progress counter is added to aws_h1_connection.synced_data to track client streams that have been submitted but not yet completed. The counter is incremented when a stream is activated and decremented in the existing critical section of s_stream_complete()

An internal aws_http_connection_is_connection_idle() function is introduced, backed by a connection vtable function and wired through the connection manager's system vtable. For H1 connections, this returns false when num_streams_in_progress > 0. For H2 connections, it unconditionally returns true (H2 multiplexes streams independently and uses GOAWAY for lifecycle management).

The connection manager release path now checks both new_requests_allowed (connection alive/not errored) and is_connection_idle (no orphaned streams). If the connection is not idle, it is destroyed instead of recycled, preventing the stall.

Tests:
Check that new requests are allowed during pipelining.
Check that is_connection_idle returns false during active/stalled streams and true after completion.

CI:
split out the downstream ci job into its own workflow to allow us to retry the significantly shorter running jobs if they failed a flaky test.
fail-fast on matrix CI jobs to false so we can allow other jobs to continue running if one flaky test fails in one matrix element.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 65.38462% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.63%. Comparing base (8acf604) to head (db242fb).

Files with missing lines	Patch %	Lines
source/h2_connection.c	0.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #561      +/-   ##
==========================================
- Coverage   79.79%   79.63%   -0.16%     
==========================================
  Files          28       28              
  Lines       12085    12104      +19     
==========================================
- Hits         9643     9639       -4     
- Misses       2442     2465      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TingDaoK]

Overall, I am not against the idea when user releases the connection back to the pool, close it when there is still streams alive or even error it out.

If there is outstanding request/response and user release it back to the pool, it to me like freeing an object that is still actively in use. We can be proactive to either prevent it, or be nice to close the connection so that the requests will be closed as well.

[TingDaoK]

i don't know what this means...

I'd say either we raise error that it's not currently supported in http/2 or actually support this helper (not for the reason to control the lifetime from connection manager, but it is nice to have to check if the connection is idle or not)

[sbSteveK]

HTTP/2 doesn't run into the issue because it multiplexes streams (preventing a stream not completed from blocking the connection entirely) and uses GOAWAY (

aws-c-http/include/aws/http/connection.h

Lines 79 to 93 in 8acf604

	/**
	* Invoked when an HTTP/2 GOAWAY frame is received from peer.
	* Implies that the peer has initiated shutdown, or encountered a serious error.
	* Once a GOAWAY is received, no further streams may be created on this connection.
	*
	* @param http2_connection This HTTP/2 connection.
	* @param last_stream_id ID of the last locally-initiated stream that peer will
	* process. Any locally-initiated streams with a higher ID are ignored by
	* peer, and are safe to retry on another connection.
	* @param http2_error_code The HTTP/2 error code (RFC-7540 section 7) sent by peer.
	* `enum aws_http2_error_code` lists official codes.
	* @param debug_data The debug data sent by peer. It can be empty. (NOTE: this data is only valid for the lifetime of
	* the callback. Make a deep copy if you wish to keep it longer.)
	* @param user_data User-data passed to the callback.
	*/

) from peer to prevent a new_request_allowed() from returning true. It also uses the stream manager to check if it has streams assigned to it before allowing it to be released. Re-reading this description, it's pretty weird without context...

This function being added to the v-table was a result of needing to get this info from a testing perspective in our test framework. That meant having to add it to HTTP/2 and figured it'd be easiest to just make it a no-op.

It makes sense to allow it to return the actual idle state of an HTTP/2 connection instead of just returning true since it's here. I've added a best-effort check on the idle state but removed this from the being checked on HTTP/2 connections during the recycle check because HTTP/2 connections have different behavior that doesn't require it.