Coalesce socket write when sending data with HTTP/2

[nickva]

Previously we wrote each element of the accumulator separately with one send() call for every DATA frame header with the file bytes in between and in ranch ssl sendfile fallback we read the file in 8KB chunks and called ssl:send() on each so for a 100KB file it might call ssl:send() about 30 or so times.

To speed things up gather up to 64KB chunks before calling Transport:send(). Also, avoid re-opening and closing the file repeatedly, instead cache the opened Fds. If the file cannot be opened, read or get truncated unexpectedly we terminate the connection with a GOWAY INTERNAL_ERROR.

If there is TCP sendfile support then we don't do any of this and let sendfile handle it with zero-copy support.

This was inspired from the Erlang forums post [1] comparing a variety of web servers and noticing the the same optimization could apply to cowboy as well.

With h2load on macos (intel), otp27 with 8 connections, 32 streams, 3s runs: got a decent improvemnt in file uploads:

    100KB:  1014 -> 2632 req/s (2.6x), ssl:send 29 -> 5
    1MB:     116 ->  358 req/s (3.1x), ssl:send 259 -> 19
    10KB:   4901 -> 5715 req/s (1.1x), ssl:send 6 -> 4

[1] https://erlangforums.com/t/livery-high-performance-http-1-1-http-2-http-3-server-for-erlang-otp-27/5693/5

[essen]

Thanks for the PR.

ranch_transport:sendfile has an option to set the chunk size, it could be set to 64KB without changing anything else. I don't think we gain much from caching fds.

Sending the frame header and data in one call likely is valuable but mostly for smaller files. The higher the file size the less likely the extra call is going to matter. That said this could probably be solved by a ranch_transport:sendfile option that has header data to be sent along with the first call.

If my suggestions prove correct then we don't need most of this extra code.

[nickva]

Sorry the the delayed response. I tried it on again on another macos machine with [{chunk_size, 65536}] for the transport in the mix. It did speed things but up to 1.5x only. With the PR it went much faster (4x)

100 KB (req/sec )

master : 1056
chunk_size 64 KB : 1496 (1.42x)  
PR  : 4232 (4.01x) 

1 MB (req/sec)

master  : 109
chunk_size 64 KB : 173 (1.59x)
this PR : 511 (4.67x)

With HTTP/2 the file is split up in frames so each one is called as a separate Transport:sendfile call it seems. Each call to send then is doing an open/close and a header send. For HTTP/2 at least a larger file wouldn't amortize that much because of HTTP/2 framing.

Maybe Transport (ranch) can have something like sendv or send_multi(Sock, [iodata | {sendfile, Path, Offset, Bytes}]) so we can hand it a whole accumulator? Then ranch can group them, re-use file open handles or batch them, maybe send a bunch of them in one ssl:send() call like here