We can remove all custom scripts and Dockerfile instructions

[sudhirj]

We're currently adding scripts to do two things:

• check the mount size and write it to the $CACHE_SIZE variable: this might not be required, NGINX can fill the disk and then manage itself after that. See https://www.nginx.com/blog/nginx-caching-guide/ under max_size.

• periodically resolve the other nodes in the same app, update the nginx.conf and reload NGINX: we can use the DNS resolver and variables to achieve this inside normal NGINX with no extra scripts or reloading. See https://www.nginx.com/blog/dns-service-discovery-nginx-plus/ in the "Setting the Domain Name in a Variable" section.

If we drop the scripts we're maintaining to do this, then we can drop other dependencies and hivemind as well. The repo then becomes the example fly.toml, the example nginx.conf and a Dockerfile that just reads:

FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf

and that's it.

[sudhirj]

@jsierles @mrkurt any thoughts / objections to doing this?

[jsierles]

Unfortunately, DNS discovery is only supported by Nginx Plus, their commercial product. This is one of the reasons that I tend to prefer using Varnish over nginx for HTTP caching. The other reason is being able to perform cache purges on wildcard paths. Both of these require commercial versions of Nginx.

[sudhirj]

Proper DNS discovery is only a Plus feature, but

resolver 10.0.0.2 valid=10s;

server {
    location / {
        set $backend_servers backends.example.com;
        proxy_pass http://$backend_servers:8080;
    }
}

is a hack-ish way to do that job that should work in the free version, I'm testing to confirm.

Selective purging seems to be a Plus feature, but we're not really promising that to anyone right now. I'm asking more from the p.o.v of simplifying the repo and the required configs.

[jsierles]

Ah nice - I remember this now. DNS sounds like a good change if it works!

For caching, the text is not too clear here on whether it will involve the cache manager even without setting a limit. I'd guess it would though.

When the cache size reaches the limit, a process called the cache manager removes the files that were least recently used to bring the cache size back under the limit.

[jsierles]

IIRC this may only have worked with resolver ipv6=on

[sudhirj]

Think I saw that ipv6 is on by default, testing anyway.

Harder to test the cache limit / max_size but the nginx.conf needs to be changed by whoever is setting up the volumes anyway — so they could just set it up without an automated script either way.

Will also see if I can do a test, will fill most of the disk with junk and see how the cache is managed.