Delivery attempt failed, reason: "context deadline exceeded"

[cuu508]

Describe the bug

I'm running maddy 0.9.4, and noticed that about 12 hours ago deliveries to @hotmail.com addresses started to fail with:

May 11 06:34:29 maddy[286019]: submission: RCPT ok        {"msg_id":"71eef592","rcpt":"[redacted]@hotmail.com"}
May 11 06:34:29 maddy[286019]: submission: accepted        {"msg_id":"71eef592"}
May 11 06:34:36 maddy[286019]: delivery attempt failed        {"msg_id":"71eef592","rcpt":"[redacted]@hotmail.com","reason":"context deadline exceeded"}
May 11 06:34:36 maddy[286019]: will retry        {"attempts_count":{"[redacted]@hotmail.com":1},"msg_id":"71eef592","next_try_delay":"14m59.99999862s","rcpts":["[redacted]@hotmail.com"]}
May 11 06:49:41 maddy[286019]: delivery attempt failed        {"msg_id":"71eef592","rcpt":"[redacted]@hotmail.com","reason":"context deadline exceeded"}
May 11 06:49:41 maddy[286019]: will retry        {"attempts_count":{"[redacted]@hotmail.com":2},"msg_id":"71eef592","next_try_delay":"14m59.999998479s","rcpts":["[redacted]@hotmail.com"]}
[...]

The mesage would go back in the queue, and subsequent attempts would fail as well. In the logs there were no other interesting bits, no error message from the remote mail server.

After restarting maddy, the "context deadline exceeded" errors ceased, and emails to @hotmail.com addresses started to go out again. This indicates to me that there's potentially something wrong on the maddy side – it somehow got itself in a bad state it could not recover from.

Unfortunately I do not have more specific details for now. I realize that from what I described it may be impossible to figure out what precisely went wrong. But I want to report what I have for now anyway, and I will update if I find out more. I've now enabled debug logging in maddy configuration, and will look out for any reoccurences of the "context deadline exceeded" error.

Steps to reproduce

Unfortunately I do not know yet. The server has been running fine for a long time, this is the first time I've noticed something like this. Perhaps there's some weirdness from hotmail.com mail servers that triggered a bug in maddy, but no idea what it is.

Log files

See above

Configuration file

$(public_ip) = 1.2.3.4

hostname mail.example.org

openmetrics tcp://127.0.0.1:9749 { }

tls {
    loader acme {
        hostname mail.example.org
        email contact@example.org
        agreed
        challenge dns-01
        dns cloudflare {
            api_token "secret"
        }
    }
}

msgpipeline webhook {
    default_destination {
        check {
            command /usr/bin/curl -fsS -o /dev/null --data-binary @- https://example.org/hanbdle_bounce/
            command /usr/bin/cp /dev/stdin bounces/{msg_id}.txt
        }

        deliver_to dummy
    }
}

smtp tcp://$(public_ip):25 {
    limits {
        # Up to 20 msgs/sec across max. 20 SMTP connections.
        all rate 20 1s
        all concurrency 20
    }

    destination mail.example.org {
        deliver_to &webhook
    }

    default_destination {
        reject 551 5.7.1 "Delivery not authorized, message refused"
    }
}

submission tls://$(public_ip):465 {
    limits {
        # Up to 50 msgs/sec across any amount of SMTP connections.
        all rate 50 1s
    }

    auth pass_table static {
        entry smtpuser "secret"
    }

    modify {
        # Bounces should be delivered to us, not the main MX 
        replace_sender regexp "(.+)@example.org" "$1@mail.example.org"
    }

    source mail.example.org {
        default_destination {
            modify {
                dkim {
                    domains example.org
                    selector mail
                    sign_subdomains yes
                }
            }
            deliver_to &remote_queue
        }
    }

    default_source {
        reject 551 5.7.1 "Bad from address, we only accept email from example.org"
    }
}

target.queue remote_queue {
    # debug true
    target &outbound_delivery
    # retry for ~32 hours:
    max_tries 17
    max_parallelism 100

    autogenerated_msg_domain mail.example.org
    bounce {
        deliver_to &webhook
    }
}

target.remote outbound_delivery {
    # debug true
    local_ip $(public_ip)
    force_ipv4 true
    connect_timeout 1m
    conn_max_idle_time 30
    limits {
        # Up to 20 msgs/sec across max. 20 SMTP connections for each recipient domain.
        destination rate 20 1s
        destination concurrency 20
    }
    mx_auth {
        dane
        mtasts
    }
}

Environment information

• maddy version: 0.9.4 linux/amd64 go1.26.2