Limitation of closures returning futures

Author: nt54hamnghi

src/part-guide/more-async-await.md

@@ -133,9 +133,145 @@ A function which returns an async block is pretty similar to an async function.
 
 You would usually prefer the async function version since it is simpler and clearer. However, the async block version is more flexible since you can execute some code when the function is called (by writing it outside the async block) and some code when the result is awaited (the code inside the async block).
 
-
 ## Async closures
 
+Closures are anonymous functions that capture data from their environment. They provide expressiveness by allowing passing behavior as a value in functional style APIs, notable examples of which include various methods on `Iterator` (e.g., `filter`, `map`, etc.) or spawning a thread with `std::thread::spawn`.
+
+If a closure needs to await an async operation in its body, it has to return a future (just like async functions). A simple way is to return an async block from the closure, for example `|| async {}`. This often works, but because futures capture the data they use, closures returning futures don't work in all situations:
+
+```rust,norun
+#[tokio::main]
+async fn main() {
+    let mut logs = Vec::new();
+    run(|line| async {
+        logs.push(line);
+    });
+}
+
+async fn run<F, Fut>(f: F)
+where
+    F: FnMut(&str) -> Fut,
+    Fut: Future<Output = ()>,
+{
+    todo!()
+}
+```
+
+```
+error: captured variable cannot escape `FnMut` closure body
+ --> examples/async_closure.rs:6:12
+  |
+5 |       let mut logs = Vec::new();
+  |           -------- variable defined here
+6 |       run(|| async {
+  |  __________-_^
+  | |          |
+  | |          inferred to be a `FnMut` closure
+7 | |         logs.push("complete".to_string());
+  | |         ---- variable captured here
+8 | |     });
+  | |_____^ returns an `async` block that contains a reference to a captured variable, which then escapes the closure body
+  |
+  = note: `FnMut` closures only have access to their captured variables while they are executing...
+  = note: ...therefore, they cannot allow references to captured variables to escape
+```
+
+The problem is that `line` and `logs` are captured by the closure and only available during the execution of the closure, but the returned future needs to reference them when it is later awaited. And "later" is not soon enough, as it may be after the closure finishes execution. As a helpful mental model, you can think of the future created by the async block as a struct that implements the `Future` trait and is generic over the lifetime of the data it borrows.
+
+```rust,norun
+#[tokio::main]
+async fn main() {
+    let mut logs = Vec::new();
+    run(|line| AsyncBlock {
+        line,
+        logs: &mut logs,
+    });
+}
+
+struct AsyncBlock<'line, 'logs> {
+    line: &'line str,
+    logs: &'logs mut Vec<String>,
+}
+
+impl<'line, 'logs> Future for AsyncBlock<'line, 'logs> {
+    type Output = ();
+
+    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
+        todo!()
+    }
+}
+```
+
+Here, the returned `AsyncBlock` value holds on to `line` and `logs`, which are only accessible to the closure while it executes.
+
+However, that is not the only issue. Let's replace our inline closure with an actual async function and pass it to `run`. Note that the bound on `F` has also changed to `for<'a> FnMut(&'a str) -> Fut`, meaning that for all lifetimes `'a`, `F` is a closure accepting a `&str` that lives for `'a`. This is called a Higher-Rank Trait Bound (HRTB) in Rust, and `F` is said to be higher-ranked over the lifetime of its input.
+
+```rust,norun
+#[tokio::main]
+async fn main() {
+    run(do_something);
+}
+
+async fn do_something(s: &str) {}
+
+async fn run<F, Fut>(f: F)
+where
+    F: for<'a> FnMut(&'a str) -> Fut, 
+    // ^------
+    // HRTB used here
+    Fut: Future<Output = ()>,
+{
+    todo!()
+}
+```
+
+```
+error: implementation of `FnMut` is not general enough
+  --> examples/async_closure.rs:8:5
+   |
+ 8 |       run(do_something);
+   |       ^^^^^^^^^^^^^^^^^
+...
+13 | / async fn run<F, Fut>(f: F)
+14 | | where
+15 | |     F: for<'a> FnMut(&'a str) -> Fut,
+   | |        ----------------------------- doesn't satisfy where-clause
+16 | |     Fut: Future<Output = ()>,
+   | |_____________________________- due to a where-clause on `run`...
+   |
+   = note: ...`for<'a> fn(&'a str) -> impl Future<Output = ()> {do_something}` must implement `FnMut<(&'a str,)>`
+   = note: ...but it actually implements `FnMut<(&'0 str,)>`, for some specific lifetime `'0`
+```
+
+This happens because the concrete types for `F` and `Fut` are determined by the caller, and there is no way to express that `Fut` may capture the higher-ranked lifetime `'a`. In `main`, `Fut` is inferred to be the future produced by `do_something`, which must capture the lifetime of its `&str` input to use it, but `for<'a>` requires `F` to work for any lifetime, not just the one tied to that specific input.
+
+If your implementation doesn't need to reference its input `s`, you can use [opaque type precise capturing](https://doc.rust-lang.org/nightly/edition-guide/rust-2024/rpit-lifetime-capture.html) to indicate that:
+
+```rust
+fn do_something(s: &str) -> impl Future<Output = ()> + use<> {
+    async {}
+}
+
+async fn run<F, Fut>(f: F)
+where
+    F: for<'a> FnMut(&'a str) -> Fut,
+    Fut: Future<Output = ()>,
+{
+    todo!()
+}
+```
+
+We desugar the async function to a regular sync function that returns a `Future` value. The `use<>` syntax tells the compiler that the opaque type should not capture any generic lifetime parameters, because the corresponding hidden type does not use them (notice that we return an empty async block).
+
+Of course, you can't always avoid holding on to the input references, so the issue still remains.
+
+As we have seen, to quote [the RFC for async closures](https://rust-lang.github.io/rfcs/3668-async-closures.html#motivation), two major limitations when using closures in async code includes
+
+> 1. That closures cannot return futures that borrow from the closure captures.
+> 2. The inability to express higher-ranked async function signatures.
+
+
+#### TODO
 - closures
   - coming soon (https://github.com/rust-lang/rust/pull/132706, https://blog.rust-lang.org/inside-rust/2024/08/09/async-closures-call-for-testing.html)
   - async blocks in closures vs async closures