Feature request
DragonOS should implement an OOM killer, or an equivalent Linux-compatible OOM handling path, for memory allocation failures reached from user page faults.
Motivation
When a user-mode page fault reaches the memory fault handler and the fault result contains VM_FAULT_OOM, returning to user space without resolving the fault or terminating/selecting a victim can cause the same instruction to fault repeatedly. Linux does not silently ignore this condition: the x86 page fault path calls into the OOM handling flow (pagefault_out_of_memory()), after which user space either retries successfully or the task is killed if selected by OOM policy.
DragonOS currently has page fault paths that can observe VM_FAULT_OOM, but there is no complete OOM killer behavior to make this path robust and Linux-compatible.
Expected behavior
- User-mode
VM_FAULT_OOM should not silently return with the fault unresolved.
- The kernel should select and terminate an appropriate victim, or otherwise provide a clearly defined Linux-compatible OOM resolution path.
- Kernel-mode faults should continue to follow exception-table fixup or kernel fault handling rules.
- The implementation should avoid test-specific workarounds and fit the existing memory-management and process-exit architecture.
Suggested validation
- Add a regression test that forces allocation failure during a user page fault and verifies that the task does not livelock on the same faulting instruction.
- Validate behavior against Linux 6.6 semantics for user page fault OOM handling.
- Ensure normal SIGSEGV/SIGBUS fault reporting remains unchanged.
Feature request
DragonOS should implement an OOM killer, or an equivalent Linux-compatible OOM handling path, for memory allocation failures reached from user page faults.
Motivation
When a user-mode page fault reaches the memory fault handler and the fault result contains
VM_FAULT_OOM, returning to user space without resolving the fault or terminating/selecting a victim can cause the same instruction to fault repeatedly. Linux does not silently ignore this condition: the x86 page fault path calls into the OOM handling flow (pagefault_out_of_memory()), after which user space either retries successfully or the task is killed if selected by OOM policy.DragonOS currently has page fault paths that can observe
VM_FAULT_OOM, but there is no complete OOM killer behavior to make this path robust and Linux-compatible.Expected behavior
VM_FAULT_OOMshould not silently return with the fault unresolved.Suggested validation