Skip to content

feat(mm): implement an OOM killer for user page fault allocation failures #1976

@fslongjin

Description

@fslongjin

Feature request

DragonOS should implement an OOM killer, or an equivalent Linux-compatible OOM handling path, for memory allocation failures reached from user page faults.

Motivation

When a user-mode page fault reaches the memory fault handler and the fault result contains VM_FAULT_OOM, returning to user space without resolving the fault or terminating/selecting a victim can cause the same instruction to fault repeatedly. Linux does not silently ignore this condition: the x86 page fault path calls into the OOM handling flow (pagefault_out_of_memory()), after which user space either retries successfully or the task is killed if selected by OOM policy.

DragonOS currently has page fault paths that can observe VM_FAULT_OOM, but there is no complete OOM killer behavior to make this path robust and Linux-compatible.

Expected behavior

  • User-mode VM_FAULT_OOM should not silently return with the fault unresolved.
  • The kernel should select and terminate an appropriate victim, or otherwise provide a clearly defined Linux-compatible OOM resolution path.
  • Kernel-mode faults should continue to follow exception-table fixup or kernel fault handling rules.
  • The implementation should avoid test-specific workarounds and fit the existing memory-management and process-exit architecture.

Suggested validation

  • Add a regression test that forces allocation failure during a user page fault and verifies that the task does not livelock on the same faulting instruction.
  • Validate behavior against Linux 6.6 semantics for user page fault OOM handling.
  • Ensure normal SIGSEGV/SIGBUS fault reporting remains unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions