Skip to content

Heap exhaustion returns null to the allocating thread instead of throwing OutOfMemoryError #647

Open
Open
Heap exhaustion returns null to the allocating thread instead of throwing OutOfMemoryError#647
Labels
agent/skipThe agent decided this issue is out of scope; see comment for reason.area/vmJVM internals: JIT compilers, GC, VM magic, type system.kind/bugConfirmed bug report with a repro or stack trace.

Description

@LSantha
LSantha
opened on Sep 17, 2026

Summary

When the Java heap is exhausted, the allocator returns null to the allocating thread instead of throwing OutOfMemoryError in it. The thread stumbles on with a null reference, producing cascading secondary failures, and the guest freezes hard (no prompt, Ctrl-C ineffective, no panic) instead of failing the offending allocation cleanly.

Environment

  • JNode 0.2.9-dev (x86 32-bit), VirtualBox 7.2.6, guest 1024 MB then 2048 MB RAM
  • Observed 2026-09-17; KDB log on UART1 + VGA screenshot

Reproduction

Run a workload that accumulates unreachable-but-uncollectable state until the heap fills — in our case, repeated whole-class JIT compiles (VmType.compileRuntime) across ~25 mauve testlets in one VM lifetime (no class unloading, so compiled code/IR/loaders accumulate). Eventually the KDB log shows repeating:

<oom/><mark/><sweep/><cleanup/><mark/><sweep/><cleanup/>

and the VGA console shows e.g.:

Out of memory in allocObject(0007D00C)ret null.01000000
Debug stacktrace: org.jnode.vm.VmStackReader::debugStackTrace
    ...
    org.jnode.vm.memDef.DefaultHeapManager::allocHeap
    org.jnode.vm.memDef.DefaultHeapManager::allocObject
    ...
    java.util.LinkedList::addBefore
    java.util.LinkedList::addLast
    org.jnode.driver.serial.console.RawKeyboardReader$InputPump::run
    ...
<oom/><mark/><sweep/><cleanup/><mark/><sweep/><cleanup/>

Note the victim here is the serial console's own keyboard-pump thread dying inside LinkedList.addLast — i.e. once the heap is full, even the management console cannot allocate, so the guest is unmanageable (serial + VGA dead, only poweroff works). The allocating thread got null, not an exception.

Expected behavior

After a full GC, the allocating thread should receive a catchable java.lang.OutOfMemoryError (per JLS 11.1 / JVM spec), letting the application fail fast while the rest of the VM — especially console/management threads — stays alive. At minimum, system threads should be insulated from user-heap exhaustion.

Suspected component

org.jnode.vm.memDef.DefaultHeapManager::allocObject / allocHeap OOM path (returns null; the <oom/> marker + debugStackTrace suggest the detection exists but the delivery as a thrown error does not).

Impact

Any long-running guest workload that grows the heap eventually bricks the VM with no diagnosis and no recovery short of poweroff + snapshot restore. This turned a routine 50-test sweep into repeated freeze/restore cycles during L2-compiler differential testing.

Triage Addendum (auto, 2026-09-24)

  • Area / Kind: area/vm + kind/bug
  • Blast radius: L5 system (heap allocation/GC path, VM-wide failure mode, boot/QEMU proof required)
  • Expected scope: DefaultHeapManager.allocObject / allocHeap OOM delivery and focused regression coverage; exact lines and minimal fix pending; no ASM, public API, or unrelated heap-policy changes unless required.
  • Test path: Tentative sh build.sh x86 plus a QEMU/VirtualBox boot test that fills the heap and verifies OutOfMemoryError delivery; exact commands pending.
  • Merge signal: human-review (bug on an L5 VM-wide allocation/GC path)
  • Gaps: Blocking: exact build target and commit; exact boot/run commands; numbered reproducible workload/test attachment.
  • Full report: ## Triage comment below.

✅ Ticket Runner Status

Field Value
Phase DONE
Turn 0/3
Retries 0/3
PR -
Started 2026-09-24T22:09:43.501Z

Activity

  1. LSantha
    added
    kind/bugConfirmed bug report with a repro or stack trace.
    area/vmJVM internals: JIT compilers, GC, VM magic, type system.
    on Sep 17, 2026
  2. LSantha
    added
    no-autoSkip all automation; human drives via /oc and /run only.
    on Sep 24, 2026

  3. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    /oc triage

  4. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    Triage

    • Area: area/vm (suspected org.jnode.vm.memDef.DefaultHeapManager GC/allocation path; root cause unconfirmed)
    • Kind: kind/bug (stack and actual/expected behavior are reported)
    • Repro: needs more info from reporter
    • Blast radius: L5 system (heap allocation/GC path, VM-wide failure mode, boot/QEMU proof required)
    • Expected scope: DefaultHeapManager.allocObject / allocHeap OOM delivery and focused regression coverage; exact lines and minimal fix pending. Do not change ASM, public API, or unrelated heap policy unless required.
    • Test path: tentative sh build.sh x86 plus a QEMU/VirtualBox boot test that fills the heap and verifies OutOfMemoryError delivery; exact commands pending.
    • Merge signal: human-review (bug on an L5 VM-wide allocation/GC path)
    • Gaps: blocking: exact build target and commit; exact boot/run commands; numbered reproducible workload/test attachment
    • Suggested next: needs-info
    • Split proposal: none yet; keep as one VM allocation-path issue if the focused fix stays within the heap manager and regression tests
    • Labels applied: kind/bug, area/vm, no-auto (no removals; no-auto retained)

    Needs the following before work can start:

    1. Provide the exact commit SHA and sh build.sh <target> command used.
    2. Provide the complete QEMU or VirtualBox boot/run command, including memory and architecture, and confirm whether boot reached System has finished.
    3. Provide numbered shell/test-harness steps or the attached workload/testlet artifact, including how the heap is sized, plus a 20–60 line UART1 excerpt around the failure if available.

  5. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    Triage

    • Area: area/vm (suspected org.jnode.vm.memDef.DefaultHeapManager GC/allocation path; root cause unconfirmed)
    • Kind: kind/bug (stack and actual/expected behavior are reported)
    • Repro: needs more info from reporter
    • Blast radius: L5 system (heap allocation/GC path, VM-wide failure mode, boot/QEMU proof required)
    • Expected scope: DefaultHeapManager.allocObject / allocHeap OOM delivery and focused regression coverage; exact lines and minimal fix pending. Do not change ASM, public API, or unrelated heap policy unless required.
    • Test path: tentative sh build.sh x86 plus a QEMU/VirtualBox boot test that fills the heap and verifies OutOfMemoryError delivery; exact commands pending.
    • Merge signal: human-review (bug on an L5 VM-wide allocation/GC path)
    • Gaps: blocking: exact build target and commit; exact boot/run commands; numbered reproducible workload/test attachment
    • Suggested next: needs-info
    • Split proposal: none yet; keep as one VM allocation-path issue if the focused fix stays within the heap manager and regression tests
    • Labels applied: kind/bug, area/vm, no-auto (no removals; no-auto retained)

    New%20session%20-%202026-09-24T21%3A55%3A25.873Z
    opencode session  |  github run

  6. LSantha
    added
    agent/needs-infoThe agent has posted clarifying questions; waiting for the reporter.
    and removed
    no-autoSkip all automation; human drives via /oc and /run only.
    on Sep 24, 2026

  7. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    /run

  8. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    /oc Please proceed with this task.

  9. LSantha
    added
    agent/in-progressThe OpenCode agent is currently working on this issue.
    on Sep 24, 2026

  10. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    🤖 Working on this. Run: 2026-09-24T22:09:43.501Z
    Plan: trace OOM delivery, add focused regression coverage, then build and verify the VM path.

  11. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    Plan

    • Trace DefaultHeapManager.allocObject and allocHeap failure semantics.
    • Compare OOM handling across allocation entry points and heap managers.
    • Add the smallest regression test for exception delivery.
    • Run the focused core test suite and x86 build.
    • Review the final diff for Java 1.6 and VM-path safety.

  12. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    🤖 Refusal — needs more info

    • Why this matches: The reported null result does not match the current allocator control flow. DefaultHeapManager.allocObject performs a synchronous GC and retry, then throws the preallocated OutOfMemoryError at core/src/core/org/jnode/vm/memmgr/def/DefaultHeapManager.java:320-338; its normal exit at line 378 is a non-null object.
    • Log interpretation: ret null. and debugStackTrace() are emitted by the low-level native-block allocator at core/src/core/org/jnode/vm/MemoryBlockManager.java:123-131. DefaultHeapManager.allocHeap converts that sentinel to null internally, but allocObject does not return it after exhaustion.
    • Likely observed failure: The serial input pump itself allocates at core/src/driver/org/jnode/driver/serial/console/RawKeyboardReader.java:85 and has no OutOfMemoryError recovery. This explains loss of the management console after OOME, but preserving system threads requires a separate heap-reserve/resilience design and is outside the claimed null-delivery fix.
    • What I did not do: I made no source changes and did not run a build/boot test because the reported behavior cannot be reproduced from the current source and no exact build commit or workload was provided.
    • Needed to proceed:
      1. Exact commit SHA and ISO build target used on 2026-09-17.
      2. Exact QEMU/VirtualBox command and heap-sizing configuration.
      3. Numbered workload or test artifact showing the workload thread catching OutOfMemoryError, separate from the serial input pump failure.
      4. UART excerpt showing the OOME delivery/failure point before any console input allocation.

    Confidence: high for the current-source control flow; the reported guest behavior remains tied to the missing revision/repro.

  13. LSantha commented on Sep 24, 2026

    @LSantha
    LSantha
    on Sep 24, 2026
    OwnerAuthor

    No code changes made. The current allocator already throws OOME at core/src/core/org/jnode/vm/memmgr/def/DefaultHeapManager.java:338; posted a needs-info report explaining the low-level ret null log and requesting the exact commit/reproduction.

    New%20session%20-%202026-09-24T22%3A11%3A33.781Z
    opencode session  |  github run

  14. LSantha
    added
    agent/skipThe agent decided this issue is out of scope; see comment for reason.
    and removed
    agent/needs-infoThe agent has posted clarifying questions; waiting for the reporter.
    agent/in-progressThe OpenCode agent is currently working on this issue.
    on Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/skipThe agent decided this issue is out of scope; see comment for reason.area/vmJVM internals: JIT compilers, GC, VM magic, type system.kind/bugConfirmed bug report with a repro or stack trace.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions