LOOKUP join failing with OOM error

[richardkchapman]

According to @dabayliss and @odmitriev, jobs that used to succeed in 702 systems are failing with memory errors in 3.8.2 with errors such as:

"0: System error: 0: Graph[2132], lookupjoin[2135]: SLAVE 10.222.64.5:8400: Out of memory, allocating row array, had 22379215, trying to allocate 52218184 elements"

[dabayliss]

I am beginning to suspect this is a 'bug' rather than 'poor resource usage'. Within: W20120910-144402 on dataland I am getting:System error: 0: Graph[74], lookupjoin[78]: SLAVE 10.241.7.26:16600: Out of memory, allocating row array, had 58717966, trying to allocate 137008603 elements
on a lookup join with 200K small (10s of bytes) elements
This now makes 4 different places within SALT blowing up on OOM on ,LOOKUP joins - I am having a hard time believing I was 'getting lucky' fours times before ...

[richardkchapman]

Sounds like a leak, if 200k records in the join causes 137 million rows to be allocated.

[jakesmith]

The RHS of the Lookup is trying to read 100M records, not 200K (LHS).
Still, with 10b records, there should be enough room, ignoring all overheads it should have needed ~2GB to accommodate.
I am wondering whether the problem/difference is that, the records read so far (58M) somehow fragmented the pages sufficiently, that it could not find enough contiguous pages to allocate the expanded row ptr attr (~1GB).
In the past (702) - it guessed, how much it could fit max and pre-allocated the ptr table.

[richardkchapman]

Hmmm - yes thinking about the Roxie memory manager will fragment badly in this situation. It was designed around the 'allocate largest then shrink' paradigm

[richardkchapman]

While fragmentation is an issue, I think the leak described in #3321 is more of an issue