Infinite stall when worker pools don't finish work in the order they consume it

[craigfe]

For example, this map request never terminates:

* 543270a worker_NAlAM16I Removed pending <multiply> on key l (map--Vwi4x9WY)
* 5e45b28 worker_NAlAM16I Performed <multiply> on key l
* b2c86a1 worker_NAlAM16I Consuming task on key l
* 0e31409 worker_z20xqzz3 Removed pending <multiply> on key k
* 8b1ceea worker_z20xqzz3 Performed <multiply> on key k
* 69ffc5d worker_z20xqzz3 Consuming task on key k
* e4ccd3c worker_NAlAM16I Consuming task on key j
* 5f18026 worker_cwDZRugP Consuming task on key i
* dcee305 worker_NAlAM16I Consuming task on key h
* 5c3d90e worker_cwDZRugP Consuming task on key g
* d152305 worker_cwDZRugP Removed pending <multiply> on key f
* f86fbc8 worker_cwDZRugP Performed <multiply> on key f
* e516a0f worker_cwDZRugP Consuming task on key f
* 644d2dc worker_cwDZRugP Removed pending <multiply> on key e
* 64b07a9 worker_cwDZRugP Performed <multiply> on key e
* 76a9a76 worker_cwDZRugP Consuming task on key e
* 3716a2b worker_cwDZRugP Removed pending <multiply> on key d
* b3b3922 worker_cwDZRugP Performed <multiply> on key d
* f62fd25 worker_cwDZRugP Consuming task on key d
* bad1636 worker_z20xqzz3 Removed pending <multiply> on key c
* 4d156b4 worker_z20xqzz3 Performed <multiply> on key c
* c4fc93e worker_z20xqzz3 Consuming task on key c
* d4d1baf worker_NAlAM16I Removed pending <multiply> on key b
* 82f5f9d worker_NAlAM16I Performed <multiply> on key b
* 942ce23 worker_NAlAM16I Consuming task on key b
* f1a6e3e worker_cwDZRugP Removed pending <multiply> on key a
* f5c95d0 worker_cwDZRugP Performed <multiply> on key a
* b7ca1b1 worker_cwDZRugP Consuming task on key a
* 419d1fb map Specifying workload
* 1aeb6a8 map Add map--Vwi4x9WY to job queue (HEAD -> master)
* 21ca40c client Committing to key l
* e0ced83 client Committing to key k
* 769e64e client Committing to key j
* 66ba3ec client Committing to key i
* e9d05e5 client Committing to key h
* eba79ed client Committing to key g
* a465f18 client Committing to key f
* 27e5c4e client Committing to key e
* c103b7c client Committing to key d
* 0874129 client Committing to key c
* 1c4029b client Committing to key b
* efd03d8 client Committing to key a

An initial attempt at fixing this should probably include a refactoring of the worker to allow for stalls to be injected at test time (so that this is reproducible).