Skip to content

[EXPERIMENTAL] Prefetch Engine for AA hybrid stores#15

Open
sushantmane wants to merge 1 commit into
mainfrom
li-aa-dcr-rmd-val-prefetch
Open

[EXPERIMENTAL] Prefetch Engine for AA hybrid stores#15
sushantmane wants to merge 1 commit into
mainfrom
li-aa-dcr-rmd-val-prefetch

Conversation

@sushantmane

@sushantmane sushantmane commented Mar 18, 2024

Copy link
Copy Markdown
Owner

Introduce Prefetch Engine for AA hybrid stores

This PR introduces a prefetch engine for hybrid stores to hide latency when
fetching RMD and values for incoming keys from RocksDB for use in DCR.

Here's how it works:

  1. After fetching a batch of records from the consumer, we arrange TopicPartitions
    to be processed in the following fashion:
+---------+-----+---------+---------+-----+---------+
| VT-TP-0 | ... | VT-TP-N | RT-TP-0 | ... | RT-TP-N |
+---------+-----+---------+---------+-----+---------+
  1. We fire prefetch requests for Keys belonging to RT-TPs in sequential order.

  2. Record processing for each TP begins with VT-TP-0. By the time we reach
    RT-TP-0, ideally, RMD and values for the keys will already be in memory.

How was this PR tested?

WIP

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@sushantmane sushantmane changed the title Prefetch Engine for AA hybrid stores [EXPERIMENTAL] Prefetch Engine for AA hybrid stores Mar 18, 2024
@sushantmane sushantmane force-pushed the li-aa-dcr-rmd-val-prefetch branch 2 times, most recently from da33b22 to 6a3450d Compare March 23, 2024 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant