[BUG] Chunk cleanup logic should make sure RMD CF data is deleted for AAWC

[ZacAttack]

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Venice version

0.4.139

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.0): Mariner 5.15.111.1-1.cm2
• JDK version: 17

Describe the problem

When the server resolves to clean up a chunked record, it calls the ReplicationMetadataRocksDBStoragePartition#deleteWithReplicationMetadata API. Unfortunately, it will not actually delete the RMD CF, but instead leave 4 bytes garbage behind for the value schema ID in that CF. A fix should make sure that the API properly cleans things up. Note: This only happens for those stores which have active/active enabled as this is the only mode which actually uses RMD's.

Tracking information

No response

Code to reproduce bug

No response

What component(s) does this bug affect?

• Controller: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
• Router: This is the stateless query-routing layer for serving read requests.
• Server: This is the component that persists all the store data.
• VenicePushJob: This is the component that pushes derived data from Hadoop to Venice backend.
• VenicePulsarSink: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
• Thin Client: This is a stateless client users use to query Venice Router for reading store data.
• Fast Client: This is a stateful client users use to query Venice Server for reading store data.
• Da Vinci Client: This is an embedded, stateful client that materializes store data locally.
• Alpini: This is the framework that fast-client and routers use to route requests to the storage nodes that have the data.
• Samza: This is the library users use to make nearline updates to store data.
• Admin Tool: This is the stand-alone client used for ad-hoc operations on Venice.
• Scripts: These are the various ops scripts in the repo.