[Bug]: Incremental index fails in merge stage (FtsRocksdbReducer: source postings is not BitPacked) and leaves the index unopenable

[vchen12]

Description

On a large Markdown/plain-text workspace (~20,000 files), incremental zg index runs repeatedly fail in the merge stage with ZVEC_INTERNAL_ERROR. The CLI message observed for one of these failures was:

[ZVEC_INTERNAL_ERROR] FtsRocksdbReducer: source postings is not BitPacked. field=text

After such a failure the index is left unopenable: every subsequent zg status / zg query returns Failed to open zvec collection storage (ZVEC_GREP.ENGINE.STORAGE.ZVEC_OPEN_FAILED), index.zvec is left with unfinished *.tmp directories, and the on-disk index had grown from 3.3 GB to 6.3 GB. The only recovery we found is a full zg index --rebuild, which takes about 5 hours for this corpus.

Expected: an incremental index job that fails should leave the previous index intact and queryable, and should be retryable without a full rebuild.

This is not a one-off. The daemon log on this machine records 117 ZVEC_INTERNAL_ERROR job failures between 2026-09-04 and 2026-09-14, spanning zg 0.2.1 and 0.2.2.

Three related problems found while recovering

1. A stale in-process RocksDB lock blocks --rebuild. Running zg index --rebuild directly after the failure deletes the old storage but then fails to create the new one (create id map failed); files.zvec/idmap.0/LOG says lock hold by current process, i.e. the still-running daemon holds a lock on the already-deleted storage. Recovery requires zg server off && zg server on before --rebuild. It would help if the rebuild path released and reopened the storage itself, or if the error named the holding daemon.
2. The daemon log never persists the error message. ~/.zvec-grep/daemon/logs/server.log records only "error_code":"ZVEC_INTERNAL_ERROR" on job.finished - no message, no stack. The FtsRocksdbReducer text above was only visible on the CLI at the time, so for scheduled or background runs the cause is unrecoverable after the fact. Please log the message (and ideally the failing field/segment) next to the code.
3. A rebuild is neither resumable nor reported as partial. A rebuild interrupted by daemon exit (the machine slept overnight) left 674 MB written under index.zvec, but zg status then reports Coverage 0% - 0 / 20,357 files with 20,357 added pending. The written data is not reused, and nothing indicates that an earlier rebuild was interrupted.

One local modification worth disclosing: the index home is redirected out of the workspace to a local disk path, via ~/.zvec-grep/workspace-homes.json plus a small patch to dist/engine/service/root.js, because the workspace itself lives in a cloud-sync folder where RocksDB writes time out. The patch only changes where the index home resolves to; indexing and merging are untouched.

Steps to Reproduce

1. Install zg (npm install @zvec/zvec-grep plus @zvec/zvec) and the local/potion-multilingual-128m model.
2. Index a workspace of roughly 20,000 .md / .txt files:
   zg index -t markdown -t txt -g '!<excluded-dir>/**' --embedding local/potion-multilingual-128m
3. Let a daily incremental run add or modify a few hundred files, then run zg index --mode server again. Repeat daily.
4. Observe: after about 100 s the job ends with state=failed, error_code=ZVEC_INTERNAL_ERROR, and the CLI shows FtsRocksdbReducer: source postings is not BitPacked. field=text.
5. zg status --mode server now returns Failed to open zvec collection storage; the index stays unusable until a full --rebuild.

We have not isolated a specific triggering document. The failures happen in the merge stage, not during extraction.

Version and Diagnostics

zg --version         0.2.2          (also seen on 0.2.1)
@zvec/zvec           0.7.1          (official @zvec/bindings-darwin-x64 prebuilt)
node --version       v24.13.0
sw_vers              macOS 12.7.6 (21H1320), x86_64 (Intel)
embedding            local/potion-multilingual-128m - 256 dimensions - cosine
workspace            20,357 files, type=markdown|txt, one directory excluded by glob
index size           2.8-3.3 GB when healthy; 6.3 GB after the failed merge

Logs or Error Output

Daily breakdown of job.finished events from ~/.zvec-grep/daemon/logs/server.log (paths redacted):

2026-09-09 {'ZVEC_INTERNAL_ERROR': 1, 'succeeded': 52}
2026-09-11 {'ZVEC_INTERNAL_ERROR': 1, 'ZVEC_GREP.ENGINE.STORAGE.ZVEC_OPEN_FAILED': 1, 'succeeded': 8}
2026-09-12 {'succeeded': 48, 'ZVEC_GREP.ENGINE.INDEXING.CONTENT_HASH_FAILED': 1}
2026-09-13 {'succeeded': 20, 'ZVEC_GREP.ENGINE.INDEXING.CONTENT_HASH_FAILED': 3}
2026-09-14 {'ZVEC_INTERNAL_ERROR': 1}

The 2026-09-11 pair is the corruption sequence - the internal error, after which the storage can no longer be opened:

{"timestamp":"2026-09-11T16:14:31.361Z","event":"job.finished","state":"failed","attempt":1,"error_code":"ZVEC_INTERNAL_ERROR","duration_ms":1090528}
{"timestamp":"2026-09-11T21:55:42.661Z","event":"job.finished","state":"failed","attempt":1,"error_code":"ZVEC_GREP.ENGINE.STORAGE.ZVEC_OPEN_FAILED","duration_ms":807}

Most recent occurrence (2026-09-14), showing that only the code is logged, never the message:

{"timestamp":"2026-09-14T15:04:41.023Z","event":"job.started","job_id":"d4bb4341-...","reason":"manual","attempt":1}
{"timestamp":"2026-09-14T15:06:27.037Z","event":"job.finished","job_id":"d4bb4341-...","state":"failed","attempt":1,"error_code":"ZVEC_INTERNAL_ERROR","duration_ms":106014}

Aggregate failure counts in this log (2026-09-04 to 2026-09-14, 393 failed jobs in total):

ZVEC_GREP.ENGINE.STORAGE.ZVEC_OPEN_FAILED      244   (index unopenable after a failed merge)
ZVEC_INTERNAL_ERROR                            117
ZVEC_GREP.ENGINE.DAEMON_LEASE_ACTIVE            17
ZVEC_GREP.ENGINE.LOCK.BUSY                       9
ZVEC_GREP.ENGINE.INDEXING.CONTENT_HASH_FAILED    4
ZVEC_GREP.ENGINE.INDEXING.STATUS_FAILED          2

Because the message is not logged, we can only confirm the "FtsRocksdbReducer ... not BitPacked" text for the failure of 2026-09-11T16:14Z, which was observed on the CLI. The other ZVEC_INTERNAL_ERROR entries share the code and the same merge-stage timing, but we cannot prove they are all the same defect.

Operating System

macOS 12.7.6 (Intel x86_64)

Installation Method

npm project dependency

Affected Area

Indexing or file extraction, Local server

Pre-submission Checklist

• I checked the zvec-grep troubleshooting guide and existing issues.
• I removed credentials, private paths, and sensitive file contents from this report.

[egolearner]

Thanks for the report. Our initial investigation found two issues:

• In zvec: A durability issue can leave an index in an inconsistent state if the process exits unexpectedly during optimization. We reproduced the same error and implemented a fix locally, which passes crash-recovery tests. This is a possible explanation for the reported failure, though we haven’t confirmed the exact trigger in your case.
• In zvec-grep: Cleanup currently retries optimization, and if that fails, it can prevent the underlying stores from closing. We’ll adjust this so an optimization failure doesn’t block cleanup or leave database locks behind.

The zvec fix should prevent this failure in future runs, but an already affected index may need to be rebuilt. We’ll fix these problems ASAP.

[egolearner]

Thank you again for the detailed report. And we’ve opened two PRs related to this issue:

• [zvec #759](fix(fts): persist sealed postings before dropping side column families alibaba/zvec#759) fixes a durability issue that we reproduced by interrupting optimization.
• [zvec-grep #160](fix(storage): release collections after optimization failures #160) improves storage cleanup after optimization failures and adds error details to daemon logs.

We also identified a possible trigger in v0.2.2: stopping the server normally waits for active work to finish, but may forcibly terminate the daemon if shutdown takes too long. This could interrupt optimization and trigger the durability issue, even when using the normal stop command. We haven’t yet verified that full sequence end to end or confirmed whether it explains your experience.

To help us narrow this down, could you let us know whether, before the first source postings is not BitPacked error, you stopped or restarted the server while indexing was still running—for example, with zg server off? If you recall, did the stop command take around 30 seconds or longer? Any unexpected daemon exits or machine restarts around that time would also be helpful.

You mentioned a daemon exit during the later rebuild; we’d like to understand whether anything similar happened before the initial merge failure. Approximate timing is perfectly fine, and we appreciate any additional context you can share.

[vchen12]

Thanks for the PRs. I went back through the daemon log (2026-09-03 → 2026-09-16) and can answer fairly precisely.

Short answer: we never stopped the server while indexing was running. Every recorded stop was clean. What we do have is the daemon being killed abruptly, with a job still in flight.

1. zg server off during indexing: no.
The log has 15 server.stopped events. Reconstructing each daemon session (ready → stopped/disappearance) and tracking job.started / job.finished inside it, none of those 15 clean stops had an unfinished job at the time. So the "normal stop that escalates to a forced kill" path does not appear to be what hit us — at least not through a stop we issued.

2. Stop duration: not measurable from the log, but all restarts were fast.
The daemon log records the stop event but not how long the stop took. Every server.stopped → next server.ready interval was 1–8 s (1, 2, 4, 5, 5, 7, 7, 8, 8 s …). Nothing near 30 s. If the stop path does have a 30 s escalation timer, we have no evidence of reaching it.

3. Unexpected daemon exits: yes, frequently — and one lands right before our first ZVEC_INTERNAL_ERROR.
22 daemon sessions ended with no server.stopped line at all (next thing in the log is a new server.ready with a new pid). Of those, 10 ended while a job was still running:

pid 31816  2026-09-03T16:13:21Z → 16:28:06Z   job in flight since 16:13:36Z
pid 33533  2026-09-03T17:14:41Z → 17:28:58Z   job in flight since 17:15:05Z
pid 35808  2026-09-03T17:32:30Z → 22:37:41Z   job in flight since 20:53:16Z
pid 50094  2026-09-03T22:37:41Z → 09-04T03:56:35Z  job in flight since 23:49:38Z
pid 63135  2026-09-04T03:56:35Z → 04:00:33Z   job in flight since 03:59:09Z   <-- see below
pid 2497   2026-09-05T03:19:10Z → 15:55:43Z   job in flight since 13:49:16Z
pid 1191   2026-09-06T09:49:36Z → 12:17:22Z   job in flight since 10:32:51Z
pid 10231  2026-09-06T12:17:22Z → 14:41:05Z   job in flight since 14:29:46Z
pid 16055  2026-09-13T15:02:23Z → 09-14T15:02:17Z  job in flight since 09-13T17:02:48Z
pid 1694   2026-09-14T15:09:26Z → 09-15T10:27:53Z  job in flight since 09-14T15:09:36Z
pid 2202   2026-09-15T10:27:53Z → 15:59:26Z   job in flight since 10:50:27Z

The one marked above is the closest correlation we have: pid 63135 disappeared at 2026-09-04T04:00:33Z with an index job running, and our first ZVEC_INTERNAL_ERROR is at 2026-09-04T04:09:06Z — about 9 minutes later.

Cause of those disappearances, from macOS: machine reboots (local time: 2026-09-10 17:29 and 22:56, 09-11 17:48, 09-12 21:58 and 22:44, …) and, for the two most recent ones, the machine sleeping overnight while a rebuild was running. This is a laptop that sleeps and reboots regularly, so "the daemon is killed mid-optimization" is a routine event here, just not via the stop command.

4. The specific not BitPacked failure (2026-09-11T16:14:31Z) was not itself interrupted.
That job started 15:56:20Z and failed 18 min later. Its daemon (pid 10641) had been up since 15:02:07Z, with no stop or restart during the job, and macOS logged no sleep or shutdown in that window. The preceding daemon session also ended with 0 jobs in flight. So that particular failure reads as hitting damage that was already on disk, rather than being the interruption itself — consistent with your durability hypothesis, with the interruption having happened earlier (most plausibly 09-04).

5. Caveat on attribution.
Because the daemon log stores only error_code and not the message, we can confirm the literal FtsRocksdbReducer: source postings is not BitPacked. field=text text only for the 2026-09-11T16:14Z failure, which we saw on the CLI at the time. The other 116 ZVEC_INTERNAL_ERROR entries share the code and the same merge-stage timing, but we cannot prove they are the same defect. zvec-grep #160 adding error details to the daemon log will fix exactly this blind spot for us.

Happy to run any specific reproduction you'd like once the fixes land — this machine reproduces the "killed mid-optimization" condition very easily.

[laihenyi]

Independent confirmation from a different platform, a different integration path, and a case where no stop/restart was involved. Posting in case it helps narrow the trigger.

Environment

• @zvec/zvec-grep 0.2.2, @zvec/zvec 0.7.1 (@zvec/bindings-darwin-arm64), Node 26
• macOS on Apple Silicon (the reporter above is Intel macOS 12), no CLI usage at all
• indexing is driven entirely by an agent extension that runs the engine in-process: @sugarforever/pi-zvec-grep 0.1.1 (locks carry "instanceToken": "direct-<pi-pid>")
• embedding local/potion-code-16M-v2, 256 dims, cosine
• the indexed root was an entire home directory (recursive, no excludes)

Same failure, but the index stayed openable

• index.zvec reached 226 segments / 13 GB. 0.tmp and 1.tmp each held ~3.3 GB of recovery leftovers (one of them contained embedding.index.2.proxima at 1.83 GB plus scalar.0.ipc at 1.74 GB). That is where the 13 GB went, not into searchable data.
• Locked by the wedged process: locks/home.write/lock.json with {"operation":"index"}, plus a daemon.guard.
• Unlike the report above, status/query kept working here: the storage opened and searches returned results, and only merging failed. So "unopenable" may be a second-order effect rather than part of this defect.

The agent-facing consequence

This integration derives workspace status from job runtime state, and a workspace whose optimization always fails alternates between "job running" and "job failed". Any sampling of the agent's search tool then returns:

{"status":"indexing","retryable":true,"message":"The workspace index is not ready. ..."}

retryable is hardcoded true for the indexing/updating phases, and the integration has no consecutive-failure counter or backoff. The failed pass sets phase = "error" with the real message, but the next queued pass immediately resets the phase and clears the error, so the sampling almost always lands on indexing. The net effect for an agent is that it never learns the index is permanently broken and never falls back to grep — it just retries. I saw this run for 3+ hours.

Breaking the loop required SIGTERM on the holding process. That is worth knowing operationally: the terminal had been closed long before, so stdio was revoked and the process had been reparented to launchd. It was still holding the index and still cycling, and nothing in the product would have stopped it.

On the trigger: no forced stop involved

Your zg server off escalation hypothesis does not apply here — no stop command was ever issued on this host. What I do have is a long-lived in-process watcher: a pass that fails leaves the index not-ready, which makes the integration keep enqueuing passes (reconcile timer plus change batching), and each pass writes fresh segments that can never merge. So "damaged once by an earlier interrupted optimization, then failing forever" fits my data better than "every incremental pass independently produces segments the reducer rejects". I cannot date the original damaging event because I only investigated after the fact.

Confirmation of the zvec-grep half of the fix

Reading the 0.2.2 dist, the two cleanup paths you describe are exactly:

• dist/engine/storage/zvec.js:199 — finalizeWrites() awaits collection.optimize() and rethrows, so the entire index pass fails.
• dist/engine/storage/zvec.js:207 and :359 — close() retries optimization before closing, so a second failure skips this.files.close() and this.collection.closeSync().

I reproduced the lock-leak half by injecting the native message into a real collection on a temp path (the native Collection methods are read-only, so interception goes through a Proxy): pristine close() throws and never reaches files.close(); with the shape from #160 applied, both stores are always released. Happy to share the harness if it is useful as a regression case.

One smaller thing, relevant to #146: the embedding model cache resolves to join(home, "models") (dist/engine/service/zvec-grep.js:900), i.e. it lands in the index home even when the index is redirected elsewhere, so moving the home re-downloads the model unless --model-cache is passed.

[laihenyi]

A note for anyone validating PR #759 locally, because I got this wrong myself first.

I built the darwin-arm64 addon from this PR's head baa29d6e and used it with the
released @zvec/zvec@0.7.1 wrapper. The addon loads fine and npm test passes,
but any real indexing fails immediately:

ZVEC_INTERNAL_ERROR: Create vector column indexer failed: embedding)
ZVEC_GREP.ENGINE.STORAGE.ZVEC_UPSERT_FAILED: zvec failed to upsert entity documents

That looks like the PR broke indexing. It did not — it is a core/wrapper pairing
problem:

• @zvec/zvec@0.7.1 was published from zvec-ai/zvec-node@63c6fafd (tag v0.7.1).
• Both zvec-node main and tag v0.7.1 pin the src/zvec submodule to
  58b95626f9c30e52b8fd9041dda1bcc63543041e ("feat(quantization): add uniform
  uint4 reformer/quantizer (#698)").
• This PR's head baa29d6e is 44 commits ahead of 58b9562 (status=ahead,
  behind=0, merge base 58b9562), including segment/vector internals such as
  #731, #726, #722, #718.

So building from the PR head gives a newer core with an older JS facade. Nothing in
the loader catches that: @zvec/zvec/src/index.js and src/prebuilt.js resolve the
addon by path only, with no version or hash check, so a mismatched binding loads
silently and only fails later, at vector-column-indexer creation. Easy to misread as
a regression in the PR.

To validate this PR against a released wrapper, backport it onto 58b9562
instead of checking out the branch. I checked the applicability of the PR diff on
that base (git apply --check -3):

src/db/index/column/fts_column/fts_column_indexer.cc    cleanly
src/db/index/column/fts_column/fts_column_indexer.h     cleanly
src/db/index/column/fts_column/fts_indexer.cc           cleanly
src/db/index/column/fts_column/fts_indexer.h            cleanly
tests/db/index/column/fts_column/fts_column_indexer_test.cc   cleanly
tests/db/index/column/fts_column/fts_rocksdb_reducer_test.cc  one conflict

The four source files — the actual fix — apply without drift; only one test file
conflicts. (I am still validating that backported build end to end, so I am not
claiming it is confirmed working yet, only that this is the correct route.)

For what it is worth, the PR's own C++ suites are green on the PR head build:

fts_column_indexer_test   Passed  25.3 s   93 tests / 10 suites, including
  SealPaths/FtsSealRecoveryDeathTest, CleanupStages/FtsSealCleanupDeathTest,
  FtsSealStatsTest, DropFailures/FtsSealRetryTest
fts_rocksdb_reducer_test  Passed   7.9 s   18 tests
100% tests passed, 2 tests

Two suggestions that would have saved me this detour:

1. Note the core commit each published binding is built from (the @zvec/zvec
   package.json or a runtime accessor would do it), or have the loader refuse an
   addon whose core revision does not match the facade.
2. When a fix targets a specific core revision, mention the base commit in the PR
   description — otherwise "build from the branch" silently mixes revisions.

Related note: I also see the FTS seal-durability failure in the wild without any
zg CLI involvement (an agent extension drives the engine in-process), which is
consistent with the interrupted-optimization trigger discussed above.

[vchen12]

Two contrast points from the original report (Intel macOS 12, CLI + daemon), in case they help bound the failure.

1. Here the storage really did become unopenable, and the log shows the ordering. The internal error came first, and the next job — 5.5 h later — could not open the storage at all:

{"timestamp":"2026-09-11T16:14:31.361Z","state":"failed","error_code":"ZVEC_INTERNAL_ERROR","duration_ms":1090528}
{"timestamp":"2026-09-11T21:55:42.661Z","state":"failed","error_code":"ZVEC_GREP.ENGINE.STORAGE.ZVEC_OPEN_FAILED","duration_ms":807}

From then on every zg status / zg query returned Failed to open zvec collection storage until a full --rebuild. So "unopenable" being second-order fits our data too — it just went further here than on your host, where only merging failed and search kept working.

2. Redirecting the index home did not cause a model re-download here. We redirect the workspace index home out of the cloud-sync folder (~/.zvec-grep/workspace-homes.json plus a small root.js patch), and the model cache still resolved to ~/.zvec-grep/models (506 MB); no models directory ever appeared under the redirected index home, and we never pass --model-cache. That suggests the re-download depends on which home is moved — the global one versus a per-workspace index home — which may be worth distinguishing in #146.

For completeness: we rebuilt on 2026-09-17 on unpatched 0.2.2 / zvec 0.7.1 (19,799 files, 1 h 50 m, 0 pending · 0 failed) and it has been healthy since. We are waiting for a released build rather than validating the PR locally, so no findings from us on that front.