Skip to content

Improving Implement buffered document importing for improved performance#137

Merged
tnaum-ms merged 22 commits into
nextfrom
copilot/fix-130
Jun 11, 2025
Merged

Improving Implement buffered document importing for improved performance#137
tnaum-ms merged 22 commits into
nextfrom
copilot/fix-130

Conversation

@tnaum-ms

@tnaum-ms tnaum-ms commented Jun 6, 2025

Copy link
Copy Markdown
Collaborator

Closes #130
Fixes #136

Copilot AI and others added 14 commits June 3, 2025 09:53
Co-authored-by: xingfan-git <50465320+xingfan-git@users.noreply.github.com>
Co-authored-by: xingfan-git <50465320+xingfan-git@users.noreply.github.com>
- Copy documentBuffer.ts from sample PR, excluding cosmos-related code
- Update ClustersClient.ts InsertDocumentsResult and insertDocuments method
- Update importDocuments.ts with buffering logic following sample PR
- Rename insertDocumentIntoCluster to insertDocumentWithBufferIntoCluster
- Update l10n bundle with new error message

Co-authored-by: xingfan-git <50465320+xingfan-git@users.noreply.github.com>
Co-authored-by: tnaum-ms <171359267+tnaum-ms@users.noreply.github.com>
Co-authored-by: tnaum-ms <171359267+tnaum-ms@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 6, 2025 11:01

This comment was marked as outdated.

@tnaum-ms tnaum-ms changed the base branch from main to next June 6, 2025 15:32
@tnaum-ms tnaum-ms requested a review from Copilot June 10, 2025 08:12

This comment was marked as outdated.

@tnaum-ms tnaum-ms requested a review from Copilot June 10, 2025 15:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a buffering layer for batched document inserts to improve performance and updates import logic to leverage that buffer, while also enhancing error handling and progress reporting.

  • Introduces DocumentBuffer with configurable thresholds and a factory for MongoDB defaults.
  • Extends ClustersClient to handle unordered bulk writes and log detailed write errors.
  • Refactors importDocuments to use the buffer for bulk inserts, adjust progress increments, and localize final messages.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
src/utils/documentBuffer.ts Adds new DocumentBuffer class and createMongoDbBuffer utility for batched writes.
src/documentdb/ClustersClient.ts Wraps insertMany in try/catch, logs bulk write errors.
src/commands/importDocuments/importDocuments.ts Refactors import flow to use buffering, updates progress reporting and error accumulation.
l10n/bundle.l10n.json Updates localized strings for new final messages and write errors.
Comments suppressed due to low confidence (3)

src/utils/documentBuffer.ts:93

  • Add unit tests for DocumentBuffer to cover insertion, flushing, and error scenarios (e.g., oversized documents, empty documents).
export class DocumentBuffer<T> {

src/utils/documentBuffer.ts:138

  • [nitpick] documentsToProcess is undefined on success but an array in other cases; consider always returning an array (empty on success) for consistency and simpler downstream handling.
            return { ...insertResult, documentsToProcess: undefined };

src/documentdb/ClustersClient.ts:496

  • Calling ext.outputChannel.show() inside the loop can trigger the UI repeatedly; move this call after logging all write errors to show the output channel only once.
                    ext.outputChannel.show();

Comment thread src/commands/importDocuments/importDocuments.ts
Comment thread src/commands/importDocuments/importDocuments.ts
@tnaum-ms tnaum-ms merged commit b52e9c1 into next Jun 11, 2025
5 checks passed
@tnaum-ms tnaum-ms deleted the copilot/fix-130 branch June 11, 2025 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import Fails on Azure Cosmos DB for MongoDB (RU) Due to Throttling Improve Document Importing with Buffer

4 participants