fix: reduce batch size to prevent SDK timeouts and rewrite README

Author: Olow304

.docs.mv2.GG3ypp

@@ -1,6 +1,6 @@
 <div align="center">
 
-<img src="s_maw.svg" alt="maw" width="400">
+<img src="https://raw.githubusercontent.com/memvid/maw/main/s_maw.svg" alt="maw" width="400">
 
 **Crawl any site. Search it forever.**
 
@@ -14,174 +14,172 @@
 
 ---
 
-Feed the maw. It never forgets.
-
 ```bash
 npx @memvid/maw https://stripe.com/docs
 ```
 
-That's it. The entire Stripe docs are now in a 40MB file you can search and ask questions to. Offline. Forever.
-
-## Why?
+30 seconds later you have the entire Stripe documentation in a single 40MB file. Search it. Ask it questions. Works offline. Works forever.
 
-Because you shouldn't need to keep 47 browser tabs open or bookmark links you'll never read again. Crawl once, query forever.
-
-```bash
-# later, when you actually need it
-maw ask stripe.mv2 "how do webhooks work?"
-```
+No more bookmarking docs you'll forget about. No more 47 browser tabs. No more "I read this somewhere but can't find it."
 
 ## Install
 
 ```bash
 npm i -g @memvid/maw
 ```
 
-Or just use `npx @memvid/maw` without installing.
+Or just `npx @memvid/maw` if you don't want to install anything.
 
-## What it does
+## The basics
 
-```
-maw https://react.dev           → react.mv2 (312 pages, 18s)
-maw https://docs.python.org     → python.mv2 (2,847 pages, 4min)
-maw .                           → repo.mv2 (your local git repo)
-maw https://news.ycombinator.com/item?id=12345  → just that page
+```bash
+maw https://react.dev              # crawls entire site → maw.mv2
+maw find maw.mv2 "useEffect"       # instant search
+maw ask maw.mv2 "when should I use useCallback vs useMemo?"  # AI answers
 ```
 
-Smart defaults:
-- **Single page URL?** Fetches just that page
-- **Domain root?** Crawls the whole site
-- **Local path?** Reads your git repo
-- **Protected site?** Auto-switches to stealth browser
+That last one needs `OPENAI_API_KEY` in your env. The first two work out of the box.
 
-## Commands
+## What can you crawl?
 
-### Crawl
+Basically anything.
 
 ```bash
-maw <url>                           # → maw.mv2
-maw <url> -o docs.mv2               # custom output
-maw <url> docs.mv2                  # same thing (appends if exists)
-maw <url> --depth 5 --max-pages 500 # go deeper
+maw https://docs.python.org        # documentation (2,847 pages, ~4 min)
+maw https://paulgraham.com         # blogs
+maw .                              # your local codebase
+maw https://github.com/user/repo   # any git repo
+maw "https://news.ycombinator.com/item?id=12345"  # single pages
 ```
 
-### Query
+It figures out the right approach automatically:
+- Single page URL → fetches just that page
+- Domain root → crawls everything it can find
+- Local path → reads your files directly
+- Cloudflare/bot protection → switches to stealth browser
 
+## Commands
+
+**Crawl**
 ```bash
-maw find docs.mv2 "authentication"  # full-text search
-maw ask docs.mv2 "how does X work?" # AI answer (needs OPENAI_API_KEY)
-maw list docs.mv2                   # see what's inside
+maw <url>                             # saves to maw.mv2
+maw <url> -o docs.mv2                 # custom output file
+maw <url> docs.mv2                    # same (appends if file exists)
+maw <url> --depth 5 --max-pages 500   # go deeper
 ```
 
-### Preview
-
+**Search**
 ```bash
-maw preview stripe.com              # shows sitemap, estimated page count
+maw find docs.mv2 "authentication"    # keyword search
+maw ask docs.mv2 "how do I do X?"     # AI-powered answers
+maw list docs.mv2                     # see what's in there
 ```
 
-### Export
+**Preview before crawling**
+```bash
+maw preview stripe.com                # shows sitemap, page count estimate
+```
 
+**Export**
 ```bash
-maw export docs.mv2 -f markdown -o docs.md
-maw export docs.mv2 -f json -o docs.json
+maw export docs.mv2 -f markdown       # dump everything to markdown
+maw export docs.mv2 -f json           # or json
 ```
 
-## Embeddings
+## Semantic search
+
+By default you get keyword search (BM25). It's fast and works well for most things.
 
 Want semantic search? Add `--embed`:
 
 ```bash
-maw https://docs.whatever.com --embed openai
+maw https://kubernetes.io/docs --embed openai
 ```
 
-Uses OpenAI embeddings for semantic search. Costs ~$0.01 per 1000 pages. Without it, you get BM25 keyword search (still good, just different).
+Costs about $0.01 per 1000 pages. Your queries will understand meaning, not just keywords.
 
-## How it works
-
-Most sites work with a simple fetch. When that fails (Cloudflare, JS-heavy SPAs), maw falls back to a real browser. When *that* fails (aggressive anti-bot), it uses stealth mode.
+## How it handles protected sites
 
 ```
-fetch (fast) → playwright (slower) → rebrowser (stealth)
-     ↓              ↓                      ↓
-   works?        blocked?              blocked?
-     ↓              ↓                      ↓
-    done          retry                  done
+fetch (fast) → playwright (real browser) → rebrowser (stealth)
 ```
 
-90% of sites never need the browser. The 10% that do, just work.
+90% of sites work with a simple fetch. The other 10% get a real browser. If that's blocked too, stealth mode usually gets through.
+
+You don't have to think about this. It just tries each approach until something works.
 
-## Options
+## All the flags
 
-| Flag | Description | Default |
-|------|-------------|---------|
+| Flag | What it does | Default |
+|------|--------------|---------|
 | `-o, --output <file>` | Output file | `maw.mv2` |
-| `-d, --depth <n>` | Crawl depth | `2` |
-| `-m, --max-pages <n>` | Max pages to crawl | `150` |
+| `-d, --depth <n>` | How deep to crawl | `2` |
+| `-m, --max-pages <n>` | Stop after this many pages | `150` |
 | `-c, --concurrency <n>` | Parallel requests | `10` |
-| `-r, --rate-limit <n>` | Requests per second | `10` |
-| `--include <regex>` | Only crawl matching URLs | - |
-| `--exclude <regex>` | Skip matching URLs | - |
+| `-r, --rate-limit <n>` | Max requests/second | `10` |
+| `--include <regex>` | Only crawl URLs matching this | - |
+| `--exclude <regex>` | Skip URLs matching this | - |
 | `--browser` | Force browser mode | - |
 | `--stealth` | Force stealth mode | - |
-| `--embed [model]` | Enable embeddings | - |
+| `--embed [model]` | Enable semantic embeddings | - |
 | `--no-robots` | Ignore robots.txt | - |
-| `--no-sitemap` | Skip sitemap discovery | - |
+| `--no-sitemap` | Don't use sitemap.xml | - |
 
 ## Examples
 
 ```bash
-# documentation sites
+# grab some docs
 maw https://react.dev
 maw https://docs.python.org
 maw https://stripe.com/docs
 
-# news/blogs
+# archive a blog
 maw https://paulgraham.com/articles.html
-maw "https://news.ycombinator.com/item?id=40000000"
 
-# your own repos
-maw . -o my-project.mv2
-maw https://github.com/user/repo
+# your own code
+maw .
+maw https://github.com/your/repo
 
-# combine multiple sources
+# combine sources into one file
 maw https://react.dev https://nextjs.org -o frontend.mv2
 
-# deep crawl with embeddings
+# big crawl with semantic search
 maw https://kubernetes.io/docs --depth 4 --max-pages 1000 --embed openai
 ```
 
 ## Limits
 
-Files up to **50MB** work without any API key. That's roughly 500-2000 pages depending on content.
+Up to **50MB** works without an API key. That's roughly 500-2000 pages depending on how much text is on each page.
 
-For bigger crawls, get a key at [memvid.com](https://memvid.com).
+Need more? Get a key at [memvid.com](https://memvid.com).
 
 ## FAQ
 
 **Is this legal?**
 
-Respects robots.txt by default. Use `--no-robots` at your own discretion.
+Respects robots.txt by default. What you do with `--no-robots` is your business.
 
-**Why .mv2?**
+**What's an .mv2 file?**
 
-It's a [memvid](https://memvid.com) file — single-file database with full-text search, embeddings, and temporal queries baked in. Think SQLite but for documents.
+A [memvid](https://memvid.com) file. Single-file database with search built in. Like SQLite but for documents and memory.
 
-**Can I use it programmatically?**
+**Programmatic usage?**
 
 ```javascript
-import { crawl, query } from 'maw'
+import { maw, find, ask } from '@memvid/maw'
 
-await crawl('https://example.com', { output: 'site.mv2' })
-const results = await query('site.mv2', 'search term')
+await maw(['https://example.com'], { output: 'site.mv2' })
+const results = await find('site.mv2', 'search term')
+const answer = await ask('site.mv2', 'explain this to me')
 ```
 
-**What about rate limiting?**
+**Will I get rate limited?**
 
-Default is 10 req/sec with automatic backoff. Most sites won't notice you. If you're hitting APIs, consider `--rate-limit 2`.
+Default is 10 req/sec with backoff. Most sites won't notice. If you're worried, use `--rate-limit 2`.
 
-**Does it handle JavaScript-rendered content?**
+**JS-rendered content?**
 
-Yes. If fetch fails, it automatically tries Playwright. For heavily protected sites, use `--stealth`.
+Works. Falls back to a real browser automatically when needed.
 
 ---
 

README.md

@@ -10,7 +10,7 @@ import { maw, find, ask, list, preview, exportDocs } from '../src/index.js';
 import { setLogMode } from '../src/utils/logger.js';
 import * as ui from '../src/utils/ui.js';
 
-const VERSION = '1.0.4';
+const VERSION = '1.0.5';
 
 // Global error handlers to prevent crashes
 process.on('uncaughtException', (err) => {

bin/maw.ts

@@ -1,6 +1,6 @@
 {
   "name": "@memvid/maw",
-  "version": "1.0.4",
+  "version": "1.0.5",
   "description": "Crawl any site. Search it forever.",
   "type": "module",
   "main": "dist/index.js",

package.json

@@ -126,8 +126,8 @@ export async function ingestToMv2(
   let skippedDupes = 0;
   let stoppedAtLimit = false;
   let estimatedFileSize = 0;
-  // Batch size for ingestion (default 10)
-  const batchSize = options.batchSize || 10;
+  // Batch size for ingestion (default 3 - larger batches can cause SDK timeouts)
+  const batchSize = options.batchSize || 3;
   const batch: Array<{ title: string; label: string; text: string; metadata: Record<string, any> }> = [];
 
   // Size limit (default 40MB to stay safely under 50MB free tier with buffer)

src/ingestor/index.ts

@@ -0,0 +1,53 @@
+import { create, use } from '@memvid/sdk';
+import { existsSync } from 'fs';
+
+async function test() {
+  const filePath = '/tmp/test-append.mv2';
+
+  // Create new or open existing
+  let mem;
+  if (existsSync(filePath)) {
+    console.log('Opening existing memory...');
+    mem = await use('basic', filePath);
+  } else {
+    console.log('Creating memory...');
+    mem = await create(filePath, 'basic');
+  }
+
+  // Generate 10 docs with realistic content (like web pages)
+  const docs = [];
+  for (let i = 0; i < 10; i++) {
+    docs.push({
+      title: `Page ${i + 1} - Sample Web Page`,
+      label: 'web',
+      text: `# Page ${i + 1}\n\nThis is sample content for page ${i + 1}. `.repeat(50) +
+            `\n\nThis is a longer document with more text to simulate real web pages.`,
+      metadata: {
+        url: `https://example.com/page-${i + 1}`,
+        crawledAt: new Date().toISOString(),
+      },
+    });
+  }
+
+  console.log(`Trying putMany with ${docs.length} docs...`);
+  const start = Date.now();
+  try {
+    const ids = await mem.putMany(docs);
+    console.log('putMany() succeeded in ' + (Date.now() - start) + 'ms');
+  } catch (e) {
+    console.error('putMany() failed:', e.message);
+  }
+
+  // Now test find
+  console.log('\nTesting find...');
+  try {
+    const results = await mem.find('sample content', { k: 5 });
+    console.log('find() returned', results?.hits?.length || 0, 'results');
+  } catch (e) {
+    console.error('find() failed:', e.message);
+  }
+
+  console.log('Done');
+}
+
+test().catch(console.error);