768 Example gRPC Upload Script

This script uploads vectors from a Parquet file to Pinecone using the gRPC client for improved performance. It uses parallel processing with multiple worker threads to efficiently upload large datasets.

What It Does

• Reads vector data from a Parquet file
• Uploads vectors to a Pinecone index using the gRPC protocol
• Automatically creates the index if it doesn't exist
• Uses parallel processing with configurable batch sizes and worker threads
• Provides progress tracking (if tqdm is installed)

Installation

Required Dependencies

Install the required packages using uv:

uv add "pinecone[grpc]" pandas pyarrow python-dotenv

Optional Dependencies

For progress bars during upload:

uv add tqdm

Configuration

Before running the script, you need to configure the following:

1. Environment Variables

Set your Pinecone API key either:

• Option A: Set as an environment variable:

  export PINECONE_API_KEY="your-api-key-here"

• Option B: Create a .env file in the project root:

  PINECONE_API_KEY=your-api-key-here
  

2. Script Configuration

Edit the configuration variables in 768_example_grpc.py:

# --- Configuration ---
PARQUET_FILE = "768_example.parquet"  # Path to your Parquet file
INDEX_NAME = "768-example-grpc"       # Name of your Pinecone index
VECTOR_DIM = 768                      # Dimension of your vectors

BATCH_SIZE = 400                      # Number of vectors per batch
MAX_WORKERS = 16                      # Number of parallel worker threads
READ_CHUNK_SIZE = 50000               # Number of rows to read from Parquet at once

What to change:

• PARQUET_FILE: Update to the path of your Parquet file. The file should contain columns:

  • id: Vector IDs (will be converted to strings)
  • values: Vector embeddings (list of floats)
  • metadata: Optional metadata (can be JSON strings or dicts)

• INDEX_NAME: Change to your desired Pinecone index name

• VECTOR_DIM: Set to match the dimension of your vectors (default: 768)

• BATCH_SIZE: Adjust based on your needs:

  • Larger batches = fewer API calls but more memory usage
  • Default: 400 (good for gRPC)

• MAX_WORKERS: Number of parallel upload threads:

  • More workers = faster upload but higher resource usage
  • Default: 16 (good for saturating gRPC connections)

• READ_CHUNK_SIZE: How many rows to read from Parquet at once:

  • Larger chunks = less file I/O but more memory
  • Default: 50,000

3. Index Configuration

The script creates a serverless index with these settings:

pc.create_index(
    name=INDEX_NAME,
    dimension=VECTOR_DIM,
    metric="cosine",
    spec={"serverless": {"cloud": "aws", "region": "us-west-2"}}
)

To change the region or use a different index type, modify the create_index call in the main() function.

Usage

Run the script:

python 768_example_grpc.py

Parquet File Format

Your Parquet file should have the following structure:

• id: Vector identifier (any type, will be converted to string)
• values: Vector embeddings as a list/array of floats
• metadata: (Optional) Metadata as JSON strings or dicts

Example:

import pandas as pd
import pyarrow.parquet as pq

df = pd.DataFrame({
    'id': ['vec1', 'vec2', 'vec3'],
    'values': [
        [0.1, 0.2, ..., 0.768],  # 768-dimensional vector
        [0.3, 0.4, ..., 0.768],
        [0.5, 0.6, ..., 0.768]
    ],
    'metadata': [
        '{"category": "A"}',
        '{"category": "B"}',
        '{"category": "A"}'
    ]
})

df.to_parquet('768_example.parquet')

Performance Tips

• For large datasets: Increase MAX_WORKERS (try 32 or 64) if you have sufficient network bandwidth
• For memory-constrained systems: Reduce READ_CHUNK_SIZE and BATCH_SIZE
• For faster uploads: Use gRPC (this script) instead of REST API
• Monitor progress: Install tqdm to see real-time upload progress

Troubleshooting

• ImportError for gRPC: Make sure you installed pinecone[grpc] not just pinecone
• API Key not found: Verify your .env file is in the project root or the environment variable is set
• Index already exists: The script will use the existing index if it matches the name and dimension
• Memory issues: Reduce READ_CHUNK_SIZE and BATCH_SIZE