Live product · Evidence & limitations
The evidence page connects the interactive demo to its inspectable source-row grounding, deterministic validation, benchmark protocol, raw run status, and explicit claim gate.
The public demo uses 46 wholly fictional people and companies generated by scripts/build_synthetic_demo.py. Role, industry, and business-need summaries are combined with offline TF-IDF/LSA vectors; no embedding or description API call is required. The research timing summaries describe separate evaluation runs.
The hosted example supports clustering, overlap controls, and source-row inspection. Run the news workflow locally for development; public deployment requires a production-permitted NewsAPI plan and NEWSAPI_PRODUCTION_LICENSED=true. See https://newsapi.org/terms.
- Inspect the schema, missingness, uniqueness, and candidate semantic views.
- The public example combines all selected field values using offline TF-IDF/LSA. An optional local OpenAI embedding backend caches vectors by a hash of model plus source text.
- Combine the currently selected fields with an equal-weight vector mean and L2 normalization. Changing to a combination of already embedded fields stays local and does not make another embedding request.
- Explore K=3 through K=7 using Fuzzy C-Means. Compare argmax silhouette as a diagnostic only, fuzzy partition coefficient, partition entropy, Xie-Beni compactness/separation, and centroid separation.
- Convert distances to each learned fuzzy prototype into tie-aware, within-prototype percentile affinities and sharpen them to avoid high-dimensional distance concentration. These scores do not sum to one, so a row can belong to several clusters at the same cutoff. Native Fuzzy C-Means memberships are retained in the CSV for auditability.
- Generate a report, cutoff sensitivity table, overlap pairs, examples, and a PCA SVG.
The independent affinities are prototype scores, not calibrated probabilities of persona truth. They are appropriate for evaluating overlap behavior, but a production model should validate calibration against human-labeled examples.
From the project directory, using a Python environment with NumPy and pandas:
python -m src.run_prototypeThe clustering and reporting code does not use a generative model to assign memberships. OpenAI supplies semantic vectors; Fuzzy C-Means and the independently calibrated prototype affinities run locally.
Copy .env.example to .env, then add both keys without quotes:
NEWSAPI_KEY=your_key_here
OPENAI_API_KEY=your_key_here
OPENAI_MODEL=gpt-5-mini
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_MODEL and OPENAI_EMBEDDING_MODEL are optional; the local server defaults to gpt-5-mini and text-embedding-3-small respectively.
The repository-level .gitignore excludes .env files. Run the local dashboard server with:
python -m src.serve_dashboardOpen http://127.0.0.1:8765/clustering_explorer.html. The browser sends the generated query and the visible text of the selected persona rows to the local proxy. The proxy authenticates with the X-Api-Key header and searches English-language coverage from the last 30 days, newest-first. It fetches additional result pages when needed and retains up to 25 distinct events by comparing cached title-and-description embeddings plus local entity, event-type, amount, date, token, and title signals. Conflicting event evidence blocks merges, and every accepted merge retains auditable pair-level provenance.
The repository root is ready for a Python Heroku buildpack. requirements.txt triggers Python detection, .python-version selects Python 3.13, and Procfile starts the web process on the platform-provided PORT:
web: python -m src.serve_dashboard --host 0.0.0.0 --port $PORT
On Northflank, select its Heroku/buildpack build type with this directory as the build context. Expose the same injected application port and use /health for the HTTP health check. Configure OPENAI_API_KEY and NEWSAPI_KEY as secrets; OPENAI_MODEL and OPENAI_EMBEDDING_MODEL remain optional.
The proxy then sends only those visible persona descriptions and the retained articles' titles, descriptions, source names, dates, and URLs to the OpenAI Responses API. Structured output is used to produce ranked operational pain hypotheses, an evidence ledger, and a compact article-by-article audit. The dashboard presents that synthesis instead of a raw article feed. Both API keys stay in the server-side .env file and are never embedded in the HTML; OpenAI response storage is disabled for these requests.
The NewsAPI query builder receives persona-specific context only from the selected candidates' visible names and per-column evidence. It does not inject fixed market assumptions such as B2B, AI, SaaS, startup, or martech. Its generic event vocabulary is inferred from role and theme words that actually appear in that visible text, and the generated JSON includes the exact persona_descriptions used for auditability.
The demo dashboard's CSV-column controls can combine Job Title, company, location, headline, summary, and enriched profile summary text. For the public demo, offline TF-IDF/LSA vectors, role, industry, and business-need summaries, and K=3–7 models are precomputed. Loading it and changing the persona-count or overlap controls use the saved data without model calls. The clustering endpoint serves the same fixed payload and rejects other trait selections. Non-demo builds retain the explicit Update workflow for batching trait changes. A newly selected field makes an embedding request only for values missing from the local vector cache. Model-generated persona descriptions are validated to include a summary for every selected field. Each populated summary and overall description must cite deterministic, candidate-scoped evidence IDs; the server rejects unknown, duplicate, wrong-field, and cross-candidate references before resolving them to bounded exact CSV values. The dashboard keeps generated interpretation visually separate from inspectable source row IDs, fields, and values. Identifier, URL, numeric, duplicate enrichment, and all-null fields are excluded from these controls.
The public demo is sample-only. Upload customer CSV remains visible but disabled, and the server rejects upload requests before reading their content. The bundled fictional dataset loads with Use demo CSV. Upload support remains implemented for non-demo builds through UPLOADS_ENABLED in src/config.py.
Uploads are parsed in memory by the local server and support 3–500 non-empty rows, up to 80 columns, and files up to 8 MB. The server infers descriptive text columns while excluding likely identifiers, names, URLs, email addresses, and numeric measures. It also chooses initial high-signal fields and three display columns; every inferred descriptive field remains available as a checkbox.
After OpenAI embeddings and local fuzzy memberships are computed, one batched generative call receives bounded representative values—not the complete CSV—and generates natural persona names, descriptions, and a summary for every selected field. If that call fails, clustering remains usable with deterministic fallback labels. Uploaded source datasets live only in the current server process and are not written to the project; the vector cache contains hashes and numeric vectors rather than source text.
The dashboard's Workflow trace makes all external boundaries visible. The complete call-site and data-flow contract is documented in MODEL_CALLS.md: customer-field embedding, persona description generation, NewsAPI query generation, news-event embedding, and pain-point synthesis are model-assisted boundaries; CSV parsing, field inference, vector combination, fuzzy clustering, merge decisions, and validation are local.
Event deduplication has a versioned labeled calibration fixture and a deterministic evaluator in tests/fixtures/event_deduplication_v1.json and src/event_deduplication.py. Its reported metrics describe that deliberately small fixture only, not broad production accuracy.
Edit the obvious setting in src/config.py:
MEMBERSHIP_THRESHOLD = 0.40Then run:
python -m src.analyze_from_savedThat command reuses outputs/run_state.npz and refreshes memberships.csv, cluster_report.md, and embedding_projection.svg.
outputs/data_inspection.md: schema, inferred roles, missingness, samples, and semantic-view candidates.outputs/memberships.csv: original rows plus independent memberships, native FCM memberships, threshold flags, and stable source row IDs.outputs/cluster_report.md: K diagnostics, cutoff sensitivity, overlap pairs, and per-cluster examples.outputs/embedding_projection.svg: PCA projection with strong-cluster color and overlap outlines.outputs/run_state.npzandoutputs/run_metadata.json: saved embeddings, centroids, memberships, and metadata for threshold-only reruns.
Keep each future semantic view separate. Change SEMANTIC_FIELD, write its results to a distinct output directory, and compare memberships later by row_id. Do not concatenate employee count or other numeric attributes into embedding text. Numeric and categorical attributes can be analyzed after clustering as cluster descriptors.
See PRIVACY.md and THIRD_PARTY_NOTICES.md. The software license does not relicense third-party data or publications.
The dashboard links to /upload.html; review results open at /groups.html#<private-id>. CSV and Excel (.xlsx, .xls, first worksheet) are supported. The Excel reader is served locally from outputs/vendor/. Raw uploads, generated results, and upload embeddings stay in process memory, are scoped to a browser-session cookie, and expire after one hour since last dataset access, swept every 15 seconds. Private uploads bypass disk embedding caches and cannot be saved as a shared simulation. Timings are returned with the private result in memory. Optional JSONL diagnostics (DEMANDMAP_TIMING_LOG) contain only stage names, durations, statuses, random run IDs, and batch IDs, without customer values or filenames; disk logging is off by default. Existing bundled demo fixtures are separate from private uploads.
Set DEMANDMAP_ALLOWED_HOSTS to the deployment hostname(s), comma-separated, for additional domains. Set DEMANDMAP_HTTPS=1 behind the production HTTPS proxy to mark session cookies Secure.