Skip to content

Doc needs to be refined and discussed #20

Closed

Description

@zaoxing
  • LLM Internal Observability Infrastructure or something? e.g., DMI is an observability layer for LLM inference, giving real-time access to ANY internal model states with minimal overhead.
  • starting scripts need to be unified, e.g., including model name or independent of models.
  • adding a road map w/ call for contributions.
  • add a section of "why should I care", as the placeholders for potential uses.
  • a short visual demo or screenshots are helpful
  • missing performance claims. adding overhead numbers. Ideally, a comparison table to Transformer Lens, PyTorch Hooks, and others.
  • In addition to a script, maybe adding a short 10-line python example of usage.
  • When doc is a bit long, adding table of contents.
  • No need to specifically mention ClickHouse. Mentioning of visualizer/grafana.

e.g., some headline pitch

If you're:

  • debugging hallucinations and model bugs in production
  • studying safety/interpretability and steering
  • studying attention collapse
  • building distillation datasets
    and more...

You need internal visibility without rewriting your model or slowing your model by 10x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions