rishi

Rishi gives local and hosted models one Chat API. A conversation is portable Python history, so one backend can continue where another stopped.

Install

Base rishi includes LiteRT and the hosted backends. Add an extra for another local runtime:

pip install rishi              # LiteRT, hosted APIs, Claude Code, and GitHub Copilot
pip install 'rishi[llama]'     # local GGUF models
pip install 'rishi[mlx]'       # Apple Silicon
pip install 'rishi[ollama]'    # Ollama models
pip install 'rishi[all]'       # every available backend

Extras combine, for example rishi[llama,mlx]. Hosted providers need their usual credentials. Backend modules load only when selected.

Chat

Chat(model) creates one conversation. Each call appends to chat.hist. resp_text returns the answer text.

chat = Chat(gemma4_e2b)
print(resp_text(chat('Give one fact about lobsters.')))
print(resp_text(chat('Give one more.')))
chat.close()

Lobsters are crustaceans, which means they have a hard exoskeleton and eight legs.
Lobsters are known for their ability to change their color and texture to blend in with their surroundings, a behavior called camouflage.

Streaming

Pass stream=True to iterate markdown strings rendered by Urai. Rishi normalizes each backend’s events; use stream='raw' to receive those chunk dictionaries instead.

chat = Chat(gemma4_e2b)
for markdown in chat('Write a two-line poem about rain.', stream=True):
    print(markdown, end='', flush=True)
chat.close()

Soft drops tap on the pane,
Washing the world clean again.

Image

Place an image beside the prompt. Path, bytes, and PIL.Image inputs use the same message API.

chat = Chat(gemma4_e2b)
image = repo_root()/'nbs/images.jpeg'
display(Image.open(image))
print(resp_text(chat(['Describe this image in one sentence.', image])))
chat.close()

A medium-sized, reddish-brown German Shepherd is looking attentively off to the side with its mouth slightly open.

Audio

Place an audio file beside the prompt. Use a model with an audio input tower, such as Gemma 4 LiteRT.

chat = Chat(gemma4_e2b)
audio = repo_root()/'nbs/speech.wav'
print(resp_text(chat(['Transcribe this clip.', audio])))
chat.close()

Dancing in the masquerade, idol truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems asleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

One conversation across seven backends

Every backend stores the same canonical messages in chat.hist. Each leg below receives the previous leg through messages=, adds one turn, and passes the enlarged history on.

The example downloads four local models. The remote, Claude Code, and Copilot legs require their usual credentials.

legs = [
    ('LiteRT', gemma4_e2b, {}),
    ('Ollama', f'ollama/{ollama_qwen}', {'think': True}),
    ('MLX', mlx_qwen, {}),
    ('llama.cpp', llama_qwen, {'n_ctx': 4096}),
    ('cloud', 'gpt-4.1', {}),
    ('Claude', 'claude/haiku', {}),
    ('Copilot', 'copilot/gpt-4.1', {}),
]

history = []
for i, (name, model, options) in enumerate(legs):
    chat = Chat(model, messages=history, **options)
    prompt = (
        'The code word is amber. Start a travel log with one short sentence.'
        if i == 0 else
        f'What is the code word? Add one short sentence saying {name} received this conversation.'
    )
    reply = chat(prompt)
    print(f'{name}: {resp_text(reply)}')
    history = list(chat.hist)
    chat.close()

print([m['role'] for m in history])

LiteRT: Amber awaits.
Ollama: The code word is **amber**. Ollama received this conversation.

Fetching 9 files:   0%|                                                                                                                 | 0/9 [00:00<?, ?it/s]Fetching 9 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1912.59it/s]

MLX: The code word is **amber**. MLX received this conversation.

llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

llama.cpp: The code word is **amber**. llama.cpp received this conversation.
cloud: The code word is amber. Cloud received this conversation.
Claude: The code word is amber. Claude received this conversation.
Copilot: The code word is amber. Copilot received this conversation.
['user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant']

Backend guides

notebook	backend-specific topics
00_core.ipynb	runtime registration, compatibility names, skill install
01_llama.ipynb	GGUF loading, GPU offload, KV cache, mtmd media
02_litert.ipynb	.litertlm loading, GPU and NPU, bench()
03_mlx.ipynb	MLX text and media models, speculative decoding, caches
04_remote.ipynb	hosted providers and server-side tools
06_claude.ipynb	Claude Code sessions, transcripts, MCP policy
07_copilot.ipynb	authentication, editor headers, model listing
08_ollama.ipynb	daemon lifecycle, thinking levels, /api/show