Models
DioscHub does not ship a model of its own — it runs on the LLM providers you bring. You register a provider’s credentials once, list the models you want available, and then each Assistant picks one of those models. DioscHub supports OpenAI, Anthropic, Azure OpenAI, Google, AWS Bedrock, Ollama, and any OpenAI-compatible endpoint.
Register a provider credential
Section titled “Register a provider credential”A provider credential is a named record — “Production OpenAI”, “Azure — East US” — holding the provider
type and its connection details (API key, base URL, deployment, and so on). Create one in the admin portal
(or POST /admin/llm-provider-configs).
You can validate a key before you rely on it: DioscHub can run one minimal completion against the provider to confirm the credential works (Test key).
Base URL rules
Section titled “Base URL rules”A provider’s base URL (Ollama, LM Studio, Azure, or any OpenAI-compatible endpoint) follows the same egress rules as MCP server URLs. They apply when you create the credential, when you change its endpoint, and in Test key. Embedding uses the same stored credentials, so it is covered too.
- Always refused: link-local and cloud-metadata addresses (
169.254.0.0/16,metadata.google.internal, and similar). - Private and loopback ranges (RFC 1918, CGNAT, ULA, loopback) are allowed only when listed in
EGRESS_TRUSTED_NETWORKS. With it unset, outside production only loopback is trusted; in production nothing is. - Plain
httpis allowed by default outside production. In production it is refused for targets outside the trusted networks unlessEGRESS_ALLOW_HTTP=true.
A refused URL on save is a 400 with code EGRESS_BLOCKED and the reason. A hostname that does not
resolve at save time is accepted. Test key fails closed: for a refused target or a network failure
(connection refused, DNS failure, timeout) it shows only “The endpoint could not be reached or is not
allowed.”; provider HTTP errors such as a 401 invalid key are still shown. Saved credentials keep
working, and nothing is checked again at chat time. The variables are described in production
hardening.
List the models you want available
Section titled “List the models you want available”A model entry names a specific model — gpt-4o, a Claude model, a Gemini model — and records what
DioscHub needs to route and budget it: its provider, its context window, its maximum output, and its
per-token cost for usage accounting. Add the models you intend to use (POST /admin/models), and optionally link each one to a provider credential you registered above.
How DioscHub finds the credential for a model, in order:
- The provider credential linked to that model, if you set one.
- A shared credential registered for that provider.
- The provider’s environment variable (for example
OPENAI_API_KEY), as a fallback.
Choose a model per Assistant
Section titled “Choose a model per Assistant”Each Assistant selects its model in llmConfig:
| Field | What it controls |
|---|---|
model | The model id to run — this is what picks the provider and credential at run time. |
maxTokens | A ceiling on tokens generated per turn (capped further by the model’s own maximum). |
reasoningEffort | How hard a reasoning-capable model thinks: minimal, low, medium, or high. |
temperature | Sampling temperature — best-effort; reasoning models ignore it (below). |
fallbackModels | An ordered list of up to 2 backup models, tried when the primary fails (below). |
The model id is the selector. DioscHub looks up that model to decide which provider and credential to use — so pointing an Assistant at a different provider is a matter of choosing a model that belongs to it. If an Assistant has no model set, it falls back to the deployment’s environment defaults.
Fallback models
Section titled “Fallback models”An Assistant can name up to two fallback models in fallbackModels, an ordered list of
{ provider, model } entries. If the primary model’s provider fails, DioscHub answers the turn from the
first fallback, then the second. Set them on the Assistant’s Setup tab; see Assistants &
Roles.
A turn moves to the next model only when the failure is a provider error (a 5xx, an overloaded provider, a timeout or a transport failure), a rate limit, or an exhausted quota. A rejected request (bad request, authentication, context window exceeded) or a content-filter refusal is not retried on another model. A turn also never moves on once the failing model has started streaming text, or after the user presses Stop. If every model fails, the user sees the usual “couldn’t answer just now” message with a reference.
Retries follow the order: every model that has a successor is called with provider retries off, so a failure hands over at once; the last model in the list keeps the provider SDK’s default retries. An Assistant with no fallbacks behaves as it always has.
- Fallbacks count against your active models. A fallback must be an active model, and active models are
capped by your tier (
activeModels; see Licensing & tiers). The Free tier allows one active model, so it cannot hold a second one as a fallback; there is no separate tier switch for fallback. On Pro, the primary and its fallbacks must fit within the active-model cap. - A fallback must be compatible. It cannot be the primary or repeat another entry, must belong to the
provider named in its entry, and must support
tool_useandvisionwhenever the primary does. Saving an Assistant that breaks these rules is refused with an error naming the entry. - A model in use as a fallback is protected. Deactivating or deleting it is refused while an Assistant lists it, and replacing a model when you activate another one rebinds the Assistants’ fallback entries too.
- Routing wins. While model routing is on for the Assistant, its own fallback
order is used and
fallbackModelsis ignored.
When a fallback answers, the chat shows no fallback notice; the widget’s context details name whichever model answered, as they do for a routed turn. You see the failover in the admin portal: a marker on the reply in the Conversations explorer, “Answered by” on the model health list, and a Failovers count on the Usage page.
Reasoning models need room to think
Section titled “Reasoning models need room to think”reasoningEffort turns on a model’s internal reasoning, and DioscHub maps it to each provider’s own
control (extended thinking on Anthropic, reasoning effort on OpenAI and Azure, thinking level on Google).
Two consequences to plan for:
- Reasoning consumes your token budget. The thinking a model does counts against
maxTokens. SetmaxTokenstoo low and the reasoning eats the budget before any answer is produced, and the reply comes back truncated. Give reasoning models generous headroom. - Reasoning models ignore
temperature. When reasoning is active, the provider fixes sampling itself and yourtemperatureis not applied. Treattemperatureas effective only on non-reasoning models.
Next: Knowledge bases.