Skip to content

Models

DioscHub does not ship a model of its own — it runs on the LLM providers you bring. You register a provider’s credentials once, list the models you want available, and then each Assistant picks one of those models. DioscHub supports OpenAI, Anthropic, Azure OpenAI, Google, AWS Bedrock, Ollama, and any OpenAI-compatible endpoint.

A provider credential is a named record — “Production OpenAI”, “Azure — East US” — holding the provider type and its connection details (API key, base URL, deployment, and so on). Create one in the admin portal (or POST /admin/llm-provider-configs).

You can validate a key before you rely on it: DioscHub can run one minimal completion against the provider to confirm the credential works (Test key).

A provider’s base URL (Ollama, LM Studio, Azure, or any OpenAI-compatible endpoint) follows the same egress rules as MCP server URLs. They apply when you create the credential, when you change its endpoint, and in Test key. Embedding uses the same stored credentials, so it is covered too.

  • Always refused: link-local and cloud-metadata addresses (169.254.0.0/16, metadata.google.internal, and similar).
  • Private and loopback ranges (RFC 1918, CGNAT, ULA, loopback) are allowed only when listed in EGRESS_TRUSTED_NETWORKS. With it unset, outside production only loopback is trusted; in production nothing is.
  • Plain http is allowed by default outside production. In production it is refused for targets outside the trusted networks unless EGRESS_ALLOW_HTTP=true.

A refused URL on save is a 400 with code EGRESS_BLOCKED and the reason. A hostname that does not resolve at save time is accepted. Test key fails closed: for a refused target or a network failure (connection refused, DNS failure, timeout) it shows only “The endpoint could not be reached or is not allowed.”; provider HTTP errors such as a 401 invalid key are still shown. Saved credentials keep working, and nothing is checked again at chat time. The variables are described in production hardening.

A model entry names a specific model — gpt-4o, a Claude model, a Gemini model — and records what DioscHub needs to route and budget it: its provider, its context window, its maximum output, and its per-token cost for usage accounting. Add the models you intend to use (POST /admin/models), and optionally link each one to a provider credential you registered above.

How DioscHub finds the credential for a model, in order:

  1. The provider credential linked to that model, if you set one.
  2. A shared credential registered for that provider.
  3. The provider’s environment variable (for example OPENAI_API_KEY), as a fallback.

Each Assistant selects its model in llmConfig:

FieldWhat it controls
modelThe model id to run — this is what picks the provider and credential at run time.
maxTokensA ceiling on tokens generated per turn (capped further by the model’s own maximum).
reasoningEffortHow hard a reasoning-capable model thinks: minimal, low, medium, or high.
temperatureSampling temperature — best-effort; reasoning models ignore it (below).
fallbackModelsAn ordered list of up to 2 backup models, tried when the primary fails (below).

The model id is the selector. DioscHub looks up that model to decide which provider and credential to use — so pointing an Assistant at a different provider is a matter of choosing a model that belongs to it. If an Assistant has no model set, it falls back to the deployment’s environment defaults.

An Assistant can name up to two fallback models in fallbackModels, an ordered list of { provider, model } entries. If the primary model’s provider fails, DioscHub answers the turn from the first fallback, then the second. Set them on the Assistant’s Setup tab; see Assistants & Roles.

A turn moves to the next model only when the failure is a provider error (a 5xx, an overloaded provider, a timeout or a transport failure), a rate limit, or an exhausted quota. A rejected request (bad request, authentication, context window exceeded) or a content-filter refusal is not retried on another model. A turn also never moves on once the failing model has started streaming text, or after the user presses Stop. If every model fails, the user sees the usual “couldn’t answer just now” message with a reference.

Retries follow the order: every model that has a successor is called with provider retries off, so a failure hands over at once; the last model in the list keeps the provider SDK’s default retries. An Assistant with no fallbacks behaves as it always has.

  • Fallbacks count against your active models. A fallback must be an active model, and active models are capped by your tier (activeModels; see Licensing & tiers). The Free tier allows one active model, so it cannot hold a second one as a fallback; there is no separate tier switch for fallback. On Pro, the primary and its fallbacks must fit within the active-model cap.
  • A fallback must be compatible. It cannot be the primary or repeat another entry, must belong to the provider named in its entry, and must support tool_use and vision whenever the primary does. Saving an Assistant that breaks these rules is refused with an error naming the entry.
  • A model in use as a fallback is protected. Deactivating or deleting it is refused while an Assistant lists it, and replacing a model when you activate another one rebinds the Assistants’ fallback entries too.
  • Routing wins. While model routing is on for the Assistant, its own fallback order is used and fallbackModels is ignored.

When a fallback answers, the chat shows no fallback notice; the widget’s context details name whichever model answered, as they do for a routed turn. You see the failover in the admin portal: a marker on the reply in the Conversations explorer, “Answered by” on the model health list, and a Failovers count on the Usage page.

reasoningEffort turns on a model’s internal reasoning, and DioscHub maps it to each provider’s own control (extended thinking on Anthropic, reasoning effort on OpenAI and Azure, thinking level on Google). Two consequences to plan for:

  • Reasoning consumes your token budget. The thinking a model does counts against maxTokens. Set maxTokens too low and the reasoning eats the budget before any answer is produced, and the reply comes back truncated. Give reasoning models generous headroom.
  • Reasoning models ignore temperature. When reasoning is active, the provider fixes sampling itself and your temperature is not applied. Treat temperature as effective only on non-reasoning models.

Next: Knowledge bases.