Chat files & indexing
Every file in a user’s library is stored first and indexed in the background: files uploaded in the chat,
files your client-side tools deliver,
files an MCP tool writes with put_file, and files the assistant generates. Indexing reads the file’s
text so the assistant can read it with read_file, and so file_search can return passages from it.
Each file reports a status that says how far this got and, when something went wrong, why. How the
assistant then reads a file is in How the assistant reads files.
This page is for integrators building on the widget or the files API. The upload limit, quota and retention an administrator sets are in File storage limits, quota & retention.
How a file is indexed
Section titled “How a file is indexed”Once an upload is accepted, the file is processing and a background job indexes it:
- Read. The hub reads the stored file as a stream. Plain text, Markdown, JSON and CSV are decoded as they are read. PDFs are read page by page and Excel workbooks sheet by sheet. A Word document is read in one piece. Other formats are read by their own parsers; see Supported formats.
- Store the text. The extracted text is written to the file store, next to the file, as the file’s stored text. There is no length cap: a long document is stored and indexed in full.
- Split. The text is cut into overlapping chunks. Each chunk records where it came from: the PDF page, the sheet, row range and A1 range, or the line range. Spreadsheet and CSV chunks also record their column headers.
- Embed. Chunks are embedded and written in batches, so memory use doesn’t grow with the file’s size.
The new index becomes searchable only when the whole run completes. Until then, any earlier index of the same file keeps serving search, so re-indexing a file never leaves a gap. A run that fails leaves the earlier index in place.
Indexing runs in the hub’s background job queue. Temporary failures, such as an embedding provider’s rate limit or a storage error, are retried with backoff. Permanent ones, such as a damaged file, settle at once. At most two indexing jobs run at once per library (a user’s library, an anonymous session, or a knowledge base), so one user uploading a hundred files doesn’t hold up everyone else.
DioscHub reads text from documents (PDF, Word, PowerPoint .pptx, OpenDocument, EPUB, RTF, mail), tables
(Excel .xlsx, OpenDocument spreadsheets, CSV, TSV) and text, code and markup files. Images are not
indexed: an image is ready, a vision model can view it, and file_search doesn’t return it. Media,
archives and unknown types are stored without being read, and executables are refused at upload. The full
list, and how an administrator adds types of their own, is in File types, kinds & parser
limits.
File status
Section titled “File status”Every file is in one of six states. The hub also derives two flags from the state: canRead (the
assistant can read the file) and canSearch (file_search can return passages from it). Read the flags
instead of working them out from the state yourself.
status | What it means | canRead | canSearch |
|---|---|---|---|
processing | Being indexed. | Only when text from an earlier run is stored | No |
ready | Read and indexed. An image is ready but not indexed. | Yes | Yes. No for an image. |
search_unavailable | The text was read and stored, but the search index wasn’t built. | Yes | No |
no_text | The file was read and holds no text, for example a scanned PDF with no text layer. | No | No |
stored_only | DioscHub doesn’t read this kind of file, or its content doesn’t match its type. It is stored and can be downloaded. | No | No |
failed | The file couldn’t be read. | No | No |
Reasons
Section titled “Reasons”A file that isn’t ready or processing carries a reason: a fixed code and a fixed message. The
message is the one the user sees. It never contains the underlying error text, which stays on the hub.
reason.code | status | reason.message |
|---|---|---|
over_quota | search_unavailable | Not searchable: the library is over its storage quota. |
embedding_failed | search_unavailable | Not searchable: the search index could not be built. |
no_text_layer | no_text | No text could be found in this file. |
stored_only_kind | stored_only | This kind of file is stored but not read. |
type_mismatch | stored_only | The file’s content doesn’t match its type. |
corrupt | failed | The file could not be read; it may be damaged. |
encrypted | failed | The file is password-protected. |
parser_limit | failed | The file is too large to read. |
timeout | failed | Reading the file took too long. |
source_missing | failed | The file’s stored content is missing. |
retry_exhausted | failed | Processing kept failing. Try Re-index. |
Warnings
Section titled “Warnings”A file that was mostly read is still ready (or search_unavailable), and lists what it skipped in
warnings. Each warning is a fixed code, a location and a retryable flag:
code | location | What happened |
|---|---|---|
page_unreadable | page N | A PDF page couldn’t be read, or took too long (30 seconds by default). The stored text holds the line [page N could not be read] in its place. |
sheet_unreadable | sheet <name> | An Excel worksheet couldn’t be read. The stored text holds [sheet "<name>" could not be read] in its place. |
invalid_encoding | line L | The text had bytes that aren’t valid UTF-8. They became replacement characters (U+FFFD), and L is the first line affected. A file gets at most one of these. |
warnings_truncated | <n> more | The file had more than 100 warnings. The 100th entry counts the rest. |
text_truncated | none | Only on files indexed before the upgrade: the earlier version stopped reading a file at a size cap, so the stored text is incomplete. Re-index the file to read all of it. New indexing never writes this code. |
A warning marked retryable (a page that timed out) is tried once more at the end of the run. If the page
reads then, its text is added to the end of the stored text after the line [page N, recovered], and the
warning goes. If it fails again, the warning stays with retryable: false.
A file is failed only when nothing could be read at all: every page, or every sheet. Plain text is never
split into sections, so it is either read whole or not at all.
Treat the code lists as open: show a code you don’t recognise as it is. Files whose text was cut short
by the length limit of earlier releases can carry a text_truncated warning (location null) until they
are indexed again.
Progress
Section titled “Progress”While a file is processing, progress is { step, done, total }; otherwise it is null.
step | done / total |
|---|---|
indexing | Bytes read for text formats, pages for a PDF, worksheets for an Excel workbook, 0 then 1 for Word |
retrying | Retryable sections tried again, out of how many there are |
Progress is written at most once every two seconds per file, and done never goes down within a step.
Where the status appears
Section titled “Where the status appears”The same fields are on every file the hub returns:
- the user’s library,
GET /api/filesandGET /api/files/:id; - a session’s files,
GET /api/sessions/:id/files; - the
files:statusevent; - the files of a knowledge base, in the admin API.
An abridged file from GET /api/files:
{ "id": "9b2f6c1e-4d0a-4f7e-9a51-2c8d6e0b7f13", "filename": "handbook.pdf", "mimeType": "application/pdf", "sizeBytes": 2483021, "origin": "UPLOAD", "status": "ready", "reason": null, "warnings": [{ "code": "page_unreadable", "location": "page 14", "retryable": false }], "progress": null, "canRead": true, "canSearch": true, "isIndexed": true, "indexBytes": 412880, "derivedBytes": 0, "createdAt": "2026-11-06T09:12:44.000Z", "expiresAt": null}Live status: the files:status event
Section titled “Live status: the files:status event”Each time the hub writes a file’s status or progress, it pushes files:status with the file’s status
fields. The widget applies it in place, and your code can listen to it too:
diosc('on', 'files:status', (file) => { // { fileId, filename, mimeType, status, reason, warnings, progress, canRead, canSearch, isIndexed } if (file.status === 'failed') showProblem(file.fileId, file.reason?.message);});- Who gets it. For a signed-in user, every widget that user has open on that Assistant, whichever session uploaded the file, and also for files uploaded outside any session. For an anonymous visitor, that visitor’s session only. Knowledge-base files push nothing.
- What’s in it. The status fields, so you never need to refetch to learn them. It never carries a
storage key, error text or any of the file’s text.
isIndexedis the deprecated alias, as above. - Pushes aren’t replayed. A widget that was disconnected catches up by reading the list.
files:updated still fires, with { fileId, filename }, to the session that uploaded a file once that
file is indexed. It says only that something changed. Prefer files:status.
Re-index
Section titled “Re-index”A user can re-index one of their own files when it is failed or search_unavailable: after freeing
storage for an over_quota file, say, or once the embedding provider is back. The widget’s Files dialog
shows a Re-index button on those rows.
A ready spreadsheet or CSV file with no workbook outline can
be re-indexed too, to build its outline: a file indexed before outlines existed, or a CSV whose outline a
redaction removed. This applies to .xlsx, CSV and TSV files that have text stored, as the file was
classified. An .ods file, or a spreadsheet with no text stored, can never have an outline, so its
re-index is refused. The widget has no button for this case yet; call the API, or ask an administrator for
a bulk Re-index.
Over the API:
POST /api/files/:id/reindex
The request has no body and is authenticated like the other /api/files routes: the widget’s REST
token, or an anonymous visitor’s files token. The Role needs the
files feature.
| Response | When |
|---|---|
200 with the file, now processing | The Re-index is accepted and an indexing job is queued. |
404 | The file isn’t in the caller’s library, or has expired. |
409 with code: "not_reindexable" | The file isn’t failed, search_unavailable, or a ready spreadsheet or CSV without an outline: for example a ready file that isn’t a table, a spreadsheet that already has its outline, an .ods file, or a file that is processing, no_text or stored_only. Another Re-index, or an overwrite, may have got there first. |
429 with a Retry-After header | The caller spent the Re-index limit: by default 5 per minute, counted per caller like the other request limits (REINDEX_LIMIT_PER_MINUTE). |
The 409 body is:
{ "code": "not_reindexable", "message": "Only a failed file, one whose search is unavailable, or a ready spreadsheet or CSV with no workbook outline can be re-indexed"}The Retry-After header is readable from a cross-origin page, so the widget can say how long to wait:
“Too many re-index requests. Try again in 42 seconds.”
Re-index is not a new upload, so the upload quota check doesn’t run. The indexer checks only growth: re-indexing a file that hasn’t changed never trips the quota. A file that now produces a larger index can still stop at the quota.
Storage quota and indexing
Section titled “Storage quota and indexing”When a library has a storage quota, the indexer checks it
before each batch of chunks is embedded. If the batch would take the library past the quota, embedding
stops and the file becomes search_unavailable with reason over_quota. Its full text is still stored, so
the assistant can read the file, but file_search can’t return passages from it. To make it searchable,
free space, then Re-index it.
Knowledge-base files never count against a user’s quota, and the background re-indexing an administrator starts (below) never stops at one.
How the assistant reads files
Section titled “How the assistant reads files”The model reaches a file’s content only through tools it calls. Nothing from a file is pasted into the prompt: a user message lists its attached files, and the model then reads, searches or queries them.
| Tool | What it does | Available when |
|---|---|---|
read_file | Reads one of the user’s files, page by page: its extracted text, or its original bytes | The licence includes file handling and the user’s Role has the files feature |
file_search | Finds passages across the user’s files, or inside one file | Same as read_file |
describe_workbook, load_table, query_table | Sums, filters and other table maths on the user’s own spreadsheets and CSV files; see Spreadsheets | Same as read_file |
read_kb_file | Reads a knowledge-base file, page by page | The licence allows knowledge bases |
kb_search | Finds passages in the knowledge bases linked to the Assistant and Role | Same as read_kb_file |
The reading and searching tools are read-only and never ask for approval. The licence and the Role are checked again on every turn, so a change applies from the next message.
Every piece of file text that read_file, read_kb_file, file_search, kb_search and the table tools
hand the model is framed as data, not instructions; see File content the model reads.
The model’s file list
Section titled “The model’s file list”A user message with attachments carries a <files> block that lists them for the model:
<files> <file id="9b2f6c1e-…" name="handbook.pdf" mimeType="application/pdf" sizeBytes="2483021" kind="document" status="ready" pages="42" approxTokens="31800"/></files>| Attribute | What it holds |
|---|---|
id, name, mimeType, sizeBytes | The file’s id, name, type and size |
kind | The file’s kind, other when it has none |
status | The file’s status when the message was sent |
reason | The reason code, only when the status isn’t ready |
pages | The page count, when known |
approxTokens | The size of the extracted text in tokens, rounded up to a multiple of 100 |
hint | The hint of the admin file type rule that matched the file, cut to 200 characters |
The list is a snapshot: the model is told that the status may have changed since, and that read_file
reports the current one. pages and approxTokens let it decide how much of a file to read. Every value
is escaped, so a file name or hint can’t close the block or add a row.
Read the text
Section titled “Read the text”read_file with the default text view returns one page of the file’s stored
text:
{ "fileId": "9b2f6c1e-4d0a-4f7e-9a51-2c8d6e0b7f13", "name": "handbook.pdf", "kind": "document", "view": "text", "text": "<file-data boundary=\"…\" origin=\"read_file\" file=\"handbook.pdf\">\n[page 1]\nEmployee handbook…\n</file-data boundary=\"…\">", "location": { "from": { "page": 1 }, "to": { "page": 3 } }, "nextCursor": "eyJmaWxlSWQiOi…", "progress": { "readBytes": 30412, "totalBytes": 412880, "done": false }}- Markers. The text starts with a marker and carries one wherever a new section starts:
[page N]for a PDF,[sheet "Name" rows A-B]for a spreadsheet,[line N]for text and code. They come from the file’s search index. A file without one (search_unavailable) gets a line marker at the start of each page. Inread_filetext a spreadsheet is marked by sheet and row range. Search hits also carry the chunk’s A1 range and column headers (see From a search hit into the file), and the table tools work by A1 area. - Paging. While
nextCursorisn’tnull, the model callsread_fileagain with the same file, the same view and that cursor.progresssays how much has been read;doneistrueon the last page. The full text can be read, however long it is. A page never splits a character. - Starting somewhere.
atstarts the read at{ "page": N },{ "sheet": "Name", "row": N }(the row is optional),{ "line": N }or{ "offset": N }. A location that doesn’t exist is refused withlocation_not_foundand a message that says what does exist, such as the page range or the sheet names. A cursor, when given, wins overat.
Page size
Section titled “Page size”A page, the whole JSON result included, stays within a page budget the hub works out each turn:
- a quarter of the smallest context window among the models the session can use (the Assistant’s model, its routing candidates and its fallbacks), counted at 4 characters per token;
- never more than the tool-result limit,
DIOSC_MAX_TOOL_RESULT_CHARS(default 65,536 characters), and never less than 4,096 characters.
A model whose context window isn’t known counts as 32,768 tokens. With a 128K-token model alone, a page is 65,536 characters; if one fallback has a 32K window, every page is 32,768 characters, so a page still fits whichever model answers. Nothing the model sends changes the budget.
Cursors
Section titled “Cursors”A cursor is opaque. It is tied to the file, the view and the version of the stored object it was issued for, and it grants nothing: the file’s owner is checked on every call.
status | When |
|---|---|
cursor_stale | The file was re-indexed or redacted since the cursor was issued. |
cursor_invalid | The cursor isn’t one the hub issued for this file and view. |
changed | The file changed while the page was being read. No page is returned. |
Each refusal tells the model to start again without a cursor. Text already read is never mixed with text from the new version.
The raw view
Section titled “The raw view”view: "raw" returns the file’s original bytes instead of its extracted text. It works for every file,
whatever its kind or status, including files the text view can’t read. It’s for questions the extracted
text can’t answer, such as why a strict server rejects a PDF: the version, encryption, scripts or forms,
embedded fonts, incremental updates, a broken cross-reference table.
Printable ASCII, line feeds, carriage returns and tabs come through as they are, a backslash as \\, and
every other byte as \xHH. Each result carries totalBytes, the byte range it covers, kind,
mimeType, nextCursor and progress, within the same page budget. The raw view always starts at the
first byte and moves only by cursor; an at offset is refused.
After a redaction deleted a
binary file’s original bytes, the raw view answers original_removed, and reads nothing else in its
place.
From a search hit into the file
Section titled “From a search hit into the file”Every file_search and kb_search hit names its file and where it sits:
{ "fileId": "9b2f6c1e-4d0a-4f7e-9a51-2c8d6e0b7f13", "name": "handbook.pdf", "location": { "kind": "page", "page": 14 }, "startOffset": 18842, "score": 0.71, "content": "<file-data boundary=\"…\" origin=\"file_search\" file=\"handbook.pdf\">\n…\n</file-data boundary=\"…\">"}location is a page, a sheet and row range ({ "kind": "sheet", "sheet", "rowFrom", "rowTo", "range" }),
a line range ({ "kind": "lines", "lineFrom", "lineTo" }) or null. For a spreadsheet hit, range is the
A1 area the chunk covers, such as "C2:E40"; it is left out when it isn’t a plain A1 range. To read around a
hit, the model calls read_file (or read_kb_file for a kb_search hit) with at: { "offset": startOffset }.
Hits no longer carry storage keys or raw chunk metadata.
Column headers are file text, so they never go in location. A hit from a spreadsheet or CSV file whose
header row is known starts its framed content with a Columns: line instead:
Columns: Region, Quarter, RevenueEast,Q1,1200…The line holds at most 64 headers, each cut to 200 characters. The same headers go into the text that is embedded for each chunk, so a question that names a column, such as “Revenue for East”, finds the right rows.
file_search also names the files it couldn’t search, with their status and reason:
- with
scope: "file", infile:{ fileId, name, kind, status, reason, canRead }; - with
scope: "user", innotSearchable(the 10 newest) andnotSearchableTotal(how many there are).
Files that aren’t ready
Section titled “Files that aren’t ready”read_file never returns empty text for a file it can’t read. It answers with the file’s status and a
fixed message: the reason when the file has one, or a fixed text for the status, such as
“This file is still being processed and has no readable text yet. Try again shortly.” for a file being
indexed for the first time.
The text view reads a file when canRead is true: ready and search_unavailable files, and a
processing file that already has text from an earlier run. For a no_text or stored_only file, and
for a readable file whose stored text is missing, the message adds that the raw view can still read the
original bytes. A no_text or stored_only answer also carries the hint of the file type rule that
matched the file, when there is one.
A file that isn’t in the caller’s library gets one answer, whatever the cause: another user’s file,
another session’s, an expired file, a file DioscHub derived for its own use (other than a workbook
picture of the caller’s own workbook), or an id that doesn’t exist.
It is not_found, “File not found. If this id came from kb_search, use read_kb_file instead.” An
anonymous visitor reads only their own session’s files, and only when the session’s Role had files on
when the session started.
Images
Section titled “Images”read_file on an image doesn’t return its bytes. When the image can be shown, the hub adds it to the
conversation right after the tool results, as one message that carries the image the way a user-attached
image is carried. That message is framed as file data from read_file and marked as the hub’s own: it is
not a user turn, and the widget’s history, the admin conversation transcript, session search and
compaction leave it out. The read_file result names the frame’s boundary, so the model can tell the real
image from text that only looks like one.
An image is shown only when every model the session can use (the Assistant’s model, its routing
candidates and its fallbacks) has the vision feature and comes from a provider whose message order
DioscHub has verified for it: OpenAI, Azure OpenAI, OpenAI-compatible servers, Anthropic and Ollama. A
chain that includes AWS Bedrock, Google, Mistral, DeepSeek, Groq or xAI doesn’t get images from
read_file. Images a user attaches to a message are not affected.
status | When |
|---|---|
image_attached | The image is shown. |
vision_unavailable | A model the session can use can’t view images, or its provider isn’t one of those above. |
image_unsupported | The format isn’t PNG, JPEG, WebP or GIF. |
image_too_large | The stored image is over the vision limit, VISION_MAX_INPUT_BYTES (default 32 MiB). |
image_limit | Two images have already been shown since the user’s last message. |
image_repeat | The same image was already shown since the user’s last message. |
image_unavailable | The image couldn’t be loaded. |
A shown image is scaled down to at most 1,568 pixels on its longer side, as attached images are; one
that is still over 4.5 MiB, or is over the pixel limit (VISION_MAX_INPUT_PIXELS), is image_unavailable.
Each image counts as 1,600 tokens in the usage quota and compaction estimates. The turn always goes on: an
image that can’t be shown leaves the model a fixed message saying why. If an MCP tool is bound under the
name read_file as well, images from read_file are off.
Knowledge-base files: read_kb_file
Section titled “Knowledge-base files: read_kb_file”read_kb_file reads knowledge-base files the way read_file reads a user’s: the same views, pages,
markers, cursors, at locations, page budget and refusals. It reads a file only when kb_search could
search it: its knowledge base is active and linked
to the Assistant or to the user’s Role, and, when the bind carried claims, the link’s permissions allow
it. That is resolved again on every call. Any other id, a user’s chat file
included, gets the same not_found. An image in a knowledge base is not shown; the raw view can still read
it.
What users and administrators see
Section titled “What users and administrators see”- The widget’s history. When a conversation is reopened, each tool result in it, a
read_filepage included, is cut to 4,000 characters, followed by… [N more characters]. An imageread_fileshowed never appears as a message. - The conversation transcript in the admin Conversations explorer
shows up to 4,000 characters of each tool result too,
read_filepages included. - Compaction. When older turns are summarised, the summariser reads at most 4,000 characters of each tool result, so ten full pages can’t overflow the background model.
What the assistant read stays in the conversation after the file changes or is deleted; see Read file content stays in the conversation.
Spreadsheets
Section titled “Spreadsheets”read_file and file_search read a spreadsheet as text. For numbers (a sum, a count, a filter, a group
by) the assistant uses three table tools instead, in this order:
describe_workbookshows how the file is laid out, without reading its data.load_tableloads one table from it, so it can be queried.query_tableruns one read-only SQLSELECTover the loaded tables and returns rows.
What each format supports:
| Format | read_file, file_search | Table tools | Pictures |
|---|---|---|---|
Excel .xlsx | Yes | Yes | Yes |
| CSV, TSV | Yes | Yes, the whole file is one table | No |
OpenDocument .ods | Yes | No | No |
Excel 97–2003 .xls | Stored, not read: only read_file’s raw view | No | No |
Which row applies is decided from how the file was classified
when it was stored, never from its name alone. The table tools take the id of one of the user’s own files,
such as one from the message’s file list. They come with the Files tools: the
licence must include file handling and the user’s Role needs the files feature.
The workbook outline: describe_workbook
Section titled “The workbook outline: describe_workbook”When a spreadsheet or CSV in a user’s library is indexed, the hub also stores an outline of it.
describe_workbook reads only that outline; it never opens the file itself.
For an Excel workbook, the outline lists every sheet in workbook order, with:
- its visibility (
visible,hiddenorveryHidden) and used range; - merged cells, and hidden rows and columns;
- the table regions found on the sheet (at most 20), each with its A1 range, a confidence (
high,mediumorlow), its header rows and its column names; - pictures, with the cell each one sits on (see Pictures in workbooks);
- charts, with their title, type and the ranges they plot (charts are listed, not drawn);
- a preview of the first 10 non-empty rows and 10 columns, each cell labelled with its A1 reference and cut to 80 characters. A cell in a hidden row, column or sheet is labelled as hidden.
For a CSV or TSV file, the outline holds its columns, its record count (the header line included) and a preview.
The answer stays within 24,000 characters. When a workbook is larger, the first sheets are shown in full and the later ones by name and visibility only.
Load a table: load_table
Section titled “Load a table: load_table”For an Excel workbook, the model gives the sheet name exactly as describe_workbook listed it, the
table’s A1 area (for example B2:F40) and, when the header isn’t exactly one row, headerRows (0 to 10,
default 1). For a CSV or TSV file it gives only the file id: the whole file is loaded, with its first record
as the header, and a sheet, range or headerRows it sends anyway is ignored, with a note saying so.
The answer describes the table it loaded:
{ "name": "s1_a1_d20", "sheet": "<file-data …>Sales</file-data …>", "range": "A1:D20", "headerRows": 1, "rowCount": 19, "columns": [ { "name": "revenue", "header": "<file-data …>Revenue</file-data …>", "type": "DOUBLE", "column": "D", "profile": { "nulls": 0, "min": "<file-data …>120.5</file-data …>", "max": "<file-data …>980</file-data …>", "distinct": 19 } } ], "samples": [[{ "ref": "A2", "value": "<file-data …>East</file-data …>" }]], "notes": ["Distinct counts in the profiles are approximate."]}- Names. The table is
s<sheet number>_<top-left>_<bottom-right>, such ass1_a1_d20; a CSV’s table iscsv. Columns are named from their headers, in lower case with_for other characters; an empty header becomescol_<letter>. - Types. Each column is
BOOLEAN,DATE,TIMESTAMP,BIGINT,DOUBLEorVARCHAR, decided from its values. A column that mixes numbers and text, such as one with a strayN/A, isVARCHAR. In a CSV, codes with a leading zero, such as ZIP codes, stay text. - Rows. Every table has
_row, the sheet row number (for a CSV, the record number, the header being 1). Hidden rows are loaded and marked in a_row_hiddencolumn, hidden columns are markedhidden, and a hidden sheet loads with a note giving its state. - Samples. Up to 5 rows of up to 30 columns, each value cut to 80 characters and labelled with its A1 reference.
- Size. The answer is at most 16,000 characters. On a wide table, the columns past what fits are listed by name and type only, and a note says so.
A loaded table is kept as a derived file of the spreadsheet, so a later turn or session can query it without loading it again. A spreadsheet keeps at most 8 loaded tables; loading a ninth drops the one used least recently. The kept tables count toward the user’s storage quota. When the quota is full, the table still answers but isn’t kept: it stays usable on that hub instance for up to an hour, and a note tells the model so. When the file changes (it is overwritten or redacted), its tables are dropped and built again on the next load.
Query it: query_table
Section titled “Query it: query_table”query_table takes the file id and one SQL SELECT (a WITH … SELECT is fine) over the tables
load_table returned for that file. Anything else is refused. It answers with the rows:
{ "rowCount": 3, "truncated": false, "result": "<file-data … origin=\"query_table\" …>{\"columns\":[\"region\",\"total\"],\"rows\":[…]}</file-data …>"}- At most 1,000 rows and 16,000 characters. Values are cut to 200 characters. When rows are cut,
truncatedistrueand a note tells the model to narrow the query. - A query runs for at most 10 seconds.
- On an error the model gets one fixed sentence for the kind of error (syntax, an unknown column, a value that can’t be converted, the time limit) followed by the loaded tables and their columns, so it can fix the SQL. The database’s own error text never reaches the model.
- A binary column is refused with “Binary columns can’t be shown; select
length(x)or cast to VARCHAR.”
Queries run in a sealed, read-only database that can read only the tables loaded from this file. It has
no file or network access and its settings are locked. Table functions (other than range and
generate_series), schema-qualified names, file paths or URLs in FROM, and functions that describe the
database itself are all refused with a fixed Rejected: … sentence. The table build is locked down the
same way: no extensions are installed or loaded.
Pictures in workbooks
Section titled “Pictures in workbooks”When an Excel workbook in a user’s library is indexed, its pictures are extracted, so the assistant can
look at a logo, a scanned form or a chart pasted as an image. describe_workbook lists each picture with
its cell and name, and either a fileId or the reason it wasn’t stored. read_file with that fileId
shows the picture to the model in the same way as any image, under the same vision rules and
limits. The result also names the workbook, the cell the picture sits on, and the sheet’s state when the
sheet is hidden.
- At most 20 pictures per workbook are extracted, in workbook order. Pictures on hidden sheets are included and labelled.
- PNG, JPEG, GIF and WebP pictures are stored, scaled down to at most 1,568 pixels on the longer side. A GIF keeps its first frame.
- A picture isn’t stored when its format is another one (EMF, WMF, SVG and so on), when it is too large
(over 16 MiB in the workbook, over the pixel limit, over 4.5 MiB
once scaled, or past 32 MiB of scaled pictures for the workbook), when it can’t be read, when it is a link to a file outside the workbook, when it is past the
twentieth, or when the user’s storage quota is full.
describe_workbookgives the reason in words. A picture that isn’t stored never stops the workbook from being indexed. - Pictures count toward the user’s storage quota, under Derived in the Files dialog.
- Pictures are internal: they are not in the file list, and they can’t be downloaded, renamed, deleted or re-indexed on their own. They are deleted with the workbook, and when it is overwritten, redacted or erased, or its anonymous session ends. A re-index of the workbook extracts them again.
read_fileshows a picture only to the owner of its workbook, and only while the workbook it came from is still the same file. After the workbook is deleted, overwritten or redacted, the picture’s id isnot_found.
What the table tools refuse
Section titled “What the table tools refuse”Every refusal is one fixed sentence. None of them repeats the model’s input, file text or an error.
| When | The model is told |
|---|---|
| The id isn’t one of the user’s own files: a knowledge-base file, another user’s or session’s file, a derived file, or an id that doesn’t exist | ”No file with this id is in your files.” The same words for every cause, so the answer reveals nothing about other files. |
The file isn’t a spreadsheet or CSV, an .xls included | ”describe_workbook works on spreadsheets and CSV files only.” (load_table and query_table say the same with their own name) |
| The file is still being indexed | ”This file is still being indexed; try again shortly.” |
| Indexing failed | ”This file could not be read, so it has no outline.” (load_table: “…so it cannot be loaded as a table.”) |
An .ods file, or a spreadsheet with no text stored | ”This file has no workbook outline, because of its type or because no text is stored for it; use read_file or file_search to read it.” load_table on an .ods file: “load_table works on .xlsx workbooks and CSV/TSV files only.” |
An .xlsx or CSV file with no outline yet: indexed before outlines existed, or changed since (for example redacted) | “This file has no workbook outline yet (…); re-indexing the file builds it.” See Re-index. A CSV can still be loaded without one. |
| The file’s stored content was removed, as a redaction does for an Excel file | ”The source of this file has been removed.” Its loaded tables are deleted too. |
| The file is over 256 MB | ”The file is too large to load as a table (limit 256 MB).” |
| The area is over 1,000,000 rows, 512 columns or 20,000,000 cells, or a CSV has more than 1,000,000 records | ”The range is over the table cap (1,000,000 rows, 512 columns).” |
The sheet name, A1 area or headerRows is wrong, or the area is outside the sheet’s used range | A sentence that says which, such as “The range is outside the sheet’s used range A1:F200.” |
query_table before any load_table | ”No table of this file is loaded; load it with load_table first.” |
| The file changed while its table was being built | ”The file changed while the table was being built; call load_table again.” |
The table tools work on the user’s own files only. For a spreadsheet in a knowledge
base, the assistant uses kb_search and read_kb_file: knowledge-base files
have no outline, no table tools and no extracted pictures in this release.
Limits on reading a file
Section titled “Limits on reading a file”- No length cap. The full text of a file is stored and indexed, however long it is.
- Word documents are read whole, so their size is bounded by
FILE_INDEXING_DOCX_MAX_BYTES(default 268,435,456 bytes, 256 MiB). A larger.docxisfailedwith reasonparser_limit, without being read. - PDF pages that take longer than
FILE_INDEXING_PDF_PAGE_TIMEOUT_MS(default 30,000 ms) are skipped with a retryablepage_unreadablewarning. - Concurrent indexing per library is
INDEXING_PER_OWNER_CONCURRENCY(default 2). - Every format has its own input size limit, and zip-based formats, mail and legacy Office files have more; see Parser limits.
Set these on every hub instance. A value that is empty, not a number, or not greater than zero falls back
to the default. The two file limits are read when a file is indexed, and the concurrency at startup.
Earlier releases’ EXTRACT_MAX_CHARS and FILE_INDEXING_MAX_BYTES are gone; setting them has no effect.
Status overview and bulk Re-index
Section titled “Status overview and bulk Re-index”Administrators see how files are doing across the whole deployment on System → Files in the admin portal. It shows counts only, never one user’s files: chat files and knowledge-base files, each by status, by reason, how many have warnings, and by the embedding model of their current index.
The same tab starts a bulk Re-index of the files that match a filter: scope (chat, knowledge base or
both), one or more states (any except processing), a reason, files with warnings only, or an embedding
model, and the kind, extension or MIME type (see Re-classify stored
files). Use it to retry the files a provider outage left
search_unavailable, or to move every file onto a new embedding model. To give spreadsheets indexed before
this release their outline and pictures,
re-index the chat files of kind table.
The model name none means two slightly different things. In the counts, it covers files with no index
and files whose index was built before DioscHub recorded the model. As a filter, it matches only files
with no index. To reach files indexed before the model was recorded, filter by state instead.
- Choose the filter and select Estimate. The portal shows how many files match, how much text they hold and about how many embedding tokens re-indexing them will use.
- Confirm. The job starts in the background.
A bulk Re-index runs under the same per-library limit as uploads, and users’ uploads are indexed ahead of it. It never stops at a user’s storage quota, but the bytes it stores still show in that user’s usage afterwards. A file whose re-index fails keeps its previous status and index. Only one bulk Re-index can be queued or running at a time, and there is no cancel in the portal.
Viewing the counts and the estimate needs settings:read; starting a bulk Re-index needs
settings:manage, and each start is recorded in the audit trail. Over
the admin API:
| Endpoint | Returns |
|---|---|
GET /api/admin/file-indexing/status-counts | chat and kb: total, byStatus, byReason, withWarnings, byEmbeddingModel |
GET /api/admin/file-indexing/reindex-estimate?scope=&status=&reason=&hasWarnings=&embeddingModel= | { fileCount, textBytes, approxTokens, filesWithoutText, sourceBytes } |
POST /api/admin/file-indexing/reindex with { filter, confirm: true, expectedFileCount } | 201 with { jobId, fileCount } |
The start is refused with 400 confirm_required without confirm: true, 400 invalid_filter for a
filter it doesn’t accept, 409 estimate_changed (with the new estimate) when more files match than you
confirmed, and 409 bulk_running while another bulk Re-index is queued or running.
Known limitations
Section titled “Known limitations”- If a data erasure or redaction
gives up waiting for a file’s indexing to stop, it fails with
index_job_still_running, and that file can be left inprocessingwith nothing to finish it. Run the erasure or redaction again. - After a redaction removes a binary file’s original bytes, downloading it (
GET /api/files/:id/content) fails with500instead of a clear error. The assistant still reads the redacted text. read_filemarks only pages, sheet row ranges and lines. A PowerPoint, OpenDocument presentation or EPUB file gets line markers, not slides or chapters, andat: { "page": N }on one is refused; read it by line or from a hit’sstartOffset. Search hits from these files havelocation: null, and still carrystartOffset.- A sheet whose name contains
"or a line break gets no sheet marker, and can’t be reached withat: { "sheet": … }.
Related: Frontend integration · File storage limits, quota & retention · Knowledge bases · Read and write files from a tool