Dashboards

Four views for whoever runs the deployment. They read the app’s usage ledger, its activity trail in _internal, its operational logs, and its KV Store collections, all of which are readable by administrators - other roles see empty panels. Token Usage, Activity Trail and Health are tabbed, and a tab runs its searches when you open it. Token Usage has a time-range picker and a user filter, Activity Trail adds a surface filter (chat, | osai, MCP), Health has the time-range picker alone, and the Conversation inspector takes a conversation id and reads that conversation’s whole history. A row on Token Usage or Activity Trail, or a feedback comment on Health, opens the Conversation inspector for that conversation.

Token Usage

Who is spending, and on what, from the usage ledger. Tokens and cost per user and per day, tokens by hour of day and weekday, the most expensive conversations, prompt-cache hit rate and the dollars it saved at the rates frozen on each record, the reasoning share of output per model, how full each model’s context window got, requests served by more than one provider, tool-call depth against tokens, and each user’s last 24 hours against the configured spend limits next to the deployment-wide budget. An Unattended tab attributes spend to the alert or saved search that triggered it, and a panel lists the | osai runs as the activity trail saw them, beside the ledger’s own record of them. See Costs and limits.

Reading it:

• Requests counts one chat run once, however many worker passes or providers it took. Tokens counts every token under its provider’s own cache convention - the number the daily token limits count.
• Recorded cost sums only requests a priced provider served. not priced means no provider in the window has rates configured, which is not $0; no data means the window holds no usage. Unpriced requests is the share no recorded cost covers - set the provider’s Input price and Output price (price_input_per_mtok, price_output_per_mtok) to price them.
• Cache hit rate is cached prompt tokens as a share of all prompt tokens. OpenAI-compatible and Azure OpenAI providers count cache reads inside the input total; Anthropic and Bedrock report them beside it, which is why Tokens can exceed input plus output.
• Last 24 hours against the daily caps and Whole deployment read the rolling 24 hours the [limits] budgets enforce, whatever the time range says. A percentage appears only where a cap is set; it turns yellow at 50%, orange at 80% and red at 95%. The deployment-wide daily_cost_usd refuses every new request once all users’ recorded spend passes it.
• Served by more than one provider counts requests with ledger rows from two or more providers: the first answered part of the run, then failed. A target that failed before serving a token leaves no row, so the failover count is on Health. Requests over several passes are runs the worker yielded and resumed.
• Cost by provider and model: cost/1k out is the blended price of an answer once tool loops and cached context are counted. When priced reqs is below reqs, rates covered only part of the window. Rates in force during the window shows more than one rate for a model when its price changed: each ledger row keeps the rate it was priced at, so repricing never rewrites past spend.
• Requests by how full the context got is the largest prompt a request sent against its model’s context window. A p95 near 100% means long runs are being trimmed to fit.
• Unattended runs by trigger names the alert or saved search splunkd says started each run (alert:<name>, search:<name>). alert (unverified) is an alert an ordinary user owns, whose name is not recorded; alert (unresolved) and search (unresolved) mean splunkd could not say; search (ad hoc) is a | osai task= search typed rather than saved; rest (direct) is an unattended job created over REST.

Activity Trail

Who runs which tools, and what changed, from the activity trail. Turns by outcome and time-to-answer per model, tool calls per user by tool, tool failure rates with tail latency and the error each failed call returned, every change per object with its approver (including approvals made on another user’s behalf), refusals classified as declined by a person, blocked by policy, budget, concurrency cap, or license, prompt-injection flags by pattern and by tool, and the searches the agent ran. See Security.

Reading it:

• A turn is one pass of the agent on a request: a chat message, a | osai run, an alert investigation. A request that runs past a worker pass, or pauses for an approval, records a turn each time it stops and another when it resumes, so turns can outnumber the answers people got; done turns count those. Outcomes: done answered, error failed, canceled was stopped by a person, awaiting_approval paused for consent, yielded ran out of a worker pass and resumed later.
• Changes made counts successful creates and updates of dashboards and saved searches. Splunk’s own _audit index records that an object was written; this records which conversation asked and who approved.
• How a tool came to run: auto - the conversation’s policy let a read-only tool run; approved and denied - a person decided; blocked - policy refused with nobody involved. A refusal that never reached a tool (a budget, the concurrency cap, the license) shows as (none).
• Tool usage: fail % is green under 1%, yellow from 1% and red from 10%; p95 is the slow tail a user waits on. A rising splunk_search p95 usually means wider searches, not a slower splunkd.
• Approvals on someone else’s behalf: an administrator approved a tool call in another user’s paused run. The run keeps its owner’s credential - approving lends it no access - so user is whose access the tools ran with, and approved_by is who released them.
• Runs that read something suspicious and then changed an object is the row a security reviewer asks for first; empty is the expected state. Flags (instruction_override, role_marker, prompt_disclosure, exfiltration, hidden_characters) are what the detector saw in a tool result before the model did - never a block.
• Searches that named sensitive fields lists queries naming password, secret, key, card or SSN-like fields. They are noted, never refused.
• Failed unattended runs is where runs nobody was watching are noticed. An error naming a credential means the run-as user could not be given a token: check that token authentication is on (Settings - Tokens). A run-as user missing a capability never gets this far - the alert action refuses the alert and says why in its log.
• MCP calls carry no conversation, so their rows open an empty inspector.

Health

For whoever operates the search head. Jobs queued, running and awaiting approval now, queue-wait and run-time percentiles, provider retries, failovers, and circuit-breaker events, the worker’s housekeeping, the newest warnings from the worker, the REST handlers and the alert action, new conversations and active users per day, the Helpful and Not helpful votes from Chat (counted as thumbs up and down) with the comments people wrote, the approval modes and tool allowlists in use, and every KV Store collection’s size next to the retention settings that bound it.

Reading it:

• The time range bounds the logs and the trail, and the KV lookups by their own timestamps: jobs that finished in the range plus every unfinished job, conversations started in the range plus every live one, feedback left in the range. The now tiles, the collection sizes, the retention settings and the template count read the current state.
• Queue wait and Run time cover jobs that finished in one pass without pausing for approval: a resume restarts the clock, and a canceled job’s clock includes whatever it waited on. On a search head cluster every member runs a worker; a rising wait with idle providers means the worker is disabled or busy elsewhere.
• Provider retries, failovers and breaker events: retry - a transient failure retried on the same provider; failover - the chain moved to the next target; breaker opened - a target was taken out of rotation after repeated failures; breaker skip - a target skipped while its breaker was open; served by fallback - a fallback answered. Empty is the healthy state. Turns by serving provider and model stamps each turn with the provider that actually served it.
• Worker housekeeping: retention sweeps, stale jobs and orphaned claims reaped, the tokens minted and revoked for unattended runs, orphaned secrets and tokens cleaned up, and revokes or cleanups that failed - a growing count there means tokens linger until the orphan sweep retries. A growing reaped count means jobs are dying mid-run.
• Latest warnings and errors lists what needs an administrator: an ERROR for a job that failed outside a run (it ran out of continuation passes, or its credential could not be used), a WARN for splunkd refusing or not answering the worker. The agent’s own search mistakes, which it corrects itself, are not here; Activity Trail counts them as failed tool calls.
• KV Store collections: Splunk ages indexes, KV Store does not; the retention sweep, at most once an hour from the worker, is what bounds them. Jobs hold a request’s whole message state and are the largest rows.

The dependency checks are not on this dashboard; they are one REST call - GET /os_ai_agent/health reports whether the worker is enabled and when it last ran, whether KV Store is readable, the license state, whether any provider is configured, whether token authentication is on, and whether the trail is being written, with an overall field a saved search or an external monitor can watch.

Conversation

Not in the navigation: clicking a row on Token Usage or Activity Trail, or a feedback comment on Health, opens it for that conversation, or paste a conversation id. It lists every turn, tool call, and refusal behind the conversation in order, its usage rows, and its feedback, with a link to open it in Chat. The inspector reads the app’s stored records, which the admin, sc_admin and os_ai_agent_admin roles can read; Chat opens a conversation only for its owner; anyone else following the link lands in their own Chat.

Reading it:

• The trail is read over all time for that one conversation, back as far as Splunk keeps _internal (30 days by default). When retention is on and [retention] trash_days is above 0 (7 by default), a conversation deleted longer ago than that has no Thread row; its trail remains as long as _internal keeps it, and its usage rows as long as [retention] usage_days does.
• In the event list a tool row is one call, and a turn row closes the answer it belongs to, carrying that answer’s tokens and cost; the answer’s tool calls are the rows above it.
• Tokens is input plus output as the trail records them per turn; the usage rows carry the cache-aware total, so the two can differ. Input grows turn by turn because the history is re-sent, until context windowing trims it.