This page covers where to look when something does not work and the failures people hit most often.
Start with the health check
The state of everything the app depends on is one REST call on the app’s API - GET /os_ai_agent/health, reachable through the Splunk Web proxy at /splunkd/__raw/servicesNS/nobody/os_ai_agent/os_ai_agent/health with your browser session, or against the management port with curl. It says whether the chat worker is enabled and when it last ran, whether KV Store is readable, the license state, whether a provider is configured and the default still exists, whether token authentication is on (with the alerts affected while it is off), and whether the activity trail is being written. overall is ok only when nothing is degraded.
Where the logs are
| Surface | Where |
|---|---|
| Chat and the setup page | index=_internal sourcetype=os_ai_agent:worker (the chat worker) and sourcetype=os_ai_agent:admin (the setup and chat handlers) |
| The activity trail | index=_internal sourcetype=os_ai_agent:audit - every turn, tool call, and refusal |
| The alert action | index=_internal source=*os_ai_agent_alert.log - every enqueue and every refusal |
| osai | The search’s own messages and its search.log in the Job Inspector; on a self-managed search head also osai.log in the search’s dispatch directory |
To raise the detail while reproducing an issue, set the log level to debug on the Setup page’s Settings tab, or target one component with levels.<component> in os_ai_agent.conf. Put it back afterwards.
Common issues
Chat is not available, or you get a capability error. Using the agent requires the use_os_ai_agent capability. Assign the user os_ai_agent_user (or os_ai_agent_admin). See Security.
Every page says the app is not configured. Splunk gates the app’s views until setup is done. Open the Setup page and save a provider; that clears the gate. Splunk checks the gate at sign-in, so reload Splunk Web or sign out and back in afterwards.
Test fails with a connection error or timeout. The search head cannot reach the provider’s endpoint. Check the base URL’s shape for the kind (an OpenAI-compatible URL ends in /v1; an Anthropic URL must not), the network path from the search head, and on Splunk Cloud the stack’s outbound port rules (see Splunk Cloud). A local model on a laptop is not reachable from a Cloud stack.
Test fails with 401 or 403. The key is wrong or for another endpoint, or, for Azure, the key belongs to a different resource. Re-enter it on the provider form (blank keeps the stored key; type to replace). For Bedrock, check that the model is enabled in the region.
Test succeeds but the model returns nothing useful, or errors on tools. The model does not support tool calling, or the local server is not exposing it. Pick a model that does; every hosted model above does.
| osai refuses to run right after an upgrade and says the REST handlers predate the command. splunkd’s handler processes outlive an app reload, and the command will not run under one that does not admit it as a search-command run. Restart splunkd, or wait for the handler processes to recycle.
A | osai search waits and then fails after fifteen minutes. Its run is a chat job, and no worker picked it up: the same cause as a queued message that never runs, below.
A message sits at “queued” and never runs. The chat worker is not running. Check the Health dashboard, then Settings - Data inputs for os_ai_agent_worker; enable it if it is disabled. The worker log (sourcetype=os_ai_agent:worker) shows why it stopped.
A conversation fails with a context-length error. The conversation outgrew the model’s window. Set the provider’s Context window so long conversations are trimmed before each call; press Load models to have it detected where the provider reports it. Leave tool-result compaction on.
A local model is slow and the call times out. Raise the provider’s Request timeout - it includes the quiet period before the first token, which on a local model is prompt processing. The Stream idle timeout is a separate bound on silence mid-answer and should stay short.
A second provider or a second user is refused. The free tier allows one of each. See Licensing.
An alert investigation never appears. Read index=_internal source=*os_ai_agent_alert.log for the refusal. The usual causes: the alert is app-shared and names no Run as user, the run-as user lacks use_os_ai_agent or (for an app-shared alert) run_os_ai_agent_alerts, or token authentication is off on the search head. See Alerts and schedules.
A run was refused for a budget or the concurrency cap. The Activity Trail dashboard’s refusals panel says which limit, and the Token Usage dashboard shows the user’s last 24 hours against it. Raise the limit on the Settings tab or wait for the window to advance.
The dashboards are empty. They read KV Store collections and _internal data that only administrators can search. Open them as an administrator.
Something changed after an upgrade but the app behaves as before. Splunk keeps the app’s REST handler processes alive across an app reload, and a browser tab keeps its old bundle. Restart Splunk after upgrading, and reload any tab you had open.
Verify the chain end to end
The Test button on the Providers tab runs a real completion through the whole chain - credential, endpoint, model - and reports the latency. From the search bar:
| osai prompt="Reply with the single word OK"If that returns an assistant event, the provider works and any Chat problem is about the worker, permissions, or the tools, not the model.