Troubleshooting

This page covers where to look when something does not work and the failures people hit most often.

Start with the health check

The state of everything the app depends on is one REST call on the app’s API - GET /os_ai_agent/health, reachable through the Splunk Web proxy at /splunkd/__raw/servicesNS/nobody/os_ai_agent/os_ai_agent/health with your browser session, or against the management port with curl. It says whether the chat worker is enabled and when it last ran, whether KV Store is readable, the license state, whether a provider is configured and the default still exists, whether token authentication is on (with the alerts affected while it is off), and whether the activity trail is being written. overall is ok only when nothing is degraded.

Bulletin messages

The app posts Splunk bulletin messages - the bell in the Splunk bar - for problems that would otherwise surface only when a chat or a search fails, and removes each one when its problem ends. They are shown to roles holding edit_os_ai_agent_providers or admin_all_objects, and to roles that import one of those; never to chat users.

The license and provider messages link to the Setup tab that fixes them - License (/app/os_ai_agent/setup#license) for the key, Providers (setup#providers) for the limits and a refused key; the worker message links the Health dashboard, or the input under Settings - Data inputs. The check behind them is the scripted input os_ai_agent_health.py, which runs every 5 minutes on each search head and logs one line per run to splunkd.log (index=_internal "os_ai_agent health"). It calls no provider: a refused key is reported from what the worker saw, so after you update the key the message clears with the next request the provider accepts.

Each message is posted once. Dismissing it hides it until the problem ends and comes back, or until splunkd restarts; a restart clears every message, and the next check posts what still holds. To stop them, disable the input under Settings - Data inputs - Scripts; the ones already posted last until splunkd restarts.

Where the logs are

To raise the detail while reproducing an issue, set the log level to debug on the Setup page’s Settings tab, or target one component with levels.<component> in os_ai_agent.conf. Put it back afterwards.

An error names a request ID

When a request to the app fails in a way it did not plan for, the answer is a plain sentence and a request ID, never the failure’s own text - which could carry a provider’s response, a path on the search head, a configuration value or part of a key. The chat and Setup pages show the ID with the error, and | osai adds it to the search’s error message. The cause, with its full traceback, goes to splunkd.log, every line tagged request_id=<ID>; the ID also tags the request’s lines in os_ai_agent_admin.log and its Activity Trail events. So one search finds everything the request left behind:

index=_internal "<request ID>"

Every REST answer carries the ID in its X-Request-Id header. A client that sends its own X-Request-Id - up to 128 letters, digits and . _ : -, from a proxy or a script correlating its own logs - gets that ID used instead of a new one. When splunkd fails a call the app makes on your behalf, the answer names splunkd’s HTTP status rather than repeating splunkd’s response.

Common issues

Chat is not available, or you get a capability error. Using the agent requires the use_os_ai_agent capability. Assign the user os_ai_agent_user (or os_ai_agent_admin). See Security.

Every page says the app is not configured. Splunk gates the app’s views until setup is done. Open the Setup page and save a provider; that clears the gate. Splunk checks the gate at sign-in, so reload Splunk Web or sign out and back in afterwards.

Test fails with a connection error or timeout. The search head cannot reach the provider’s endpoint. Check the base URL’s shape for the kind (an OpenAI-compatible URL ends in /v1; an Anthropic URL must not), the network path from the search head, and on Splunk Cloud the stack’s outbound port rules (see Splunk Cloud). A local model on a laptop is not reachable from a Cloud stack.

A model on your network stops answering once Splunk has a proxy configured. Provider requests follow [proxyConfig] in server.conf, and only localhost and loopback addresses go direct. Add the model’s host, domain or address range to no_proxy, or set use_splunkd_proxy = false; see Providers. A proxy setting the agent cannot use, such as a SOCKS proxy, fails the request with a message naming the setting.

Test fails with a certificate error. The app’s TLS client refuses a certificate signed with SHA-1 or carrying an RSA key under 2048 bits. Replace the certificate; Insecure TLS on the provider turns verification off, for an evaluation only.

Test fails with 401 or 403, or the bell says a provider refused its key. The key is wrong or for another endpoint, or, for Azure, the key belongs to a different resource. Re-enter it on the provider form (blank keeps the stored key; type to replace). For Bedrock, check that the model is enabled in the region.

Test succeeds but the model returns nothing useful, or errors on tools. The model does not support tool calling, or the local server is not exposing it. Pick a model that does; every hosted model above does.

| osai refuses to run right after an upgrade and says the REST handlers predate the command. splunkd’s handler processes outlive an app reload, and the command will not run under one that does not admit it as a search-command run. Restart splunkd, or wait for the handler processes to recycle.

A | osai search waits and then fails after fifteen minutes. Its run is a chat job, and no worker picked it up: the same cause as a queued message that never runs, below.

A message sits at “Queued” and never runs. The chat worker is not running, which the bell also reports after 15 minutes. Check the Health dashboard, then Settings - Data inputs for os_ai_agent_worker; enable it if it is disabled. The worker log (sourcetype=os_ai_agent:worker) shows why it stopped. A message that waits only while other conversations run is queued behind them: the worker runs four at a time per search head by default (worker_concurrency; see Configuration).

A conversation fails with a context-length error. The conversation outgrew the model’s window. Set the provider’s Context window so long conversations are trimmed before each call; press Load models to have it detected where the provider reports it. Leave tool-result compaction on.

A local model is slow and the call times out. Raise the provider’s Request timeout - it includes the quiet period before the first token, which on a local model is prompt processing. The Stream idle timeout is a separate bound on silence mid-answer and should stay short.

A second provider or a second user is refused. The free tier allows one of each. See Licensing.

An alert investigation never appears. Read index=_internal source=*os_ai_agent_alert.log for the refusal. The usual causes: the alert is app-shared and names no Run as user, the run-as user lacks use_os_ai_agent or (for an app-shared alert) run_os_ai_agent_alerts, token authentication is off on the search head, or the alert sets only one of Decisions and Decision index. See Alerts and schedules.

A decision event never appears. Open the run’s conversation and read its Decision recorded note: it says whether the event was written and, if not, why. The usual causes: the user the run acts as holds neither run_collect nor edit_tcp, or the run did not conclude - one that fails or is stopped records nothing, by design. A failed write is also in the worker log (sourcetype=os_ai_agent:worker). If the note says the event was written and it is still not there, check that the index exists wherever your search head’s own data is indexed. See Decisions.

The Decision recorded note says the write could not be confirmed. The app could not tell whether its | collect search wrote the event - it timed out while Splunk kept running the search, for example - and deliberately did not write again, because a duplicate would fire your alert twice. Search the decision index for an os_ai_agent:decision event with the job_id the note names before you act on the verdict.

A decision event exists, but its run shows as failed. The event is written when the run concludes, just before its conversation is saved. If saving the conversation then fails, the run ends as an error naming “persisting thread messages” although its decision was recorded, and the Activity Trail records the turn as failed. The investigation behind the verdict did complete.

A run was refused for a budget or the concurrency cap. The Activity Trail dashboard’s refusals panel says which limit, and the Token Usage dashboard shows the user’s last 24 hours against it. Raise the limit on the Settings tab or wait for the window to advance.

The dashboards are empty. They read KV Store collections and _internal data that only administrators can search. Open them as an administrator.

A page says it hit an error, with a Reload page button. The chat or the Setup page failed while drawing itself, and says what failed instead of going blank. A chat message, tool call or approval card that cannot be displayed is replaced on its own, and the rest of the conversation stays usable; a Setup tab that fails leaves the other tabs working. Include the message when you report it.

Conversations appear in _internal on Splunk 10.6. Splunk 10.6’s KV Store logs document values into _internal at its default log level. See Security for the override that stops it.

Something changed after an upgrade but the app behaves as before. Splunk keeps the app’s REST handler processes alive across an app reload, and a browser tab keeps its old bundle. Restart Splunk after upgrading, and reload any tab you had open.

Verify the chain end to end

The Test button on the Providers tab runs a real completion through the whole chain - credential, endpoint, model - and reports the latency. From the search bar:

| osai prompt="Reply with the single word OK"

If that returns an assistant event, the provider works and any Chat problem is about the worker, permissions, or the tools, not the model.