The osai command

| osai calls the agent from SPL. It is a generating command - it goes first, after the leading | - and it has three modes: a free-form prompt, an explanation of a search, and a full read-only investigation. Because it is ordinary SPL, it can be saved, scheduled, and delivered by any alert action, which is how you get a recurring agent report with no extra infrastructure.

Running it requires the use_os_ai_agent capability. See Security.

| osai (prompt=<string> | explain=<spl> | task=<instruction>)
       [provider=<name>] [model=<id>] [system=<string>]
       [temperature=<float>] [max_tokens=<int>] [run_as=<user>]

Quote values that contain commas or spaces - prompt="...", system="...". SPL splits unquoted arguments on both, and rather than answering a truncated prompt the command rejects a stray fragment with an error that names it and shows the quoted form. Unrecognized parameters are rejected the same way. The command refuses inline credential parameters (api_key=, token=, and similar): credentials come from the configured provider only. Never type a secret into a search. Splunk records the search text in search history, the audit index, and the dispatch directory before any command sees it, so a secret the command refused has still been written down - rotate it.

Output

One event per provider call, not one per streaming token. Fields:

Drop the reasoning row from a thinking model and keep just the answer:

| osai prompt="..." | where role="assistant"

prompt= - a completion

A prompt= run is a plain completion: no tools are offered to the model, and the system message is exactly your system=, or none.

| osai prompt="Summarize the three most common causes of a 503 from an API gateway, in one paragraph"

| osai prompt="Classify this stack trace" provider=local-lmstudio model=qwen/qwen3.6-35b-a3b

| osai prompt="Generate five search tips for a new analyst" provider=openai-prod temperature=0.4 max_tokens=300

explain= - explain a search

The agent verifies that the search parses, breaks the pipeline down stage by stage - consulting the embedded SPL reference for any command it is not certain about rather than guessing - optionally grounds the explanation against a small, tightly bounded sample of real events, and returns a markdown explanation with the fields each stage creates, pitfalls, and performance notes. Only read-only tools run during an explain.

| osai explain="index=payments sourcetype=api | stats count by status | sort -count"

task= - an unattended investigation

The agent runs a full investigation with its read-only tools - search, discovery, verification - and emits a self-contained markdown report. It cannot create or modify anything, and it never pauses for an approval nobody is there to give. Designed for scheduled saved searches: put it on a cron and attach an email or webhook action to deliver the report.

| osai task="Check every index for ingestion gaps in the last 24 hours and report anything unusual"

| osai task="Review the alerts that fired in the last week: which fire most, which never trigger, and which have no suppression"

Every run is a chat job

The command does not call the model from inside the search process. It creates a conversation in Chat - owned by the user running the search, titled after the request - submits the run as a job, and waits for the app’s background worker to finish it, the same way the chat page does. Three things follow.

• The run is gated and metered like a chat turn. The [limits] budgets and max_active_jobs_per_user apply to it, it is written to the usage ledger, and a refused search returns no rows and names the budget in the job inspector. Its turn event in the activity trail carries tokens and cost:

  Copy

  index=_internal sourcetype=os_ai_agent:audit event=turn surface=spl
  | stats sum(tokens_in) sum(tokens_out) sum(cost_usd) by user

• The full answer stays readable in Chat, which also means a daily scheduled report leaves a conversation per run in that user’s list; [retention] threads_days ages them out, or the user deletes them.

• The search holds a search slot for as long as the run takes. The worker picks a job up within seconds, so the added latency is small, but a run nobody picks up within fifteen minutes is cancelled and the search says what the health check knows about the worker. The command needs the worker modular input running, as chat does.

A run acts as a person. A scheduled search owned by nobody - one shared at app level - runs as splunk-system-user, so give it task="..." run_as=<user> and the run acts as that user, on a token minted for them, the way an app-shared alert action does. As for such an alert, the named user must hold run_os_ai_agent_alerts - the os_ai_agent_alerts role ships for exactly this - the search head needs splunkd token authentication enabled, and run_as= is accepted with task= only. prompt= and explain= run as the search’s own user.

Right after upgrading the app without restarting splunkd, a | osai search may refuse to run and say that the REST handlers predate the command: splunkd’s handler processes outlive an app reload, and the command will not run under a handler that does not admit it as a search-command run. Restart splunkd, or wait for the handler processes to recycle.

Why the change: a search command runs under the calling user’s identity, so running the model call inside it meant the provider API key had to be in that process. See Security for what that cost and how this closes it. See Alerts and schedules for the alert-triggered counterpart and Costs and limits for the accounting.