The agent also works with nobody watching, in two ways. Both are strictly read-only - the authoring tools are not even offered to the model - and neither can pause for an approval nobody is there to give.
Scheduled reports with | osai task=
Put | osai task="<instruction>" in a saved search, schedule it, and attach an email or webhook action. Every run is a full agent investigation - search, discovery, verification - that emits a self-contained markdown report as its result. “Check every index for ingestion gaps in the last 24 hours” on a daily cron, delivered by email, is a recurring agent report with no extra infrastructure. The syntax is in The osai command.
A scheduled search runs as its owner, with that user’s Splunk permissions and that user’s schedule_search capability. A search shared at app level has no owner - it runs as splunk-system-user - so give it run_as=<user>: the run acts as that user, on a token minted for them, and the conversation lands in their Chat, the way an app-shared alert action’s does. As for such an alert, that user must hold run_os_ai_agent_alerts, the search head needs splunkd token authentication enabled, and run_as= is accepted with task= only. Every run is a chat job, gated and metered like a chat turn; see The osai command.
The “Ask OS AI Agent” alert action
Attach the Ask OS AI Agent action to any Splunk alert. When the alert fires, the agent investigates it - armed with the alert’s name, a link to its results, and a bounded sample of the triggering rows - and the findings appear as a new conversation in Chat, with the full tool activity recorded. The action itself only enqueues the job, in well under a second; the app’s background worker does the work, so slow model calls never run inside Splunk’s alert-action time budget. An alert that triggers for each result gets one investigation per result, each carrying only the row that fired it.
The action’s parameters, in the alert editor:
| Parameter | What it does |
|---|---|
| Investigation prompt | What the agent should figure out. The alert’s name, owner, results link, and result sample are appended automatically. Default: determine what happened, whether it looks expected or anomalous, and what to do next. |
| Run as | The Splunk user the investigation runs as, and whose Chat receives the findings. See below. |
| Provider, Model | Overrides for this alert. Default: the app’s default provider and its default model. |
| Include triggering results, Max sampled results | Attach a sample of the alert’s result rows (1 to 50, default 20, capped at 64 KiB) so the agent reasons over real data. The rows are read by the worker, as the run-as user, so the sample is exactly what that user may see. |
Who the investigation runs as
An investigation runs as a named user, with that user’s own Splunk permissions, and its findings land in that user’s Chat:
- A user-owned alert runs as its owner. Leave Run as blank.
- An app-shared alert - one shared at app level, with owner
nobody, which is how every alert an app ships appears (all of Monitoring Kubernetes’, for instance) - fires assplunk-system-userand has no person behind it. It needs a Run as user, and that user must hold therun_os_ai_agent_alertscapability, which the shippedos_ai_agent_alertsrole carries and which only a Splunk administrator can grant. Without one, the action refuses the alert with a clear line in its log.
The reason for that capability is worth knowing: a shared alert fires as the system user, whose session would authorize any “Run as”, so whoever can edit the alert could otherwise have the agent act as anyone. Holding the capability is an administrator’s consent to be that principal. It is granted to no built-in role on purpose, and it says what may be done in its holder’s name, not what the holder may do.
Naming a user other than the alert’s owner on a user-owned alert takes edit_tokens_all, Splunk’s own authority to mint tokens for other users, which an ordinary user does not hold - so a private alert always runs as its owner.
Token authentication
The session key Splunk hands an alert action stops working seconds after the action exits, so the app’s worker mints a short-lived splunkd authentication token for the run-as user when the job runs, and revokes it when the job settles. This requires token authentication to be enabled on the search head (Settings - Tokens). It is on by default on Splunk Cloud and off by default on Splunk Enterprise. The app’s health endpoint reports the scheme’s state and names the alerts affected while it is off, and a job that cannot mint a token fails with that reason in its conversation instead of running.
Throttle alerts that fire often
A user is limited to a number of active chat jobs at once (10 by default; see Configuration), so an alert that can fire rapidly should set throttling in the alert editor, or investigations will be dropped rather than queued without bound.
Where to look
- The findings: a new conversation in the run-as user’s Chat, titled after the alert.
- Every enqueue and every refusal: one line in the action’s log,
index=_internal source=*os_ai_agent_alert.log. - The run itself, with every tool it called: the Activity Trail dashboard’s Unattended tab, or
index=_internal sourcetype=os_ai_agent:audit task_mode=true. - Spend attributed to the alert that triggered it: the Token Usage dashboard’s Unattended tab.