Security

Two questions govern what the agent can do:

1. Can this Splunk user use the agent at all? Answered by Splunk roles and capabilities.
2. What can the agent do once it runs? Answered by the calling user’s own permissions and the approval model.

Get both right and OS AI Agent is a faster way to do what a user could already do, with a record of every step.

Splunk roles and capabilities

The app defines three capabilities:

And three roles that bundle them:

These are the names Splunk shows under Settings - Roles; in authorize.conf the stanzas carry a role_ prefix, as in [role_os_ai_agent_user]. A full Splunk administrator (admin_all_objects, or sc_admin on Splunk Cloud) passes the first two checks without the app roles. The third is never implicit: no built-in role holds it, and only an administrator can grant it.

None of the app roles inherits from a Splunk role, and os_ai_agent_admin needs none of Splunk’s administrative capabilities to run the Setup page, the three administrator-only settings above excepted: the app checks edit_os_ai_agent_providers and then saves with its own permissions, API keys included.

Neither role holds list_storage_passwords, and the app’s credential store is restricted to administrator roles. Chat, | osai, alert investigations and the Setup page’s Test button have a provider’s key read for them in a privileged context and handed straight to the agent, so no process a user controls ever holds a key: use_os_ai_agent lets a user spend a key without being able to take one. If you granted list_storage_passwords to a role of your own in order to use the agent, remove it - it is no longer needed, and it grants clear-text read of every credential the app holds. The capabilities are also independent, so a role you create that holds only edit_os_ai_agent_providers can configure the app without being able to use the agent. Test and Load models send requests to the provider, so those two buttons need use_os_ai_agent as well.

Your permissions are the boundary

Every tool the agent runs executes against splunkd with the calling user’s own session - never a system account. The agent can read only the indexes and objects your Splunk roles allow, and scheduling a search or alert requires your own schedule_search capability. An unattended run acts as a named user, on a short-lived token minted for the job and revoked when it settles.

Two more rules narrow what the agent may do even within your permissions:

• Writes need your approval. Every authoring tool pauses on an approval card that shows exactly what will be written; you can edit the parameters before approving. Updates are limited to your own private objects - the agent refuses to modify shared or other users’ dashboards and saved searches. Permission to approve is checked when you approve, not only when the run started. Approving somebody else’s paused run does not lend it your access: the run continues with its owner’s credential, and is refused if that credential has expired.
• Searches the agent runs are read-only. The search tool rejects SPL that writes or leaves the search sandbox - delete, collect, outputlookup, sendemail, script, rest, and others - and refuses real-time ranges. The check reads command names, so it cannot see inside a macro, nor into the saved search a savedsearch: dataset runs; rather than run either unchecked, the agent refuses the search and asks for the expanded SPL or the explicit command. A from=... search term and an SPL comment are unaffected, and so are searches you run yourself. Your Splunk permissions remain the authoritative boundary.

Conversations are isolated by ownership: only a conversation’s owner or an administrator can read it or watch its live progress, and conversation and job identifiers are minted by the server, never chosen by a client, so nobody can guess or squat on another user’s conversation.

Data that talks back

Everything the model reads from your data - search results, knowledge objects, error messages that echo indexed text - is wrapped in an explicit untrusted-data boundary with instructions not to follow anything found inside it. That is the defense, and it does not depend on noticing an attack.

The app additionally scans what the agent reads for content aimed at it: text telling it to ignore its instructions, chat-transcript markers trying to close the boundary, requests for its prompt or credentials, directions to send data somewhere, and invisible characters positioned to hide text from a human reading the same event. Findings appear on the tool card in Chat, on the audit event, and in the app’s log. A finding never blocks the result - a security analyst investigating injection attempts in their own logs is reading attacker text on purpose - except invisible characters, which are stripped before the model reads them because their whole purpose is to make the model’s view differ from yours. Searches that read credential- or PII-shaped fields (password, api_key, ssn, credit_card, and a few more) are noted the same way, with an instruction to the model to summarize rather than quote.

What a model writes cannot become script in your browser either. Chat, the HTML export and the print view render model output through one sanitizer: headings, tables, lists, code blocks and links render as written; images, forms, embedded media and inline styles do not; a link may point only at http(s), mailto, or a path within this Splunk instance, and opens in a new tab with no handle back on the Splunk page. The “Open dashboard” link on a tool card is offered only when it is a real dashboard path. This matters most for text the agent did not write itself: search results it read back, and imported conversations.

Secrets and storage

• Provider API keys live in Splunk’s encrypted credential store (storage/passwords), restricted to administrator roles; the setup API never echoes them back, and | osai refuses inline credential parameters. A key is identified by its provider’s name, never by a reference the caller chooses, and is only ever sent to that provider: changing a provider’s base URL means entering the key again, and a redirect is never followed, so the address a provider redirects to receives nothing.
• Conversations, messages, jobs, feedback, and usage records live in app-scoped KV Store collections readable only by administrators and the system context. Users reach their own data through the app’s REST handlers, which authenticate the caller and scope every read and write to that user.
• Per-job credentials - a captured session or the principal an unattended job runs as - live in a separate collection only Splunk administrators and the system context can write, and are cleared when the job completes.
• The archive signing key is created on first export and kept where only administrators can read it, so a user cannot re-sign a conversation they edited.
• Nothing phones home. The only outbound traffic is to the provider endpoints you configured.

The activity trail

The app writes one event per agent turn, tool call, and refusal to $SPLUNK_HOME/var/log/splunk/os_ai_agent_audit.log; Splunk’s own default monitor indexes it into _internal as sourcetype=os_ai_agent:audit, with no index to provision. Each event carries the user, the surface (chat, | osai, MCP), the conversation and job, the provider and model, the decision behind each tool call, the object a change touched, the approver when a person approved on someone else’s behalf, injection flags, and the alert that triggered an unattended run.

This records what nothing else can. Splunk’s own audit index already logs that a dashboard was written; it cannot know which conversation asked for it, which tool call made it, whether a person approved it or a policy did, or what the agent was refused - a tool blocked by a conversation’s allowlist, a person declining an approval, a request turned away by the license, a budget, or the concurrency cap. Plain Splunk authorization failures are not duplicated here; Splunk’s audit index has them.

index=_internal sourcetype=os_ai_agent:audit event=tool mode=mutation ok=true
| table _time user approved_by tool target

The Activity Trail dashboard reads the same data; see Dashboards. Writing is best-effort and never fails a request; [audit] disabled = true turns it off, and the Settings tab has the switch.