Security

Two questions govern what the agent can do:

1. Can this Splunk user use the agent at all? Answered by Splunk roles and capabilities.
2. What can the agent do once it runs? Answered by the calling user’s own permissions and the approval model.

Get both right and OS AI Agent is a faster way to do what a user could already do, with a record of every step.

Splunk roles and capabilities

The app defines three capabilities:

And three roles that bundle them:

These are the names Splunk shows under Settings - Roles; in authorize.conf the stanzas carry a role_ prefix, as in [role_os_ai_agent_user]. A full Splunk administrator (admin_all_objects, or sc_admin on Splunk Cloud) passes the first two checks without the app roles. The third is never implicit: no built-in role holds it, and only an administrator can grant it.

None of the app roles inherits from a Splunk role, and os_ai_agent_admin needs none of Splunk’s administrative capabilities to run the Setup page, the four administrator-only settings above excepted: the app checks edit_os_ai_agent_providers and then saves with its own permissions, API keys included.

Neither role holds list_storage_passwords, and the app’s credential store is restricted to administrator roles. Chat, | osai, alert investigations and the Setup page’s Test button have a provider’s key read for them in a privileged context and handed straight to the agent, so no process a user controls ever holds a key: use_os_ai_agent lets a user spend a key without being able to take one. If you granted list_storage_passwords to a role of your own in order to use the agent, remove it - it is no longer needed, and it grants clear-text read of every credential the app holds. The capabilities are also independent, so a role you create that holds only edit_os_ai_agent_providers can configure the app without being able to use the agent. Test and Load models send requests to the provider, so those two buttons need use_os_ai_agent as well.

The app’s administrator role reads the app’s KV Store collections, which the dashboards need - all but the job credentials below - but writes only one of them, the query-template library. Conversations and their jobs, the usage records the spend limits are summed from, the per-user run slots, the worker’s job claims and retention markers, and feedback are written only by the app, in its own privileged context, and by Splunk’s own administrators (admin, sc_admin). An app administrator therefore cannot delete usage records to get back under a budget, free another user’s run slots, or rewrite another user’s queued request or conversation.

Your permissions are the boundary

Every tool the agent runs executes against splunkd with the calling user’s own session - never a system account. The agent can read only the indexes and objects your Splunk roles allow, and scheduling a search or alert requires your own schedule_search capability. An unattended run acts as a named user: an alert investigation, and a scheduled | osai task= that names run_as=, run on a short-lived token minted for the job and revoked when it settles; any other | osai run acts on the session of the user who ran the search.

Two more rules narrow what the agent may do even within your permissions:

• Writes need your approval. In Chat every authoring tool pauses on an approval card that shows exactly what will be written; you can edit the parameters before approving. The MCP server has no approval card: its authoring tools stay off unless OS_AI_AGENT_MCP_ALLOW_MUTATIONS=1 is set, and then run without one (The MCP server). Updates are limited to your own private objects - the agent refuses to modify shared or other users’ dashboards and saved searches. Permission to approve is checked when you approve, not only when the run started. Approving somebody else’s paused run does not lend it your access: the run continues with its owner’s credential, and is refused if that credential has expired.
• The agent searches on its own only with commands that read. Before splunk_search or profile_data runs a search without asking, it asks splunkd’s own search parser what the search runs - your macros, saved searches and data model searches expanded, every subsearch included - and checks each command against Splunk’s built-in commands that only read, plus any add-on commands a Splunk administrator allowed on the Settings tab (see Configuration). It then runs the expansion it checked rather than the macro or saved-search reference, so a definition edited in the meantime cannot change what runs. Anything else is not run on the agent’s authority: a command that writes (collect, and its other names stash, summaryindex and sumindex; outputlookup; delete), sends (sendemail, sendalert), runs a script (script, also spelled run) or calls the REST API (rest), any add-on command nobody allowed - one that opens a ticket, say - and a search the check cannot fully see into, such as a savedsearch: dataset the parser leaves unexpanded. In Chat the agent can ask you to run such a search with splunk_search_with_approval: you see the SPL on an approval card, and it runs only if you approve, with your own permissions. A search the parser rejects outright never reaches an approval card - Splunk would fail it once approved - so the agent gets the parser’s error back and fixes the search. Alert investigations, | osai task= runs and the MCP server do not have that tool at all. Real-time ranges are refused either way. Searches you run yourself are unaffected, and your Splunk permissions remain the outer boundary.

An unattended run’s one write, its decision event, is made by the app when the run concludes, not by any tool the agent calls - an unattended run’s search tools never run collect. It is written as the user the run acts as, on that run’s own credential, so it needs that user’s own run_collect (or edit_tcp) capability; the app grants itself nothing for it. An interactive Chat turn never writes one.

Conversations are isolated by ownership: only a conversation’s owner or an administrator can read it or watch its live progress, and conversation and job identifiers are minted by the server, never chosen by a client, so nobody can guess or squat on another user’s conversation.

Data that talks back

Everything the model reads from your data - search results, knowledge objects, error messages that echo indexed text - is wrapped in an explicit untrusted-data boundary with instructions not to follow anything found inside it. That is the defense, and it does not depend on noticing an attack.

The app additionally scans what the agent reads for content aimed at it: text telling it to ignore its instructions, chat-transcript markers trying to close the boundary, requests for its prompt or credentials, directions to send data somewhere, and invisible characters positioned to hide text from a human reading the same event. Findings appear on the tool card in Chat, on the audit event, and in the app’s log. A finding never blocks the result - a security analyst investigating injection attempts in their own logs is reading attacker text on purpose - except invisible characters, which are stripped before the model reads them because their whole purpose is to make the model’s view differ from yours. Searches that read credential- or PII-shaped fields (password, api_key, ssn, credit_card, and a few more) are noted the same way, with an instruction to the model to summarize rather than quote.

A run that decides picks only from the answers you listed, so nothing in your data can put an answer you did not write into a decision event. It can still argue for one of yours, because the model weighs what it reads. Treat a decision like any other alert input: throttle the alert that acts on it, and wire it only to remediations you would accept firing on a wrong answer.

What a model writes cannot become script in your browser either. Chat, the HTML export and the print view render model output through one sanitizer: headings, tables, lists, code blocks and links render as written; images, forms, embedded media and inline styles do not; a link may point only at http(s), mailto, or a path within this Splunk instance, and opens in a new tab with no handle back on the Splunk page. The “Open dashboard” link on a tool card is offered only when it is a real dashboard path. This matters most for text the agent did not write itself: search results it read back, and imported conversations.

Secrets and storage

• Provider API keys live in Splunk’s encrypted credential store (storage/passwords), restricted to administrator roles and private to the app, so they are not listed among another app’s credentials; the setup API never echoes them back, and | osai refuses inline credential parameters. A key is identified by its provider’s name, never by a reference the caller chooses, and is only ever sent to that provider: an edit that changes where the key goes or what it is used as - the base URL, the kind, the AWS region, turning TLS verification off, or an Azure provider’s authentication - means entering the key again (see Providers), and a redirect is never followed, so the address a provider redirects to receives nothing. A base URL that carries a user name and password (https://user:password@gateway/v1) is never repeated with them in an error message or a log line.
• An inline key. A provider you write by hand in os_ai_agent.conf may carry api_key = ... instead of a stored key. It sits in the configuration file in plain text, readable by anyone who can read the file, so keep it to local evaluation; the Setup page always stores the key in storage/passwords.
• Conversations, messages, jobs, feedback, and usage records live in app-scoped KV Store collections readable only by administrators and the system context. Users reach their own data through the app’s REST handlers, which authenticate the caller and scope every read and write to that user.
• Per-job credentials - your session key for a queued chat turn or | osai run, or the token the worker mints for an unattended run, and the principal that run acts as - live in a separate collection only Splunk administrators and the system context can read or write. The credential is encrypted (AES-256-GCM) with a key the app creates in storage/passwords the first time a job is queued, so the collection never holds a usable credential, and it is cleared when the job settles. In a search head cluster that key replicates to every member like the app’s other credentials.
• The archive signing key is created on first export and kept where only administrators can read it, so a user cannot re-sign a conversation they edited.
• Nothing phones home. The only outbound traffic is to the provider endpoints you configured.

Splunk 10.6: KV Store values in _internal

Splunk 10.6 runs the KV Store on PostgreSQL behind a service, kvstore-pdl, that at its default log level logs every database operation, document values included, to $SPLUNK_HOME/var/log/splunk/sup-pkg-kvstore-pdl-stdout.log - and Splunk indexes that file into _internal. So by default on 10.6, conversations, tool results (search results among them) and conversation titles are also readable by anyone who can search _internal, and stay there for that index’s retention period: deleting a conversation in the app does not remove those copies. Job credentials are encrypted before they are stored, so no usable credential reaches that log either way. Earlier Splunk versions keep KV Store contents out of _internal.

On Splunk Enterprise, stop it by lowering that service’s log level with Splunk’s supported override. On every search head - each member of a cluster - create this file and restart Splunk:

1[kvstore-pdl]
2rootCategory = ERROR,rootAppender

New writes then stay out of that log and out of _internal (verified on 10.6.0.5); the service still logs its errors, and what was indexed before stays until it ages out. Until the override is in place, and until those earlier copies have aged out, limit who can search _internal.

On Splunk Cloud Platform you cannot edit that file or restart the search heads yourself. Ask Splunk Support whether your stack logs KV Store values this way and to apply the override - we have not measured Splunk Cloud - and limit who can search _internal in the meantime.

PostgreSQL also cannot store a NUL character, and Splunk’s upgrade to 10.6 drops any stored row that contains one. The app stores U+0000 as U+FFFD (�) in everything it keeps, on every Splunk version, so nothing it stored is lost to that upgrade. What we’ve learned so far about the KV Store in Splunk 10.6 has the details.

The activity trail

The app writes one event per agent turn, tool call, refusal, and configuration change to $SPLUNK_HOME/var/log/splunk/os_ai_agent_audit.log; Splunk’s own default monitor indexes it into _internal as sourcetype=os_ai_agent:audit, with no index to provision. Each event carries the user, the surface (chat, | osai, MCP), the conversation and job (MCP calls have neither), the provider and model, the decision behind each tool call, the object a change touched, the approver when a person approved on someone else’s behalf, injection flags, and the alert that triggered an unattended run.

This records what nothing else can. Splunk’s own audit index already logs that a dashboard was written; it cannot know which conversation asked for it, which tool call made it, whether a person approved it or a policy did, or what the agent was refused - a tool blocked by a conversation’s allowlist, a person declining an approval, a request turned away by the license, a budget, or the concurrency cap. Plain Splunk authorization failures are not duplicated here; Splunk’s audit index has them.

index=_internal sourcetype=os_ai_agent:audit event=tool mode=mutation ok=true
| table _time user approved_by tool target

Every write through the Setup page is an event=config_change: a provider created, updated, deleted or made the default, the settings (the license key and seat included), a query template, a guidance note. It names the user, the route and method, the object as target (provider:openai, settings, template:<id>, guidance:<name>), the names of the fields the request wrote as keys (such as limits.daily_cost_usd), and the outcome (ok, http_status). It never carries a value: an API key or a license key appears by name only. A request the Setup page refused is recorded too, with ok=false.

index=_internal sourcetype=os_ai_agent:audit event=config_change
| table _time user method route target keys ok http_status

An event written while the app answered a REST request - a configuration change, a synchronous chat turn, or a refusal by the license, a spend budget or the concurrency cap - carries the request_id that request answered with, so one search finds everything a failed request left behind; see Troubleshooting.

The Activity Trail dashboard reads the same data; see Dashboards. Writing is best-effort and never fails a request. [audit] disabled = true turns it off, and the Settings tab has the switch, which takes a full Splunk administrator. A change that switches the trail off through the Setup page is written anyway, as the trail’s last event, so the trail says who silenced it; an edit to the file is not.