This page covers where to look when something does not work and the failures people hit most often.
Where the logs are
Warnings and errors a command raises appear as messages on the search itself, and the command’s stderr is captured in the job’s search.log, both readable from the Job Inspector. Each search additionally writes its own aws_search.log in the search’s dispatch directory on the search head, under $SPLUNK_HOME/var/run/splunk/dispatch/<sid>/; splunkd cleans these up as the search expires, and on Splunk Cloud they are not reachable. At the default info level it holds the search’s warnings and failures, including every message the search shows once it has started. A search that ends before its settings are read writes no log, and says why on the search: one refused for the way its arguments are written or for a missing run_aws_search capability, and one that cannot load aws_search.conf at all. At debug it also says which account and region pairs the search targeted, how each one authenticated (the credential type, the access key ID or the role and its session name - never a secret), what came from the cache, from AWS, or was resolved locally with no AWS call, and how long each target took. The app writes argument names and never their values, and no result rows, and debug applies to its own lines only - the HTTP and AWS clients underneath stay at info, because their debug output carries headers and response bodies. An access key ID or a role ARN is written only when it has the shape of one, in case a secret was pasted into the wrong field. A URL is written as its scheme and host in every line of the file, since an endpoint override can carry a password or a token and a connection that fails quotes the address it was sent to. That keeps the file safe to send to support. The one thing it carries from AWS is the error message of a target that failed, in which AWS can quote part of a statement it rejected.
The app’s setup handlers and the cache sweeper log to Splunk’s internal index:
index=_internal aws_searchTo raise the detail while reproducing an issue, set the log level to debug on the Setup page’s Settings tab, or set level = debug under [logging] in aws_search.conf. Put it back afterwards - debug output is verbose.
Common issues
A command is not recognized, or you get a capability error. Running the commands requires the run_aws_search capability. Assign the user the aws_search_user role (or aws_search_admin). See Access control.
Every page says the app is not configured. Splunk gates the app’s views until setup is done. Open the Setup page and save an account; that clears the gate, whoever saves it - the app marks itself configured with its own permissions. If the pages still say so afterwards, reload Splunk Web or sign out and back in: Splunk checks the gate at sign-in.
Test fails with InvalidClientTokenId or SignatureDoesNotMatch. The access key id or secret is wrong, or the key was deactivated in IAM. Re-enter both on the account form (a secret field left blank keeps the stored value, so type into it) and test again.
Test succeeds but a search returns AccessDenied. The identity authenticates but lacks the permission for that API. Attach the IAM policy, and remember that Cloud Control calls the underlying service’s describe and list APIs, so those read permissions are needed too - ReadOnlyAccess is the simplest way to cover them.
Assume role fails with AccessDenied. The base identity lacks sts:AssumeRole on the role, the role’s trust policy does not name it, or the external id does not match the trust policy’s condition. The error names which. Cross-account assume-role also requires a paid license.
A region is rejected. The account has an allowed-regions list and the search named a region outside it. Add the region to the list on the account form, or search within the list. See Accounts - allowed regions.
region=* warns that it used the default region. The account has no allowed-regions list, so * is not a fan-out; the app uses the default region and says so rather than pretending. Set a list to make * mean exactly those regions.
Results have fewer fields than the console shows. Many Cloud Control types list only an identifier; the full state needs detail=true (one extra call per resource, cap it with limit=). Association details - which instance a volume is attached to, an Elastic IP’s target - come from the association kinds (volume-attachments, eip-associations). Some console-only fields (an instance’s status checks, a load balancer’s state) are not in the Cloud Control model at all; use the service’s CloudWatch metrics instead.
Results are empty. Check the obvious first: the right account=, region=, and kind=. Then whether the identity can list that type (a denied listing shows as a _target_error row, not as an empty result). Confirm with a kind you know exists, like | awsget kind=vpcs.
| awscost fails or returns nothing. Cost Explorer must be enabled in the AWS account (the Billing console does it; the first data appears about 24 hours later), and the identity needs ce:GetCostAndUsage. Remember every call is billed.
The region picker on a dashboard is empty. The picker runs | awsget kind=regions, which needs ec2:DescribeRegions when the account has no allowed-regions list. Grant it, or set the list.
A search times out, or reports a per-request timeout. The search head cannot reach the AWS endpoint, or a single request took longer than the request timeout (default 30 seconds). Check the network path first - on Splunk Cloud, the stack’s outbound port rules (see Splunk Cloud) - then raise the timeout on the Settings tab if the API is simply slow.
A second account is refused. The free tier allows one account. Registering more requires a paid license. See Licensing.
Searches are blocked with a license message. The app is running in a search head cluster without a paid license, the license expired past its grace period, or more accounts are registered than the license allows. The License tab shows which; nothing is deleted.
Data looks stale. Results may be served from the short-lived cache. Add cache=0 to force a live call, or clear the cache on the Cache tab. See Concepts - caching.
An account is skipped with a message about its endpoint override. The override is not HTTPS, or it names an instance-metadata address; both are refused when the configuration is loaded as well as when it is saved, so a value edited into awsaccounts.conf by hand is caught too. Fix the value on the Setup page. A plaintext endpoint is only for a local mock, and then allow_insecure_endpoint = true on the stanza allows it - see Accounts.
The search says your credential override is ignored. An override in your awsauth.conf is the instance role, or an assumed role with no base keys; either would run as the search head, so the account’s own credential is used instead. Save an override with your own keys, or delete it on the Credentials tab.
One account fails during a fan-out. When you target several accounts with account=*, an account whose AWS call fails emits a single row carrying _target_error and the others still return; search for that field to see which account failed and why. An account that fails before any call is made - its stored secret cannot be read, or its role cannot be assumed - has no row: it is skipped with a warning in the search’s messages, which the Job Inspector also shows.
Verify a connection
The quickest end-to-end check of a registration is the Test button on the Accounts tab, which calls sts:GetCallerIdentity. From the search bar, list a kind every read identity can see:
| awsget kind=vpcs account=prod region=us-east-1If that returns rows, the credential, permissions, and network path are all good, and any empty result from a more specific query is about the query or IAM, not the connection.