Field names and searching

Collectord adds indexed fields to every event it sends - kubernetes_namespace, kubernetes_pod_name, docker_container_name, syslog_component, a Prometheus label as metric_label_<label>. This page covers how Collectord spells them and how to write searches that find them.

Underscores or dots

[general] fieldsSeparator chooses how Collectord spells the names it composes:

A Splunk metrics index always gets the dotted names, which is also how Splunk’s Analytics Workspace groups the fields. With an Elasticsearch or OpenSearch output, the default is .. Collectord reads the separator when it starts: a configuration reload does not change it, and Collectord logs that the change waits for a restart.

Fields you name yourself are sent as you wrote them under either separator, on both indexes: fields.cost_center stays cost_center, and so does a workload’s collectord.io/logs-userfields.cost_center. Only Collectord’s own names are respelled.

Before 26.10, a metrics-index record carried your own fields with dots too: cost_center arrived as cost.center.

Monitoring Kubernetes, OpenShift, Docker, Linux and macOS read both spellings. Every indexed field an app knows has a search-time alias to its other spelling, in both directions, so a search, an alert or a dashboard filter written against kubernetes_namespace finds events that carry kubernetes.namespace, and the reverse, in search, stats and eval.

Move to dotted names

Your own searches, alerts and data models name the fields one way. To move to dotted names while they still name the underscore ones, send both for a while:

1[general]
2fieldsSeparator = .
3
4[output.splunk]
5duplicateFieldsWithUnderscore = true

With Helm, set fieldsSeparator under collectord.configuration.general and duplicateFieldsWithUnderscore under collectord.configuration.outputs.splunk.default.

duplicateFieldsWithUnderscore sends every built-in field a second time under its underscore name, on the events index only, and Collectord refuses it unless the separator is .. Measured on 20,000 container-log events, it adds nothing to the license, since Splunk licenses the event text and not its indexed fields, but 36% to the compressed HEC traffic and about 25% to the index on disk, for as long as it is on. Turn it off once your searches name the dotted fields, or both.

A Prometheus label’s field is named after the label, so the apps alias only the labels they read. A search of yours on any other metric_label_* field needs the duplicate mode until you rewrite it.

Search Collectord’s fields

Measured on Splunk 9.4 and 10.4:

• field=value is the one to use. kubernetes_namespace=payments narrows the scan to the matching events just as kubernetes_namespace::payments does, without any fields.conf declaration, and through the apps’ aliases it also finds the events that carry the other spelling.

• field::value reads only the exact indexed name, never an alias: syslog_component::sshd misses every event sent as syslog.component. Where both spellings arrive, name both: (syslog_component::sshd OR syslog.component::sshd).

• tstats reads the index files and never an alias. Name both spellings, in the where and in the by, and take whichever is set:

  text

  Copy

  1| tstats fillnull_value="-" count where index=kubernetes_logs by kubernetes_namespace, kubernetes.namespace
  2| eval kubernetes_namespace=if(kubernetes_namespace=="-", 'kubernetes.namespace', kubernetes_namespace)
  3| stats sum(count) as count by kubernetes_namespace

  The apps’ own tstats searches already do. duplicateFieldsWithUnderscore keeps yours working while you change them.

• Don’t declare the apps’ fields in a fields.conf of your own. A field declared INDEXED = true is answered from the index files alone, which stops the alias: a search on the spelling the events lack finds nothing. An undeclared indexed field narrows a search just the same.

Before 26.10, the apps’ fields.conf declared their fields INDEXED = true.