Skip to content

Hosts

Data Engineering, Data quality, Hosts answers three questions a SOC asks every day: which hosts are sending each log, which have gone silent, and which are new. A log can look healthy while one in ten of its hosts has stopped. The rules still run and still alert, and they see nothing from those hosts.

What you see

Hosts sending Distinct hosts that sent at least one log within their own usual gap, out of all hosts seen in the last 7 days
Silent hosts Hosts that have stopped sending at least one log. Each host is counted once
Stopped on every log Silent hosts that are silent on every log they send. The host or its forwarder has most likely stopped
New hosts this week Hosts first seen in the last 7 days, with nothing in the 23 days before
Hosts sending a day, 30 days Distinct hosts with at least one event on each day
Silent hosts by index Where the silent hosts are
Silent hosts table Each silent host with when it was last seen, the logs affected, how many rules read those logs, and the owner
Hosts by log table Each log with its hosts sending, silent and new, and the rules that read it

A log's own page has a Hosts tab with every host of that log, sending and silent.

When a host counts as silent

Vyrux judges each host on its own history, log by log. A flat limit such as 24 hours is wrong for most hosts: it is far too long for a domain controller and too short for a server that sends one batch a week.

  • A host is silent on a log when it sent that log in the last 7 days and has now been quiet for longer than its own limit.
  • The limit is 1.5 times the host's longest quiet spell on that log in those 7 days.
  • It is never under 4 hours (the floor; change it under Admin, Calibration) and never over 3 days.
  • The log's own arrival delay is added, up to a day. If the events of a log always reach Splunk four hours after they happen, every host of that log is given four more hours. Without this, such a log would show all its hosts as silent.
  • Only a host with a steady pattern on that log is judged: one seen in at least 24 separate hours of the 7 days, or seen in three or more hours at regular intervals (its longest gap is no more than 1.5 times its average gap). A host that sends now and then, as most hosts do on a log of alerts, is never called silent. After a day with nothing it shows as No steady pattern on the log's page.
  • Only on a log whose hosts send steadily. When the typical host of a log sends in fewer than 15 percent of the time periods of the 7 days, as on a log of alerts, hosts send only when something happens, so none is judged. Hosts by log shows such a log as No, sends on events under Judged.
  • A host that has sent nothing for 7 days leaves the read. Its issue closes, marked as not repaired.

Two examples. A firewall that sends every minute has a longest quiet spell of an hour at most, so it is silent after 4 hours. A scanner that reports once a day has a longest quiet spell of 24 hours, so it is silent after 36 hours.

A host that was down for two days this week keeps a limit of 3 days until that outage is more than 7 days old. That is the price of judging each host on its own history.

Two cases are not covered. On a log where hosts send only when something happens, such as an alert feed, some hosts look steady by chance and then show as silent when they are only quiet. On the lab this left 477 of 8,147 hosts of a Defender alert log marked silent. Raise Open a host issue from under Admin, Calibration for such an estate, or suppress that log's host issue.

The second: A feed that arrives in batches further apart than the limit, with event times that fill the gaps, looks continuous in its history and quiet between batches. Its hosts can show as silent until the next batch lands. Raise the floor under Admin, Calibration, or plan a window, if you have such a feed.

How it is read, and what it costs

The background scan runs one search over the last 7 days. It uses tstats, which reads index, sourcetype and host names from the index files and never opens an event:

| tstats max(_time) as lt count where index=* NOT index=_* by index sourcetype host _time span=1h

A second, small tstats inside the same search reads each log's arrival delay: how far its newest event is behind the time it was indexed.

The counts by log, by index and for the whole estate are worked out inside that search. Only names, counts and times leave it: no event text and no field value. At most 5,000 silent host rows and 500 new host rows are kept; the counts are always whole. Two more tstats searches run once a day: new hosts (30 days) and hosts sending a day (30 days).

On a Splunk Enterprise 10.6 lab with 11,064 hosts, 358 logs with events and 30,244 host and log pairs, the 7 day search took 22 seconds with one hour spans inside the background scan (15 to 37 seconds when run by hand), and 11 to 13 seconds with four hour spans. The two daily searches took about 16 seconds each. So:

  • A read that takes longer than 20 seconds is made once an hour, not at every scan.
  • Above 20,000 host and log pairs the 7 days are cut into four hour spans, and above 100,000 into one day spans. The floor is never under two spans, so on a very large estate the shortest silence Vyrux can find is 8 hours, or 2 days.

These figures come from one lab. Measure on your own search head: Admin, System health shows how long the last host read took, and has a switch to turn it off.

Issues and alerts

When hosts stop sending to a log that at least one enabled rule reads, the background scan opens one issue for that log, such as "12 hosts stopped sending to WinEventLog:Security". The issue names the hosts (up to 50), the rules that read the log, and the ATT&CK techniques those rules are mapped to. It follows the count, and closes when the hosts send again.

  • A log that is silent as a whole has its own issue, and no host issue.
  • A log no rule reads opens no issue. Its silent hosts are still on the Hosts page.
  • One alert is sent for each change: when hosts go silent on a log, and again when more do. Nothing is repeated while nothing changes. More than ten changes in one scan go as ten messages and one that counts the rest.
  • Alerts go to the channels that have Hosts go silent turned on under Admin, Integrations. It is off until you turn it on.
  • Alerts do not depend on the licence.

Planned work

Under Admin, Maintenance, plan a window for a log or for one host. A silent host in a window opens no issue and sends no alert, on every log it sends. You can plan a window for a host only when one of your own Splunk roles can search a log that host sends.

Find a host

Type a host name, or part of one with *, to see every log it sends, when each last arrived, how long it may be quiet, and whether it is silent there. This read runs with your own Splunk permissions, so it lists only logs you can search.

Who sees what

Host names come from every index. They are kept where only Splunk admins and the Vyrux service can read them, and the service gives each person the rows for indexes their own Splunk roles can search. Totals for the whole estate, the 30 day trend and the report figures go to people whose roles search every index. Anyone else gets totals read with their own session, by the same search.

In reports

The weekly and monthly report decks have a Hosts slide, and the board pack and the Posture page have the same figures: hosts sending and silent hosts at the end of the period, hosts that went silent and came back, the longest silence, the daily trend, and the logs and rules affected. The record of silences starts with the first background scan after build 82, so a period before that has no figures.

Ask Chuka

"Which hosts stopped sending?" is answered from the scan's figures. Host names are replaced with placeholders before the question leaves Splunk, unless an admin turned network masking off under Admin, AI.