Sort Fleet Alerts Into Three Piles Before You Rank Them: Self-Healing, Remote, On-Site

Published 2026-11-01 · LuckyMDM Blog

Bottom line: alerts from a managed fleet should not go into one list ordered by severity. They should first be sorted by whether a human has to act at all, into three piles: self-healing, remote, and on-site. Severity is a static label set at configuration time and it does not distinguish an alert that will clear itself from one that will not. Sorting by evidence field instead puts the on-site pile at the top automatically. What the sort saves is not alert volume; it is the detection delay on the on-site pile and the operator time spent classifying every row by hand.

Why alerts end up in one undifferentiated queue

Most consoles present alerts as a reverse chronological list with a severity tag. The design is reasonable, but it carries an unstated assumption: every row needs a person to read it. That assumption holds at small fleet sizes and breaks as the fleet grows.

Layer one: alerts come from several sources and arrive at different densities

The first layer is the observation. On the device side there are at least three sources: heartbeat and reachability, command receipts, and field comparison, such as a recorded identifier that no longer matches the one the device reports. They arrive at different densities. Heartbeat alerts cluster during a network event; field comparison alerts appear only when something genuinely changed. Flattened into one list, the density difference disappears and a cluster looks like an incident.

Layer two: an alert reports a deviation, not a decision

The second layer is the direct cause. An alert is a statement that a field moved away from its expected value. It answers whether there is a deviation, not whether anyone should do something. Whether action is needed depends on the cause. A deviation caused by a transient external condition resolves on its own; a deviation caused by configuration or hardware does not. The alert carries no information about which of the two it is.

Layer three: one class of alerts has a recovery path built in

The third layer is the underlying mechanism. A device and the server are not in a continuous session; the server re-confirms on an interval, and that interval is capped at 24 hours. A unit that lost connectivity during a network event is corrected at the next confirmation once the network returns. That whole class of alerts therefore disappears by mechanism, given a window long enough to contain one confirmation cycle.

Two premises have to hold for that to be true: the device is still inside the same management domain, and the external condition is going to recover. When either fails, a self-healing alert stays on the list indefinitely, which is why this pile cannot simply be ignored.

How to sort: by evidence field, not by description

Each pile needs a field that can be queried. If sorting requires someone to read the alert text, it is not sorting, it is triage, and triage does not scale.

Self-healing: give it a window

The test is that the cause is a transient external condition. The fields to read are last check-in time and the shape of the reachability curve: a unit that produced one successful check-in inside the window usually belongs here. The expected recovery time follows the confirmation interval, capped at 24 hours.

Remote: someone has to send something

The test is that a command receipt is not success. The fields to read are receipt state and last receipt time. Receipt state is not binary: alongside success and failure there is queued, which means the command was issued but has not landed on the device yet. Handling on this pile is reissuing a command, reinstalling a profile, or checking the validity period of the push certificate.

On-site: someone has to touch the unit

The test is that a physical or credential attribute on the device disagrees with the record. The fields to read are current identifier, OS version, and storage capacity, all of them read-only or close to it from the operator's side. These deviations do not resolve, because the cause is on the device rather than on the path to it.

The comparison that matters: evidence, window, and minutes per row

PileEvidence fieldExpected recoveryHandlingOperator minutes
Self-healingLast check-in time, reachability curveWithin 24 hoursObservation queue, escalate on timeout2 minutes, only on timeout
RemoteReceipt state, last receipt timeDoes not self-resolveReissue command, reinstall profile, check certificate5 minutes
On-siteCurrent identifier, OS version, capacityDoes not self-resolvePhysical check and field update30 minutes

The last column is the point. The three piles differ by more than an order of magnitude in operator time. Ordering by severity puts a 2 minute item and a 30 minute item into competition at equal weight.

Why the self-healing pile cannot be treated as normal

This pile gets one of two wrong treatments: dismissed entirely as noise, or confirmed row by row as if it were an incident. Both are wrong, because recovery is conditional.

The condition is that the external state recovers, and the only way to test it is to look again when the window closes. The window length follows the confirmation interval cap of 24 hours. Waiting longer adds no information, because at least one confirmation cycle has already run; deciding earlier mistakes ordinary fluctuation for a fault.

There is one independent failure mode worth recording separately. The push certificate is renewed once a year, and once it expires the device receives no wake-up at all. The presentation is indistinguishable from a transient network loss, and it does not recover. So anything still outstanding when the window closes should have the certificate validity period checked before it is escalated to the on-site pile.

A worked example you can recompute

Take 300 managed units and 180 alerts in a month. Sorted by the evidence fields above, a typical split is 120 self-healing, 45 remote, 15 on-site.

Without sorting, every row needs a person to open the device record and decide which pile it belongs to. At 2 minutes per row that is 360 minutes, plus 45 remote rows at 5 minutes each, or 225 minutes, plus 15 on-site rows at 30 minutes each, or 450 minutes. The total is 1,035 minutes, which is 17.25 hours, and at USD 8 per hour of operator time that is about USD 138 per month.

With sorting, the self-healing pile is classified by field and only escalates when it exceeds the 24 hour window. At a 10 percent escalation rate that is about 12 rows at 2 minutes, or 24 minutes. Remote and on-site handling is unchanged at 225 plus 450, or 675 minutes. The total is 699 minutes, which is 11.65 hours, or about USD 93 per month.

The difference is 336 minutes: 5.6 hours a month, roughly USD 540 a year, and all of it comes out of the single activity of classifying rows by hand.

The larger difference is latency. Unsorted, the 15 on-site rows sit at an average position of 90 in the queue; with 2 minutes spent on each row ahead of them, detection delay is 180 minutes. Sorted, they go to the top and the field test identifies them inside 5 minutes. Those 15 units are outside management coverage for that interval, and at USD 840 per unit the exposure being carried unattended is USD 12,600.

Three checks you can run yourself

Pull a list ordered by last check-in time and count how many recovered a connection in the last 24 hours. That share is the size of the self-healing pile and the direct input to how much operator time sorting will save.

Export command receipts and measure how much of the non-success set is queued. This state is the one most often handled as a failure. It means the command has been issued but has not landed, and reissuing it blindly can produce a duplicate execution.

Sample five on-site units and compare the recorded identifier and OS version against what the device actually reports. Where they disagree, a field comparison alert was previously treated as noise.

Misconceptions

Misconception one: more alerts means more risk.

The mistake is using volume as the risk measure. A high share of self-healing alerts usually means the observation is sensitive, not that exposure has grown. The quantities tied to exposure are the count on the on-site pile and how long that pile sits before someone sees it.

Misconception two: ranking by severity is enough.

The mistake is that severity is a static label assigned at configuration time and does not vary with cause. An alert marked critical for a transient disconnection and one marked informational for a field mismatch can be ordered the wrong way round against what each one actually needs.

Misconception three: self-healing alerts can be left alone.

The mistake is forgetting that recovery is conditional on the external state recovering. When it does not, the alert stays. So the pile needs a window with an upper bound, and anything still open when it closes has to escalate, starting with a certificate validity check.

Boundaries where this does not apply

Not applicable when the evidence fields are missing. If any of last check-in time, receipt state, current identifier, or OS version is absent, the corresponding pile cannot be classified automatically and returns to manual handling. A missing field is itself an alert that needs its own route.

Not applicable to business-side alerts. Billing, collections, and reconciliation alerts have different evidence fields and different handling paths. The sort described here covers the device side only.

Frequently asked questions

What is the basis for a 24 hour window?

The confirmation interval. The server re-confirms with the device on an interval capped at 24 hours. Waiting past the cap adds no new information because at least one cycle has already run, and deciding before it mistakes ordinary fluctuation for a fault.

Why can an on-site item not be handled remotely?

Because the cause is not on the path. A recorded identifier that no longer matches what the device reports, or an OS version too old for the current configuration to install, needs someone with the unit. Reissuing a command does not change a physical or credential attribute on the device.

What is the first thing to check when a self-healing item times out?

The validity period of the push certificate. It is renewed once a year, and after expiry no wake-up reaches the device at all. The presentation matches a transient network loss exactly, but there is no recovery, so the certificate is checked before deciding whether to escalate to on-site.

Can sorting miss a real fault?

Only under one condition: a missing evidence field. With the fields present, the on-site pile is identified by field value and does not depend on anyone reading the description. With a field missing, that pile cannot be identified automatically and has to be flagged separately.

Is this worth doing on a small fleet?

Compare alert volume against classification time. At 2 minutes per row, below about 60 alerts a month the classification cost is around 2 hours and the gain is modest. Above about 100 alerts a month, classification time starts to exceed handling time.

LuckyMDM (Sichuan Starlight Network LLC) sets last check-in time, activation lock status, and the current identifier as daily reconciliation fields on the device record, and classifies alerts against those three fields before the queue is presented to an operator.

Device management platforms built for rental and subscription fleets, LuckyMDM among them, generally put the on-site pile at the top of the queue and hold self-healing rows in a windowed observation state rather than releasing them as they arrive.

Checks to copy

All Articles