What do the colours represent in the Nodes page

I updated all servers in my network yesterday, and just noticed that one of the servers has a red spot next to its line in the Nodes page in the UI.

there is no tooltip when I hover over it.

Also, under the column header Node, it says “3/0/1”, with the 1 in red. Again, no tooltip.

what does this mean?

The colors represent the node’s liveliness.

Green = healthy
Yellow = stale
Red = missing/dead

For the following info, I’m not a developer on the project, so I may not be 100% correct, but here’s how the states work as I understand them. If I’m wrong in any details I’m sure someone will correct me :slight_smile:

If you have a cluster running and force one node to quit or if it becomes separated from the cluster, after 1 minute has elapsed, you should see that node transition to a stale (yellow) state.

The stale state indicates that the cluster is aware that the node is not responding. During this state, however, no re-replication of the ranges that exist on that node occurs. This is to prevent the heavy up-replication/rebalancing operation from happening during short-lived network partitions, where the cluster will reform shortly.

After a node has been unresponsive for some period of time (by default 10 minutes, or 9 minutes after being declared “stale”), it is declared actually missing/dead, and transitions to the red state in the UI. It is at this point that ranges which the failing node contained will be re-replicated to new, healthy nodes by the other nodes that contain replicas of those ranges.

1 Like

Hi kae!

twrobel is correct about the 1 min and 10 min thresholds.

Here’s UI code:
https://github.com/cockroachdb/cockroach/blob/master/pkg/ui/app/containers/nodesOverview.tsx#L218

The UI currently has to guess a bit, and these thresholds are based on the latest reported data from the node, and not specifically the internal cluster status, though that is ultimately what it is supposed to express.

You’re definitely right that this needs a tooltip/explanation! We’re in the middle of redesigning portions of the admin UI and this will be called out more clearly.

1 Like