Witness Node Setup¶

Section: High Availability | Article 18
Audience: System Administrators
Last Updated: 2026-04-07

Overview¶

A witness node is a lightweight RP-PAM instance that participates in leader election voting but does not store data, serve API requests, or run the web portal. Its sole purpose is to provide a tiebreaker vote in clusters with an even number of full nodes.

When to Use a Witness Node¶

Cluster Size	Witness Needed?	Reason
2 full nodes	Yes (recommended)	Without a witness, a network partition leaves both nodes unable to determine which should be leader
3 full nodes	No	Odd number already provides a natural tiebreaker
4 full nodes	Yes (recommended)	Prevents a 2-vs-2 split from causing a stalemate
5+ full nodes	No	Odd number provides a natural tiebreaker

What a Witness Node Does NOT Do¶

Does not store any database data
Does not serve the web portal or REST API
Does not handle user authentication or access requests
Does not require an RP-PAM license seat
Does not need significant CPU, RAM, or disk

Prerequisites¶

Requirement	Detail
A separate host	The witness must run on a different server (physical or virtual) than your full RP-PAM nodes to provide genuine fault independence
Network access	The witness must reach Redis (port 6379) and be reachable by all full nodes on port 5201
Docker (recommended)	The witness is most easily deployed as a Docker container
Redis	Running and accessible (see Redis Setup)

Resource Requirements¶

The witness is extremely lightweight:

Resource	Minimum
CPU	1 vCPU
RAM	256 MB
Disk	100 MB
Network	1 Mbps (heartbeat traffic only)

Option A: Deploy as a Docker Container (Recommended)¶

Step 1: Pull the Image¶

Linux:

docker pull ravenphyre/rppam-witness:1.0

PowerShell:

docker pull ravenphyre/rppam-witness:1.0

Step 2: Create the Configuration File¶

Create a minimal rppam-witness.config file on the Docker host.

Linux:

sudo mkdir -p /etc/rppam
sudo tee /etc/rppam/rppam-witness.config > /dev/null <<'EOF'
{
  "node": {
    "nodeName": "witness1",
    "grpcPortBase": 7001,
    "role": "witness"
  },
  "redis": {
    "enabled": true,
    "connectionString": "10.0.1.20:6379,password=YOUR_REDIS_PASSWORD,ssl=false,abortConnect=false",
    "keyPrefix": "rppam:",
    "tlsEnabled": false
  },
  "cluster": {
    "leaderLockTtlSeconds": 30,
    "leaderRenewalIntervalSeconds": 10,
    "heartbeatIntervalSeconds": 5,
    "peerEndpoints": [
      "https://10.0.1.10:5201",
      "https://10.0.1.11:5201"
    ]
  }
}
EOF

PowerShell:

New-Item -ItemType Directory -Force -Path "C:\ProgramData\Ravenphyre\RP-PAM"

@'
{
  "node": {
    "nodeName": "witness1",
    "grpcPortBase": 7001,
    "role": "witness"
  },
  "redis": {
    "enabled": true,
    "connectionString": "10.0.1.20:6379,password=YOUR_REDIS_PASSWORD,ssl=false,abortConnect=false",
    "keyPrefix": "rppam:",
    "tlsEnabled": false
  },
  "cluster": {
    "leaderLockTtlSeconds": 30,
    "leaderRenewalIntervalSeconds": 10,
    "heartbeatIntervalSeconds": 5,
    "peerEndpoints": [
      "https://10.0.1.10:5201",
      "https://10.0.1.11:5201"
    ]
  }
}
'@ | Set-Content "C:\ProgramData\Ravenphyre\RP-PAM\rppam-witness.config" -Encoding UTF8

Step 3: Run the Container¶

Linux:

docker run -d \
  --name rppam-witness \
  -p 5201:5201 \
  -v /etc/rppam/rppam-witness.config:/app/rppam.config:ro \
  --restart unless-stopped \
  ravenphyre/rppam-witness:1.0

PowerShell:

docker run -d `
  --name rppam-witness `
  -p 5201:5201 `
  -v "C:\ProgramData\Ravenphyre\RP-PAM\rppam-witness.config:/app/rppam.config:ro" `
  --restart unless-stopped `
  ravenphyre/rppam-witness:1.0

Step 4: Verify the Container Is Running¶

docker ps --filter name=rppam-witness
docker logs rppam-witness --tail 20

docker ps --filter name=rppam-witness
docker logs rppam-witness --tail 20

Look for log lines such as:

[INF] Witness node "witness1" started
[INF] Connected to Redis at 10.0.1.20:6379
[INF] Heartbeat published: witness1 (witness)

Option B: Deploy as a Native Service¶

If Docker is not available, you can run the witness as a native service using the full RP-PAM binary in witness mode.

Linux¶

Install RP-PAM normally (see Linux Installation).
Edit /etc/rppam/rppam.config with the same witness configuration shown above (the node.role set to "witness").

Start the service:

sudo systemctl enable rppam
sudo systemctl start rppam

Windows¶

Install RP-PAM normally (see Windows Installation).
Edit C:\ProgramData\Ravenphyre\RP-PAM\rppam.config with the witness configuration.
Start the service:
```
Start-Service RpPam
```

When node.role is "witness", RP-PAM automatically disables the web portal, REST API, database connections, and all module processing.

Step 5: Update Full Nodes to Know About the Witness¶

On each full RP-PAM node (node1, node2, etc.), add the witness to peerEndpoints in rppam.config:

{
  "databaseSync": {
    "peerEndpoints": [
      "https://10.0.1.11:5201",
      "https://10.0.1.30:5201"
    ]
  }
}

In this example, 10.0.1.30 is the witness node. Each full node lists all other nodes (full and witness) in its peerEndpoints.

Restart each full node after updating:

Linux:

sudo systemctl restart rppam

PowerShell:

Restart-Service RpPam

Step 6: Verify the Witness in the Cluster¶

Check Cluster Health¶

Linux:

curl -s http://localhost:7101/system/health/cluster \
  -H "Authorization: Bearer $ADMIN_JWT" | jq .

PowerShell:

$cluster = Invoke-RestMethod -Uri "http://localhost:7101/system/health/cluster" `
  -Headers @{ Authorization = "Bearer $adminJwt" }
$cluster | ConvertTo-Json -Depth 5

Expected output includes the witness:

{
  "clusterHealthy": true,
  "leaderNode": "node1",
  "nodes": [
    {
      "nodeName": "node1",
      "role": "primary",
      "status": "healthy",
      "lastHeartbeat": "2026-04-07T10:00:00Z"
    },
    {
      "nodeName": "node2",
      "role": "standby",
      "status": "healthy",
      "lastHeartbeat": "2026-04-07T10:00:02Z"
    },
    {
      "nodeName": "witness1",
      "role": "witness",
      "status": "healthy",
      "lastHeartbeat": "2026-04-07T10:00:01Z"
    }
  ],
  "redisConnected": true,
  "quorumMet": true
}

Verify Heartbeat in Redis¶

You can directly inspect the witness heartbeat in Redis:

redis-cli -h 10.0.1.20 -a YOUR_REDIS_PASSWORD \
  GET "rppam:heartbeat:witness1"

docker run --rm redis:7 redis-cli -h 10.0.1.20 -a YOUR_REDIS_PASSWORD `
  GET "rppam:heartbeat:witness1"

Expected: a JSON string containing the node name, role, and last heartbeat timestamp.

How the Witness Participates in Leader Election¶

The witness votes in leader election but cannot become the leader. During an election:

All nodes (full + witness) vote for the candidate they believe should lead.
A candidate needs N/2 + 1 votes (majority) to win.
In a 2-node + 1-witness cluster (3 voters total), a candidate needs 2 votes.
If node1 fails, node2 and the witness both vote for node2 -- node2 wins with 2/3 votes.
If the witness fails, the cluster still operates normally (2 full nodes can elect a leader with 2/2 votes from each other). The witness failure is logged as a warning.

Split-Brain Prevention¶

Without a witness in a 2-node cluster: - If the network between node1 and node2 splits, each node sees the other as failed. - Both nodes might try to become leader simultaneously (split-brain). - The leaderLockTtlSeconds in Redis prevents both from holding the lock, but if Redis is unreachable from one side, the situation is ambiguous.

With a witness: - The witness provides the tiebreaker vote. - The partition side with the witness (and therefore the majority) wins the election. - The other side steps down gracefully.

Troubleshooting¶

Problem	Cause	Solution
Witness shows `"status": "unreachable"`	Firewall blocking port 5201 to/from the witness	Open port 5201 between the witness and all full nodes
Witness container exits immediately	Invalid configuration file	Check `docker logs rppam-witness`; validate JSON syntax in config file
Witness not appearing in cluster health	Full nodes do not list witness in `peerEndpoints`	Add the witness address to `peerEndpoints` on all full nodes and restart
`"role": "witness"` missing in config	Node starting as full node instead of witness	Ensure `"role": "witness"` is set in the `node` section
Heartbeat key missing in Redis	Witness cannot connect to Redis	Verify Redis connection string in witness config; check password
Leader election still stalls with witness	Witness on the same host as a full node	Move the witness to a genuinely separate host for fault independence

Next Steps¶

Failover Testing -- Test failover with the witness in place
VIP Failover Configuration -- Add a virtual IP for client-transparent failover
HA Multi-Node Setup -- Return to the main HA guide