Linux-Specific Troubleshooting¶

Section: Troubleshooting | Article 47
Audience: System Administrators (Linux)
Last Updated: 2026-04-07

Overview¶

This article covers RP-PAM issues specific to Linux environments. For general troubleshooting, see General Troubleshooting.

Permission Denied on /var/lib/rppam¶

Symptoms¶

Service fails to start with "Permission denied" in the journal
Log shows UnauthorizedAccessException: Access to the path '/var/lib/rppam/...' is denied

Cause¶

The RP-PAM service user does not have ownership or write access to the data directory.

Solution¶

Step 1: Check current ownership:

ls -la /var/lib/rppam/
ls -la /etc/rppam/
ls -la /var/log/rppam/

Step 2: Fix ownership (the RP-PAM service runs as the rppam user by default):

sudo chown -R rppam:rppam /var/lib/rppam
sudo chown -R rppam:rppam /etc/rppam
sudo chown -R rppam:rppam /var/log/rppam

Step 3: Set correct permissions:

sudo chmod 750 /var/lib/rppam
sudo chmod 750 /etc/rppam
sudo chmod 750 /var/log/rppam

# Config file should be readable by the service user only
sudo chmod 640 /etc/rppam/rppam.config

Step 4: Verify the service user:

# Check which user the systemd unit runs as
sudo systemctl show rppam -p User

If the service runs as a different user, adjust the chown commands accordingly.

Step 5: Restart:

sudo systemctl restart rppam
sudo systemctl status rppam

keyctl Not Found¶

Symptoms¶

Error during startup: keyctl: command not found
Log shows: Failed to initialize kernel key retention service

Cause¶

The keyutils package is not installed. RP-PAM uses the Linux kernel keyring (via keyctl) for secure in-memory key storage.

Solution¶

Debian/Ubuntu:

sudo apt-get update
sudo apt-get install -y keyutils

RHEL/Fedora/CentOS:

sudo dnf install -y keyutils

Verify installation:

which keyctl
keyctl show

Restart RP-PAM:

sudo systemctl restart rppam

Note: If you cannot install keyutils (e.g., in a minimal container), configure RP-PAM to use in-process key storage instead:
{
  "security": {
    "keyStorage": "in-process"
  }
}
This is less secure than kernel keyring storage but functional.

OOM (Out of Memory) Kills¶

Symptoms¶

Service stops suddenly without errors in RP-PAM logs
systemctl status rppam shows Main process exited, code=killed, status=9/KILL
dmesg shows Out of memory: Killed process

Diagnosis¶

Step 1: Check dmesg for OOM events:

sudo dmesg | grep -i "oom\|killed process" | tail -20

Step 2: Check current memory usage:

free -h

Step 3: Check the RP-PAM process memory:

ps aux | grep rppam

Solution¶

Approach	How
Increase server memory	Add RAM to the VM or physical server
Set memory limits in systemd	See below
Tune the .NET garbage collector	See below
Add swap space	Temporary fix — see below

Setting systemd memory limits (to prevent full system OOM):

sudo systemctl edit rppam

Add:

[Service]
MemoryMax=2G
MemoryHigh=1.5G

This limits RP-PAM to 2 GB and triggers pressure warnings at 1.5 GB, preventing it from consuming all system memory.

Tuning .NET GC for lower memory: Add to rppam.config:

{
  "runtime": {
    "gcMode": "workstation",
    "gcConserveMemory": 9
  }
}

Adding swap space (temporary):

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Port Conflicts¶

Finding Port Conflicts¶

sudo ss -tlnp | grep 7101

Example output:

LISTEN  0  128  0.0.0.0:7101  0.0.0.0:*  users:(("nginx",pid=1234,fd=6))

This shows nginx is using port 7101.

Resolution¶

Either stop the conflicting service or change the RP-PAM port:

# Option 1: Stop the conflicting service
sudo systemctl stop nginx

# Option 2: Change RP-PAM port
sudo nano /etc/rppam/rppam.config
# Change "port": 7101 to "port": 7102

sudo systemctl restart rppam

After changing the port, update: - Firewall rules (ufw or firewalld) - Reverse proxy configuration (nginx, Apache, etc.) - Client configurations

Firewall Rules¶

UFW (Ubuntu):

sudo ufw allow 7101/tcp comment "RP-PAM API"
sudo ufw status

firewalld (RHEL/Fedora):

sudo firewall-cmd --permanent --add-port=7101/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports

systemd Journal Analysis¶

The systemd journal is the primary log source for service lifecycle events on Linux.

Viewing RP-PAM Journal Entries¶

# Last 50 lines
sudo journalctl -u rppam -n 50 --no-pager

# Since a specific time
sudo journalctl -u rppam --since "2026-04-07 14:00:00" --no-pager

# Follow in real time
sudo journalctl -u rppam -f

# Errors only
sudo journalctl -u rppam -p err --no-pager

# Since last boot
sudo journalctl -u rppam -b --no-pager

Checking for Crashes and Restarts¶

# Show all start/stop events
sudo journalctl -u rppam | grep -E "Started|Stopped|Failed|killed"

Exporting Journal for Support¶

# Export to a file
sudo journalctl -u rppam --since "3 days ago" > /tmp/rppam-journal.log

# Or in JSON format
sudo journalctl -u rppam --since "3 days ago" -o json > /tmp/rppam-journal.json

SELinux Issues (RHEL/Fedora)¶

If SELinux is enforcing, it may block RP-PAM from accessing files or network ports.

Check if SELinux Is Blocking¶

# Check SELinux status
getenforce

# Check for RP-PAM denials
sudo ausearch -m avc -ts recent | grep rppam

Temporary Fix (for testing only)¶

sudo setenforce 0   # Sets SELinux to permissive mode

Permanent Fix (create a policy module)¶

# Generate a policy from recent denials
sudo ausearch -m avc -ts recent | audit2allow -M rppam-policy

# Install the policy
sudo semodule -i rppam-policy.pp

File Descriptor Limits¶

If RP-PAM manages many concurrent connections (SSH, database modules, HA), it may hit the default file descriptor limit.

Symptoms¶

Log shows Too many open files
Connections start failing under load

Check Current Limits¶

# Check the RP-PAM process limit
RPPAM_PID=$(pgrep -f rppam)
cat /proc/$RPPAM_PID/limits | grep "Max open files"

Increase Limits¶

sudo systemctl edit rppam

Add:

[Service]
LimitNOFILE=65536

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart rppam

Troubleshooting Summary¶

Problem	First Check	Command
Permission denied	Directory ownership	`ls -la /var/lib/rppam/`
keyctl not found	Package installed?	`which keyctl`
OOM killed	dmesg for OOM	`sudo dmesg \\| grep -i oom`
Port conflict	What's on the port	`sudo ss -tlnp \\| grep 7101`
Service won't start	Journal errors	`sudo journalctl -u rppam -n 50`
SELinux blocking	AVC denials	`sudo ausearch -m avc -ts recent`
Too many open files	File descriptor limit	`cat /proc/<PID>/limits`

Next Steps¶

General Troubleshooting — Platform-independent issues
Windows-Specific Issues — If you also run Windows nodes
Log Collection and Submission — Collect logs for support