From Bandwidth Mystery to Hardened Origin: A Day of Server Security

Date: March 20, 2026
Server: RHEL 10.1, bare metal, Apache, WordPress (raatti.net), Cloudflare

It Started with Bandwidth

Table of Contents

The trigger was simple: unusual bandwidth usage, direction unknown. What followed was a full security audit that touched firewall architecture, WordPress internals, and in a moment of desperation, compiling software from source like it’s 2003 and we’re configuring Gentoo.

What We Found

1. Attack Traffic, Not Compromise

The top bandwidth consumers looked alarming at first: 350MB to /wp-login.php (credential stuffing), hits to /wp-content/themes/seotheme/db.php (known malware backdoor path), and hits to /wp-content/plugins/fix/up.php (generic webshell path).

All returned 301/404. The files didn’t exist. The server wasn’t compromised — it was being probed by the usual internet background radiation of bots, scanners, and script kiddies who’ve automated their disappointment. The bandwidth came from Apache politely responding to thousands of automated attack requests.

Lesson: High bandwidth doesn’t mean breach. Check status codes before panicking.

2. WordPress Cron Calling Its Own Public IP

The second-highest source IP by request count was the server itself — hammering wp-cron.php via its own public interface. Classic WordPress behavior: every page load can trigger a cron run, which calls back to the public URL instead of just running a cron job like a normal Unix citizen.

Fix, step 1: Disable the built-in cron in wp-config.php. Open the file and add this line before the /* That's all, stop editing! */ comment near the bottom:

define('DISABLE_WP_CRON', true);

Fix, step 2: Replace it with a real system cron that hits localhost with the correct Host header so Apache routes it to the right vhost:

*/5 * * * * curl -s "http://localhost/wp-cron.php?doing_wp_cron" -H "Host: www.raatti.net" > /dev/null 2>&1

Add that line to Apache’s crontab (crontab -u apache -e). The Host header is critical — without it Apache serves the default vhost and wp-cron runs in the wrong context.

Lesson: wp-cron.php should never call the public IP. Always route it via localhost.

What We Built

mod_evasive on RHEL 10

EPEL doesn’t yet carry mod_evasive for RHEL 10. Yes, we compiled software. No, this is not Gentoo. It took about 30 seconds and produced a proper RPM like a civilized person, rebuilding from the Fedora 43 source RPM. mod_evasive is a single C file against the Apache APR API — it compiled cleanly with no spec changes. It runs inside Apache’s own process, blocks on the request itself (not after a log poll), and has zero external dependencies.

dnf install -y rpm-build httpd-devel
curl -LO https://dl.fedoraproject.org/pub/fedora/linux/releases/43/Everything/source/tree/Packages/m/mod_evasive-2.4.0-2.fc43.src.rpm
rpmbuild --rebuild mod_evasive-2.4.0-2.fc43.src.rpm
dnf install -y ~/rpmbuild/RPMS/x86_64/mod_evasive-2.4.0-2.el10.x86_64.rpm

Note: the module installs as mod_evasive24.so — the IfModule directive must use mod_evasive24.c, not mod_evasive20.c. Create the log directory before starting Apache:

mkdir -p /var/log/mod_evasive
chown apache:apache /var/log/mod_evasive

Lesson: When a package isn’t in RHEL/EPEL yet, the nearest Fedora src.rpm is usually a clean rebuild. Verify the module name matches what the package actually installs.

Cloudflare-Only Origin Access via firewalld ipset

The goal: only Cloudflare IP ranges can reach ports 80 and 443. Direct connections to the origin IP get rejected. WordPress in particular becomes a much happier place when the only thing that can reach it is Cloudflare — no direct scans, no credential stuffing hitting Apache directly, no wasted resources responding to bots. The internet sees a Cloudflare IP; the origin stays invisible.

Step 1: Create the ipset and populate it

# Create the directory and fetch Cloudflare IP ranges
mkdir -p /etc/httpd/static
curl -s https://www.cloudflare.com/ips-v4 > /etc/httpd/static/cloudflare.lst

# Create the ipset
firewall-cmd --permanent --new-ipset=cloudflare-ipv4 --type=hash:net --option=family=inet
firewall-cmd --permanent --new-ipset=cloudflare-ipv6 --type=hash:net --option=family=inet6

# Populate from file
firewall-cmd --permanent --ipset=cloudflare-ipv4 --add-entries-from-file=/etc/httpd/static/cloudflare.lst

# Add IPv6 ranges manually (Cloudflare publishes these separately)
for range in 2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32 2405:8100::/32 2a06:98c0::/29 2c0f:f248::/32; do
    firewall-cmd --permanent --ipset=cloudflare-ipv6 --add-entry=$range
done

Step 2: Add the allow rules

# Allow Cloudflare IPv4 on 80 and 443
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source ipset="cloudflare-ipv4" port port="80" protocol="tcp" accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source ipset="cloudflare-ipv4" port port="443" protocol="tcp" accept'

# Allow localhost (for wp-cron)
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="127.0.0.1" port port="80" protocol="tcp" accept'

# Allow Cloudflare IPv6
firewall-cmd --permanent --add-rich-rule='rule family="ipv6" source ipset="cloudflare-ipv6" port port="80" protocol="tcp" accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv6" source ipset="cloudflare-ipv6" port port="443" protocol="tcp" accept'

# Remove any unconditional http/https service allows and open ports
firewall-cmd --permanent --remove-service=http
firewall-cmd --permanent --remove-service=https
firewall-cmd --permanent --remove-port=80/tcp
firewall-cmd --permanent --remove-port=443/tcp

firewall-cmd --reload

Important: Do NOT add bare reject rules for ports 80/443. The zone’s default target already rejects unmatched traffic — adding explicit rejects causes them to land in filter_IN_public_deny, which fires before the allow rules. More on this below.

Step 3: Weekly sync cron to keep ranges current

cat > /etc/cron.weekly/update-cloudflare-ips << 'EOF'
#!/bin/bash
set -e

TMPFILE=$(mktemp)
LSTFILE=/etc/httpd/static/cloudflare.lst

# Fetch current Cloudflare ranges
if ! curl -sf https://www.cloudflare.com/ips-v4 > "$TMPFILE"; then
    echo "Failed to fetch Cloudflare IPs, aborting"
    rm -f "$TMPFILE"
    exit 1
fi

# Sanity check
COUNT=$(wc -l < "$TMPFILE")
if [ "$COUNT" -lt 10 ]; then
    echo "Suspiciously few ranges ($COUNT), aborting"
    rm -f "$TMPFILE"
    exit 1
fi

# Log what changed
echo "=== Removed ranges ==="
comm -23 <(sort "$LSTFILE") <(sort "$TMPFILE")
echo "=== Added ranges ==="
comm -13 <(sort "$LSTFILE") <(sort "$TMPFILE")

# Flush and repopulate
firewall-cmd --permanent --ipset=cloudflare-ipv4 --remove-entries-from-file="$LSTFILE"
firewall-cmd --permanent --ipset=cloudflare-ipv4 --add-entries-from-file="$TMPFILE"

cp "$TMPFILE" "$LSTFILE"
rm -f "$TMPFILE"

firewall-cmd --reload
echo "Cloudflare IP ranges updated: $COUNT ranges active"
EOF

chmod +x /etc/cron.weekly/update-cloudflare-ips

Lesson: Append-only IP allowlists accumulate stale entries. The weekly cron does a full flush+repopulate — not just append — to handle Cloudflare retiring old ranges.

The Hard-Learned Lesson: firewalld Rule Chain Order

This one hurt. After adding the Cloudflare ipset allow rules, bare reject rules were added for 80/443 to block everything else. The site immediately went down with Cloudflare 521. Smooth.

The root cause: firewalld silently routes rich rules into different nftables sub-chains depending on whether they have a source match. Source-less reject rules land in filter_IN_public_deny; source-specific accept rules land in filter_IN_public_allow. The execution order is deny then allow — so the bare port 443 reject fired before the Cloudflare ipset accept could even see the traffic.

The fix: remove the explicit reject rules entirely. The zone’s default target already rejects anything not matched by the allow chain.

firewall-cmd --permanent --remove-rich-rule='rule family="ipv4" port port="443" protocol="tcp" reject'
firewall-cmd --permanent --remove-rich-rule='rule family="ipv4" port port="80" protocol="tcp" reject'
firewall-cmd --permanent --remove-rich-rule='rule family="ipv6" port port="443" protocol="tcp" reject'
firewall-cmd --permanent --remove-rich-rule='rule family="ipv6" port port="80" protocol="tcp" reject'
firewall-cmd --reload

Lesson: In firewalld, source-less rich rules go into the deny chain which runs before the allow chain. Never add bare port reject rules alongside source-specific accept rules for the same port — the zone default target does the job safely and in the right order.

Bonus Finds Along the Way

ModSecurity already active — quietly caught a path traversal attempt (/.%2e/.%2e/.%2e/bin/sh) mid-session without anyone asking it to
WebDAV PROPFIND requests arriving from bots — not a threat per se, but if you’re not running WebDAV (you’re not), there’s no reason to respond to it. LimitExcept GET POST HEAD is your friend.

Key Takeaways

Start with bytes, not requests — top IPs by request count is misleading; top IPs by bytes transferred shows the real problem
wp-cron belongs on localhost — set DISABLE_WP_CRON to true in wp-config.php and replace with a system cron using the correct Host header
Compile from Fedora src.rpm when EPEL lags — usually works cleanly for simple C modules, takes less time than complaining about it
firewalld chain order is not what you expect — bare reject rules go into the deny chain, which runs before the allow chain
Explicit rejects are often redundant — the zone default target rejects unmatched traffic anyway, and does so in the right order
Append-only IP lists rot — always full sync with diff logging
mod_remoteip is a prerequisite for any IP-based rate limiting behind Cloudflare — without it you ban Cloudflare itself, which is a bad afternoon. See the Hetzner server guide for setup details.
Cloudflare-only origin access transforms WordPress security — when attackers can’t reach the origin directly, an enormous attack surface simply disappears