# Server Hardening Guide — IMS (Laravel + Next.js) Companion to the source-level security work (see `doc/security/payment-integrity-finding.md` and the application security audit). That work closes vulnerabilities *in the code*; this document closes the gap around it — the OS, network, web server, database, and deployment pipeline the code runs on. **A hardened app on an unhardened server is still compromised the moment the server is.** Written for the actual deployment shape in this repo: Ubuntu/Debian servers, Laravel 12 (PHP-FPM) behind Apache or Nginx, PostgreSQL, Next.js served by Node/PM2, multiple client backends per host (`/var/www/html/api__backend`), deployed by `git reset --hard` from GitHub Actions (`.github/workflows/ci-cd.yml`). Adjust package-manager commands (`apt`) if you're on a different distro. Work through this section by section on **every** production and staging host. Each section ends with a way to verify it actually worked — don't just run the commands and assume. --- ## 0. Threat model (why each section exists) | Attacker goal | Sections that stop it | |---|---| | Get a shell on the box (SSH brute force, leaked key, weak password) | 1, 2, 3 | | Reach a service that should be internal-only (DB, Redis, queue) | 2, 6, 7 | | Read source, `.env`, or `.git` over HTTP | 4, 8 | | Exploit an unpatched OS/PHP/Node/package CVE | 1, 5, 9, 11, 13 | | Pivot from one client's app to another on the same host | 12 | | Tamper with a deploy, or ship a compromised dependency | 13 | | Exfiltrate or ransom the database | 6, 10 | | Go undetected long enough to do real damage | 14, 15 | --- ## 1. Operating system baseline ```bash # Keep the OS patched — do this weekly at minimum, and subscribe to security-only auto-updates sudo apt update && sudo apt -y upgrade sudo apt install -y unattended-upgrades apt-listchanges sudo dpkg-reconfigure -plow unattended-upgrades # enable "Unattended-Upgrade::Automatic-Reboot" for kernel updates in the config it opens # Time sync (auth log correlation, TLS validation, JWT exp/iat all depend on correct clocks) sudo apt install -y chrony sudo systemctl enable --now chrony # Minimal package set — every extra service is attack surface sudo apt autoremove --purge -y ``` **Create a non-root deploy user.** Nothing should run or deploy as `root`. ```bash sudo adduser deploy sudo usermod -aG sudo deploy # only if this box needs occasional admin sudo; prefer per-command sudoers rules otherwise ``` **Verify:** `uname -r` and `apt list --upgradable` show a current kernel and package set; `timedatectl` shows `System clock synchronized: yes`. --- ## 2. SSH hardening Edit `/etc/ssh/sshd_config`: ``` PermitRootLogin no PasswordAuthentication no PubkeyAuthentication yes KbdInteractiveAuthentication no X11Forwarding no AllowTcpForwarding no MaxAuthTries 3 LoginGraceTime 20 ClientAliveInterval 300 ClientAliveCountMax 2 # AllowUsers deploy # uncomment and list only the accounts that need SSH ``` ```bash sudo sshd -t # validate syntax before restarting — a typo here locks you out sudo systemctl restart ssh ``` Put each engineer's own public key in `~deploy/.ssh/authorized_keys` (never a shared key), `chmod 600` on the file, `chmod 700` on `.ssh`. Consider moving SSH off port 22 only as a minor noise-reduction measure — it is not a substitute for the above. **Verify:** from a fresh terminal (don't close your current session first), `ssh -o PreferredAuthentications=password deploy@host` must be refused, and `ssh deploy@host` with your key must work. --- ## 3. Firewall + brute-force protection ```bash sudo apt install -y ufw fail2ban sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow OpenSSH # or your custom SSH port sudo ufw allow 80/tcp sudo ufw allow 443/tcp # Do NOT open 5432 (Postgres), 6379 (Redis), 3306 (MySQL) — those stay loopback/private-network only (section 6, 7) sudo ufw enable sudo ufw status verbose ``` `fail2ban` for SSH (enabled by default in most distro packages) and, since this app is behind Apache/Nginx, add a jail for repeated 401/403s so credential-stuffing against `/api/v1/auth/login` gets IP-banned in addition to the app-level rate limits already in the code: ```ini # /etc/fail2ban/jail.local [sshd] enabled = true maxretry = 4 bantime = 1h [nginx-limit-req] # or apache-equivalent if you're on Apache enabled = true ``` **Verify:** `sudo ufw status` shows default-deny with only 22/80/443 open; `nmap -Pn ` from an **external** host confirms nothing else answers. --- ## 4. Web server — Apache / Nginx The repo already ships the app-level rules: - `IMS-Backend-New/.htaccess` (root safety net) + `IMS-Backend-New/public/.htaccess` - `IMS-Backend-New/deployment/nginx-hardening.conf` Server-level responsibilities on top of those (see `deployment/README.md` for the full docroot + verification checklist — do that first, it's the most commonly-missed step and the one that leaks `.git`/`.env` if skipped): ```nginx # One server block per client, docroot MUST end in /public server { listen 443 ssl http2; server_name api..com; root /var/www/html/api__backend/public; include /var/www/html/api__backend/deployment/nginx-hardening.conf; client_max_body_size 55m; # match the largest upload limit in the app (learning materials = 50MB) server_tokens off; # don't advertise the nginx version location ~ \.php$ { include fastcgi_params; fastcgi_pass unix:/run/php/php8.2-fpm.sock; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_hide_header X-Powered-By; fastcgi_read_timeout 300; # payment webhooks / PDF generation can be slow — don't cut them off mid-request } location / { try_files $uri $uri/ /index.php?$query_string; } } server { listen 80; server_name api..com; return 301 https://$host$request_uri; # HTTP -> HTTPS, always } ``` Apache equivalent for the same points: `ServerTokens Prod`, `ServerSignature Off` in `/etc/apache2/conf-available/security.conf`, one `` per client with the docroot from `deployment/README.md`, and a `` that only serves a redirect to HTTPS. **Do this for every client vhost, not just one** — this stack runs multiple client backends per server (see the `DIRECTORIES` array in `.github/workflows/ci-cd.yml`); a hardened template you forget to copy to client #6 leaves client #6 exposed. **Verify per vhost:** ```bash curl -sI https://api..com/ | grep -iE 'server|x-powered-by' # must not reveal versions curl -s -o /dev/null -w '%{http_code}\n' https://api..com/.git/config # 404 (see deployment/README.md section 2 for the full list) curl -s -o /dev/null -w '%{http_code}\n' http://api..com/ # 301 to https ``` --- ## 5. TLS / certificates ```bash sudo apt install -y certbot python3-certbot-nginx # or python3-certbot-apache sudo certbot --nginx -d api..com --agree-tos -m ops@.com --redirect sudo systemctl status certbot.timer # auto-renewal, confirm it's active ``` Do this for the frontend's domain too if it terminates TLS on the same host rather than behind a CDN. Minimum TLS policy (Nginx `ssl_protocols TLSv1.2 TLSv1.3;`, disable TLS 1.0/1.1 and weak ciphers — `certbot`'s default Nginx/Apache config already does this on recent distros, just don't override it with something weaker). **Verify:** run the domain through [SSL Labs](https://www.ssllabs.com/ssltest/) — target grade A, no TLS 1.0/1.1, valid chain, HSTS present (the app already sends `Strict-Transport-Security` — see `SecurityHeaders` middleware — confirm it survives the reverse proxy: `curl -sI https://api..com | grep -i strict`). --- ## 6. PostgreSQL hardening ```bash # postgresql.conf listen_addresses = 'localhost' # never '*' unless the DB is genuinely on a separate host + firewalled to app servers only ``` ```bash # pg_hba.conf — only allow local/loopback, password-authenticated connections local all all peer host all all 127.0.0.1/32 scram-sha-256 host all all ::1/128 scram-sha-256 # no 0.0.0.0/0 entries, ever ``` Least-privilege app user, one per client database (never share credentials across clients — see section 12): ```sql CREATE ROLE api__backend WITH LOGIN PASSWORD ''; GRANT CONNECT ON DATABASE _db TO api__backend; GRANT USAGE, CREATE ON SCHEMA public TO api__backend; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO api__backend; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO api__backend; -- NOT superuser, NOT createrole, NOT createdb, NOT DROP on tables the app never drops ``` ```bash sudo systemctl restart postgresql ``` **Verify:** ```bash # from outside the server — must time out / refuse, never connect psql -h -U postgres -d postgres # on the server, confirm the app role can't do superuser things psql -U api__backend -d _db -c "CREATE ROLE test_escalation SUPERUSER;" # must be permission-denied ``` --- ## 7. Redis hardening (if `CACHE_STORE=redis` / queue driver = redis) ``` # /etc/redis/redis.conf bind 127.0.0.1 -::1 protected-mode yes requirepass rename-command FLUSHALL "" rename-command FLUSHDB "" rename-command CONFIG "" ``` ```bash sudo systemctl restart redis-server ``` Set the matching `REDIS_PASSWORD` in every client's `.env`. The API cache middleware (`CacheApiResponses`) stores per-user response data here for up to an hour — an unauthenticated Redis is a direct read of that. **Verify:** `redis-cli -h ping` from an external host must fail to connect; `redis-cli -a ping` locally must return `PONG`. --- ## 8. File system permissions & secrets Run per client app directory after every fresh deploy setup (already in `deployment/README.md`, repeated here as part of the full server checklist): ```bash cd /var/www/html/api__backend sudo chown -R deploy:www-data . sudo find . -type d -exec chmod 750 {} \; sudo find . -type f -exec chmod 640 {} \; sudo chmod -R 770 storage bootstrap/cache # PHP-FPM (www-data) needs to write here sudo chmod 600 .env sudo chmod 600 storage/app/firebase/firebase-credentials.json 2>/dev/null || true ``` Never commit real secrets to the repo (`.env` is git-ignored; `.env.example` must stay placeholder-only — this is checked in the app audit). Generate a **distinct** `APP_KEY`, `JWT_SECRET`, `JWT_REFRESH_SECRET`, DB password, and Redis password **per client** — a single shared secret across all 8+ client deployments means one leak compromises every client at once. **Verify:** ```bash find /var/www/html/api__backend -perm -o+w -type f # should print nothing (no world-writable files) stat -c '%a' /var/www/html/api__backend/.env # 600 ``` --- ## 9. PHP-FPM hardening `/etc/php/8.2/fpm/php.ini`: ```ini expose_php = Off display_errors = Off display_startup_errors = Off log_errors = On error_log = /var/log/php8.2-fpm-errors.log disable_functions = exec,passthru,shell_exec,system,proc_open,popen,curl_multi_exec,parse_ini_file,show_source allow_url_fopen = Off allow_url_include = Off session.cookie_httponly = On session.cookie_secure = On session.cookie_samesite = Lax upload_max_filesize = 55M post_max_size = 55M ``` Each client's pool in `/etc/php/8.2/fpm/pool.d/.conf` should run as its **own** system user (ties into section 12 — isolation between clients): ```ini [api__backend] user = api__backend group = www-data listen = /run/php/php8.2-fpm-.sock listen.owner = www-data listen.group = www-data ``` ```bash sudo systemctl restart php8.2-fpm ``` **Verify:** `php -i | grep expose_php` → Off; a deliberately-triggered 500 in the app returns the app's generic JSON error (round 3 of the code audit — `shouldRenderJsonWhen` — already enforces this), not a PHP stack trace. --- ## 10. Backups The application already writes DB backups to the **private** disk only (see `DatabaseBackupController`, hardened in the code audit — never `public`/`s3`). Server-level responsibilities on top of that: ```bash # Encrypt at rest and ship off-box — never rely solely on local disk backups 0 2 * * * pg_dump _db | gpg --encrypt -r backups@.com | \ aws s3 cp - s3:////$(date +\%F).sql.gpg --sse ``` - Backup storage credentials must be **separate** from the app's own AWS/S3 credentials, and that bucket must not be writable/deletable by the app server's role (a compromised app server should not be able to destroy its own backups). - Retention: keep at minimum 30 daily + 12 monthly restore points. - **Test the restore, not just the backup**, at least quarterly: `gpg --decrypt backup.sql.gpg | psql ` end to end. **Verify:** confirm a backup from the last 24h exists in the off-box bucket; confirm the app server's IAM role/credentials cannot `s3:DeleteObject` on the backup bucket. --- ## 11. Node.js / Next.js process (frontend) ```bash # Run as a dedicated non-root user, never root sudo adduser --system --group nextapp sudo -u nextapp pm2 start npm --name "ims-frontend" -- start sudo -u nextapp pm2 save pm2 startup systemd -u nextapp --hp /home/nextapp # run the printed command as root once ``` - Terminate TLS in front of Node (Nginx/Apache reverse proxy, or a CDN) — don't expose the Node process directly to the internet. - Keep Node itself patched: `sudo apt install -y nodejs` from NodeSource's current LTS channel, not a stale distro package. - `NEXT_PUBLIC_*` env vars ship to every browser — confirm none of them are secrets (the code audit already checked the current set; re-check this every time a new one is added). - `next.config.js` already sets `poweredByHeader: false` and the `next.security.js` headers — confirm they survive the reverse proxy the same way as section 5: `curl -sI https://app..com | grep -iE 'x-frame-options|content-security-policy'`. **Verify:** `pm2 list` shows the process running as `nextapp`, not `root`; `curl -I http://127.0.0.1:3000` from the server works, but the port is not reachable externally (`ufw` from section 3 only opened 80/443). --- ## 12. Isolation between client deployments This server hosts multiple clients' backends (`.github/workflows/ci-cd.yml`'s `DIRECTORIES` array). A vulnerability or compromise in one client's app must not reach another's data. - **Separate DB credentials, DB, and (ideally) DB user per client** — never a shared Postgres role across `api_gsc_backend`, `api_mcas_backend`, etc. (section 6). - **Separate PHP-FPM pool + system user per client** (section 9) so a file-write vulnerability in one app can't read another app's `.env` via the filesystem. - **Separate `APP_KEY` / `JWT_SECRET` / `JWT_REFRESH_SECRET` per client** (section 8) — a leaked secret from one client must not forge tokens for another. - **Separate S3 prefixes or buckets per client**, with bucket policies scoped accordingly. - Consider containerizing each client (Docker) as the next step up if the client count keeps growing — it makes this isolation the default instead of something to remember per-client. **Verify:** as the `api__backend` PHP-FPM user, attempt to read `/var/www/html/api__backend/.env` — must be denied by filesystem permissions (section 8 + this section's separate system users). --- ## 13. CI/CD & deployment pipeline The code audit already fixed the workflow's `permissions:`, `concurrency:`, and push-only deploy trigger (see `.github/workflows/ci-cd.yml`). Remaining server-side pieces: - **Deploy key scope**: `PROD_SERVER_01_SSH_KEY` in GitHub Secrets should be a key whose `authorized_keys` entry is restricted (`command="/opt/deploy/run.sh",no-port-forwarding,no-X11-forwarding`) rather than a general-purpose login key, if your deploy script's actions can be wrapped that way. - **Least privilege**: the deploy user should only be able to `git reset --hard` inside `/var/www/html/*`, run `composer`/`artisan` there, and restart the app's own queue/PHP-FPM pool — not arbitrary `sudo`. - **Pin GitHub Actions to a commit SHA**, not a floating tag (`@v4`) — a compromised upstream Action otherwise runs with your deploy secrets. Get the current SHA and pin it deliberately: `gh api repos/actions/checkout/commits/v4 --jq .sha`, then use `actions/checkout@ # v4`. Do this for every third-party action in the workflow. - **Secrets never in logs**: audit `.github/workflows/ci-cd.yml`'s `script:` blocks — nothing should `echo` a secret; GitHub masks values that exactly match a registered secret, but a transformed/concatenated secret can leak around that. **Verify:** trigger a deploy and confirm nothing in the Action's log output resembles a credential; confirm the deploy SSH key's `authorized_keys` entry (on the server) is restricted, not a blank general-purpose key. --- ## 14. Logging & monitoring - **Centralize logs off the app servers** (they're the first thing an attacker tampers with). At minimum, ship `storage/logs/laravel.log`, PHP-FPM error logs, Nginx/Apache access+error logs, and `auth.log`/`fail2ban.log` to a separate log host or a managed service (CloudWatch, Datadog, Papertrail, self-hosted Loki — whatever fits your budget). - **Alert on**: repeated 401s on `/auth/login` from one IP (credential stuffing beyond what `fail2ban`/app rate-limits already catch), a spike in 403s from `RoutePermissionMiddleware` (a role probing for access), any `CRITICAL`-level log line — the app already emits these for unconfigured webhook secrets (`FringerWebhookController`, `WhatsAppWebhookController`, `ZoomWebhookController`, `MyFeesPaymentController`) and for the payment amount-mismatch guard (`GeniePaymentController`, `MyFeesPaymentController`) — those `CRITICAL` lines mean a real gap is being hit in production and should page someone, not just sit in a log file. - **Disk space & log rotation**: `logrotate` for Nginx/Apache/PHP-FPM logs; Laravel's own log channel should also rotate (`LOG_CHANNEL=daily` with a sane `LOG_DAILY_DAYS`). - **Uptime + error-rate monitoring** (UptimeRobot, Pingdom, or self-hosted) on every client's public endpoint. **Verify:** kill a service (e.g. `sudo systemctl stop php8.2-fpm` on staging) and confirm an alert actually fires within a few minutes, not just that a dashboard *could* show it. --- ## 15. Ongoing hygiene — put these on a calendar | Cadence | Task | |---|---| | Weekly | `apt update && apt list --upgradable`; review `fail2ban` ban list for patterns | | Weekly | `composer audit` / `npm audit` in CI (already wired, currently `continue-on-error` — triage findings here) | | Monthly | Review server user accounts + SSH `authorized_keys` for anyone who's left the team | | Monthly | Confirm backup restore still works (section 10) | | Quarterly | Run `/code-review ultra` or an equivalent full review against `main` on both repos | | Quarterly | Re-run the SSL Labs check (certs/ciphers drift as browsers/servers update defaults) | | Quarterly | Rotate `JWT_SECRET`/`JWT_REFRESH_SECRET`/DB passwords per client (this logs everyone out — schedule a low-traffic window) | | On any staff departure | Revoke their SSH key, GitHub access, and any shared credentials within 24h | | On any suspected incident | Rotate every secret for the affected client immediately; see section 14 for what to check in logs first | --- ## Quick reference: full post-setup verification Run this against every client vhost after completing the sections above: ```bash BASE=https://api..com echo "-- headers --"; curl -sI "$BASE/" | grep -iE 'server|x-powered-by|strict-transport|x-frame|content-security' echo "-- git/env exposure --" for p in /.git/config /.env /composer.lock /storage/logs/laravel.log; do printf '%s -> ' "$p"; curl -s -o /dev/null -w '%{http_code}\n' "$BASE$p" done echo "-- open ports (run from an external host) --"; nmap -Pn "$BASE" 2>/dev/null || echo "(run nmap from outside)" echo "-- TLS --"; echo | openssl s_client -connect "${BASE#https://}:443" 2>/dev/null | openssl x509 -noout -dates ``` If any of these come back wrong, stop and fix it before considering this server production-ready — everything above it in this document exists to make these checks pass.