Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
<!-- BANNER_END -->

<!-- telemetry-svg-start -->
![Workflow Timeline](workflow_status.svg)
<!-- telemetry-svg-end -->
# BrokenOps

Expand Down
12 changes: 10 additions & 2 deletions labs/nginx-broken/cloud-init.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
#cloud-config
packages:
- nginx
- policycoreutils
- selinux-utils
- policycoreutils-python-utils
- auditd

runcmd:
# Intentional break: configure nginx to listen on invalid port to break it
- sed -i 's/listen 80 default_server;/listen 80808 default_server;/' /etc/nginx/sites-available/default
# Intentional break: move nginx to 8080, but do not allow SELinux http_port_t on 8080.
- sed -i 's/listen 80 default_server;/listen 8080 default_server;/' /etc/nginx/sites-available/default
- sed -i 's/listen \[::\]:80 default_server;/listen [::]:8080 default_server;/' /etc/nginx/sites-available/default
- systemctl enable --now auditd || true
- setenforce 1 || true
- semanage port -d -t http_port_t -p tcp 8080 || true
- systemctl restart nginx || true
16 changes: 8 additions & 8 deletions labs/nginx-broken/lab.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
id: nginx-broken

name: Nginx Service Down
name: SELinux Denies Nginx on Custom Port

category: linux
difficulty: beginner
Expand All @@ -17,18 +17,18 @@ vm:
cloud_init: cloud-init.yaml
verify_script: verify.sh
exposed_ports:
- 80
- 8080

verify: verify.sh

description:
summary: "Nginx is not running after reboot"
story: "A server reboot caused nginx to stop."
summary: "Nginx runs but app is unreachable on port 8080"
story: "Nginx was moved to port 8080, but SELinux policy was never updated."
objectives:
- Start nginx
- Enable service on boot
- Confirm HTTP response
- Investigate SELinux denials for nginx
- Map TCP port 8080 to http_port_t
- Confirm HTTP response on port 8080

tags:
- systemd
- selinux
- nginx
14 changes: 10 additions & 4 deletions labs/nginx-broken/question.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
### Scenario

You have just joined a new company as a junior sysadmin. A developer frantically messages you:
> "Hey! The main Nginx web server is completely down on our internal staging box. The service won't start at all after I accidentally rebooted it! I need to test my new frontend right now, can you please fix it?"
Nginx was reconfigured to use port **8080** on a host where SELinux is expected to remain enforcing.
The service appears configured, but the app is unreachable.

### Objective

Investigate why the `nginx` service is failing to start on this machine, fix the root cause, and ensure the service is running.
Once running, the web server should be accessible via the **"Open Port 80"** button in the top navigation bar. When clicked, it should display the default Nginx welcome page.
Find and fix the SELinux policy issue preventing HTTP service on the custom port while keeping SELinux enforcing.

When solved:
- Nginx should run on **port 8080**
- SELinux should allow HTTP traffic on 8080 (`http_port_t` mapping)
- The **"Open Port 8080"** button should load the Nginx default page

### Useful Commands
- `systemctl status nginx`
- `journalctl -xeu nginx.service`
- `ausearch -m avc -ts recent`
- `semanage port -l | grep http_port_t`
42 changes: 20 additions & 22 deletions labs/nginx-broken/solution.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,40 @@
### The Issue

The developer or someone else accidentally configured Nginx to listen on a port that it doesn't have permission to bind to, or they made a syntax error in the configuration file!
Nginx was moved to TCP port `8080`, but SELinux policy did not allow HTTP services on that port.
So nginx fails to bind when SELinux is enforcing.

### Step-by-Step Fix

1. **Check the service status to get a hint:**
1. **Check nginx status and logs:**
```bash
systemctl status nginx
journalctl -xeu nginx.service
```
You should see it failing.

2. **Test the Nginx configuration for syntax errors:**
2. **Check SELinux denials:**
```bash
sudo nginx -t
ausearch -m avc -ts recent
```
This will pinpoint exactly which file and line number contains the bad configuration. You'll likely see an error pointing to `/etc/nginx/sites-available/default` trying to listen on port `80808` or a typo like `lissten 80`.

3. **Fix the configuration file:**
Open the file with your favorite editor:
3. **Confirm the custom port is not mapped to `http_port_t`:**
```bash
sudo nano /etc/nginx/sites-available/default
```
Find the incorrect line and fix it back to listening on a standard port like `80`:
```nginx
server {
listen 80 default_server;
listen [::]:80 default_server;
# ...
}
semanage port -l | grep http_port_t
```

4. **Restart the Nginx service:**
4. **Map TCP 8080 for HTTP services:**
```bash
sudo systemctl restart nginx
sudo semanage port -a -t http_port_t -p tcp 8080
```
If it already exists with a different type, modify it instead:
```bash
sudo semanage port -m -t http_port_t -p tcp 8080
```

5. **Verify it's running:**
5. **Restart nginx and verify:**
```bash
systemctl status nginx
sudo systemctl restart nginx
ss -tuln | grep ':8080 '
curl -I http://127.0.0.1:8080
```
It should now say **active (running)**. You've solved the lab!

You have solved the lab once nginx is reachable on port `8080` with SELinux still enforcing.
13 changes: 13 additions & 0 deletions labs/nginx-broken/solution.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash
set -e

# Keep nginx on custom port 8080
sed -i 's/listen 80 default_server;/listen 8080 default_server;/' /etc/nginx/sites-available/default || true
sed -i 's/listen \[::\]:80 default_server;/listen [::]:8080 default_server;/' /etc/nginx/sites-available/default || true

# Ensure SELinux allows HTTP traffic on tcp/8080
if ! semanage port -l | awk '/^http_port_t/ && /(^|[^0-9])8080([^0-9]|$)/ {found=1} END {exit found?0:1}'; then
semanage port -a -t http_port_t -p tcp 8080 || semanage port -m -t http_port_t -p tcp 8080
fi

systemctl restart nginx
24 changes: 19 additions & 5 deletions labs/nginx-broken/verify.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,30 @@
# Check if nginx service is active
systemctl is-active --quiet nginx
if [ $? -ne 0 ]; then
echo "FAILURE: The nginx service is not running."
echo "FAILURE: nginx service is not running."
exit 1
fi

# Check if it's actually listening on port 80
ss -tuln | grep ":80 " > /dev/null
# Check if nginx is listening on port 8080
ss -tuln | grep ':8080 ' > /dev/null
if [ $? -ne 0 ]; then
echo "FAILURE: Nginx is running, but it's not listening on port 80!"
echo "FAILURE: nginx is not listening on port 8080."
exit 1
fi

echo "SUCCESS: Nginx is running and listening on port 80!"
# Check SELinux port context mapping for HTTP on 8080
semanage port -l | awk '/^http_port_t/ && /(^|[^0-9])8080([^0-9]|$)/ {found=1} END {exit found?0:1}'
if [ $? -ne 0 ]; then
echo "FAILURE: SELinux is not allowing HTTP service on tcp/8080 (http_port_t mapping missing)."
exit 1
fi

# Check HTTP response over localhost
curl -sSf http://127.0.0.1:8080 > /dev/null
if [ $? -ne 0 ]; then
echo "FAILURE: nginx is running on 8080 but HTTP request failed."
exit 1
fi

echo "SUCCESS: nginx is serving on 8080 and SELinux http_port_t mapping is set for tcp/8080."
exit 0
50 changes: 33 additions & 17 deletions workflow_status.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.