Preventing fork bombs in Ubuntu

I think a sign of security-mindedness for a server GNU/Linux distribution is whether it can withstand simple attacks out of the box. At the moment Ubuntu fails this test as a simple fork bomb from any user or any compromised service can render the system useless.

It doesn’t get much simpler than running the following command from a bash prompt:

• :(){ :|:;};:

On an unprotected system this will cause the system to become unresponsive. To determine the vulnerability of a system to this attack run:

• ulimit -u

This shows the maximum number of processes allowed for any 1 user. In Ubuntu this is wisely not set to be “unlimited”. Unfortunately my tests show that it is set at an unreasonably high number. On a Xen domain with 128MB of ram this is set to “1088” which is way too high for a machine with such limited resources. To find the rest of the user limits run:

• ulimit -a

Here are the defauls for a machine with 128MB of ram:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1088
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1088
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

On machines with more RAM the “max user processes” is raised. I’d say that at each step it’s 5-10 times higher than I would set it for a secure system. I suspect the discrepancy only increases with RAM.

It’s also important to pay attention to the “unlimited” settings. These represent other potential denial of service attack vectors. Even an arbitrarily high limit might be better than no limit.

To address these issues I added the following lines to /etc/security/limits.conf:

# prevent core dumps
*       hard    core    0

# limit user processes per user to 150
*       soft    nproc   100
*       hard    nproc   150

# limit memory
*       hard    data    [Size of RAM in KB]
*       hard    rss     [Size of RAM in KB]
*       hard    as      [(Size of RAM x 2) in KB]

# limit max file size to 10GB
*       soft    fsize   1048576
*       hard    fsize   10485760

The reason system defaults are so difficult is that systems may be quite different in their needs. Most of these values are experimental and so I would not recommend using them blindly.

• nproc - 150 seems reasonable and stops the forkbomb from working though this may need to go higher in certain circumstances.
• data/rss - These are currently ignored so I just set them to max physical RAM.
• as - I haven’t seen many recommendations on this as it tends to be so system specific. I think twice the physical RAM should be good for now.
• fsize - A hard limit of 10GB seems reasonable. The soft limit may need to be raised if programs handling files larger than 1GB cannot do it automatically.
• cpu - Trying to limit users by their CPU time seems problematic for system services. I think this could work by assigning limits to specific users or groups but not as a default.
• locks - I also neglected setting this one since I could not find any recommendations on what a reasonable limit would be.

Here is what the ulimit -a command produces on a 128MB system after these changes:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 131072
scheduling priority             (-e) 0
file size               (blocks, -f) 1048576
pending signals                 (-i) 1088
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) 131072
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100
virtual memory          (kbytes, -v) 262144
file locks                      (-x) unlimited