Posted Thursday, December 10, 2015
If you’re seeing errors on a server (say, npm complaining about an ENOSPC error, or apt failing to install something) that suggest the hard disk is full, use this guide to resolve the issue.
All of the following commands assume you are logged in as a sudo user on a Linux machine. Some of the commands, like
df -hT, won’t work exactly as advertised on something like, say, Solaris.
First, check the amount of available disk space.
If one of the available drives (such as
/dev/xvda1) shows as 100% full, then you’ll need to clear some space. The first place to look is
/var/log/, especially if nginx is writing log files to disk. Clear out all old logs (anything with a .gz file extension is fair game for deletion). If that doesn’t free up enough space, the next place to check is the application directory for whatever app is on the server. If there are more than 3 releases on the server, you can safely delete the oldest ones (until you have at least 2 remaining). You can also remove all the files in
/tmp/, although that should be a last resort.
If you’ve confirmed that the hard drive isn’t full (or cleared enough space that it isn’t), and you’re still seeing “disk is full” type errors, then you almost certainly have too few inodes available on the server. This means, basically, that there are too many files on the server.
Run the following command to check inode usage:
You’ll see something like the following:
Filesystem Inodes IUsed IFree IUse% Mounted on udev 121K 409 120K 1% /dev tmpfs 125K 479 124K 1% /run /dev/vda1 1.9M 237K 1.7M 13% / tmpfs 125K 2 125K 1% /dev/shm tmpfs 125K 3 125K 1% /run/lock tmpfs 125K 15 125K 1% /sys/fs/cgroup tmpfs 125K 4 125K 1% /run/user/1000
IUse% column is the one you’re interested in. If
IUse% is at or near 100%, then you have an inode usage problem.
Often, you can free up a ton of inodes if you delete old kernels. This is done with a pretty simple command:
sudo apt-get autoremove -y
Try that first. If that doesn’t remove the old kernels, this page may help.
This script will help find directories with large numbers of files. Put the following into a file
list-files.sh in your home directory:
#!/bin/bash # count_em - count files in all subdirectories under current directory. echo 'echo $(ls -a "$1" | wc -l) $1' >/tmp/count_em_$ chmod 700 /tmp/count_em_$ find . -mount -type d -print0 | xargs -0 -n1 /tmp/count_em_$ | sort -n
Then make the file executable:
chmod +x ./list-files.sh
Finally, go to the root directory on the server and run the script:
cd / ~/list-files.sh
It may take a couple minutes to run, but it will present you with a list of all the directories with a number of files next to each, sorted in ascending order of number of files. Take a look at each of the high ones; that might give you an idea of what you need to clean up.
Log files are safe to delete. So are old releases. Also check the
/tmp/ directory; it can often be a hiding place for tons of files.