Sometimes your server might run out of space, necessary for its ability to serve clients. Here comes a typical situation: you have too many files, and your server has no space to write temporary files. This way you cannot even run Putty to determine the problem (if your server is remote). Here is a brief and clear guide I’ve been searching for some hours before I fixed the issue.
First of all, let’s see how the error message looks like:
This message was received using KVM. I have already told that server was inaccessible via SSH, though httpd, named and other daemons were working as usual.
The first step you need is to reboot your server in rescue mode. My dedicated server provider has an option in its panel. If you can gain physical access to the server, it means you should boot it from CD or DVD. Your server will not start properly if there is no space for temporary files, so it’s the best solution to boot it from another source.
After your server has been placed to rescue mode, you should first check its partitions for errors. You can do it using the following command:
e2fsck -D -f /dev/hdXY
Where hdXY is the name of your partition. This process can take a lot of time, so be patient, especially if you have many files. After you ensure your partition is ok, it’s time to check what causes the problem. Let’s mount our partition:
mkdir /mnt/problem; mount /dev/hdXY /mnt/problem
Now your partition files are accessible through the folder /mnt/problem. Next action you need to perform is to find our the number of free inodes (most probably your server runs out of free inodes, and not of the space itself). Here is how:
You should see something like this:
Note, this is a screenshot that is done after the problem is fixed. Most probably your free inode percentage will be about 0-1%. That’s extremely low value. You can have about 50% free space shown by df:
That’s the screenshot from the same server. As you can see, there are 48% of free space in blocks, and just 6% of free inodes. What does it mean and how can it be?
The reason of this phenomenon is that some directories on your server have lots of small or even zero-byte files. Due to ext3 filesystem properties, this way of file placement is taking many free inodes, making your server space-free.
Now we need to identify, which folders consume so many inodes. First of all, go and clear your /tmp directory.
rm -rf tmp/
You can recreate the /tmp after it’s deleted, of use mc to delete all the files in it. Next we need to know where are those lots of files:
for i in /*; do echo $i; find $i |wc -l; done
This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.
for i in /home/*; do echo $i; find $i |wc -l; done
You should replace home with the directory where you consider that little files are placed. Once you found the directory causing problems, you should delete it:
rm -rf /home/bad_user/directory_with_lots_of_empty_files
I prefer doing that with mc (if there are too many files, it performs faster than rm command). Repeat this procedure until you have at least 4-5% of free inodes (you can check that by using df -i command at any time).
After that try to reboot your server in usual mode. It should now start without error messages.
In order to ensure you won’t have problems in the future, I’d recommend you to disable smartd disk monitoring if you don’t use it:
/sbin/service smartd stop /sbin/chkconfig --del smartd
Now your server should be up and running. I recommend you to have at least 10% of free inodes at any time. If you’re dealing with a large number of temporary files, you should set up some cronjobs to delete them in time.
Feel free to ask and share your ideas. If you have faced another “Directory index full” error and know how to fix it, please, let me know.