How to delete tone of files easily and quickly on Unix

So today, I no discovered my root partition was almost filled up. But it wasn't because of some huge file or directory, but very small files that were being created on the system -- millions of them. As a result, my disk space was just 39% but inode(meta data about regular files) was showing 87%. I needed to quickly free-up space by deleting several million files.

/dev/ada0p2 4.8G 1.7G 2.7G 39% 625k 97k 87% /

Unfortunately rm -r dir/* wasn't working. It was complaining that "Argument list too long.". This usually happens when you run a command on several thousands of files. Even ls l | grep pattern will bring the same error. Apparently it's because there is a relatively small buffer of memory allocated to storing this list of arguments and if it is filled up, the shell will not execute the program.

So I had to search for an alternative and safe way of doing it.

My first solution is to write a one-liner for loop. I do this a lot times when working with several files or directory. So, I run command that searches for specific files I wish to delete, then run results through a for loop and then delete result in each iteration.

for i in $(ls /root/ | grep "cronscheduled.php"); do rm -v $i; done;

The second and probably the best is to use the inglorious find command. Yes, the find command is really one robust command that you want to use each time you are working with files for basically one reason; you can pipe results from the find command to another set of commands.

So again we find our files based on a certain pattern, then pipe the output to another command -- in this case rm which deletes our files. You could do something like;

find /root -iname 'crontopupnow.php*' -type f -exec rm -v {} \;.

The problem is that this takes a bit of more time. It's slow. Fortunately, the find command has a convenient -delete option which achieves the same and more faster. -print option just makes it verbose so we see what the command is actually doing.

find /root -iname 'crontopupnow.php*' -type f -print -delete.

Finally, you could use the famous rsync. I know rsync is majorly used for copying files or syncing directories, but we can use it to delete files too. That's only if the files we want to delete are all in the same directory. If they aren't or if you have files you don't want to delete in the same folder as those you want to, then don't use this approach.

So the trick is to create an empty directory and then sync your empty directory with the one with files you want to delete.

mkdir empty_dir. rsync -a --delete empty_dir/ yourdirectory/

Be very very careful whenever you are deleting files or directory because such operations can end up with disastrous results when done in a rush. Always test out the output of the command with things like -print or -v or --dry-run which print out results of the command. I usually also like to add sleep 2 especially in a while/for loops so at least I have 2 seconds to see what's happening before I remove the sleep.

David Okwii

David Okwii is a Systems Engineer who currently works with Uganda's country code top-level domain registry.

Kampala Uganda http://www.davidokwii.com

Subscribe to oquidave@geek:~ #

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!