Linux/Unix filesystem is not reflecting the change of a deleted file on disk?
Have you ever deleted a large file or files successfully using the rm command but didn’t see any change in disk usages of the filesystem? It happened to me last week when I received a PagerDuty alert for high disk usages on one of our production servers. To resolve the issue quickly, I deleted few old large log files using the rm command and immediately checked the disk usages of the filesystem using the df -h command. Gotcha!! No change in disk usages of the filesystem.
This made me curious to look into the /opt folder and check what is occupying all the spaces. Then I ran du -sh /opt to see the actual disk usages on the mounted folder /opt. Again Gotcha!! Disk usages of /opt folder was just 124GB out of 493GB.
On further investigation, I found that there is a list of deleted files (not present in the directory tree) but still held open by the running applications.
Resolution:-
Graceful shutdown of the relevant process: Use the lsof +L1 to get the list of the deleted files which are still held by the process and do a graceful shutdown of the process.
Truncate file size: Alternatively, it is possible to force the system to de-allocate the space consumed by an in-use file via the proc file system. This is an advanced technique and should only be carried out when you are certain that this will cause no adverse effects to running processes.
$ echo > /proc/pid/fd/fd_number
Root cause:-
On Linux/Unix filesystem, filenames are just pointers (inodes) that point to the memory where the file resides which could be a hard drive or even a RAM-backed filesystem. Each file records the number of links to it — the links can be either the filename (plural, if there are multiple hard links to the same file) and also every time a file is opened, the process actually holds the “link” to the same space.
So whenever you use the rm command to delete a file it actually unlinks the file from the filesystem’s directory structure. Space/memory is physically freed only if there are no links left to the deleted file (therefore, it’s impossible to get to it). So, while the processes have the files still opened, you shouldn’t expect to get space/memory back. It’s not freed, it’s being actively used. This is also one of the reasons that applications should really close the files when they finish using them.
