Truncate log files without a shutdown

Anybody who handles Java applications for a living will have encountered that log from hell that keeps on growing and does not automatically roll over to a sane size. Cleaning that log is going to require you to shutdown the application since deleting it will not work.

To understand why the delete will not work you need to understand that while you might delete the inode to the file, the file descriptor to that file will still be in use. Meaning you can no longer see the file, but still see it fill up your disks and continue to grow

What I’ve learned to do is execute the following for older distros like CentOS 5 to truncate log files:

echo “” > /log/file

This will basically change the file contents to an empty string, removing all the current data in the file.
Advantage is that all filedescriptors will remain open and the file will be continue to be written to without interrupting the owning process.

Modern systems also include the truncate command, which does something similar, but much better.

So far this little trick has helped me a lot, but according to a discussion on StackOverFlow it should not.
Anybody want to clue me in why it works for me, but not everbody? Or chime in with your own experiences 🙂

Verify your package integerity with yum-verify

If you’ve been following the news you may have heard of a new backdoor that replaces your Apache binary.
While it’s a great idea to install something like Tripwire or CSF to let you know when a binary changes, ad hoc queries using yum are also possible.

A short and to the point example..

For this example we are going to “infect” the GCC binary as if we are creating a backdoored compiler

Step 1: Install yum verify

yum install yum-verify -y

Step 2: Let’s do some damage..

echo “.” >> /usr/bin/gcc

Step 3: Running and understanding yum verify-rpm

You can invoke yum verify  to check the package integerity

yum verify-rpm gcc
Loaded plugins: fastestmirror, presto, priorities, verify
==================== Installed Packages ====================
gcc.x86_64 : Various compilers (C, C++, Objective-C, Java, …)
File: /usr/bin/gcc
Problem: checksum does not match
Current: sha256:c2ddfb51a2e631e6176d550c035ab9484043a4be6732cb405a92bad3ecfd1e35
Original: sha256:173b459de91d3d08fb04ec72eafac6a3fad87c5a75bdfb381e071e48e8c16fa8
——–
Problem: size does not match
Current: 263890 B
Original: 263888 B
——–
Problem: mtime does not match
Current: Thu May 9 16:05:48 2013 (77 days, 0:29:20 later)
Original: Thu Feb 21 15:36:28 2013
File: /usr/bin/x86_64-redhat-linux-gcc
Problem: checksum does not match
Current: sha256:c2ddfb51a2e631e6176d550c035ab9484043a4be6732cb405a92bad3ecfd1e35
Original: sha256:173b459de91d3d08fb04ec72eafac6a3fad87c5a75bdfb381e071e48e8c16fa8
——–
Problem: size does not match
Current: 263890 B
Original: 263888 B
——–
Problem: mtime does not match
Current: Thu May 9 16:05:48 2013 (77 days, 0:29:20 later)
Original: Thu Feb 21 15:36:28 2013
verify-rpm done

Well there is your problem. It would seem the binary no longer matches the original hash. Please note that this tool will also flag configuration changes that you made to the original configs after install.

Step 4: Remediate the problem by reinstalling the old package

A reinstall is simple:

yum reinstall gcc
Loaded plugins: fastestmirror, presto, priorities, verify
Setting up Reinstall Process
Loading mirror speeds from cached hostfile
* base: www.fedora.is
* extras: www.fedora.is
* updates: www.fedora.is
Resolving Dependencies
–> Running transaction check
—> Package gcc.x86_64 0:4.4.7-3.el6 will be reinstalled
–> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================
Reinstalling:
gcc x86_64 4.4.7-3.el6 base 10 M

Transaction Summary
==============================================================================================================================
Reinstall 1 Package(s)

Total download size: 10 M
Installed size: 19 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 10 M
gcc-4.4.7-3.el6.x86_64.rpm | 10 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : gcc-4.4.7-3.el6.x86_64 1/1
Verifying : gcc-4.4.7-3.el6.x86_64 1/1

Installed:
gcc.x86_64 0:4.4.7-3.el6

Complete!

 

Step 5: Verify that the package is now correct

yum verify-rpm gcc
Loaded plugins: fastestmirror, presto, priorities, verify
verify-rpm done

No output means that the packages are now fine.