IEEE Transactions on Dependable and Secure Computing (TDSC); Volume 8, Number 5, pages 670--684; September/October 2011.
Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify non-control data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits.
This article presents a novel technique to detect rootkits that modify both control and non-control data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously-known rootkits, including those that modify non-control data structures.
Code: [ Gibraltar on Xen ]
DOI: [ 10.1109/TDSC.2010.38 ]
Conference version: This article is a revised and expanded version of Automatic Inference and Enforcement of Kernel Data Structure Invariants, which appeared in Proceedings of ACSAC 2008, the 24th Annual Computer Security Applications Conference, December 2008.