Host based Attack Detection using System Calls Jestin Joy (141103) Under the Guidance of Ms Anita John Rajagiri School of Engineering & Technology
April 3, 2012
Jestin Joy (RASET)
Attack Detection
April 3, 2012
1 / 24
Outline 1
Motivation Approaches to Intrusion Detection
Jestin Joy (RASET)
Attack Detection
April 3, 2012
2 / 24
Outline 1
Motivation Approaches to Intrusion Detection
2
System Call
Jestin Joy (RASET)
Attack Detection
April 3, 2012
2 / 24
Outline 1
Motivation Approaches to Intrusion Detection
2
System Call
3
Current Work Kernel Level Rootkits Attacks Host Based Anomaly Detection Our Approach Results & Discussion
Jestin Joy (RASET)
Attack Detection
April 3, 2012
2 / 24
Outline 1
Motivation Approaches to Intrusion Detection
2
System Call
3
Current Work Kernel Level Rootkits Attacks Host Based Anomaly Detection Our Approach Results & Discussion
4
Future Work
Jestin Joy (RASET)
Attack Detection
April 3, 2012
2 / 24
Motivation
Motivation Motivation As systems are connected to internet, they became more vulnerable to attack. These attacks can be handled at both network and host level. Some attacks can be from sources other than network. This calls for better host based intrusion detection[1]. Intrusion Detection Intrusion detection is the process of monitoring the events occurring in a computer system or network and analysing them for signs of possible incident.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
3 / 24
Motivation
Approaches to Intrusion Detection
There are two basic approaches: 1
Misuse Detection : A system using this approach detects intrusion events which follow well-known patterns.
2
Anomaly Detection : A system using this approach looks for deviation from normal behaviour. A system call is a request for service that a program makes to the kernel. The service is generally something that only the kernel has the privilege to do, such as doing I/O. Attackers make use of system calls for initiating the attacks
Jestin Joy (RASET)
Attack Detection
April 3, 2012
4 / 24
System Call
System Call in Linux
Jestin Joy (RASET)
Attack Detection
April 3, 2012
5 / 24
Current Work
Initial work focussed on rootkits. Rootkits Rootkits[2] are a set of software tools used by an attacker to gain unauthorized access into a system, thereby providing him with privilege to access sensitive data, conceal its own existence and allowing him to install other malicious software. Rootkits are the most challenging malware to detect due to their elusive nature. Modern rootkit attacks mainly focus on modifying operating system kernel.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
6 / 24
Current Work
Kernel Level Rootkits
Kernel Level Rootkits
Kernel level rootkit[3] modifies the kernel itself. Kernel being the lowest level of operating system makes it a good choice for the intruder to attack, since an attack on it is very difficult to detect. Also being at the kernel level provides the attacker with complete freedom to access all most all areas of an operating system.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
7 / 24
Current Work
Attacks
Attacks
Figure: System Call Table Modification Jestin Joy (RASET)
Attack Detection
April 3, 2012
8 / 24
Current Work
Attacks
Attacks
System Call Table Modification : Attacker replaces original system call with his own custom version System Call Target modification: Legitimate code in the target address is modified System Call Table Redirection: Attacker replaces the call to System Call Table with his own custom version.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
9 / 24
Current Work
Host Based Anomaly Detection
Detection
Due to its efficiency and easiness attackers increasingly use system calls for attack Current techniques for detection mainly rely on archived copy of system files for detection[4] Need a detection approach that takes less prior information about the state of the system
Jestin Joy (RASET)
Attack Detection
April 3, 2012
10 / 24
Current Work
Our Approach
Our Approach
The proposed technique detects system call redirection attack which is a widely employed kernel level rootkit attack[5]. Our approach doesn’t need prior information about the state of the system to function[5] Detection mechanism uses statistical information from the infected system System call redirection attacks cause anomaly to the host system. Creation of clusters[6] could help to detect the attacks
Jestin Joy (RASET)
Attack Detection
April 3, 2012
11 / 24
Current Work
Our Approach
Our Approach Main Idea When a LKM based attack using new system call occurs, it gets an unusually high memory address as compared to other system calls. For the detection purpose the running kernel should be disassembled. We scan the system call table and the unconditional/conditional jump instructions are analysed and their operands, which are memory addresses are collected. Attackers make use of higher level system calls which are loaded earlier in memory for manipulation. We do clustering to differentiate between normal and malicious system calls
Jestin Joy (RASET)
Attack Detection
April 3, 2012
12 / 24
Current Work
Our Approach
Our Approach
Figure: Data flow diagram
Jestin Joy (RASET)
Attack Detection
April 3, 2012
13 / 24
Current Work
Our Approach
Our Approach
Figure: Data flow diagram
Jestin Joy (RASET)
Attack Detection
April 3, 2012
14 / 24
Current Work
Our Approach
Results & Discussion 2500000 +3.238e9
System Call Distribution
Memory Addresses
2000000 1500000 1000000 500000 00
50
100
150 200 System Calls
250
300
350
Figure: System Call Distribution
Figure represents system call distribution in an uninfected system. (kernel 2.6.38 ) Jestin Joy (RASET)
Attack Detection
April 3, 2012
15 / 24
Current Work
Our Approach
Results & Discussion No 1 2 3 4 5 6 7 8 9 10 11 12
System Call No 102 253 277 278 279 280 281 282 286 287 288 337
System call SOCKETCALL LOOKUP DCOOKIE MQ OPEN MQ UNLNK MQ TIMEDSEND MQ TIMEDRECEIVE MQ NOTIFY MQ GETSETATTR ADD KEY REQUEST KEY KEYCTL RECVMMSG
Address c11f6c23 c1111456 c1119279 c1119117 c1118eac c1118af3 c11186e2 c11185ac c111be4e c111bfb0 c111c22b c11f58aa
Figure shows system calls that fell into false cluster Jestin Joy (RASET)
Attack Detection
April 3, 2012
16 / 24
Current Work
Our Approach
Results & Discussion System Call Distribution
4.2 1e9
Memory Addresses
4.0 3.8 3.6 3.4 3.20
50
100
150 200 System Calls
250
300
350
Figure: System Call Distribution
Figure represents system call distribution in an infected system. (kernel 2.6.38 ) Jestin Joy (RASET)
Attack Detection
April 3, 2012
17 / 24
Current Work
Our Approach
Results & Discussion
False positive rate is 0.0366% Analysis could be done on a machine other than the host system. The presence of natural outliers is a bottleneck In-order to detect other type of system call based methods like, system call target modification, the contents of the system call table need to be disassembled. Disassembling each system call gives us information about different points in the call that are candidates of hooking.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
18 / 24
Current Work
Our Approach
Results & Discussion GDB can be utilized to disassemble our running kernel. # gdb / u s r / s r c / l i n u x − 2 . 6 . 3 8 . 8 / v m l i n u x / p r o c / k c o r e ( gdb ) d i s a s s s y s w r i t e Dump o f a s s e m b l e r code f o r f u n c t i o n s y s w r i t e : 0 xc10d2c83 < s y s w r i t e +0>: push %ebp 0 xc10d2c84 < s y s w r i t e +1>: mov %esp ,% ebp 0 xc10d2c86 < s y s w r i t e +3>: push %e s i 0 xc10d2c87 < s y s w r i t e +4>: mov $ 0 x f f f f f f f 7 ,% e s i 0 x c 1 0 d 2 c 8 c < s y s w r i t e +9>: push %ebx 0 xc10d2c8d < s y s w r i t e +10>: s ub $0x10 ,% e s p 0 xc10d2c90 < s y s w r i t e +13>: mov 0 x8(%ebp ) ,% ea x 0 xc10d2c93 < s y s w r i t e +16>: l e a −0xc(%ebp ) ,% edx 0 xc10d2c96 < s y s w r i t e +19>: c a l l 0 xc10d34bd < f g e t l i g h t 0 xc10d2c9b < s y s w r i t e +24>: t e s t %eax ,% ea x Jestin Joy (RASET)
Attack Detection
April 3, 2012
19 / 24
Current Work
Our Approach
Results & Discussion
The vmlinux file is the uncompressed kernel image The optional /proc/kcore parameter acts as a core file, to let gdb actually peek into [7] the memory of the running kernel. There are jump instructions, which can be abused by an attacker to divert the normal control flow. Attack simulation is a challenge
Jestin Joy (RASET)
Attack Detection
April 3, 2012
20 / 24
Future Work
Future Work
Detection daemon is invoked every 30 seconds, which induces an extra load in the system. Embedded devices are also victims of these types of attacks. Study should be done on them. The effect of detection mechanism on low memory devices should be thoroughly studied. Our prototype has included only system call addresses for the detection purpose. System call sequences could also be studied [8]
Jestin Joy (RASET)
Attack Detection
April 3, 2012
21 / 24
Future Work
References I
S. Freeman, A. Bivens, J. Branch, and B. Szymanski, “Host-based intrusion detection using user signatures,” in Proceedings of the Research Conference. RPI, Troy, NY, 2002. J. G. Levine, J. B. Grizzard, and H. L. Owen, “Detecting and categorizing kernel-level rootkits to aid future detection,” IEEE Security and Privacy, vol. 4, pp. 24–, January 2006. J. Levine, J. Grizzard, and H. Owen, “A methodology to detect and characterize kernel level rootkit exploits involving redirection of the system call table,” in Information Assurance Workshop, 2004. Proceedings. Second IEEE International, pp. 107–125, IEEE, 2005.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
22 / 24
Future Work
References II G. H. Kim and E. H. Spafford, “The design and implementation of tripwire: a file system integrity checker,” in Proceedings of the 2nd ACM Conference on Computer and communications security, CCS ’94, (New York, NY, USA), pp. 18–29, ACM, 1994. J. Joy, A. John, and J. Joy, “Rootkit detection mechanism: A survey,” Advances in Parallel Distributed Computing, pp. 366–374, 2011. J. Joy and A. John, “A host based kernel level rootkit detection mechanism using clustering technique,” in Trends in Computer Science, Engineering and Information Technology (D. Nagamalai, E. Renault, and M. Dhanuskodi, eds.), vol. 204 of Communications in Computer and Information Science, pp. 564–570, Springer Berlin Heidelberg, 2011.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
23 / 24
Future Work
References III
R. Stallman, R. Pesch, S. Shebs, et al., Debugging with GDB. Gnu Press, 2002. S. Forrest, S. Hofmeyr, A. Somayaji, and T. Longstaff, “A sense of self for unix processes,” in Security and Privacy, 1996. Proceedings., 1996 IEEE Symposium on, pp. 120 –128, may 1996.
Jestin Joy (RASET)
Attack Detection
April 3, 2012
24 / 24