Host based Attack Detection using System Calls Jestin Joy (141103) Under the Guidance of Ms Anita John Rajagiri School of Engineering & Technology

April 3, 2012

Jestin Joy (RASET)

Attack Detection

April 3, 2012

1 / 24

Outline 1

Motivation Approaches to Intrusion Detection

Jestin Joy (RASET)

Attack Detection

April 3, 2012

2 / 24

Outline 1

Motivation Approaches to Intrusion Detection

2

System Call

Jestin Joy (RASET)

Attack Detection

April 3, 2012

2 / 24

Outline 1

Motivation Approaches to Intrusion Detection

2

System Call

3

Current Work Kernel Level Rootkits Attacks Host Based Anomaly Detection Our Approach Results & Discussion

Jestin Joy (RASET)

Attack Detection

April 3, 2012

2 / 24

Outline 1

Motivation Approaches to Intrusion Detection

2

System Call

3

Current Work Kernel Level Rootkits Attacks Host Based Anomaly Detection Our Approach Results & Discussion

4

Future Work

Jestin Joy (RASET)

Attack Detection

April 3, 2012

2 / 24

Motivation

Motivation Motivation As systems are connected to internet, they became more vulnerable to attack. These attacks can be handled at both network and host level. Some attacks can be from sources other than network. This calls for better host based intrusion detection[1]. Intrusion Detection Intrusion detection is the process of monitoring the events occurring in a computer system or network and analysing them for signs of possible incident.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

3 / 24

Motivation

Approaches to Intrusion Detection

There are two basic approaches: 1

Misuse Detection : A system using this approach detects intrusion events which follow well-known patterns.

2

Anomaly Detection : A system using this approach looks for deviation from normal behaviour. A system call is a request for service that a program makes to the kernel. The service is generally something that only the kernel has the privilege to do, such as doing I/O. Attackers make use of system calls for initiating the attacks

Jestin Joy (RASET)

Attack Detection

April 3, 2012

4 / 24

System Call

System Call in Linux

Jestin Joy (RASET)

Attack Detection

April 3, 2012

5 / 24

Current Work

Initial work focussed on rootkits. Rootkits Rootkits[2] are a set of software tools used by an attacker to gain unauthorized access into a system, thereby providing him with privilege to access sensitive data, conceal its own existence and allowing him to install other malicious software. Rootkits are the most challenging malware to detect due to their elusive nature. Modern rootkit attacks mainly focus on modifying operating system kernel.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

6 / 24

Current Work

Kernel Level Rootkits

Kernel Level Rootkits

Kernel level rootkit[3] modifies the kernel itself. Kernel being the lowest level of operating system makes it a good choice for the intruder to attack, since an attack on it is very difficult to detect. Also being at the kernel level provides the attacker with complete freedom to access all most all areas of an operating system.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

7 / 24

Current Work

Attacks

Attacks

Figure: System Call Table Modification Jestin Joy (RASET)

Attack Detection

April 3, 2012

8 / 24

Current Work

Attacks

Attacks

System Call Table Modification : Attacker replaces original system call with his own custom version System Call Target modification: Legitimate code in the target address is modified System Call Table Redirection: Attacker replaces the call to System Call Table with his own custom version.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

9 / 24

Current Work

Host Based Anomaly Detection

Detection

Due to its efficiency and easiness attackers increasingly use system calls for attack Current techniques for detection mainly rely on archived copy of system files for detection[4] Need a detection approach that takes less prior information about the state of the system

Jestin Joy (RASET)

Attack Detection

April 3, 2012

10 / 24

Current Work

Our Approach

Our Approach

The proposed technique detects system call redirection attack which is a widely employed kernel level rootkit attack[5]. Our approach doesn’t need prior information about the state of the system to function[5] Detection mechanism uses statistical information from the infected system System call redirection attacks cause anomaly to the host system. Creation of clusters[6] could help to detect the attacks

Jestin Joy (RASET)

Attack Detection

April 3, 2012

11 / 24

Current Work

Our Approach

Our Approach Main Idea When a LKM based attack using new system call occurs, it gets an unusually high memory address as compared to other system calls. For the detection purpose the running kernel should be disassembled. We scan the system call table and the unconditional/conditional jump instructions are analysed and their operands, which are memory addresses are collected. Attackers make use of higher level system calls which are loaded earlier in memory for manipulation. We do clustering to differentiate between normal and malicious system calls

Jestin Joy (RASET)

Attack Detection

April 3, 2012

12 / 24

Current Work

Our Approach

Our Approach

Figure: Data flow diagram

Jestin Joy (RASET)

Attack Detection

April 3, 2012

13 / 24

Current Work

Our Approach

Our Approach

Figure: Data flow diagram

Jestin Joy (RASET)

Attack Detection

April 3, 2012

14 / 24

Current Work

Our Approach

Results & Discussion 2500000 +3.238e9

System Call Distribution

Memory Addresses

2000000 1500000 1000000 500000 00

50

100

150 200 System Calls

250

300

350

Figure: System Call Distribution

Figure represents system call distribution in an uninfected system. (kernel 2.6.38 ) Jestin Joy (RASET)

Attack Detection

April 3, 2012

15 / 24

Current Work

Our Approach

Results & Discussion No 1 2 3 4 5 6 7 8 9 10 11 12

System Call No 102 253 277 278 279 280 281 282 286 287 288 337

System call SOCKETCALL LOOKUP DCOOKIE MQ OPEN MQ UNLNK MQ TIMEDSEND MQ TIMEDRECEIVE MQ NOTIFY MQ GETSETATTR ADD KEY REQUEST KEY KEYCTL RECVMMSG

Address c11f6c23 c1111456 c1119279 c1119117 c1118eac c1118af3 c11186e2 c11185ac c111be4e c111bfb0 c111c22b c11f58aa

Figure shows system calls that fell into false cluster Jestin Joy (RASET)

Attack Detection

April 3, 2012

16 / 24

Current Work

Our Approach

Results & Discussion System Call Distribution

4.2 1e9

Memory Addresses

4.0 3.8 3.6 3.4 3.20

50

100

150 200 System Calls

250

300

350

Figure: System Call Distribution

Figure represents system call distribution in an infected system. (kernel 2.6.38 ) Jestin Joy (RASET)

Attack Detection

April 3, 2012

17 / 24

Current Work

Our Approach

Results & Discussion

False positive rate is 0.0366% Analysis could be done on a machine other than the host system. The presence of natural outliers is a bottleneck In-order to detect other type of system call based methods like, system call target modification, the contents of the system call table need to be disassembled. Disassembling each system call gives us information about different points in the call that are candidates of hooking.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

18 / 24

Current Work

Our Approach

Results & Discussion GDB can be utilized to disassemble our running kernel. # gdb / u s r / s r c / l i n u x − 2 . 6 . 3 8 . 8 / v m l i n u x / p r o c / k c o r e ( gdb ) d i s a s s s y s w r i t e Dump o f a s s e m b l e r code f o r f u n c t i o n s y s w r i t e : 0 xc10d2c83 < s y s w r i t e +0>: push %ebp 0 xc10d2c84 < s y s w r i t e +1>: mov %esp ,% ebp 0 xc10d2c86 < s y s w r i t e +3>: push %e s i 0 xc10d2c87 < s y s w r i t e +4>: mov $ 0 x f f f f f f f 7 ,% e s i 0 x c 1 0 d 2 c 8 c < s y s w r i t e +9>: push %ebx 0 xc10d2c8d < s y s w r i t e +10>: s ub $0x10 ,% e s p 0 xc10d2c90 < s y s w r i t e +13>: mov 0 x8(%ebp ) ,% ea x 0 xc10d2c93 < s y s w r i t e +16>: l e a −0xc(%ebp ) ,% edx 0 xc10d2c96 < s y s w r i t e +19>: c a l l 0 xc10d34bd < f g e t l i g h t 0 xc10d2c9b < s y s w r i t e +24>: t e s t %eax ,% ea x Jestin Joy (RASET)

Attack Detection

April 3, 2012

19 / 24

Current Work

Our Approach

Results & Discussion

The vmlinux file is the uncompressed kernel image The optional /proc/kcore parameter acts as a core file, to let gdb actually peek into [7] the memory of the running kernel. There are jump instructions, which can be abused by an attacker to divert the normal control flow. Attack simulation is a challenge

Jestin Joy (RASET)

Attack Detection

April 3, 2012

20 / 24

Future Work

Future Work

Detection daemon is invoked every 30 seconds, which induces an extra load in the system. Embedded devices are also victims of these types of attacks. Study should be done on them. The effect of detection mechanism on low memory devices should be thoroughly studied. Our prototype has included only system call addresses for the detection purpose. System call sequences could also be studied [8]

Jestin Joy (RASET)

Attack Detection

April 3, 2012

21 / 24

Future Work

References I

S. Freeman, A. Bivens, J. Branch, and B. Szymanski, “Host-based intrusion detection using user signatures,” in Proceedings of the Research Conference. RPI, Troy, NY, 2002. J. G. Levine, J. B. Grizzard, and H. L. Owen, “Detecting and categorizing kernel-level rootkits to aid future detection,” IEEE Security and Privacy, vol. 4, pp. 24–, January 2006. J. Levine, J. Grizzard, and H. Owen, “A methodology to detect and characterize kernel level rootkit exploits involving redirection of the system call table,” in Information Assurance Workshop, 2004. Proceedings. Second IEEE International, pp. 107–125, IEEE, 2005.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

22 / 24

Future Work

References II G. H. Kim and E. H. Spafford, “The design and implementation of tripwire: a file system integrity checker,” in Proceedings of the 2nd ACM Conference on Computer and communications security, CCS ’94, (New York, NY, USA), pp. 18–29, ACM, 1994. J. Joy, A. John, and J. Joy, “Rootkit detection mechanism: A survey,” Advances in Parallel Distributed Computing, pp. 366–374, 2011. J. Joy and A. John, “A host based kernel level rootkit detection mechanism using clustering technique,” in Trends in Computer Science, Engineering and Information Technology (D. Nagamalai, E. Renault, and M. Dhanuskodi, eds.), vol. 204 of Communications in Computer and Information Science, pp. 564–570, Springer Berlin Heidelberg, 2011.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

23 / 24

Future Work

References III

R. Stallman, R. Pesch, S. Shebs, et al., Debugging with GDB. Gnu Press, 2002. S. Forrest, S. Hofmeyr, A. Somayaji, and T. Longstaff, “A sense of self for unix processes,” in Security and Privacy, 1996. Proceedings., 1996 IEEE Symposium on, pp. 120 –128, may 1996.

Jestin Joy (RASET)

Attack Detection

April 3, 2012

24 / 24