New Run-time Heuristics for Effective Shellcode Detection

Nowadays, code injection attack is one of the most common types of attacks. In every code injection attack there is a payload, called shellcode. So enhancing the accuracy ofintrusion detection systems by instrumentation of their shellcode detection ability, leads to detection of more classes of codeinjection attacks. One of the best approaches to detect code injection attacks is the emulation-based approach in which theinput stream is executed and compared with multiple maliciousbehaviours of shellcodes. Most of the existing code injection attack detection systems that are based on payload execution,concentrate on detecting polymorphic shellcodes. Thus, detection of plain shellcodes is an important issue since there isno self-decrypting behaviour in such shellcodes. One of the recent proposed systems can detect four classes of plainshellcodes by using some heuristics derived from the behaviour of shellcode during the execution. In this paper we have proposed new behaviours of shellcodes which none of them canbe detected by existing systems. Also we have designed appropriate run-time heuristics by which we can detect theproposed shellcodes. The experimental result shows the highaccuracy of the proposed detection system in contrast to the existing one.

کلیدواژه ها:

Code injection ، Shellcode ، Egg-hunter ، Emulation