19th-20th November 2011 - Melbourne, Australia
Static string signatures in Antivirus don't effectively fingerprint unknown malware variants. One approach which has seen some success is using the structural information of a program's control flow to build a signature. The control flow describes the possible flow of execution a program may take. It is represented by what's known as a directed graph - basically a network diagram of how execution moves from one set of instructions to another. Control flow doesn't change much in variants even if the byte level content changes like in polymorphic and metamorphic malware. A real advantage of using graphs is that we can compare these graphs to show if they are approximately similar. We can quantify how similar two programs are and set a threshold to identify related or mutated malware. I have implemented a system using these ideas to perform malware detection in real-time.
The system improves previous work by performing more efficiently and detecting more variants. It replaces the classification system that I presented at Ruxcon 2010 and uses several new ideas that make it better. This presentation discusses how the system works, its implementation, and its evaluation.
Silvio Cesare is a PhD student at Deakin University. His research interests include Malware detection and automated vulnerability discovery using static analysis of executable binaries. He has previously spoken at conferences including Blackhat, Cansecwest, Ruxcon, and academic outlets.
Silvio spoke at the first Ruxcon in 2003 on open source kernel vulnerabilities. He has worked in industry including time as the scanner architect of the vulnerability management company, Qualys.