CSCI 8730: Advanced Operating System and System Security

Course Objection

Keep graduate students updated and involved in the latest research topics and projects in the field of operating system and system security.

Course Description

This is the 8000 level course on operating systems. Topics include system software testing, kernel internals, kernel module debugging and development, isolations and protections in virtual machines.

General Information

Web Page:

Instructor:   Kang Li (kangli AT (421 Boyd GSRC)


Grading Policy (tentative)

The class grades will be based on the following components:
    40% Class Projects  
    45% Demo/Presentation  
    15% Class Participation  

In-Class Presentation/Demo Guidelines

Please don't put too much details in slides. Audience usually won't be able to absorb complex information in a short time. Make sure you cover the important points and essential information. Conclusions and broad flows are often more useful to audience than the details of every fields of a protocol. If appropriate, graphs and charts are better than text. You can borrow figures from the original paper, its presentations or if necessary, create it by your own. Make sure they are readable.

Suggestion for Class Participation

Since the class is about the on-going research topics, active student participations are highly encouraged and are important for the learning process of all students in this class. Students can contribute to class participation in many ways. These include participating the in-class discussion of the state-of-the-art, helping to organize competitions, refining project details, and actively searching recent research work that are related to the topics for class discussion.


Temporary Course Topics  (more paper assignment will be given as we progress through this semester)




Review of System Security

(threat modeling,  control flow hijacking,

buffer overflow attack and defense,  privilege escalation)

 We will use the following online resources for a quick review of these subjects


Installing a Linux VM

Fuzzing and SW Testing (1)

1. B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32–44, 1990.

1. P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 1 volume 40, pages 213–223. ACM, 2005.

1. N. Nethercote and J. Seward. Valgrind: a framework for heavyweight dynamic binary instrumentation. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), volume 42, pages 89–100. ACM, 2007.

M. Zalewski. American Fuzzy Lop.


Fuzzing and SW Testing (2)

T. Wang, T. Wei, G. Gu, and W. Zou. TaintScope: A checksum-aware directed fuzzing tool for automatic software vulnerability detection. In Proceedings of the IEEE Symposium on Security and Privacy, 2010.

M. Neugschwandtner, P. Milani Comparetti, I. Haller, and H. Bos. The BORG: Nanoprobing binaries for buffer overreads. In Proceedings of the ACM Conference on Data and Application Security and Privacy (CODASPY). ACM, 2015.

I. Haller, A. Slowinska, M. Neugschwandtner, and H. Bos. Dowsing for overflows: A guided fuzzer to find buffer boundary violations. In Proceedings of the USENIX Security Symposium, 2013.

P. Godefroid, M. Y. Levin, and D. Molnar. SAGE: Whitebox fuzzing for security testing. Communications of the ACM, 55(3):40–44, 2012.


Symbolic Execution

· KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs (Cristian Cadar, Daniel Dunbar, Dawson Engler)


D. Engler and D. Dunbar. Under-constrained execution: Making automatic code destruction easy and scalable. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA). ACM, 2007.


C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: Automatically generating inputs of death. ACM Transactions on Information and System Security (TISSEC), 12(2):10, 2008.


Dynamic Test Generation To Find Integer Bugs in x86 Binary Linux Programs  David Molnar, Xue Cong Li, and David A. Wagner. Usenix Security 2009.


T. Avgerinos, A. Rebert, S. K. Cha, and D. Brumley. Enhancing symbolic execution with veritesting. In Proceedings of the International Conference on Software Engineering (ICSE), pages 1083–1094. ACM, 2014.

D. A. Ramos and D. Engler. Under-constrained symbolic execution: Correctness checking for real code. In Proceedings of the USENIX Security Symposium, 2015.




·       S2E: A Platform for In-Vivo Multi-Path Analysis of Software Systems by Chipounov, Vitaly; Kuznetsov, Volodymyr; Candea, George



Kernel Modules


Develop a simple Linux kernel module


Monitoring and Traces into OS Kernel

·  Proc and Debug FS

·  kprobes, ftrace

Observing Linux Kernel Internals at Run-time


Kernel Initialization


Hook linux kernel functions


Debugging kernel modules

·  gdb and kernel debugger

Debug kernel crashes


Virtualization and Cloud

·  Intel-VT and AMD-V, SGX



I/O Virtualization

·  WindRiver Simics

·  KVM and QEMU HW

Kernel Debugging with QEMU/Simics






Device Drivers Vulnerability

·  Testing Closed-Source Binary Device Drivers with DDT (Volodymyr Kuznetsov, Vitaly Chipounov, and George Candea)

·  Tolerating Malicious Device Drivers in Linux

Hook windows kernel functions










Other Topics

·  TBA

Projects TBA


Academic Honesty

Students are responsible for informing themselves about the UGA Culture of Honesty before performing any academic work. All academic work must meet the standards contained in the Culture of Honesty. The standard is available at