IC Colloquium: Evaluating Fuzz Testing (An Adventure in the Scientific Method)
Event details
| Date | 22.10.2018 |
| Hour | 16:15 › 17:30 |
| Location | |
| Category | Conferences - Seminars |
By: Michael Hicks - University of Maryland
Video of his talk
Abstract:
IFuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.
This is joint work with George Klees, Andrew Ruef, and Benji Cooper (all at UMD) and Shiyi Wei (UT Dallas)
Bio:
Michael W. Hicks is a Professor in the Computer Science department and recently completed a three-year term as Chair of ACM SIGPLAN, the Special Interest Group in Programming Languages. His research focuses on using programming languages and analyses to improve the security, reliability, and availability of software.
He has explored the design of new programming languages and analysis tools for helping programmers find bugs and software vulnerabilities, and explored technologies to shorten patch application times by allowing software upgrades without downtime. Recently he has been looking at synergies between cryptography and programming languages, as well techniques involving random testing and probabilistic reasoning. He also led the development of a new security-oriented programming contest, "build-it, break-it, fix-it," which has been offered to the public and to students of his Coursera class on Software Security. He blogs at http://www.pl-enthusiast.net/.
More information
Video of his talk
Abstract:
IFuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.
This is joint work with George Klees, Andrew Ruef, and Benji Cooper (all at UMD) and Shiyi Wei (UT Dallas)
Bio:
Michael W. Hicks is a Professor in the Computer Science department and recently completed a three-year term as Chair of ACM SIGPLAN, the Special Interest Group in Programming Languages. His research focuses on using programming languages and analyses to improve the security, reliability, and availability of software.
He has explored the design of new programming languages and analysis tools for helping programmers find bugs and software vulnerabilities, and explored technologies to shorten patch application times by allowing software upgrades without downtime. Recently he has been looking at synergies between cryptography and programming languages, as well techniques involving random testing and probabilistic reasoning. He also led the development of a new security-oriented programming contest, "build-it, break-it, fix-it," which has been offered to the public and to students of his Coursera class on Software Security. He blogs at http://www.pl-enthusiast.net/.
More information
Practical information
- General public
- Free
- This event is internal
Contact
- Host: Mathias Payer