Application Security Vendors Need Help With Reporting
I've been reading web application vulnerability reports from tools and services for 6-7 years and found that 99% of these reports are geared towards security engineers or system administrators. Many of the reports I see focus on
- The type of flaw and what it its impact is
- The URL affected
- Links to references and additional reading
Security engineers typically aren't the ones fixing application level security defects and often file them into a development bug tracking system for development to address. A lot of the text is 'fluff' that is useful to security folks or those wanting a detailed explanation but overkill for those in development/qa. The fluff aside none of them seem to provide manual reproduction instructions (beyond reproducing the issue in their penetration testing tool) which doesn't allow for QA to regression test an issue in the
future. In my own experiences I've found myself needing to 'translate' these tool/services reports into language that makes sense to a developer/QA engineer (including writing manual reproduction instructions).
Many vendors are shifting to security testing throughout the development life cycle yet still aren't creating reports for the consumers within development. Development and QA are most interested in
- Knowing what application is affected.
- Knowing what parameters are affected.
- Knowing the flaw's impact on the safety of users, the site, and the application.
- Knowing how to test/reproduce the issue.
- Knowing the manual steps for reproduction.
- Knowing the vulnerable expected result.
- Knowing the non vulnerable expected result.
- Knowing how to fix the flaw.
I'm asking all vendors (product and services) reading this post to
- Consider your audience in your vulnerability summaries.
- Add *good* reproduction instructions
- Manual reproduction instructions so we can verifying things without needing to use your tool or service.
- Assume that the people consuming these reports have zero knowledge of tools such as Paros or burp proxy. Walk them through each step to reproduce the issue.Create reports even my mother could reproduce.
This site gets its fair share of product and service vendors readers so I'm considering this an open request to all of you. While this post can be seen as targeted towards a small audience I'm sure others have had the same experiences and frustrations and I welcome others to post what their experiences are.
Comments are open!
Great post, I also see one point that you may discuss in the future: the Executive Summary
These reports are almost always a piece of unreadable mysterious box for the executive team which are actually paying for the tests and need to be convinced to support the recommended corrections. Shows a two-page document with a lot of beautiful graphics and non-sense tables may help in some cases, however translate the technical problems to the business language will help us all.
Posted by: Eduardo V. C. Neves | Feb 10, 2009 3:56:26 AM
Hi,
Why do you think that the fluff in the advisories is irrelevant to developers? many developers learn from these advisories on the security problems, and how to write secure code, so next time they won't do the same mistake.
Could it be that since you are a web app sec expert, you think this is fluff? not everybody know what XSS or SQLi is these days...especially not developers, who oftentimes don't get proper security education.
If you don't want to read the fluff, just don't print it (most scanners allow you to disable it).
BTW - I do agree with the rest of your claims.
Posted by: Richard A. Barley | Feb 10, 2009 4:08:18 AM
I completely agree with the sentiment here. Please consider using the detailed reporting requirements in the Application Security Verification Standard (ASVS) from OWASP. The standard covers reporting at all four levels of rigor, and applies to findings made with automated tools (scan, sca), manual code review, manual penetration testing, and security architecture review as well.
Posted by: Jeff Williams | Feb 10, 2009 7:11:47 AM
@Richard
"Why do you think that the fluff in the advisories is irrelevant to developers?"
I suppose I'd be fine with the fluff if they got the rest of the reporting right, but they usually don't.
I do agree some developers will read further (this is good) but for qa people/testers they generally don't care/read them.
Posted by: Robert | Feb 10, 2009 9:04:22 AM