"The Web Application Security Consortium is pleased to announce the release of version 1 of the Web Application Security Scanner Evaluation Criteria (WASSEC).  The goal of the WASSEC project is to create a vendor-neutral document to help guide information security professionals during web application scanner evaluations.  The document provides a comprehensive list of features that should be considered when conducting an evaluation.  The WASSEC project does not promote any specific products or tools, but instead provides valuable information to help you make your own decision about which of these tools best meets your needs.
 
The WASSEC document be found here in both wiki and PDF formats:
http://projects.webappsec.org/Web-Application-Security-Scanner-Evaluation-Criteria"

From the project page

"Web Application Security Scanners are automated tools to test web applications for common security problems such as Cross-Site Scripting, SQL Injection, Directory Traversal, insecure configurations, and remote command execution vulnerabilities.  These tools crawl a web application and locate application layer vulnerabilities and weaknesses, either by manipulating HTTP messages or by inspecting them for suspicious attributes.

A large number of web application scanning tools are available, both commercial and open source.  Effective use of these tools is an important part of a thorough web application security assessment, and regular security scans are required to comply with security requirements such as section 6.6 of the Payment Card Industry Data Security Standard (PCI-DSS).

The Web Application Security Scanner Evaluation Criteria (WASSEC) is a set of guidelines to evaluate web application scanners on their ability to effectively test web applications and identify vulnerabilities.  It covers areas such as crawling, parsing, session handling, testing, and reporting. 

The goal of the WASSEC is to create a vendor-neutral document to help guide web application security professionals during web application scanner evaluations.  This document provides a comprehensive list of features that should be considered when conducting a web application security scanner evaluation.  Different users will place varying levels of importance on each feature, and the WASSEC provides the user with the flexibility to take this comprehensive list of potential scanner features, narrow it down to a shorter list of features that are important to the user, assign weights to each feature, and conduct a formal evaluation to determine which scanning solution best meets the user's needs.

The aim of this document is not to define a list of requirements that all web application security scanners must provide in order to be considered a "complete" scanner, and evaluating specific products and providing the results of such an evaluation is outside the scope of the WASSEC project.  Instead, this project provides the tools and documentation to enable anyone to evaluate web application security scanners and choose the product that best fits their needs. "

Link: http://projects.webappsec.org/Web-Application-Security-Scanner-Evaluation-Criteria