tssci has a VERY long post about crawling in relation to vuln assessments.

"This post isn’t intended to be a retort to Jeremiah Grossman’s post last month on Why crawling matters, but more of a follow-up post to my latest blog entry on Why pen-testing doesn’t matter. Hint: both pen-testing and crawling are still important/matter, but my CPSL process described in my last post leans towards a new, idealistic approach on the vulnerability problem.

I’ve covered web crawling before, in Scraping the web for fun and profit as well as when I mentioned papers written by Shreeraj Shah in my longest-yet post on 2007 Security Testing tools in review. Ajax crawling was only partially covered, so we’ll continue to expand on that topic (as well as other topics) in this blog entry. Another motivator for posting information about crawling is that I’m seeing some repeated information over at blogs such as securitytechscience.com/blog — especially in their latest on Screen Scraping."

Article Link: http://www.tssci-security.com/archives/2007/12/02/why-crawling-doesnt-matter/