Web Servers and HTTP (for beginners)
Web browsers interact with web servers using the Hypertext Transfer Protocol (HTTP). When you click on a link on a page, fill out a form or start a search, the browser sends an HTTP request to the server.
This request includes:
- The path that defines the target server and the resource (e.g. a file, a specific data point on the server, the service being started, etc.).
- A method that defines the required action (e.g. to retrieve a file, save or update some data). The different methods/commands and the related actions are listed below:
- GET - to get a certain resource (for example, an HTML file containing information about a product or a list of products).
- POST - create a new resource (for example, a new wiki article, add a new contact to the database).
- HEAD - to get metadata about a certain resource without getting content, as GET does. You can, for example, use a HEAD query to find out when a resource was last updated and only then use the (more "costly") GET query to load a resource that was modified.
- PUT - update an existing resource (or create a new one if it does not exist).
- DELETE - delete the specified resource.
- TRACE, OPTIONS, CONNECT, PATCH - these commands are used for less popular/advanced tasks, so we will not consider them.
- Additional information can be encoded in the query (e.g. form data). The information can be encoded as:
- URL parameters: GET requests encrypt data in the URL sent to the server by adding name/value pairs at the end, e.g. http://mysite.com?name=Fred&age=11. You always have a question mark (?) separating the rest of the URL from the URL parameters, a sign equal (=) separating each name from its corresponding value, and an ampersand (&) separating pairs. URL parameters are inherently "unsafe" because they can be changed by users and then sent again. As a result, the /GET request parameters are not used for requests that update data on the server.
- POST data. POST queries add new resources whose data is encrypted in the query body.
- Cookies of the client part. Cookies contain session data about a client, including keywords that the server can use to determine its authorization status and access rights to resources.
Web servers wait for messages with requests from clients, process them when they come in and respond to the web browser via a HTTP reply message. The response contains an HTTP response status code that shows whether the request was successful (e.g. "200 OK" means successful, "404 Not Found" if the resource cannot be found, "403 Forbidden" if the user is not allowed to view the resource, etc.). The body of a successful response to a GET request will contain the requested resource.
Once the HTML page has been returned, it is processed by the browser. The browser can then examine the links to other resources (e.g. the HTML page usually uses JavaScript and CSS files) and send a separate HTTP request to upload these files.
Both static and dynamic websites (discussed in the following sections) use exactly the same protocol / communication templates.
Example of a GET request / response
You can generate a simple GET request by clicking on a link or through a site search (e.g. a search engine page). For example, an HTTP request sent during a "client server overview" query on an MDN site will be much like the text below (it will not be identical because parts of the message depend on your options.
The format of the HTTP message is defined in the web standard (RFC7230). You don't need to know this level of detail, but at least now you know where it came from!
Request
Each line of the request contains information about the request. The first part is called a header, it contains important information about the request, just as the HTML head contains important information about the HTML document (but not the content of the document that is located in the body):
GET https://developer.mozilla.org/en-US/search?q=client+server+overview&topic=apps&topic=html&topic=css&topic=js&topic=api&topic=webdev HTTP/1.1
Host: developer.mozilla.org
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Accept: text/html,application/xhtml+xml;q=0.9,image/webp,*/*;q=0.8
Referer: https://developer.mozilla.org/en-US/
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8,es;q=0.6
Cookie: sessionid=6ynxs23n521lu21b1t136rhbv7ezngie; csrftoken=zIPUJsAZv6pcgCBJSCj1zU6pQZbfMUAT; dwf_section_edit=False; dwf_sg_task_completion=False; _gat=1; _ga=GA1.2.1688886003.1471911953; ffo=true.
The first and second lines contain most of the information mentioned above:
Request type (GET).
- URL of the target resource (/en-US/search).
- URL parameters (q=client%2Bserver%2Boverview&topic=apps&topic=html&topic=css&topic=js&topic=api&topic=webdev).
- The target website (developer.mozilla.org).
- The end of the first line also contains a short line identifying the version of the protocol (HTTP/1.1).
The last line contains information about client cookies - in this case you can see the cookies that include id for managing sessions (Cookie: sessionid=6ynxs23n521lu21b1t136rhbv7ezngie; ...).
The remaining lines contain information about the browser used and some of its supported features. For example, you can see here:
- My browser (User-Agent) Mozilla Firefox (Mozilla/5.0).
- It can accept information packaged by gzip (Accept-Encoding: gzip).
- It can accept the specified encodings (Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7) and languages (Accept-Language: de,en;q=0.7,en-us;q=0.3).
- The Referer line identifies the address of the web page containing a link to this resource (i.e. the source of the original request, https://developer.mozilla.org/en-US/).
An HTTP request may also contain a body, but in this case it is empty.
The answer is
The first part of the answer to the request is shown below. The header contains the information described below:
- The first line contains the 200 OK response code indicating that the query was completed successfully.
- We can see that the answer is in text/html format (Content-Type).
- We can also see that the answer uses the UTF-8 encoding (Content-Type: text/html; charset=utf-8).
- The header also contains the size of the response (Content-Length: 41823).
At the end of the message we see the body content containing the HTML code of the returned response.
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: developer1.webapp.scl3.mozilla.com
Vary: Accept,Cookie,Accept-Encoding
Content-Type: text/html; charset=utf-8.
Date: Wed, 07 Sep 2016 00:11:31 GMT
Keep-Alive: timeout=5, max=999
Connection: Keep-Alive
X-Frame-Options: DENY
Allow: GET
X-Cache-Info: caching
Content-Length: 41823
<!DOCTYPE html>
<html lang="en-US" dir="ltr" class="redesign no-js" data-foopensanslight=false data-foopensans=false >
<head prefix="og: http://ogp.me/ns#">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">.
<script>(function(d) { d.className = d.className.replace(/\bno-js/, ''); })(document.documentElement);</script>.
...
The rest of the answer header contains information about the answer (e.g. when it was generated), the server and how it expects the browser to process the page (e.g. the X-Frame-Options: DENY string tells the browser not to allow this page to be embedded if it is embedded into <iframe> on another site).