I’ve been doing web development for a pretty long time, but just over the last few years I’ve come to really appreciate the fundamentals of HTTP, what’s going on under the hood when I’m building web applications. There are two sides of this. The first is that HTTP is in one sense a very simple protocol. It’s just little text messages going back and forth between your browser and the web server. Whether I’m using Node or Django or some huge WSDL-driven Java XML-Beans monstrosity, what it’s doing isn’t rocket science; it’s just taking care of a bunch of tedious, nit-picky bookkeeping that I don’t want to be bothered with. If I really wanted to, I could just type the messages myself (and we’ll get to that in a minute).
The practical upside of that is that you can use really simple tools to debug big, hairy, complex web applications. A few years ago, I was working in one of those Big Web Services systems with WSDL files and auto-generated Java code and layers and layers of middleware. We’d get some kind of error at the front end, and it’d be really hard to tell which piece had broken. So I ended up writing a bunch of really simple shell scripts to test the web services in isolation. I’d spackle together something using
sed that built up and picked apart the messages as text, without dragging in all that mess of Java code.
The flip side is that HTTP is actually a richer protocol than I’d realized. There’s a lot I didn’t know about it until I started building RESTful web services and trying to understand the “right” way to do it. There’s all this stuff you can do with status codes and headers that I’d been re-implementing at the application level.
To take a recent example, I’ve been working on a web service that talks to other web services. Someone would make a call to us, we’d call the back-end services, they’d time out or barf up some sort of error, and we’d pass back a 500 error to our client. They’d see it and email us asking what was wrong with our service. It’d be nice to let them know it’s not our fault and that they should pester the back-end systems people instead. We could send back a message body that says something like, “Back-end systems failure. Original error message follows,” but it turns out we can say that just by returning a different status code. Not only is there a 502 status code, which means that a back-end system failed, but there’s also a 504, which means that we timed out trying to contact it. That tells our client that they can try again in a little while and the request might go through.
Ok, enough talking. Now code.
Goin’ all Mechanical Turk on this
To illustrate the first point, that this is all just text, I’m going to play human web server, using
netcat. If you’re not familiar with it, it’s a standard unix utility that just opens a network connection. Anything you type gets sent along it; anything that comes back gets printed out on your screen. I open up a terminal and type:
That starts up
netcat listening on port 3333. Then I switch to my browser and tell it to go to
http://localhost:3333/. The “page loading” indicator starts spinning. In the
netcat terminal, I see:
1 2 3 4 5 6 7 8
That’s Chrome telling me it wants the root resource (
/, which Apache or whatever would normally interpret as
index.html). It’s also telling me a lot about what kind of response it can handle. I’m going to ignore all that for now and just type:
1 2 3 4 5
Pretty straightforward. The content length is 7 because it includes the return character after “Hello!” Here’s what we see in Chrome:
Switch back to the browser and go to
http://localhost:3333/index.html. In the
netcat terminal, we get a request that’s much the same as before, except the first line is:
Since they asked for HTML, I’ll give them HTML. I type:
1 2 3 4 5
And in Chrome we see:
So at some fundamental level, that’s all a web application is. It’s a program that listens for a connection, gets little text messages, interprets them, and sends back responses. How simple can we make that?
RESTful Web Services in Bash
How about this?
1 2 3 4 5
uptime is a standard unix utility that reports how long the computer has been running and what the 1, 5, and 15 minute system load averages are. That’s marginally useful - I’ve actually used a script much like this for basic server monitoring. Put it in a file, make it executable, run it from the command line, and it’ll spit out something like:
1 2 3
From here, if you want to follow along, you’ll need to have Apache set up and configured to let you run CGI scripts in the directory you’re working in. (That’s a whole tutorial on its own, but here’s some instructions for Mac OS X. Otherwise, Google for “apache enable cgi” and your operating system.)
On my machine, this script is saved as
public_html/api/v1/load/index.cgi. That lets me access it as
http://localhost/~colin/api/v1/load/, as we can see in Chrome:
We can also use
netcat in place of Chrome. Instead of listening on a port, we open a connection to the web server’s port:
Then I type:
And I get this back from Apache:
1 2 3 4 5 6 7 8 9 10 11
You can see that Apache includes a bunch of header fields that I didn’t bother to when I was playing web server. (I’ll trim most of these out of later examples to cut down on the clutter.) The more interesting thing is that it doesn’t have a
Content-length header. What it has instead is
Transfer-Encoding: chunked. That says that its content will be in chunks, prefixed by their size (in hexadecimal). 46 hex is 70, which is the length of the next line (again, counting the return character at the end). The ‘0’ for the next chunk says, “that’s all, folks!”
We can make this a little easier on ourselves by using
curl instead of
netcat. It’s a somewhat more custom tool for making HTTP requests. We can just run
curl -si http://localhost/~colin/api/v1/load/ from the command line, and get back:
1 2 3 4 5
That’s the same as what
netcat gave us (minus the header clutter), but notice that it combined the chunked response for us. Even at this level, some of the details are being hidden.
Let’s take this a step further. The status script gets a status message (“GREEN”, “YELLOW”, or “RED”) from a file, and prints it out like so:
1 2 3 4 5 6
It also lets us set a new status like so:
1 2 3 4
Note that we used the same URL, but changed the HTTP method to
PUT (instead of the default
GET - don’t ask me why that’s the
-X option) and specified “GREEN” as the data (
-d) to be sent along with the request. We get back an exciting new response code: 204! Since we’re telling not asking, it doesn’t make much sense for the server to send anything back. The 204 status just says, “That thing you were doing? It worked.” No reason to have a message body saying “Success!” when the code already tells you that. I’ve definitely been guilty of reinventing that wheel before I ran across this.
What if we try to send a bad status, like ‘BLUE’?
1 2 3 4 5
400 is the “your mistake” error code, which is pretty generic, so we include a descriptive message in the response body. Since it’s a user error, it’s reasonable to just have a human-readable message.
If you look at the script, you’ll see references to environment variables like
$REQUEST_METHOD. That’s how Apache makes information about the request avaliable to the script (as part of the CGI standard). In case you want to see all of them, I’ve added an env script, which dumps them all out, plus the content. You can hit it with Chrome or
curl, or even
netcat. See what’s different between them.
Ok, great! Now we have two simple yet useful web services. But they’re not so simple that they don’t need any documentation, so let’s add some. We could have some sort of parallel hierarchy for documentation, like
/api/docs/v1/load/, etc., but that’s kinda clunky. Instead, let’s rework our services so they give you data when you ask for data, and text when you ask for text. For that, we take advantage of the
Accept header. Take a look at the script to see all the details, but it’s basically a bunch of if-then-else clauses checking
Now when we point
curl at the new version of the ‘load’ service, we normally get:
1 2 3 4 5 6 7 8
But if we add the
Accept header saying we want JSON data, we get:
1 2 3 4 5 6 7 8
Sweet! And when we use Chrome, which asks for
text/html, we get:
If you hit the
env script with Chrome, one of the environment variables you see is
HTTP_ACCEPT_LANGUAGE, which corresponds to the
Accept-language header. So by setting a header field, we can tell the server what language we want the response in. Again, something that I probably would have re-implemented in the message body in a totally ad-hoc way, instead of using the standard language codes. For my browser, the language code is
en-US - American English. Just for kicks (since Google Translate makes it easy and my girlfriend knows enough German to sanity-check it), let’s translate the
load script documentation into German, and display that if the
Accept-language header begins with ‘de’. Take a look at that version of the load script to see the details.
When I hit
v3/load/ in the browser, it looks the same. But then I (temporarily) switch my language preference to German (Google to see how to do that) and reload the page, and I get:
If I go back and hit the
env script, I see
HTTP_ACCEPT_LANGUAGE is now
q is strength of preference, so
de at 100%,
en-US at 80%, plain
en at 60%. I cheat and just look at the first two letters.)
This is one that I recently avoided re-implementing. We were developing an iPhone app that used a data file that wouldn’t change often, but did need to be kept up to date. It was a couple megabytes - more than we’d want to have to fetch every time the app starts up. We were talking about all sorts of ways of doing that before I thought, “Hey, isn’t this essentially a content caching problem? I bet there’s some sort of mechanism built into HTTP for that.” It turns out that’s an understatement - there’s all kinds of caching schemes built into HTTP.
The simplest and most generally useful involves the
ETag header. This is a unique identifier generated by the server and included in the response headers. Apache does it automatically for static files. For example, if you fetch this file (the one you’re reading) with
curl, you’ll see a header block like this:
1 2 3 4 5 6 7 8
Then the next time you request that page, include an
If-None-Match header with that
ETag value. That tells the server to only return the contents if the
ETag value it calculates doesn’t match. (Note that the quotes are important!) If it’s still the same, it returns a 304 status code, like so:
1 2 3 4 5
Apache doesn’t set an
ETag header for script responses (because they’re expected to be different every time), so we’ll have to implement this ourselves. Fortunately, that’s pretty easy. I just run the status contents through the
sha1sum utility, which calculates a unique number based on them. (How it does that is a whole ‘nother article.) Amazingly, this turns out to be slightly faster than getting the modification time for the file. That section of the code looks like:
1 2 3 4 5 6 7 8 9 10 11
Pretty simple, huh?
If you want to learn more about this, Wikipedia is a good place to start, as always. They’ve pulled together info on all the status codes and header fields, as well as an in-depth overview of HTTP. Yahoo! also has a nicely concise summary of what the status codes mean and when to use them.
So if you’re new to web development and web services, hopefully they’re a little less intimidating now. If you’re an old hand, maybe you’ve picked up some new tools and techniques for debugging. If you haven’t already, play around with
netcat, be a Human Web Server, get it into your fingers. Make
curl part of your toolbox. For both, take some time to explore what else they can do. Next time you’re designing web applications or services, consider whether you’re reinventing the wheel or making things more complicated than they need to be.
Have the courage to build simple things.