Adventures in HttpContext All the stuff after 'Hello, World'

Building for the Web: What Are We Trying to Accomplish?

Jan 4 2012

The web technology landscape is huge and growing every day. There are hundreds of options from servers to languages to frameworks for building the next big thing. Is it nginx + unicorn + rubinus or a node.js restful service on cassandra running with ember.js and html5 on the front end? Should I learn or ? What’s the best nosql database for a socially powered group buying predicative analysis real-time boutique mobile aggregator that scales to 100 million users and never fails?

It is true there are many choices out there but web technology boils down to a very simple premise. You want to respond to a user action as quickly as possible, under any circumstance, while easily changing functionality. Everything from the technology behind this blog to what goes into facebook operates on that simple idea. The problem comes down to doing it at the scale and speed of the modern web. You have hundreds of thousands of users; you have hundreds of thousands of things you want them to see; you want them to buy, share, create, and/or change those things; you want to deliver a beautiful, customized experience; everything that happens needs to be instantaneous; it can never stop working; and the cherry on the cake is that everything is constantly changing; different features, different experiences, different content; different users. The simple “hello world” app is easy. But how do you automatically translate that into every language with a personal message and show a real-time graph with historical data of every user accessing the page at scale? What if we wanted to have users leave messages on the same page? Hopefully by the end of this series you will get a sense of how all the pieces fit together and what’s involved from going to 10 users to 10 thousand to 10 million.

The Web at 50,000 Feet

The web breaks down to three distinct areas: what happens with the client, what happens on the server, and what happens in between. Usually this is rendering a web page: a user clicks a link, a page delivered, the browser displays the content. It could also involve handling an ajax request, calling an API, posting a search form. Either way it boils down to client action, server action, result. Doing this within a users attention span given any amount of meaningful content in a fail-safe way with even the smallest amount of variation is why we have all this technology. It all works together to combine flexibility and speed with power and simplicity. The trick is using the right tool in the swiss-army knife of tech to get the job done.

Let’s break all this down a bit. Handling a user action well comes down to:

  • Having the server-side handle the request quickly (Serving)
  • A quick travel time between the client and server (Delivering)
  • Quickly displaying the result (Rendering)

And developing and managing all this successfully comes down to:

  • Changing any aspect of what is going on quickly and easily (Developing)

As sites grow and come under heavier load doing any one of these things becomes increasingly difficult. More features, more users, more servers, more code. There is only so much one server can do. There is also only so much a server needs to do. Why hit the database if you can cache the result? Why render a page if you don’t need to? Why download a one meg html page when it’s only 100kb compressed? Why download a page if you don’t have to? How do you do all this and keep your code simple? How do you ensure everything still works even if your data center goes down?

Http plays an important role in all of this. I didn’t truly appreciate http until I read Restful Web Services by Sam Ruby and Leonard Richardson. Http as an application protocol offers an elegant, scalable mechanism for transferring data and defining intent. Understanding http verbs, various http headers and how http sits on top of tcp/ip can go along way in mastering the web.

Making it all work together

So how do you choose and use all the tools out there to serve, deliver, render and develop for the web? What does client action/server action/result have to do with rack and wsgi? I naïvely thought I could write everything I wanted to in a single post: from using sass for compressed, minimized css to sharding databases for horizontal scalability. It will be easier to spread it out a bit so stay tuned. But remember: any language, framework or tool out there is really about improving client action/server action/result. Even something like websockets. Websockets eliminates the client request completely. Why wait for a user to tell you something when you can push them content? Knowing your problem domain, your bottlenecks, and your available options will help you choose the right tool and make the right time/cost/benefit decision.

I’ll dig into the constraints and various techniques to effectively deliver, serve, develop and render for the web in upcoming posts.