Technology Evolves.
Many of us have chosen to work in technology because we are pioneers. I speak in the classical sense of pioneering: those of us who venture into the wilderness to make a path for others. We forge and shape new tribes; new communities; new products and new industries. To me, the most interesting thing about pioneering is -- when it is done well, others join you on the path you have made. A pioneering product will take off with the users spreading the news by word of mouth, experimenting, contributing, helping to make it better. I will elaborate on this because, as we all know, the objectively best technology does not always win. "Done well" is a complex matrix of good (enough) technology, good timing, vocal advocates and broad scale adoption.
The Nginx bookshelf. There was a time when every developer building client-server applications knew Stevens by heart
In the early years of computing, every network tool had its own protocol and to develop software we had to live and love those low-level languages and protocols. As client/server evolved as the dominant model for extending computing beyond the capabilities of a single piece of hardware, we had to grok the fundamental language of networked computing -- TCP/IP -- and the voodoo of Berkeley sockets. On top of TCP new protocols flourished and grew, masking some of the more mundane and esoteric complexities that could be left as 'good enough' for many of our use cases.
Tools and developer communities flourished around these higher order protocols for emerging use cases that did not need the granularity of a tuned TCP stack. Like pioneering ants sharing and reinforcing paths to find food discovered in the wilderness, HTTP gathered enough mindshare of both developers and consumers to become codified and ubiquitous; and then extended.
HTTP is the ubiquitous protocol
The standard configuration of firewalls promiscuously passing traffic destined to ports 80 and 443 helped reinforce this path. For many years when I needed to make an outbound SSH connection from a client's network, I tunneled it over SSL. As habits like these developed and the community coalesced around HTTP, innovation moved upwards in the stack from low-level client/server implementations speaking custom protocols to web applications being accessed by out-of-the box browsers or thin clients making API calls via web services running over HTTP and HTTPS.
Internet Protocols via Wikipedia
Of this list of common protocols that run the Internet, as we know it, there are a very small number of developers who are actively working on software that directly manipulates or writes to these protocols. Those who do so are usually maintaining legacy code or working deep in the network stack on tooling that is rapidly commoditizing -- operating systems; network peering and routing; data stores; IaaS; PaaS; BaaS. As consumers of the Internet application stack, the majority of traffic that we interact with is HTTP or HTTPS.
TCP is now the wire and HTTP is the protocol.
Peak Traffic Composition via SandVine
Network connectivity has stabilized and, for most developers, dithering over the choice between UDP and TCP as the best transmission protocol is a thing of the past. Even streaming media (a classic use case for UDP) has been adapted to run within HTTP over TCP.
None of this means that we have reached the frontier.
Recently, RFC2616 was obsoleted by six new RFCs cleaning up the definitions of HTTP/1.1. The idea of these new documents was to only to improve the articulation of the protocol except in cases of egregious security or interoperability issues. There is further work on all sorts of innovations and extensions to HTTP. And this work, fundamentally, is why HTTP remains the frontier. Websockets means that HTTP connections can be upgraded to a direct two-way socket connection, and HTTP/2 and SPDY is a re-envisioning of HTTP fixing some of the challenges that exist in serialized textual content transfer. And beyond the evolution of the protocol standard itself, we've started wrapping new functionality in HTTP, just like HTTP is wrapped inside TCP or as we hear over and over at technology events, platforms upon platforms upon platforms - it's turtles all the way down.
But, why do we care?
The standardization on HTTP has democratized development. Which in turn has disrupted the proprietary model of innovation. Gone are the days where the only technology choice was a solution from single company who sold vertically integrated stacks of hardware, operating systems, end user software, and peripherals.
- "In order to change an existing paradigm you do not struggle to try and change the problematic model. You create a new model and make the old one obsolete."- Richard Buckminster Fuller
We now have smaller, more nimble, individuals and companies innovating by fixing a customer need. Consuming data available from other products or services made available over HTTP through APIs, we're building single use or narrow use tools that can plug into or augment other products. These smaller products are built and evolved faster. They are building blocks to be assembled into masterpieces.
The modern Internet is made of compose-able and interoperable components all speaking HTTP. Go forth and build the pieces that you think are missing, just do so respecting the power that exists in the ever-evolving protocol of HTTP.