Haystack Home

It is an occasionally useful maxim of mine that standardization cuts two ways. It decreases variety and increases volume, almost without exception. The sometimes counter-intuitive result of standardization is an increase in human variety. The development of TCP/IP vastly reduced the kinds of interaction between computers while simultaneously multiplying the amount of information exchanged. Humans used this new volume for a dizzing array of webpages, services, games and most importantly organization of these kinds of information. Search engines are webpages, but in certain ways, they are above them. You go to webpages for specific content, even if of a dynamic kind. Search engines are fundamentally different, and belong in a different kind of category. services like finger and ssh are similar. They provide information about other services, or provide increased levels of interopability. They transcend simple transaction, but operate on the same mechanical level.

These specialized kinds of variety all operate on a single level from the standpoint of the computer, packets of information with particular addressing and content. This similarity allows the computer to transmit with impunity the great array of ideosyncratic creations that we call the internet. The key is that the standardization exists on a transparent level to the computer. If the english language or picture color balance were standardized, little increased volume would result. Because a computer lacks the ability to take advantage of the regularities in such standardization. Without extra potential volume, little new variety would result, and the standardization process would be a useless venture, from the standpoint of computers. Teaching english and photography may be easier in that hypothetical case.

If software was several orders of magnitude smarter, regularities in english and picture color balance, might be tractable to increased natural language parsing, and intelligent search of photos. But I digress.

The key is to provide increased potential usable volume by reducing underlying complexity up to the point of transparency. the encroaching standardization can never get in the way of human variety. This key point explains why standards that work with TCP/IP work up to a point. html, XML, dynamic content, standard port addresses, IPv6, these are all good things and useful to many people. But technologies like OS specific net connectivity, social exchange technologies, and even tiny simple things like standard quoting rules in forums and email lists fail miserably or just lead to balkinization.

But standards on the edge of human variety and machine parsibility is where the exciting ideas are. The question of which side of the line a technology falls under is an open question that often depends on effort, cleverness and popular interest. A wonderful example is the above 'success story' of XML. XML is another child of a larger technology called SGML. This miserably complex, highly disordered description framework had hundreds of interpretations which were almost entirely opaque to one another. But the central concept was such a good one that clever folks were compelled to snip usable portions of it into nice sub technologies that enjoyed much more success, like html, certain vector graphics formats, and document description frameworks. Enter some years later, faster computers, more complex software environment. XML, a specification that's nearly as powerful as SGML, and in many ways just as opaque is a runaway success. The line of machine-useful information is higher, and the human variety imparted to XML applications is truly stunning. Almost all SGML applications are now ported to XML(affirming their equivalency, though XML has a much greater ease-of use and cleverness factor). The difference is very slight, but enough for the differential of widely used and deprecated.

The moving line of standardization vs. human variety has reached new heights with the advent of complex software and UIs that reach into complimentary arenas and make available the unions to their users. The above link is a software project that may or may not succeed, but shows the approximate position of what kinds of information may be succeptable to machine organization for human benefit.

I was surprised by the possiblities and the sheer pleasure of interacting with a UI like this. It's pleasant surprises and steady advances that make me hopeful for informational technology like this. None of the underlying technologies are new. IMs, Emails, websites, text searching. But treating these superficially different services as a standardized commodity brings new potential USABLE volume to the table. It's theoretically possible to have infinite webpages, but only a small finite amount are useful. technologies like google and haystack bring that usable amount much higher, and haystack links that usable volume to other usable volumes, like private correspondence and personal data. The resulting human variety from this standardization may be used for... well, whatever you want to use it for :-)