New blog site! Update your bookmarks/RSS feeds.

I have set up a new blog and I'm calling it Happy Chaos Freedom Machine. I've migrated some of the posts from here already and will be making all my new posts over there from now on. The reason for this is that I'm finding a focus emerging in my posting here that I want to hone and develop. For years I've wanted to write a book about what excites me about the way the world is changing, but I've been unable to find the right entry point and focus to get it done. Instead of a book, I'm going to blog it. Makes sense, right?

Anyway, please update any bookmarks. See you over there.

China's New State Investors - Newsweek: International Editions - MSNBC.com

The Chinese are freeing up some of the capital they've had in t-bills to use for higher risk and higher return - some $200B to start.

Two thoughts occur to me - first, couldn't you make money just chasing their money around? That huge a chunk of cash is going to make some very big ripples. Second, it seems to be part of an emerging trend of privatizing the capital markets. More on this on later.

BarCamp / iPhoneDevCamp

I have to hand it to my friends William and Raven - 3 weeks ago they decided they wanted to have the first iPhone-centric hacking session, and lo and behold: the iPhoneDevCamp.
It sounds like fun - 300 attendees with a bunch of iPhones, trying to figure out some useful/fun/elegant hacks. Wish I could be there - have fun guys!

Not a good time to be a trader

There is yet another technology-driven disruption brewing - this time in the realm of stock trading. The shift towards "algorithmic trading" is accelerating - in fact, according to this Reuters article, "About a third of U.S. equities trading is already being done using algorithmic trading, with that figure expected to soar to more than 50 percent by 2010..." Which may be an underestimation. Apparently an IBM study from 2006 estimated a tenfold reduction in the ranks of traders, as in, for every 10 traders today there will be 1 by 2015.

While it is inevitable that disruption hurts someone, it is also true that someone will benefit -- in this case I believe the benefits will come eventually in an Internet-style disintermediation of financial markets. After all, if one can execute strategies through a piece of software, it should be trivial to provide access to that software to whoever wants to use it, right?

Kraftwerk Kulturzentrum

I am consistently impressed by the quality of Bruce Sterling's thought. His recent Wired piece, Dispatches From the Hyperlocal Future, is no exception. It's a fictional set of blog posts from 2017 made by a hyperconnected Cory Doctorow stand-in who specializes in the applications of computing power embedded in everyday objects and ubiquitous network connectivity. He calls this "hyperlocal" - hyper as in linked and local as in wherever you happen to be right now.

Every paragraph actually touches on a distinct ground-shaking implication of the ultra-connected computational wonderland we're heading for, almost too many to keep track of, which is sort of appropriate. One theme popped out at me in particular, an enhanced ability to discover the complex set of interactions that produce a social phenomenon. In Sterling's piece it's a torrent of real estate speculation in Dubai that his main character is hired to analyze and understand. What he uncovers is the convergence of a loophole in a treaty to unify Taiwan with China that allowed wealthy Taiwanese to hide their assets in the houses they own and the legal attributes unique to Dubai.

This is the same effect the Internet had, with its distributed architecture and resources rendered networked applications effectively transparent, exposing the guts of the machine to whoever wanted to dig around in them. In the "hyperlocal" world Sterling describes, the flows of people and capital will be similarly transparent and exposed to those who wish to look. What a contrast to the mysterious ways of our world as they have been, accessible only to those "masters of the universe" with skills and wealth and connections.

Also, who can resist the idea of the "Kraftwerk Kulturzentrum," a "vast, perpetual, swirling technorave ... [an] homage to the 20th century's supremely influential band"

Individual Exceptionalism and the building Hedge Fund backlash

When asked if we believe we possess attributes that are above or below the average, the majority of us will answer that we are above. More than a mathematical impossibility, this is an insight into the insatiable desire to be unique and superior to the crowd that fuels so much of human society. Lottery tickets, fear of dying in an airplane crash, and The American Dream all have their roots in the fallacy that I am somehow exempt from the statistical realities that shape our world. The financial markets, in particular depend on this belief, and there exists no greater concentration of Individual Exceptionalism than in the world of Hedge Funds.

From the promise of outsize returns made by the fund managers to the seemingly limitless appetite for extraordinarily high fees and limited investors' rights, the unregulated world of Hedge Funds depends very much on the belief that somehow the laws of the universe don't apply to me. The pitch is basically this: a bunch of phenomonally smart guys are going to invest your money in such clever ways that they will consistently and massively out-perform the market. In exchange for this privilege, you will pay them a management fee, typically 2% of your investment per year, plus 20% off the top of whatever returns they make for you. There are more than 10,000 such funds now, with well over a trillion dollars under management. The catch? Most times you can't just take your money back, and the investment strategies are almost totally opaque even to investors. What makes us think we know under which shell the pea is hidden?

After a decade of growing Hedgemania, a backlash is now brewing. There's more talk of regulation and, as this fun, fact-filled article in The New Yorker details, a growing body of academic work that points out the lie at work. The author, John Cassidy, profiles one professor in particular, Harry Kat of the Cass Business School in London. Kat has been systematically dismantling the myth of Hedge Funds and Hedge Fund managers. In one study, he and a colleague looked at 77 funds' performance between 1990 and 2000 and found that when adjusted for fees, 72 of them underperformed the market. Over 90% of them.

Other academics studying Hedge Funds have looked at "survivor bias" -- the positive skew put on performance statistics by the dropping out of failing funds. Factoring the failures' results back into the mix, they found that between 1996 and 2003 Hedge Funds made a not-so-impressive average return of 9.32%. Further research showed that 40% of Hedge Funds' returns actually depend on the performance of the market as a whole. In other words, of that 9.32%, only about 5.59% came from the ultra-secret Hedge Fund strategies cooked up by those super-smart managers.

Is this a case of the pool of talent being diluted by the appeal of the vast fees fund managers can garner? This is a tempting perspective to take, since it preserves the appealing myth of the uber-manager with massive returns and the idea of the heroic, generally. However, the article hints at the more probable answer:

[Kat] argued that even some of the most prestigious funds owe much of their success to luck. “You can be fortunate,” he said. “You can live off market trends for quite a while. As in credit spreads”—the difference in yields between different types of bonds. “Credit spreads start to come down, and you make lots of money in credit. A couple of guys from an investment bank’s credit desk jump out and start a fund. If they are lucky, the trend continues for another couple of years, and they will look like masters of the universe. But when the trend reverses, or when there is no trend left, they are in trouble. If a guy has done well for two years, what does that mean? He could be really smart, or he could be really lucky. If I had bought stocks at the end of 1997 and you had looked at me at the end of 1999, I would have looked brilliant.”

In other words, clever people find opportunities in the market and create strategies to exploit them, then create a Hedge Fund around those strategies, or incorporate them into an existing fund. Then the opportunity booms, fizzles or bombs. Only in retrospect do the successful strategies look like the product of brilliant insight. The failures are swept under the rug or publicized as the products of less-talented fund managers. The luck machine rolls on, and we continue to mistake it for a game that can be won, if only by the exceptional among us. Kat sums it up well

"... if you know that eighty per cent of hedge-fund managers aren’t worth the fees they charge, then the rational thing to do is to give up trying to find a super manager, and just go for a good, efficient diversifier instead."

The Music of Chance

The most puzzling & even troubling aspect of market dynamics - that the best experts can be out-performed by happenstance. This story of the leading contender in a CNBC stock picking contest is a good illustration. She's a waitress who has no trading experience, who describes her method:

"Part of this was luck," she says. "A lot of it was a gut feeling, some eenie, meenie, minie, moe, and common sense."

This can be seen as an example of what Nassim Nicholas Taleb describes in The Black Swan: The Impact of the Highly Improbable - the most impact comes from events that are impossible to predict based on previous experience. As the story about the winning waitress puts it:

... it's also sign of what Paul Auster once called the "music of chance." Picking stocks is about luck as much as strategy. In a field of 375,000 contestants with 1.6 million portfolios, someone has to finish first.

To me this means that the best fund managers and stock pickers and quant models at best can match the performance of the market, over the long term. If they beat the market occasionally, it's random luck.

What does that mean for our understanding of large system dyanmics and financial markets in particular? Clearly the best experts and models should be heeded in order to ensure at least the performance of the market, but the holy grail of consistent market beating performance is a long way off, if it is achievable at all.

ETFs based on intelligent indexing

A more recent obsession of mine are Electronically-Traded Funds, or ETFs. I'm particularly interested in the rapid evolution they are undergoing, from recreations of relatively static indexes like the S&P 500 or the Russell 1000 to more sophisticated, creatively constructed indexes.

A good example is the Claymore/Sabrient Insider ETF, or NFO, which tracks an index of companies experience a heavy volume of insider buying, taking the famous Peter Lynch quote to heart:

"Insiders might sell their shares for any number of reasons, but they buy them for only one: they think the price will rise"

Sabrient, an independent analyst firm in California, has a number of interesting indexes that form the basis for their collaboration with Claymore, but NFO does seem to be the most interesting, as this Smartmoney article concurs. Backtesting shows almost 18% gains over the past ten years, but that and $5 will buy you a Starbucks grande chili-frapp-ca-ching-weeno or whatever passes for coffee these days.

More interesting is this comparison of NFO against the S&P for the past year, which shows it basically doubling its performance and clocking in a healthy 22% or so. Not that past performance is any indicator of future triumphs. I'm also intrigued by Claymore sector cyclical ETF, but I haven't looked into it deeply enough to make a determination.

This kind of simple tweak to narrow a broader index makes intuitive sense to me. All the data is out there - all the SEC filings, all the econometric indicators - why not correlate selected indicators to cherry pick the stocks that have a better chance of outperforming the herd? Seems more like the Web 2.0 way to invest, to me.

Disclaimer: I own NFO and I'm besides I have very little idea what I'm talking about, generally.

The iPhone cannot succeed

Yes, I prognosticate at my peril, but I think it's actually a pretty safe bet that it is impossible for Apple's forthcoming iPhone to be deemed a success -- at least, in relation to the level of hype and in the time frame of Wall Street's ADD.

New York magazine published a profile of Steve Jobs by John Heileman which serves as a terrific backgrounder to the launch. Apple's been blazing trails for personal computing since the beginning, and that pioneering spirit has often come at a considerable price. Each major product has initially met with derision, from the "GUIs are for children" Mac to the iPod. Yes, the iPod. Remember the first reactions to the scroll wheel? The general lack of interest in yet-another MP3 player from The Street? It took 18-24 months and some solid sales figures to turn it into the blue chip stock AAPL has now become.

The other price Apple pays for innovation is fallout from not-quite-perfect first versions of its most appealing technologies. With the Mac it was a lack of a hard drive, with the iPod, poor battery life. Even with evolutionary releases of products like laptops Apple seldom gets the 1st generation right. This is hardly a unique failing in high-tech, as these incredibly complex artifacts are cranked out with shorter and shorter product cycles that don't allow for rigorous engineering testing.

Which brings us to the iPhone. The level of anticipation for this Apple product surpasses that of any previous products, and even if every detail is perfect it's hard to believe that people won't be disappointed. The first mention of a cracked screen, corrupted memory, or simply the frustration of not having a keyboard is going to deflate the hype at least somewhat.

Beyond the initial introduction period, it seems very unlikely to me that the iPhone will ever achieve the iPod-level of complete market domination in the smart phone market. The iPod's appeal is simple - it is a very well-designed tool that does one thing extremely well (no one buys an iPod because it can display photos or play videos).

The iPhone suffers from the lack of focus that plagues all smart phones - communication device, information appliance, media player, camera, web browser, email client, instant message client, which one is it? Like a personal computer, by trying to be everything to everyone, the iPhone can't be the best of anything for anyone.

The Web is the playground of youth

Fascinating chart over at Businessweek depicting online activity. It validates my instinctual reaction to the common criticism about the Web, especially in the late 'nineties and early oughties - that the minority of people will want to be active creators. That was an age-biased view, born of the passivity bred into the boomers by mass media and mass consumption.

In every the category except "spectator" and "inactives", the older boomers (51-61) don't show up more more than seniors, and even the "young boomers" (41-50) are pretty absent.


Click the image for a full-size, highly readable chart on BusinessWeek's site. Oh, and who knew stodgy old BusinessWeek had it in them to create such a tufte-y chart?

Lively books about dull subjects, first installment: Against the Gods

Recently I realized that my favorite type of book is the type that allows me to learn about something I know is important and interesting but that I am prejudiced against as dull. I thought I'd share my impressions of these remarkable books on my blog, hoping that others can similarly expand their horizons without becoming stodgy tweed-wearing professors in the process.

Expecting a book about statistics to be on the deadly end of the dull spectrum, Peter L. Bernstein's Against the Gods sat on my "to read, someday" shelf for quite some time. When I finally did get around to it, though, I found it thoroughly enjoyable. Tracing the history of risk management may not sound scintillating, but in fact the themes it explores are deeply embedded in the very nature of our modern world.

Once I understood the basic premise - that quantifying risk allowed man to step out from the idea that his life was predetermined by the will of the gods - I had a hard time putting it down. Math and statistics have seldom been rendered so approachable. Instead of getting bogged down in equations or detailed explanation, Mr. Bernstein makes the very wise decision to trace the evolution of the concepts behind the science rather than the science itself. The result is a highly readable, deeply profound piece of work that will mess with your frame of reference.

Trivial theory of human productivity

Whipped up this little chart after a fleeting thought I had while driving. It occurred to me that everyone I know has a unique tradeoff between chaos and order. Some strive for perfect order, cleaning messes as they occur, crafting perfect filing systems, trying to keep their email inbox empty. Others simply forge ahead, valuing progress and production over cleanliness and structure.

From my point of view, pushing too far on either end of the scale bears a heavy price. Organizing and cleaning all the time leave a mind rigid and too many aspirations unfulfilled. Doing doing doing is ultimately just as counter-productive, since the damage you leave behind for others to fix eventually alienates you from your family and society.

The two dotted vertical lines, delineate the middle ground where the tradeoff is optimal - you have the raw creative energy that comes from chaos and drives innovation, intuitive leaps and superhuman productive spurts, and you also have the structure that sustains productivity by nurturing precision, calm and poise.

It's no secret to my loved ones that I tend too far towards chaos generally, but as I grow older and wiser I think (I hope!) I'm trending towards the middle.

2007 US Farm Bill

The importance of the Farm Bill due to come before congress this summer is difficult to overstate. Michael Pollan provides a good overview in this NY Times Magazine article.

The health of the American soil, the purity of its water, the biodiversity and the very look of its landscape owe in no small part to impenetrable titles, programs and formulae buried deep in the farm bill.

I was talking with my friend Ray the economist the other day - he was marveling at the apparent inability of our political system to remove such obviously distorting economic incentives. Pollan has a theory to explain this:

...most of us assume that, true to its name, the farm bill is about “farming,” an increasingly quaint activity that involves no one we know and in which few of us think we have a stake. This leaves our own representatives free to ignore the farm bill, to treat it as a parochial piece of legislation affecting a handful of their Midwestern colleagues. Since we aren’t paying attention, they pay no political price for trading, or even selling, their farm-bill votes.

George Orwell's Writing Advice

A scrupulous writer, in every sentence that he writes, will ask himself at least four questions, thus:

1. What am I trying to say?
2. What words will express it?
3. What image or idiom will make it clearer?
4. Is this image fresh enough to have an effect?
   

And he will probably ask himself two more:

1. Could I put it more shortly?
2. Have I said anything that is avoidably ugly?
   

One can often be in doubt about the effect of a word or a phrase, and one needs rules that one can rely on when instinct fails. I think the following rules will cover most cases:

1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do.
3. If it is possible to cut a word out, always cut it out.
4. Never use the passive where you can use the active.
5. Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent.
6. Break any of these rules sooner than say anything outright barbarous.

1. 
   

* From “Politics and the English Language” by George Orwell.

Our desperate need for a revolution in military affairs

Contemporary terrorist tactics require a radical rethinking of our society's response. Our current strategy can be summed up in two ways. First, we are projecting our military power into foreign states that conform roughly with our conception of the ideological threat posed by terrorism, the so-called "rogue states." Second, we are attempting to create the perception of tightly secured borders and air travel. Both of these efforts are not merely futile wastes of money, but are actually serving the terrorists' ends.

The military response to terrorism in the 21st century would be laughable if it didn't represent such a tragic, senseless waste of human life and society's wealth. The idea that a grossly bureaucratic, slow-moving, monolithic organization poses any threat to a decentralized, informal, ad hoc community is delusional. There is a reason why the idea of an elephant being afraid of a mouse holds such persistent appeal, regardless of whether it's true in nature. Similarly, the success of the independent Internet in the face of the combined efforts of the telecommunication industry's incumbents to control it was a foregone conclusion. The founding of the United States of America owes a debt to the advantages of the nimble vs. the monolithic, as does the most spectacular military defeat the USA ever suffered, in Vietnam.

Contemporary terrorism has gone one step further and explicitly utilizes the adverse effects of military force as a rallying call. The US attacks in the gulf states in the name of the "War on Terror" are proof that Al Qaeda is a legitimate, powerful foe of the US, one that the US fears. For an individual who has lost a livelihood or a family member to the effects of war, this makes the terrorist cause simpatico with his or her own motivations and emotions. The War on Terror yields more terrorists the more fervently it is persecuted, and the terrorists' goal of spreading fear is fulfilled.

Domestic security in the form of tougher border and airport controls has similarly paradoxical effects. Preventing a terrorist attack is quite simply impossible. There are infinite scenarios for taking over and/or taking down a flight, and infinite ways to circumvent borders. There are only finite resources for safeguarding against these scenarios. Therefore, no matter how hard we try there will always be a way for a determined individual to defeat all our security measures.

The terrorists' second and equally important goal is to cause economic damage to the West. This modern form of siege warfare is fought on a very broad battlefield. The economic harm done by 9/11 is just as important as the psychological harm, and probably has longer lasting effects. Therefore, the resources expended in a futile attempt to secure air travel and borders actually aides the terrorists.

What then are appropriate responses to modern-day terrorism? What can the West do to defend itself against the harm that terrorists seek to inflict? It can seem hopeless when faced with an enemy who is made stronger by one's attempts to weaken him, and whose goals are achieved through one's efforts to mount a defense. However, a way to defeat terrorism does exist if we are willing to take a radically different view of the challenge.

Firstly, our offense cannot take the form of any type of conventional military campaign. One counters a distributed network with another distributed network. Only a loosely organized network of semi-autonomous operatives will be able to speed and agility of the terrorist networks. These covert operatives need to match as closely as possible the profile of the target organizations, and be appropriately compensated. They also must be given the power to carry out attacks at will without pre-clearance. However, in order to maintain control over such a force, every attack must be transparently documented and justified and reviewed after the fact. If an operative takes a life that cannot be justified he or she will face criminal charges.

This force will represent a blending of military and intelligence forces operating with a new degree of autonomy and decentralization. It may be difficult for us to stomach putting such a force into the field, but if appropriate protocols are defined and maintained through oversight, the precision of its efforts will put the conventional military to shame. Most importantly, however, is the publicizing of these efforts. We want the world to know that instead of rolling tanks into their towns we have shadowy enforcers targeting only those among them who mean us harm. It is essential that the terrorists themselves know that if they engage in training or planning for an attack they make themselves targets of an unseen foe.

In fact the entire effort against terrorism should focus on increasing the cost of that strategy incurred by those who pursue it. As a methodology, terrorism is appealing because of its extremely low barriers to entry and extremely high returns on investment. A few random miscreants can merely talk amongst themselves about wanting to mix explosives from their component parts in the bathroom of a commercial aircraft and as a result Heathrow airport is virtually shut down for days and travelers are limited to three ounce bottles of any liquids kept separate in a one quart ziplock bag. Good news for the makers of toiletries and ziplock bags, perhaps, but not good news for the economy at large. The additional drag on the efficiency of air travel has incalculable knock-on effects and the public's mood as a whole is degraded by the peculiar mix of fear and disgust at our own institutional reaction to that fear.

That kind of a result for such a paltry effort makes terrorism completely irresistible to those who seek to attack the west. Until we increase the cost of their participation in terrorist plots and decrease the expected return on those plots, it is inevitable that the plotting will continue. Public statements of intent to pursue and eliminate terrorists with our own decentralized secretive forces - better trained and equipped than any terrorist cell will ever be - is the first step. The casual recruit will think twice if he believes that there may be a sniper waiting for him after the meeting of the cell. Even more effective, calibrating our response to uncovered plots and even successful acts of terrorism to minimize societal and economic disruption will introduce the only brinksmanship likely to succeed against terror tactics - to make a significant impact the terrorists will need to plan ever bigger and more elaborate attacks. The more complex the plot the more people involved, and the more likely their efforts are to being discovered by own counter-terrorism forces.

No matter how effective they may be in theory, these are not easy strategies to adopt. The use of covert operational forces to fight a foe is troublesome in a nominally open and democratic society such as ours. The spectre of death squads accountable to no-one for their use of tactics such as kidnapping, torture and assassination instantly springs to mind. However, a bureaucratic, centralized, hierarchical force will always be bested by a determined, decentralized foe using guerrilla tactics. The answer is to purposefully create a blend of the structural benefits enjoyed by the terrorist network with the oversight and accountability an open and free society deserves and demands. This is challenging simply because it has never been attempted before in military history, but there are examples from other fields that may help guide our pioneering efforts.

In computer networking for example, the Internet has defeated all other models for global telecommunications and computing, and yet it has no centralization of control or bureaucratic oversight. It accomplishes this by a clearly stated set of protocols for interactions to which all parties must adhere. Communications that do not conform to the standards of the Internet are simply discarded with little or no effect to the network as a whole. If a code of conduct were drawn up for our counter-terrorist forces to which they would be held accountable in post-operational review, that should be sufficient to ensure that our standards for human rights are not violated.

Perhaps the proposed 180 degree shift in domestic security measures is more emotionally troubling, since it admits that attacks and loss of life are inevitable and preventing them should not the priority for our investment of time and resources. It is difficult to accept the loss of any human life when presented with the possibility. Certainly if I thought there were a threat to a friend or family member I'd do everything in my power to try to deflect the risk to their life and limb. The difficulty here is that the threat is diffused and generalized across the entire population and therefore cannot be dispelled - protecting one individual or group merely shifts the threat to another until the necessary adaptations are made at which point the threat returns in a newly mutated form.

What is required is a kind of mental jujitsu where the threat's own power is turned against itself. Terrorism is only powerful because of its ability to disrupt and frighten. Take away that power through near-instantaneous recovery of normalcy after an attack and the likelihood of the attack ever happening is diminished. In other words, do away with the hours-long security checks at the airport and create instead a commercial air travel system that recovers from a disaster in hours not days, and we will have effectively defended against future attacks.

It bears mentioning that such a recovery function would serve our society equally well in the event of a natural disaster. In fact the threat of terrorism to society is best understood in comparison to the threat of a hurricane. You can't stop the wind, so you make sure that when it blows you are affected as little as possible. Terrorism is, thankfully, weaker than a force of nature - by increasing the personal risk of being a terrorist while simultaneously diminishing the rewards of their attacks, we can render the strategy ineffective.

Open Source is not Commoditizing Software

Perusing the title of this essay I imagine the reader bursting out, "what? Preposterous!" Or at least, that's what I imagine if I was writing this a hundred years ago, and my subject was petroleum oil. If that were the case, you'd be right, of course. The development of the oil industry did result in the reduction of its product to a near-perfect commodity. In my actual context, the first decade of the 21st century and the continuing evolution of the software industry, however, I stand by the statement: Open Source software is not turning software into a commodity.

"But," you might say, "you can download and use for free any flavor of Linux or BSD, OpenSolaris, Apache projects, Mambo, Drupal, Wikimedia, SugarCRM, Compiere, Nagios and the list goes on and on." The acceptance of the commoditizing effect of Open Source software (OSS) has gained credence through the statements of various industry luminaries, such as Johnathan Schwartz. If the new CEO of Sun says OSS is commoditizing his own products, who am I to disagree.

Except, at the risk of repeating myself, what is happening to software is not that it is turning into a commodity. Without wanting to sound too pedantic, a commodity is defined as a good that can be obtained from any supplying vendor without preference. The suppliers market interchangeable 'versions' of the commodity. No one will claim that SugarCRM and Apache HTTP server can be used interchangeably. Even restricted to a single category, choosing OpenSolaris or RedHat Linux involve very different skills requirements and deployment practices.

To draw the contrast even more sharply, we can return to oil. Software is not oil, a sea of bits to be stored in enormous steel tanks, pumped through pipes (Senator Ted Stevens notwithstanding) and purchased without preference at whatever retail outlet is most convenient to your computing needs. Network connectivity may be oil, but software most definitely is not.

So what is OSS doing to software? If not commoditization, then what? Extending our analogy to the oil industry may prove illustrative. Finding oil, extracting it from the earth, refining it and distributing it required the development of very sophisticated tools and techniques. As the technology of oil exploration, production and distribution developed, innovative applications of the newly available energy source developed in parallel. Oil-fired ships, airplanes and automobiles all drove demand for oil quicker than the industry could create the supply. Each new application had broad direct and indirect effects on our society, and created massive amounts of new economic value.

The technology of oil production also made some people very very wealthy -- the Hughes fortune, for example, came not from the ownership of oil fields but from the manufacture and sale of innovative drill bits and well equipment. However, as innovation solved the initial set of hard problems, the value slowly leaked out of the tools and equipment. They transitioned from exotic, differentiating advantages to everyday appliances, utilitarian, indispensible, but simply table stakes for an oilman.

I submit this is what the emergence of OSS represents to the software industry. The hard problems of multi-user operating systems, of relational databases, of office productivity, of systems management, and the like, have been solved. Or at least the solutions to the problems that most people want to solve most of the time have been solved. Those tools are no longer exotic. Linux doesn't drive innovation in and of itself, no more than Sugar does. They adopt and implement the hard-won lessons learned in the R&D labs, Comp Sci departments, and IT departments over the past forty years.

This is in no way meant to denigrate what these OSS projects represent -- only a fool spends time reinventing the wheel. And if these OSS projects do not innovate at the core of what they are, by standing on the shoulders of giants they free up energy and resources to innovate all around the OS or the database or the office suite. Building a new consumer electronics device? Grab an embedded Linux distro. Want to host Web Services on it? Embed Apache in it. Tie it into live, network-delivered customer service? I hear SugarCRM works quite well in a hosted mode, maybe you can leverage that and extend the customer-facing function onto your new toy.

Solving the hard problems doesn't mean the solution becomes a commodity. It just becomes commonplace. When formerly exotic tools become commonplace, it enables exploration at deeper and deeper levels, unlocking the potential for value creation in previously unimaginable ways.

Why the enterprise software industry should be afraid

What I like best about 37Signals, poster child of the Web 2.0 "movement" because in an odd, tangential way, it captures the essence of the Internet-flavored gung-ho attitude that is fueling the trend.

As a complete aside, check out the etymology of phrase gung-ho. Who knew that the most reproduced GI Joe figure was named "industrial workers' cooperative?" Or, for that matter, that a kiwi founded early industrial unions in China? File under "Truth stranger than fiction." For the record, I am using the phrase in the "work together"/"we can do it!" sense, not the GI Joe, workers' collective, or even anime sense.

Anyway, what I find is the distinguishing trait of the Web 2.0 crowd is a determination to return to the roots of the founding of the Internet, with its insistent focus on the simplest solution for maximum leverage of affect. This set the Internet right from the start at odds with the cathedral builders who had brought the computing industry into existence. Not surprising given the animosity between the hackers and the CS department at MIT that tilled the soil from whence the Internet would spring.

When the enterprise IT world woke up to the Web in the 'nineties, a slow process of subverting the "more with less" thinking of the Internet began. E-mail became Outlook or Notes, HTTP servers became application servers, awk and perl became Visual Basic and Java, and the whole thing came to an overly-complex head around J2EE and .Net. The entire transformation was in the service of adding complexity to justify IT investments by the customer firms of the IT industry. Enterprise License Agreements, upgrade cycles, professional support and services, bigger servers, more storage, faster, newer, more spending in a relentless cycle. Sometime around the dot-com bubble bursting the enterprises paused and asked "what am I getting for all this money, exactly?"

What they found is that quite often they are forking out millions of dollars in software licenses to fix the problems created by last year's spending on software licenses. The original "sure, we can do that" spirit of the Internet was dead.

Wait, or was it? The argument is often raised that Web 2.0 is nothing new. That argument is correct, there is nothing new in the collection of technologies and trends collectively referred to as "Web 2.0." At its heart there is something actually quite old - the original spirit of harnessing just the right amount of technology to get the job done. Not enough to justify charging the customer more than you did last year, perhaps, but just enough. And so the software industry faces a threat from "good enough" thinking once again, this time dressed up with rounded corners, neat acronyms that evoke mythical heros and cute made-up names that could be Star Wars characters.

In fact this is a very old struggle, going back through Bill Gates' angry letter to the hobbyists telling them that if they don't stop copying software, no-one will get rich; through the MIT hackers leaving their source tapes in the desk drawers for the next person; through all the way to deep, ancient issues about human ownership, dominion, rights and privileges. But we'll leave the mythic roots for another day.

I found this slide on 37Signals site, in a discussion thread about the relative merits of this slide's design and message. The accompanying discussion centered on its obvious PowerPointiness - "it's too vague," "it's marketing fluff," "programmers suck!" "designers suck!" etc. All the while missing the simple message conveyed by the slide. Marketing message aside, (I know nothing about the AppSight BlackBox software from BMC that this slide is promoting and therefore am ignoring the middle bit with the "Hummer tires" as one comment put it) the important point is: if you dedicate 80% of your time to simply thinking about and analyzing the problem, and 20% to the fix, you will save 70% off your time, cost and effort. I believe this 70% to be a magic number to watch in particular. More than once I've heard CIOs mention numbers in this neighborhood as the potential savings off their current IT budget that they hope to realize by taking another look at how they use technology. A slow-boil approach that emphasizes thinking deeply about the problems and goals, then attacking the fix with a "less is more" attitude.

A market opportunity that is up to 70% reduced, that's a trend about which the enterprise software industry should be very nervous.

There is no Open Source bubble

I've heard rumblings from some quarters about a return to the salad days of the '90's for the IT industry, this time with Open Source and Web 2.0 inflating the bubble. Setting aside the question of whether the dot-com bubble was something to which we'd actually want to return (it isn't), the possibility of a repeat is pure fantasy.

The rocket fuel at the core of the dot-com bubble had two main constituents: first, the availability of the IPO as an exit. A public offering made the VC a 10X return on his investment, made investment banks mountains of cash for nothing, and allowed institutional investors and fund managers to pad their return rates and bump up their bonuses and credibility.

Second, the fear of the telcos that they were becoming obsolete. In the early days of the Web the telco, the enterprise and Wall Street didn't take it or firms based on Web technologies seriously. The telcos in particular allowed ISPs to spring up using their phone lines, IP-based network equipment and Web-based businesses to develop, all without lifting a finger. Probably because that finger was bloated on all the free cash on which the telcos were gorging.

Then, seemingly overnight, the enterprise woke up to the fact that thousands of web servers were springing up internally, matching the grassroots growth of Web-based companies like Yahoo!, Lycos, and Amazon. The IT analysts like Gartner and Forrester smelled enterprise IT spending and started talking up the Web, VCs began funding an increasing number of companies with ".com" at the end of their names, and The Street started taking the whole phenomenon seriously.

The Street's interest was really the boost that got the bubble truly under way, because it was their efforts to promote the IPOs they were underwriting to their big institutional investors as a quid pro quo that got the telcos annoyed. Here were nothing little software companies like Netscape with market caps that rivaled the firms whose phone lines upon which their business model depended. The telcos struck back with the ridiculous "ASP" story around 1997, claiming they were going to sell enterprise applications as services to the biggest companies on the planet. As soon as they could figure out their billing systems. And build the infrastructure to host the applications. And figure out what an enterprise application was, and where to find one. But anyway! The important part was the "build the infrastructure" stuff, because it meant a massive surge in telco spending on IP-based equipment and Web technologies attracting the VCs and investment banks. Sure, there were detours down deadends like ATM and WAP, but it was still money. The quote that is seared into my mind from this period was from the CEO of a systems management company who, during a workshop on future scenario planning, when the opinion was expressed that the telcos would all fail at their ASP plans, said, "sure, but they're going to spend a lot of money failing." Which to me sounded like a business plan focused on polishing the marble floors of the ballroom in the Titanic.

The other crucial ingredient brought by The Street and their promotion of the building pipeline of IPOs was new money the equity markets, supplied by a new class of investor: Joe and Sally Lunchpail. The massive influx of capital into the equity markets triggered by the migration from pension funds to 401(k)'s and mutual funds was encouraged to play the Las Vegas version of The Street. High risk, high gain lottery tickets that got all the publicity like Amazon and Netscape, where an individual could make 40% gains or lose everything. Of course, the banks didn't fully disclose the way underwriting works - the bank gets to pre-allocate big chunks of the offering, handing out candy to the Big Boy customers to lure in more of their money. That meant that by the time Joe and Sally got a chance to buy shares the price was already inflated. The Lunchpails paid a premium and by the end of their buying wave the Big Boys would start to take profits, sucking the momentum out of the overvalued stock. In the technical argot this is called a Ponzi scheme, more colloquially "bait-and-switch."

This long-winded analysis of the dot-com bubble is really just a setup to explain why Open Source and Web 2.0 won't create another cycle of overvaluations and overnight millionaires. In case you haven't noticed, there isn't an IPO market, so that exit is effectively closed to startups. The Lunchpails licked their wounds after the 2001 bust, then started sinking their money into their homes and real estate in general. That's the current bubble that's about to burst. With middle class families' "savings" tied up in increasingly illiquid assets, there is little possibility of pools of capital flooding the equity markets and goosing the growth rate.

With no 5-year path to a 10X exit, the VCs are not nearly as excited about startups these days. In fact being a VC is much less glamourous in the 'naughties than it was in the 'nineties. Frankly, from the VCs I have spoken with it sounds like it kind of sucks. While The Street would like the VCs to find the golden egg that would hoover more pocket change out of the heartland and into their bank accounts, they still have lots of M&A activity, consumer spending, real estate and energy prices to drive their income.

So what is behind the buzz around Open Source and Web 2.0 startups, then? It is a little more than a nostalgic wish to return to "the halcyon '90s, when Pets.com ruled the world." There is an exit that allows a lucky few to realize some significant gains, but the emphasis is on "few." Both Open Source and the Web 2.0 obsession with quick, simple results threaten the incumbents in the IT industry - IBM, Microsoft, SAP, and Oracle. These giants are slugging it out as the industry consolidates around three big platforms - .Net, J2EE and SAP itself - and the last thing they want is to have to fight over a diminishing pie. But that's exactly the net effect on the market of the commoditization effects of Open Source and the 'do more quicker with less' approach of Web 2.0. Enabling the big guys' best customers to spend less on IT and get the same results is the best way to ensure an exit through acquisition. The buyer can be the incumbent you are hurting the most, the competitor most interested in hurting the incumbent, or a 2nd or 3rd tier IT company seeking a way to be more attractive to one of the big platforms and therefore ensure either their continued existence, or future acquisition for a bigger price tag. That was the force behind Red Hat's acquisition of JBoss, IBM's acquisition of Gluecode, Oracle's near acquisition of MySQL, and many more acquisitions to come.

If it sounds like I'm describing a world where little fish are gobbled up by successively larger fish, well, that's what a consolidating market looks like. Two important points, to circle back to our topic: first, that exit by acquisition nets 2-4 times your invesment, maybe 6X tops, but not the astronomical returns that a 'nineties IPO reaped. This is bad news for the typical VC portfolio management model of "one grand slam, two singles and seven strike outs," which depends on those 10X grand slams to even out the losses from the rest. Second, being a consolidating market, there is a finite and rather close horizon for this acquisition market and the exit it represents. So instead of a throwing a lit cigarette into a National Park, this flurry of activity is more like holding a lighter to a pile of trash in an barrel.

Of course, this analysis is not directly addressing the new face of the IT industry: the Google/Yahoo/eBay struggle. For the moment I think Cringely has written most of what needs to be said on that topic, and for those interested I can think of no better essays to read.

Communicating

I am the product of two extremes - my father's family only gets together once a decade, at most, and if they are in contact with one another in between those instances it is an occasion for maximum awkwardness, guilt and discomfort. My mother's family has organized weekly teleconferences in order to plan for an upcoming semi-annual reunion, and if more than a week goes by without some contact among any given pair of the nine siblings, it is an occasion for growing anxiety and guilt until the breach in communications has been repaired.

Is it any wonder, then, that the degree to which I stay in contact with friends and family varies wildly and wanders aimlessly from diligent and thoughtful to callous and distracted?

What does it mean when we fall out of touch with a friend or family member? Does it mean that we don't care for them anymore? Or does it merely mean we don't have enough time to spare them? Is that prioritization something that should be taken personally by the out-of-touchee? In other words, should the radio silence be interpreted as meaning that everything else in that person's life is more important than you are at this moment?

Of course this is an absurd point to contemplate. Of course the other aspects in another's life are more important to them than you are. They are the demands on his time, the requirements of his job, the needs of his children, his spouse, his own needs, desires and goals. You are external and peripheral to all that, and by definition take a secondary role in his life.

In fact, this is the root of my suspicion of those who are so diligent with their communication with others. There is a dishonesty at work, implying that you do occupy some level of importance in this person's life that demands regular and intense sessions of communication. In fact I suspect that one of the person in question's own intimate needs to be fulfilled is to feel as though he is doing a "good job" with keeping in touch. If he did not prioritize the weekly phone calls, the semi-annual reunions, the special trips and the like, he would feel the other party would think him negligent in his communication duties, and would feel bad about himself. In other words, the real priority is always the self, not the other. Interpersonal relations quite naturally obscure and confuse this basic point of human nature, because it entails a commingling of two impulses with diametrically opposed goals - each to fulfill its own, self-contained needs.