Identifying The Blind Spots

The Computational Beauty of Romanesque Cauliflower

The New Computer Scientists

A new article out from BusinessWeek entitled "Math Will Rock Your World" is well worth reading. It talks about the huge business opportunities for the mathematically inclined:

The world is moving into a new age of numbers. Partnerships between mathematicians and computer scientists are bulling into whole new domains of business and imposing the efficiencies of math. This has happened before. In past decades, the marriage of higher math and computer modeling transformed science and engineering. Quants turned finance upside down a generation ago. And data miners plucked useful nuggets from vast consumer and business databases. But just look at where the mathematicians are now. They're helping to map out advertising campaigns, they're changing the nature of research in newsrooms and in biology labs, and they're enabling marketers to forge new one-on-one relationships with customers. As this occurs, more of the economy falls into the realm of numbers. Says James R. Schatz, chief of the mathematics research group at the National Security Agency: "There has never been a better time to be a mathematician."

While it gets the vision thing mostly right, it drops the ball on a key point: it's not the people coming out of mathematics departments that are primarily shaking things up. It's the computer scientists who have mathematics training, people with backgrounds in fields like machine learning, data mining, computational statistics, and computational social networks. You just have look at the technical and research heads of companies like Google, Yahoo, Amazon, and Microsoft to see that this is true.

To make this mistake is to miss the radical transformation that has taken place in computer science departments across the country. While a decade ago, 90-95% of all computer science professors were working in the areas of theory (e.g. discrete algorithms, computability), hardware, database, and languages, now you have 20-30% of the people working in new applied fields like computer security, bioinformatics, computer vision and machine learning, many of which require strong mathematical training.

The Great Google Heist

Ever since reports started coming out that there was massive click fraud taking place on Google's AdSense platform, I remember thinking that it was the end for Google. After all, virtually all of Google's revenues come in the form of what are essentially commissions for getting people to look at other companies' advertisements. And given how easy it is to build a simple web bot to randomly click through those ads, I had a hard time understanding how those clickstreams weren't getting polluted enough with massive amounts of phony clicks that the companies bank rolling Google's financial future wouldn't be up in arms and all but drop Google in the water. I have since come to the realization that Google must have some very smart folks, probably dozens of PhDs, working on combatting click fraud using intelligent statistical algorithms and luckily have thus far have been able to stem the tide by deducting the majority of any revenues coming from fraudulent clickthroughs as chargebacks. But still no algorithm will ever be 100% fool proof against click fraud, not even close.

I mean let's think for a moment about what kinds of things you have at your disposal to combat click fraud if you're Google. You can look at where the ip addresses are coming from and thus set up detectors to monitor any clickstreams above a certain threshold all coming from the same ip. You can look at who the referrer is, the page that was visited before clicking on the ad, and assume any clickthroughs that didn't have accompanying referrers are fraudulent. You can look at historical trendlines and detect any highly bursty clickstreams whose volumes are extreme and assume that some of those ips are probably fraudelent and should be further investigated. Probably the best way to do it and the way I would imagine Google is doing it collect a large historical dataset of traffic signatures for ips that are known to be real humans as well as  those that are known to be fraudulent, and build a machine learning model that will automatically classify ips as being either fraud or not fraud.

But even if you use advanced machine learning algorithms, it will always be a game of cat and mouse since 1) machine learning algorithms are trained on historical data and thus can only detect patterns it has already seen and 2)  if fraud signatures are indistinguishable from non-fraud isgnatures then they can't be detected. So all a fraudster has to do is figure out what the signature for a non-fraud ip looks like, e.g. no more than one click per ip of a given ad and no more than twenty ads clicked on per ip per day, and he can always stay under the radar. See this article for an example of what I'm talking about.

Now one might think the hardest part for an individual fraudster may be collecting enough ip addresses so that each one can stay below Google's radar yet the fraudster could use them collectively to perpetrate massive amounts of fraud. But this is simply not the case. Talking about the why IP banning is ineffective for stopping spam, Adam Kalsey writes:

... IP addresses are very easy to get or fake for spammers who care about such things. There are hundreds of thousands of open proxies that will let anyone direct Web traffic through them. When I’m using an open proxy, my IP address is effectively masked. And I can use simple software to switch to a different open proxy (and thus a different IP address) every few minutes. So my spamming activity isn’t tied to a specific IP address.

Hypothetically speaking, if the problem of open proxies were to disappear overnight, there are two other mechanisms that provide a limitless set of IP addresses to spammers: dialup and spoofing.

Which all leads me to believe one day a rogue trader and a rogue programmer (might even be the same person) might conspire to carry out what I call "the great Google heist." The idea is simple. Slowly accumulate millions of dollars worth of of out-of-the-money puts on Google under different false pseudo-names with long enough expiration dates for the earlier trades so that they don't expire too soon. Simultaneously pollute Google's ad clickstream so heavily, while staying below Google's fraud detectors, that Google's ad revenues rise far above what they were before the heist. Then wait. First for Google's corporate clients to start complaining and eventually drop Google in the water for the ridiculous amounts of money they are being asked to pay. Then wait for the news media to develop such a frenzy around the issue that many of Google's shareholders feel the panic to such an extent that they begin to sell their shares. Once the whole situation hits a blazing crescendo, start selling all of those puts.

While it is relatively unlikely that such a scenario will ever be carried out as I have envisioned it, given the likelihood of getting caught, the fact that such a scenario can even be envisioned has always made me scared of ever buying any Google stock. While so far it's all been to my detriment, I still contend that Google, so long its has no alternative sources of revenue, and even if had a P/E ratio closer to 30 instead of above 100, will always be an extremely risky investment.

My Predictions for 2006

There's nothing worse than predictions that can not be meaningfully validated. Saying 2006 will be the year of hydrogen is not a prediction, I'm sorry to report to all you would-be Nostradami. Having that out of the way, here are my top 10 predictions for 2006:

1. Total revenues generated by the sale of smart drug modafinil will exceed 1 billion dollars in part due to its effectiveness in treating one of the fastest growing social epidemics in the US: information overload. (Here's an excellent article on the subject.)
   
2. A major high-tech manufacturer will introduce a sub $500 cell phone that can store roughly one year's worth of phone calls (over 10K minutes). Although it will need to be recharged much more frequently than a normal cell phone, people can use it to automatically record every voice conversation they have ever had since using the phone. Lawyers, of course, will recommend this feature be turned off.
3. More than 3 of the 10 most highly cited scientific papers will be published in online journals that allow authors to retain copyright ownership.
4. More than 40% of all US science/engineering PhD graduates will be foreign born increasing from 36% in 2000 and 23% in 1966.
5. The average rate on a US 30-year mortage will be over 7.25% (which is slightly less than one more point than it is now).
6. Eliot Spitzer will be elected governor of New York and Arnold Schwarzenegger will be ousted as governor of California.
7. According to Alexa, Wikipedia will enter the elite top 10 club of most highly trafficked web sites on the Internet, moving up from it's current position of 34 (as of the time of this writing).
8. At least one of the major search engines will be able to explicitly distinguish between people, places and organizations in their search results.
9. The number of man-made catastrophes will for the first time since the 1970s be more than double the number of natural catastrophes. However, the number of lives lost due to natural catastrophes will continue to exceed the number lost to man-made catastrophes by a factor of 10. Catastrophes here are defined the same way they are by official Swiss Re reports.
10. Contrary to my friend Paul Kedrosky's prophecy, my employer Amazon will not get bought out by Walmart. On the contrary, I've done the math and forecast revenues for Amazon which are currently just under 8 billion a year will exceed those of Walmart, currently 305 billion a year, by 2018, a long ways off I know :)

On Model Building

Some people think building a good machine learning classifier is easy. You just grab some data, the thinking goes, do some feature selection, if necessary, pump the data through some black box classifier, such as a naive bayes classifier or a decision tree, and presto, you have yourself a model.

In fact, nothing could be further from the truth. Building a good predictive model is often painfully hard work. The more common scenario I find in the business world is you have gobs of data, more than you would ever know what to do with. Not only that but you have an incredibly large number of potential predictors you can draw from, often in the hundreds, sometimes in the tens of thousands.

Where things get difficult is that most of the data, in its raw form, is often unusable, containing little predictive value. So what is required is knowledge of the domain and a sometimes painstaking process of examining the variables you have and figuring out the correct representation, i.e. how to transform those variables into variables that are predictive. In my opinion, this process is not only the hardest part, but also is critical in squeezing every last bit of accuracy into the model and taking what would otherwise be barely better than using some common sense heuristic rules into one that is gold-standard, leaps and bounds better than anything that could every built manually without using machine learning techniques.

Unfortunately, assuming you have all the right variables in place, you're barely half way done. In reality, you have to go through many iterations of of tuning the classifier. There are many different kinds of tuning that may be required. They include making sure the model does not overfit the data and making sure the learning gradient is as steep as possible so training finishes as quickly as possible.

Actually the type of tuning most people forget to do is by far the most important one: calibrating the model to the real world. This means, for one, making sure the predicted class distribution matches the actual class distribution. Novices often forget this, run their model against a validation set, see that they get a very high accuracy and think they have a winning model when in fact all they have done is learn to say "yes" on every decision where 95% of all decisions are yes.

Beyond that, you also have many other things to take into account, including the cost of different types of incorrect predictions. Sometimes these costs are equal but not always. The 5% of cases that are no, in the last example, could be worth a lot more than the yeses in terms of cost however it's defined.  Building a good cost model can in some cases be extremely difficult and can require a lot of tedious analysis.

Of course, all the satisfaction comes at the end once you've built the model, deployed it into production, and are able to verify that yes the damn thing actually works.

The Nature of Incumbents

There is a telling paragraph at the end of the first chapter of "Seattle and the Demons of Ambition: A Love Story" written by Fred Moody (same guy who wrote "I Sing the Body Electric") that speaks to the question of how revolutionary, young startups are often perceived and actually change once they become the incumbents they sought to replace, having succeeded in their quests to radically transform the marketplace:

"I was standing within canister's throw of four Northwest companies - Nordstrom, Nike, Starbucks, and AT&T Wireless that had  all at one time  been brash, romantic startups determined to rebel against the status quo in their businesses and deliver something previously forbidden to the beleaguered and deprived consumer-citizen. Now, all four were reviled as oppressors of customers, competitors, employees, former employees, contracted third-world employees, or all of the above. I remembered too that Microsoft, Amazon.com, and McCaw Cellular (before AT&T bought it and turned it into AT&T Wireless) had once been popular Seattle startups, freedom fighters in the corporate age, wresting power over information and communication from the hands of previously indomitable corporations and putting it in those of ordinary citizens."

It's an interesting point that Moody makes and I think it can be broadened outside of the Northwest.  In the same way people forget that every religion was once a cult, people often forget that virtually every large company that people so despise for one reason or another was once a revolutionary startup cheered on by the masses to replace the incumbents that came before them.

In many cases, I think, it's the story of youthful idealism being corrupted by the forces of power and greed and the way companies are perceived aptly mirrors the reality of the situation. Think Walmart, Microsoft (at least leading up to the antitrust case), Enron, and so forth.

But then there are the other cases, where perceptions are misaligned with the reality on the ground. Take Amazon, for example, a company I happen to work for. There is a lot of resentment by people against Amazon because they have supposedly driven many small bookstore owners out of business. But what these people may not realize is that they allow businesses of any size to sell their products on Amazon's website so a customer can choose to buy a book (or any other product) directly from Amazon or alternatively pay a lower price for a used or new one  from a small mom and pop. Perhaps Amazon is evil in other ways I am unaware (that they have flaws is not in question), but at least in this regard there is a large gap between perception and reality.

A Study In Pattern Recognition

Just a few weeks ago I made one of the biggest hedges I've made in my entire life. I decided to quit my job as senior manager of a small text mining company in San Diego, move out of my house, leave my friends and relatives, and move all the way up to Seattle where I know hardly anyone and work as a machine learning scientist at Amazon. What I find particularly surprising about the whole experience is how rapidly I've adjusted to my new surroundings.

For example, the whole process of moving one's self, wife, child and belongings thousands of miles north seemed to have gone by in a heartbeat. My new digs with views of the Seattle skyline and overlooking the Puget sound seems to have already lost its sense of newness without diminishing how exhilirating it is to see the barges shipping thousands of tons of goods in and out of the harbor, the ferries taking passengers to and from all of the islands in the sound, and all the other boats leisurely meandering this way and that.

Although it's only been a few weeks, somehow it feels like I've been walking a few blocks to Pike's market for years to pick up some of the spectacular culinary delights offered there whether it be a mouth-watering piroshki and borscht soup for $5 from the Russian bakery, Piroshki Piroshki, or one of Uli's famous sausage sandwiches for $6, or a baguette and cheese from Beecher's Handmade Cheese where you can watch the cheese being made the old fashioned way right there in the store.

Even though I heard about the extremely high percentage of bookstores and coffee shops in Seattle, I never imagined seeing so many pubs, bars, and restaurants with a literary theme. I'm thinking of places like Library Bistro, Bookstore Bar, and the antique bookstore in Pike's market, whose name I cannot recall, with a full bar that is open on weekend evenings. But now that I have, it's all seems like part of the normal fabric of life in Seattle.

And when I take the ten minute bus ride to my office at Union Square in the international district of Seattle and people other than low income workers ride the bus, not only does it feel refreshing that public transportation is not stigmatized the way it is in San Diego, but it feels like this is how I've been commuting for years.

I certainly never thought the time would come that I would have gear for different kinds of rain. But now that I have my thick down jacket, my lightweight Goretex jacket, and all the other gear that I do, nothing seems out of the ordinary. When I walk outside, the cold November wind that streaks against my face and the light drizzle that sometimes beats against my head feels like it's been blowing and beating like that for time eternal.

I think part of the reason I've been able to adapt so quickly to my new surroundings is that all of the experiences that I'm having as a new Seattelite are not really all that new. Between the long stretches of time that I've spent in various cities and countries, with their own climates, cultures, and ways of life, in one way or another I've experienced it all before, perhaps not exactly the same but close enough to it that my brain is able to recall what it was like and thereby lessen the sense of surprise and sometimes shock. This isn't to say that there is nothing new to experience here in Seattle. Quite to the contrary.  Every day feels like a new adventure where my brain is constantly overwhelmed by all of the familiar patterns.

I sometimes joke that Portland is my new TJ, Vancouver is my new LA, and Montana is my new Arizona, part of which is to say you have to like water to live up here.  Actually it's also partly a boast since I'd choose Portland over TJ and Vancouver over LA any day.  I've been in Seattle for just over a few weeks now and despite the rainy weather I think I am happier here than I've ever been since living in Paris in my early twenties or Germany in my middle teens. Part of it's the adventure. But also part of it is that Seattle really is a spectacular city. In fact, I've come to realize that it's Seattle's rain that not only keeps everything a lush green but also acts a sort of gatekeeper keeping those afraid to to get a little wet from being able to enjoy all that Seattle has to offer. And should the secret ever get out, that it doesn't actually rain as much, at least in total rainfall, as most people might imagine, I suspect the barbarians would be at the gates and all the empires, be they culinary, musical, literary, or outdoors, would start to crumble. So when I read a Seattle blog talking how about the supposed real estate bubble in Seattle, specifically how absurd the growth rate has been in real estate here over the last few years rising as high as the low teens, in terms of the year-over-year median sales price , rather than dismiss the outcry, having lived in San Diego where growth in the mid to low twenties was the norm for a good half-decade and housing prices are about 40% more expensive than they are here, I find myself in complete solidarity.

Do It Yourself Donations

Why is it that people always feel the need to give used clothing and uneaten foods to a third party intermediary rather than give it directly to the poor? It's not like they are hard to find.  Perhaps they think they know what would happen. They start unloading food and clothing from their SUV and next thing you know there is a band of thugs hijacking their car, placing them at gunpoint, searching them for anything they can use to buy themselves a bottle of whiskey or a heroin fix. In certain neighborhoods this may very well happen, but not in any of the ones I've lived in.

When I was moving out of my house in San Diego, I walked down to the local Vons one morning with a bag of used clothes and another bag of food where there was a recycling center where the homeless would congregate to receive a small amount of money for collecting trash bags full of crushed cans. I offered them what I had and apart from some collared shirts and shorts which none of them had any use for, about half a dozen men took just about all the clothes I brought. Because some of the men had very few teeth left due to the fact they had gone for so long without healthcare, they couldn't accept some of the food I offered them such as some apples I was giving away. But all the food was taken and they evenly distributed it amongst themselves. They were all extremely thankful and appreciative. Actually later that day, I saw one of the same men I met earlier, wearing the red sweatshirt I gave him.

Compare this experience with what normally happens. You call some non-profit to come and pick up your stuff and that's that. You never hear from them again unless they call to see if you have more stuff to give away. You never find out who received any of the stuff you gave away and whether they liked it or not. As far as you know, all the stuff could just be sitting in a giant storage room.

I'd love to see a day when many more people apply the DIY mindset to helping those less well off than themselves. Especially if it involved acting locally and helping any people they might encounter on a daily basis, such as the guy always rumagging through trash at the supermarket.

It's Not Easy Letting Go

There is a deep rooted human instinct in all of us to not want to make hard choices if they can avoid being made. One trivial example is shopping. We often buy clothes we don't actually wear but think we might some day only to acquire a warddrobe full of a few things that we wear all the time and full of many things we hardly ever wear. It is only when placed in different circumstances that require us to make hard decisions, such as what to pack when traveling abroad, that we acknowledge this reality and learn to our surprise that not only are we capable of getting by on a small subset of what we thought needed, but life is so much simpler and worry free when you have fewer things demanding your attention and cluttering your life.

One of the reasons we choose not to make hard decisions is that as humans, we are terrible predictors of our own future circumstances. With so little control of external events and even our own future moods and whims, it is understandable that we take on as much as we do. "Who knows what good tidings may come our way if instead of narrowing future possibilities we expand them?" the thinking goes. "Never mind the extra burden of taking on responsibilities that in the end will prove needless."

But viewed in terms of the total amount of time and energy that ends up going misspent, the consequences of our unwillingness to make hard choices is staggering. All the easy choices we gave in to at some point or another eventually end up cumulating in our information delivery boxes, whether it be our mailboxes, voicemail, feed readers, email inboxes or cognitive memory, as what is essentially spam, whether or not we can recognize it as such at the time. The way to tell is if it ends up forcing you to incur an opportunity cost, distracting your attention away from something you were doing before that was not only interesting but also oriented towards a worthwhile goal. Because we fail to make the hard choices early on when it counts the most, we end up paying the price of having to contend with a growing stream of unnecessary distractions, each one tearing out a little piece of us.

That's why I'm a big fan of things like the slow email movement, the slow blogging movement, taking time to actually read selected books rather than thumbing through magazines, not watching television unless there is something specific I want to watch, and generally not committing to things unless I'm sure I can follow through on them. I'm all for having fun and seeking out stimulation. But I'm also for keeping things fulfilling and purposeful.

A Question For Ebay

I don't use Ebay a lot and my experience with it in the last few weeks has been frustrating to say the least. I'd been trying to find a used high quality cyclocross bike frame for a good price and had made probably close to  a dozen bids, only to find myself each time being outbid at the very last moment by someone who had not previously made any bids. In some cases, my max bid was way too low and I was hoping for a real bargain so I have no right to be frustrated by such a maneuver. But in many cases, I was quite surprised at how much was being paid for some of these bike frames. For example, I saw a used frame that was four years old from a good brand but which was in not so good shape sell for about $305 when the 2005 model brand new model cost only $370.

This got me thinking about the following question: do people have any sense of how much something is actually worth? Observe that answering this question requires two things: 1) knowing how to determine the worth of something and 2) determining whether people have a sense of that worth. Most people, I think, have the following notion of worth: something is worth whatever is lower: the cheapest price for which the thing they want to buy is available or the price they are willing to pay. In cases where it's easy to compare competing prices, I think the answer is yes, most people do have a pretty good sense of how much something is worth since they can compare the cheapest price they can find for what they want to buy across many different vendors and evaluate that vis a vis how much they are willing to spend. The fact that prices tend to cluster around a narrow range generally gives a good indication of the costs involved in building, distributing, and marketing something.

But where things get trickier is when the product is used and there is very little data on bid and ask prices. For example, try typing in "1999 used Gunnar Crosshair Frame" into Ebay and you will likely get 0 responses because it is very hard to find this frame used let alone for that year. The fact that a product is used makes its worth that much harder to determine when price data is hard to come by. Every product has its own price time decay behavior and a lot of knowledge is required to determine what the time decaying path of a product's worth should look like. So if you see a used niche product come on the market and you want to evaluate what your max bid should be, you're most likely not going to be comfortable that you have a good handle on how much such a frame is actually worth. Put differently, if someone gave you the frame and you had to price it at a fixed price, your sense of uncertainty would likely be quite high. Thus for markets where bid and ask data is hard to come by I think people's sense of worth is quite poor, doubly so for products that are used.

There are many businesses, not just the used, niche product business, where getting good information on prices is hard to come by. Any business, product or service where an average person with access to the Web can not figure out how much something is worth within a reasonable amount of time is a business you don't want to be transacting in unless you know what you're doing or you can afford to get ripped off.

But imagine if Ebay allowed you to search on how much any product that had ever been sold on Ebay sold for. If someone was able to find a long history of prices for which something he wanted to buy or sell was sold for in the past that would dramatically decrease his sense of uncertainty. It seems to me decreased uncertainty would in turn increase volume of transactions since fewer people would be hesitant to make a bid or sell an item and instead would be more confident about the bids they were making. So the question is why doesn't Ebay make this data available? Is it for prosaic reasons, e.g. they don't want to spend the money on additional hardware, or is there a deeper motive? I for one would love to know.

Brief Thought On Global Pandemics

If there's one piece to read on the current bird flu pandemic scare, it's this one.

One thing that I was continually reminded by in reading this article is how much of the reason for the growing rampancy of this and other viruses can be attributed to the squalid conditions of the developing countries in which the virus was able to succesively mutate, often across multiple species, unchecked. 

In this new era of globalization in which we all now live, where everyone is connected through just a few degrees of separation, the butterfly effect has an important corollary:  one man's cough is another man's illness, even if the two are separated by thousands of miles of land and ocean.

When the founders of the United States declared that among man's inalienable rights is the pursuit of "life, liberty and the pursuit of happiness," they left out the part about access to modern sanitation, clean water supplies, and education about  good hygienic practices. Because the less everyone has access to these, the less anyone else can ultimately be reasonably assured they will be able to live at all let alone live happily or freely.

 

Is it 1999 all over again?

Brian Dear has a great post on the infectious greed permeating the Web 2.0 crowd. It reminds me of something Tom Wolfe or even Michael Lewis would write. But instead of the big swinging dicks in Liar's Poker who were greedy, take-no-prisoners bond traders, the characters in this story are greedy nerds looking to flip their toy company for a quick buck  even if it means having to explain to their employees that the whole change-the-world vision thing they were pitched to get them to work crazy hours was, in the end, just a gimmick.

One of the problems with flipping companies is that acquisitions almost always are much worse for the employees than they are for the management of a company. Because after all, the 4 years that employees have to put in to get their options vested typically get revested at the time of acquistion for another 4 years, meaning in the worst case 8 years in total vesting time. I have met only a few people, and I have met and know many, who after working for an exciting start-up that was subsequently acquired, did not feel the change-the-world adrenaline-filled sense of anything-is-possible come to a screeching halt, only to be replaced by a morbid feeling that life was too short to stick around at a larger, bureaucratic company to recoup the capital that they had worked so hard to earn. For the few who do end up sticking around, those last 4 years generally tend to be very bitter and unhappy times.

Whether or not Brian's observations are fair in their characterizations, there is no doubt that the many of the ideas coming out of the Web 2.0 community are going to have a huge impact on the future of the Web.

The Pen is Mightier Than The Sword

Apparently the whole DeLay indictment has some at Blogs for Bush actually talking about civil war. Hunter at Daily Kos responds with the best partisan political rant I've read in some time.

Gerin Oil

Richard Dawkins has an article in this month's Prospect magazine which is well worth reading, if nothing more than for its unique literary approach. In it he argues, that Gerin oil, a highly addictive drug, was responsible for many of the world's worst calamities, including September 11th, the Salem witchhunts, and many very bloody wars.

If you haven't heard of Gerin oil before, try reading the piece a second time. Make sure to "free your mind."

Collective Imagery of Katrina

One way to experience viscerally the devastation of Katrina is to watch this slideshow over at Flickr of tens of thousands of images collected over the last week and a half. You can adjust the speed in the top right to 1 second per image and minimize the window so it sits in the corner of you screen. After watching this for an hour or so, I think you can get a pretty good sense of the devastation inflected as well as the hardships people are going through.

Here are just a few images I found particularly striking:
Dead Man Watched On By a Dog
Crying Boy in A Shopping Basket
Woman In Curlers Recovering at the Astrodome
Bush Playing Guitar In Front of A Crying Mother
Last Of the Holdouts
Two Families Waiting For Help On a Roof
People Clammering To Get On Buses
SOS Sign on The Roof Of A Flooded House
Restaurant Gone But The Sign Remains
Girl Who Lost Her Parents

Multiply Your Donation

How to Become a Hero has compiled a list  of organizations that have matching programs for Hurricane Katrina disaster relief. Before you donate directly through organizations like the Red Cross, you should think about going through one of these matching programs since your donations will instantly be doubled.

The Aftermath of Katrina

Six months ago Goldman Sachs predicted there would be a multi-year super spike in oil prices with prices reaching $105 a barrel or more within the next few years. At the time, such an estimate seemed over the top but looking at where oil prices are now and where they are likely to be in the short term, the estimate might turn out to be incredibly accurate. Consider this chart (another chart can be found here)  which shows inflation adjusted monthly crude oil prices since 1946 :

In just the last two months, since the last data point on that chart, July 2005, oil prices have soared from $50 a barrell to around $70 a barrel. Now consider that the Gulf of Mexico supplies 1.3 million barrels per day of the roughly 20 million barrels per day supplied to the nation in total. That is around 6.5 % of the nation's daily supply of oil comes from the Gulf of Mexico. Recent forecasts estimate that in the long term (over 30 days disruption), overall production coming from the Gulf of Mexico is likely to be reduced by 22.4% (85.6% in the next 10 days and 50.1% in the next 30 days). Who knows exactly what the figures will turn out to be. But assuming they are in the right ballpark, that means the nation's supply of oil is likely to be reduced by rougly 1.5%. This may not seem like a big deal but keep in mind that we're just talking about reductions in crude oil. It still is unclear what the reducion in refined gasoline will be and in the wake of a period where markets are stretched and supply/demand for oil is still very tight, this could have cascading effects shooting gasoline prices up to $4 a gallon within the next year. Several economists and analysts have made similar remarks in the press.

The question of how much damage Katrina will cause to the nation's economy relative to September 11th, which was in the hundreds of billions, is far from clear. On the face of it, it might seeem the damage is much less, but if we see cascading effects due to the oil shortages in the Gulf of Mexico, with the result being that Goldman Sachs' prediction comes true and oil-intensive businesses such as airlines companies get hit hard by a spike in oil prices, it is conceivable that the damage caused by Katrina could actually be comparable to that caused by September 11th. While I do not consider myself a fear mongerer and often find most doomsday scenarios absurd, given the current oil market and a distinct memory of what a severe blow to the economy 9/11 was, where most of the damage was through cascading effects, I am more than a little concerned.

Update: The Economist has a much more thorough analysis here.

The Q-Entropy Controversy

There is a good article in the New Scientist this weekend by Mark Buchanan about Constantino Tsallis and his work on non-extensive statistical mechanics and q-entropy. Buchanan, I think, does a great job of laying out the debate on q-entropy, fairly addressing both sides. For those not in the know, q-entropy is an attempt to generalize  Boltzmann-Gibbs statistical mechanics so that it can describe systems other than those which exhibit thermal equilibriums. Think chaotic systems where things like turbulence prevent the system from ever settling down. While there are a number of physicists who think this work is bunk, with criticisms ranging from complaining that the equations are unnatural to complaining that it's nothing more than fitting power laws to complaining that the equations have led to no new predictions, there are also literally thousands of physicists from all over the world who think this work is deeply significant not only for its theoretical interest but also because it gives them an arsenal of tools they can use to characterize phenomena that exhibit some degree of chaos that are outside the scope of traditional statistical mechanics. Since Tsallis first published his ideas, thousands of papers have been written on the subject and the community of physicists working in the area of non-extensive statistical mechanics has been growing leaps and bounds. This, of course, has those firmly based in the establishment up in arms since they can't help but pick up a copy of any of the top physics journals and not see some article on the subject. Having read Tsallis' draft letter to the editor of the New Scientist this weekend wherein he addresses the concerns of his critics, I am obviously favorable to his research programme even if I don't fully understand all the intricate mathematical minuatie. What is clear to me is that most of the critics haven't done their homework and simply have not read enough of the theoretical contributions to understand how rigorous much of the work is and also are not familiar with many of the predictions that have been made and then later empirically substantiated. There are still holes in the research programme, as Tsallis will be the first to admit, but the research programme is still early on and when looked at in the context of how long it took for traditional statistical mechanics to reach a level of maturity where most of the holes were kinked out, one shouldn't be too harsh.

Update: The New Scientist published Tsallis' response but chopped off about 90% of it. Some of the strongest arguments were excluded.

 

Do Real Estate Brokers Always Make a Bundle When Prices Soar?

Interesting article by Chicago econ professor Austan Goolsbee over at Slate about why real estate brokers are not getting rich off the housing bubble.  His argument is essentially that as a market ramps up more and more brokers get into the market and the # of transactions the brokers are able to complete goes down proportionately. Unlike law or medicine where it takes many years of education and training before you can even begin to see earnings, in real estate you can get a license to sell houses in a matter of months. He points to a study by two professors at Berkeley to back up his case:

A recently published study bears this out. Enrico Moretti and Chiang-Tai Hsieh of the University of California, Berkeley, studied the real-estate agent business in 282 metropolitan areas during a 10-year period. They compared agents in inflated markets to agents in flat-lining markets and found overwhelming evidence of the zero-profit condition in action. When housing prices rose, the number of agents did as well, and this, in turn, reduced the number of houses each agent sold by almost exactly the same proportion as the price increase. In Moretti and Hsiesh's data, for example, houses cost 5.9 times more on average in San Francisco than they do in Steubenville, Ohio. But the average full-time agent working in Steubenville sells more than 22 houses per year, whereas the same agent in San Francisco sells less than one-fifth as much.* The average income for real-estate work in the two locales is virtually identical. Moretti and Hsieh found that the direct correlation between housing prices and agent productivity held true across all markets. A rise in housing prices in an area has no significant impact on the average wage of the brokers in that market. It's the oldest line in the economics book: No barriers to entry mean no big profits.

While I more or less agree that brokers in fast, growing real estate markets may not be making as you would think off their commissions, I think Goolsbee discounts how much money they make on the side buying and selling houses for their own personal gain. I recently spoke to a real estate broker who operates in one of the most expensive areas in San Diego, Mission Hills, where the average house price has been going for just under a million. She told me that over the course of 5 moves to successively larger houses, after starting out in a small two bedroom dump, she was able to end up in a $2 million dollar property solely based on making wise investment decisions about which houses were undervalued. Of course, real estate agents are not the only ones doing this, but surely they have a significant edge on the rest of us by virtute of their experience, contacts, and access to information such as houses that are going to go up for sale but are not yet on the market.

Dabblers, Blowhards and Hemingway Tests

There's nothing quite like hitting the nail on the head so smack-dab center that the proverbial nail pierces right through so that all you can hear is a faint whimper of resistance from the other side. This explains in part why I just love this piece by Maciej Ceglowski entitled "Dabblers and Blowhards." It's informed, it's biting, and it's ruthlessly on point. I just feel sorry that Paul Graham has to be on the receiving end of most of it.

While it's all to easy to call bullshit on other people's ruminations on some topic or another, especially the pundits, I think it's fair to say with the rise of blogging and other opportunities for self-expression there has been a concomitant rise in the amount of sheer dribble with people pretending they know a lot more about something than they actually do. The medium of blogging, where you are your own editor,  makes it's all to easy to put stuff out there without ever running it through a good BS meter. Occasionally I find myself falling into this trap of wanting to write about stuff I actually know nothing about. Luckily, I catch myself always remembering something Hemingway once wrote (paraphrasing just slightly): strip away the fluorishes and write what you know in the simplest and truest terms.

It's a shame there aren't more English lit teachers in the blogopshere because if there were, you would see a lot of desperately needed berating going on. In the meantime, I think, if more people, applied the Hemingway test to their own writing, we would see a dramatic decrease in dribble and attending increase in authentic self-expresison. Of course, having more critics like Maciej wouldn't hurt either.