Identifying The Blind Spots

Leopard Sharks, Red Tides, and Natural Hallucinogens

So I was planning on meeting my wife today after work to go snorkeling at the La Jolla Underwater Park and play with the leopard sharks which are currently out in droves there only to be foiled by a massive bloom of phytoplankton known as the red tides. CDNN (not to be mistaken with CNN) has a good article on it here. Although they kill the visibility during the day, apparently the night time underwater visuals are incredible (above water photos can be seen here):

"If you turn off your lights and wave your hands around in 10 feet of water, the light display is unlike anything that exists above water," [a nightdiver] said. "You're surrounded by sparkling blue lights, zillions of them. It's absolutely phenomenal."

Note to self: Gotta get me some night gear...

Sidenote: According to the local life guard, this is one of the largest and densest blooms we've had in decades, going all the way from LA to Baja.
 

Putting the Brakes On The Inexorable Flow of RSS Time

Before the RSS movers and shakers of the world get too far along in transporting us to an information-rich nirvana, I hope they stop and think about one thing. Does all RSS content really need to be parcelled up in chronological order of when the content was created? It's important to remember that RSS came out of work done on weblogs which was and still is, in large part, modeled on news and radio. In both of those industries, it has made a lot of sense to focus on what's new because that's how you made money at least according to popular belief.  And while a lot of apps are emerging that are centered around making money by getting people excited about discovering what's new in the world, RSS is still young enough that it need not be driven by the same economics and overly obsessive focus on the new.  Most of the interesting content out there, at least for me, is not about what happened this morning or what so and so said yesterday. No, most of the interesting content spans a much longer event horizon into the past. Sure it's weighted towards the present but I'm constantly discovering interesting articles or audio clips that are more than a few years old.  So I'm making a personal plea to those in the RSS community (Mark Fletcher, are you reading this?), please give us users alternative ways of consuming content which are not solely fixated on the new.  I waste way too much time, the way things currently stand, going through my RSS feeds, deleting stuff that is obsessively focused on the now, and thus unnecessarily adding to my sense of Information Overload.

But that is how RSS was designed you say? There is no going back. Nonsense.  For example, just because an RSS feed may be indexed on the server by creation time doesn't mean you have to read them in that order.  You could instead read them based on a personalized topic distribution of what topics the subscriber is feeling particularly interested in that day and then rerank all the rss items according to how closely they match the subscriber's topic distribution. Ideally, you could even use some machine learning techniques to detect if something is time-sensitive and if the user wanted to could filter that out.

Nic Harcourt & the Characteristics of Succesful Trend Spotters

Great profile in this Sunday's NY Times Magazine on Nic Harcourt, the public radio station DJ who happens to be one of the most influential arbitrators of what I call "smart music." Not nearly as well known as he should be. It's interesting though. As much as most people hate to admit it, when it comes to being able to spot emerging trends, there are a few people whose talents rise far above the rest of us. If you think about all the Nic Harcourts, Robert Parkers, or even John Udells of the world, what is it that these people have in common beyond just having a deep fascination (or obsession in many instances) with their discipline and lots of time to wade through it all? Well, for one I would guess (as I'm sure most social network researchers would) their social connectivity patterns all look very much alike with lots of short-range connections to other discriminating authorities of taste in the same communities of influence but then they also have all these  long-range connections out to more obscure, highly-specialized experts spanning many different niche communities, in some cases, all over the world. Thus I suspect the following description of Harcourt's social connectivity patterns is typical of other succesful trend spotters as well:

Harcourt stands at the hub of an interconnected web of opinion and advice that helps guide him through the avalanche of material constantly coming his way: friends in the U.K. who keep him current on English and European releases; producers and musicians he has grown to trust over the years; the producer of ''Morning Becomes Eclectic'' and KCRW's assistant music director, Ariana Morgenstern, who was born in Argentina and feeds Harcourt Spanish-language rock discs as well as jazz vocals selected to work in the morning mix; other KCRW D.J.'s with specialized knowledge; and the English music magazine Uncut, which he reads cover to cover.

Of course, most trend spotters are not merely succesful by having the right social relations. That is perhaps only a necessary precursor.  Rather they are succesful by having a Heisenbergian ability to help accelerate, if not bring about, whatever it is they are spotting.

Thinking Through The Implications Of Data (Dis-)Ownership

Paul Kedrosky has a great followup to my last piece where he poignantly observes:

The more people that figure out that it's not about proprietary applications, but about proprietary data, then we merely move from one walled garden to another. Because a walled garden of data, however pretty, still has walls, just like old algorithmic garden of shrink-wrapped software. All that we are doing by over-focusing on data being the Intel inside is ensuring that data becomes the Intel inside in every sense -- another proprietary layer for litigation, instead of something that can be readily extracted and combined to create better apps & services.

That's exactly it. Replacing the OS with the browser as the main site of interaction with one's data, merely moves the problem of lock-in up the stack. Rather than trying to figure out how to export your data from one desktop app to another, the problem has simply been transmuted to how do you move your data out of web site A over to B.  In cases of lock-in in the latter case, not only can you not get your data in a format you want it, but you can't even get access to the binaries since they no longer live on your hard drive.

What I really wonder is of all the users currently being locked in to some service out there, how many users realize they are being locked in. I bet it is surprisingly few. Luckily, for some of the big apps where it's absolutely clear that it is your data, e.g. email, most people aren't locked in because if they were, there would be an outcry. But there are many services where it may be a little less unclear that the data you copied or manually entered into some site, for example, is yours and that you should be able to export it to say an xml or plain text file but you currently can't. 

This raises a lot of interesting legal issues. For example, if I enter data into a site after accepting a license that requires handing ownership of the data over to the company, then legally it would seem I don't have any right to claim access to the raw data. But what about instances where the license says nothing about who owns the data placed on the site by the user? It's like I walk into a store and staple the only copy of a poem, that I happen to own and have all legal rights to, to the bulletin board and leave. Does the store now own the poem merely because I left it there regardless of what my intentions were? It would probably depend on whether I made an effort to reclaim it. Now imagine a situation where millions of people are posting millions, if not billions, of poems to the bulletin boards of stores all over the world. Most of these people don't try and reclaim the poems, although some of them do. But if you explained to these people that if they didn't try and reclaim what was originally theirs, they would lose ownership, and possibly all intellectual property rights if that was the only copy and no evidence had been recorded to the contrary, it is likely most people would try and reclaim them and might regret they had put it there in the first place. Similarly, if you explained to them that that there was a really small sign on the wall saying anything left in the store is no longer theirs, it is likely, in many instances, they would stop leaving their poems there. The example of poem as data, of course, is meant to be illustrative chosen in part to reflect on the owners since most poets aren't known to be particularly savvy when it comes to real-world legal issues.

So this is the problem that I see we're currently facing. There's a huge lack of awareness on the part of the users about the legal implications of putting their data on someone else's servers. Companies don't mind that there's this huge lack of awareness because it works to their benefit. Most people aren't demanding that the company allow them to export to their data out should and when they decide to stop using the company's services. And so many companies, perhaps without realizing it at the outset, are reaping huge additional benefits, which is fine, except for the fact that it disempowers the user, stifles innovation by preventing the user from leveraging the data in other ways, and makes it harder for the companies that didn't get a head start on this data monopolization strategy to compete.

What I would love to see happen is for companies to be clearer about their data ownership policies. For example, a sentence in large, bold font reading "By placing data on our site, you are (or are not) legally giving up your rights of ownership to this data" would be helpful. In cases where data is owned by the user, companies should allow them to freely export the data to a standard format, either in xml or plain text. More awareness on the part of the users would go a long way to forcing companies to be clearer about their policies. The solution I proposed in my previous post was meant to help address this problem.

An Idea for Jump Starting the Open Data Movement

Thinking through a little more about the general problem related to my last post of of how to encourage companies to avoid trying to lock in their customers, it occurs to me that one solution is to start an initiative loosely modeled on something like EnergyStar where companies that comply with some set of Open Data guidelines can then use the initiative's copyrighted logo for marketing purposes or what have you as a way to attract users that have concerns about being locked in to a vendor's platform. Rather than have a heavyweight regulatory system where companies apply for licensing and the application is reviewed, there could just be a lightweight community-based framework (think Wikipedia) where people that use the logo but fail to meet the simple guidelines would be informed that their license has been revoked. If just a few highly visible people started promoting the idea, it's possible that that could create enough awareness about the issue that some vendors might have enough incentive to start using it. There are already a lot of vendors which do a good job of not locking in their customers so this could be a free lunch for them.   If that happened, then we'd have a wonderful virtuous circle where non-compliant vendors would feel the pressure from compliant vendors in the same space to become compliant, uninformed users would become aware of the problem based on the visibility of the logo and information describing the initiative, and the rest is history. Of course this all assumes a handful of people are willing to work on setting up an organization to back the idea and carefully define a minimal set of Open Data principles and compliancy requirements.  Easier said than done.

On Disintermediating You From Your Data

Interesting, if not catty, back and forth (also see comments) a few weeks back between Dave Winer and Rick Klau of Feedburner as a result of Dave alleging that Feedburner is trying to lock in their customers. Very similar remarks were made by Rogers Cadenhead who fretted that "if FeedBurner goes out of business, which is the most likely outcome for any Internet startup, they'll instantly lose that entire audience." More serious is that by handing over your RSS feed to a third party, you are literally disowning some of your own data, allowing an intermediary to possibly cash in on it, for example, through advertisements and leaving you all the poorer. Dave Winer's solution?

Perhaps we should start an open source project that provides software you can install on your own server that does log analysis on feeds. The technology isn't very hard. We would endorse a competitor who offered the benefits of Feedburner without ceding control of the subscribers.

And he's right about the technology not being very hard. Producing reports of log activity is not exactly quantum physics.

But what's more interesting to me is the larger issue of data ownership. We have seen incredible innovations from the open source movement, but what about the open data movement where your own works and data, whether they be book reviews, RSS feeds, photos,  or articles are under your control copyrighted by your own licensing needs rather than stored on some company server where you're locked in because you can not easily export your data elsewhere? John Udell and others have been writing about this for some time, talking, for example, about the need for companies to be competitive in the market place based on offering innovative services that don't require ownership of your data. But clearly it's unrealistic to expect for-profit companies to modify their behavior until their is a strong incentive to do so. To a limited extent, we've already seen that by simply modifying your behavior many of the benefits of disintermdiation can be achieved. For example, instead of posting a comment or review on someone else's site such as Amazon, you can post it on your own and then reference it on the other site. If at any time you want to modify, redistribute or remove said review, you don't need someone else permission. But this only works for a small subset of all the data one is constantly putting out in the world. Furthermore, most people don't want to be their own part-time system administrator wasting time doing data management, especially when the data lives in various forms with different intents of use.

What is really needed are innovative companies to come around who make it their business to take care of your data and whose mindset is centered around bringing you in control of that which is yours. Not just to host it,  but also to provide tools to make it searchable, to reformat it, to redistribute it, even to export it. The vast majority of the companies that claim to be in the business of content distribution, data management, and related services rarely have your best interests at heart.Yes, companies like Yahoo! and Amazon, generally come close by doing a reasonably good job of storing data you put on their site and allowing you and others to search it, but there are some areas of their business where they lock customers in. In general, I suspect that the innovative companies that emerge with the right mindset and strategy regarding how to unlock your data while allowing them to make a profit will do better than those whose interests are not as closely aligned eventually forcing them to follow suit.

Update: It turns out Feedburner has of June 10th gone ahead and provided an option for users to stop using their service without incurring serious hardship. So a small victory for the Open Data movement and I'm sure Dave Winer had a lot to do with it. My bad for noticing this when writing the original post.

Damn You For Speaking Out

Anyone hear about this? Quite frightening, really. Luckily, it was all caught on C-SPAN so skeptics like me can see it ain't no conspiracy theory.

A Command Line for the Web

This is one of the coolest ideas for a web service I have ever seen, a KISS solution to building a federated web OS and it all came out of the 24 hour Ruby on Rails programming competition. Actually what Jonathan, the creator, has built here is, technically speaking, just a Unix-style shell for performing web queries but with just a little more work I can see myself using this on a daily basis. I shouldn't say "just" though because it, as Jonathan, points out it has some very special properties:

The beauty of YubNub is that anyone can help to extend it. If there is an existing web service with a submit form, they can add it pretty easily (like I did with the Amazon example above). But even more interesting is the adding of complex data-processing services (like validating an RSS feed, or converting webpages to audio using text-to-speech).

It's not unlikely that this will inspire people to build similar types of shells and eventually we'll have to have a .webrc file to tell our browser which shell we want to use and how we want it configured.  This, in turn, might motivate someone to build a web-based browser configuration tool so that when I start using a new computer, it'll automatically download my configuration preferences, and I'll be up and running. I'm a little skeptical that one can take the whole Unix mantra and map it to the Web as Jonathan seems to want to do, by, for example, stringing pipes of Web queries together. I just don't think the Web is sufficiently self-describing that it can be done with arbitrary content, even if it's all in XML and your making smart use of XQuery or XPath, but I'd love to be proven wrong.

 

The Pot Calls The Kettle Black

The headline of a frontpage NY Times article reads: "A Congressionally mandated panel will report that the U.N. suffers from poor management, lack of accountability and professional ethics." I'm so glad my tax dollars aren't being spent trying to figure out why the US government suffers from the same problems because that would be just so, what's the word: causeless, exorbitant, expendable, extraneous, extrinsic, fortuitous, futile, gratuitous, haphazard, inessential, irrelevant, lavish, needless, noncompulsory, nonessential, optional, prodigal, profuse, random, redundant, supererogatory, superfluous, surplus, uncalled-for, uncritical, undesirable, unessential, unneeded, unrequired, useless, wanton, worthless.  It's comforting to know we have responsible leaders in charge who know just where the pressing quandaries that are that our nation faces lie and are able to take a purely rational, efficient approach to addressing them head on, no matter who is reponsible.   

The (Ultima)te DIY Project

My friend and ex-boss, Craig Dillon, head of scoring solutions at Fair Isaac, is working on what is arguably, the mother of all Do-It-Yourelf projects: building a high performance racing car virtually from scratch (it's based on a high-end kit). The particular car he's building is called, no pun intended, the Ultima. Some photos of this bad boy can be found on his online build diary here. The funny thing is that's he's never been into cars per se as he says on his online diary:

I am not a car person. Here’s my proof – here is the list of car’s I’ve owned: a big old 1980’s station wagon, a second-hand Ford Taurus (what a piece of junk), a Honda minivan, a Honda Civic, and a Ford Focus. I don’t watch car racing, I don’t read car magazines, but I am interested in the technology behind automotive engineering. Every car I’ve had is automatic, so I don’t even really know how to drive a manual! The most exciting car maintenance I have done is replacing the radiator on the Ford Taurus in a car park. I have driven a Nascar, though, which was absolutely amazing.

Knowing Craig, it's all about the joy of building and designing non-trivial engineering marvels. As he puts it:

A car is a project of great precision and great strength. Engines are built within thousandths of an inch accuracy. Drive train components that fit in the palm of your hand can literally handle the pulling power of 500 wild horses (seriously, that's why they call it horsepower). Chassis are welded together at thousands of degrees temperature, and body panels are formed in giant molds. No matter how you look at it, building a car is a dream for a tool guy.

He's documented this project every step of the way with detailed photographs and narratives and 10 months into it, it seems he's pretty far along. It will be interesting to see how the final product turns out.

What I especially like about this project is the notion of building something non-trivial and documenting your work along the way, not just at the very end, so others can learn from it and enjoy the engineering challenges involved. Before Craig pointed me to his online build diary, I had a preconception that putting together a kit car was one of those things you did if you were hung up on impressing people with a fancy car but didn't have the money to afford a real one. But after reading his diary, I can appreciate the hackeresque desire to want to build something cool, ripe with design/build conundrums,  where the reward is something entirely unique and, in this case, awe-inspiring.

Update: Craig has since told me the car he is building will be faster than every production car in the world including the F1, Enzo, and Mclaren.