One of the good things about not being able to blog about my work is that I can upload random charts and not have to exert the keyboard bashing required to explain my thoughts.[1]

X axis would be current account balance.[2]
Y axis is loan to deposit ratios for random banks

---

[1] Which are largely derived from another happily married Aussie bloke.
[2] Dada thanks to another Aussie bloke.

Just so folks know, I'm heading on honeymoon for almost two weeks starting Friday. My posting frequency will be undisturbed by this event.

Should be fun.

despite getting married outside in pouring rain, a good time was had by all.

I spent Saturday afternoon at a birthday picnic for one of S.'s good friends who works in the film biz. The great thing about that crowd is that I get to both live vicariously through their stories and also I am reminded of the brief period when I worked in film post production. No offense to my current coworkers, but there is something special about working with creative creative people, rather than technically or financially creative people.

In a conversation with a guy who does indy film distribution, after fawning over his current project with Werner Herzog, we got into a discussion about the future of the industry with all the youtube and whatnot. I made some throw-away line about how branding isn't yet well established online and there may be some possibilities there. But what really got me thinking was the lack brand identities in the re-democratization of arts and entertainment.

Directors, record labels, musicians and producers succeed when they become brands. As much as I wish it weren't so, I would have never sat through all of Gerry if I didn't know it was a Gus Van Sant film. Countless songs would have missed out on the critical 5th replay required for them to catch on if I didn't know that they were written by X or released by Y. Sure, it is not a hard and fast rule and I discover new and unknown stuff all the time. But we all go the extra mile in accepting familiar brands. I am loyal.

If it weren't for the coin slot, people would have no qualms in stealing their morning newspaper, or so would be the case if we were all Homo Economicus. Plenty-o-folk download music without worrying about paying their dues to the artists. But would we do so if the band was watching us at the time? Radiohead's great experiment in behavioral economics showed us that when given the option of free, many people still elected to pay. I posit that it was merely the act of making the consumer aware of the option at the time of download that resulted in this. Not a really stunning observation, but I think it is key.

And it wouldn't work for bands who weren't brands. If you don't know what you are getting, you probably won't pay. However, if you were aware of the artists situation, you might come back and pay. Online, this might be as simple as putting up a download link with a big message along side: "The band remains poor and starving. Please help." But being a technologist, I think we can do better.

Imagine a system whereby independent artists collected payments, voluntary or not, through a central clearinghouse. That way each artist could accurately display how much income they are making from their music, and various stats about downloads. Starving artists can use their prandially challenged status to help convince listeners to share a few dollars. And popular artists will get all the benefits of popularity.

This could easily be another feature of last.fm or similar. But they are busy building a walled garden of listener behavior data. However there is no reason why this needs to be a centralized service. Distribute this sucker. That way record labels can do whatever it is that online record labels do, and can collect the cash. They can then distribute the cash to the artists, which I hear they do, on rare occasion, do.

Production costs are decreasing. Anyone with talent & a computer can make a great film or album. Companies yearn for strong brands as consumers can then align their personal images with these brand identities. Art and entertainment is a visceral identifier of personal characteristics. I like the music that I like because it says something about me. I strongly believe that given repeated exposure anyone can at least enjoy, if not love, any form of art. At some stage I chose the music, film and art that I wanted to be identified with, and later did the deep connection and enjoyment grow. As individuals the clothes we buy, the food we eat and the entertainment that we enjoy often provides a shortcut to our own deeper identity.

So, why do entertainer incomes follow a power distribution? Why does Coldplay have the benefit of being able to turn down multi-million dollar advertising deals when countless other musicians have to rely on government subsidies to feed themselves? These poorer artists are not always lesser in their art. I can understand that in a world where search costs are high, attention is focussed on a smaller set of well marketed artists. However with the diminished cost of search, consumers should be more selective in their choice of art brands. And if the audience is free to sample unrestricted digital works, but are aware of the status, needs and micro-brand of the artists, I would like to think that we can arrive at a more even distribution of wealth across the arts.

And then I'd find a way of sticking a fiver in this guy's cap.

After some email goading from Jerry[1] I have decided to share a few of my thoughts on the problems with twitter. First off, let me say that I don't know the people behind twitter and I have no access to the sort of information I'd want to see before I could even come close to forming a cogent opinion. So everything from here on is most definitely wrong and bone-headed.

In my reply to Jerry I made some claims based on my initial rumination about Twitter's problems. I decided to do some further digging and found that Jeff Atwood's[1] opinion is pretty close to mine. Namely, that the first order of business is to stop blaming Ruby or Rails (neither of which I have used), and instead thinking about the underlying platform. For me, this means thinking long and hard about their database.

Whatever I'm actually paid to do in my career, I always seem to end up working with sizeable datasets. And I think databases are totally swell. Bees knees. I'm one transaction shy of putting a poster of Codd on my wall. But sometimes, not due to lack of effort, I give up on databases and end up rolling my own data crunching application. Databases are so swimmingly handsome because they let you approach them and make general queries on data. Sure, there is an art and science to optimizing a database for specific queries, but 90% of the time you don't have to think about that.

c.f. The Twitter API specification[1]

When Twitter was just an Obvious twinkle in someone's eye, a database made perfect sense. Who knew what Twitter would become and the flexibility of a database is a huge advantage. But now Twitter is Twitter: A messaging service. And save the discovery of a business model, the announced coming features don't dramatically impact architectural requirements for the service. Oh, and since the original implementation of the prototype service, Twitter has experienced massive growth and downtime is crushing goodwill. I say, ditch the general purpose database, implement a custom solution. The great thing about a custom database is that it really doesn't take much time to build if you know your usage patterns. Whereas a RDBMS has to make guesstimates as to how to execute a query, and optimize that execution against a set of generic index types, a custom solution can use custom index algorithms. For example, as part of a recent bit of hacking I got a few thousand fold speed improvement from moving an app from a well managed commercial database to some hand rolled C. For a key join, the RDBMS was performing its best, but we were able to perform better by making a merge-bsearch-linear scan algorithm that made perfect sense for this application.

In short, if you are database bound, and your database is well managed, and your application has known query patterns - ditch the database. People seem to forget the computational bandwidth at their fingertips. Grab the nearest napkin, scrible down a rough estimate of the bandwidth requirements of your app and then compare that to your computer. If A << B and you are suffering performance issues, then you are in a happy place - it should be easy and rewarding to solve.

Step away from your computer and just think about what it means to have 2 GILLION CYCLES PER SECOND. Thems be a lot. Use them.

My napkin calculations based on over-heard numbers suggest that a platform change would suffice and make Twitter as happy as Larry. But let's say that someone goes along and implements what I'm talking about and that's still not enough. Well, then it gets to be _real_ fun. I took a wee bit of graph theory at school, but I also have the pleasure of having friends who took an unhealthy amount of graph theory. It is a well studied domain. If platform level optimization isn't enough there is a wealth of knowledge in this space on optimially placing resources to maximize bandwidth. Twitter is probably pretty close to a fully connected graph. But where are the cut points? How lumpable is the distance matrix? How does the matrix evolve over time? (Pretty slowly, I guess) What is the optimal time period for relocating clusters of users across your horizontally scaled system? And why am I not hearing any discussion online about these questions? (Probably because I haven't looked.)

OK. It is 4:30AM and I should be packing for the move tomorrow. So I'll cut it short. Edgar[2] and Edsger[3]. People have been thiking about these types of problems for a long time. I know how daunting it can be when you are running a system and everything goes to shit. Immediate reaction is to patch stuff. Sometimes that just makes it worse. Read the literature, mine your operational data and find a better way.

And whatever you do, don't listen to some wacky Australian on the blog-o-sphere.

---

[1]: I'll come back tomorrow and add links to this blog after I become a Brooklynite.
[2]: Codd
[3]: Dijkstra

I was looking for a big list of blogs, so I decided to make one. Here is a random selection of 20,000 blogs from my big list of blogs.

Unrelated to this, I cancelled my twitter account today. I know there has been a lot of back-seat commentary about how to fix their problems, and I certainly have my own ideas. I don't feel like sharing them as I still find it very difficult to understand how they managed to get their implementation so wrong - thus being the kinda guy I am, I think the best of others and assume I have a flawed understanding of some subtle complexity in running a messaging service. [1]

---

[1] While it is true that I cancelled my twitter account, it is a total lie that I honestly think I'm missing some crucial detail as to how to make a scalable messaging service. The honest truth is that I'm too lazy to write it down. And if I did write it down, I'd be adding to a flame-war that seems to consist of people who seem to know little of what they speak.

I just received my Kindle. My initial complaints [1] with the technology are pretty much along the same lines as everyone else: low contrast glossy screen with low enough resolution to make the jaggy fonts unpleasant, buttons in all the wrong places, and software that seems all too beta for a hardware device. Technology aside, the device has highlighted a few of my personal reading foibles. I often read in bed before I go to sleep, and I like to know many pages I have to go before reaching the end of a chapter. I balance the number of pages against my sleepiness and if I think I can make it, I like to put the book down at the end of a chapter. It is always awkward returning to a book mid-chapter. While I understand that variable font sizing on the Kindle erases the notion of a 'page', the device does provide a progress bar to indicate how far along one is in the scope of the entire book. I want the same thing, but on the chapter level. Or at least a way of flicking to the next chapter and looking at the 'page number' and comparing that against where I currently am in the book.

So I read the manual.

At first, I started reading it on the Kindle itself, but after reading the first 3 chapters (within which I was instructed on how to change the font size on 4 separate occasions), I gave up and went to my desktop. I quickly realized that the manual made no mention of my chapter-progress-bar feature.

So I called Amazon's Kindle customer support (1-866-321-8851)

Me: Hi
AmaDroid: Email address?
Me: Huh, um, xxxx@i2pi.com
AmaDroid: Billing address?
Me: Is this the Kindle help desk?
AmaDroid: Yes. Billing address?
Me: XX Horatio St, New York
AmaDroid: ZIP?
Me: 10014
AmaDroid: How can I help you today?
Me: Well, first you can explain why you asked for my personal details without saying 'hello' to me.
AmaDroid: Um...
Me: Or maybe you can tell me if there is any way to work out how far along you are in a given chapter when you are reading a book on the Kindle?
AmaDroid: There is the progress bar that tells you how far along you are in your book.
Me: But can you tell how far along within a chapter?
AmaDroid: No.

At this point two things come to my mind:

1. Seth Godin's piece about in-bound calls
2. And instructions for getting root access on the Kindle. FedEx tells me that my parts will arrive at the office tomorrow. Yay.

Me: [HANGUP]

:wq

---

[1] As a prematurely grumpy old man, I'll leave compliments to others.

Thanks to a recent message on twitter I learned that David Bryne is blogging. Not only do I really dig Talking Heads, but I love his work with Ryuichi Sakamoto and it is great to peek into the mind of a musician through writing that is as well composed and shares an unsurprisingly concordant asesthetic and ethic as his music. Go read his blog: http://journal.davidbyrne.com/. And if you do go read, you will find out about this:

Playing the Building, a 9,000-square-foot, interactive, site-specific installation by David Byrne, will transform the interior of the landmark Battery Maritime Building in Lower Manhattan into a massive sound sculpture that all visitors are invited to sit and “play.” Byrne’s project will consist of a retrofitted antique organ placed in the center of the building’s cavernous second-floor gallery that will control a series of devices attached to its structural features—metal beams, plumbing, electrical conduits, and heating and water pipes. These machines will vibrate, strike, and blow across the building elements, triggering unique harmonics and producing finely tuned sounds. As Byrne explains, it is an elaborate system for “activating the sound-producing qualities that are inherent in all materials.”

Playing the Building marks the first time in decades that the second floor of the Battery Maritime Building will be accessible to the public. The space will be open and free to all visitors on Friday, Saturday, and Sunday throughout the summer of 2008. Everyone will be invited to sit at the organ, tap on the keys, and create a unique array of sounds that travel through the space. In addition, David Byrne and Creative Time will invite guest musicians to challenge his creation through a series of performances and jam sessions.

Now, I don't really know much about art, but I really like it when I see a simple idea executed well and I end up walking away saying "Wow. I could have done that. If only I thought of it first." I guess I'll be heading down to Battery Maritime this summer to check it out.