One of the best things about being crowned a romance buff by the prestigious New York Post is that I now can get away with pretty much anything at home.

“Honey, can you do the dishes?”
“But I’m busy knitting you a scarf”
“Yes, but doing the dishes would be romantic. I declare it so.”

I now have a Frank Bruni-esque[1] power to declare anything romantic and it must be so. Anything else would be denying the Post’s role as an arbiter of New York cultural icons. Of which I am one.

—
[1]: Or whomever the NY Post’s equivalent may be.

Update: According to one of Shianling’s patients, we also made it to The Gothamist. Now I’m truly a cultural icon.

Postgres:

 think=# select 'hello' = 'hello'; ?column? ---------- t (1 row) think=# select 'hello' = 'goodbye'; ?column? ---------- f (1 row) think=# select 'hello' = 'HELLO'; ?column? ---------- f (1 row) 

MySQL:

 mysql> select 'hello' = 'hello'; +-------------------+ | 'hello' = 'hello' | +-------------------+ | 1 | +-------------------+ 1 row in set (0.00 sec) mysql> select 'hello' = 'goodbye'; +---------------------+ | 'hello' = 'goodbye' | +---------------------+ | 0 | +---------------------+ 1 row in set (0.00 sec) mysql> select 'hello' = 'HELLO'; +-------------------+ | 'hello' = 'HELLO' | +-------------------+ | 1 | +-------------------+ 1 row in set (0.00 sec) 

This is not kindergarten people, this is UNIX. Do what I say, not what you think I mean.

Since appearing on the popular section of del.icio.us, a number of people have asked me to explain how it is, in fact, possible to replicate what we see in the following video:

Whats going on here is really quite simple. All you need to do is hold the video camera with one hand and wrap the foil etc. using your free hand. It definitely helps if you have a small, hand-held video camera. If you have one of these it might be good to have a friend around while attempting the above.

Mike called my bluff. The book I refer to in this post was actually from 1988, not the 1960’s. The book is Dynamic Graphics for Statistics and can be had for a princely sum of $149 from Amazon. I’m certain that I paid approximately 1/100th of that for it.

In a semi-unrelated meme, I was chatting with my friend Aaron a few weeks back about why I disliked 3D graphics, especially in games. To his credit, I don’t play computer games so I miss out any much of the emotional nuance that comes from interacting with simulated gore and I probably shroud my argument with rationales purely to hide my irrational aesthetic bias. But that said, I don’t like 3D graphics for two interrelated reasons. Firstly, I assume that the designers of these graphics curse themselves at the lack of rendering power at their fingertips. Sure, they think their blood spatter looks better than any game that came before it, but if only they had 64 GPU pipelines instead of 32, it would look so much better. George Suerat never complained about the number of lines per inch in his Gillot paper. The paper was chosen for its capabilities and its characteristics were exploited to promote the aesthetic that Seurat was seeking. I have long felt that art and constraints go hand in hand and successful creators express themselves within boundaries. For the most part, game graphics are not timeless. Today people look back on games from 2003 and chuckle at the lack of realism. In 2009 the creators of Halo will secretly wish that they could go back and remake their games with 2009 era technology. Who knows if the makers of Galaga would prefer to recreate their game using today’s technology, my guess is not.

Fundamentally, I find it a little offensive when people make graphics and aren’t happy with the outcome. Sure, game creators (and players) probably really enjoy their work. But whenever they admit that they need a better computer to ‘get the full experience’, they are stating that they are not happy with what they have. Computers are immensely powerful. But I fear that much of that power is being dedicated to doing things very uncomputer-like. If people adapted to and adopted the medium, we may have graphics that look very different to reality but in the abstraction away from reality I think we can find a fluid, informative, and medium appropriate representations. People don’t have an innate understanding of what data looks like, and in the world of statistical analysis and visualization we can afford to break the mold. Perhaps not in many games, but I have seen some games that take this approach and invent new visual realities that are (to my untrained eyes) as emotionally engaging and poorly rendered trees and bump-mapped blood spatter.

I won’t dwell too much on my second concern with contemporary computer graphics as it is well covered by Tufte. Namely, the core concern is that too many pixels and graphic elements are used for non-informative purposes. Shading a chart, or purposelessly extruding a bar chart in 3D adds no new information to a graphic. This is why on any new install of Excel I spend a good hour or so setting up new chart defaults to get rid of all the junk it tends to decorate my charts with. (Yes, at times, I use Excel.) Going beyond the world of images, the same principle can be extended in interface design. Just as one needs fewer distractions from the information, interfaces need fewer mouse clicks and menu selections to play with the data. In this age of computer power, people often neglect the value of exploratory data analysis. More, now than ever, is EDA important. I have terabytes of data at my fingertips, and most visualization tools just don’t keep up. I want to be able to slice, dice, plot and histogram data at a breeze. Data retrieval isn’t the bottleneck, but rather poor interface design.

What this world needs is a nice, simple, stand alone, does-one-thing-and-does-it-well exploratory data analysis tool. Maybe I should make one.

A movie about exactly that. A documentary about the world we live in.
Thee spirit ov thee times.

I watched this movie recently, it is a melding of many different themes that are current in todays society and prevelant in everyones mind. It centres around Money , War, Terrorism and how we are all fooled by the mass media. On the surface this kind of description may seem to be rather conspiracy thoerist and alarmist , but infact i found this movie to be a rather sobering roller coaster journey back to reality. It puts together alot of factual information and presents it in a way which just makes you think “this is whats going on” and it indeed is.

Although the movie is focussed largely on America and its historical impercial capitalist value system, it is equally relevant for Australia , since as the famous old addage goes “When America sneeze’s the rest of the world gets a cold” , seems particularly true in light of the information presented. Particularly regarding some of the more notable “Black flag” operations , such as 911 , 7/7, and bali bombings ,the date of which , sadly escapes my current frame of mental reference; and the strikingly similar circumstances in which those so called “terrorist” attacks occured. I have always had my doubts on the validity of the information the media presented on these generic operations. This movie has served only to assist me in strengthening my resolve to think for myself and question authority. I must be getting old.

sense

This weekend was my girlfriend’s birthday and my parents happened to be passing through New York. I figured the aligning of the stars was a sign that it was about time for me to propose. Being unabashed nerds, our first date involved spending a decent chunk of the day browsing through the math book section at Strand. In a very xkcd spirit, we hunted down erotically suggestive math titles. Our favourite was “Tight and Taught Sub-manifolds.” Topology always seemed a little kinky.

Being particularly unimaginative I decided to propose at the site of our first date by constructing some books that spelled out the proposal along their spines. The ruse was completed by convincing her that I was unable to get her present delivered to the house on time, and thus we would have to drop by the store to pick it up. Rather than setting the books up in the humid and always busy basement of Strand, I arranged to set things up in the Rare Book collection on the third floor, away from most traffic. The store was very keen to help out as the New York Post had recently been looking for love stories to tie into an article celebrating the stores 80th birthday.

Given that my now-fiance’s interests are more aligned around the natural world rather than mathematics, I planned to set the books up next to the rare natural history collection. Its a lovely nook in the store and seemed like the perfect place. Having nonchalantly covered my tracks for the previous month or so, it all broke down when we entered the book store. My plan was to casually peruse books and gently lead her to the natural history collection. Instead, it was more a case of yanking her towards the books and me childishly pointing to my proposal books and saying “Look!” She was stunned and I went on one knee to ask the big question. Despite being a relatively capable public speaker, I had considered the possibility that my mouth might stop working at that moment, so my goal was to get out at least four words; “Love”, “Light”, “Sweetness” and “Joy”. I got as far as “Love…. Marry me”. She said yes.

After some crying and hugging I realized that the books were set up, not in the natural history section, but next to it - in the medical section. In the photos, the book closest to my proposal is a title on gynecology. None of that mattered, of course.

It took a few moments to compose ourselves and then we wandered through Union Square to the W hotel where I had booked a suite. Thanks to some friends who work for Starwood we got comped up to their largest suite. Champagne, Strawberries, Bubble Bath. Nap.

Then we walked down to Casimir in the East Village for a surprise dinner with 25 close friends and family. A group of us returned to the suite for some more drinks, dancing and fried food. By midnight we were exhausted. It was a fantastic day. My only regret was not seeing my friends from Australia and around the world. Hopefully I can rectify this shortly.

Read it for yourself.

Q: How does one calculate a p value?

Update Some kind soul has done the right thing and calculated a princely p value of >> 0.05. In other words, lottery draws are random. Of course, it could be a huge cover up as if lotteries were not random I can think of a whole bunch of statistics books that would have to be re-written with new examples.

And now for some R code:


#Get big.dat at http://www.state.nj.us/lottery/data/big.dat
big=read.table("big.dat",sep="%",fill=T)
big$date=as.Date(apply(big[,1:3],1,paste,collapse="-"))
big$type=ifelse(big$date>"1999-1-13",ifelse(big$date>"2002-3-15",ifelse(big$date>"2005-06-22",4,3),2),1)
big$maxnorm=c(50,50,52,56)[big$type]
big$maxspecial=c(25,35,52,46)[big$type]
maxnorms=table(big$maxnorm)
p=rep(0,56)
for(i in 1:nrow(maxnorms))
p[1:as.numeric(names(maxnorms)[i])]=p[1:as.numeric(names(maxnorms)[i])]+maxnorms[i]*5
maxspecial=table(big$maxspecial)
for(i in 1:nrow(maxspecial))
p[1:as.numeric(names(maxspecial)[i])]=p[1:as.numeric(names(maxspecial)[i])]+maxspecial[i]
p=prop.table(p)
allnum=unlist(big[,5:10])
t=table(allnum)
chisq.test(t,p=p)
plot(t/p)


As a quick hint: read.csv can accept a URL in addition to local files.

TED quotes a very memorable statement from Derman’s “My Life as a Quant”:

Quantitative finance “superficially resembles physics,” he says, “but the efficacy is very different. In physics, you can do things to 10 significant figures and get the right answer. In finance, you’re lucky if you can tell up from down.”

Derman likes to bring up the analogy often:

Now I think that maybe “financial engineering” suggests too much precision, like using too many decimal places in specifying your height as 6′ 1.2345″. Maybe FE is a politically correct word for something more fuzzy.

Computer scientists should be called computer engineers. Nutritional scientists should be called nutritionists, without the science. Maybe people shouldn’t be allowed to self-describe and name their occupation.

What is the most accurate job name to describe what financial engineers do?

As someone who never made it beyond the national level in the Physics Olympiad due to my poor mathematics, I can’t wax lyrical about heat diffusion equations and their place in finance. However, I can guesstimate that a decent financial engineer spends 90% of their time trying to profit from other people’s mathematical approximations of human behavior. The best financial engineers spend 90% of their time convincing others that their models better correspond to reality.

TED’s post reminds me of an anecdote about the early mortgage market. At that time there were no models for prepayment or default risk. A prominent player in the market (with no formal mathematical education), hired a PhD student to build a rudimentary model and then applied the model to find a basket of underpriced loans. After amassing the portfolio he circulated the model to the street, creating demand for his loans. Now, thats arbitrage.

I just happened to come across this article while working on some code for analyzing financial data:

In short, the picture of Wall Street designers that comes across is revealing. The designers in are smart, able, savvy. But they make up an distinct community of practice, one with lower status, limited financial knowledge and one that does not seem to fully communicate with the traders and bankers. In terms of innovation, they also seem to be paralyzed by the needs of their users. As as we know from Christensen’s “The Innovator’s Dilema,” users are a conservative group.

I’m quite busy right now, but let me add some random bullet points to the discussion:

• My current codebase has ~300 lines of R code dedicated to analysis
• A further 150 lines are for visualization
• Thats a 2:1 ratio for the mathematically inclined
• My charts have black backgrounds, no extraneous borders or labels.
• Every single pixel is important (See Tufte)
• No offense people, but 99% of the designed I have worked with in the past have none of the technical caliber of John Maeda
• My favourite book on statistical visualization was written in the 1960’s and I have yet to come across a visualization package as powerful as the one they describe.

In addition to poor underwriting standards, Ranieri says the mortgage market has brought in too many “middlemen with no axe to grind other than to collect money.” By this, he means that the traditional relationship between a local thrift or savings-and-loan and a depositor, who might have borrowed from the thrift, had changed.

From an article in Investment Dealers’ Digest about my former gig.