Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious.

We’ve chosen to use relational databases, specifically PostgreSQL, for storing some of our data. We like ACID, we like the ease of ad-hoc query-ability and we like the fact that databases add an additional layer of security and data quality control. To make the most of this we should adopt some conventions, so that when we are accessing PG from an ORM, we don’t bring too many ORM-isms into our data model. This ensures that other staff, who might be using other ORMs, can still work with our data and it also prevents us from relying too much on the ORM or application layer for doing work that RDBMs can already do.

These are my preferred conventions, we will evolve them over time. I’ll try to provide a justification for each one, but this is a discussion.

1. All names (table, column, sequence, index, constraint, role, etc.) should be lowercase with underscores. Postgres does support AnYSortOF casing that you’d like, but it makes manual querying painful.
2. Table names should be a singular noun that describes one row. “account”, not “accounts”. Some people prefer plural, we just need a standard, my vote is for singular as it makes SQL a little more natural to read
   e.g., “SELECT * FROM account WHERE account.balance > 5000;”.
3. We’re using a relational database. Have relations. Very few tables should be islands.
4. Foreign keys should be named “<table>_id”, e.g., if the “account” table links to the “person” table, there should be a column in “account” called “person_id”. In the case where there are multiple foreign keys to the same table, prefix the ids, e.g. “from_person_id” and “to_person_id”
5. Foreign keys must have foreign key constraints. It makes the schema more readable, both by humans and introspection tools. It also prevents mistakes at the application layer.
6. Serial columns should have the sequence as the default value for that column. E.g., if the “account” table has a primary key of “id”, it should be defined (in SQL) as “id SERIAL PRIMARY KEY”, which is a shortcut for “INTEGER NOT NULL DEFAULT nextval(‘account_id_seq’)”.
7. Never expose serial columns outside of the model layer. If any table is going to be exposed in any way via an API, it should have a UUID column that will be exposed instead of using the "id".
8. Index, constraint and sequence names should take the form of table_column_[idx | uidx | seq | ck] for indexes, unique indexes, sequences and constraints.
9. Unique indexes should encompass all the rules for uniqueness. If the “user” table can only have one copy of each user, consider a unique constraint on first_name, last_name, address and zip, also on SSN, or whatever. There is nothing wrong with having the front-end, back end and database all check this.
10. Constraints should reflect business rules. Just because your application does sanity checking, it doesn’t mean that some bozo at the terminal will do it. FYI Josh has access to most of our machines and is a bozo.
11. Postgres has a rich selection of native types (IP Addresses, UUIDs, Time intervals, Polygons). Use them where appropriate. If your data is an IP address, stick it in an INET. If it is a UUID, there’s a type for that.
12. Postgres also supports enumerated types. If we have a relatively immutable small list of possible values for a column, use an enum.
13. If the SQL type is not descriptive enough of the type of data that is stored in a column, use the units of measurement in the column. E.g., "height_meters" if we are storing a height, in meters. God knows why we'd do that, but you get the idea.
14. Don’t be afraid of TEXT. If you want to store free form text, VARCHAR (2048) isn’t what you want. Postgres is smart enough to move large chunks of text outside of the table and into a blob, so there VARCHARs end up taking more space than TEXT. If there are strict length constraints, don’t use TEXT.
15. Don’t be afraid of NUMERIC. We are dealing with money. Bigint’s are fine, but we need to then rely on the application layer to do the right thing. Each application needs to know what 12345 means in dollars. When we start having interest bearing accounts, 4 decimal places may not be enough. Postgres supports arbitrary precision numbers. We should standardize on NUMERIC (18,6) for money. And please be sure that your application doesn’t silently translate arbitrary precision numbers into IEEE-754 floats or similar. We all saw Superman III.
16. Set reasonable DEFAULTs. If you have a column called “created” which records when a row was created, a reasonable default would be now(). I saw this as a default on one of our tables “not null default ''::character varying” Not reasonable. If its not supposed to be null, setting the default as ‘’ is silly. At the very least, decide whether each column should be NULL.
17. Don’t be afraid of schemas. Postgres supports multiple object namespaces within the same database . If you’re unaware of schemas, you are probably creating objects in the “public” schema. If we ever get to a point where any database has dozens of tables, schemas are a good way to clarify the roles of each table. Look into it.
18. By default, don’t denormalize. At our scale, its bad form to have the same column in two tables that are joined by a 1:1 relationship. Doing this means less logic at the application layer to enforce consistency.
19. Many to many tables should be named with the name of the two tables they join.
20. Log modifications. If tables have mutable columns, provide a _history table that keeps track of changes. If you’re so inclined, you can do this with triggers.

Comments (0)

I threw this slide up as part of my talk yesterday at the IA Ventures Big Data conference. The talk was titled "Big Data with Small Data" and was my attempt at describing how we at BankSimple apply big data techniques to relatively small data sets.

It was a short talk and while this was my key slide, I didn't have as much time to discuss it as I would have liked. What do you think?

Discuss.

Comments (0)

I've spent the last 37 minutes trying to send $104 to a friend for tickets to a party. He specifically asked me to send him the money via PayPal. I have a PayPal account, so it shouldn't have been a problem.

The problems began when I first attempted to send him the money. PayPal complained that they weren't able to confirm that I owned the account. I'm currently on a business trip and using the hotel's internet connection, so I figure the odd IP address is confusing PayPal. None of the suggestions provided suggested that they would fix the problem, so I decided to verify my account by linking PayPal directly to my checking account.

My checking account is with USAA, and linking the two required providing PayPal not only with the username and password to my bank account, but also answers to three security questions and my card's PIN. It took me a good 10 minutes of switching back and forth between USAA and PayPal to sort that out. (It's more difficult than it should be. Why don't banks support OAuth?..)

By linking the two accounts I assumed that I would have provided PayPal with enough evidence that I owned my account. This wasn't the case.

I also re-verified my email and set up and verified my phone number with PayPal. It took more than 5 minutes for their verification SMS to reach my phone which currently flutters between one bar and 'Searching...' while I'm out here in the mid-west. Not PayPal's fault, but clunky.

None of this helped convince PayPal that I was who I said I was. If I gave anyone else all the information I had just shared with them, they could walk away with my money. But I couldn't send $104 to pay for party tickets.

Its two thousand and ten. Money is electronic. Sending money between American banks, while clunky, is cheap(*) and easy. Doing it via PayPal is hard because PayPal's supports international transfer and thus, rightfully, expends more effort fighting fraud than they do sending money. Horror stories, like mine, are common. But while PayPal could certainly improve their web interface, the majority of the experience failures are due to their interminable vigilance against fraud.

---

(*) Cheap for the banks. While it may cost them a tiny fraction of a penny, they will readily charge customers far more.

Comments (0)

Although I'm really a C programmer, I've been doing more work over the past three years in Python. Today, for the first time, decided to write a digital signal processing program in Python. C is usually my go to language for these types of tasks and I felt like a fish out of water.

You can check out the code, if you are so inclined, to my spectrum analyzer at gist. There are few sections where the code reads more like what a C programmer would write, rather than that of a native Python programmer. I was having trouble clearly expressing the following line of code:

[ord(s[2*i]) | ord(s[2*i+1])<<8 for i in range(0,len(s)/2)]

Essentially, that code did exactly what I wanted, but I feel that there was probably a simpler way of expressing the same intent in Python. I asked my followers on Twitter for some help with the above segment, and got some useful answers. I then re-phrased the exact same question, exposing my intent, as:

Do you know of an inbuilt way to convert a byte stream containing unsigned 16 bit integers into an array of python ints?

"convert a byte stream containing unsigned 16 bit integers into an array of python ints". Seventy-two characters to type. The code I was using consumed fifty-three characters. In some way, the code was 25% more efficient at expressing the underlying intent. And that brings me to the moral of this blog post.

I woke up this morning to a flurry of news stories about an SEC proposal to include Python code with asset backed securities (ABS) filings. The idea that while ABS documents are chock full of legalese, a computer program can provide a very concise way of understanding how a financial instrument is supposed to operate. I really liked the idea, but needed to know more.

This recommendation comes in a 667 page pdf. I just finished scanning through it, trying to find more details about the proposed implementation. You see, I spent a good chunk of last week writing a retail banking simulator in Python, and I have some questions about how they intend to do it. Of course, completely missing the point of their very sensible recommendation, no where in the document is there any Python code. Rather than making me go through hundreds of pages of text, I would have really appreciated a link to hundreds of lines of code.

Oh well. Their heart is in the right place.

So, why am I writing a retail banking simulator in Python? Well, at banksimple we have an ever growing Excel spreadsheet. Given the limitations of Excel, we make lots of broad assumptions about the distributions of things like account balances and daily spending. Given the non-linearities of both our business rules and human behavior, I want to get a sense of the sensitivity of our model to various risks. And the best way I know to do that is via simulation.

Rather than hiding code throughout gnarly cell references, I can clearly express business rules and customer responses in code and from there I can tweak inputs and asses the impact of distribution assumptions on our revenue model. Essentially, I build a universe with millions of bank customers and let them do the things that people do with money for a few years and see what happens. This is a very different approach to modelling in Excel. It is much better at capturing non-linearities.

I wanted to know whether the SEC proposal was for including full simulations of securities that can be composed of thousands of other instruments, or whether it was more like an Excel model, just written in Python. The answer is probably somewhere in those 667 pages, but I can't find it.

---

Sorry I haven't been blogging here recently. We are quite busy getting things ready with BankSimple. You can follow along over at our blog at http://banksimple.net/blog

Comments (11)

A decade ago I was pitching web analytics software to a number of retail banks back in Australia. The pitching process was intense. Not only did we have to explain what web analytics was about all to marketing executives who had just discovered the internet, but we also had to get past operations teams who were held to a five nines standard. Five nines, or 99.999% uptime means six seconds of downtime per week. This attitude pervades the development of banking products, both in ways that are beneficial to the user and in ways that make what should be simple experiences painful.

Running a large teller machine network requires a serious investment in uptime. With thousands of transactions running through the American card networks each second, the fallout of losing even a fraction of a percent is too serious a risk to face light-heartedly. As a whole, the network is amazingly reliable. But point of sales machines don't adhere to the same reliability levels as the network as a whole. The machines that you read your card swipes at stores are subject to risks from power failure, phone network outage or even running out of paper before coming close to hitting a system-wide network failure. In the event of a single failure at your local bodega, affected customers have the option of walking across the street to use an ATM or make some other minor adjustment to cope with a local problem. If the network as a whole were to go down, even for a short period of time, there would be chaos.

An individual POS machine meets local demand at 99.9% availability. This is how the system is meant to work. Likewise, the internet. The internet is designed to be fault tolerant, with the realization that it is cheaper (and often plain better) to tolerate failure instead of going to great length to avoid it. Your POS terminal could reach 4 nines if it were housed in a fancy co-location center, but that would be hardly be convenient. Yet when we were pitching marketing analytics to the banks we got the impression that if banking technologists had their way, all bodegas would have multi-homed candy aisles and drip coffee machines with backup generator power.

While this attitude has shifted since the formation of independent internet banking groups as distinct from core banking operations, tight coupling between legacy systems results in a development process that is just completely wrong. Banking is complex. Retail operations are tightly regulated. Applying this mentality to every aspect of banking leads to unnecessary inflexibility. Yet successful online experiences are defined by development processes that rapidly iterate and evaluate. The idioms for online interactions are rapidly changing – and at this point in our short history of internet usage it is difficult to see many points of conversion. This is the zeroth hour and iteration begets discovery of new interactions and continuing evolution of a common language for working with the web.

As many of you know, I'm working with a great group reinventing retail banking. A big part of what we want to get right is our user experience. In researching this project, I've signed up with a bunch of different banks across America. The typical process for doing so heavily reflects the forms that you fill and experience that you would have opening an account at a branch. While many of these form fields are required by law, the end user experience is heavily informed by legacy development processes. Branches were satellite offices, connecting to mainframes via expensive WANs. The process feels unwieldy to new customers as their applications move in lock step through a system that was designed as a series of incremental improvements over a pen and paper driven process of just two decades ago.

Most banks operate their core transactional processing systems on a batch cycle. This stems from the way the Fed works with banks for overnight lending. And as the internet groups budded off from core banking operations, the process of institutional mitosis resulted in a bunch of useless DNA being carried over. One of the banks I have accounts with (a top 5 US bank) regularly closes their internet banking site on Sunday nights for scheduled maintenance. When did Facebook last shut down for maintenance, scheduled or otherwise?

One thing we are working very hard towards is the distinction between parts of our banking service that must be highly reliable/available and those where we can iterate quickly. For example, take call center operations. A good phone system, simply due to variability in customer demand, will need staffing slack to reach given quality of service – in terms of expected time on hold, and time to resolution. You can spend a tonne on reaching a certain level of technological availability, but have little impact in key measures of customer satisfaction due to the different costs involved in staffing. With a five nines phone system you can still deliver a shitty service if your call center capacity is capped out by 100% staff utilization.

When making decisions about technology behind a retail bank, such as the call center or web site, we choose to trade iterability for early wins in the cost of scaling. Large, complex and thus rigid systems make it difficult to evaluate competing operating procedures. Short sighted metrics for success lead to short sighted incremental improvements. Free from the constraints of public markets we are able to take a risk and try something different – even if we don't know, a priori, how different it has to be. We believe it is critical to be able to try new things quickly, learn from our customers and improve their experience based on data.

I'm always scared to have our audience ask "What about feature X?" during our early technology demos. More often than not "feature X" is already in our feature tracking system, but market as "Will not fix." Even very simple features that would take only minutes of development time to implement have far reaching consequences. If we added the ability to filter transactions by date, for example, a number of quick decisions would be made about how to implement the user interface - with no possible implementation resulting in an interface less complex than simply leaving the feature out.

Additional complexity, without a clear use case, is bad. The flexibility to add new features as they are justifiably demanded is good. Complex systems work best when they are adaptive. Desinging new features in bulk and dumping them on users after a 12 month development cycle is just cruel. Especially so in banking, where mistrust is rampant and fear of making a mistake is justified. Better to iterate quickly and support an adaptive complexity landscape.

People often ask us what it is that makes us better than other banks. Glibly, I respond that we are just a plain old retail bank – but we don't suck. Not sucking is our killer app.

I don't know what that means in terms of fine grained details for future features. Sure, we have prior beliefs as to what experiences suck more than others under the current banking model, and where we should appropriately spend our valuable time. But we are also fine with being wrong - hell, we expect it. The only thing that we believe is that by setting ourselves up to respect and learn from our customers experience, we win. Other banks just can't do that.

---

Photo by: riv / flickr
Comments (0)

According to Mr. Meara, 90 percent of all transactions with bank tellers involve checks. If everyone had an iPhone deposit app, people wouldn't come into the branch as often. That would be fine had banks not invested so much time and energy in training branch workers to persuade checking account customers to move into more profitable products.

"One the one hand, fewer deposit transactions could mean a headcount reduction," he said. "But it invites the erosion of store profitability. The banks are struggling with the enormity of what it means."

Hurry up & credit my account New York Times, September 18, 2009

Brand apathy is rampant in retail financial services. The number one sales channel is the branch. That expensive, increasingly empty, retail space is how banks sell new products to customers. "Is there anything else that I can help you with today" is the alpha and omega of retail bank marketing, with a small epsilon for banners of smiling families plastered inside branches.

Below is a screenshot of what I see when I log into my American Express account. A small portion of the screen is stuff I care about. A small, but significant, portion of my emotional well being rides on those numbers. The bulk of the page is dedicated to selling me stuff. How about this? Make understanding and working with my money easier - make me happier, and then I'll be far more receptive to upsells. But if you can't even get basic information and interaction right, then I'm too busy worrying about my current state of affairs to consider new fangled products and the incremental complexity they entail.

Comments (2)

As someone who has spent most of their working career selling data to advertisers, I'm suprised by the number of businesses that are predicated on the model of selling data to advertisers. If you have a great widget, it is easy to get it in the hands of millions of people, especially if you are giving it away for free. As programmers and scientists we deify data. What we don't do is understand advertising. Sure we understand that advertising is about selling stuff, but we don't seem to get that the advertising industry exists to sells ways of selling stuff.

Whether an external agency or an internal group, advertising professionals have to convince others that they are adding value. And if your model is simply to sell data to advertisers, you have to convince them that your data is worth at least as much as what you are selling it for. Your data needs to be useful. Smart techniques are popping up for doing interesting things with large amounts of data. But interesting isn't always useful. If you think your customer data is useful, you should use your data to make your product better. Data that is valued by your users for the richer experience it provides is likely to be valued by others. If you can't improve your customers experience with their own data, your data is worth nothing.

Comments (2)

While I'm ranting, let me ask you something, Randall. At the risk of sounding like Glenn Beck Jr. - what the fuck has gone wrong with our country? Used to be, we were innovators. We were leaders. We were builders. We were engineers. We were the best and brightest. We were the kind of guys who, if they were running the biggest mobile network in the U.S., would say it's not enough to be the biggest, we also want to be the best, and once they got to be the best, they'd say, How can we get even better? What can we do to be the best in the whole fucking world? What can we do that would blow people's fucking minds? They wouldn't have sat around wondering about ways to fuck over people who loved their product. But then something happened. Guys like you took over the phone company and all you cared about was milking profit and paying off assholes in Congress to fuck over anyone who came along with a better idea, because even though it might be great for consumers it would mean you and your lazy pals would have to get off your asses and start working again in order to keep up.

And not just you. Look at Big Three automakers. Same deal. Lazy, fat, slow, stupid, from the top to the bottom - everyone focused on just getting what they can in the short run and who cares what kind of piece of shit product we're putting out. Then somehow along the way the evil motherfuckers on Wall Street got involved and became everyone's enabler, devoting all their energy and brainpower to breaking things up and parceling them out and selling them off in pieces and then putting them back together again, and it was all about taking all this great shit that our predecessors had built and "unlocking value" which really meant finding ways to leech out whatever bit of money they could get in the short run and let the future be damned. It was all just one big swindle, and the only kind of engineering that matters anymore is financial engineering.

http://www.fakesteve.net/2009/12/a-not-so-brief-chat-with-randall-stephenson-of-att.html
Comments (2)

The message is pretty clear - competition, the pillar of capitalism, results in better products and services for consumers. When lenders compete, you win. While this is the slogan for one major mortgage lead generator, the methodology is common to the industry as a whole. And people believe that magic technology fosters competition, with the net benefit of better lending rates.

The reality is a little different. When I oversaw the operations of a mortgage marketplace, the competition was not in terms of the products offered, but rather, the price paid for getting a person's attention. Lenders would bid for leads and the lenders who paid the highest price received the most leads. Thus the incentives were counter to people's rational goals. The lenders with the highest margins were able to spend the most on customer acquisition, while lenders with more affordable products were unable to reach the same audience.

Google recently publicized their direct entry into this space. Prior to their entry they captured only a portion of the marketing dollars - with lead generators buying keyword ads on Google, funneling the traffic to their site, collecting lead information and selling to the highest paid mortgage providers. When the lead generators spend on Google was lower than their revenue from the mortgage companies, they profited. A mercenary and highly unregulated bunch, the lead generators would go to great lengths to screw the consumer.

Google's product appears better in that rather than selling out the consumer for the highest price, they display a targeted list of options - clearly outlining the competing offers - letting the consumer decide which companies to contact for a quote. As is always the case, transparency leads to a better outcome for people.

Despite the numerous and simultaneous failures in the mortgage marketplace that has so deeply scarred the American economy, one upside that is often forgotten is the benefit of standardization of lending products. Prior to the development of the mortgage backed security market mortgage contracts varied greatly in their structure and terms. And while they remain complex financial contracts, standardization means that a consumer is able to properly evaluate the bulk of the financial impact of their mortgage choice by simply examining a handful of parameters.

I have a home equity loan. I also have credit cards, savings accounts and brokerage accounts. The simplest account that I have, and the one that sees the most action, is the humble checking account. My checking account has a 36 page introductory preamble that outlines the terms and conditions. These terms are fully documented on a corner of my bank's web site, and change on a semi-monthly basis. No one reads these terms.

I spent my summer reading not only the terms of my checking account, but of all of my accounts and the accounts at other major banks in America. You'd be terrified to know what they actually contain. That is, if you could find them. The GAO found that 65% of banks do not make these documents available on the web, and 35% fail to produce them if you visit a branch.

And these terms matter.

Unlike most people's mental model of retail banking operations, banks do not make most of their money on the difference between the rates at which they lend versus the rate they offer for savings. American banks, quite distinctly from banks elsewhere in the world, make the bulk of their money from fees and charges. Invisible and often unavoidable consequences of little clauses in contracts that no one ever reads.

This stands in stark contrast to the message that we hear in bank marketing. Retail bank marketing is dominated by APR: Best rate savings! Lowest rate on credit cards! Yet the largest financial impact to the consumer is fees and charges.

Fees and charges that consumers have no hope of simply understanding.

Lead generation is rife across the financial product landscape. Some companies try to offer Google-like services for better helping consumers choose financial products - but these services fall into the trap of not taking into account the obscure and non-standardized terms that most impact financial well being. And as such no one believes the offers they see in sites like Mint.com. If people honestly believed they could "Save $2,000 by switching to Bank XYZ's credit card" then the conversion rates on these offers would be vastly better than the prevailing rates.

And so with all the technology that we have at our disposal, people are no better off. Banks have no incentive to increase transparency, lead generators have no incentive to provide real offers and immense brand apathy prevails resulting in short sighted decisions further driving down customer experience. The cycle continues.

Until it stops.

Comments (3)

Last week I paid my bills. As part of the regular bill-paying process, I take any funds left over that are not required as cash over the coming weeks, and pay down my home equity facility. I pay my credit cards in full. The only rate I concern myself with is 3.8% rate on my loan. There is nothing particularly unusual about this process.

A few days after I paid my bills my bank, JPM Chase, emailed me to tell me that my account was overdrawn. I logged into the web site and saw that they had put a bunch of payments through twice. Most importantly, they put my loan principal payment through twice. In their wisdom, they helped me correct this by withdrawing from my credit card to pay down my loan. My credit card has a purchase rate of 12% and an cash rate of 19%.

Follow me for a moment: They withdrew from an account charging 19% to pay down an loan at 3.8%. And along the way they charge an overdraft fee for the 'service.'

I have three simple requests for Chase:

1. Reverse the double payment, restoring my checking account to its intended balance.
2. Reverse the overdraft fee.
3. Return the interest they are earning at 19% on the credit card account.

I was first routed to the online banking group as it was clear that the error originated within their domain. The timestamps on the transactions are identical, the transaction numbers are near sequential; there is a clear indication that my intent was to only pay once, but they processed the transactions twice. From there I was transferred to the credit card fiefdom who told me that they would correct the overdraft issues - but it would take four days.

Four days later I called to discover that no work order was placed and there was no note of my original call. I went through the same call center waltz, but instead was routed to the home equity group. The same group that asked me to forge a signature during the application process - but that is another story for another day. During this call I was told that the home equity group would take care of everything as they were the final destination of the funds. I made sure that the work order included the three key points listed above. I also took note of the work order number and the names and times of all the people I spoke with.

Meanwhile, mind you, I still have a zero checking balance and am unable to make other payments. I am loathe to draw down the debt facilities at my disposal, as it would just make it even more complex to reverse these transactions. Friends have been helpful and luckily I have enough cash on hand.

Today I called the home equity group directly to check on the status of the work order. They gave me the same estimate as the last time I spoke with them four to five days. At which point the interest would have accumulated to $11. Not a great deal in the scheme of things, but it is my goddamn $11. I'd be happy if they simply paid me the $11 and moved on - they have earned a good deal of revenue from me over the years, and this is clearly their error. But of course, they can't return me the $11 as they have no mechanism for doing so.

"You cannot dispute this transaction for this reason: 5102 - Bank Releated[sic] Fees / Charges - Not Eligible"

As of now it appears my only option to force the return of my overdraft fee and for me to receive any accrued interest is to take action in the New York small claims court. There is no way Chase will defend this - that would cost hundreds for legal approval alone. Of course, they make an order of magnitude more than that from my family in fees, charges, interchange and net interest margin each year. As soon as I get my $11 back, they will no longer hold any of my accounts.

Customer experience matters.

Comments (4)

Last night we had our fourth NY R Statistical Programming meetup. The topic was Bayesian Methods + MCMC. We had two presenters, Jake Hofman and Suresh Velagapundi, both of whom did an admirable job of presenting a very broad topic to an audience with diverse backgrounds. I want to use this post to bridge a gap between the background material and day to day utilization. This is catered towards the audience who may have some experience with R, but aren't very familiar with the Bayesian Way. While it is a simple example, the steps involved extend on to the issues that are faced in real world applications.

The source for this example can be downloaded using the internet!

---

We are going to step through Jake's coin flip example to get a sense of what is involved in doing Bayesian inference. There are a number of packages on the CRAN Bayesian Inference view that do all of what you will see below. I decided against using them for two reasons. First off, the coin flip example is a little too trivial for using many of the techniques that rely on multivariate parameter estimation to see any utility. But more importantly, I want to use the opportunity provided by a nice simple example to step through the underlying mechanics. My hope is that after reading through this you can have a look at the available packages and be a better judge of what they are used for and where one package may stand out over another. In the course of doing this write up I went through the MCMCpack package and it is a good exercise to compare how they implement the MCbinomialbeta() against the first half of this walk through. For the curious, the MCMCmetrop1R() function is far more advanced than the simple implementation of Metropolis-Hastings shown below, and it is a good exercise to understand their tuning parameters.

As a quick recap, the point of the exercise is to go from prior belief in a distribution (in this case we believe that the coin is fair) and use observed data to arrive at a posterior distribution using both the prior and the data. There are three things that we need to know to calculate the posterior distribution:

1. The likelihood of seeing the new data given our estimate of the bias
2. Our prior distribution
3. The 'evidence' or the integral of the likelihood and prior for each possible estimate

I won't step through the derivation of the likelihood, as this should be easy enough to derive from the binomial probability distribution function. In this case our likelihood, with N trials and h heads is:

likelihood <- function (N, h, theta) theta^h * (1 - theta)^(N-h)	

Check that the likelihood function makes sense:

t <- (0:100) / 100

png ("figure1.png", width=800, height=600)
par (mfrow=c(2,2))
par (bty='n')
par (col='red')
plot (t, likelihood(100, 50, t), type='l', xlab='Theta Hat', ylab="Likelihood", main='Likelihood (t=0.5)')

Great, the maximum likelihood for 50 heads from 100 flips is a theta of 0.5. (See chart below).

Jake uses the Beta distribution as his prior as it has some neat analytic properties; namely that the posterior will be of the same distribution family as the prior. We call these types of priors conjugate priors.

prior <- function (theta, a, b) dbeta (theta, a, b)		

a <- 2
b <- 2
plot (t, prior(t, a, b), type ='l', xlab='Theta Hat', ylab='Pr(theta|a,b)', main='Prior')

If we do the integration, we can arrive at the analytic form of the evidence and thus the posterior:

evidence  <- function (N, h, a, b) beta(h + a, N - h + b) / beta (a, b)
posterior <- function (theta, N, h, a, b) likelihood (N, h, theta) * prior(theta, a, b) / evidence (N, h, a, b)

plot (t, posterior(t, 100, 50, a, b), type ='l', xlab='Theta Hat', ylab='Pr(theta|Observations,a,b)', main='Posterior (t=0.5)')
plot (t, posterior(t, 100, 70, a, b), type ='l', xlab='Theta Hat', ylab='Pr(theta|Observations,a,b)', main='Posterior (t=0.7)')

dev.off()

Let's say we don't know what the analytic form for the evidence (denominator in Bayes' rule) is, and replace it by a numerical integration over all possible theta's from 0 to 1:

evidenceN  <- function (N, h, a, b) integrate (function(t) likelihood (N,h,t) * prior (t,a,b), 0, 1)$value
posteriorN <- function (theta, N, h, a, b) likelihood (N, h, theta) * prior(theta, a, b) / evidenceN (N, h, a, b)

N <- 100 	# Trials
h <- 70		# Heads

png ("figure2.png", width=800, height=600)
par (mfrow=c(1,1))
analytic  <- posterior (t, N, h, a, b)
estimated <- posteriorN(t, N, h, a, b)
plot (t, analytic, type ='l', xlab='Theta Hat', ylab='Pr(theta|Observations,a,b)', main='Posterior (t=0.7)')
lines (t, estimated, type ='l', xlab='Theta Hat', ylab='Pr(theta|Observations,a,b)', col='blue', lty=2)

err <- (analytic - estimated)^2
lines (t, (err - min(err)) / diff(range(err)) * max(analytic), lty=3, col='black')
legend (0,2, c('Analytic','Estimated','Error^2 (scaled)'), col=c('red','blue','black'), lty=c(1,2,3), bty='n', text.col='black')
dev.off()

While things are pretty simple with this toy example, Jake made the point that real difficulty with Bayesian inference is twofold:

1. Integrating across theta to find the evidence (denominator)
2. Once you have the posterior, integrating it to calculate summary statistics (mean, variance, etc.)

In the above example we used the integrate() function to apply adaptive quadrature to find the evidence. We could use this method for 2, but lets not. Instead, let us use MCMC - which is at its core, a way to draw samples from a distribution that is otherwise hard to sample from.

Given that this example is rather trivial, with just one parameter in question (theta), I won't step through the implementations of vanilla Monte Carlo methods (uniform, importance & rejection sampling) These implementations are pretty much straight forward from Jake's presentation.

I will however, implement a simple Metropolis-Hastings MCMC sampler using a simple and symmetric Gaussian proposal density (q in Jake's notes).

MHstep <- function (pdf, prevCandidate)
{
	# Effectively we are taking a random walk.
	newCandidate <- prevCandidate + rnorm (1, mean=0, sd=0.1)

	# NB: Because we are using the normal distribution
	# as our proposal density, which is symmetrical,
	# we cancel out the q terms on the numerator and 
	# denominator, as q(x|y) = q(y|x)

	a <- pdf(newCandidate) / pdf(prevCandidate)		

	# Draw a uniform random number from 0 to 1	
	u <- runif(1)									

	if (a > u)
	{
		# This candidate is likely to be a better sample
		return (newCandidate)
	}

	# Else, stick with our previous candidate
	return (prevCandidate)
}

Let's use our numerical approximate to the actual posterior function as the PDF we want to draw samples from:

posteriorPDF <- function (t) posteriorN (t, N, h, a, b)

Time to go on a random walk down coin flip street.

steps <- 1000
samples <- matrix(NA, steps)

samples[1] <- 0.5		# initial guess
for (i in 2:steps)
{
	samples[i] <- MHstep (posteriorPDF, samples[i-1])
}

And how did we do?

png ("figure3.png", width=800, height=600)
par(bty='n')
par(col='red')
plot(cumsum(samples)/1:steps, type='l', xlab='Step', ylab='Estimated Mean', main='Drawing samples by Metropolis-Hastings')
dev.off()

Nice.

Comments (1)

the basics

"If you are not embarrassed by the first version of your product, you've launched too late."

On Monday I released predict.i2pi.com, a statistical learning web service. Designed to deal with common classification and regression problems, it takes input data in the form of a CSV file and returns to the end user a set of predictive models. For example, if you have a list of store locations, local weather data, and store revenue, you could use the service to see if location and weather impact store revenue. predict.i2pi tries to determine whether predictions are possible by running your data against a growing number of user contributed statistical learning algorithms and finding the ones that work best with your data.

In planning this I went through a range of features, bells and whistles but have decided to strip it all back. This is the simplest thing I could build to support what I wanted. It takes a file, runs predictive algorithms against the file, and returns performance measures. Data and predictions.

data

The data provided is expected to be in the form of a number of observations, with one observation per row. Each column contains measurements for these observations. One or more of the measurements we are interested in predicting. For example:

|<------ Explanatory Variables ------>|   /----- Response Variables (dentoed by *)
X1,    X2,    X3,    Name,  Date,        *Y
12.3,  13.4,  8.32,  Terry, 2008-10-12,  736.0
 9.3,  34.1,  1.21,   Josh, 2008-10-12,  NA     <-- NB: NA response variables will have
...    ...    ...    ...    ...          ...            will have predictions available 
 8.7,  38.7,  8.17,   Jess, 2009-01-07,  1823.1         subsequent download.

Data may include observations for which we do not know the response. These observations can be included, with the response left blank. Once satistfactory models are found, end users can download spreadsheets containing our best predictions for that data. On my todo list is adding confidence intervals to these values.

Once uploaded we try to best detect the following data types:

• Numeric (floating point numbers)
• Integers
• Dates (YYYY-MM-DD works best)
• String Factors (e.g., State or letter scores)
• Text (longer text than factors, with analytic interpretation as language text instead of as factors)

learning

Internally, predict.i2pi performs a standard test / training protocol. Data is loaded and a random half of that data is used to train the learning algorithm. The remaining half is used to test how good the learned algorithm works against previously unseen data. Robust algorithms will do almost as well on the test as during training, while less robust approaches will lead to far poorer performance during testing. The system continues this process of picking a training sample, training and the testing as many times as possible in an allotted time. During each of these cycles, predictions are tested against the actual responses in the corresponding observation. Performance is then measured using the R-squared metric for regressions and simple classification accuracy for classification problems. The system supports user defined performance measures with the goal being to let those who supply data decide on which performance measure is best for their application. However, at the moment I'm concentrating on opening up the ability for users to upload their own learning algorithms.

Currently learning algorithms are specified in small snippets of R code that can be dynamically loaded into the main R subsystem that is responsible for coordinating training cycles. See, for example, rpart.R which links in a recursive partitioning algorithm from the rpart library.

#requires(rpart)

myModel <- function (formula, data) rpart(formula, data, na.action=na.exclude)
myPredict <- function (model, data)
{
	p <- predict (model, data)
	as.numeric(apply(p, 1, function(r) order(-r)[1]))
}

All learning algorithms must contain two function definitions: myModel and myPredict. myModel takes a model formula and data, returning a model object that can be used to make predictions against new data. myPredict takes two parameters, the model object returned by myModel and a set of data that may not have been seen during training. We call the prediction function with one randomly ordered half of the data for training. For testing, we provide myPredict with the model object generated from the training set, but provide it with the as yet unseen testing portion of the data.

Users are also able to define transforms that take a matrix of explanatory variables and returns a new matrix with the same number of observation rows but with one or more of the explanatory columns transformed into a new space. For example one could take a 100 column matrix and apply some form of dimensionality reduction that returns a new matrix, with the same number of rows, but only 10 columns. The transform function is not shown the response variable to ensure that no funny business occurs whereby the response is somehow embedded in the explanatory variables. These same transform functions can then be applied to response variables alone, allowing the system, for example, to construct a model log(Y) ~ PCA(X1, X2, ... , Xn).

The following example shows a transform function that replaces any columns that are more than 50% NA with an indicator variable:

myTransform <- function (x)
{
	if (is.null(dim(x))) return (x);
	if (ncol(x) == 1) return (x);

	bad_idx <-  apply(x,2,function(c) sum(is.na(c)) / length(c)) >= 0.5
	if (any(bad_idx))
	{
		y <- x
		y[,bad_idx] <- is.na(y[,bad_idx])*1	# replace NA's with an indicator variable
		return (y)
	} else
	{
		return (x)
	}
}

coming soon

As for uploading code, at the best way to do this right now is via email. I hand rolled my own sandbox environment to prevent 3rd party code from hijacking my system - but as with any security code that I write myself, I loathe testing it in the real world until I've had a good chance to be as close to 100% sure that it is safe. In reality, I'll probably stop trying to reinvent the wheel, and use a pre-existing solution.

Given long term plans, and issues around data privacy, I didn't want to set up a system whereby data leaves the system for testing on external machines. While it works well for very large datasets, e.g., the Netflix Prize, the potential of over fitting is higher for smaller datasets when random portions of that data are often reused in validation cycles. That said, developing new learning algorithms (or plugging in ones from existing CRAN libraries) is fairly straight forward so you should be able to develop locally and upload.

There already is an API, but it is not at all documented. This is my next priority. Currently I'm running into some issues with using RCurl to interact with my API - issues which would not exist in any other language - but I really would like to get the R API out of the door before I open up wider access. In short, there are are 3 methods which are currently used by the web site (inspect my horrid JavaScript code to see them.) These allow you to upload data, make edits to meta data and receive predictions. Each prediction includes links to the R source that was involved in performing the learning + any transforms used. The prediction meta data also includes the quartiles for the measure after a number of test/train cycles, plus a sample of 250 predictions vs. actuals.

It has been suggested that I also include a small downloadable example snippet for each file to allow developers to get a better flavor of what they are working with. For larger files, I think this is a perfectly swell idea. In fact, I really do want to hear more of your suggestions. I took a knife to a slew of functionality before I released this, but I have code ready to go. But I want to wait for real life suggestions to see what I should be working on next.

The original plans for this project included complex routines for doing unsupervised schema detection and meta modelling to help identify which algorithms might work best with particular shapes of data. Also I had built a framework for combining multiple learnings algorithms in a boosting type environment. All of these features remain possible and will hopefully be released in the not to distant future.

One of the big issues I struggled with in deciding to release this is the nature of my target audience. At the moment there is an impedance mismatch between the sophistication required to understand what the system does and the utility of the system to sophisticated users. To those with any experience in predictive analytics, everything here should be your bread and butter - and most likely far simpler than what you do on a daily basis. However there is a large audience of people in the information business who currently make do with the 'Add Trendline' option in MS Excel. To this audience, this service would be greatly valued, but in its current form is probably a bit too much. This deeply embarrasses me, but I'm not going to let that stop me from publicizing what I'm up to. There is a plan, and it exists in increments.

For the lay information worker, there are hurdles both in providing understandable explanations of how the learning algorithms work and were applied but yet also difficulties in adapting my format to the natural shape of the data that they often work with - not to mention data cleaning. As an example time series models pose an interesting problem. They do break the model of one independent observation per row, but it is difficult to come up with a way of training and testing that is consistent with my current implementation. Even if I were to develop special case handling for time series data, it can be difficult for a computer to find appropriate periods over which to lag variables. At this point I think the simplest route is to let people include previous observations that they deem important, at lags that they think might be interesting, with each row. That way each row can be treated independently from the others and I don't have to build a lot of machinery to guess appropriate treatment of temporal dependency.

Likewise there are other problems whose natural representations don't map neatly to the one row = one observation representation - think of collaborative filtering or graph based problems. I am quite keen on keeping the one row representation as it affords me some nice system scaling properties without becoming too domain specific. That said, there is nothing stopping me from building front-ends that take data from these problem domains in their natural representation and map them to one that works better for my system.

When it comes to explaining the models, well. That is another story.

Comments (6)

A few months back I caught up with a fellow Aussie in New York, who I first met once ten years ago. It is amazing how social network dynamics change as an expat. He is currently teaching Architecture at Columbia while completing his doctorate in the nature of representation in architecture. It was the sort of long conversation that lingers for a few months before finding a resting place in your mind. At first glance we spent quite some time discussing the work at the Spatial Information Design Lab as this most closely bridged the gap between our worlds. The deeper conversation was that of representation.

Engineers build things. They use sciences to make sure that the things they build don't fall over. Architects design things - they take ideas of the world and represent them. Their audience is both the client and the engineering and construction teams. Different representations serve different purposes. Engineering: Representation to World. Architecture: World to Representation.

Finanical engineers take what they know about how companies work and built new things to serve other companies. Economists take the real world and make model representations of reality. However there is a void in economics, between the macro and the micro, in the domain of the company. Likewise, there is a void in financial engineering. Financial engineering is currently dominated by time-series analysis. I posit a weak form of the Black Swan theorem - namely that we currently don't know enough about the past to even pretend to predict the future. We have financial historians, in the form of data providers, but we don't have the architects to take this repository of past knowledge and build representations of how companies operate. Accountants across the globe set and implement the rules of this complex system, but we don't understand it's dynamics.

Can financial engineers shed the instrumentation of time series analysis and take on this role? Or will it come from a new group - the type of people who build Googles?

Or will their buildings leak?

---

Image by ken mccown on Flickr.
Comments (0)

The interwebs has made it really easy for those who are looking for data to find it. Or at least a close approximation to it. Those who have the tools to scrape the web and reverse out interesting data are typically part software developers, statisticians and hackers. Mix these three together and one is genetically predisposed to collect as much data as possible. But there must come a point when the collection stops and the inference begins. Inference is difficult, in that it requires making statements about the mushy world, whereas coding systems to collect data deals with deterministic computers. It is easy to fall into the trap of simply collecting data to avoid dealing with mush.

In my previous job I was overseeing a project which involved scraping a large publicly traded e-commerce site to find interesting information to support investment decisions. The problem was that everyone on the street was also scraping the same site. Our code was top notch and having done this before we were able to avoid common pitfalls and our system was gathering oodles of potentially useful data. Faced with all this data one of my developers had a tough time working out where to start the inference process. Sure, there are obvious places, like predicting revenue from site activity, but they are obvious. So obvious that even the sell-side researchers were doing it. Faced with the task of finding something less obvious, my advice to my colleague was to pick a pair of columns at random and come up with a model for the dynamics of the relationship. In such an exercise one picks the boring stuff, like transaction numbers instead of transaction values - and begins decoding from there. The goal is to look at the data sideways and see if anything interesting pops up.

Most often this fails. But it is a good way of breaking out of the data rut. When I do this exercise, I typically find myself desperate for one other set of observations to help explain what I am seeing. While the result may be boring, it is not failure as it gives you a direction with which to approach the data.

Currently I have a few projects that involve social networks in one shape or another. While my clients are generally looking for the standard orthogonal projection of the data, I can't resist the urge to look at things sideways.

A client was walking through the important data that they collect about social network activity, but when I talked with their developers they also mentioned in passing that they also collected profile pictures. Not for analysis, but for another part of their suite. Pictures. Cool. Thems be data. Excitedly I professed, as if I actually knew what I was talking about, that there is probably heaps of juicy stuff inside profile picture data. Intruiged at my own confidence I decided to tackle this by scraping 250,000 profile pictures from MySpace and grabbing a few key stats about each account. The first thing I wanted to examine was whether profile pictures in any way informed the number of friends that a user had. They do.

As I have other plans for this data, I didn't scrape MySpace with the complete intention of doing this project, thus only 19,214 of the images have associated friend lists. But this was enough to get started.

First off I wrote a short C program to calculate 32 features from each image. These features are pretty typical image processing functions, like size, average color levels, number of colors, smoothness, symmetry and a few keys points from the luminosity histogram. MySpace pictures tend to be a mix of faces, icons and general photos - to rougly help identify faces (without commiting to facial specific measures) I also calculate a subset of these values for the central portion of the image, including recording the location of vertical axis of greatest symmetry. Most of the values have been normalized to some image specific reference to increase variation and limit covariance, for example the average R,G,B values are expressed as a % of the images luminosity.

To get a sense of the variation of these features, I constructed an image based on the first two principal components of the feature space. At this point some kind folks on Twitter (starting with Mark Reid) pointed me towards t-distibuted stochastic neighborhood embedding. Someone mentioned that I could simply forgoe my feature calculation code and simply use tSNE on the pixel data, which sounded exciting, but after reading the paper I decided against it. In the paper the authors do demonstrate this technique, but their first step after reading in the pixel data is to perform PCA to reduce the dimensionality of the problem. And their image set was much more well behaved than the images I was working with. Maybe I'll take another look at tSNE in this context when I next have some free time.

Visualization aside, the next step was fairly simple. I divided my data into a 75% training set and reserved 25% for testing, attempting to predict log(# of friends) by my image fetures. Using a linear model was pretty poor, but not terrible. In sample I got an R^2 of 0.17, out of sample it was far worse. Using an SVM, I limited the training to classification rather than regression - trying to classify in groups by quantiles of log(# of friends). For a simple binary classification (more or less friends than the median sampled MySpace user), the accuracy was 70% - with errors evenly distributed across the two classes. I also tried 3 and 4 classes, and the lift was similar.

To visualize this I performed the regression using an SVM and, as expected by the results of the classification, got a decent R^2 (0.25) on the out of sample test set. To get a better sense of the outliers, I produced the following visualization. Note in this visualization I have sacrified some positional accuracy by enforcing a constraint that no images may overlap.

I also used a similar approach to test whether I could predict other, more interesting, network features like measures of centrality, and my initial results are positive.

At this point, if I were to run with it, I'd like to make some assesments as to the underlying process that relates social network features from image features. Until I have more time, my current hypothesis is Boobs.

---

For those of you with more time on your hands, I have packaged up some datasets. Grab them here.

Comments (4)

This is a comment on Jesper & Toby's recent E-Tech presentation. For some reason I posted it twice to the blog but it never made it through. My current theory is that the gubberment is out to get me.. Or I suck at technology.

(I guess my previous comment got lost in the system. Maybe someone can bail it out)

First up, congrats Toby & Jesper on tackling this issue. It is a shame that the Money:Tech conference was cancelled this year as that would be the perfect venue to address a good mix of quants and techs and spark some serious discussion. Barring that, I wanted to chime in with my own 2c as a techy/quant guy.

[disclaimer]In my previous life I did a fair bit of equity research on the ratings agencies, and the views here are mine and probably not shared by others. I have no positions in any CRAs right now[/disclaimer]

While the talk focussed on corporate bond ratings, the largest growth area for these agencies was in structured finance. And much of this was mortgage backed securities and similar derivatives. So to understand the mess we are in now we need to look at the history of these instruments.

MBS's were born for two key reasons. First off was the realization that in 'normal' times the dominant risks were idiosyncratic in nature and as such could be minimized through the application of portfolio theory and diversification, leading to pooled entities with smoothed cash flows and tranches providing for the needs of various risk profiles. In my opinion this story is primarily the sizzle. The real steak was the fact that by aggregating together whole loans new tradeable instruments could be formed.

The problem with whole loans was that their pricing was highly dependent on a large vector of unstandardised parameters whose diversity precluded the formation of any depth necessary to support liquidity in traditional market designs. By eliminating the idiosyncratic risk components these pooled instruments could theoretically be summarized by a small set of parameters and relatively simple models for prepayment risk meant that traders could respond to bids and asks against them.

Faced with these simple models quants took off in a great fantastical leap and applied ever more complex techniques to model out pricing. For one take, see Felix Salmon's recent piece in Wired. Or look to Paul Wilmott's take on the ever escalating departure into a mathematical wonderland that ignored the realities of the underlying loans and their associated risks.

Somewhere along the line practitioners forgot that the technology underpinning the frothy new market was based on 1970's financial and computational technology. Back then a bank of associates armed with HP-12C's could price out MBS's using a small set of descriptive parameters.

Over the next 20 years more computing power was thrown at the problem, but the basic data was still confined in scope. Sure, some funds were taking apart these pools and doing a deep analysis of the components, but there wasn't much reward in doing so as the market was moving at such an upward clip.

Even worse, if you look at the papers from Frank Partnoy, the credit ratings agencies - who were supposed to be taking a deeper look at these securities, without the demands of second by second trading - were using plainly silly assumptions. There was a huge amount of mathematical and financial stupidity going on. Not even going to mention the conflicts of interest and the regulatory arbitrage at play...

Anyway... To address Falafulu's point about MPT - I agree, MPT is great stuff; a very powerful framework by which to understand finance. But just look at the assumptions. Sure, these assumptions make the math tractable, but modern computing power enables us to take a more nuanced view of the world. We no longer have to rely on single parameters of 'default risk' to price these instruments. The market would be far better served if all available data for the underlying components and use their own information about their own risk profile to come up with better measures of value. Just compare David Einhorn's spreadsheet with a report put out by Moody's. It is night and day. Give me the data, not some puff piece of pseudoscientific nonsense passing itself off as high finance.

The original problems with trading whole loans, namely that there were too many parameters to support liquid markets, is no longer an issue. Look at WeatherBill. Look at Robin Hanson's work on combinatorial market mechanism design. Falafulu, sure some smart people were recognized for their ground breaking work of decades ago. But the most recent winner of the John Bates Clark medal in Economics went to Susan Athey, who is doing some fantastic work in mechanism design.

Computational power is such that we no longer need to pretend that all financial instruments have to be priced on with a slide rule. We have new marketplaces that can effectively support trade in financial instruments with high dimensionality. We have the computational power to let traders value these instruments. What we don't have is the data.

Give us the data and we will trade.

For a less ranty take on my world view, check out my blog post Data trades inversely to liquidity
Comments (5)

I recently voluntarily left my job running a 'renegade' equity research group to start an independent 'big data' consultancy business and in this economic environment this fact regularly gets me odd looks. Who in their right mind leaves a great position at a successful fund when half of Wall Street is battling to keep their jobs? This is not the first time I've done this, having run a similar consulting business right after the burst of the dot-com bubble. I don't have a great deal of life-wisdom, but if I did my only credo would be that when life hands you strawberries, it is time to go hunting for lemons. Don't rest on your laurels and the best time to be risk seeking is when everyone else is risk averse.

Having had to explain my rationale for launching i2pi as a consultancy frequently, I've come to rely on the phrase 'data trades inversely to liquidity.' This notion holds true in both my prior world of investment management and especially now in the data collection and analysis business. In finance, when markets are liquid price discovery is cheap. With all the talk right now of mark to market accounting treatments, it is clear that the converse is also true. Holders of illiquid securities can no longer rely on quoted prices to manage their portfolio risk. As the current crisis began to unfold earlier last year there was a very visible Mexican standoff while shops with CDO/etc. exposure refused to trade as the act of trading would force everyone else to reprice their own portfolios. Doing so could only last so long and the inevitable write downs began to occur as margins were being called. And thus the house fell.

The premise that led us to this mess was that with only a modicum of data and some threadbare models trading would be the final arbiter of value and the collective intelligence of efficient markets would result in fundamentally sound pricing. Now that liquidity has gone from the markets, traders of these illiquid instruments are bulking up their data and models to try and better their understanding of fundamental value. And so it is that when markets are liquid the market relies on trading to assimilate the information of individual agents. Without this method of price discovery these agents need to gather their own data as the market no longer performs the role of grand aggregator. Data trades inversely to liquidity.

While my work at the fund was phenomenally diverse and deeply intellectually stimulating, there was no fire. I've never had a real job. I've only ever worked at start ups where there is no time for a 'job.' In a constant state of conflagration, everything at a start up requires immediate attention. Early on in my career I worked as a back-end system engineer and 'fire' usually involved dealing with scalability and general growing pains. Late nights implementing features that were sold to paying clients well before the development team was consulted. Later I spent more time selling these features and there was a constant fire to come up with new and interesting things to attract clients and revenue. At the fund our financial stability was near certain and while there was a drive for deeper insight, the fire was luke-warm at best.

The current financial crisis is, at its core, rooted in the debt markets and this dislocation has clearly negative consequences for start up financing. Contemporaneously, new technologies and operational methods allow technology start ups to scale efficiently. Cloud Computing, as distinct from the similar buzz about Web Services just a few years ago, provides a platform upon which small companies can grow their operations in proportion to their needs without large capital investments in hardware or expensive, unwashed and hirsute systems administrators. Hadoop, Memcache and their ilk let developers build applications that operate on huge data sets without investing in the expensive vertical scaling solutions of Oracle & Co. And social networking results in network driven growth patterns that can be much steeper than products or services that live on an island. However, the skills required for scaling analysis systems are quite different to those needed to scale operations; part statistician, part database administrator, part computational micro-economist, and then and understanding of business to tie together a narrative that tells stories with numbers rather than purely stories about numbers.

The environment I see around me for technology start ups is one whereby funding is hard to come by and series B's are even more painful to founders. These companies need to be smart. They need to focus some of their attention on what to do with all the data that they gather as part of daily operations. Development teams, while facing the world with more appropriate tools than those available a few years ago still focus on operations. Someone needs to focus on research. Data driven research. In times when funding was easier and valuations were higher, companies could focus on operations and hope that that operational scale would lead to revenue. I firmly believe that in this environment data rather than scale alone is of immense value.

Beyond the empirical sciences of revenue optimization, behavioral targeting, customer segmentation and the discovery of on-line arbitrage there lies a need for basic research that develops fundamental models for understanding the environment in which a firm operates and how to question the unknown. This type of science is closer to applied sociology or experimental physics than the tradition of ponderous economics in that it is driven by and dependant on the immense mass of data that new businesses are generating. But as with all science the role is to simplify; to construct a narrative upon which new questions can be found and companies can learn to change how they think about their operations rather than simply what they know.

i2pi seeks to bring this science to companies.

Comments (3)

Sometimes, when trying to optimize a computer system, you get to a point whereby on one hand the next x% of optimization will take orders of magnitude more time than the previous x%. And on the other hand, you can completely change your marginal optimization cost if you take a different approach by approximation. Good systems can have hardware thrown at the problem to scale. As has been mentioned elsewhere, hardware is often cheaper than programmers, so we tend to go as far as we can by taking this approach - until it no longer works. Perhaps a more efficient algorithm can be implemented to lower the marginal cost of scaling. Unfortunately there comes a point whereby traditional algorithmic optimizations fail to change the equation enough to manage your costs. At this point you may need to cheat to scale.

Knowing when to approximate requires an understanding of the costs of doing so. If your system is responsible for proprioception in an automated x-ray system, then getting it wrong is to be avoided at all possible costs. Everyone would like to be as accurate as possible, but this is not always cost efficient. If you are running an analytics system being wrong costs less than you might think. Any system that produces statistics based on sampled observations has room for approximate solutions. Programmers tend to forget that such statistics contain error. Programmers tend to think that if there code is free of logical flaws, then the output must be error free. If you observe 12 users clicking on an ad that was displayed 1,000 times then the click through rate must be 1.2%. Click through rate, in this case is a statistic - a way of summarizing raw data. However, click through rate is also a measure of some innate clickability that is driven by the ad, its context and its viewers. Note that we can't possibly measure these abstract quantities, so we use our observed behavior of 12 clicks from 1,000 ads as guide to navigating the underlying complex abstract system of interactions that determine the clickability of an ad.

As soon as we produce a report that states that click through rates are 1.2% on all Fridays then we are implicitly giving the caveat of 'based on all Fridays we have seen.' But we must admit some error as soon as anyone tries to generalize about all future Fridays from that one statistic. A statistic used as an estimate must come with an acknowledgement that without seeing the entire population there will always be the chance of being wrong. A correct software system coupled with perfect data capture ensures that the sample statistic will be correct but there will always be a chance for error in its estimates. In evaluating the cost of deviating to algorithms that are only probabilistically correct or involve some form of approximation, we must concentrate on the increased cost caused by a possible increase in the rate or nature of errors.

There are a number of things to consider when determining the amount of error introduced when producing estimates, but all methods involve two numbers, a confidence interval and confidence level. These will depend on what you are estimating and how you go about it. Estimating a population maximum behaves differently in the presence of approximation than does the estimation of the mean or median. Approximating by taking every second sample may have a different impact than only looking at the first half of a sample. To begin one needs to look at the baseline error in your estimate as produced by an algorithm that uses no approximation. We may find that our estimate of Friday click through rate of 1.2% has a confidence interval of +/- 0.2% at a confidence level of 95%. This tells us that 95% of the time, when we try to estimate click through rate it will fall between 1.0% and 1.4%. We then examine how changing the method of measurement by introducing approximations will change the confidence interval for a given confidence level.

From this point we compare the marginal cost of error against the difference in cost of optimization by approximation and the cost of scaling using the current cost to scale. Approximation may chance the confidence interval from +/- 0.2% to 0.3%. How much this costs you is a question of risk management and depends on the economics of the business that you are in. Click through rates might be used to tweak an advertising budget. If you are buying ads on a cost per click basis, then the cost of the increased uncertainty in click through rate will be determined by the cost of each click. The cost of implementing an approximation versus traditional scaling will be determined by how analytically optimal your current system is and the costs associated with scaling be it by hardware, software or approximation. If the increase in cost caused by approximation is lower than the cost of buying more hardware or implementing more complex algorithms, then cheat.

Comments (1)

Apologies about the formatting here, LaTeX doesn't look so pretty as HTML. For a pdf version, click on the words 'pdf version'.

People often ask me what i2π means. To me, i2π represents the pleasure of stringing together simple principles to arrive at a beautiful understanding of the nature of something. Here I am going to share with you the way I think about i2π in a mathematical context. This doesn’t represent the full history or complete derivation, but it is how I like to think about things. If you want to learn more, Wikipedia is a good place to start. If you want to see things through my eyes, for just a moment, stick around. If you scroll ahead it may look a little scary to the mathematically uninitiated, but we won’t be using anything more complicated than mid-range high school math to get there. And I think the journey is worthwhile.

To understand what i2π is all about we really have to start with Euler’s constant, e. Euler’s constant is a special number in mathematics and it appears in many equations across mathematics, physics, statistics and economics because it has a number of unique properties. My favourite property is a variation on what is know as Euler’s identity:

And that is why we are starting our i2π journey with e. My next favourite property of e is the relationship between e^x and its derivative , namely

Those who have taken high-school calculus take this for granted. Those who didn’t or who have forgotten it are probably scratching their heads. For their sake, and my own, lets quickly review derivatives.

Derivatives are a convenient way of describe the slope of a line. Take the equation for the line y = 2x, then for each unit increase in x, we get 2 units of increase in y. The slope of a line is the ratio between the change in the output of the function that describes it with respect to the change in the input. In this case, x is the input and y is the output, and increasing x by 1 increases y by 2, so the slope is 2:1 or simply 2. The notation describes the slope of some function f(x) as x changes. The equation y = 2x has the same slope for all values of x, so we say the slope is a constant. More complicated functions, like f(x) = x² (in the figure below) are curved, so the slope changes as we change x.

Without going too deep into calculus, it is known that = 2x. In other words, for any point x, the slope is 2x. If you look in the figure above, when x = 0, the slope is 0. As x increases to the right of zero, the slope gets steeper and steeper. To the left, as x becomes increasingly smaller than zero, the slope becomes increasingly negative. In fact, for any k

For example, = 7x⁶.

When you take the derivative (find the slope) of most functions, the answer is usually some modified form of the function you started with. However, the exponential function, e^x is the simplest example of the case where the derivative is equal to the function. Up to this point, we haven’t even worked out what the numerical value of e is, but let us try to define e by starting with the fact that = e^x. To do this, we will need to take a detour into the world of factorials.

Imagine that we have 5 books that we want to place on a shelf. How many different ways can we arrange them? Working methodically, from left to right, there are 5 possible books we could put in the leftmost spot on the shelf. Once we choose the first book to place there, we are left with 4 possible books to place along side it. And once we choose that book, 3 books remain. After placing the third book, only 2 more remain, and so on. This means there must be 5 × 4 × 3 × 2 × 1 = 120 ways of arranging those 5 books. If we had 100 books, it would take up too much paper to write down 100 × 99 × 98 ×… × 1, so we use the shorthand 100!, which we pronounce ’100 factorial’. One hundred factorial is a big number. If we built a book arranging robot that could do one billion arrangements per second, and had one billion of them running in parallel since the big bang, they still wouldn’t be finished trying all the possible ways to arrange the books. Thank god for the Dewey Decimal system, eh?

Ok. So factorials are just a shorthand way of writing down a special type of successive multiplication. We can use the formula for the slope of x^k to find the slope of

We can re-arrange this to be

If you multiply some function by a constant, C, then the slope is also multiplied by C. From the rule we saw 2 paragraphs ago, we now that

We also know that k! = k × (k - 1)!. For example, 6! = 6 × 5! = 6 × 5 × 4 ×… × 1. So

Cancel out the k in the numerator and denominator, and we get

Now lets tackle a slightly more complicated function,

The first 2 terms of this function are pretty simple. Recall that factorials count the number of ways of arranging objects. There is only one way to arrange zero books, so 0! = 1. Also recall from math class that x⁰ = 1, so

There is only one way of arranging one book and x¹ = x, so

Therefore

I know this comes out of nowhere, but just go with flow. What is the derivative of this function? Derivatives are additive so we can just do each bit individually and add them together. The derivative of 1 is 0, as it is a flat line - hence no slope. The derivative of x is 1, as it is a line with a 45 degree slope. And we just worked out the rest in the previous paragraph, so

Hey, wait up. If we drop the leading 0, which we can, then that’s just g(x) again.

To see this requires some majorly deep and majorly simple insight. When things are deep AND simple, they are beautiful. What just happened was that even though the first term of the sum disappeared by turning into zero, the rest of the sum remained. Because we defined g(x) to go on from + … to infinity, for each term that drops off the front, there are an infinite number of terms to make up for it. Infinity less one is still infinity.

Recall that we defined e^x to have the property that

and we now have a function where

These properties are the same. The function has itself as its derivative. And it just so happens that g(x) is e^x. To understand why, we need to look at Taylor’s Theorem. So point your browser to Wikipedia if you are so inclined and join us in the next paragraph to continue.

Great. Welcome back. The point of this post is to explain i2π and so far we have only covered e, so let’s get a move on and have a look at i and π. I assume everyone is cool with 2.

What is i? i is the square root of negative one.

So, i² = -1. When I first encountered i I asked a family friend / math professor to explain it to me. All the books I had read just talked about ’complex numbers’ and I wanted to understand what made them ’complex.’ She explained to me that they aren’t complex, in the sense that complex means difficult. They are just different to the normal numbers we usually encounter. In school you would have learned that the square root of negative numbers is undefined. But they turn up so frequently that man invented a new class of numbers to allow us to define them. Numbers are just symbols for the abstract concept of quantity. And i is just a symbol for the square root of negative one. While it is not possible to have i books or bananas, we can still do mathematics with i and end up with real world numbers. For example,

And i⁴ = i² ×i² = -1 ×-1 = 1. So while we can’t buy i bananas, we can buy i⁴ bananas, because i⁴ is 1. As you keep on raising i to higher and higher powers, a pattern emerges. i¹ = i, i² = -1, i³ = i² ×i = -i and i⁴ = 1. When we look at i⁵ we find i⁵ = i⁴ × i = 1 × i = i, and the pattern repeats. For no apparant reason, lets sum up all the powers of i:

To see the pattern more clearly, lets split up the odd an even terms

Out of sheer curiosity, lets find out what pattern would we get if we expanded e^ix using the formula we found before.

We know that the pattern in the i’s comes out nicely if we split out the evens and odds, so lets call the even part of the right hand side of the equation a(ix) and b(ix) for the odds:

Now using the -1 + 1 - 1 + 1 -… pattern, we get

Likewise for the odd terms,

Now when you went across to Wikipedia to check out Taylor’s Theorem, you will have seen that

Which is totally the same as our a(ix). I may not be hip and fresh with the rock'n'roll and skateboarding, but I know cool when I see it, and that is cool. We also know that

Which means that b(ix) = i × sin(x). Putting a and b together, we find that

Now that we have gone from summing polynomials to trigonometry, it may be coming clear where the π fits in. π is a special number that defines the ratio between a circle’s diameter and its circumfrence. If a circle has a diameter of one furlong, then its circumfrence will be π furlongs. π is also used to measure angles, the same way as degrees are. In a circle there are 360 degrees, but mathematicians like to say that a circle has 2π radians. That is, if you have a circle with a radius of 1 foot, then the circumfrence will be 2π feet. If you were to walk around this circle through 1/8ths of its circumfrence, you will have moved 45 degrees, or radians.

The sine and cosine take an angle in radians and tell you the x and y coordiants of that point on a circle. If you walk 45 degrees anti-clockwise around a circle starting from the point (1,0), then you will end up at position

If you walk 2π radians around the circle, you will have gone 360 degrees and end up where you started. So = (1,0)

So the function e^iω tells us the coordinates of where one ends up after walking ω radians around a circle of radius 1. We have been writing down our coordiates as (x,y), where x = cosω and y = sin(ω), but as we found out earlier,

If we think of i as being a symbol to represent our distance in the y direction, then we can convert from x + iy to (x,y). And if we walk around a full circle and end up at the beginning point of (1,0), we can convert back to 1 + i × 0 = 1. Therefore

I like i2π as this term comes up across a range of mathematical equations and you can go a long way in learning about mathematics by looking in the origins of this famous identity. Simplicity, depth & beauty.
Comments (4)

I recently rejoined twitter and was completely blown away by the amount of discussion on twitter that is about twitter. I am reminded of a friend who once told me that the only thing you learn by taking drugs is how to take drugs. Conscious of this, I have refrained from mentioning twitter on twitter, but given that I have talked about it before on this blog, I feel OK bringing it up here.

This morning I saw a post from my friend Aaron:

neonarcade I wish Twitter was more about what people are doing again, and less a generally boring conversation about Twitter and social media.

I was compelled to act. My initial thought was to reskin twitter, or make a firefox plugin that removes all mentions of twitter from the feed. However, by the time I reached the office I settled on a single service site, as is the style of the time. So I now present Twit Or Love. Right now, the word 'twitter' has been mentioned 1.54x more frequently than the word 'love.' I use a highly accurate Markov-Chain Monte-Carlo technique to arrive as this unbiased estimate. My Gibbs sampler (unrelated to melting snow with salt) is coded in Erlang with the front end dynamically generated by a Scala program that writes out Ruby on Rails code. It is hosted on EC2. And uses map reduce. Enjoy

Comments (2)

This weekend marked the first time I shoveled snow. To cope with the odd sensation of being very cold while being very hot & sweaty, I thought about the mechanism by which salt melts ice. It has been a while since I studied physics and chemistry, but I had a basic notion of what was going on. I don't want to give it away early on in this post, so I'll share with you the chronology of my process of trying to confirm my understanding of why salt melts ice.

I started off with a simple Google search for "salt snow." The top link was to a page on About.com (as often is the case.) And the article was totally useless (as often is the case.) The page gave me some information, specifically it told me

Freezing point depression is a colligative property of water. A colligative property is one which depends on the number of particles in a substance. All liquid solvents with dissolved particles (solutes) demonstrate colligative properties. Other colligative properties include boiling point elevation, vapor pressure lowering, and osmotic pressure.

All true. But no knowledge is contained within that paragraph. I followed some links and came across another page trying to explain the mechanism. Coupled with the animation, the crux of the argument is presented in the paragraph:

Adding salt to the system will also disrupt the equilibrium. Consider replacing some of the water molecules with molecules of some other substance. The foreign molecules dissolve in the water, but do not pack easily into the array of molecules in the solid. Try hitting the "Add Solute" button in the animation above. Notice that there are fewer water molecules on the liquid side because the some of the water has been replaced by salt. The total number of waters captured by the ice per second goes down, so the rate of freezing goes down. The rate of melting is unchanged by the presence of the foreign material, so melting occurs faster than freezing.

Here we get a better sense of why - namely it has something to do with equilibrium between the solid and liquid phases of water. They don't mention it in the paragraph, nor does the animation give any indication, but the solid phase of water (ice) is a crystal. Water molecules share electrons between oxygen and hyrdogen forming H20. In crystalline form, these molecules align to form an organized structure. Wikipedia starts heading us along an interesting route as a full third of the discussion of Ice Ih (common ice) is dedicated to Proton Disorder:

The protons (hydrogen atoms) in the crystal lattice lie very nearly along the hydrogen bonds, and in such a way that each water molecule is preserved. This means that each oxygen atom in the lattice has two protons adjacent to it, and about 101 pm along the 275 pm length of the bond. The crystal lattice allows a substantial amount of disorder in the positions of the protons frozen into the structure as it cools to absolute zero. As a result, the crystal structure contains some residual entropy inherent to the lattice and determined by the number of possible configurations of proton positions which can be formed while still maintaining the requirement for each oxygen atom to have only two protons in closest proximity, and each H-bond joining two oxygen atoms having only one proton. This residual entropy S₀ is equal to 3.5 J mol⁻¹ K⁻¹. There are various ways of approximating this number from first principles. Assuming a given N water molecules each has 6 possible arrangements this yields 6^N possible combinations. Given random orientations of molecules, a given bond will have only a ½ chance that it has exactly one proton, or in other words, each molecule has a ¼ chance that its protons lie on bonds containing exactly one proton, leaving a total number of (3 / 2)^N possible valid combinations. Using Boltzmann's principle, we find that S₀ = Nkln(3 / 2), where k is Boltzmann's Constant, which yields a value of 3.37 J mol⁻¹ K⁻¹, a value very close to the measured value. More complex methods can be employed to better approximate the exact number of possible configurations, and achieve results closer to measured values.

Hmmm. Entropy.. Equilibrium.. Sounds like Thermodynamics. Some more clicking gets us to an explanation in terms of Gibbs Free energy. The answer given on the page basically states that by the first and second laws of thermodynamics, salt melts ice. Why? Well, by the second law, entropy wants to increase and salty water has greater entropy than ice so it is a lower energy state, and for energy to be conserved (the first law), ice will turn into water to maintain the total energy of the system. This is certainly a more satisfactory answer than 'because water has colligative properties.'

But why does salty water have greater entropy than pure water? What is entropy?

To understand this, we have to think at the level of statistical mechanics. Entropy is a convenient probablistic measure of complicated stochastic processes. Entropy (in the thermodynamic sense) is a measure of disorder in a system. As we saw from the description of proton disorder in ice, entropy measures the number of different ways the components of a system can be arranged to firm different structures. Without diverging too far into a discussion of urns filled with different numbers of red and blue balls, it is easy to see that for ice to be ice there are only a relatively small number of ways by which the hydrogen and oxygen atoms of water can be arranged with respect to eachother. However, in liquid water, there are many many, but not an uncountably infinite number of, ways by which you can arrange the water molecules and still have liquid water.

The key thing about the role of entropy in melting snow is that each water molecule is indistinguishable from another. Thus, when considering the number of states (or arrangements) that would result in liquid water, swapping two molecules of water would not count as a new state. As you add a solute such as salt to the liquid the number of distinguishable states increase even as the total number of molecules in the system remain constant.

We began this search with a proof by definition, namely that salt melts ice because the fact that salt melts ice is a colligative property of water. We then saw found that salt melts ice because it minimizes the Gibbs free energy of the system. We then found out that this was due to increasing entropy. Along the way, I may have inadvertantly introduced a proof by obfuscation.

Of course, nothing I have said during my hand-waving explains why salt melts ice. But we know that it does. The thermodynamic explanation relies on a model of the underlying statistical mechanics - entropy being a descriptive statistic of an underlying complex system. And the underlying statistical mechanics is just a model of the statistical quantum dynamics, and so forth until turtles. Salt doesn't always melt ice, it is just very likely to do so. Much of microeconomics follows a similar path of evolving explanations, but the predictive power of the results depends on the quality of the model of the individual agents. Water 'wants' to maximize entropy and conserve energy, but not as much as people want to maximize a simple model of utility.

---

It would be remiss of me to end this Wikipedia link fest without mentioning Shannon entropy from information theory.
Comments (0)

Via God Plays Dice I came across a great list of mathematical proof techniques, of which my favorite is probably Proof by Induction:

Proof by Induction

Proof by Induction claims that

where is the number of pages used to contain the proof and is the time required to prove something, relative to the trivial case.

For the common, but special case of generalising the proof,

where is the number of pages used to contain the proof, is the number of things which are being proved and is the time required to prove something, relative to the trivial case.

The actual method of constructing the proof is irrelevant.

Great jokes don't need explanation, but I just can't help myself. The equation listed above is actually Faraday's law of induction, which describes how a changing magnetic field induces an electrical field. C.f. mathematical induction, which is a method of mathematical proof typically used to establish that a given statement is true of all natural numbers. It is done by proving that the first statement in the infinite sequence of statements is true, and then proving that if any one statement in the infinite sequence of statements is true, then so is the next one.

Mathematical induction is often misunderstood by those trying to apply it, often resulting in what Uncyclopedia calls Engineer's Induction:

Suppose P(n) is a statement.

1. Prove true for P(1).
2. Prove true for P(2).
3. Prove true for P(3).
4. Therefore P(n) is true for all .

And to snuff out any life remaining in the joke, the technique above only proves P(n) for n = 1, 2 & 3. To make it a real proof you would need to show that if P(n) is true for some n then it must be true for n+1, which coupled with the proof for P(1) would show that it was then true for all natural n.

Comments (0)

I like Felix Salmon. He's a pretty awesome finance and economics blogger and on the few occasions I've spoken with him, he seems like a genuinely nice guy. But a recent post left me scratching my head:

John Gapper worries that taxing non-diet sodas would be regressive. I worry that it wouldn't even do what it is designed to do, which is reduce obesity: all it would do is increase the amount of diet sodas consumed, and there's a strong link between diet-soda consumption and increased obesity.

Unless and until there's some empirical evidence that switching from non-diet to diet sodas helps people lose weight, this should be filed under "very bad ideas".

If you follow that strong link link, you'll see:

Fowler is quick to note that a study of this kind does not prove that diet soda causes obesity. More likely, she says, it shows that something linked to diet soda drinking is also linked to obesity.

"One possible part of the explanation is that people who see they are beginning to gain weight may be more likely to switch from regular to diet soda," Fowler suggests. "But despite their switching, their weight may continue to grow for other reasons. So diet soft-drink use is a marker for overweight and obesity."

Right. Correlation is not causation. In fact, if the elasticity of demand for soda is not correlated with propensity for obesity, then a Pigovian tax on soda would make for a great instrumental variable to see whether diet soda has any causal relationship with obesity.

---

Today's xkcd comic is fantastic. Check it out.
Comments (4)

I was looking for some code that would allow me to automagically populate my database with transactions from Wesabe. It needed to work with Postgres, and Perl would be a decent language for the job. I found it here. Enjoy.

Comments (0)

It is good to know that $1,793 power cables are useful even without replacing the miles of standard copper between the power station and your house.

George,

Shortly after receiving two 5ft Quadlink power cables, I dragged them to my office. I am employed in a 24/7 engineering support environment and thought my "always on" computer would be a good starting point to burn in the cables. Working 12 hour night shifts I have grown accustomed to eye fatigue while working with a computer. Several busy days later (currently on my 9th straight 12 hour night) other priorities and a busy work schedule had caused me to forget about the cables burning-in beneath my desk. Yesterday I noted how uncharacteristically good my eyes felt. Today, just an hour ago, a coincidental look at my feet put me face to face with the Quadlink.

I have tried to identify another cause for this remarkable change but am unable to attribute the effect to anything but the Quadlink power cable. I trust you'll find that satisfying.

Needless to say I intend to purchase a replacement cable for my home audio use as soon as possible - I don't think this one will be leaving my computer monitor anytime soon. I am anxious to loan my cables to others in the office who have complained of similar eye fatigue problems.

Best Regards, Wade

See here. And then here.

---

WTF: Where does a night-shift support desk worker get the $ to spend on such crap?

Comments (0)

A few weeks ago at lunch we were discussing forensic accounting and Benford's law came up. I've always been a little uncomfortable with the law, which stipulates that the distribution of leading digits in 'natural' measurements follow a non-uniform distribution. I believed the law but never had a good grasp of why. In undergrad I recall proving the existence of the law by starting with the assumption that if such a law exists, then it must be scale invariant. That is, if there is some law that describes the way leading digits of measurements are distributed, then it shouldn't matter whether you measure the underlying quantity in Celsius or Farenheit; that is, it should remain after any linear transformation. From that point its pretty simple to derive the distribution. But it was a fairly weasely way to answer the question as I started with the assumption that the law must exist. Mathematical proofs frequently start with the assumption that the converse is true and then show that if the converse were true it would be non-sensical, hence the converse of the converse must be true - this is called proof by contradiction. Proof by assumption of truth is not really proof. Either way, my 'proof' was accepted and my discomfort with the law remained until today.

Last night I was browsing through Hacker's Delight, a compendium of neat coding tricks that would get you fired at most workplaces with the charge of being too clever and abstruse. And in my reading I came across Bedford's law and decided that today I would finally take the time to grok it. As has become my default browsing behavior, I started at Wikipedia and came across a description and picture that finally let the pieces click. I also got to feel a little bit better about the shame that I have carried from my undergrad 'proof', as they include my same argument as evidence of the law:

The law can alternatively be explained by the fact that, if it is indeed true that the first digits have a particular distribution, it must be independent of the measuring units used. For example, this means that if one converts from e.g. feet to yards (multiplication by a constant), the distribution must be unchanged -- it is scale invariant, and the only distribution that fits this is one whose logarithm is uniformly distributed.

To neatly close off the issue, I logged into a database at the office and grabbed 123 million financial values taken from around 65 thousand companies from around the world. It is nice having access to this kinda data. And lo and behold the distribution of leading digits follows the law perfectly. The black line in the picture at the top of this post is the observed frequency and the red dotted line is what the law would have one expect. Cool.

 digit   observed   expected 
     1 0.30200305 0.30103000
     2 0.17712325 0.17609126
     3 0.12545546 0.12493874
     4 0.09689768 0.09691001
     5 0.07977275 0.07918125
     6 0.06637933 0.06694679
     7 0.05717501 0.05799195
     8 0.05021505 0.05115252
     9 0.04497842 0.04575749

Comments (1)

I have a love/hate relationship with Mint and Wesabe, two Web 2.0 online financial tracking applications. They do a few things well and have great potential but still manage to leave a bad taste in my mouth. The strength of both services is that they consolidate all of your transactions from your various banks and credit providers and present the information in a unified format. This alone is huge for me. Even within Chase's own website, I can't get a single consolidated view of all my accounts. However, the sizzle on the steak is that they take advantage of network effects in tagging your transactions. For example, when one user tags a cryptic line item such as '238927 12/04SOU EQUIN' as 'Gym Membership', all future transactions across all user accounts will benefit. That said, I still find most of my time using these sites (I have accounts on both) spent tweaking their automatic suggestions for tags. However I understand that me spending 5 minutes correcting their 10% classification error rate is better than spending an hour hand classifying every transaction [1]. I also take issue with some of the interface design choices, especially on Mint, whereby quick searching and sorting is prevented by their excessive use of 'pretty' transitions and AJAX style effects. I usually keep my mouth shut about UI issues, but given that I've spent the last week designing a visualization app, I feel suitably qualified.

But my biggest grief is one I raised both with the founder of Wesabe and on this blog at one time. I can't seem to find my original post on the topic, as it was lost when I went through the Blogger to Wordpress switch - I'll try and extract it later. But anyway, my point was that I was disheartened by the lack of double entry book-keeping. This problem manifested itself on Wesabe when it congratulated me for earning $5k when in reality I had actually just paid off a $5k balance on my credit card - net economic impact of far less than $5k. This annoys me.

This morning I was doing my semi-regular perusal of my accounts and came across a balance advance from a line of credit and I was curious as to where the money went. It is a royal pain to do arbitrary searches with either of the sites, so I opened up my SQL clone of their site that I scrape on a regular basis. From there it was easy to find all transactions within a time window that matched the dollar amount of the balance advance. This is a non-GAAP technique, but even for the 8 accounts my wife and I have, joining by transaction timestamp, amount and debit/credit is a pretty accurate way to build a financial journal - in the accounting 101 sense of the word 'journal.'

I don't know why these services don't offer this feature. It is easy to see useful extensions that bring much more color to our financial activity. Just in the same way that session management gives you oodles more insight into online marketing transactions than simply looking at individual requests in a web server log file. Now maybe I'm the only one nerdy enough to specifically request double entry accounting, but in these times everyone wants to know more about their financial data. Couple this data with logs from my cell phone bill, cable service, Netflix and we'd soon be able to fairly accurately recognize the rationale being each transaction with our banks. If only there was some kind of online Vault service that could own this data for me...

---

[1]: It is also really difficult to determine their false positive rate. It's easy enough to work out when they can't work out a tag, but sometimes it takes quite a while to realize a $20 purchase tagged as 'Pet Food' was really 'Movie Tickets'.

PS: Note to Mint.com, stop calling your inane pie-charts 'Trends', you are starting to piss me off.

Comments (1)

The real take away from my last post is that I am a man of mystery. To that effect:

Comments (0)

Sometimes I forget that Knuth is not the be all and end all of algorithms. So when I couldn't find what I was looking for last night, I slept on it hoping that something would come to me in the morning. And as hoped, on the train I found what I was looking for. It is a great feeling when your mind is jumbling around so many concepts and then everything clears away to reveal a solution. Of course, as soon as I got into the office and did some googling, I found out that this was all worked out years ago. Nonetheless, it is fun to think about.

I was looking for a spatial index for high dimensional data. I know a bit about various spatial indexes and the best approach that I knew of was an R-tree or one of their variants. For the uninitiated, R-trees are multi-branching trees that extend from the B-tree concept, but instead of indexing linear regions, R-trees index multi-dimensional rectangles. Postgres uses R-trees and previously I had made some minor contributions to that codebase. From my possibly flawed understanding of their implementation, the tree is arranged only with consideration of the marginal distributions of the dimensions. In cases where you have correlated variables, you can do better than that.

Lets say we have a 2-dimensional data set, where y = f(x) + e, where e iid. Normal. In this case, let f(x) be a simple linear transform, but everything that follows should work for any monotonic function:

If we want to find all points that are within a query box {(x_min,y_min), (x_max,y_max)} we can use our knowledge of the correlation to get us there. The x-range of our selection is [x_min, x_max]. This range is shown as the solid green lines. We can use our estimated inverse of the function y = f(x) + e to turn find our y'-range - the dashed green lines representing [f'(y_min), f'(y_max)]. (NB: I'm using f' to denote the inverse of f, not the first derivative.) Now instead of storing these points in a 2-dimensional tree, like a quadtree, we can use our estimate of f(x) and store our points in a binary tree. To find all the points inside our query box we can take the range that encloses where we expect the points to be, given our estimate of f(x), and use that to search the space. Naively, this range is [max(x_min, f'(y_min)), min(x_max, f'(y_max))], assuming f(x) is increasing. Call this range the selection range - it is shown in orange below: This gets us part of the way there. However, we want a selectivity and specificity of 1 - that is, we want to find every point that is inside the box and reject any points that happen to fall out of the box. We need to improve on our selection range to take into account the error term, e. In the figure below I have included the prediction interval as two dotted red lines above and below the best fit. I have arbitrarily set the prediction level to 80%. This means that 80% of the y values fall within these lines for any given x. Knowing this, we can take 80% of our points and index them only using the x value. We adjust our search range using the prediction interval and our selectivity increases.

We can now apply this approach to design a tree structure that utilizes our knowledge of the relationship between the variables. Start by dividing the dataset by the median of x so that half the points are to the left, and half to the right:

Note: we could have divided by y, but I just chose x by convention.

Most of the points to the left of the orange line have y values less than f(median(x)), and conversely for the points to the right. We can quantify 'most', by looking at where the 80% prediction interval intersects the y=f(median(x)) line. To the right of the right-most dashed line, 90% (= (100% + 80%)/2) of the points are above the y ordinate where the prediction interval intersects the line, shown as a dashed purple line. We call the points below outliers and highlight them with red circles.

Using this we can now construct a binary tree much like what you would do if you were only keeping track of x values. The only difference is that at each node you keep a list of the outliers in the y direction.

---

NB: the R source code for this page is here
Comments (2)

I finally got around to reading Memos from the Chairman today after having it languish on my ever growing book inbox for the past few months. Perhaps my purchase was prescient of Bear Stearns' demise, but nothing more so than the first memo in the book, written by Alan Greenberg on Oct 5 1978:

Bear Stearns is moving forward at an accelerated rate and everybody is contributing. It is absolutely essential for us to be able to talk to our partners at all times. All of us are entitled to eat lunch, play golf and go on vacation. But you must leave word with your secretary or associates where you can be reached at all times. Decisions have to be made and your input can be important!
I conducted a study of the 200 firms that have disappeared from Wall Street over the last few years, and I discovered that 62.349% went out of business because the important people did not leave word where they went when they left their desk if even for 10 minutes.
That idiocy will not occur here.

Flash forward 30 years and we have stories like Where in the World is Bear's Jimmy Cayne? Playing Bridge.

Last year, when he was still chief executive of Bear Stearns Cos., James Cayne took heat for hitting the bridge circuit during troubled times for his firm. Will the same rules apply to Cayne now that he's chairman?
We'll soon find out. Thursday and today, as Bear fought off a pending cash crisis that threatened to ruin its business, Mr. Cayne who relinquished his CEO title in January and become the firm's non-executive chairman - has been in Detroit, playing in the North American Bridge Championship.

Three days later, Bear is acquired by JP Morgan.

Comments (0)

(click for more)
Comments (0)

A few years ago I started specing out a project to build my own customizable video camera rig. After writing up some code to simulate various sensor types I decided to get my hands dirty with this project, which was a great success. In the following year or so I got busy with a new job and didn't have the same resources as a certain eyewear mogul and didn't follow up. While it was a fun project for me to work on, in the back of my mind I was thinking about the possibilities of commercializing my idea. Since then, RED was launched and is a smashing success.

Maybe I should dust off the old calculator project and get cracking before Dov Charney decides that it is a hot opportunity.

Comments (0)

Los Angeles was the Australia of the art world.

Comments (0)

The web is full of bullet-point lists of handy tips for job applicants. Some of them are OK, some are crap. I'm not sure which direction my post is going to go, but it's certainly what I am looking for in applicants.

1. Read the job ad. If the ad asks you for a particular piece of information, give it. If you don't you will end up looking like one of the RAND_MAX people who apply for everything that appears on Craigslist.
2. Write a cover letter. Especially if the ad asks for good communication skills. Don't send your cover letter as an attachment, just treat the email body as your cover letter. If you aren't feeling particularly creative a standard 3 paragraph letter works well:
   1. Who are you / what attracts you to the job
   2. What skills do you have? A small anecdote, e.g., "In my last job I used my awesome C# skills to fix the blah blah blah system..."
   3. Request to be contacted
3. You must custom write the cover letter to the job you are applying to. You want to spend the next few years working with me, then spend the next few minutes putting some effort in to getting the job.
4. This probably doesn't apply if you are dealing with some company with a dedicated team of HR drones who stick your resume in a database, but I'm not a drone.. Therefore, send your resume as a PDF or as plain text. I like people who pay attention to detail and MS Word resumes always look crap on my computer because it is not your computer. All of your funny spelling and odd grammar shows up with squiggly lines. A PDF lets you format the resume exactly how you want me to see it. And you get bonus points if your resume was generated in LaTeX.
5. Keep your resume short. I don't care what you were doing 10 jobs ago, especially if it is pretty much identical to what you are doing now. I don't care about the details of the various internal systems you have worked on. Give me your last 2 or 3 jobs, and a short summary of previous work if you want.
6. Spell correctly and use a close approximation to correct grammar. And don't misspell a computer related term. Have someone read your resume before you send it out.
7. God save me from those who include a list of 100+ technologies they know. If you are truly proficient in 35 languages, then say "Proficient in 35 languages, ranging from Ada to Fortran." And unless the job is specifically asking for expert knowledge in some dead technology like dBaseII, don't bother listing it. I don't care.
8. This probably means that you should tweak your resume to each job you are applying to.
9. Please include a little bit on your resume showing something about who you are rather than simply what you do. I'd like to know if you ballroom dance, play the cello or play Go.
10. Most resume tips warn you against anything that might result in your application being ignored. I don't ignore applications - while I may not read them all completely, I do want to find the right employee so everything gets at least a glance. And if the application is egregiously bad I'll certainly pay attention; the laughs alone are worth it.

OK. That's 10 and while I could go on, I feel better already.
Comments (4)

I've tweaked my blog 'system' a little bit this afternoon. Instead of being a pile of bash scripts, now it is a pile of bash and perl scripts. Unfortunately, all my blog feeds will be reposted, but on the plus side I now have a copy of all my old posts from various other platorms. Oh, and RSS feeds should now work again. They have been broken since the last time I messed around with the system. I spent a little bit of time writing some javascript to scrape my old blog posts from inside Google Reader, which had the only copy of my old posts from wordpress and blogger.com. There is probably an easier way to extract the posts, but I can always do with some practice on the reverse engineering front.

Ok. That's enough housekeeping. Back to my regular blogging schedule

Comments (5)

It has to be lower than McCains, which as far as I know is infinite.

Comments (0)

Let me just reiterate how much I like Amazon's recommendation service. Collaborative filtering at it's finest, if you ask me. Collaborative filtering is the process of using what a system knows about others to make recommendations for an individual. If most people who like A and B also like C, then if you like A and B, you will probably like C. It is a simple concept and it works well in many applications. Netflix implements a similar system, which also looks at people's ratings of the movies that they have rented. If it turns out that I like A and B, but really hated C, the algorithm will find other people like me and perhaps recommend E instead of D.

Most of this is widely known. And then the human interest stories started popping up, the most famous of which covering the 'My Tivo thinks I'm gay' phenomenon:

Mr. Iwanyk, 32 years old, first suspected that his TiVo thought he was gay, since it inexplicably kept recording programs with gay themes. A film studio executive in Los Angeles and the self-described "straightest guy on earth," he tried to tame TiVo's gay fixation by recording war movies and other "guy stuff."

"The problem was, I overcompensated," he says. "It started giving me documentaries on Joseph Goebbels and Adolf Eichmann. It stopped thinking I was gay and decided I was a crazy guy reminiscing about the Third Reich."

While these stories are cute, they highlight the pitfalls of making recommendations that just don't fit. I've been noticing more news stories and blogs including related links at the bottom of their posts. They do this to increase the stickiness of their site, by redirecting traffic back to the original web site. It is a nice idea, but with the prevalence of good recommendation technologies, the slightest hint of bad is enough to train users to ignore such related links sections.

Computer generated related links tend to suck because, unlike collaborative filtering, the technology tries to use high-power semantic analysis to understand the text and come up with related articles. And text analysis technology just isn't that good right now. At my firm we use some fairly run of the mill text processing to identify similar companies based on SEC filing text. This tends to work well as we have a large corpus of highly idiosyncratic text to work from. Short blog posts wouldn't work well with simple techniques, so more complex techniques are used. And they suck. Even worse, none of these technologies attempt to use any user feedback to help train the algorithms. I don't want to see explicit, click-here-if-you-think-this-story-is-related buttons, but at least track which links are clicked on to help optimize your algorithms.

My gut feel is that for the developers to prove the value of their technology they are compelled to include related links in as many stories as possible. This is a wrong-headed move. They should only include links when relevance is well established; otherwise users (like myself) will quickly learn to ignore that junk text that appears at the bottom of blog posts and articles. It took me a few days to subconsciously ignore the RSS feed 'flair' at the bottom of most posts. And now that my ignoring neurons are well trained, it took me all of nano-seconds to realize that 'Related Links' are junk.

Related Links:

• Price of tea in china
• Northenmost point in Europe
• List of Star Trek episodes (Polish)

Email this   Digg this   Stumble it!   Share on Facebook   Share on MySpace   Link to me on LinkedIn  

Comments (0)

Comments (0)

Comments (0)

This is a bit of a long post. It stems from a conversation with friends last night where I tried my best to explain what is going on with the American financial system. My central theme is the dangers that arise from the decoupling of risk from reward and I take a tour of mortgage brokers and borrowers, accounting standards, credit rating agencies and finally the credit default swap market. The opinions, misunderstandings, gross simplifications, omissions and outright errors in the piece are wholly my own.

Mortgage Risks

Two main risks concern those bankers who originate mortgages: prepayment risk and default risk. Prepayment risk is the chance that during a decline in interest rates, borrowers will pre-pay their mortgages as they refinance at a lower rate. This hurts lenders as they originated the mortgage expecting 30 years of juicy interest payments. If a borrower pre-pays then the lender receives the principal balance, and possibly some fee income, but loses out on the opportunity for future interest income.

Default risk is the chance that a borrower will just stop paying altogether. Lenders would compensate themselves for this risk by looking at the credit rating of the borrower and charging a higher rate for those who show quantitative signs of default risk – as determined by their credit score, income and debt level and the size of the loan relative to the value of the property.

Or at least that is how the models work. At a higher level all risks can be placed on a spectrum from idiosyncratic to the systemic. Idiosyncratic risks are those that arise from situations that only impact the individual borrower, such as falling ill and having medical bills that take priority over mortgage payments. Systematic risks impact the mortgage market as a whole, such as volatility in interest rates. The whole idea of pooling together mortgages is that idiosyncratic risks are uncorrelated. Joe in Montana getting hit by a car is a statistically independent event from June in Miami getting fired. By pooling together these uncorrelated risks, the portfolio of loans has a lower idiosyncratic risk than any individual loan.

The cash flows generated by these pools of mortgages could be then segregated into tranches with different guarantees as to their payment priority. If there were, say, 1,000 loans in the portfolio, each paying $1,000 per month then the total monthly income would be $1m. The highest priority tranche would be set up so that each month $800k of the $1m would be paid out to those holders of the top tranche. Thus it would require 20% of the individual loans to default before the regular flow of $800k per month would be disrupted. Subsequent tranches would have higher risks of defaulting.

This structure meant that the top tranches would end up with a far higher credit rating than any of the individual mortgages, or even a simple pool of mortgages. As highly rated instruments institutions that traditionally were unable to buy assets as risky as mortgages could now enjoy the benefits of higher interest rates on their money whilst maintaining a portfolio of highly rated debt.

Consequently these products were in high demand. Couple this with the desire to allow more Americans to fulfill the dream of home ownership and an environment was created that allowed more and more people to attain mortgages. But somewhere along the line the risk-reward relationship that sits at the cornerstone of capitalism became dislodged.

Brokers + Regulatory Arbitrage

First there were the mortgage brokers. As poorly trained and lightly regulated commission sales agents, their role was to pre-qualify potential borrowers for mortgages. But as their income stream was tied to the number of applicants that passed the qualifications and ended up as borrowers, they had every incentive to make sure that those who applied for loans got them. And as the mortgage brokers bore no personal liability, apart from being charged of criminal mortgage fraud, many bent the rules and flat out lied along with applicants to get these deals done. This is what happens when people with no fiduciary duty are executing large financial transactions. However, on paper fiddling of applicant details wouldn't solve the problem of some of these applicants who simply couldn’t afford the monthly payments. For these borrowers complex ‘option ARM’ mortgages were offered, where initial payments were low but jumped after a certain number of years. These were issued to borrowers on the assumption that house prices would go up, giving borrowers the opportunity to refinance before their payment rates rose.

The brokers bore no liability from qualifying unqualified applicants, as they were not the ultimate source of cash that funded the loan. This cash was sourced by the banks that were happy as long as they could earn higher interest than what was available from other debt instruments with the same credit rating. Even better, from the banks' point of view, was that these mortgage backed securities and collateralized debt obligations were built in the form of special purpose vehicles and, as such, were not consolidated on the banks' balance sheet. If the account rules that govern consolidation of such vehicles sound familiar, you probably are remembering a thing or two from the Enron scandal.

As these new financial instruments weren’t listed on the balance sheet, important leverage ratios were not impacted. Regulation, such as the Basel accord, limits how much debt can be used to finance operations. Off balance sheet financing gave a way to game the system, a game known as regulatory arbitrage. Leverage is great for financial institutions when their bets are working for them. They can juice up their returns, borrowing at X% and earning money at 20 x (Y – X); which is all fine and well when Y is greater than X. It was this strict interpretation of the letter of the law, but not the spirit, that drove the demand for mortgages to be repacked. And as long as the mortgage backed securities received high credit ratings, this was a great way for bankers and brokers to make money whilst giving affordable housing to those who would not usually qualify for loans.

Credit Ratings

But the models that were used, both by the banks and by the credit rating agencies (often working together from the same bed) underestimated the impact of systemic risk. If house prices are artificially elevated by an environment where everyone is suddenly able to qualify for a mortgage due to diminished underwriting standards, then as soon as the music stops home prices should return to their natural, unsubsidized level. And that they did.

Across America home prices began to collapse in 2006 as lenders began to see the impact of the declining quality of borrowers. And anything that has an effect across America is no longer an idiosyncratic risk. Add on to that the fact that many borrowers had borrowed more than their house was worth, under the assumption that home prices would continue to rise, suddenly found themselves owing far more than their house would be worth at any point in the near future. This brought to an end the domestic spending fueled by home equity withdrawals, a significant and growing part of GDP over the past 10 years. At its peak between 2004 and 2006, mortgage equity withdrawals were about 9% of GDP, nearly five times larger than they were just ten years prior.

While corners of the press/punditsphere were predicting a severe decline in house prices, the prognosticators at the credit agencies seemed to lack the same foresight. Even worse, the agencies had no problem with slapping AAA credit ratings on vehicles filled with dodgy loans. Here again we see the consequences of a disconnect between risk and reward. In a situation reminiscent of Arthur Andersen, the consulting / accounting firm that audited Enron's books, the credit rating agencies also provided consulting services. If you were a bank looking to get a AAA rating, the rating agencies were more than happy to provide consulting services to ensure that the rating arm of the firm would get your desired rating.

As an aside, I don't really understand why ratings agencies even exist. Investors are more than happy to do their own leg-work and purchase stock in companies without any seal of approval from third parties.

Credit Default Swaps

There is another piece of the puzzle that also leaves me somewhat confused: credit default swaps. When you decide to loan money to someone, you are at risk of not receiving all of your money back if the other party somehow defaults on the loan. To get around this, you will often charge a higher interest rate to make up for the credit risk. But if you lend them money today, and they go bankrupt tomorrow, then you won't even get your first interest payment. Enter credit default swaps (CDSs).

In a CDS, you essentially buy insurance from a third-party that will pay out if your counter-party (the person you loaned money to) defaults. But CDS's are not regulated as insurance. As a homeowner it is fraudulent to take out multiple policies against your single home. If not, I could insure my house 20 times, burn it down tomorrow and buy 20 more the following day. This isn't the case for credit default swaps. I can buy as many as I like, and insure as much money as I want against the chance of some company going bankrupt. Although not regulated as insurance policies, many insurers got into this game – including AIG. It seemed like a no-brainer, here were companies looking to buy insurance against the possible default of AAA rated companies and bonds. With AAA ratings the purported likelihood of default was de minimus so the insurers were happy to oblige. And so the CDS market grew to cover about 50 trillion dollars worth of insurance - about the same size as the total sum of the entire US corporate bond market.

This worked great for the insurers of mortgage bonds until they started to default. But that's the risk you take as an insurer. However, the dollar value of the insurance can be much larger than the dollar value of the loss when more people are insured against a default than there are counter-parties exposed to that default. This becomes particularly interesting when insuring against corporate bond default. As more and more people buy insurance against the risk of a company going bankrupt, the cost of that insurance goes up – simple supply and demand. And as the cost of insurance goes up, the market perceives a greater risk of default and demands higher interest rates on bonds issued by the target company. Thus the cost of financing goes up and it can get to the point whereby the company's cost of capital is greater than their return on equity, and the company can no longer operate profitably. The more reliant a company is on debt to fund its operations, the more risk they are exposed to from their cost of financing. And companies that rely of high levels of leverage, by definition, are highly exposed to such risks.

And then...

This is what was playing out over the past few weeks. Banks, using their low cost of capital, to trade in highly-(mis)rated mortgage instruments, earned highly levered returns to fuel their profits. Seeing this as a problem, people took the opportunity to buy credit default swaps against these banks, and at the same time short sold their equity to doubly profit from the demise of the bank. Depending on your point of view, this is either capitalism at its finest, preying on the weak to allow the strong to survive. Or it is rampant manipulation, using credit default swaps to force banks into bankruptcy. In reality, it is probably a bit of both.

Short-selling, while still not widely understood by the folk outside of finance, has been blamed by many for the market activity of late. This is like a poor student failing a test and blaming the teacher for making the test too hard. As best as I can tell, there is hardly any impact on a firm's ability to operate in the face of declining stock prices. So unlike speculative purchases of default swaps, which negatively impact a firm's ability to borrow, short-selling stock should not decrease the ability of a firm to continue operating. [See here] Nonetheless, regulators in the USA, the UK and Pakistan(!) have intervened in the market to limit short-selling. But they have done nothing to address the underlying disconnects that led to our current predicament. Credit default swap markets remain lightly regulated and highly opaque.

The Treasury wants to buy up 'toxic' mortgage debt to the tune of half a trillion dollars and while this will probably solve the problem quickly, it may not be the best way of doing so. The way I see it, doing so is simply an extension of the stimulus check program, but with benefits going to the 5 million odd borrowers from the past decade who already spent that money. It is like putting a helicopter in a time machine and handing out free money to those who responded positively to the requests of fraudulent mortgage brokers to lie on their loan applications.

I'm not sure what the right solution is. There probably isn't one. Certainly nothing that won't be painful. But I do hope that capitalism restores the relationship between risk and reward and aligns incentives so that we don't find ourselves in the same miss any time soon.

:wq

Comments (0)

“Panics do not destroy capital; they merely reveal the extent to which it has been destroyed by its betrayal into hopelessly unproductive works.”

-- John Stuart Mill
Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

---

*Only traders and mothers know how to send out emails containing .bmp image attachments.
Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

18 / 06 =  3  (i)
18 - 08 = 10  (ii)
 3 x 10 = 30  (i x ii; QED)

4eyes

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Via Junk Charts:

(Thanks to reader Josh R. for the tip.) The "plucky statisticians" at Urbanspoon decided to tackle the political hot potato: is Barack Obama an elitist? Scratch that -- what they actually did was to determine if Obama supporters were elitists (of course, Obama would then be, due to guilt by association.) Scratch that -- what they actually analyzed was if there tended to be more Starbucks per capita in those states in which Obama won Democratic primaries.

Hey, that Josh R. character sounds like a mighty fine bloke.

Mr. Urbanspoon, the statistics professor is here and he disapproves.  As discussed before (and here), plotting two series of data on the same chart and applying two different scales is a recipe for disaster.  Not reaching immediately for the scatter plot when one has two data series is another serious misstep.  (Indeed, Josh sent the link in with a note wondering why "people dislike scatter plots so much".)

Wow. I'm famous.

---

[1] I have no idea what that plot is of other than being the first image returned from my home directory search for scatter*
Comments (0)

Comments (0)

Comments (0)

Comments (0)

I tend not to speak up when I should. Especially when it comes to matters where I feel I lack the requisite expertise. I overcompensate by chiming in where I have an inkling that I know what I'm talking about. It seems that the wedding process is continually reminding me of the problems with this tactic. I'm not a graphic designer, but I feel like I know enough about Adobe Illustrator to know when our invites were incompetently composed. Come on, if you are going to pretend to draw a Banksia bush by using the vectorize tool with a photo you downloaded off the web, you better do it properly and fix up the artifacts. In the end I just gave up and just let them make it however they wanted. And it looked terrible, in my opinion, but hey it's only an invite - an immaterial part of the wedding.

For my suit, I want perfection. But I am by no means a tailor. I like the Duncan Quinn suits, and figured that they would take care of me. And maybe they will, I only just got back from my first fitting. However, I suspect that I may not actually wear the $5,000 suit on the big day. It is far easier to go through off the rack selections to find something that fits me the way I want than to be stuck with a piece of fabric that is constrained by the original cut.

You see, I have child-bearing hips. I can't help it, I just have wide and prominent hip bones. Whenever it comes up in conversation, usually constrained to clothes buying situations, but also the odd well lubricated party, I refer to my 'child-bearing hips.' Always good for a laugh. I definitely mentioned my concern at my initial meeting at Duncan Quinn. Perhaps not with the vociferousness concomitant with my level of concern.

"Make sure the back of the jacket doesn't flare out too much."
"Keep the pants low on my waist so that pockets don't flare."
"I like slim pants and sleeves, but my shoulders need to be broad -- I have child bearing hips, you see! tee hee hee."

I wore my Costume National off-the-rack suit in to the fitting today. The idea was to take a suit that I like, and ensure that the significantly more expensive 'bespoke' one would be better. It's not. On my way there I had my conversation all mapped out. When it came down to it, all I could manage was a moping mumle. The first thing that stood out to me was the shoulder breadth. It turns out that they can't let them out any farther as there is not enough fabric inside to allow it. I didn't mention the height of the pants - I was too concerned with the jacket.

Of course, I'm now sitting at home stewing over this. Not so much the cut of the suit, but more that I didn't kick up a bigger fuss. One day I'll learn.

Comments (0)

Every few weeks I clear out my Amazon shopping cart. It seems like every other day I find a book or a CD that I am interested in and pop it in the cart to buy later. Today is that day. I have about $300 in books on American economic history, four minimal (a.k.a highly repetitive) electronic music CDs and a Bollywood movie that I found on YouTube. My co-worker suggested that I check out half.com to see what I could find at a discount. I have no problems with saving money, but the thought of distributing my purchase history to another site irks me. However, I like the fact that Amazon knows who I am and I get just as much joy browsing my years of purchases as I do from looking at my bookshelves. I like reading comments on books that I have read. Seeing the same commenters on different products that I own. Amazon is by far my favourite social network.

That said, Amazon should really do more to embrace this. I was trying to follow up on some comments last night and Amazon frames everything inside an 'improve your recommendations' context rather than as a social activity. I noticed that both LinkedIn and Facebook recommend friends to me; "Do you know X?" And more often than not they are right, and it amazes me. I am not amazed when Amazon recommends stuff to me as they are quite transparent on that front. It would be amazing if Amazon embraced the social context and let me tackle my valuable metadata socially.

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

	Album II - Kem $12.99 - Quantity: 1 - In Stock - Eligible for Amazon Prime shipping rates: join now Condition: new Sold by: Amazon.com  Gift options None
	The Album Formerly Known As... - Carl Craig $18.98 - Quantity: 1 - In Stock - Eligible for Amazon Prime shipping rates: join now Condition: new Sold by: Amazon.com  Gift options None
	Kemistry - Kem $12.99 - Quantity: 1 - In Stock - Eligible for Amazon Prime shipping rates: join now Condition: new Sold by: Amazon.com  Gift options None
	George Duke - Greatest Hits - George Duke $8.98 - Quantity: 1 - In Stock - Eligible for Amazon Prime shipping rates: join now Condition: new Sold by: Amazon.com  Gift options None
	Cendre - Fennesz Sakamoto $16.98 - Quantity: 1 - In Stock - Eligible for Amazon Prime shipping rates: join now Condition: new Sold by: Amazon.com  Gift options None
	Sessions - Carl Craig $17.98 - Quantity: 1 - Available for Pre-order - Eligible for Amazon Prime shipping rates: join now Condition: new Sold by: Amazon.com  Gift options None Pre-order Price Guarantee.
	Secret Rhythms - Burnt Friedman $15.57 - Quantity: 1 - In Stock Condition: used Sold by: doolicity
	The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation (Springer Texts in Statistics) - Christian P. Robert $53.70 - Quantity: 1 - Usually ships within 1-2 business days Sold by: newbook_networks
	The Anthology - Loleatta Holloway $19.13 - Quantity: 1 - Usually ships within 1-2 business days Sold by: caimanoutlet

Comments (0)

Here. Have some free music*. Thanks.

• A: There is a sickness [7:58]
• B: Dolphin Hotel [6:56]

---

Craque:"Steve Reich meets Thomas Brinkmann in an alley with John Adams"
Twerk:"Farben meets Zamfir meets Kenny G"

---

*NB: Not mastered by Audible Oddities. Milage may vary.

Comments (0)

Comments (0)

Comments (0)

Supposedly the _real_ reason behind the recent market volatility is some French schlub.

From the Times Colonist[1]

PARIS (Reuters) - A "massive fraud" by a junior rogue trader has punched a $7 billion hole in the finances of French bank Societe Generale , leaving its credibility in tatters and forcing it to get emergency cash. France's central bank and government scrambled to shore up confidence in the banking system after Societe Generale, France's second-biggest bank, said on Thursday it had been the victim of massive and "exceptional" fraud resulting in losses of 4.9 billion euros.

Supposedly this guy was earning less that $145k, working in risk management, taking offsetting positions to hedge against index exposure. SG was quoted as saying that it doesn't appear that he took on the rouge trades for personal gain. Go figure.

Update:
Let me pip the wiley New Zealander and point out that

Risk Magazine named the bank "Equity Derivatives House of the Year", while The Banker magazine named it as "Financial Institutions Group Asset-Liability Management House of the Year" in October.

---

[1] I know its from Reuters, but I just like the name 'Colonist'.
Comments (0)

Comments (0)

Comments (0)

Comments (0)

Comments (1)

Comments (0)

Comments (0)

Comments (0)

Comments (0)

Items not yet shipped:
Delivery estimate: January 11, 2008

1 of: Stock Market Liquidity: Implications for Market Microstructure and Asset Pricing (Wiley Finance) Sold by: Amazon.com, LLC 1 of: Empirical Market Microstructure: The Institutions, Economics, and Econometrics of Securities Trading Sold by: Amazon.com, LLC 1 of: Parimutuel Applications In Finance: New Markets for New Risks (Finance and Capital Markets) Sold by: Amazon.com, LLC

Comments (0)

Reading delicious/popular has become quite a habit for me. It is often a good place to find interesting and obscure programming tricks. The down side is that there is a lot of web-dev stuff in there that really doesn't interest me too much. Even so, the latest and greatest color picker is no where near as evil as this article by a Rock Star coder about storing tree structures in relational databases. Rather than going on a long rant , i'll try to keep this simple:

STORE DATA IN YOUR DATABASE IN A FORMAT THAT YOUR DATABASE UNDERSTANDS.

Done.

OK. Maybe I'll flesh that out a lil' more.

People much smarter than you or me write databases. And amongst those people lurk a hardcore group who write query optimizers. Their job is to understand your SQL and find the fastest execution plan given available indices and related statistics. And to do this they tend to rely on the fact that an INTEGER column is used to store INTEGERS and that all the various rules of mathematics apply. Now, unles your database supports the INTEGERS-SEPERATED-BY-PERIODS-REPRESENTING-AN-ORDERED-PATH data type, then you are SOL if you want any help from the query optimizer.

(Of course, if you are using PostgreSQL, then you will find an integer array datatype which supports ordered lists of integers. And the query optimizer knows how to use properly constructed indices to find stuff.)

According to Rock Star, the standard way of doing things fails because "It involves recursion, which is slow and complex." Hmm. Maybe people who find recursion complex shouldn't be re-inventing the wheel? And slow, maybe if you told us more about your application, we could help you work out why a perfectly normal solution doesn't work for you. Rocky then names two more problems with using MySQL's susggested Nested Set Model. OK. Fine, that model does suck. Even so, the only complaint he has listed against the standard 'Adjaceny List Model' is that it uses recursion.

I guess I knew his proposal would suck when one of his requirements was that "Selection of items needs to be possible with SQL.". Dude. You are using a database, so SQL _should_ work. You are just entering a world of pain by totally ignoring the role of the query optimizer, which will have no idea what to do with your janky path representation. And even if thats not a concernt, you have made it super difficult for any DBA to work with this schema as you'll end up with bastardized SQL to do anything apart form your original purpose. If you are 100% sure that your requirements will never change, fine. But what happens when one of your PHP scripts screws up and you have to manually correct the data? What about referential integrity? What do you do when your LIKE's are not of the form '1.2.3%' - e.g., find me all Speakers? Using your format, that turns into a SELECT * FROM junk WHERE parent LIKE '%.2.%'. Ugh. Throw out any misconceptions you may have about LIKE being fast.

In short, if you have exhausted the benefits that you get from using a database as a database you have 2 options:

1. Stop using a relational database if you don't want any of the benefits of a relational database.
2. Still use a relational database, but store your base level data properly. And then create materialized views (or similar) with optimized views for particular use cases.

In your case, I recommend #1. Stick it all in a text file. Or maybe a PDF that you print out and stick on a nice wooden table.
Comments (0)

Comments (0)

Known Joshua Reichs:

1. Me
2. My preferred alternate [Congratulations!]
3. New Zealand Josh
4. Arizonan Preacher
5. Christian Graphic Designer [Crazy Skillz!]
6. The Dead One [RIP]

Comments (0)

Release Date: December 19, 2007
For release at 10:00 a.m. EST

On December 17, 2007, the Federal Reserve conducted an auction of $20 billion in 28-day credit through its Term Auction Facility. Following are the results of the auction:

Stop-out rate: 4.65 percent [Josh: Only 10bp less than discount]

Total propositions submitted: $61.553 billion [Mo Money Please!]
Total propositions accepted: $20.000 billion
Bid/cover ratio: 3.08

Number of bidders: 93

Bids at the stop-out rate were prorated at 1.96% and resulting awards were rounded to the nearest $10,000 (except that all awards below $10,000 are rounded up to $10,000).

The awarded loans will settle on December 20, 2007, and will mature on January 17, 2008. The stop-out rate shown above will apply to all awarded loans.

Institutions that submitted winning bids will be contacted by their respective Reserve Banks by Noon EST on December 19, 2007. Participants have until 3:00 p.m. EST on December 19, 2007 to inform their local Reserve Bank of any error.

Let me go out on a limb and say that the markets will be down on this. A rate of 4.65% basically confirms that institutions haven’t been willing to go to the discount window because any such action becomes public knowledge. The TAF is a silent auction and thus avoids this stigma. Interesting stuff.

Comments (0)

The BBC at least went to some effort in trying to describe the mechanics of the current credit crunch, but they start off on the entirely wrong foot by conflating the ’sub-prime model’ with securitization. What they describe in the image above is securitization. Portfolio Magazine has a much better graphic describing the details of recent innovations in securitization.

Arguably, securitization lowered rates for all borrowers. It also re-distributed the fees from local lenders to larger investment banks who could earn commissions for writing and selling CDOs. But along the way information asymmetry increased and everyone had an incentive to keep filling the pipes with new whole loans. Mortgage brokers got a quick buck from selling a refinance to an originator. The originator made a quick buck flipping the loan to the secondary market. Banks and funds made a quick buck flipping MBSs into CDOs. And investors (usually via other banks and funds) made some money holding onto paper that was paying out a rate marginally higher than similarly rate debt.

The only problem with making so much money is that enough is never enough. And so more brokers needed to originate more loans to be repackaged into more CDOs. However, nearly everyone who could already afford a mortgage had already refinanced into a lower rate and so the only people left to be refinanced were those who couldn’t afford a mortgage. Brilliant! Mortgages for people who can’t afford them!

But they can. You see, a few years ago adjustable rate mortgages (ARMs) were invented to allow borrowers who were both financially savvy and creditworthy to structure their cash flows so that the bulk of their payments would occur a few years in the future. Perfect for up and coming hedge fund analysts who knew their bonus 2 years from now would be considerably bigger than their entire earnings to date. But as of 3 years ago, the same product became perfect Joe Trailerpark who never learned that nothing comes for free. And that a 3% rate this year means a 10% rate 2 years from now.

Now, I don’t fully blame Mssrs Trailerpark, I’m quite sure a large number of brokers glossed over the latter detail. (Did I ever tell you about the time I was sitting at John Wayne Airport, in the heart of mortgage origination land, and overheard 2 brokers talking about the finer points of faking W2s ? Maybe another time…) Either way, everyone was happy as Trailerpark & Co. provided the much needed raw materials for new CDOs.

Of course, this all goes to shit when people stop paying their mortgages. Statistical models suck when faced with a small number of observations and flawed assumptions. And humans are all too happy to ignore the fundamental problems with their risk models when they are just making so much gosh darned money.

And that’s how it all went wrong.

(PS: Did you get your bid in today? I may post some thoughts on that in 2 days when the results are released).

Comments (0)

I, for one, whole heartedly support the writers in their strike. In fact, I hope they remain on strike. Forever. The internet has never before had such a great range of funny videos to watch. However after a while it gets a little boring to continually watch videos about the strike. So, writers, we get your point but please move on to some other topics. You have so much talent not to waste it all as one trick ponies. Or LoLcats.

Comments (0)

Books I have read in the past few weeks:

1. Twinkie Deconstructed. Really boring. Last time I trust Tyler Cowen for a book recommendation.
2. The Electric Kool Aid Acid Test - Great. The closest thing fiction that I have read in a while, without actually being fiction. It’s the story of Ken Kesey and his merry pranksters who suffered heavily from the burden of being some of the earliest LSD addicts. And as such weren’t enlightened by the now commonplace knowledge that _everything_ you do on acid seems really interesting and important.
3. Moneyball From the author of Liars Poker comes the story of Billy Beane and how he used refined statistical analysis to lead the Oakland A’s to a world series victory. Or at least thats what I thought until I got to the end chapter. Turns out that they don’t win, but they get close. Either way, its a good read. I walked away even more convinced of the fact that most established knowledge is right, but every now and then its very wrong.
4. The Informant. My dad left this book with me when he was over in New York last week. On the 600-odd pages of this book we learn the story of international price fixing in the lysine market and the development of a huge white collar crime case by the FBI and DoJ, centering on the Archer Daniels Midland company. I had dinner on the weekend with some FBI agents [[Why is it that a statistically significant portion of my friends work in law enforcement in America, yet most of my Australian friends are criminals?]] and they had never heard of the case. Supposedly they are making a film of this book, which doesn’t surprise me as it reads like a script. Should be a great film, very exciting story.

Book(s) I am currently reading:

1. Judgement under Uncertainty. After my disappointment with Behavioural Finance, by James Montier, I decided to go straight to the horses mouth and finally read the works of Kahnemann and Tversky. So far the book is confirming my prior belief that behavioral finance/economics is simply plain old economics with more complex utility functions than the analytically pleasing forms that are often assumed to describe rationality.

Books I probably won’t ever read:

1. The Tipping Point, by Malcolm Gladwell
2. Anything by Richard Dawkins

Machines I am currently building:

1. Soup! - I’m not quite sure what it does yet. But its fun to be building machines again. I really suck at drilling holes in a straight line. I want a drill press for Christmas. And a lab in which to use it.

Comments (0)

Jerry just wrote an interesting post about the impact of behavioral targeting on media prices. Half of our ads work, we just don’t know which half, so the adage goes. Jerry asks:

If advertisers could put their ad only in front of the fifty people who are likely to buy the product rather than the hundred that might or might not, those other fifty ad slots go begging. Half the advertising inventory will not be bought. This 50% drop in demand should drastically cut prices.

On the other hand, if an advertiser can target better, she should be willing to spend more to get in front of that targeted audience. But how much more? Twice as much? Does the advertiser end up paying less, resulting in lower overall media revenues? Or does media benefit by being able to charge premium prices for all of their inventory?

Ceteris paribus, net revenues to the publisher should remain the same. They are selling half the units at twice the cost. And the advertiser ends up paying the same amount for the same number of customers. Of course, if this were the case, behavioral targeting companies would not exist - the industry would be indifferent as to whether to target or not. Behavioral targeting companies market their technology with the premise that there is no need to pay $50 CPM’s for premium placements when the same eyeballs also see remnant inventory at $0.50 CPM. The targeting platform makes sure that the right eyeballs see the right ad at the right time, regardless of placement. That said, as Jerry recognizes, if everyone switches to behavioral targeting, then it all works out in the wash, and pricing per placement becomes a thing of the past. But net, the cash flows remain the same.

“Nobody counts the number of ads you run;they just remember the impression you make.”
–
William Bernbach

Where things get interesting is if we allow for the possibility that premium placements really deserve higher CPM’s on the basis of placement alone. In which case, we can no longer lump together the n units of premium inventory with the N units of remnant/run of site, into a fungible (n+N) size pool. If there is anything fundamentally different about premium inventory then ceteris stops looking paribus. There are plenty of smart cookies, with more PhD’s than you can poke a University of Phoenix marketing diploma at, who continue to pay a premium for front page ad runs. I reckon they might be onto something, behavioral targeting be damned.

Comments (0)

I’ve been following the 20×200 blog / store since inception. I love the idea and and many of the works. A few weeks ago Shianling and I decided to buy a print (above). Unfortunately I’m not terribly happy with the print quality. I don’t know much about printing, but am familiar with the problems of viewing art online, where the gamut of a monitor is greatly limited compared with what can be printed. However, the print we received lacks much of the vibrancy of the online picture. Oh welp. We supported an artist and have a shiny new thing to hang on our rapidly crowding walls.

Complaints about the print quality aside, I am very tempted to buy a new work that became available today (below) as I seem to have a deep emotional connection to custom bound books.

Comments (0)

One of the best ways to improve your Page Rank is to slavishly follow other blogs and provide witty responses to the posts of more credible bloggers. In that spirit, let me one-up Felix Salmon and raise his correlation analysis with a p-value. It turns out that Felix holds annual Pinot parties. Sorry, not parties, but scientific evaluations of Pinot Noir. Eschewing issues arising from normality, rank scoring, sample size, and underrepresentation of Australian wines, I present the following confirmation of Felix’s guesstimate that Pinot rankings and price are uncorrelated. Rather than rely on the unwieldy crutch of an R-square, I prefer to look at the p-value on the regression of score against price:

 > score=c(115,196,137,146,175,212,193,184,180,167,154,143); > price=c(18,23,29,52,33,12,26,18,10,50,13,12); > l<-lm(score ~ price); > summary(l) Call: lm(formula = score ~ price) Residuals: Min 1Q Median 3Q Max -53.982 -19.423 8.385 17.913 41.085 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 174.7818 17.4850 9.996 1.60e-06 *** price -0.3222 0.6200 -0.520 0.615 — Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 29.36 on 10 degrees of freedom Multiple R-Squared: 0.0263, Adjusted R-squared: -0.07107 F-statistic: 0.2701 on 1 and 10 DF, p-value: 0.6146 

The above analysis, done in my BFF language - R, demonstrates that price is not a significant predictor of wine score. With a p-value of 0.615, the null hypothesis is looking a lot like a Pinot Party. A place where we can all be comfortable.

For bonus points, I leave it to my good readers to perform a contingency table analysis.

Comments (0)

One of the best things about being crowned a romance buff by the prestigious New York Post is that I now can get away with pretty much anything at home.

“Honey, can you do the dishes?”
“But I’m busy knitting you a scarf”
“Yes, but doing the dishes would be romantic. I declare it so.”

I now have a Frank Bruni-esque[1] power to declare anything romantic and it must be so. Anything else would be denying the Post’s role as an arbiter of New York cultural icons. Of which I am one.

—
[1]: Or whomever the NY Post’s equivalent may be.

Update: According to one of Shianling’s patients, we also made it to The Gothamist. Now I’m truly a cultural icon.

Comments (0)

Postgres:

 think=# select 'hello' = 'hello'; ?column? ---------- t (1 row) think=# select 'hello' = 'goodbye'; ?column? ---------- f (1 row) think=# select 'hello' = 'HELLO'; ?column? ---------- f (1 row) 

MySQL:

 mysql> select 'hello' = 'hello'; +-------------------+ | 'hello' = 'hello' | +-------------------+ | 1 | +-------------------+ 1 row in set (0.00 sec) mysql> select 'hello' = 'goodbye'; +---------------------+ | 'hello' = 'goodbye' | +---------------------+ | 0 | +---------------------+ 1 row in set (0.00 sec) mysql> select 'hello' = 'HELLO'; +-------------------+ | 'hello' = 'HELLO' | +-------------------+ | 1 | +-------------------+ 1 row in set (0.00 sec) 

This is not kindergarten people, this is UNIX. Do what I say, not what you think I mean.

Comments (0)

Since appearing on the popular section of del.icio.us, a number of people have asked me to explain how it is, in fact, possible to replicate what we see in the following video:

Whats going on here is really quite simple. All you need to do is hold the video camera with one hand and wrap the foil etc. using your free hand. It definitely helps if you have a small, hand-held video camera. If you have one of these it might be good to have a friend around while attempting the above.

Comments (0)

Mike called my bluff. The book I refer to in this post was actually from 1988, not the 1960’s. The book is Dynamic Graphics for Statistics and can be had for a princely sum of $149 from Amazon. I’m certain that I paid approximately 1/100th of that for it.

In a semi-unrelated meme, I was chatting with my friend Aaron a few weeks back about why I disliked 3D graphics, especially in games. To his credit, I don’t play computer games so I miss out any much of the emotional nuance that comes from interacting with simulated gore and I probably shroud my argument with rationales purely to hide my irrational aesthetic bias. But that said, I don’t like 3D graphics for two interrelated reasons. Firstly, I assume that the designers of these graphics curse themselves at the lack of rendering power at their fingertips. Sure, they think their blood spatter looks better than any game that came before it, but if only they had 64 GPU pipelines instead of 32, it would look so much better. George Suerat never complained about the number of lines per inch in his Gillot paper. The paper was chosen for its capabilities and its characteristics were exploited to promote the aesthetic that Seurat was seeking. I have long felt that art and constraints go hand in hand and successful creators express themselves within boundaries. For the most part, game graphics are not timeless. Today people look back on games from 2003 and chuckle at the lack of realism. In 2009 the creators of Halo will secretly wish that they could go back and remake their games with 2009 era technology. Who knows if the makers of Galaga would prefer to recreate their game using today’s technology, my guess is not.

Fundamentally, I find it a little offensive when people make graphics and aren’t happy with the outcome. Sure, game creators (and players) probably really enjoy their work. But whenever they admit that they need a better computer to ‘get the full experience’, they are stating that they are not happy with what they have. Computers are immensely powerful. But I fear that much of that power is being dedicated to doing things very uncomputer-like. If people adapted to and adopted the medium, we may have graphics that look very different to reality but in the abstraction away from reality I think we can find a fluid, informative, and medium appropriate representations. People don’t have an innate understanding of what data looks like, and in the world of statistical analysis and visualization we can afford to break the mold. Perhaps not in many games, but I have seen some games that take this approach and invent new visual realities that are (to my untrained eyes) as emotionally engaging and poorly rendered trees and bump-mapped blood spatter.

I won’t dwell too much on my second concern with contemporary computer graphics as it is well covered by Tufte. Namely, the core concern is that too many pixels and graphic elements are used for non-informative purposes. Shading a chart, or purposelessly extruding a bar chart in 3D adds no new information to a graphic. This is why on any new install of Excel I spend a good hour or so setting up new chart defaults to get rid of all the junk it tends to decorate my charts with. (Yes, at times, I use Excel.) Going beyond the world of images, the same principle can be extended in interface design. Just as one needs fewer distractions from the information, interfaces need fewer mouse clicks and menu selections to play with the data. In this age of computer power, people often neglect the value of exploratory data analysis. More, now than ever, is EDA important. I have terabytes of data at my fingertips, and most visualization tools just don’t keep up. I want to be able to slice, dice, plot and histogram data at a breeze. Data retrieval isn’t the bottleneck, but rather poor interface design.

What this world needs is a nice, simple, stand alone, does-one-thing-and-does-it-well exploratory data analysis tool. Maybe I should make one.

Comments (0)

A movie about exactly that. A documentary about the world we live in.
Thee spirit ov thee times.

I watched this movie recently, it is a melding of many different themes that are current in todays society and prevelant in everyones mind. It centres around Money , War, Terrorism and how we are all fooled by the mass media. On the surface this kind of description may seem to be rather conspiracy thoerist and alarmist , but infact i found this movie to be a rather sobering roller coaster journey back to reality. It puts together alot of factual information and presents it in a way which just makes you think “this is whats going on” and it indeed is.

Although the movie is focussed largely on America and its historical impercial capitalist value system, it is equally relevant for Australia , since as the famous old addage goes “When America sneeze’s the rest of the world gets a cold” , seems particularly true in light of the information presented. Particularly regarding some of the more notable “Black flag” operations , such as 911 , 7/7, and bali bombings ,the date of which , sadly escapes my current frame of mental reference; and the strikingly similar circumstances in which those so called “terrorist” attacks occured. I have always had my doubts on the validity of the information the media presented on these generic operations. This movie has served only to assist me in strengthening my resolve to think for myself and question authority. I must be getting old.

sense

Comments (0)

This weekend was my girlfriend’s birthday and my parents happened to be passing through New York. I figured the aligning of the stars was a sign that it was about time for me to propose. Being unabashed nerds, our first date involved spending a decent chunk of the day browsing through the math book section at Strand. In a very xkcd spirit, we hunted down erotically suggestive math titles. Our favourite was “Tight and Taught Sub-manifolds.” Topology always seemed a little kinky.

Being particularly unimaginative I decided to propose at the site of our first date by constructing some books that spelled out the proposal along their spines. The ruse was completed by convincing her that I was unable to get her present delivered to the house on time, and thus we would have to drop by the store to pick it up. Rather than setting the books up in the humid and always busy basement of Strand, I arranged to set things up in the Rare Book collection on the third floor, away from most traffic. The store was very keen to help out as the New York Post had recently been looking for love stories to tie into an article celebrating the stores 80th birthday.

Given that my now-fiance’s interests are more aligned around the natural world rather than mathematics, I planned to set the books up next to the rare natural history collection. Its a lovely nook in the store and seemed like the perfect place. Having nonchalantly covered my tracks for the previous month or so, it all broke down when we entered the book store. My plan was to casually peruse books and gently lead her to the natural history collection. Instead, it was more a case of yanking her towards the books and me childishly pointing to my proposal books and saying “Look!” She was stunned and I went on one knee to ask the big question. Despite being a relatively capable public speaker, I had considered the possibility that my mouth might stop working at that moment, so my goal was to get out at least four words; “Love”, “Light”, “Sweetness” and “Joy”. I got as far as “Love…. Marry me”. She said yes.

After some crying and hugging I realized that the books were set up, not in the natural history section, but next to it - in the medical section. In the photos, the book closest to my proposal is a title on gynecology. None of that mattered, of course.

It took a few moments to compose ourselves and then we wandered through Union Square to the W hotel where I had booked a suite. Thanks to some friends who work for Starwood we got comped up to their largest suite. Champagne, Strawberries, Bubble Bath. Nap.

Then we walked down to Casimir in the East Village for a surprise dinner with 25 close friends and family. A group of us returned to the suite for some more drinks, dancing and fried food. By midnight we were exhausted. It was a fantastic day. My only regret was not seeing my friends from Australia and around the world. Hopefully I can rectify this shortly.

Comments (0)

Read it for yourself.

Q: How does one calculate a p value?

Update Some kind soul has done the right thing and calculated a princely p value of >> 0.05. In other words, lottery draws are random. Of course, it could be a huge cover up as if lotteries were not random I can think of a whole bunch of statistics books that would have to be re-written with new examples.

And now for some R code:


#Get big.dat at http://www.state.nj.us/lottery/data/big.dat
big=read.table("big.dat",sep="%",fill=T)
big$date=as.Date(apply(big[,1:3],1,paste,collapse="-"))
big$type=ifelse(big$date>"1999-1-13",ifelse(big$date>"2002-3-15",ifelse(big$date>"2005-06-22",4,3),2),1)
big$maxnorm=c(50,50,52,56)[big$type]
big$maxspecial=c(25,35,52,46)[big$type]
maxnorms=table(big$maxnorm)
p=rep(0,56)
for(i in 1:nrow(maxnorms))
p[1:as.numeric(names(maxnorms)[i])]=p[1:as.numeric(names(maxnorms)[i])]+maxnorms[i]*5
maxspecial=table(big$maxspecial)
for(i in 1:nrow(maxspecial))
p[1:as.numeric(names(maxspecial)[i])]=p[1:as.numeric(names(maxspecial)[i])]+maxspecial[i]
p=prop.table(p)
allnum=unlist(big[,5:10])
t=table(allnum)
chisq.test(t,p=p)
plot(t/p)


As a quick hint: read.csv can accept a URL in addition to local files.

Comments (0)

TED quotes a very memorable statement from Derman’s “My Life as a Quant”:

Quantitative finance “superficially resembles physics,” he says, “but the efficacy is very different. In physics, you can do things to 10 significant figures and get the right answer. In finance, you’re lucky if you can tell up from down.”

Derman likes to bring up the analogy often:

Now I think that maybe “financial engineering” suggests too much precision, like using too many decimal places in specifying your height as 6′ 1.2345″. Maybe FE is a politically correct word for something more fuzzy.

Computer scientists should be called computer engineers. Nutritional scientists should be called nutritionists, without the science. Maybe people shouldn’t be allowed to self-describe and name their occupation.

What is the most accurate job name to describe what financial engineers do?

As someone who never made it beyond the national level in the Physics Olympiad due to my poor mathematics, I can’t wax lyrical about heat diffusion equations and their place in finance. However, I can guesstimate that a decent financial engineer spends 90% of their time trying to profit from other people’s mathematical approximations of human behavior. The best financial engineers spend 90% of their time convincing others that their models better correspond to reality.

TED’s post reminds me of an anecdote about the early mortgage market. At that time there were no models for prepayment or default risk. A prominent player in the market (with no formal mathematical education), hired a PhD student to build a rudimentary model and then applied the model to find a basket of underpriced loans. After amassing the portfolio he circulated the model to the street, creating demand for his loans. Now, thats arbitrage.

Comments (0)

I just happened to come across this article while working on some code for analyzing financial data:

In short, the picture of Wall Street designers that comes across is revealing. The designers in are smart, able, savvy. But they make up an distinct community of practice, one with lower status, limited financial knowledge and one that does not seem to fully communicate with the traders and bankers. In terms of innovation, they also seem to be paralyzed by the needs of their users. As as we know from Christensen’s “The Innovator’s Dilema,” users are a conservative group.

I’m quite busy right now, but let me add some random bullet points to the discussion:

• My current codebase has ~300 lines of R code dedicated to analysis
• A further 150 lines are for visualization
• Thats a 2:1 ratio for the mathematically inclined
• My charts have black backgrounds, no extraneous borders or labels.
• Every single pixel is important (See Tufte)
• No offense people, but 99% of the designed I have worked with in the past have none of the technical caliber of John Maeda
• My favourite book on statistical visualization was written in the 1960’s and I have yet to come across a visualization package as powerful as the one they describe.

Comments (0)

In addition to poor underwriting standards, Ranieri says the mortgage market has brought in too many “middlemen with no axe to grind other than to collect money.” By this, he means that the traditional relationship between a local thrift or savings-and-loan and a depositor, who might have borrowed from the thrift, had changed.

From an article in Investment Dealers’ Digest about my former gig.

Comments (0)

I really dislike youtube’s compression and random dropping of frames.

For full slashion [25MB]

–
footer

Comments (0)

Yes, I know its been a while, but after a short vacation from blogging I want to kick things off with a rather detailed reply from one of the successful stock or not players. JP writes the following:

Someone else said: “If distinguishing stock market data from random data is a skill that takes study and or talent, no amount of testing the untalented or unstudied would give you accurate results in determining if stock market data is distinguishable from random data.”

I don’t think so, assuming your method is right novices will learn the game over time. Furthermore, if you give people with absolutely no stock experience a minute of coaching, you can dramatically improve their score. My girlfriend could care less about the stock market, but after I told her the trick about high volatility at high prices and low volatility at low price she scored a decent 68%, or 34/50.

If you want to win at Stock Or Not, download this great document which describes all my failings at generating fake stock prices. Well done JP, and sorry that it took me a while to getting around to publishing this.

Comments (0)

Comments (0)

me on twitter as jdigittl

Comments (0)

A number of peeps have now played with my phone - including an ex-Apple, reverse engineering expert. He was pretty useless, as he had far too many beers in him and required my girlfriend to do the typing for him. Everyone seems pretty impressed. My girlfriend noticed a few UI bugs. Nothing major.

I want to lay down a $599 bet that, at some point in the phuture, we will have a low cost iPhone Shuffle. One button - call a random friend. It’ll sell like hotcakes!

Comments (0)

This is coming from my phone. I couldnt work out how to edit my old post . Text area has no scrollbar so i cant append as the keyboard has no arrows . Predictive text doesnt work on webpages as far as i can tell. I am currently at dinner being very rude. As i walked here we saw lpts of iphones.

I hope this gets less frustrating.

Comments (0)

Not quite sure how it happened but I now have 2 iPhones. A coworker (lets call him Luke) had paid someone to wait in line all day, but then that guy sold his spot - leaving my colleague down some cash, an umbrella and one lunch. Oh well, this is how free markets work, right? I went along with Luke to observe the feeding frenzy and somehow got sucked in myself. We ended up paying a guy $50 to take his spot (which was considerably less than what Luke had paid his first schlub.) At about 5 to 6, the line started moving. It took about 20 minutes to go through around 200 people. As we approach the cube of the 5th St. store, Apple hipster staff were waiting outside applauding people as they came in. Inside the store it was a swift process to part with $1.2k and we were out withing minutes of being inside.

After activating my first phone (which required a download of the latest iTunes), I was up and running in no time. However I have _zero_ reception in my apartment. This wasn’t completely un-expected as I had no reception on my old T-Mobile GSM phone. I tried getting on to my WiFi connection, but in my buzzed state I seem incapable of remembering my password. Surprisingly, unlike on my BlackBerry 8800, it was very easy to enter in my password. As Mossberg says, the keyboard is a complete non-issue. I can type just as fast on the iPhone, using 2 thumbs, as on the berry. As for the WiFi reception, I live in a small apartment, and I am 6ft away from my airport and I only have 2 out 4 bars.

The screen is great. I took my keys to it and tried to scratch it in vain. My original plan was to take it apart and work out what was inside, but I decided against that course and will just wait for someone else’s dissection later tonight.

I had to take my phone outside to get any reception and EDGE is god awful. It took me over 1 minute just to load the eBay homepage. Same for Slashdot. YouTube - YouGottaBeKidding. I tried making some calls and without the headset, using speaker mode, I found it very difficult to hear the other end of the line. I will try again shortly.

I took some shots with the camera, both inside and out, and it’s lackluster. It looks like every other crappy phone camera. Yes, the UI is beautiful, I can pinch and flick to see how noisy the image is with ease.

Despite my mixed review another co-worker pounced on the chance to buy my second phone. That’s what the hype is like. Luke and I went to grab a coffee after we bought the phone to reflect on our participation in frenzied consumerism and I have never seen wait staff behave like they did. It was a fairly high end coffee joint (no Starbucks) and every waiter came to see what it was. Then the maitre de, and managers. No qualms in taking the phones out of our hands to touch and feel it. Mind you, at that point they were yet to be activated and so there wasn’t much playing to be had with them.

General score: meh. UI is everything and more I could want from a phone. Technically, its not that great. Keyboard - great. Phone - meh. Camera - pfft. Internet (EDGE) - argh.

But then again, it just could be buyers remorse. I wasn’t planning on getting an iPhone, and I could be actively looking for flaws. I’ll use the phone more over the next few hours and provide updates if my feelings change.

Comments (0)

Ok. I’m totally sucked into the iPhone. And today I officially cross over into the world of fanboyism (or is that fanboyizm?) by posting this screenshot from Apple’s recently released activation video. Who knew a video about setting up a mobile phone account could be so exciting?

Far be it for me to cast the first stone, but could it possibly be that a first version of an Apple device/software contains bugs!?

Comments (0)

Wired	Tired
Pulp Fiction	Pulp Fiction on Bravo. Translated from English to Bravlish. And about 2 hours shorter than the original

Comments (0)

One of the upsides of not smoking is that rather than using a cigarette to snap into crystal lucidity as soon as I wake, I often find myself mumbling around the house while I get ready in the morning.

“The ironical thing about Funky Town is that, as a song, its not so funky.”

“I don’t think ‘ironical’ is a word. Irregardless of that, you are misusing the concept of irony.”

“Yeah. Whenever I say irony, I can’t help thinking of all the times I’ve heard other people comment about its misuse.”

“I’ll tell you what irony is: people misusing the word ‘irony’.”

“Maybe thats the joke.”

“Maybe.”

“Hm.”

“What?”

“Did you hear that Queen Margret died?”

One of the great things about having a slightly stocky Arizona preacher as my digital doppelganger (who is more highly ranked on Google than I) is that when people meet me in real life, they are somewhat pleasantly surprised. [1]

---

[1] My fact checkers inform me, that as of writing time I have now clawed my way above the ‘other’ Joshua Reich in Google rankings. But Joshua Reich, the Jewish swarm intelligence researcher from Columbia University still outranks me, but that is the way it should be.

PostScript: Other posts have been similarly fueled by lack o’ nicotine. See if you can work out which!

Comments (1)

I’m IMing an entrepreneur and something he says reminds me of a recent post by Fred Wilson. I want to paste a link in the conversation, so I go to my always open Google Reader window and find that Google doesn’t have a _search_ feature. Google no search. WTF? So I go to Fred’s site, which happens to be notoriously slow given the large number of widgets he uses. Knowing this, I figured he’d have a search widget. And he does - provided by Lijit (founded by Wandering Stan - one of my original colleagues from Root). Unfortunately, it doesn’t find what I’m looking for. Off to Technorati I head, and I have to wade through their interface to find the post.

10 minutes wasted.

Comments (0)

At Root we (I) saw a huge need for introducing structured futures contracts to the lead market. With the then impending meltdown of the mortgage market, expectations for future price volatility were much higher than historical volatility. Mortgage brokers operate primarily fixed cost businesses and in my conversations with our buyers they fundamentally understood the purpose of futures contracts and were eager to sign up for them. Brokers tend to manage their business on a cost per funded loan basis, and leave revenue management to their portfolio people. Likewise, most lead generators of note have a significant portion of their traffic driven by negotiated fixed-price advertising deals. Before the advent of lead ‘exchanges’, most advertisers sold their leads under negotiated deals. The exchanges introduced spot pricing. I’ll cut this long story short, as I have written about this before.

While we did launch something along the lines of what I was thinking, we never ended up going live with a fully fleshed out product. This was a real shame as the opportunities for this market extended beyond natural hedging to people with a speculative interest in mortgage market dynamics. Before Greg’s ignominious departure, we spent quite some time grappling with what would happen if the derivative market grew larger than that of the underlying. The unique nature of the lead market made this particularly concerning as, if you squint, a gold-chain wearing hairy-chested mortgage goon from Long Island looks mighty similar to devious Goldman trader looking to exploit loopholes in whichever market they are in. On the Root Exchange, participants were always trying the weirdest tricks. Couple that with fungibility issues and the inherent inability to store leads for any significant period of time and the gaming opportunities could be immense if certain mechanisms weren’t properly designed.

The blogger that I-most-wish-I-had-a-crush-on sums these concerns up well with her thesis regarding the eventual unwinding of the sub-prime market. I’ll say no more save pointing you in her direction.

Comments (0)

Starting a new job always means filling out lots of forms and letting loose with personal information. I can only assume that some of this information has leaked resulting in me, in addition to my usual junk mail, receiving ‘personalized’ offers from Amnesty International and the ACLU. Which database marketer worked out that new employees at hedge funds make excellent prospects for both of these organizations?

What if marketing dollars were efficiently allocated…

Comments (0)

I just walked outside to get a bagel. Along the way two police officers were standing next to a homeless guy who had passed out on the sidewalk. Walking towards me is a gentleman who walks past the scene, stops, and turns around. My quick conclusion: perhaps the man is a doctor who is going to help the poor guy. Actuality: The man pulls out a video camera to take home a slice of New York for his friends back home.

Comments (1)

Is it just me, or since retaining a new editor, has the last letter to the editor become less quirky?

It used to be such a joy, reading the letters whilst keeping the last one covered with my palm. Savoring the subtle chuckle for the end of a series of otherwise dry ripostes, more often than not from the Undersecretary of Urban Planning at the Peruvian Institute for the Future.

But now its just no fun anymore. Whats up with that?

Comments (0)

Having recently changed jobs, it is not surprising that quite a few people are asking me what I now do. It took me a good two years to get my answer to that question down pat for my previous position, so I am not sure why I expect to be able to answer this after only a few weeks at the new firm. Perhaps it is because my current employer, who (purely for the sake of maintaining some level of intrigue) will remain nameless, has been in existence for a good 40 years or so. My role at the firm is to take over management of an internal skunkworks group that traditional provided collateral equity research to supplement the work of our portfolio managers.

Let me back up a bit.

The firm is an investment manager, for the most part in equities, both locally and abroad. Our investment thesis takes a purely growth oriented, fundamental analysis approach. And to a fly on the wall, the conversations around the office would be indistinct from the stream of pitches heard at a VC, save for the fact that we deal in publicly listed companies. I like this. Despite the fact that I am very much a numbers guy, the most enjoyable and challenging aspects of my roles in the past have been understanding the stories behind the companies where I have worked. In the future I would not be upset at all if I were involved in venture capital as I think I have a good sense of story. Unfortunately, as a numbers guy, it is difficult to distill the elements of a good story into a spreadsheet. And thus the quandary with respect to describing my new role as Director of Technical Research.

Overarchingly the position entails bringing quantitative techniques to fundamental analysis; numbers to stories. Mind you, the portfolio managers and research analysts have no fear of spreadsheet modeling, so the question I have to answer is where, above and beyond valuation 101, can new techniques be applied. Day to day this involves managing a small team of technologists who have traditional flexed an artful knack for discovering unique and proprietary data sources, somewhat a la Majestic Research. Recently we have geared down such activities for a number of reasons. But being developers, rather than waste cycles spinning, attention was turned to improving internal work-flow and a number of tools were developed to aide in performance monitoring and crunching through SEC filings. As useful as these tools are the firm already has a dedicated development team, and dammit, tools are just not glamorous. And who works in finance, if not for the glamor.

The other major difficulty in defining my role is the pesky problem of efficient markets. For the past 6 years I have worked in companies that rallied behind the cry of efficient markets. At Root Markets we sought to stamp out inefficiencies in the lead generation business. Previously, at Traffion (which was eventually subsumed into Nielsen//NetRatings, where I hear my code is still alive and kickin’) our goal was to design an exchange for efficient trading of online advertising inventory (much like Google & Right Media do today.) So whenever I think about trying to out-smart the system I find myself lamenting that all of my ideas must, by definition, be for naught due to the power of markets efficiently disseminating information. Yet whenever I do, I simply have to remind myself that efficient markets don’t pay for vanity tail numbers on private jets.

My first week on the job was fairly typical; remembering and forgetting names, working out how much to tip the shoe shine guy and where to get lunch. In week two, I was full of big ideas as to how to make a difference. Week three I worked out why none of those ideas had worked in the past. Now I am getting to terms with the fact that while my directive is easy to comprehend (”help our clients make more money”) the direction I need to take is more fuzzy. And the pace is certainly very different to working at a startup - I have much more time to think about things and I hope that with time, the ideas will come.

Comments (0)

Yes. I’m still alive - having recently started my new job I haven’t had much time to post. Past performance is no indication of future performance, etc.

I came along a quote in a comment on Calculated Risk today, and I think its super:

the plural of anecedote is not data

Comments (0)

I touched down last night and with carefully timed sleep on the plane I am back in New York time.

Happy to be back.

Happy to be long YHOO (up 11% today on the rekindling of MSFT’s interest).

And happy to discover that just as Dark Side of the Moon syncs perfectly with Alice in Wonderland, Jean Michel Jarre’s Zoolook album syncs perfectly with a random walk around the West Village.

Comments (0)

After much watching from the sidelines, I signed up for twitter today. But I have no friends on twitter, so excuse me whilst I twitter here. As I shuffle towards the end of my Melbourne tour, I had a pretty interesting day today. In twitter style I’ll keep it short’n’sweet.

1. Coffee with private investor
2. Sandwich with advertising executive
3. Phone with fund startup guy
4. Dinner with TV post-production whiz
5. #4 coins the phrase “invespionage”

Two more days back home and then I return to the U.S. of A.

Comments (0)

I’ve been in Australia for 24 hours now and the biggest news here is the drought engulfing much of South-East Australia. This region, which includes my home town, not only supports the bulk of the Australian population but also provides for the majority of fresh food for local consumption and export. Both of these are supported by the fact that the rest of Australia, for the most part, is arid.

Just prior to one of my first meetings with the then to be chairman of Root Markets (my former employer), Lew Ranieri had returned from a series of meetings in Washington with Julian Robertson (ex Tiger). While these gentlemen are probably not the first names that come to mind when thinking of environmentalists, their agenda in Washington was to rally behind environmental initiatives, arguably to support their interest in various trading opportunities. Lew relayed to me Julian’s belief that Australia was selling itself short with its export food industry as his opinion was that rather than selling food to the rest of the world, Australia was really just selling water - one commodity that Australia does not have in ample supply.

From an article from today’s The Age newspaper:

“Do you like steak?” one farmer asks The Age. “City people are all worried about whether to press the half or full-flush button on their toilet, or how to save 20 litres of water in the shower. It’s pointless. It takes 55,000 litres of water to make a kilo of beef. That’s where your water goes, to make food.”

One mechanism currently in place to prioritize the allocation of water to farmers in the region is a system of water trading:

A water right is an entitlement to irrigate using a certain amount from the river system. Some bought their rights when purchasing their farms, others inherited them. […]
Water traded permanently has fetched more than $2000 a megalitre, while temporary water has sold for as much as $950 a megalitre.

For example,

Mr Lee says that if he sold his yearly allocation of water he would make $160,000 — and without growing a single grape. It would be more than double what he made last year, slogging his guts out.

As a freemarketeer I find the essence of this plan strongly appealing. However the majority of farmers hold a different opinion. It appears that the concept that their livelihood has been supported through an effective subsidy on our most precious resource is entirely foreign. If Mr Lee is unable to produce grapes with sufficient profit to pay for the water he uses, then we must learn to live without Mr Lee’s grapes. For a long time Japan has been criticized for subsidizing local rice production, given the vast tracts of land required to grow the grain. I suppose Japan supports local rice growers given their institutionalized nationalism and would never deign to admit that rice grown from the expansive paddies of China is of the same quality. Likewise, Australian farmers feel deep shame now that their inefficiency has be efficiently priced. I agree that the effects on a region can be immense when the bulk of farmers can no longer afford water, but we must understand that the environment, and the economy at large, will benefit from efficient allocation of a scarce resource.

“They’re arrogant, they’re selfish, it’s totally immoral,” shouts one farmer, John, at this week’s meeting. His anger is met with thunderous applause.

I, too, applaud this statement. Two out of three is not bad. Yes, I am arrogant. And yes, we are selfish. But free market pricing of water is not immoral as the selfish actions of individuals in a free market results in fair prices and benefit the greater good.

Update: The numbers in the article don’t seem to make sense. If it takes 55k liters to produce 1kg of beef, and annual water rights sell for ‘as much as’ $950, then 1kg of beef would use $52 of water. According to my mum (Hi Mum!), 1kg of beef costs around $25. Either the water markets are (dare I say) illiquid, or the quoted numbers are wrong. I will do some more research into this.

Comments (0)

Someone find me a Ruby on Rails developer, stat! I need a Google maps mashup that empowers my wifi-enabled, gps-aware device with an AJAX interface (REST middle tier) showing me where all the power outlets are in departure lounges.

Oh, and while we are at it can some biz-dev type craft a ‘peanut butter’ memo to the slew of operators who provide wifi access at such lounges. For some reason they think its OK to make you go through ten steps to sign up for 15 minutes of wifi access. Why do I need a password and username?! Every time I go to an airport it, without fail, takes 4+ attempts to guess a new username that I haven’t used at some airport at some time. jdigittl - NO, jdigittlcrap - NO, jdigittalJFK - NO, jdigittlareyoufuckingkiddingme - NO (too long), robochunks - YES!!! Oh, and for some reason the site helpfully remembers all of my details apart from the fact that I unchecked the “Send me lots of spam” checkbox. Fuckers.

I know that I could sign up for the $24.99 / month service to save me the hassle of having to deal with this. But the fact is that even when I am traveling more than 4x per month, each airport has a different provider. Why can’t we just have government sanctioned monopoly?

Ooh! I spot a free power outlet in the janky bar. Time to order a scotch, plug in and tune out.

Only 27 hours until I’m in Australia.

Update: Man, I’ve had faster internet access over an acoustic coupler.

Comments (0)

In my unemployed state, I am using my time to catch up on news. CNN International is my source of choice. I have also been catching up on reading too. When not watching TV I am currently reading John Waters’ Crackpot. The book is a series of short stories giving a glimpse into the authors world. The opening chapter is “John Waters’ Tour of L.A. (1985)” and includes the following paragraph:

Much more elusive was Annette Funicello’s garbageman. If you hang out all Wednesday night, the night she puts out her garbage (16102 Sandy Lane, Encino), you might spot him. His boss graciously declined to give out information, falsely assuming I wanted to look through Annette’s cans.

CNN just aired an informative interview with Cho Seung-Hui’s parent’s mailman. Paraphrasing:

Q: What can you tell me about the family?
M: Not much. They usually are not home when I deliver the mail. I guess that means they work.

I’m enriched and I’m sure Mr. Waters is enthralled.

Update:Now CNN is introducing the mailman as “Someone who has gotten to know the family quite well”.

Comments (0)

I am obliged to note Google’s recent acquisition.

Comments (0)

So it seems to me you’ve proved that stock market data is not random. Your procedure for generating “random” data generates a dataset that is far from random. Have you even looked at how many datasets from the universe of truely random data can not be generated by your “random data” procedure?

Dispite the fact that you keep trying to generate data that is more and more “like” actual stock market data (and hence less and less random) skilled players are still able to spot the differences.

The comment above highlights a common question people are asking both over this blog and email.

I’m not sure as to how someone would construct a test to see whether one dataset was more random than another set. However I am familiar with the efficient market hypothesis and I believe the method I use to generate random stock prices is in accordance with at least weak form of efficiency. Namely, the current price is the best predictor of the future price and you can’t beat the market by looking at patterns.

If I wanted to go out and test whether technical analysis can predict future values, I could simply do a test whereby I show a chart and ask people whether they would buy, sell or hold on the basis of the chart alone. We could then look at returns over certain periods and see whether some people are able to generate statistically significant returns above the market rate.

Now, if we assume that people can generate above market returns simply by looking at a chart (with no additional information about fundamentals), then we must believe that the chart itself has predictive power. In other words, a skilled examination of historical prices and trends can be used to predict future prices. The weak form of EMH states that the expected value of the price tomorrow is equal to the price today. And my method of generating random stock data observes this. So, by testing whether people can tell the difference between randomly generated data and actual data I am testing a super-set of EMH. If people can tell the difference, there are 3 possible reasons:

1. My model is imperfect. It is quite possible that there are better models that can be used to generate random stock prices that still obey the requirement that future prices are best predicted by the current market price.
2. My model is poorly calibrated. Even with a ‘correct’ model formulation, the model may not be calibrated to realistic market parameters.
3. It is impossible to generate ‘realistic’ stock price data under weak form EMH.

I believe all three factors are at play here, with many problems arising for highly illiquid stocks or stock that trade in small-penny ranges. An easy way to test this would be, as many people have suggested, to remove such stocks from the site. I will probably do this some time this week - at the moment I’m having a few technical difficulties related to the server move from the US to Europe (see my previous post).

I’d like to thank everyone for their feedback, and for continuing to play the game. I did not expect it to go this far and am enjoying all of your suggestions and critiques.

Comments (0)

I just moved this site to a server in Europe. If there were a faster way to transfer ~100GB of dada to some random datacenter in Haarlem then the rest of i2pi would be on the new server too.

Comments (0)

See here.

Comments (0)

Lets draw a line from (0,0) to (100,1):

Still with me? OK! This line takes y values along the real line from 0 to 1. Its nice and continuous. Lets pretend that instead of using all values from 0 to 1, we can only use 0 or 1. For a first try, lets round each of the values below 0.5 to 0 and everything else to 1:

Nothing surprising here. But we have clearly introduced error into our time series. This is what the error looks like:

What if we kept track of our cumulative error and whenever it became larger than 1, we rounded up instead of down. Lets call this function ns(). At first ns(4.1) => 4. After 9 more times ns(4.1) => 5. Over these 10 iterations, the average value is now (4×9 + 5)/10 = 4.1. Much better than rounding 4.1 ten times with the simple rounding operation and ending up with an average value of 4.

Do you want to see a picture?

Doesn’t look much like a line from 0 to 1, but the at any point along the chart the local average is much closer to the underlying continuous line than the simple rounding version. Reminds me of noise shaping.

Here is how I do it in R:

 ns <- function (x) { rx <- round(x); ns_err <<- ns_err + (x - rx); if (ns_err > 1) { rx <- rx + 1; ns_err <<- ns_err - 1; } else if (ns_err < -1) { rx <- rx - 1; ns_err <<- ns_err + 1; } rx } rounds <- function (x) { n <- x; ns_err <<- 0; for (i in 1:length(x)) n[i] = ns(x[i]); n } 

I think there is some market microstructure in there somewhere. Or I had one too many at the bar. Perhaps both?

Comments (0)

I’ve had 175 people play stock or not and the results have me scratching my balding-like-Fama head. Of the 3,515 games, people were right 1,832 times (52%). Now I may have a case of Friday stats trauma, but:

Alternatively, if we assume that there is a 50/50 chance of picking the right chart, the observation of successes so far is 2.51 standard deviations away from the expected mean of 50%. This leads me to think one or more of the following is occurring:

1. People have worked out that by hitting ‘refresh’ they can take a pass on making a choice until a really ‘obvious’ set of alternatives is presented
2. Implying that there is some factor I am not successful in capturing in my GARCH+Jumps+Rounding random stock generator
3. OR, there really is something funky going on

If you are one of the people who is doing amazingly well at the game, please get in touch with me. I’d love to know your strategy.

Update: OK. If I use the refresh trick, I can get a high score. There are some patterns that I see that are dead give aways. However, the person currently leading the scores did not use this trick. Curiouser & curiouser…

Update 2: My port-a-conomist is not being very helpful in identifying the flaws in my model. However, he has shared with me some wisdom:

When faced with a seemingly insurmountable problem, I tend to make clicking noises or chew my pen. I’ve decided that it might just be more effective to do this instead :

---

Session cookies, if we are feeling anal

Comments (0)

I have taken a short break from working on my calculator to vent my frustration at technical analysis. After the fact, sure, I can see the heads and shoulders and Elliot waves and Bollinger bands and resistance giving way to support. A priori, nope. If technical analysis made sense, then that would imply that there is more to asset price evolution than random innovations. I took some time out today to build a little model to generate ‘realistic’ stock prices based on pretty much standard models that are EMH-Kosher. If you are a great chartist, then you should be able to tell the difference between random prices and real data. Wander over to stockornot.i2pi.com and test your mad skillz.

If I feel so inclined, I’ll add scores and such later tonight so you can see if you can beat the magical Technical Turing Score of 50% accuracy!

Update: I felt so inclined and added some scoring and a shiny new logo! To date I’ve had 60 different people play, each playing an average of 19 turns. Across the board the average score is 51.09%. It looks like Fama is winning.

My personal best score is 57 / 100 (or 57% for the statistically inclined). My next challenge is to get this registered as an official sport at get it on betfair. With my track record (highlighting the value of knowing exactly how the random data is generated) I could make a killing!

Comments (0)

One of the great things about living so close to the Magnolia Bakery is that I can avoid eating their cupcakes and instead browse books across the road at the Biography Bookstore. However, whenever I do, I always end up buying a book or two.

This weekend I read Confessions of an Economic Hitman. I didn’t know anything about the book before I picked it up, but after reading the first chapter I felt kinda dirty. Don’t get me wrong, I love post-imperialist conspiracy theories, but this book is terribly limp. The basic premise of Perkins’ tale is that he was employed at a major consulting company which allowed him to work as an Economic Hit Man (EHM). According to Perkins, EHM’s are employed around the globe with the sole goal of selling debt to small nations. The debt serves two purposes, firstly financing the purchase of American goods and services. And secondly the debt presents such a burden on these small nations, that they become trapped as allies of America.

While this may be true - one only has to listen to Eisenhower’s final speech to get hints of this - my gut feeling is that rather than a covert conspiracy we have a situation whereby incentives are aligned to produce these outcomes. I know that statement is terribly vague, but its not what I am blogging about today. One particular detail that Perkins reveals in his book is that his first international consulting gigs was to the island of Java in Indonesia. His role was to inflate electricity consumption forecasts so as to justify the need for spending on infrastructure which in turn would be financed by foreign debt. Perkins talks of the gut wrenching he experienced by ignoring his superior, who booked an estimate of 5% year on year growth, and forecasting 15% for 25 years!

One of the great things about forecasts is that 25 years later we can get the historical stats and have a look-see at exactly how far off Perkins was. If you take a gander at the chart below hopefully you, like me, will fail to understand how Perkins lost sleep by corrupting the hapless Indonesians. My eyeballing gives a compound growth rate of 13%. I’d be pretty happy if I could pull off estimates that stood the test of time so well.

Comments (0)

Nothing pains me more than poorly-formed data. In a showcase of fantastic Web 2.0 technology, this site demonstrates how easy it is to let form corrupt function. With only three clicks, I was able to determine that there were 4 winners of the Nobel Prize in Physics who were male and female. And in an odd coincidence, all four of these people are John Bardeen.

I would copy-and-paste the self-aggrandizing text from the offending web site, but this is not possible. Because the site is so damn craptacular. Grrr.

How is it that this page gets highly ranked on various aggregators, but is so clearly broken?

Comments (0)

About 2 years ago, I was introduced to the blog-o-sphere. Seth and Greg convinced me to install a feed reader and since then keeping up to date on posts has become part of my morning routine. A few days ago Rands briefly mentioned the question “What if you had two minutes to live and could only send text messages?”. It is somewhat morbidly perverse to frame ones life within the bounds of a technology platform, nonetheless, I’ll follow the same vein and list my top five blogs:

• Marginal Revolution
• Felix Salmon
• Long or Short Capital
• Going Private
• My Blog

Perhaps with the exception of the latter, there is a clear theme here. I like finance & economics. I joined Root to bridge the gap between my background in marketing analytics & technology and finance. Two weeks ago I resigned.

As the first employee at Root it was a difficult decision to make. I first became involved in the business over 2 years ago and saw it grow from a small idea, to a number of different ideas, and back down to a core idea. Over that time we acquired two companies and launched a number of innovative changes in the mortgage lead business. Today the company clearly offers the best exchange technology and with our new leadership we have an unrivaled pedigree in the mortgage business. As we raised capital, I met with some of America’s greatest financiers and learned a good deal about building marketplaces, companies and the mortgage business in general. More so, I helped put together a fantastic team and Root is now in a position to succeed with or without me.

I came to America in 2004 to pursue my MBA, with the intention of moving into a sales & trading type role. Working in campaign analytics gave me a good sense of the value of my quantitative skills, however working with traditional advertising types was somewhat demoralizing. While I had quite a few savvy clients, for the most part the concept of ‘ROI’ didn’t mean much to your typical campaign manager. Much has changed since then, but I felt that I would be better off applying my skills in an industry where P&L is everything.

As the mortgage market is experiencing a downturn it will be the more sophisticated participants in the industry and on the exchange stand to benefit. Even today the majority of the mortgage lead business is catered towards small brokers who look to profit by turning quick refinances. I strongly believe that this will change and more top-tier lenders will enter the market. At least for the sake of the consumers, I hope that this is the case. And while I would like to be involved in this transition, no matter what lengths we go to bringing financial innovations to this space, it is essentially a marketing services opportunity. And my passions lie elsewhere.

I am planning to take April off to concentrate on some side projects and to visit friends & family in Australia. Come May, I’ll be doing something new.

Comments (0)

The tables at the diner were almost stacked on top of each other. With my burger, thanks to the vocal lawyer or architect perched dangerously close to my lap, I learned that the only real way to understand the ins and outs of litigation was to find yourself in a situation whereby you were lost in the depths of suing yourself.

The gradual erosion of inter-patron distance has lead to a distinct change in the rhythm of dining out. Waiters are now bipedal alternators, gyrating as they maneuver their way from table to table. Their constant swiveling achieves two goals. Firstly it imbues the venue with a cadence that keeps one from getting too antsy about how long it takes to get a meal in this town - clearly things are happening and its only a matter of time before your meal is served. But I feel the true purpose of sashaying whilst channeling Chubby Checker is to avoid patron-head to staff-crotch/butt contact. This is a big no-no and in these litigious times could clearly land an otherwise upright waiter in court. These are truly devolutionary times.

Comments (0)

In my calculator infused haze, I almost forgot that yesterday was Number 6’s 79th birthday! Pop goes the weasel.

link for those in bloglines
Update: Woops! Number 6’s birthday is actually 19th of March. Which also happens to be Patrick MacGoohans’ birthday too. Talk about a creepy coincidence.

Comments (0)

Update:I’ve overlaid the HP-42S keyboard on this. The function keys don’t 100% map to my layout, but it gives you an idea of what it does.

The calculator keyboard is now working and hooked up. Nothing too technical going on here. Yet it took me a while to get it to work right. I have to learn to go to bed at a reasonable hour and not keep hammering away at projects when I’m not at my best. The rational Josh knows that he should stop long before spending 2 hours writing a software work around to avoid what would have otherwise been achievable with a 15 minute soldering job. The annoying fact is that I always know when I get into such states, but unfortunately there is little I can do to get myself to walk away and get some rest. During the work day its easy to take a 15 minute break. However when its 2am, taking a 15 minute break means 15 minutes less sleep. And I always seem to believe that the problem is always solvable in 10 minutes. Thus the cycle continues, and before I know it the sun is coming up.

Oh welp. One day I will learn.

Comments (0)

In the past week I have:

• Ditched the Yagarto/Eclipse/Windows development environment. It was taking me 10x longer to do anything compared to my usual dev setup. I think this old dog is totally incapable of learning new tricks. I am very comfortable using gcc/gdb/openocd/vi/linux.
• Ported my calculator code to the ARM7. I’m pretty happy that my malloc routines worked out of the box. I had to write some minimal code to replace curses with an LCD driver. I considered using all of newlib for sprintf/floating point. In the end I wrote my own sprintf as I only needed a few small functions. I did steal some newlib floating point code.
• Placed a big order on mouser.com for random 4000 series CMOS chips, resistors, capacitors, and importantly 40 push button switches. I plan to assemble the keypad in the next few days.
• Learned that although 64k of RAM was cool for me until circa 1992, I need more RAM. The original HP-42S had 7k (IIRC), but I did spend quite some time writing SQL like functions for my language and would love to have 32MB+ of SDRAM at my disposal. The problem is that the ARM7 chip I am using has no SDRAM interface, so I’m currently deciding between using an LPC-3180 or a SAM7SE. The latter is very similar to my current chip, but with the added bonus of both NAND flash and SDRAM controllers built in. The LPC3180 is an ARM9 architecture chip which boasts memory management in addition a vector floating point unit. Ideally I would use the LPC3180, as hardware floating point would give a massive speed boost to a calculator. However, over the past week I have become fairly comfortable with the ARM7. If I were to go with the ARM9, I would have to buy a new evaluation kit and re-do all the work I have done over the past week. I need to do some soul searching to find out whether my drive for parsimony (laziness) is greater than my desire for hardware floating point.

  Once I complete the keypad, I’ll upload another fancy ’screencast’ to YouTube.

Comments (0)

As you can see, I have the screen working.

Comments (0)

If someone uses a quote from me in their signature on a Final Fantasy Legend forum, does that make me a nerd by reference?

Comments (0)

After way too many hours putzing around with a JTAG wiggler I bought from diygadget I gave up and built my own. The DIY Gadget one doesn’t come with a full schematic so it was a pain to work out exactly why it wasn’t working. My wiggler is based on the schematic from Dominic Rath (of OpenOCD fame) but without any buffering. I’m running it at a slower speed, but it seems to work a charm.

I worked through the Yagarto tutorial I got a simple blinking LED program working!

This makes me very happy. Now that my development environment is all set up and working, I can start wiring up my screen and keypad. After that I’ll work on porting the calculator OS that I have been working on.

Here is the dead simple wiggler schematic that I am using:

DB-25	Function	JTAG
3	TMS	7
4	TCK	9
5	TDI	5
11	TDO	13
18-25	GND	4-20

On pins 3,4,5 & 11 I have a 100 ohm resistor. NB: This is working with the Olimex SAM7-P256.

Comments (0)

At risk of “forfeiture of [my] entire interest“, I’d love to share some “CONFIDENTIAL INFORMATION” regarding the latest news in the Skye Ventures / Bandagro bond case. The news is that Skye has gone to extraordinary lengths to implement silly security on their website. While I will not share with you the full details (you should be able to work it out yourself), Skye has chosen to embed the password to their ’secure’ website within the source of their page. Of course, they have also ‘removed’ the ability to view the source of their page by disabling the right click mouse action.

Let me explain…

1. Skye Ventures has a documents that they only want to share with select parties. One must assume that these parties have been given a password upon signing some agreement.
2. Rather than using standard security practices, they have implemented their own security system.
3. This system checks the supplied password against a password stored within their web page (un-encrypted)
4. If the passwords match, it redirects the browser to a document. The location of this document is also stored within the web page completely un-encrypted.
5. To view any web page, your web browser must view the ’source’ of the page and then render it for you.
6. In an extraordinarily feeble attempt to prevent humans from accessing this source, the web designer has used a trick to disable the right click button on your mouse (which is usually used to access the ‘View Source’ option
7. This completely ignores the fact that you can download the source the same way the browser downloads the source - hence giving you full access to their secret documents.
8. Rather than jumping through so many hoops in attempt to build a secure web site, Skye should have used any number of standard methods for web security…

   The first mistake in writing your own security system is the belief that you should be writing your own security system.

Comments (0)

As paralleled in the world of calculating stock indices averages, yesterday I increased in age by about 1.7 years in a matter of seconds. Well maybe parallel is not the right term - my age increased by 6%, but the DJIA dropped around 4%. But both arose thanks to the power of modern computers. (NB: I don’t think the market drop was caused by a computing snafu)

As I ponder my new found years, I find myself paying more attention to the the rising commentary on mortality bonds. My question to the financial glitterati is that if I get a lump sum payment on retirement, will I be able to obtain an annuity that pays me at an increased rate if I promise to take up smoking again? If not, where is all this market efficiency I’ve been hearing about?

Comments (0)

I was just on the phone and the operator wanted to know how old I was. I said 27. I also happened to have a SQL window open, so I did a quick check.

select extract(days from now() - '1978-06-18') / 365;
?column?
------------------
28.7150684931507

It turns out that I am actually 28.

In other news, it turns out that I am actually a dork.

Comments (0)

I spent far too much time on the weekend looking into various microcontrollers for my calculator project. Rather than make a rational choice, I decided to just buy one from an online store that sold pretty much every part I needed. Its not the ideal evaluation board, but its a start.

Here’s what you got:

1 x Development Board Atmel SAM7-256 (SKU#: SAM7-P256) = $69.95
1 x Graphic LCD 128×64 STN LED Backlight (SKU#: LCD-G12864) = $19.95
1 x USB Cable A to B - 6 Foot (SKU#: Cable-USB-AtoB-6) = $2.95
1 x Wall Adapter Power Supply - 6VDC 300mA (SKU#: Tools-PS-6V) = $4.95
1 x LiPoly Charger - Single Cell 3.7-7V Input (SKU#: Batt-CG-MAX1555) = $14.95
1 x Polymer Lithium Ion Batteries - 2000mAh (SKU#: Batt-LIP2000) = $12.95

Comments (0)

If I have learned anything about promoting software, it is that a screencast makes or breaks the deal! So here is a little movie I call

How to count down from 5,000 in under one minute: A Tale of Tail Recursion.

A comment on my previous post pointed me to the Joy Programming Language, which appears to be similar to what I am developing. However, my language is designed to be used on a calculator. Which means that it must be usable with a small screen (I’m shooting for 30 characters x 4 lines). Importantly it is designed to be used on with a limited keyboard and thus with a minimal number of keystrokes.

The program above is ((32 # 1 - 32 @) (3 #) i), which can be read as

1. Store the following in register 3:
`If (register 32 less 1 (store back in register 32)) is not zero then execute register 3 '
2. Store 5000 in register 32
3. Execute register 3


I love long weekends!

Comments (0)

Since the death of my HP-42S, I’ve been fantasizing about what my ideal calculator would be like. Last week I started coding up a proof of concept for a calculator language that combines what I think is the best of RPN and Lisp. I have a deep rationale behind how and why I want to build this and the importance of designing a language specifically for a calculator and the HID considerations of such. I wont go into any of that now. But here is a quick snippet of a test session (my comments are in italics:

 push 1 onto the stack > 1 > 2 > 3  pop all numbers off the stack and turn them into a vector  > v -> [ 1.000000 2.000000 3.000000 ]  turn on Programming Mode  > ‘ Program Mode: ON > 10  In programming mode operations are pushed onto the stack, rather than executed  > * -> ‘* > ‘ Program Mode: OFF  show the stack, top down  > show (000) ‘* (001) 10.000000 (002) [ 1.000000 2.000000 3.000000 ]  The > operator turns the contents of the stack into a list  > > -> ( [ 1.000000 2.000000 3.000000 ] 10.000000 ‘* )  Execute the top of the stack  > x -> [ 10.000000 20.000000 30.000000 ] 

Comments (0)

Colin Raney wrote:
> http://www.midasoracle.org/2007/02/04/blog-of-the-day-joshua-reich/
>
> hows things man? How have you been?
>
Things be good. Living life with all the benefits that come from being blog-of-the-day-joshua-reich.

Chris, my theme is a bastardization of fun-times-inspired-by-mistylook-1. I rejigged some of the CSS and removed all of the hidden SEO tricks that the author had embedded to promote his/her own site.

Comments (0)

Technorati Profile

---

Whats this? I just discovered Technorati and found out that most of my readers came to my old blog to learn & ogle at my camera hack project (original, preblogging page is here). Here is one post that refers to me:

I have no idea what made Josh Reich decide to take apart his VX2000 and replace the lens, but I am certainly glad he did. It is very well documented minus a real purpose. I can’t see anyone else doing this, but it does produce some interesting images. Take a peak inside the mind of an optical engineer madman. Oh, and I realized the purpose of this; just because he can. I’m still amazed.

From this I learn a few (N=2) things:

1. People like my hacks (wow, I wish I knew this back in Australia when most of my spare time was spent on being a ‘useless scientist‘, to quote a dear friend)
2. People like my drunken posts.

The good news is that some books arrived from Amazon the other day, including a few on optics. My favorite would have to be Optical System Design and I have plenty of ideas. In fact last night was spent taking apart a number of 35mm lens systems to get at the glassy goodness inside. In addition to the tactile enjoyment of working with physical objects, rather than abstract concepts, I have always enjoyed browsing through engineering texts. Much of engineering leaves off from the idealism of physics and has to deal with solving real world optimization problems. Any generic text on optimization always misses out on the subtle art that evolves in particular realms of engineering. I’m quickly learning that optics is no different. There are just so many ways that one can construct a lens system and the math is just too hard to tackle analytically. I also learned that there are often competitions for experts in the art to design lenses that fit certain specifications and so many varied designs can result. When I have some cash, my next project is to make a small optics workbench to play with some designs of my own. I’ll let y’all know how it goes.

Comments (0)

I came across this story thanks to Marginal Revolution:

A few years back, Toronto-based gold mining company Goldcorp (GG) was in trouble. Besieged by strikes, lingering debts, and an exceedingly high cost of production, the company had terminated mining operations….Chief Executive Officer Rob McEwen needed a miracle. Frustrated that his in-house geologists couldn’t reliably estimate the value and location of the gold on his property, McEwen did something unheard of in his industry: He published his geological data on the Web for all to see and challenged the world to do the prospecting. The “Goldcorp Challenge” made a total of $575,000 in prize money available to participants who submitted the best methods and estimates…

Within weeks, submissions from around the world were flooding into Goldcorp headquarters. There were entries from graduate students, management consultants, mathematicians, military officers, and a virtual army of geologists….

The contestants identified 110 targets on the Red Lake property, more than 80% of which yielded substantial quantities of gold. In fact, since the challenge was initiated, an astounding 8 million ounces of gold have been found—worth well over $3 billion. Not a bad return on a half million dollar investment.

I was immediately reminded of the Netflix Prize, which I had a stab at. What I find particularly interesting about these two prizes is that they are replacing the payment and work of traditional employees with the input of competitors who are competing for a prize (or series of prizes in the Netflix case). I don’t know much about geology, but according to a quick browse through job listings, senior geologists working in mining earn a decent living. But not $575k. It seems to me that two key factors drive the success of the prize model over traditional salaries. Researchers in behavioral economics were quick to point out that rather than acting rationally, people often accept low odds of winning a large sum, rather than the risk-neutral valuation of that prize. For example, people may accept a 10% chance of winning $100 rather than a guaranteed $10. See prospect theory for more. This is probably compounded in the prize-for-work case compared to a pure lottery, whereby a selection bias induces entry only from people who strongly believe that have a better chance of winning than everyone else. In a pure lottery everyone has the same odds, but in a competition, some people are more skilled than others, but everyone is likely to believe that they are endowed with a better chance. These factors result in more people working on the task than would otherwise be involved under a salary model. Plus as a pure meritocracy the benefits in the repeated game scenario are even higher.

Of course, the Goldcorp Challenge and the still-in-progress Netflix Prize are examples where this sort of system appears to work. I also recall a number of platforms popping up a few years ago that awarded bounties for completing open source software tasks. Most of the sites appear to be dead today. To get an idea of what I’m talking about, have a look at Bounty County. They appear to be having some mediocre success, rather than a smashing revolution in the way coders get paid they most closely resemble contract posting boards. Well defined tasks, small payouts, small target audience. Perhaps the most well known blow-up of a bounty system was the fames Bounty Quest, which sought to obtain prior art to invalidate patents. I leave off with a Tim O’Reilly’s response to the question What Happened?:

I had high hopes for BountyQuest, too; it seemed like a great idea. But while I still believe that the failure to search for prior art remains a major problem for the patent system, the company was not able to make a successful business bridging the gap. Of course, this could simply have been an execution issue, or market timing. But it could also have been the fact that the patent mess is a thorny thicket that doesn’t lend itself well to penetration by amateurs. I know that when we did the bounties on the Amazon 1-click patent, there were numerous people who sent in what they thought was important prior art, but which the attorneys who reviewed it didn’t find useful. Of course, we did find some really good stuff, too–some months after we’d awarded the bounty, splitting it three ways for prior art that wasn’t completely on the money, someone sent in a killer piece of prior art. I still have it on my bookshelf, in the odd event that Amazon loses its senses and sues anyone else over 1-click. (As you may recall, Amazon and B&N settled the suit, and Jeff Bezos seems to have eschewed offensive use of patents for now, partly as a result of the awakening that he had after the Net protest I led back in 2000.)

As to the next generation, my guess is that as patent attorneys get more Internet-savvy, googling for prior art and posting queries to relevant mailing lists or newsgroups are just going to become part of the routine toolkit of anyone doing patent searches.

Tim

Comments (0)

One of the things I have always liked about C (the programming language), is that there is so little to know. I’ve been coding in C for over 15 years, and its been a very long time since I’ve learned anything new about the core of the language. Today I learned something new.

 josh:~ josh$ cat evil.c #include <stdio.h> int main (int argc, char **argv) { int i; for (i=0; i<4; i++) { printf ("%c", i["i2pi"]); } printf ("\n"); return (0); } josh:~ josh$ gcc -Wall -pedantic -ansi evil.c josh:~ josh$ ./a.out i2pi josh:~ josh$ 

Comments (0)

A few days ago I was all agog over weatherbill. Well, scratch that, now I’m all about instant messaging technologies. What sparked this turn around? Good thing you asked! Earlier I read this article on Fixed Income E-Trading:

I was chatting to a Hong Kong based colleague earlier this week. As the chat turned to specific countries and their trading habits, he mentioned Korea (South of course!).

Interestingly 30% of the trading in the Bond Market goes via the Korean Stock Exchange. The other 70%? Via YAHOO instant messaging!!!!!

Essentially the salespeople at the six main securities houses send out broadcast messages to the clients with a list of offers, axes etc. The clients react accordingly or do keyword searches when looking for a particular bond or issuer etc. Any efforts by ECN’s to break in to Korea have met with plenty of apathy & a genuine satisfaction in the way things are done now - both sellside & buyside.

I’m not suggestion this as a way forward in Europe, but viva la difference!

Compare and contrast with this post from Gen Kanai:

…Korea will only get beyond this problem by 1) applying Korean laws on open standards to the certificate authorities, 2) reassigning new certificates which work with open web standards to all Koreans, 3) reprogramming all Korean websites to support 128 bit SSL which will allow for a heterogeneous marketplace of operating systems and web browsers. This is a herculean task and thus Korea stays hostage to Redmond….

Comments (0)

Although I live in New York, I will always be a Melbourner. And one of the defining characteristics of a Melbourner is their passionate hobby of debasing Sydney given any opportunity. So I was delighted to read the following in an article from my home town newspaper, from an article entitled Intellectually bereft, obsessed with money and status … they must be from Sydney:

COULD Sydney be the saddest of cities — intellectually bereft, spiritually empty? Are its residents T.S. Eliot’s proverbial Hollow Men — heads together but whispering nothing except deadening conversations about the latest movement of the property market or fad diet?

Sydney’s culture of the relentless pursuit of property, perfect bodies and status has British psychologist and author Oliver James worried. As part of research for his recently released book, Affluenza, he travelled to seven countries to research the effect of consumerism on happiness.

He found the obsessive pursuit of money and possessions was not buying happiness. The affluenza virus was worst in Sydney, where he found interviewing locals a depressing experience. It was, he said, “the most vacuous of cities. The Dolly Parton of cities in Australia.”

Adelaide and Melbourne had a “different vibe” and did not strike James as being as materialistic as Sydney. He had not been to Sydney before and expected a “philistine nation” of “jolly, uncomplicated fun-seekers”. Instead, he found a city in thrall to American values and a puritan work ethic that robbed life of joy and meaning. Middle-class Sydney, he writes, is “packed with career-obsessed workaholics”. When they are not working the longest hours in the developed world, they pursue perfect bodies through joyless fitness regimes, or obsess about property prices. Always, they are looking around anxiously, in the hope that others aren’t doing better than them.

“(It was) full of people who place a high value on money, professional status and appearance,” he said. The result? Sydneysiders have a “greater risk of suffering from mental illness — depression and anxiety”.

“They (Sydneysiders) were like the tin man from the Wizard of Oz. They had no idea of the point of their lives, other than to get rich.” James noted Bureau of Statistics figures highlighting a rise in depression that coincided with a bullish property market, which caused stress and anxiety — particularly among young Australians.

While Britain has “its Posh and Becks”, — obvious examples of conspicuous consumption — cultural differences, including a more entrenched class system, has put the brakes on the spread of consumerism in Britain.

“The British, compared to the US or Aussies, are less easily convinced that money will get you further. The British elite have been around for an awfully long time and there is not the crassness of the Australian rich.”

While he despaired about Sydney, he found “affluenza” was not as prevalent outside the Western world. “Shanghai has the virus pretty badly but the vaccine is Confucius. Your best is good enough.” But in Sydney, a strong materialistic culture means that no one ever quite feels like they can “measure up”. His advice to Sydneysiders? “Start reading.” Starting with his book, perhaps?

That said, its somewhat odd that while I could never see myself living in Sydney, I find myself in New York - a city that defines “affluenza”. Earning more money than ever before, I live in an apartment under one-quarter the size of my place in Melbourne. I’ve always had a strong work ethic so the hours here don’t really phase me, but the gap between where I am now and where I want to be is bordering on intense. So why do I stay? Apart from the fact that I have laid roots here, I am actually starting to find comfort in, what Jon calls, the “Adversarial & Aspirational” culture here. One of the reasons why I am drawn to work in finance is that the external benchmarks of markets will quickly let you know when you are wrong. In Melbourne, it is so easy to get by without such guidelines and just float. Floating is fine for a vacation, but for now New York is both the where and how of my life.

Comments (0)

The world of blog. is complete!

Comments (0)

I’ve just moved most of my posts from my old blog over to here. There is probably a fancy way of automating this, but I did it by hand. So all the timestamps are wrong, and I haven’t moved across the comments. Maybe I’ll get around to fixing those. Anyway, I’m happy to be home on i2pi.com

Comments (0)

THIS BLOG HAS MOVED TO http://blog.i2pi.com/
Comments (0)

I’ve been trying to come up with something eloquent to say about weatherbill.com, but I’m just blown away: it’s just so damn cool! It embodies everything that I see as driving the future of online marketplaces. Just as intrade.com is giving economic derivatives a run for their money, I see this trend continuing with weatherbill as compared to the more buttoned down offerrings at CME.

Working at Root Markets I can’t truly appreciate the perspective of soemone who is taking on a pre-existing market place and attempting to replace it with something much better. I do, however, have some theories on what it takes to succeed.

Last year I was lucky enough to spend some time with one of the founders of intrade, and I was curious to learn more about what was driving their success against the incumbet economic statistics market. The lesson was simple: you need to offer contracts that serve real needs. The message that I heard was that the payoff structure of the CME Auctions Markets were not successful in doing so. Skimming over the contract specifications and current volumes, this doesn’t appear to be the case now - but it was back then. The point still remains - serve a real need - design to hedge a real risk.

Closer to home, for me, is the mortgage market. An early story that our chairman shared with me was of the evolution of the CMO market. Prior to securitization there was very thin trading in whole loans as it was commonly believed that to understand your risk exposure you needed to understand the plethora of intangible factors that could possibly drive prepayment and default risk. Buying a whole loan was thought to expose you to the risk of what side of the street the house was on, etc. Of course, with the tranche structure of the new instruments, these risks fall out leaving the holder of a CMO with exposure to the right kind of risks. Understandable systematic risks. Portfolio theory 101 at work.

As the market evolved the models improved as can be see if you compare the book for a recent issue to that of a 1980’s era issue. Nowadays, with liquidity chasing liquidity, savvy investors are seeking more specialized exposure and are slicing and these tranches looking for an edge and rebundling whole loans where once the market was super thin. Specialist funds are able to do this because of the modelling power afforded by new technology. The computer I am writing this on can do a gazillion calculations per second. Thems alotta calculations.

Compare the ability to slice, dice and model on weatherbill with the handful of contracts available on the CME. If I am a small business owner, I don’t want to hire someone with a degree in financial engineering to work out how to best hedge my risks. And even then, the hedge will be imperfect when there are only a limited number of contracts to choose from. Businesses want to be able to trade specialized and differentiated contracts to best approximate their risk exposure. Yes it may be difficult for an army of broker dealers to make a market in millions of contracts, but this is the sort of problems that computing power can be thrown at. If you have the right pricing models, you should be able to scale to a silly number of contracts far more cheaply than if you had to hire high rent dealers to make the market.

This sort of scalability is not afforded to all types of markets. In futures markets, where the cost of carry is infeasible, a prediction market forms. This is the case for weather, and likewise it is the case for consumer leads (my speciality). Prediction markets are special in that there is no first-order linkage between trading activity on the derivative and the current spot price. Compare this to storable goods: if the spot price today decreases, I can always buy a little more today and hold the good for use later thus changing the value of a forward contract, and vice versa. However, if it is hot today, I can’t store the weather for the purpose of changing the climate in 6 months time. If the price of leads decreases today, the rapid half life of consumer attention means that a marketer will have a difficult time realizing any value from a 6 month old lead, thus leads, like weather, can’t be stored.

To understand why this is relevant in determining whether a market can scale to a large number of differentiated contracts we need to look at how market makers work. If I buy a contract for a the future delivery of some good, the market maker must find some way to hedge her risk of being the counterparty to your contract. And when the volume of sales effects the spot price, liquidity risk is present. In prediction markets you don’t have to balance the effect of future market activity against spot underlying values. If you have an edge in pricing future movements, you can scale with diminished exposures to liquidity risk.

In short, prediction markets are different to standard futures markets. And there is little reason to not offer highly specialized contracts to participants. Weatherbill does this very well.

By offering the ability to upload historical revenue numbers, weatherbill makes it easy for unsophisticated businesses to asses their weather exposure and to specify a contract that exactly matches those risks. It is just so easy to use and there is something, dare I say, very Web 2.0 about it all. While it is technology that empowers this, weatherbill different from, say, hedgestreet in that there is amazing simplicity to what is otherwise a complex product.

It’s just so damn cool!

Comments (0)

I just returned from vacation in Vieques. You can see the photos from Shianling’s point of view here

My photos came out a bit funny

Adam found one of the old i2pi.com’s on the way back machine. View it here. I must admit that I’m a bit reluctant to share this with you. What can I say - I was young.

Comments (0)

What follows is a letter I wrote in high school in response to a comment made by my religious studies teacher about the absence of ghosts in Judaism. As one of the few Jews at my Anglican school, I found myself taking more pride in my roots than when I was only one of about 1,000 at my Jewish school.

I’m posting it here with a slight reference to Joel Spolsky’s latest post (which briefly talks about my current employer) and also because I am amazed that I managed to find this essay lurking in the depths of i2pi.com

It is very true to say that Judaism has very little historical associations with ghosts in the spooky spiritual sense, but to use this in the formation of the conclusion that there were simply no Jewish ghosts is pure adulteration of facts. This may seem a bold statement, but one must be so when dealing with a topic that is so ‘iffy’.

The first if in the iffiness is the definition of a ghost. To resort to a dictionary, The Macquarie defines a ghost as ‘a disembodied spirit of a dead person’ or ‘a spiritual being’; That will suffice.

I feel that there is enough evidence in traditional Jewish literature to support the claim that there were or are Jewish ghosts. But before one goes jumping into the five books one must firstly realise that the Torah only forms part of a small part of the Tanach textual trinity, containing the Torah, the Nevi’im and the Kethuvim; The Bible, The Prophets and The Writings. These three form, together with the commentaries of commentators like Rashi, the basis of Jewish ritual, custom and tradition. Without its counterparts the Torah is very much a vague thing, and it is probably in this fact where the beginnings of many a war are buried. So to form an opinion on any matter of Judaism one must at least consult the Tanach in its entirety.

If we draw our attentions to the Torah, we can find many references to spiritual beings, most of them classed under the broad heading of ‘god’, and by the second definition of ghost, we can classify these ‘beings’ as ghosts. The prime suspect for such a labelling of god himself (he is referred to as he not she in the Torah, so I will stick to that convention). Take Deuteronomy 4:12, ‘You heard the sound of words, but see no form…’. Here we have a description of god as sort of presence, who can make himself appear ghostlike to people. Now one may argue that this is a tenuous basis for supporting my aforementioned claim, and I agree, but wait, there is more.

In addition to the myriad of ghostlike appearances of god, we have the angels. These group of lucky souls have reached the court of god, and their duty is to appear on earth to carry out duties for god. In his 12th century Guide for the Perplexed, Maimonides opposed the idea that such angels are corporeal beings. So if they are not physical they must be metaphysical: ghosts. These beings pop up all through the Torah usually acting as schleppers for god. See Genesis 18:2, 3:2, and 13:6. In these references the angels appear as messengers, fire and in the last case in the form of a person; Not quite white sheet and chains material, but definitely ghosts.

It has been stated that the idea of ghosts as being merely the risen dead is just a recent concept, but if we cast our minds back to Eziekel 37 we find a very such references. In this chapter, to cut a long story short, god leads Ezi down to a valley of dry bones, whereby Ezi asks god if the bones shall every live again and god replies ‘I will cause breath to enter you and you shall live again, I will lay sinews upon you, and cover you with flesh, and form skin over you. And I will put breath into you and you shall live again’. After this Ezi was disturbed by the sounds of rising bones as they formed ghostlike human figures. Also in Daniel 12:2-3 we find further references to the dead rising from ‘sleep in the dust of the earth’ to inhabit the Earth evermore in a ghostlike presence.

So far we have had god , angels and the dead appearing in human form, or some spiritual sense inline with the conditions of the definitions stated above; We have seen ghosts, but one ghost is probably more well known within the Jewish faith than any other, Elijahu. Elijahu was a prophet, and his greatness was such that he was appointed to appear alive again before the judgment, and coming of the Messiah to prepare humanity and resurrect the dead, see Malachi 3:23. But until his appearance he is supposed to appear in Jewish custom, derived from the commentaries, annually at the household of each Jew, signified with the ritual doorpost, the Mezzuzah, where upon he would enter the house during the Passover service to drink a sip of wine especially left out for him, in his own glass. Each year Jewish children sit with one eye poised over the cup to see that fraction of a drop disappear, and each year the same joke is made countless times globally in over a hundred different languages, about how drunk Elijahu will be by the time he has drunk from so many cups, and each year the joke becomes less funny. All of this for a ghost that some suppose does not exist.

We have jumped around from one book of the Tanach to another, some pointing the finger to pillars of fire that lead the Jewish people in Ba’Midbar, others feature messengers of god coming to Earth in human manifestations. We also have god appearing to Ezikel and resurrecting the dead as ghosts. Finally we saw how Elijahu, the archetype when it comes to Jewish ghosts, and his worldwide trek drinking from the cups of Jews in apprehension of the coming of the Messiah, will be the final Jewish ghost, and with his appearance all will be revealed. Until then we will are forced to resort to one of the most holy of Jewish sources, Woody Allen; ‘There is no question that there is an unseen world. The problem is, how far is it from midtown, and how late is it open ? Unexplainable events occur constantly. One man will see a spirit. Another will hear voices, A third will wake up and find himself running in the Preakness… What is behind these experiences ? Or in front of them, for that matter? Is it true that some men can foresee the future or communicate with ghosts? And after death is it still possible to take showers?’

Comments (0)

I found this via the way back machine. I probably wouldn't be so candid today, nor would I use the same words. Nonetheless, I wrote it, and to some extent it still rings true and is available for the world to see.

response is the output of my occupation of this media volume, related to the (con/de)struction of one dimensional signals in time. or something like that. i dont really have a purpose to this site, its just a place for me to chuck my junk so that i can access it whilst i am away from my home base. i dont like going away from home, especially if i can not take at least one of my computers with me. i guess you could say i am not very well adjusted, socially. but thats ok, this is a new dawn, and there are probably many people out there like me, similarly lacking in interpersonal skills, in place for interprocess communication. why make love when you can make a remote procedure call. what is music about really ? as far as i can tell its just a tag that people can pin on them to make them different from other people, when in reality we are all very much the same. thats why if what was traditionally an 'underground' piece of music, finds itself into the 'mainstream' - the people who originally liked the music will deny that, and claim to hate it, when they are only doing so because they have been put in the position of no longer being able to use that music to make themselves special. why do people want to be special? its alot easier to become special by developing a personality than it is to become special by becoming physically attractive, or going to the gym, or cleaning behind ones ears. thats why i dont write music any more. im not very special, you see, hence any music i produce would not help others be special - hence it would fail its ultimate purpose in our consumer society. of course music can be fun to listen to and can provide enjoyment. its easy to listen to music with lyrics and walk away with a very concrete understanding of why you feel like you do after experiencing that music. when ones moves to instrumental music, certain musical constructs have evolved amongst the genres so that people can walk away easily satisified that the music that they have heard is good. howevere for the poor sods who listen to 'experimental' music its much harder for them to justify both to themselves and more importantly to others why the music is 'good' as it may not have the same constructs such melody or rhythm that allow people to judge conventional music. too often people like this 'experimental' music because it allows people to label themselves, as listeners of this music, as intellectual/special/different/horny(?). personally i like 'experimental' music for all the previously stated reasons, but also in addition, i (as a programmer and a theoritisiticsix) i like to thing about the process that was involved in the creation of the final piece. for me creation is art. i write software as art, no specific goal, just write because it allows me to hide from reality. i have no desire to earn my living from writing software. no constraints. my software does not even have to work to satisfy my. it doesnt even have to be broken in a clever way. it just has to involve me, and thinking, because, after all, its what i do best. but no better than most people.

Comments (0)

What follows is a letter I wrote in high school in response to a comment made by my religious studies teacher about the absence of ghosts in Judaism. As one of the few Jews at my Anglican school, I found myself taking more pride in my roots than when I was only one of about 1,000 at my Jewish school.

I'm posting it here with a slight reference to Joel Spolsky's latest post (which briefly talks about my current employer) and also because I am amazed that I managed to find this essay lurking in the depths of i2pi.com

It is very true to say that Judaism has very little historical associations with ghosts in the spooky spiritual sense, but to use this in the formation of the conclusion that there were simply no Jewish ghosts is pure adulteration of facts. This may seem a bold statement, but one must be so when dealing with a topic that is so ‘iffy’.

The first if in the iffiness is the definition of a ghost. To resort to a dictionary, The Macquarie defines a ghost as ‘a disembodied spirit of a dead person’ or ‘a spiritual being’; That will suffice.

I feel that there is enough evidence in traditional Jewish literature to support the claim that there were or are Jewish ghosts. But before one goes jumping into the five books one must firstly realise that the Torah only forms part of a small part of the Tanach textual trinity, containing the Torah, the Nevi’im and the Kethuvim; The Bible, The Prophets and The Writings. These three form, together with the commentaries of commentators like Rashi, the basis of Jewish ritual, custom and tradition. Without its counterparts the Torah is very much a vague thing, and it is probably in this fact where the beginnings of many a war are buried. So to form an opinion on any matter of Judaism one must at least consult the Tanach in its entirety.

If we draw our attentions to the Torah, we can find many references to spiritual beings, most of them classed under the broad heading of ‘god’, and by the second definition of ghost, we can classify these ‘beings’ as ghosts. The prime suspect for such a labelling of god himself (he is referred to as he not she in the Torah, so I will stick to that convention). Take Deuteronomy 4:12, ‘You heard the sound of words, but see no form...’. Here we have a description of god as sort of presence, who can make himself appear ghostlike to people. Now one may argue that this is a tenuous basis for supporting my aforementioned claim, and I agree, but wait, there is more.

In addition to the myriad of ghostlike appearances of god, we have the angels. These group of lucky souls have reached the court of god, and their duty is to appear on earth to carry out duties for god. In his 12th century Guide for the Perplexed, Maimonides opposed the idea that such angels are corporeal beings. So if they are not physical they must be metaphysical: ghosts. These beings pop up all through the Torah usually acting as schleppers for god. See Genesis 18:2, 3:2, and 13:6. In these references the angels appear as messengers, fire and in the last case in the form of a person; Not quite white sheet and chains material, but definitely ghosts.

It has been stated that the idea of ghosts as being merely the risen dead is just a recent concept, but if we cast our minds back to Eziekel 37 we find a very such references. In this chapter, to cut a long story short, god leads Ezi down to a valley of dry bones, whereby Ezi asks god if the bones shall every live again and god replies ‘I will cause breath to enter you and you shall live again, I will lay sinews upon you, and cover you with flesh, and form skin over you. And I will put breath into you and you shall live again’. After this Ezi was disturbed by the sounds of rising bones as they formed ghostlike human figures. Also in Daniel 12:2-3 we find further references to the dead rising from ‘sleep in the dust of the earth’ to inhabit the Earth evermore in a ghostlike presence.

So far we have had god , angels and the dead appearing in human form, or some spiritual sense inline with the conditions of the definitions stated above; We have seen ghosts, but one ghost is probably more well known within the Jewish faith than any other, Elijahu. Elijahu was a prophet, and his greatness was such that he was appointed to appear alive again before the judgment, and coming of the Messiah to prepare humanity and resurrect the dead, see Malachi 3:23. But until his appearance he is supposed to appear in Jewish custom, derived from the commentaries, annually at the household of each Jew, signified with the ritual doorpost, the Mezzuzah, where upon he would enter the house during the Passover service to drink a sip of wine especially left out for him, in his own glass. Each year Jewish children sit with one eye poised over the cup to see that fraction of a drop disappear, and each year the same joke is made countless times globally in over a hundred different languages, about how drunk Elijahu will be by the time he has drunk from so many cups, and each year the joke becomes less funny. All of this for a ghost that some suppose does not exist.

We have jumped around from one book of the Tanach to another, some pointing the finger to pillars of fire that lead the Jewish people in Ba’Midbar, others feature messengers of god coming to Earth in human manifestations. We also have god appearing to Ezikel and resurrecting the dead as ghosts. Finally we saw how Elijahu, the archetype when it comes to Jewish ghosts, and his worldwide trek drinking from the cups of Jews in apprehension of the coming of the Messiah, will be the final Jewish ghost, and with his appearance all will be revealed. Until then we will are forced to resort to one of the most holy of Jewish sources, Woody Allen; ‘There is no question that there is an unseen world. The problem is, how far is it from midtown, and how late is it open ? Unexplainable events occur constantly. One man will see a spirit. Another will hear voices, A third will wake up and find himself running in the Preakness... What is behind these experiences ? Or in front of them, for that matter? Is it true that some men can foresee the future or communicate with ghosts? And after death is it still possible to take showers?’
Comments (0)

Greg just meme-tagged[1] me! The unwavering rules of the internet now compel me to reveal five things you didn't know about me.

Here we go!

1. In the late 80's, my family went on a European vacation. Just prior to leaving, I was in the process of becoming a cool guitarist (pre-Guitar Hero days). Of course, my interpretation of cool was slightly different to other preteens. Cool for me involved making my own guitar pickup, which involved wrapping very fine wire around little ferrite cores and encasing them in hand molded plastic. The resulting pickup was a misshapen blob with wires dangling out of it. For some reason it got stuck to the bottom of my bag and traveled around the world with me, unnoticed at any of the previous security checkpoints that we passed through on our journey.
   
   At Heathrow Airport security was stepped up, as a few months earlier the Lockerbie Bombing occurred. (Of course, in the 1980's high-security at an airport post a successful terrorist attack was mild compared to the mess I recently had to go through after the foiling of the speculated liquid bomb attack a few month ago...) When my device of questionable intent was discovered by an X-ray inspection, my entire family was quickly surrounded by soldiers wielding M-16s. Upon manually searching my bag, they found my plastic explosive and a triggering device. The triggering device was a very neat Casio PB-700 calculator that my family bought me in Hong Kong a few weeks earlier. Being a nerd, by the time we arrived in London, I was half way through taking it apart. It was easy to understand how my innocent guitar pickup and disassembled calculator looked rather nefarious.
   
   After a damn stringent examination of all of our possessions and individual interviews we were allowed on the plane (which I single-handedly delayed by a few hours) to continue on our flight to Paris. My parents weren't too happy about this. The other passengers on the flight were fuming. The staff on the plane refused to serve us. Meanwhile, I was pissed that they took my calculator.
   
   
2. Needless to say, my family wasn't happy when, the following week, I demagnetized our boarding passes at Charles De Gaulle.
   
   
3. In Year 9 of school (or, as you Americans call it, Middle Junior High Sophomore?!), I was placed in the remedial math class. I had just transferred schools and I clearly didn't do so well on my placement tests. What struck me as peculiar was that rather than giving the role of remedial math teacher to someone who could impart math knowledge, instead the school had our rowing coach teach the class. What I didn't learn about simultaneous equations was more than compensated for by my practical education in how to smoke in class without getting caught. I did manage to do well enough on the semester quiz to get placed in the advanced course, but by the time I graduated I was back to being a failure of the university system. My first exam, which happened to fall on my 18th birthday, ended in a dramatic failure when for the long essay portion the only words I wrote in the answer booklet were "Schwann Cells." Despite my 51% grade average, it took me 5 more years to realize that I didn't actually want to be a doctor.
   
   
4. At an age when I was old enough to know better, but young enough to be foolish, I was caught hacking into a phone system. Somehow I got out of that one without charges being laid. *Phew*
   
   
5. I despise musical theatre. I am more than happy to sit through a 12 hour production of Angels In America, but I walked out of Rent ASAP.
   

---

[1] Wow. meme-tag, that's Web 2.0++
Comments (0)

Comments (0)