Fooled by Randomness [book]

 

In many ways this is better than Taleb’s more popular book, The Black Swan, which I’ve already noted is one of my favorite books of all time.  He started it before he was famous (before he knew it would sell), so he tried harder while writing it and it shows. The second edition (which is a third larger and better edited) is well-organized and easy-to-read, unlike Black Swan which sometimes can meander.

His central thesis is the same in both books: humans are hard-wired to demand an explanation for everything, even when there is none, leading us to be fooled into thinking that something is not random when it really is.

Much of this is easily demonstrated mathematically. My favorite example is the game where an evil investment advisor sends letters to a thousand or so prospective clients, telling half that a particular stock will go up and the other half that it will go down. The second month, repeat the same mailing to the half of the list where the prediction happened to be correct. Keep repeating each month and at the end of ten months he'll have a (short) list of people who think he was correct for ten months in a row. Apply the same math to the world's pool of actual investment advisors and mutual fund managers and you'll find that the number of people with 10 year successful track records is about what you'd expect from pure chance!

Ergodicity, the statistical concept behind the law of large numbers, says that the true properties of a process become clear only after many iterations.  The problem in life, says Taleb, is that “winners” (whether successful investment advisors or CEOs) are often there because of the survivorship bias:  since they’re graded on results, not process, many “successful” people are in their positions because of luck not skill.  If your boss thinks he’s so smart, strip him of his current position and force him to do something new without relying on his resume (which may be luck anyway).  Then you’ll see how smart he really is.

The danger of Taleb's thesis -- and one I haven't fully navigated around yet -- is that you'll succumb to fatalism. If everything is random, then why bother working hard?

Some of his suggestions are good ones, such as the idea of Buridan's Donkey: a donkey equidistant from food and water will starve unless you give him a slight nudge toward one or the other. In this case randomness is your friend, because it provides just enough of a push to get you off your duff and doing something, regardless of what it is.

You can't blame all success on luck. The investment advisor who was smart enough and worked hard enough to notice fraud in the company's annual report: I believe he'll be more successful long-term than the one who simply flipped coins.

As in the Black Swan, the weak parts of Taleb’s case are exposed in his uninformed discussion of the QWERTY problem, or so-called "path independence" : the idea that once a particular idea or product takes off, network effects kick in to give it an insurmountable advantage, even if the original is provably inferior to alternatives. Path independence theory has been refuted to my satisfaction by the works of Liebowitz and Margolis, who showed that most (all?) of the so-called examples of this are wrong, that in fact network effects are pretty good at giving the advantage to the best technology, not the worst.

As I get older, I become more humbled at how little you can trust even the smartest analysis for why and how things turn out the way they do. Newspapers and TV are so often wrong, especially on their first reports, that I wonder why we bother to pay attention at all.   I am intrigued by the Taleb style of reasoning that says randomness is at the heart of everything we do.  The trick is to internalize that fact, yet keep trying in spite it.

Greenspan Recants (or did he?)

The current financial crisis is an interesting real-life experiment to free market fans like me.  It’s far too early to draw conclusions, but much of the generally accepted wisdom right now is that “pure capitalism” has failed and that the world needs more regulated markets to protect against “excess” and prevent or lessen these crises in the future.

But my belief in free markets is more about information than it is about money.  Most regulations involve one person (a bureaucrat) telling another person (you or me) what to do.  Sometimes that’s necessary:  for example we’re all better off when a central planner decides that everyone must drive on the right-hand side of the road.  But for this to be effective, bureaucrats need to be humble about their terrible disadvantage in information.  No regulator can possibly know as much as you do about your individual circumstances, and this information disparity is the source of why regulation often causes more harm than help.

Those of us who believe this idea have long thought of Former Federal Reserve Chairman Alan Greenspan as an ally, so I was intrigued when I saw news reports (now repeated over and over) that he “changed his mind”.

Except, he didn't. When I look at the actual transcripts, it's clear to me that careless news reporters have simply spun his remarks incorrectly.  Here's the key quote:

Chairman WAXMAN. Well, where did you make a mistake then?

Mr. GREENSPAN. I made a mistake in presuming that the self-interest of organizations, specifically banks and others, were such is that they were best capable of protecting their own shareholders and their equity in the firms.

And it's been my experience, having worked both as a regulator for 18 years and similar quantities, in the private sector, especially, 10 years at a major international bank, that the loan officers of those institutions knew far more about the risks involved and the people to whom they lent money, than I saw even our best regulators at the Fed capable of doing. So the problem here is something which looked to be a very solid edifice, and, indeed, a critical pillar to market competition and free markets, did break down. And I think that, as I said, shocked me. I still do not fuIly understand why it happened and, obviously, to the extent that f figure out where it happened and why, I will change my views. If the facts change, I will change.

What follows is his attempt to defend himself against the implication that somehow he was derelict at his duties as Fed Chairman, that ideology got in the way of his performing his legal responsibilities. As someone who obviously cares about integrity, much of Greenspan's testimony is simply responding to those charges, which he thinks are unfair.

But he eventually reaches his bottom line:

Mr. GREENSPAN. I think that it's interesting to observe that we find failures of regulation all the time, and one of the reasons is a very significant amount of regulation in the economic area is based on a forecast to know in advance whether or not particular products will go bad or the cycle will turn. If we are right 60 percent of the time in forecasting, we're doing exceptionally well. That means we are wrong 40 percent of the time, and when you observe the extent of the broad failure, the difficulty is that nobody can forecast. If you try to take a look at what the private sector does it's precisely the same thing that goes on in government. We at the Federal Reserve had a much better record forecasting than the private sector, but we were wrong quite a good deal of the time and that is reflected in how one views what the appropriate regulatory authorities are because unless you can anticipate the types of problems that are going to happen, it's very difficult to know what to do. And I think that's the problem that this type of thing confronts and I don't see any way in which that's going to be fundamentally changed. We can try to do better, but forecasting is never--never gets to the point where it's 100 percent accurate.

When you look at his entire testimony, it's clear that Greenspan in no way has changed his mind or even modified his fundamental pro-market position. He maintains his strong support for an unregulated market in derivatives, for example, which "are working well" (his quote). The part he doesn't understand (where he'll change his mind if the facts change) relates to credit default swaps--a market that barely existed when Greenspan was chairman and therefore was impossible to forecast.  And who would we be regulating anyway?   He reminds us of this:

"We are not dealing with people who are dumb. We are dealing with, by far, the most sophisticated, thoughtful people about the way markets work who created the major problems".

Enacting regulations to prevent a future collapse of the credit default swaps market would be a waste of time, since (as Greenspan notes) nobody is interested in that market anymore now anyway. But enacting other regulations would simply hamper a market that is already working well 60% of the time. Does anybody think they can get better odds on an alternative system? Exactly which regulations would you enact that would improve upon that record? Greenspan doesn't recant: he just admits he doesn't know.  Do his regulation-happy opponents or the careless reporters know?   Do you?

Maxwell wins

Many of us have been watching the shifting return counts since election night, but now it’s official:  Marcie Maxwell won the election for 41st district state representative over Steve Litzow.  The margin was extremely close:  748 out of 64,394 votes cast, much tighter than the 1591 votes that separated the two of them during the primary election.

I think she won for two reasons:

1. She’s not George Bush.   In a district that Obama carried by 64%,  anybody with a “D” in their name was guaranteed at least 748 voters who wanted their state legislator to end the war in Iraq, end subsidies to Big Oil, end all that corruption in Washington D.C., stop the Born Again Fundamentalists from running the government, help those poor Katrina victims, and make abortion legal again.  If you can fog up a mirror you can vote in this country, and this year the momentum was on the side of Democrats.
   
2. Steve went negative.  Too many people were put off by the petty, irrelevant attack mails that Steve sent, implying that Marcie somehow doesn’t care about student privacy.  Or something – I’m not even sure what he was implying about her.  Anyone who knows Marcie personally (or knows somebody who does – which is half the city of Renton) looked at his ads and rolled their eyes.  Same thing with his comments about Renton school performance; there was a right way to legitimately bring this up as an issue, but Steve came across as somebody who was just picking on the good people of Renton.

The full downloadable results will be ready in early December, and I can’t wait to pour through the numbers so I can update the analysis from the August primary.  Here’s what I’ll be looking for:

• Did Steve carry Bellevue?  I bet he did.  That would be interesting because it challenges the popular explanation that the Eastside is becoming more Democratic.  In other words, it’s possible (even likely) for a Republican to win District 41, even in a terrible year.
• Did education voters make the difference?  Since this is Marcie’s main campaign theme, it will be interesting to see how much the voters agreed.  This didn’t matter much in the primary, so I want to see if anything changed in the larger turnout for the final election.

Meanwhile, Marcie deserves congratulations, and I’m proud to stand behind her as my legislator for the next two years  (Argh, is that all it is?!  After nearly a year of hard work campaigning, the prize is a lot more of those 2-hour drives to Olympia, a diddly state legislator’s salary,  and then you have to do it all over again!

Civics test: 90%

Ouch! I’m disappointed that I got three answers wrong on a new 33-question civics test created by the Intercollegiate Studies Institute, giving me a score of 90.1%.

Still, that’s better than 49% mean score of the 2500 people test-takers nationwide. In fact, it’s way better than every single category sampled:

Group	Score
Income > $100K	55
Democrat	45
Republican	55
Age 18-24	47
Age 45-64	52
Doctorate	72
Undergraduate	57

The study's authors somehow convinced 164 elected officials to take the test, and their average score was 44% – five points lower than the general public.

These questions are pretty basic: multiple choice answers to questions about the First Amendment, the purpose of the Federal Reserve Bank, names for the three branches of government. Yes, some of them were a little tricky, but I don't think you can be an informed voter if you don't know this stuff.

Take the 5-minute test yourself and let me know how you did:

http://www.americancivicliteracy.org/resources/quiz.aspx

Cheap Gas

What’s going on with gas prices?  Look what I just paid at a 76 station right off the freeway near Ellensburg:

(That’s a ‘1’ at the beginning of the price, not a ‘2’)

Purple toes in Zillah

There are no wine lovers in the San Francisco Bay Area (or in Seattle for that matter) who haven’t been to Napa Valley.  But comparatively few people know about a  similar wine-growing region just two hours east of Mercer Island (over the soon-to-be-tolled I-90 bridge), in the Rattlesnake Hills area of the Yakima Valley.  We spent our weekend there, where some good friends are getting started with their own winery.

First, we picked some grapes:

then we crushed them:

and now look at the toes on my 6-year-old:

You should go too!  Best place to stay in Zillah is the Comfort Inn, for about $100/night, including a big breakfast, a pool, and free use of their grill!

Worst President Ever

I'm a big fan of Black Swan theory, which says that although leaders are measured by the Big Things that happen, they are rarely given credit for policies or actions that prevent Big Things from happening in the first place. Since it's impossible to wind back the clock and do the redo the experiment, we are left victims of the narrative fallacy, our desperate need to explain everything even when the real cause is unknowable, and to assign credit or blame to people (like the President) who have very little real influence compared to the collective, independent actions of the rest of us.

Think how history would look back on the past eight years if we had a President who had done this:

• Instead of focusing on the economy, he wasted his first year in office on a quixotic push to pass intrusive Homeland Security laws like the one forcing airlines to put locks on cockpit doors.  Experts agree this had no effect on safety, as shown by the complete lack of hijackings since.
• Second year in office, he forced burdensome new regulations on business transparency that devastated many innocent companies, forcing needless management changes at great American companies like Enron, Worldcom, Tyco, and others.  It’s a miracle these companies still exist.
• Iraq War: rather than leave the task to international inspectors, he wasted a trillion dollars removing a dictator who experts agree presented no serious long-term threat.
• New Orleans levee protection: in a blatantly corrupt attempt to reward his cronies and win votes, he poured excessive federal money into a crash program to over-build flood levees when it’s now obvious that even during a large storm like Hurricane Katrina, the original levees would have held just fine.
• Millions of poor families remain stuck in rental housing because of his refusal to continue the wise policies of the 1990s that offered a chance at home ownership to everyone.  The economy is now paying the price, with stalled housing values and a stock market that barely rises past the inflation rate.

Nobody, not even the wisest expert out there, can predict the Biggest Thing that will happen to the next President.  But any pre-emptive thing he does will be judged unkindly by history precisely because the Biggest Disaster he prevents is the one that never happens.

Free coffee at Starbucks

I dropped in on West Mercer Elementary school this morning:

and then I went downtown and got one of these for free:

Incidentally, I disagree with the idea that irresponsible or uninformed people should vote.  If you’re unsure about a candidate or an issue, please don’t guess – you’ll likely just make things worse.

If you want to know what I think about issues, read these posts.  If I’m wrong about something, please leave comments so I can change my mind.

Sign Waving to Commuters

If you want to say something to Mercer Islanders, you can’t beat the intersection of Island Crest Way and 40th as a place to publicize your message.  A large percentage of the island drives past it every morning, and they’re your captive audience until the light turns green.  There must be an election or something this week, because here’s what we had this morning:

Can you spot the three City Council members in these photos?

I wonder how crowded it’ll be tomorrow morning?

Litzow vs. Maxwell at the PTA

The Mercer Island Legislative Team of the PTA sponsored a discussion recently between the Washington State 41st district legislative candidates, Democrat Marcie Maxwell and Republican Steve Litzow. It’s getting close to Election Day and maybe with absentee ballots the decision has already been made.  But if like me you’re planning to vote in person on Tuesday,  here’s my bottom line on the differences between the candidates:

Steve cares most about freedom: let schools, teachers, parents, and taxpayers have maximum choice and flexibility.  Marcie cares most about fairness: don’t let the “rich” districts (like Mercer Island) forget that there are less well-off places in need.  Marcie knows more details about education issues, but Steve is less beholden to vested interests and more likely to bring real change – if you think that’s necessary.

The best example of this difference is school funding.  Mercer Islanders would spend even more on our schools if we could, but Washington is one of only two states that put a maximum cap on the amount you can raise in tax levies – even if 100% of the voters in a district beg for it.  How silly is that?

Steve: would eliminate the levy lid if he could, but recognizes it’s politically difficult, so he supports a compromise that involves raising the floor on funds we get from the state.

Marcie: reminds us there are other districts out there that won’t support higher taxes for education the way we do, and it’s important not to let Mercer Island get too far ahead.

In other words, Steve wants people to be free to choose the level of funding they want, but Marcie worries that’ll lead to unfair advantages for the pro-education districts.

One nit: both candidates keep repeating the incorrect statistic that Washington scores 42nd in funding on education. That’s very old data; the actual number is 38th (as of 2006) and probably much higher by now thanks to the $2.46B added to teacher salaries in the past four years .

Both candidates say teacher strikes should be illegal, though neither gives specifics on what to do.  Steve notes that the unions seem to strike each year just before elections, and that we should expect another one two years from now, just before the next election. No suggestions for how to avoid it. Marcie repeats her union supporters’ statements about how “teachers are happiest in the classroom”  and that presumably the best way to ensure happiness is to pay them more.

But how should we pay them?  I saw a big, healthy difference:

Steve:  “I do believe it’s possible to tell the difference between good teachers and bad teachers” and the good ones should be rewarded.

Marcie: “It’s complicated”, so let’s focus on making existing teachers better, through things like National Board Certification or the Math Academy they tried in the Renton district.

Here Steve’s clearly right.  It’s ridiculous that we give the same raises to the worst 10% of teachers that we give to the top 1%.  That study people quote about National Board Certification is flawed because it didn’t distinguish between causation and causality: the certification doesn’t change the teacher. Marcie should read up on what Apple’s Steve Jobs says: today’s schools will never really improve until you fix the awful way we hire and compensate teachers.  Meanwhile, those mediocre teachers who want to keep their jobs should spend every waking minute of their day trying to ensure Marcie gets elected because she’s unlikely to propose anything new here.

Somebody asked Marcie why schools in her home district of Renton fare so poorly, and she replied with the important reminder that demographics are important: Renton is not like Mercer Island.  It’s a much bigger school district (100K students), with 44% of kids on government assisted lunch programs (in one school it’s as high as 73%).  Top-down mandates like No Child Left Behind offer little flexibility, which crushes the options for resource-constrained schools.

There is no question that Marcie is more aware of the day-to-day realities of poor districts, and that her legislative priorities are more focused on the immediate needs of the have-nots.  That’s why the Mercer Island School Board President says Steve Litzow doesn’t get it.  That may very well be true (he definitely doesn’t get it with his lame, idiotic ads implying Marcie doesn’t protect privacy—give me a break), but does it matter who “gets it” or does it matter who’s more effective?    Sure, Marcie cares about the poorest kids (so do we all) but an elected official shouldn’t be given an “A” just for effort.

One more thing: everyone blames the rising cost of education on mandates, so one question to ask is where are those mandates coming from, are they really necessary, and who is more likely to reduce them.  Marcie mentioned that we should reconsider some of those mandates (she specifically blames No Child Left Behind).  Still, it seems to me that most (all?) of the mandates come from the “fairness” people like her, not from the “freedom” people like Steve.   I don’t necessarily disagree with Marcie – I  mean, I like “fairness” too – but everything in life is a tradeoff and we can’t keep pretending that more funding is the only answer.

As for me, I’m following the lead of Surrounded by Water and voting for Steve.  Since I’m also voting for Fred Jarrett, and I know Marcie’s pretty much lockstep with Fred anyway, I think this is the best way to ensure healthy variety and flexibility in state government. 

Prius Gas Mileage

I’m having fun computing the long-term gas mileage for my 2007 Prius Touring Edition.  It’s now just under two years old, and here’s the chart for my cumulative performance:

Gas prices (blue line above) have plummeted so much lately that my fill-up last Friday ($2.499) was close to the lowest I’ve paid in two years.  The red line shows the most important number: how many miles I can travel on a dollar’s worth of gas.  Generally it hovers between 10 and 15 miles—at least double what we get from our other car (Honda Odyssey).