The Indian Economy Blog

November 17, 2005

Don’t Get Fooled By Success

Filed under: Capital markets,Miscellaneous — Amit Varma @ 12:52 pm

Consider this experiment: you are a punter, and you get a letter from me saying that I have the ability to forecast the result of all cricket matches, and if you pay me Rs. 10,000, I shall tell you what will happen in the next game so that you can make a killing — perhaps in crores — by betting on it. I invite you to pay me Rs 10,000 for the results of the next game. I promise you that I shall return the money if my prediction turns out to be incorrect. We work out an online mechanism by which you can cancel the payment right after the game if you wish to.

I get it right. I make the same offer regarding the next game. You pay me Rs 10,000 more, and this time make a modest bet with your bookie on the result I predict. I get it right again. You pay me my fee for the third game as well, and bet an even bigger amount with the bookie. I get it right again. You make a killing. I up my fee to Rs 1 lakh per game. You accept, having already made far more than that.

So are you a schmuck for having done all this?

Yes.

See, you don’t know what I did in the meantime. Here’s what I did: I wrote to 1000 punters making the same offer to them. They all accepted. I told 500 of them that Side A would win and 500 of them that Side B would win. Side A won. I returned the money of 500 people.

Then, regarding the next game, I told 250 of the 500 people that Side A would win and the others that Side B would win. Whichever side won, I returned the money of 250 people. That still left 250 people who were beginning to trust me by now.

Regarding the third game, well, you get the drift. 125 this, 125 that. After the game, 125 people, one of whom is you, trust me enough to raise my fees substantially. Not that it matters — I have already made Rs 8,750,000.

And here’s what will happen in future: even if I get my next prediction wrong, you will still look at my record, see that I’ve got 75% of my results right, and reason that my success rate justifies your continuing with me. By the time enough people see through me, I’ve made tons of money and can retire and go buy that yacht. Hell, many yachts. And I can even start the jhol all over again.

What I just described above is a typical illustration of the survivorship bias. It is the reason, for example, many mutual funds appear to have impressive records over the past few years: we ignore the possibility that they might just have got lucky, and we do not take the many failed mutual funds into account, just as you didn’t take into account — because you didn’t know about them — the many predictions that I made incorrectly.

That is why we should not be impressed by the 28% of managed funds that outperform the Sensex over a five-year term, as Gautam Chikermane points out in yet other fine article in the Indian Express. I must admit, though, that it doesn’t explain Warren Buffet. That man is something else.

26 Comments »

  1. You’re right about the survivorship bias. However, that does not imply that all non-Buffett managers just get lucky. In your example, what you’re essentially doing is improving your track record by eliminating all your “losing” bets in the next round. Were you to manage a portfolio properly, you would do the same thing – make several bets across market sectors with strict stop loss limits. When some of your bets go wrong, you take the losses on them and they fall out of the portfolio, The ones that are profitable, you let them run. It’s not difficult to outperform the index in this manner when the market’s broadly doing well. The trouble comes when all markets move adversely at the same time (from a global perspective, as in the 1987 market crash or in 1998 when Russia defaulted on their local currency debt). Then, ALL your bets lose money at the same time – which is why several hedge funds went bust at that time. Even Warren Buffett can’t do anything about those situations.

    So, point taken on evaluating funds – only the best survive and they may have been lucky; but I would also say that they’re likely to have been better managed than the ones that went bust. No, it doesn’t mean that they had greater insight into the market, just that they managed the portfolio better.

    Comment by Aniruddh Gupta — November 17, 2005 @ 1:27 pm

  2. Good points, Aniruddh.

    I’m not implying, though, that all non-Buffet managers get lucky, but just that it is fallacious to assume that just a company is successful, it is necessarily good, or that it can be successful tomorrow.

    Also, to take this analogy further in a simplistic way, if we look at strings of predictions and and say that “right-right-wrong”, “right-wrong-null” and “right-right-right” represent the decision paths of three funds, and only the third survives, it is wrong to assume that they will continue to get it “right.” Rather simplistic, and it’s obviously much more complex and nuanced than that, but you get the point.

    Comment by Amit Varma — November 17, 2005 @ 1:47 pm

  3. Sure, historical returns are never a guarantor of future performance – anywhere. Can’t take them as a predictor, but if you had to put money somewhere, and didn’t want to manage it yourself, wouldn’t you put it in a fund with a better track record? You know that they could fail, but that probability is not greater than the probability of success anyway. And perhaps, they’re less likely to fail – not more likely to succeed – than other funds because they know when to take losses.

    Anyways, good post!

    Comment by Aniruddh Gupta — November 17, 2005 @ 2:24 pm

  4. Hi Amit
    Your analogy is not perfect – but could be. Suppose you start a mutual fund company with 32 funds and you manage all of them. In 16 of them you make risky stock purchases and in the other 16 you sell the same stocks short. 16 will do well and the other 16 will tank. You proceed for 5 years until you come up with a single mutual fund with an outstanding return over 5 years.

    Two questions: (1)is that illegal? and (2)should it be? Obviously some information is missing here. You should know about all of the mutual funds that the company managed. Obviously, in the above case, the average return on all funds managed by the firm will be mediocre. This is what is relevant.

    But maybe purposefully “cooking up” a sexy fund in this way should be illegal (and maybe it is).

    Obviously, any big mutual fund company like Fidelity or Vanguard (US funds of course) will have so many funds that some will be amazing winners and some will be amazing losers by chance. But it isn’t clear to me that the money Fidelity makes on its winners by attracting new customers exceeds the money they lose on the losers from subtracting customers.

    Comment by Michael H. — November 17, 2005 @ 9:51 pm

  5. Let’s work a bit with the analogy here. You have no prior record and you want to coerce 1000 people into investing on a “prediction” that might or might not be true. That’s the tough part, and if you are a good enough salesman to make that pitch, you deserve to be a survivor.

    Also, right after your first “prediction”, half of your client base will never look at you again – ever!. There is a factor of influence and knowledged that has to be suppresed to get to the second phase of that scheme. You might have to slit a few throats to make that happen ;)

    It gets even tougher with the third round. And only worse after that.

    It’s not as easy as it sounds. Don’t believe me, try it!

    Comment by Vijay — November 18, 2005 @ 12:05 am

  6. Amit:

    Your bookie example doesn’t illustrate a survivor bias, but it illustrates a deviant of an ponzi scheme. In most countries, if this scheme you described in followed by a “registered” financial advisor, it is considered as a fraud and that advisor can be prosecuted.

    A survivor bias is a statistical concept that should be considered when analyzing populations, not individual performance (as you have mentioned). For example, if a place has ten people, and five of the ten people die at the age of, say, 20 and the present age of the rest (who are still alive) is, say 80, what is the average age of the population? If the answer is 80 years (based on 5 living people), then the sample is said to have survivor bias. The same logic applies to mutual funds.

    When you evaluate a mutual fund’s performance, it is possible for an analyst to determine the source of a fund’s returns – returns due to: asset allocation, security selection, market timing, etc. Based on this analysis, one can conclude if it is the skill of a manager that is adding alpha (excess returns over benchmark) or if it is pure luck. If a mutual fund is consistently outperforming a benchmark for five years, then the odds are it is NOT due to luck.

    Also, your comment “That is why we should not be impressed by the 28% of managed funds that outperform the Sensex over a five-year term …” is NOT an example of survivor bias. In this case, the non-survivors DO NOT matter. The author of that statement is just talking about the percentage of funds that OUTPERFORMED the market (the odds of this data having survivor bias is very low). On the other hand, if the author had said “the returns for managed funds were 28% during the period of last, say, 5 years, then there is a possibility that there is survivor bias that is resulting in an overstatement of the returns”.

    Hope this helps.

    -Madan.

    Comment by Madan Manoharan — November 18, 2005 @ 12:44 am

  7. Vijay, Madan, thanks for your comments. Don’t take my analogy too literally. My forecaster and his punts can be loosely considered equivalent to, as Michael said in his comment, a mutual fund company and the many strategies its many funds pursue. In that sense, a surviving fund, that happens to have a good five-year record, could simply be the one that got three predictions right, while all the other funds of the same company would be the ones that got one wrong somewhere along the way. Just as, in my analogy, I haven’t delivered a higher level of service to the punter who won three bets because of me than to the ones who lost money because of me, the surviving fund need not be better than the non-surviving ones.

    Madan, the analysis of the source of a fund’s returns are (obviously) done purely in hindsight. If someone happened to get his market timing right, obviously his skills at market timing will appear to be excellent. But are they really? That is a subjective call, and my feeling is what you want to believe about the fund in question might play a part in that analysis.

    I am not saying, of course, that all funds that perform well are lucky. Some might indeed be superior. But their success rate alone does not imply that.

    Comment by Amit Varma — November 18, 2005 @ 1:16 am

  8. Amit:

    You are correct in saying “But their success rate alone does not imply [they are due to manager's skills]“.

    If you are interested in this subject, I would recommend that you read Prof. Sharpe’s (yes, that same Nobel Laureate) work on style analysis. Here is the URL:
    http://www.stanford.edu/~wfsharpe/art/sa/sa.htm

    As a side note, regarding the Indian Express article you have referred to:
    There is a lot of misunderstanding about EMH; the author of that article is also sucked in that hole! EMH (in all three forms), according to Fama, is about informational efficiency, not valuation efficiency. On the other hand, when the behaviorists (Thayer, Shleifer, Kehnaman, and the rest of the gang) talk about efficiency, they are talking about valuation. Lot of people confuse between the two, and incorrectly argue their positions. The real debate between the EMH gang and the behaviorists is weather informational efficiency equals valuation efficiency.

    -Madan.

    Comment by Madan Manoharan — November 18, 2005 @ 2:00 am

  9. A few corrections (typO) in my previous posting:

    (1) “Thayer” should be “Thaler”;
    (2) “Kehnaman” should be “Kahneman”; and
    (3) “weather” (last sentence) should be “whether”.

    -Madan.

    Comment by Madan Manoharan — November 18, 2005 @ 2:18 am

  10. Madan’s point about the difference between information efficiency and valuation efficiency is fascinating. Let me see if I have understood this correctly:
    Indeed the fact that there exist funds or individuals who outperform the market consistently shows that the two efficiencies are NOT equal. Information dissemination may be 100% efficient, but prices need not reflect that since differences in behaviour based on that information may produce a systematic deviation from the intrinsic value of stocks. It is precisely this difference which talented fund managers can put to use (this IS in fact their talent – making “more” rational changes based on information available)
    However it appears that this difference is not significant, since the article says that managed funds have not consistently outperformed the index by a wide margin.
    The fact that 28% of managed funds have outperformed the index may not be impressive, but the performance of those that did definitely is impressive, and it should be easy to follow that it becomes more so with time.
    I found this very useful. It was the first time I read anything on this subject, and the last half an hour of trying to figure out stuff was enormously satisfying. Thanks Amit and Madan!

    Comment by sumeet — November 18, 2005 @ 4:29 am

  11. The question that comes to mind now is: Why is the index a comparative benchmark? Does the index somehow have greater valuation efficiency? Aren’t index components equally susceptible to valuation inefficiencies?

    Comment by sumeet — November 18, 2005 @ 4:36 am

  12. The index is ‘managed’ so that the dogs are let go everytime they look like dragging the rest of them down. The Dow Jones is rebalanced every year if I am not mistaken.

    You guys can talk all you can about theories but if you do want to win big and consistently, get on the right side of certain people. Senators and inside information!!! Search for ‘senator frist stock’ or ‘cheney halliburton stock options’. The price you pay is not rational but completely managed. The stock price of say XXXXX which may fluctuate between 900-1200 for two months is forced to be there by certain vested interests, who remain invisible. They may look at the derivatives i.e puts and calls, and say, 13000 people have made a bet that the price will be 1100, and 4000 people have bet that it will be 1000, on 1 Dec 2006 so let’s get it closer to 1000 but above 1000. This makes sure those who bet 1100 will lose, and those who bet 1000 would win but not by a large amount. This is the classic put/call flush. Most manipulation occurs due to the insane way the derivates are allowed to operate. Common people can’t leverage insane amounts and not be forced to cover their bets. BUT, big big corporations who do make big bets which are insane are allowed to esacpe, otherwise they will ruin not just that company but the economy of the host country, and possibly roil the global markets. Witness JP Morgan who shorted gold when in the early 1990′s gold was going down and down. Now that gold is back up their liabilities are in the trillions (last I checked it was $973 billion in assets vs $46.61 TRILLION in liabilities)

    scroll to pg 17-18 in this report from the “U.S. Office of the Comptroller of the gold”
    http://www.occ.treas.gov/ftp/deriv/dq205.pdf

    Shorting a stock is healthy (if anybody wants an explanation I will post here tomorrow) but playing derivatives is insane, and I wonder how many know about it.

    Comment by Amit Kulkarni — November 18, 2005 @ 5:47 am

  13. Hi Amit (Verma): Sorry to get off the main topic but one little tidbit about Mr. Buffet. In his advancing years he has taken a political left turn. He has supported tax increases, retaining the estate taxes (while is wealth is protected in tax shelter trusts) and social welfare programs.

    Why is it that men like Warren Buffet and Bill Gates have a desire to be liked by the leftists (who are predominant in the media and academia) and take to populist gesturing? Isn’t it enough that they have produced wealth and employment for so many through their vision and industry?

    But you are right, Mr Buffet is the “sage” of investing.

    Comment by Vivek G — November 18, 2005 @ 6:54 am

  14. Madan, Amit (K), Sumeet, Vijay, thanks for your quality comments, I’ve enjoyed this discussion and learnt from it.

    Comment by Amit Varma — November 18, 2005 @ 4:57 pm

  15. Amit K:

    An axe can be used to chop wood or to kill people; those who think an axe should be banned because it was used to kill do not understand the purpose of the axe.

    I don’t know what your qualms against “derivatives” are (looks like you are talking about stock options, and commodity futures – interpreted based on your stock and gold comments –, however I am sure you are aware about other types of derivatives), but I will be more than happy to understand your position and debate you on the merits of derivatives. However, let us not do it in this forum. Please suggest a place where we can have this conversation.

    Amit V: I apologize for this post.

    -Madan.

    Comment by Madan Manoharan — November 18, 2005 @ 10:02 pm

  16. Madan, good point well stated, and no need to apologize. You guys are welcome to have that debate here if you wish.

    Comment by Amit Varma — November 18, 2005 @ 10:49 pm

  17. Madan,

    I agree a debate is not relevant to this post. I will try and post something on my blog within the next 10 days, how do I contact you to join the debate? You don’t have your email id posted here, mine is a-m-i-t-k-u-l-z at Y-a-h-o-o (remove the dashes please), please just send a blank email so I will update you when I have posted something. Or I can post a comment on this particular blogpost?

    My argument is against pure speculation indexes like the Dow Jones, S&P 500, which carry no meaning to me, that axe is being used to chop people, most of whom have no clue they are being handed out a death of thousand cuts. NOT currency futures (rel. to a nation’s economy) or commodity futures (rel. to farmers) which are useful devices. I also like the concept of naked shorting of stocks which play a huge role in case of market collapses.

    To the moderators,
    Please enable sending an email to those who give their email (when posting comments here) so that I know when somebody comments here.

    Thanks

    Comment by Amit Kulkarni — November 18, 2005 @ 11:03 pm

  18. It’s an interesting topic. Thanks for your perspectives as well Amit. I appreciate it.

    Comment by Vijay — November 18, 2005 @ 11:09 pm

  19. I had to sit down and spend a straight hour before I grasped the logic behind the debate! [slow me indeed]. I guess this concept is similar to the phenomenon of outliers in regression analysis.

    I have limited knowledge of financial markets, so please consider this as a disclaimer if my query is kinda dumb.

    If I understand correctly, and if survivor bias exists, then that means that the poorly performing funds continue to perform poorly over time and therefore, they eventually disappear.

    If this is true, then we should be able to predict if a fund is going to perform poorly, because it will consistently show poor performance.Then does that mean we can predict negative trends but not positive ones?

    Comment by Ram — November 19, 2005 @ 4:16 am

  20. Amit — interesting post. It reminded me of a book I read a few months back: Fooled By Randomness by Nassim Nicholas. It’s a quick read. The guy has an axe to grind, but it’s interesting stuff nonetheless.

    Comment by Sahil Tandon — November 19, 2005 @ 6:48 am

  21. Thanks Sahil, I’ll look out for it.

    Ram, there’s nothing in my post which brings one to the conclusion that one can predict if a fund will perform poorly. The survivorship bias deals purely with how some people view funds, or whatever. It says nothing about how funds will actually perform.

    Comment by Amit Varma — November 19, 2005 @ 11:47 am

  22. The single best book on the random nature of stock markets is A Random Walk Down Wall Street — Burton Malkiel.

    During one of my past avatars, I had to review the results of about every top fixed-income money manager in the US (we were selecting about 10 money managers out of hundreds)….survivorship bias is a very real issue…. managers will show the performance records of funds and for those time periods during which they’ve outperformed the index. Became like Lake Wobegon… everyone claimed to be above average

    Comment by Prashant Kothari — November 19, 2005 @ 10:09 pm

  23. Thanks for the clarification Amit. My mistake in not being clear – I was reading the article forwarded by Sumeet that referred to Eugene Fama and his studies and got off on a tagent.

    However, between now and then, I managed to find this interview with Fama where he kind of addresses my doubt.

    Question:…the predictability of future investment success based on past success…

    “One of my students just finished his thesis on that subject, actually. What he found was that performance does repeat when it’s on the negative end! In other works, funds that do poorly, tend to do poorly persistently.

    Question : Why couldn’t one postulate that the same would be true at the other end of the spectrum?

    One could postulate it, but it doesn’t seem to be true. On the negative end of the spectrum, you have things like turnover and fees and all that kind of stuff, which can explain why you have negative persistence in poor returns.”

    Intriguing stuff. Over and out.

    Comment by Ram — November 20, 2005 @ 12:17 am

  24. Fooled by success

    Amit Varma is on form at the Indian Economy blog:Consider this experiment: you are a punter, and you get a letter from me saying that I have the ability to forecast the result of all cricket matches, and if you pay me Rs. 10,000, I shall tell you what …

    Trackback by PSD Blog - The World Bank Group - Private Sector Development — November 21, 2005 @ 5:50 pm

  25. I invite interested people to visit a debate on derivatives with Madan on my blog…

    FINANCE: The derivatives bubble

    Thanks

    Comment by Amit Kulkarni — November 27, 2005 @ 12:13 am

  26. Hi

    Talking about Indian Stock Markets and a Ponzi Scheme like the one suggested at the beginning of the thread is what fascinates me. I am analysing stocks too and feel good that I am not in that category of people who fool 50% of the people.

    After few days of being a aggressive thread this suddenly simmered down and there has been no post on it since 27th Nov 05. May be this post may trigger life into it again.

    Cheers

    Bharat Agarwalla

    Comment by Bharat Agarwalla — January 26, 2006 @ 4:43 pm

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by WP Hashcash

Powered by WordPress