Linear Regression FAIL
First I want to say that this report from Deloitte is actually quite intriguing. I hate to grub around in analysis and nit pick finding fault with every little thing. But I just couldn’t let this go.
First, take 5 minutes and go read (or skim) this Deloitte report on cost and schedule overruns in aerospace and defense programs.
Okay, you’re back. Here we go…
OMG, what the heck is going on with the linear regression in Figure 1 on page 2?
If this analysis holds then prior to 1993 we should see that budgets on aerospace and defense programs were underrun. REALLY?
The thesis of this paper is actually quite good and it is unfortunate that such frankly awful analysis as this might cloud the message.
After working with large datasets and trying to find patterns and correlations(trust me, half of what experimental physicists do is look at data and try to figure out why their experiment is not working) this kind of thing really jumps out at you.
The problem is that rather than a whole bunch of independent data points what we are looking at is really two sets of data from radically different data sets. The data sets are Clinton era programs and Bush era programs. Shortly after Bush took office there was another event that radically changed the defense and aerospace program landscape, 9/11.
So, allow me to offer an alternate interpretation:
The lesson here should be that there is a very real danger in selecting small groupings of data and then drawing trend lines through them, saying R^2 looks good, and calling it a day. This is the kind of sloppy work that gives statistics a bad name.
All that was needed was to go farther back in time. The data for 1960 to present would have been sufficient. One interpretation is that this is an honest mistake and the analyst was being sloppy in not going back farther. The other interpretation is less charitable; that the analyst did not go back farther because it did not draw as alarming a picture and so was less suited to Deloitte’s message.
Either way, this is not a good piece of work.
Shame on you Deloitte, you’re better than this.