>>> Posting number 2404, dated 17 Jul 1997 22:18:42 Date: Thu, 17 Jul 1997 22:18:42 -0500 Reply-To: Discussion of Fraud in Science Sender: Discussion of Fraud in Science Comments: Authenticated sender is From: Al Higgins Organization: Sociology Department UAlbany Subject: (Fwd) RE: (Fwd) Statisticus Ignoramus MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Here, after a delay, is a post from John Gardenier. I regret the delay. Al ++++++++++ In fact, Ted, Al, and others, the situation may not be much worse in psychology (horrifying though it is) than in many other areas of science. For the last hundred years, there has been some controversy in the statistics profession as to whether "hypothesis testing" is even conceptually valid. While most professionals support the concept, the standards of practice are continually tightening. Most scientists, and even some statisticians, seem to be unaware of the trend. There are at least two contemporaneous factors which exacerbate the problem. First, contrivers and purveyors of statistical software are tending to give less documentation than formerly on the algorithms and theoretical bases for the "statistical tests" included in their systems. At the same time, they tend increasingly to advertise that you do not need to have any understanding of statistics because the package does it all for you. This is "snake oil" of the worst kind, but it seems to sell. Second, many statistical textbooks, especially introductory ones, are at best difficult to understand and, at worst, flat wrong in their descriptions of statistical theory. In part, this seems to result from a desire by academics to use textbooks of recent vintage, regardless of the age of the actual material contained in them. The easiest way to do that is to take a few older textbooks, repeat their content with totally different wording and problems, and some difference in organization (can't have plagiarism, of course.) Errors of understanding get perpetuated, and the growing sophistication among the leading professionals is seldom reflected in a "Stat 101" type book or even in some more advanced texts. Finally, many textbooks are written by people who take pride in NOT being professional statisticians. Their credibility lies in their being a member of the X discipline, writing a text labeled "Statistical Applications in X." Nothing wrong with that if they really understand modern statistical theory and how it applies to X, but some authors do not. Other potential problems include: * failure to understand that statistics, like any other branch of mathematics, is a self-referent, algorithmic system. That means that it "works" equally well, at some level, whether or not there is any congruity whatsoever between observed patterns of real world phenomena and the math. There is both science and art in achieving truly useful congruity. * that statistics only "works" some part of the time. It is absolutely dependent on use of random inputs. This GUARANTEES that over some moderate-to-large number of applications of statistical theory, apparent statistical significance will appear purely by chance. It can be prohibitively expensive to drive the chance of error down to miniscule levels. Even then, extreme care in avoiding one type of error typically increases the chances of another type of error. This is why I, along with many other statisticians, stress the importance of "practical," as well as "statistical," significance. * that statistical analysis is only possible within the context of sample/experiment DESIGN. Too often, researchers only call in a statistical associate after they have collected data that cannot be adequately analyzed by any methodology whatsoever. * that statistical analysis can be extremely dependent on assumptions about the data and about the theory in the application discipline. Sometimes, theory in the application discipline is very weakly founded. (Feynman, among others, has discussed this problem in the context of physics.) The best possible application of statistical theory and practice cannot reliably save you from error arising from faulty theory in your own field. Of course, if you are deliberately questioning elements of theory in your own field, competent statistical analysis may be quite helpful. The danger arises when you do NOT question both the theory in your discipline and the relevance of the particular statistical methods used. * that, sadly, peer reviewers of a disciplinary paper often feel inadequate to address the statistical issues. They may all assume that "somebody else" will handle that. Often, nobody does. If fact, often, the text of a paper does not contain a description of the data and analytic procedures adequate for an assessment of statistical quality. * finally, the complex array of statistical (and other) analytic methods available may lead some researchers to fiddle with the analyses until they achieve the RESULTS they expect and want. Feigenbaum and Levy, in "The Technological Obsolescence of Scientific Fraud," have suggested that this, unlike FF&P, is a low-risk strategy because it is virtually impossible to prove how the cheating occurred or that it was done with malice aforethought. Generally, statisticians have an ethic of committing to a methodology before the analysis and living with the results afterwards, regardless of whether those results are good, bad, or indifferent. This ethic does not preclude "data mining," (searching for patterns and relevance to be revealed by the data themselves, rather than based on any a priori hypothesis.) Such data mining has to have its own means of control against self-deception or fraud. Extreme skepticism regarding the statistical support for scientific claims is not a vice, but a virtue. John Gardenier "May your results have practical, as well as (apparent) statistical, significance." __________________________________________________ A. C. Higgins ach13@cnsvax.albany.edu College of Arts and Sciences VOX: 518-442-4678 Sociology Department FAX: 518-442-4936 University At Albany Albany, NY, USA 12222