Applied statistics: Difference between revisions
imported>Nick Gardner |
mNo edit summary |
||
(11 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
{{subpages}} | {{subpages}} | ||
The term '''applied statistics''' is used both to denote the work of professional statisticians, who are responsible for the production and presentation of statistics, and to refer to the | The term '''applied statistics''' is used both to denote the work of professional statisticians, who are responsible for the production and presentation of statistics, and to refer to the processes of statistical interpretation that are undertaken by professional statisticians, by professional users of statistics and by the general public. It makes frequent use of [[statistics theory]] but it is distinct from the mathematics of that theory. It has made major contributions to scientific research and to the conduct of operational activities in a variety of fields. It has provided both a familiar source of information and a notorious source of misinformation. Misinterpretations of statistics by members of the medical and legal professions have led to false diagnoses and wrongful imprisonment, and ill-founded risk assessments were among the causes of the financial [[crash of 2008]] . | ||
==Overview: the basics == | ==Overview: the basics == | ||
Statistics are observations that are recorded in numerical form. It is essential to their successful handling to accept that statistics are not facts and therefore incontrovertible, but observations about facts and therefore fallible. The reliability of the information that they provide depends not only upon their successful interpretation, but also upon the accuracy with which the facts are observed and the extent to which they truly represent the subject matter of that information. An appreciation of the means by which statistics are collected is thus an essential part of the understanding of statistics and is at least as important as a familiarity with the tools that are used in its interpretation. | Statistics are observations that are recorded in numerical form. It is essential to their successful handling to accept that statistics are not facts and therefore incontrovertible, but observations about facts and therefore fallible. The reliability of the information that they provide depends not only upon their successful interpretation, but also upon the accuracy with which the facts are observed and the extent to which they truly represent the subject matter of that information. An appreciation of the means by which statistics are collected is thus an essential part of the understanding of statistics and is at least as important as a familiarity with the tools that are used in its interpretation. | ||
Line 10: | Line 9: | ||
==The Statistics profession== | ==The Statistics profession== | ||
The American Statistical Association defines a professional statistician as an expert in producing trustworthy data; analyzing data to make their meaning clear; and, drawing practical conclusions from data. | The American Statistical Association defines a professional statistician as an expert in producing trustworthy data; analyzing data to make their meaning clear; and, drawing practical conclusions from data. | ||
<ref>[http://www.amstat.org/careers/whatdostatisticiansdo.cfm ''What Do Statisticians Do?'', The careers center of the American Statistical Association]</ref> Professional statisticians are normally qualified in [[mathematics]] and are normally familiar with the underlying mathematics of [[statistics theory]] but their professional activity involves them in the use of the results that theory, rather than in the development of those results. They are engaged in activities that include the compilation of [[national accounts]] and the design, operation and analysis of:- | <ref>[http://www.amstat.org/careers/whatdostatisticiansdo.cfm ''What Do Statisticians Do?'', The careers center of the American Statistical Association]</ref> Professional statisticians are normally qualified in [[mathematics]] and are normally familiar with the underlying mathematics of [[statistics theory]] but their professional activity involves them in the use of the results that theory, rather than in the development of those results. They are engaged in activities that include the compilation of [[national accounts]] and the design, operation and analysis of:- social, industrial, environmental and occupational surveys; medical trials and experiments, and health statistics; industrial [[quality assurance]] systems; financial [[risk management]] systems; and, [[marketing]] pilots and programmes. | ||
The International Statistics Institute has adopted a "Declaration on Professional Ethics" that treats the provision of well-grounded information as a service to society. It requires professional statisticians to "communicate their findings, for the benefit of the widest possible community"; "not engage or collude in selecting methods designed to produce misleading results, or in misrepresenting statistical findings by commission or omission"; not to "accept contractual conditions that are contingent upon a particular outcome from a proposed statistical inquiry"; and to "take appropriate measures to prevent their data from being published or otherwise released in a form that would allow any subject's identity to be disclosed or inferred"<ref>[http://isi.cbs.nl/ethics.htm, ''Declaration on Professional Ethics'', International Statistics Institute, August 1985 ][http://isi.cbs.nl/ethics0index.htm ]</ref>. Statisticians' professional associations are listed on the [[/External Links|external links subpage]]. | The International Statistics Institute has adopted a "Declaration on Professional Ethics" that treats the provision of well-grounded information as a service to society. It requires professional statisticians to "communicate their findings, for the benefit of the widest possible community"; "not engage or collude in selecting methods designed to produce misleading results, or in misrepresenting statistical findings by commission or omission"; not to "accept contractual conditions that are contingent upon a particular outcome from a proposed statistical inquiry"; and to "take appropriate measures to prevent their data from being published or otherwise released in a form that would allow any subject's identity to be disclosed or inferred"<ref>[http://isi.cbs.nl/ethics.htm, ''Declaration on Professional Ethics'', International Statistics Institute, August 1985 ][http://isi.cbs.nl/ethics0index.htm ]</ref>. Statisticians' professional associations are listed on the [[/External Links|external links subpage]]. | ||
Line 23: | Line 22: | ||
==The collection of statistics== | ==The collection of statistics== | ||
The methodology adopted for the collection of observations has a profound influence upon the problem of extracting useful information from the resulting statistics. That problem is at its easiest when the collecting authority can minimise disturbing influences by conducting a "controlled experiment"<ref> In a controlled experiment, a "control group", that is in all relevant respects similar to the experimental group, receive a "placebo", while the experimental group receive the treatment that is on trial</ref>. A range of more complex methodologies (and associated software packages) referred to as "the design of experiments" <ref>[http://www.stats.gla.ac.uk/steps/glossary/anova.html Valerie Easton and John McCall: ''The Design of Experiments and ANOVA''. STEPS 1997]</ref> is available for use when the collecting authority has various lesser degrees of control. The object of the design in each case is to facilitate the testing of an hypothesis by helping to exclude the influence of factors that the hypothesis does not take into account. When extraneous influences cannot be eliminated by physical exclusion, their effects can sometimes be estimated by purely analytical techniques such as the "analysis of variance"<ref>[http://www.econometrics.org/. ''Anova Manova" ]</ref> For example, | The methodology adopted for the collection of observations has a profound influence upon the problem of extracting useful information from the resulting statistics. That problem is at its easiest when the collecting authority can minimise disturbing influences by conducting a "controlled experiment"<ref> In a controlled experiment, a "control group", that is in all relevant respects similar to the experimental group, receive a "placebo", while the experimental group receive the treatment that is on trial</ref>. A range of more complex methodologies (and associated software packages) referred to as "the design of experiments" <ref>[http://www.stats.gla.ac.uk/steps/glossary/anova.html Valerie Easton and John McCall: ''The Design of Experiments and ANOVA''. STEPS 1997]</ref> is available for use when the collecting authority has various lesser degrees of control. The object of the design in each case is to facilitate the testing of an hypothesis by helping to exclude the influence of factors that the hypothesis does not take into account. When extraneous influences cannot be eliminated by physical exclusion, their effects can sometimes be estimated by purely analytical techniques such as the "analysis of variance"<ref>[http://www.econometrics.org/. ''Anova Manova" ]</ref> For example, the testing of economic hypotheses has used a version of the analysis of variance termed "econometrics"<ref>[http://www.econometrics.org/ ''Econometrics'' 2005]</ref>. | ||
The taking of samples<ref>[http://www.stats.gla.ac.uk/steps/glossary/sampling.html#stratsamp Valerie Easton and John McCall: ''Sampling'', STEPS 1997],</ref> reduces the cost of collecting observations and increases the opportunities to generate false information. One source of error arises from the fact that every time a sample is taken there will be a different result. That source of error is readily quantified as the sample's "standard error", or as the "confidence interval" within which the mean observation may be expected to lie<ref>[http://stattrek.com/AP-Statistics-4/Confidence-Interval.aspx?Tutorial=AP Robin Levine-Wissing and David Thiel, ''Confidence Intervals'', AP Statistics Tutorial]</ref>. That source of error cannot be eliminated, but it can be reduced to an acceptable level by increasing the size of the sample. The other source of error arises from the likelihood that the characteristics of the sample differ from those of the "population" that it is intended to represent. That source of error does not diminish with sample size and cannot be estimated by a mathematical formula. Careful attention to what is known about the composition of the "population" and the reflection of that composition in the sample is the only available precaution. The composition of the respondents to an opinion poll, for example, is normally chosen to reflect as far as possible the composition of the intended "population" as regards sex, age, income bracket etc. The remaining difference is referred as the "sample bias"'. | The taking of samples<ref>[http://www.stats.gla.ac.uk/steps/glossary/sampling.html#stratsamp Valerie Easton and John McCall: ''Sampling'', STEPS 1997],</ref> reduces the cost of collecting observations and increases the opportunities to generate false information. One source of error arises from the fact that every time a sample is taken there will be a different result. That source of error is readily quantified as the sample's "standard error", or as the "confidence interval" within which the mean observation may be expected to lie<ref>[http://stattrek.com/AP-Statistics-4/Confidence-Interval.aspx?Tutorial=AP Robin Levine-Wissing and David Thiel, ''Confidence Intervals'', AP Statistics Tutorial]</ref>. That source of error cannot be eliminated, but it can be reduced to an acceptable level by increasing the size of the sample. The other source of error arises from the likelihood that the characteristics of the sample differ from those of the "population" that it is intended to represent. That source of error does not diminish with sample size and cannot be estimated by a mathematical formula. Careful attention to what is known about the composition of the "population" and the reflection of that composition in the sample is the only available precaution. The composition of the respondents to an opinion poll, for example, is normally chosen to reflect as far as possible the composition of the intended "population" as regards sex, age, income bracket etc. The remaining difference is referred as the "sample bias"'. | ||
Line 32: | Line 31: | ||
Published statistics are widely used for research into subjects such as economics, health and demography. A large volume of time-series statistics is published by national statistics authorities. In the United States, for example, statistical series are published regularly by over 70 federal agencies<ref>[http://www.fedstats.gov/ ''Fedstats'' (an index of statistics published by US federal agencies)]</ref>. There are also a number of databases and collations of national statistics published by international agencies such as the United Nations | Published statistics are widely used for research into subjects such as economics, health and demography. A large volume of time-series statistics is published by national statistics authorities. In the United States, for example, statistical series are published regularly by over 70 federal agencies<ref>[http://www.fedstats.gov/ ''Fedstats'' (an index of statistics published by US federal agencies)]</ref>. There are also a number of databases and collations of national statistics published by international agencies such as the United Nations | ||
<ref>[http://data.un.org/ ''UNdata'' (United Nations statistics database)]</ref> and the OECD | <ref>[http://data.un.org/ ''UNdata'' (United Nations statistics database)]</ref> and the OECD | ||
<ref>[http://www.oecd.org/statsportal/0,3352,en_2825_293564_1_1_1_1_1,00.html ''OECD Statistics Portal'']</ref>, most of which are available on the internet <ref>[http://www.lib.umd.edu/guides/intstat.html ''International Statistics Guide'', University of Maryland Library. August 2008]</ref>. | <ref>[http://www.oecd.org/statsportal/0,3352,en_2825_293564_1_1_1_1_1,00.html ''OECD Statistics Portal'']</ref>, most of which are available on the internet <ref>[http://www.lib.umd.edu/guides/intstat.html ''International Statistics Guide'', University of Maryland Library. August 2008]</ref>. The United Nations has adopted a statement that establishes ten fundamental principles governing the production of national statistics<ref>[http://unstats.un.org/unsd/methods/statorg/FP-English.htm ''Fundamental Principles of Official Statistics'', United Nations Statistics Division, 2006]</ref>, but although a number of attempts have been made to introduce common standards, nationally-published statistics are seldom directly comparable, except after extensive adjustments <ref>For example the United Nations International Comparison Programme for economic statistics[http://unstats.un.org/unsd/methods/icp/index.htm]</ref>. | ||
Many of the published series involve the quarterly or monthly collection of data concerning a specified topic such as an industry, a disease or a category of crime or of economic activity, often by a comprehensive survey (and sometimes by sample surveys) of holders of the relevant data. Another type of source, that has been of value in medical, social and criminological research, has been the occasional collection of comparative data from a group of people born in a particular place on a particular date, by a technique termed "birth-cohort study" (such as the study of the people born in Britain in one week in April 1970 <ref>[http://www.cls.ioe.ac.uk/studies.asp?section=000100020002. ''The British Cohort Study 1970'', Centre for Longitudinal Studies,2009]</ref>, the study of people born in Philadelphia in 1958<ref>[http://www.cls.ioe.ac.uk/studies.asp?section=000100020002 Robert M. Figlio, Paul E. Tracy, and Marvin E. Wolfgang''Delinquency in a Birth Cohort II: Philadelphia, 1958-1988'', Inter-university Consortium for Political and Social Research, September 1999]</ref>, and others<ref>[http://www.cls.ioe.ac.uk/text.asp?section=0001000100050009 ''Cohort Studies Throughout the World'', Centre for Longitudinal Studies, 2008]</ref>) | Many of the published series involve the quarterly or monthly collection of data concerning a specified topic such as an industry, a disease or a category of crime or of economic activity, often by a comprehensive survey (and sometimes by sample surveys) of holders of the relevant data. Another type of source, that has been of value in medical, social and criminological research, has been the occasional collection of comparative data from a group of people born in a particular place on a particular date, by a technique termed "birth-cohort study" (such as the study of the people born in Britain in one week in April 1970 <ref>[http://www.cls.ioe.ac.uk/studies.asp?section=000100020002. ''The British Cohort Study 1970'', Centre for Longitudinal Studies,2009]</ref>, the study of people born in Philadelphia in 1958<ref>[http://www.cls.ioe.ac.uk/studies.asp?section=000100020002 Robert M. Figlio, Paul E. Tracy, and Marvin E. Wolfgang''Delinquency in a Birth Cohort II: Philadelphia, 1958-1988'', Inter-university Consortium for Political and Social Research, September 1999]</ref>, and others<ref>[http://www.cls.ioe.ac.uk/text.asp?section=0001000100050009 ''Cohort Studies Throughout the World'', Centre for Longitudinal Studies, 2008]</ref>) | ||
Line 47: | Line 46: | ||
==Statistical inference== | ==Statistical inference== | ||
Although statistics is sometimes thought of as a branch of mathematics, some of its findings can be successfully interpreted by verbal inference, and there are others that | Although statistics is sometimes thought of as a branch of mathematics, some of its findings can be successfully interpreted by verbal inference, and there are others that can be tackled by a combination of verbal logic and the rules of chance<ref> The rules of chance are set out in paragraph 1 of the [[/Tutorials|tutorials subpage]])</ref>. The development of [[statistics theory]] that has enabled more complex statistical problems to be tackled, has been the work of mathematical geniuses such as Bernoulli, Laplace and Pascal, but the skills required for the effective use of statistics are different from those required for the understanding of the mathematical derivation of their theorems. Statistics theory is mainly concerned with the special circumstances under which events are governed solely by chance, without any influence from human action or recognisable natural forces, whereas applied statistics is usually concerned with its use under less restricted circumstances. Also, the terminology of statistical theory attaches precise meanings to some everyday words that may not always be applicable under less restricted circumstances. When, for example, a degree of statistical confidence is expressed in the statement that the strength of a type of steel will not fall below a stated level, that degree of confidence may apply only to certain circumstances - and not, for example, under conditions of widely varying temperatures. Thus the successful user of statistics has to combine an awareness of the theoretical tools of inference that are available with an appreciation of the extent to which they can safely be applied to a particular problem. The user who plans to employ those tools for the analysis of data must also be prepared to spend a good deal of time acquiring a grasp of the relevant theorems of statistics theory, and mastering the intricacies of the [[free statistical software]] that is available for that purpose. Managers who supervise such work, and users of its application, may seek to be excused from such expenditure of effort, but cannot escape responsibility for acquiring an understanding of statistical concepts that is at least sufficient for an awareness of the limitations of such analysis. And, as the statistician, M J Moroney has emphasised, there can never be any question of making a decision solely on the basis of a statistical test: an engineer doing a statistical test must remain an engineer, an economist must remain an economist, a pharmacist a pharmacist<ref>M J Moroney: ''Facts from Figures'',page 218, Penguin 1951</ref>. | ||
A major contribution of statistics theory to the interpretation of results obtained from a sample is the quantification of the intuitive concept of "significance" in a way that enables an objective answer to be given to the question of how likely it is that, what might appear to be information, is really only a matter of chance (although the way that question is usually put by statisticians is "whether the result is significant at the 5 per cent level"). If - and only if - it can be established by other methods that the sample used was not biassed, then one of a variety of statistical tests can be used to answer that question<ref>For example the procedure of the tutorial ''Tests of Significance'' and its following chapters , in Stat Trek Statistics Tutorials[http://stattrek.com/Lesson5/HypothesisTesting.aspx] </ref> | A major contribution of statistics theory to the interpretation of results obtained from a sample is the quantification of the intuitive concept of "significance" in a way that enables an objective answer to be given to the question of how likely it is that, what might appear to be information, is really only a matter of chance (although the way that question is usually put by statisticians is "whether the result is significant at the 5 per cent level"). If - and only if - it can be established by other methods that the sample used was not biassed, then one of a variety of statistical tests can be used to answer that question<ref>For example the procedure of the tutorial ''Tests of Significance'' and its following chapters , in Stat Trek Statistics Tutorials[http://stattrek.com/Lesson5/HypothesisTesting.aspx] </ref>. Equally useful is its quantification of the concept of the term "confidence", enabling an objective answer to be given to a question such as "how confident are we that the toxicity of a drug does not exceed a stipulate level" (or that a structure can bear at least a stipulate load, or that a variable lies within a stipulated [[confidence interval]]). Among the most powerful of the techniques of statistical analysis is the use of "correlation" (sometimes termed "regression") to explore a relationship when the available evidence is subject to errors that are attributable solely to chance. It could be used, for example, to estimate the average constant of proportionality in a (hypothesised) linear relation between IQs and examination marks. In a more complex case it is used to explore the relationship between household income and household saving, taking account of other factors believed to affect that relationship. But, besides being very powerful, regression methods are especially prone to the production of false information. Success in interpreting the data requires the tackling of problems that are often more difficult than the statistical manipulation that is required<ref>M J Moroney: ''Facts from Figures'',page 303, Penguin 1951</ref>. | ||
==Statistical misinterpretation== | ==Statistical misinterpretation== | ||
===Probability blindness=== | ===Probability blindness=== | ||
The human brain is so well adapted to intuitive judgements - of speed, distance, the pitch of a note etcetera - that it is tempting to assume that it can also make intuitive probability judgements. In fact, such judgements are usually wide of the mark<ref>Daniel Kahneman and Amos Tversky: "Prospect Theory". ''Econometrica" vol 47 No2 1979</ref> <ref>Massimo Piattelli-Palmarini: ''Inevitable Illusions: How Mistakes of Reason Rule Our Minds'', Chapter 4, "Probability Illusions", Wiley, 1994</ref>. For example, most people will agreed that it is intuitively very unlikely that two members of a group of two dozen people will have the same birthday, but in fact it is not at all unlikely<ref> A proof of the two-birthday problem is at paragraph 2.1. of the [[/Tutorials/tutorials | The human brain is so well adapted to intuitive judgements - of speed, distance, the pitch of a note etcetera - that it is tempting to assume that it can also make intuitive probability judgements. In fact, such judgements are usually wide of the mark<ref>Daniel Kahneman and Amos Tversky: "Prospect Theory". ''Econometrica" vol 47 No2 1979</ref> <ref>Massimo Piattelli-Palmarini: ''Inevitable Illusions: How Mistakes of Reason Rule Our Minds'', Chapter 4, "Probability Illusions", Wiley, 1994</ref>. For example, most people will agreed that it is intuitively very unlikely that two members of a group of two dozen people will have the same birthday, but in fact it is not at all unlikely<ref> A proof of the two-birthday problem is at paragraph 2.1. of the [[/Tutorials/tutorials subpage]]</ref>. The experimental psychologists, Amos Tversky and Daniel Kahneman have identified a variety of ways in which the use of "heuristics" in place of analysis distorts peoples' interpretation of numerical data<ref>Amos Tversky and Daniel Kahneman: ''Judgment under Uncertainty: Heuristics and Biases'', Science, New Series, Vol. 185, No. 4157. September 27, 1974</ref>. The "availability heuristic", for example, leads to overestimates of the incidence of newsworthy events such as rail disasters, and underestimates of more commonplace occurrences such as road accidents. In a well-known series of experiments it was found that subjects tended to overestimate the likelihood of death from rare causes, underestimate likely causes, and be influenced by drama or vividness | ||
<ref>Sara Lichtenstein et al: ''Judged Frequency of Lethal Events'', Journal of Experimental Psychology: Human Learning and Memory, v4 n6 p551-78 Nov 1978</ref>. Similarly, there are commonly gross discrepancies between the fear of crime and its true incidence, and a widespread conviction that the crime rate is rising at times when it is stable or falling<ref> As noted by the UK Home Office Director of Statistics in a letter to the Times of 22nnd January 2003</ref>. | <ref>Sara Lichtenstein et al: ''Judged Frequency of Lethal Events'', Journal of Experimental Psychology: Human Learning and Memory, v4 n6 p551-78 Nov 1978</ref>. Similarly, there are commonly gross discrepancies between the fear of crime and its true incidence, and a widespread conviction that the crime rate is rising at times when it is stable or falling<ref> As noted by the UK Home Office Director of Statistics in a letter to the Times of 22nnd January 2003</ref>. | ||
===Medical misinterpretation=== | ===Medical misinterpretation=== | ||
Errors in the reporting of medical research have not been uncommon. Researchers at the University of Girona in Spain, found that 38% of a sample of papers in Nature, and a quarter of those sampled in the British Medical Journal — two of the world's most respected journals — contained one or more statistical errors, some resulting in non-significant findings being misrepresented as significant | Errors in the reporting of medical research have not been uncommon. Researchers at the University of Girona in Spain, found that 38% of a sample of papers in Nature, and a quarter of those sampled in the British Medical Journal — two of the world's most respected journals — contained one or more statistical errors, some resulting in non-significant findings being misrepresented as significant | ||
<ref> Emili García-Berthou and Carles Alcaraz: ''Incongruence between test statistics and P values in medical papers'', BMC Medical Research Methodology 2004, 4:13 [http://www.biomedcentral.com/1471-2288/4/13/](reported in ''The | <ref> Emili García-Berthou and Carles Alcaraz: ''Incongruence between test statistics and P values in medical papers'', BMC Medical Research Methodology 2004, 4:13 [http://www.biomedcentral.com/1471-2288/4/13/](reported in ''The Economist'' of 30th June 2004 | ||
[http://www.economist.com/sciencetechnology/displaystory.cfm?story_id=E1_NSNQNNG])</ref>. In another study an eminent Greek | [http://www.economist.com/sciencetechnology/displaystory.cfm?story_id=E1_NSNQNNG])</ref>. In another study an eminent Greek statistician has estimated that most medical research findings are false, and that they may often be simply measures of a prevailing bias <ref>[http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 John P. A. Ioannidis ''Why Most Research Findings are False'', PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 August 2005]</ref>. It is also believed that misinterpretation of the results of medical tests is not uncommon. When the following question was put to students and staff of the Harvard Medical School "If a test of a disease that has a prevalence rate of 1 in 1000 has a false positive rate of 5%, what is the chance that a person who has been given a positive result actually has the disease." - 45 per cent gave the intuitive answer of 95%<ref> Michael Eysenck and Mark Keane ''Cognitive Psychology'' page 483 [http://books.google.com/books?id=22ZWi-LVLDcC&pg=PA483&lpg=PA483&dq=%22false+positive%22++%22Harvard+Medical+School%22+++45%25+of+students&source=bl&ots=HJH28YS9yD&sig=6FRx8zyfUaB0LR4OSg75b8_Uxso&hl=en&ei=YSVJSp6VKdShjAeBx5li&sa=X&oi=book_result&ct=result&resnum=3]</ref>: the true answer is 2%<ref> Proved in paragraph 2.2 of the [[/Tutorials|tutorials subpage]]</ref>. | ||
===Legal misinterpretation=== | ===Legal misinterpretation=== | ||
There have also been cases of statistical misinterpretation by expert witnesses and lawyers, some of which have resulted in serious injustice<ref>[http://www.dcs.qmul.ac.uk/researchgp/spotlight/legal.html ''Evaluating Legal Evidence'', Department of | There have also been cases of statistical misinterpretation by expert witnesses and lawyers, some of which have resulted in serious injustice<ref>[http://www.dcs.qmul.ac.uk/researchgp/spotlight/legal.html ''Evaluating Legal Evidence'', Department of Computer Science, Queen Mary College, University of London, 2009]</ref>. In 1999 the British Solicitor, Sally Clark was convicted of murdering her two babies on the evidence of an expert witness who told the jury that the odds against the deaths being natural were one in 73 million; a figure he obtained by taking the square of the odds against a single cot death. The UK's Royal Statistical Society issued a formal statement that the 1 in 73 million figure "...would only be valid if SIDS (cot death) cases arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong a priori reasons for supposing that the assumption will be false"<ref>[http://www.rss.org.uk/PDF/RSS%20Statement%20regarding%20statistical%20issues%20in%20the%20Sally%20Clark%20case,%20October%2023rd%202001.pdf ''Issues Raised in the Sally Clark Case'', News release by the Royal Statistical Society, 23 October 2000]</ref>. Wrongful convictions in rape cases have also arisen from a confusion between the probability of effect given cause and that of cause given effect that is known as the "prosecutor's fallacy"<ref>The prosecutor's fallacy is explained at paragraph 2.3 of the [[/Tutorials|tutorials subpage]])</ref>. | ||
===Financial misinterpretation=== | ===Financial misinterpretation=== | ||
Financial analysts have been known to use "survivor bias" to mislead potential clients by the use of statistics that demonstrate the benefits they would have enjoyed had they previously invested in their currently recommended portfolio - thus drawing attention away from previous recommendations that had been withdrawn because they had done badly. Another example of survivor bias in the interpretation of financial statistics was study of financial returns which covered only firms existing at the end of the period under review, thereby excluding those that had failed in the interim, that showed shares to have earned a return of 8.8 percent when the true return had been 3.8 per cent<ref> Dimson, Marsh and Staunton: ''The | Financial analysts have been known to use "survivor bias" to mislead potential clients by the use of statistics that demonstrate the benefits they would have enjoyed had they previously invested in their currently recommended portfolio - thus drawing attention away from previous recommendations that had been withdrawn because they had done badly. Another example of survivor bias in the interpretation of financial statistics was a well-known study of financial returns which covered only firms existing at the end of the period under review, thereby excluding those that had failed in the interim, that showed shares to have earned a return of 8.8 percent when the true return had been 3.8 per cent<ref> Dimson, Marsh and Staunton: ''The Millennium Book: a Century of Investment Returns'', ABN Amro, 2000</ref>. | ||
Probably the most damaging consequence of statistical misinterpretation was the financial crisis that followed the acceptance by many of the world's financial institutions of the ill-founded risk assessments that were issued in the early 21st century by the credit-rating agencies <ref> See ''Sources of risk-management errors''[http://en.citizendium.org/wiki/Crash_of_2008/Tutorials], on the tutorials subpage of "Crash of 2008"</ref>. The agencies had assumed that the risks would be due to random fluctuations and not to human behaviour, and that the underlying probability distribution would be the same as in a past period | Probably the most damaging consequence of statistical misinterpretation was the financial crisis that followed the acceptance by many of the world's financial institutions of the ill-founded risk assessments that were issued in the early 21st century by the credit-rating agencies <ref> See ''Sources of risk-management errors''[http://en.citizendium.org/wiki/Crash_of_2008/Tutorials], on the tutorials subpage of "Crash of 2008"</ref>. The agencies had assumed that the risks would be due to random fluctuations and not to human behaviour, and that the underlying probability distribution would be the same as in a past period | ||
Line 70: | Line 69: | ||
==Notes and references== | ==Notes and references== | ||
{{reflist|2}}[[Category:Suggestion Bot Tag]] | |||
Latest revision as of 07:00, 12 July 2024
The term applied statistics is used both to denote the work of professional statisticians, who are responsible for the production and presentation of statistics, and to refer to the processes of statistical interpretation that are undertaken by professional statisticians, by professional users of statistics and by the general public. It makes frequent use of statistics theory but it is distinct from the mathematics of that theory. It has made major contributions to scientific research and to the conduct of operational activities in a variety of fields. It has provided both a familiar source of information and a notorious source of misinformation. Misinterpretations of statistics by members of the medical and legal professions have led to false diagnoses and wrongful imprisonment, and ill-founded risk assessments were among the causes of the financial crash of 2008 .
Overview: the basics
Statistics are observations that are recorded in numerical form. It is essential to their successful handling to accept that statistics are not facts and therefore incontrovertible, but observations about facts and therefore fallible. The reliability of the information that they provide depends not only upon their successful interpretation, but also upon the accuracy with which the facts are observed and the extent to which they truly represent the subject matter of that information. An appreciation of the means by which statistics are collected is thus an essential part of the understanding of statistics and is at least as important as a familiarity with the tools that are used in its interpretation.
The basic laws of chance from which much of statistics theory has been derived are little more than a formalisation of intuitive concepts. Combined with the use of advanced mathematics, they have been used to create a toolbox of concepts and theories that are available for the use of statistics in the solution of problems arising in other disciplines.
The Statistics profession
The American Statistical Association defines a professional statistician as an expert in producing trustworthy data; analyzing data to make their meaning clear; and, drawing practical conclusions from data. [1] Professional statisticians are normally qualified in mathematics and are normally familiar with the underlying mathematics of statistics theory but their professional activity involves them in the use of the results that theory, rather than in the development of those results. They are engaged in activities that include the compilation of national accounts and the design, operation and analysis of:- social, industrial, environmental and occupational surveys; medical trials and experiments, and health statistics; industrial quality assurance systems; financial risk management systems; and, marketing pilots and programmes.
The International Statistics Institute has adopted a "Declaration on Professional Ethics" that treats the provision of well-grounded information as a service to society. It requires professional statisticians to "communicate their findings, for the benefit of the widest possible community"; "not engage or collude in selecting methods designed to produce misleading results, or in misrepresenting statistical findings by commission or omission"; not to "accept contractual conditions that are contingent upon a particular outcome from a proposed statistical inquiry"; and to "take appropriate measures to prevent their data from being published or otherwise released in a form that would allow any subject's identity to be disclosed or inferred"[2]. Statisticians' professional associations are listed on the external links subpage.
Major applications of statistics
Statistical analysis has been widely used for scientific research, drawing upon both published statistics and the results of surveys and laboratory experiments, and it has enabled major scientific advances to be made in subjects as diverse as medicine and criminology. It has also had operational uses in the fields of business decision-making. [3], climatology[4], metrology [5], cost/benefit analysis[6], and finance[7]. For those operational applications it has often had to draw upon theoretical knowledge and professional judgement, as well as upon records of past experience. As the distinguished economist Frank Knight observed in 1921, "business decisions ... deal with situations which are far too unique, generally speaking, for any sort of statistical tabulation to have any value for guidance. The conception of an objectively measurable probability or chance is simply inapplicable."[8] - and drew a distinction between risk, which can sometimes be estimated objectively; and uncertainty, which never can. Business decisions necessarily involve the use of subjective estimates, which can be combined using a "decision tree" [9] to produce a probability estimate for each of the range of possible outcomes, enabling the values of those outcomes to be combined to yield a "net present expected value"[10]. The estimation of net present value in that way, is an essential feature of cost/benefit analysis, and in those and other applications it may be necessary to base decisions upon a combination of "aleatory" (chance-related) and "epistemic" (cognitive) factors. In the practice of risk management, in particular, it is important not to neglect epistemic factors. The data used for statistical quality control[11] are usually the results of tests on samples drawn from a batch of the products of a manufacturing process and its purpose is to decide whether to accept that batch. The normal criteria for acceptance are designed to ensure, at a specified level of confidence that the quality of the batch does not depart by more than a specified percentage from a target level (but in cases where the costs of departures depend upon their magnitude the more complex Taguchi criteria [12] is sometimes adopted). Statistical analysis has also been used to devise economical sampling methods[13]
The collection of statistics
The methodology adopted for the collection of observations has a profound influence upon the problem of extracting useful information from the resulting statistics. That problem is at its easiest when the collecting authority can minimise disturbing influences by conducting a "controlled experiment"[14]. A range of more complex methodologies (and associated software packages) referred to as "the design of experiments" [15] is available for use when the collecting authority has various lesser degrees of control. The object of the design in each case is to facilitate the testing of an hypothesis by helping to exclude the influence of factors that the hypothesis does not take into account. When extraneous influences cannot be eliminated by physical exclusion, their effects can sometimes be estimated by purely analytical techniques such as the "analysis of variance"[16] For example, the testing of economic hypotheses has used a version of the analysis of variance termed "econometrics"[17].
The taking of samples[18] reduces the cost of collecting observations and increases the opportunities to generate false information. One source of error arises from the fact that every time a sample is taken there will be a different result. That source of error is readily quantified as the sample's "standard error", or as the "confidence interval" within which the mean observation may be expected to lie[19]. That source of error cannot be eliminated, but it can be reduced to an acceptable level by increasing the size of the sample. The other source of error arises from the likelihood that the characteristics of the sample differ from those of the "population" that it is intended to represent. That source of error does not diminish with sample size and cannot be estimated by a mathematical formula. Careful attention to what is known about the composition of the "population" and the reflection of that composition in the sample is the only available precaution. The composition of the respondents to an opinion poll, for example, is normally chosen to reflect as far as possible the composition of the intended "population" as regards sex, age, income bracket etc. The remaining difference is referred as the "sample bias"'.
The use by statisticians of the term "population" refers, not to people in general, but to the category of things or people about which information is sought. A precise definition of the target population is an essential starting point in a statistical investigation, and also a possible source of misinformation. Difficulty can arise when, as often happens, the definition has to be arbitrary. If the intended population were the output of the country's farmers, for example, it might be necessary to draw an arbitrary dividing line between farmers and owners of smallholdings such as market gardens. Any major change over time in the relative output of farm products by the included and excluded categories might then lead to misleading conclusions. Technological change, such as the change from typewriters to word processors has sometimes given rise to serious difficulties in the construction of the price indexes used in the correction of GDP for inflation[20]. Since there is no objective solution to those problems, it is inevitable that national statistics embody an element of judgement exercised by the professional statisticians in the statistics authorities.
Sources of statistics
Published statistics are widely used for research into subjects such as economics, health and demography. A large volume of time-series statistics is published by national statistics authorities. In the United States, for example, statistical series are published regularly by over 70 federal agencies[21]. There are also a number of databases and collations of national statistics published by international agencies such as the United Nations [22] and the OECD [23], most of which are available on the internet [24]. The United Nations has adopted a statement that establishes ten fundamental principles governing the production of national statistics[25], but although a number of attempts have been made to introduce common standards, nationally-published statistics are seldom directly comparable, except after extensive adjustments [26].
Many of the published series involve the quarterly or monthly collection of data concerning a specified topic such as an industry, a disease or a category of crime or of economic activity, often by a comprehensive survey (and sometimes by sample surveys) of holders of the relevant data. Another type of source, that has been of value in medical, social and criminological research, has been the occasional collection of comparative data from a group of people born in a particular place on a particular date, by a technique termed "birth-cohort study" (such as the study of the people born in Britain in one week in April 1970 [27], the study of people born in Philadelphia in 1958[28], and others[29])
Links to national statistics agencies are available on the external links subpage.
Adjustments, reliability, accuracy and bias
Those who regard statistics as being on a par with the proverbial "damned lies"[30] and those who believe them always to be totally objective estimates, are both mistaken. The concept of an objective statistic is a mathematical abstraction and although some, such as some financial statistics, approximate to that ideal, most published and unpublished statistics embody some element of subjective judgement. The embodiment of subjective adjustments in published statistics for the purpose of increasing their usefulness[31] is the responsibility of professional statisticians whose competence and objectivity is usually subject to oversight by an independent body (politicians in the major democracies have seldom had any influence upon the collection and publication of national statistics, and many countries have sought to allay suspicions to the contrary by delegating those functions to public bodies that are free from possible government influence.)[32].
It has seldom been feasible to assess the accuracy of published statistics, and indications of their reliability are available only when a variable is estimated from more than one source or when revisions are made. The substantial discrepancies between police-reported and victim-reported crime rates [33], for example, are presumably the result of differences in the reliability of the two methods, and the differences between income-based and expenditure-based estimates of GDP [34] may be similarly attributable. The major revisions that have been made to early estimates of GDP [35] also provide an indication of the reliability to be expected of such estimates.
The commonest cause of bias in published statistics has been faulty sampling. An early example was the "Lanarkshire Milk Trial", which yielded a perverse effect of giving milk to school children because teachers gave the milk to sickly rather than healthy children[36]. Similar errors that did not produce perverse results may have gone undetected.
Statistical inference
Although statistics is sometimes thought of as a branch of mathematics, some of its findings can be successfully interpreted by verbal inference, and there are others that can be tackled by a combination of verbal logic and the rules of chance[37]. The development of statistics theory that has enabled more complex statistical problems to be tackled, has been the work of mathematical geniuses such as Bernoulli, Laplace and Pascal, but the skills required for the effective use of statistics are different from those required for the understanding of the mathematical derivation of their theorems. Statistics theory is mainly concerned with the special circumstances under which events are governed solely by chance, without any influence from human action or recognisable natural forces, whereas applied statistics is usually concerned with its use under less restricted circumstances. Also, the terminology of statistical theory attaches precise meanings to some everyday words that may not always be applicable under less restricted circumstances. When, for example, a degree of statistical confidence is expressed in the statement that the strength of a type of steel will not fall below a stated level, that degree of confidence may apply only to certain circumstances - and not, for example, under conditions of widely varying temperatures. Thus the successful user of statistics has to combine an awareness of the theoretical tools of inference that are available with an appreciation of the extent to which they can safely be applied to a particular problem. The user who plans to employ those tools for the analysis of data must also be prepared to spend a good deal of time acquiring a grasp of the relevant theorems of statistics theory, and mastering the intricacies of the free statistical software that is available for that purpose. Managers who supervise such work, and users of its application, may seek to be excused from such expenditure of effort, but cannot escape responsibility for acquiring an understanding of statistical concepts that is at least sufficient for an awareness of the limitations of such analysis. And, as the statistician, M J Moroney has emphasised, there can never be any question of making a decision solely on the basis of a statistical test: an engineer doing a statistical test must remain an engineer, an economist must remain an economist, a pharmacist a pharmacist[38].
A major contribution of statistics theory to the interpretation of results obtained from a sample is the quantification of the intuitive concept of "significance" in a way that enables an objective answer to be given to the question of how likely it is that, what might appear to be information, is really only a matter of chance (although the way that question is usually put by statisticians is "whether the result is significant at the 5 per cent level"). If - and only if - it can be established by other methods that the sample used was not biassed, then one of a variety of statistical tests can be used to answer that question[39]. Equally useful is its quantification of the concept of the term "confidence", enabling an objective answer to be given to a question such as "how confident are we that the toxicity of a drug does not exceed a stipulate level" (or that a structure can bear at least a stipulate load, or that a variable lies within a stipulated confidence interval). Among the most powerful of the techniques of statistical analysis is the use of "correlation" (sometimes termed "regression") to explore a relationship when the available evidence is subject to errors that are attributable solely to chance. It could be used, for example, to estimate the average constant of proportionality in a (hypothesised) linear relation between IQs and examination marks. In a more complex case it is used to explore the relationship between household income and household saving, taking account of other factors believed to affect that relationship. But, besides being very powerful, regression methods are especially prone to the production of false information. Success in interpreting the data requires the tackling of problems that are often more difficult than the statistical manipulation that is required[40].
Statistical misinterpretation
Probability blindness
The human brain is so well adapted to intuitive judgements - of speed, distance, the pitch of a note etcetera - that it is tempting to assume that it can also make intuitive probability judgements. In fact, such judgements are usually wide of the mark[41] [42]. For example, most people will agreed that it is intuitively very unlikely that two members of a group of two dozen people will have the same birthday, but in fact it is not at all unlikely[43]. The experimental psychologists, Amos Tversky and Daniel Kahneman have identified a variety of ways in which the use of "heuristics" in place of analysis distorts peoples' interpretation of numerical data[44]. The "availability heuristic", for example, leads to overestimates of the incidence of newsworthy events such as rail disasters, and underestimates of more commonplace occurrences such as road accidents. In a well-known series of experiments it was found that subjects tended to overestimate the likelihood of death from rare causes, underestimate likely causes, and be influenced by drama or vividness [45]. Similarly, there are commonly gross discrepancies between the fear of crime and its true incidence, and a widespread conviction that the crime rate is rising at times when it is stable or falling[46].
Medical misinterpretation
Errors in the reporting of medical research have not been uncommon. Researchers at the University of Girona in Spain, found that 38% of a sample of papers in Nature, and a quarter of those sampled in the British Medical Journal — two of the world's most respected journals — contained one or more statistical errors, some resulting in non-significant findings being misrepresented as significant [47]. In another study an eminent Greek statistician has estimated that most medical research findings are false, and that they may often be simply measures of a prevailing bias [48]. It is also believed that misinterpretation of the results of medical tests is not uncommon. When the following question was put to students and staff of the Harvard Medical School "If a test of a disease that has a prevalence rate of 1 in 1000 has a false positive rate of 5%, what is the chance that a person who has been given a positive result actually has the disease." - 45 per cent gave the intuitive answer of 95%[49]: the true answer is 2%[50].
Legal misinterpretation
There have also been cases of statistical misinterpretation by expert witnesses and lawyers, some of which have resulted in serious injustice[51]. In 1999 the British Solicitor, Sally Clark was convicted of murdering her two babies on the evidence of an expert witness who told the jury that the odds against the deaths being natural were one in 73 million; a figure he obtained by taking the square of the odds against a single cot death. The UK's Royal Statistical Society issued a formal statement that the 1 in 73 million figure "...would only be valid if SIDS (cot death) cases arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong a priori reasons for supposing that the assumption will be false"[52]. Wrongful convictions in rape cases have also arisen from a confusion between the probability of effect given cause and that of cause given effect that is known as the "prosecutor's fallacy"[53].
Financial misinterpretation
Financial analysts have been known to use "survivor bias" to mislead potential clients by the use of statistics that demonstrate the benefits they would have enjoyed had they previously invested in their currently recommended portfolio - thus drawing attention away from previous recommendations that had been withdrawn because they had done badly. Another example of survivor bias in the interpretation of financial statistics was a well-known study of financial returns which covered only firms existing at the end of the period under review, thereby excluding those that had failed in the interim, that showed shares to have earned a return of 8.8 percent when the true return had been 3.8 per cent[54].
Probably the most damaging consequence of statistical misinterpretation was the financial crisis that followed the acceptance by many of the world's financial institutions of the ill-founded risk assessments that were issued in the early 21st century by the credit-rating agencies [55]. The agencies had assumed that the risks would be due to random fluctuations and not to human behaviour, and that the underlying probability distribution would be the same as in a past period [56]. Neither assumption was supported by the available evidence [57].
Notes and references
- ↑ What Do Statisticians Do?, The careers center of the American Statistical Association
- ↑ Declaration on Professional Ethics, International Statistics Institute, August 1985 [1]
- ↑ Kent Holzinger: Decision Making Under Uncertainty: Statistical Decision Theory, University of Connecticut, 2007
- ↑ Climate adaptation: Risk, Uncertainty and Decision-making UK Department of the Environment, Food and Rural Affairs, May 2003
- ↑ Bayesian Metrology: Overview Topics, Statistical Engineering Division, National Institute of Standards and Technology, May 2001
- ↑ Appraisal and Evaluation in Central Government, HM Treasury, UK Stationery Office
- ↑ David Harper: Introduction to Value at Risk, Investopedia.com
- ↑ Frank Knight: Risk, Uncertainty and Profit Part III, Chapter IV, paragraph 47, Boston, MA: Hart, Schaffner & Marx; Houghton Mifflin Co, 1921
- ↑ Decision Tree: How to understand it, The Quality Toolbook, 2007
- ↑ For an explanation of net present expected value, see the article on net present value
- ↑ Introduction to Statistical Process Control, Statit Software, Inc, 2007
- ↑ Taguchi Loss Function Visual and Equation, Marc Timothy Smith Elsmar Cove Forums 2007
- ↑ What is Sequential Sampling?, The Quality Toolbook, 2007
- ↑ In a controlled experiment, a "control group", that is in all relevant respects similar to the experimental group, receive a "placebo", while the experimental group receive the treatment that is on trial
- ↑ Valerie Easton and John McCall: The Design of Experiments and ANOVA. STEPS 1997
- ↑ Anova Manova"
- ↑ Econometrics 2005
- ↑ Valerie Easton and John McCall: Sampling, STEPS 1997,
- ↑ Robin Levine-Wissing and David Thiel, Confidence Intervals, AP Statistics Tutorial
- ↑ See the article on Gross domestic product
- ↑ Fedstats (an index of statistics published by US federal agencies)
- ↑ UNdata (United Nations statistics database)
- ↑ OECD Statistics Portal
- ↑ International Statistics Guide, University of Maryland Library. August 2008
- ↑ Fundamental Principles of Official Statistics, United Nations Statistics Division, 2006
- ↑ For example the United Nations International Comparison Programme for economic statistics[2]
- ↑ The British Cohort Study 1970, Centre for Longitudinal Studies,2009
- ↑ Robert M. Figlio, Paul E. Tracy, and Marvin E. WolfgangDelinquency in a Birth Cohort II: Philadelphia, 1958-1988, Inter-university Consortium for Political and Social Research, September 1999
- ↑ Cohort Studies Throughout the World, Centre for Longitudinal Studies, 2008
- ↑ The quip "lies, damned lies and statistics" is generally attributed to Benjamin Disraeli speaking in 1895, but there have been earlier attributions[3]
- ↑ Data Editing in Federal Statistical Agencies, Statistical Policy Working Paper 18, Federal Committee on Statistical Methodology, Office of Management and Budget, April 1990
- ↑ In the United Kingdom, for example, supervision is the responsibility of the UK Statistics Authority[4]. The standard adopted by European authorities are set out in an EU code of practice [5]. Arrangements in the United States are discussed in the Federal Register of 4th June 2002[6]
- ↑ An Overview of the Differences between Police-Reported and Victim-Reported Crime, Statistics Canada, 1997
- ↑ Bruce T. Grimm: The Statistical Discrepancy, WP2007-01, US Bureau of Economic Analysis, March 2, 2007
- ↑ See paragraph 3.4 of the article on gross domestic product
- ↑ "Student" (W S Gosset): The Lanarkshire Milk Experiment, Biometrika 23 1931 pp 398-406.
- ↑ The rules of chance are set out in paragraph 1 of the tutorials subpage)
- ↑ M J Moroney: Facts from Figures,page 218, Penguin 1951
- ↑ For example the procedure of the tutorial Tests of Significance and its following chapters , in Stat Trek Statistics Tutorials[7]
- ↑ M J Moroney: Facts from Figures,page 303, Penguin 1951
- ↑ Daniel Kahneman and Amos Tversky: "Prospect Theory". Econometrica" vol 47 No2 1979
- ↑ Massimo Piattelli-Palmarini: Inevitable Illusions: How Mistakes of Reason Rule Our Minds, Chapter 4, "Probability Illusions", Wiley, 1994
- ↑ A proof of the two-birthday problem is at paragraph 2.1. of the /Tutorials/tutorials subpage
- ↑ Amos Tversky and Daniel Kahneman: Judgment under Uncertainty: Heuristics and Biases, Science, New Series, Vol. 185, No. 4157. September 27, 1974
- ↑ Sara Lichtenstein et al: Judged Frequency of Lethal Events, Journal of Experimental Psychology: Human Learning and Memory, v4 n6 p551-78 Nov 1978
- ↑ As noted by the UK Home Office Director of Statistics in a letter to the Times of 22nnd January 2003
- ↑ Emili García-Berthou and Carles Alcaraz: Incongruence between test statistics and P values in medical papers, BMC Medical Research Methodology 2004, 4:13 [8](reported in The Economist of 30th June 2004 [9])
- ↑ John P. A. Ioannidis Why Most Research Findings are False, PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 August 2005
- ↑ Michael Eysenck and Mark Keane Cognitive Psychology page 483 [10]
- ↑ Proved in paragraph 2.2 of the tutorials subpage
- ↑ Evaluating Legal Evidence, Department of Computer Science, Queen Mary College, University of London, 2009
- ↑ Issues Raised in the Sally Clark Case, News release by the Royal Statistical Society, 23 October 2000
- ↑ The prosecutor's fallacy is explained at paragraph 2.3 of the tutorials subpage)
- ↑ Dimson, Marsh and Staunton: The Millennium Book: a Century of Investment Returns, ABN Amro, 2000
- ↑ See Sources of risk-management errors[11], on the tutorials subpage of "Crash of 2008"
- ↑ Written statement of Frank L. Raiter On “Credit Rating Agencies and the Financial Crisis” before the Committee on Oversight and Government Reform, United States House of Representatives, October 22, 2008
- ↑ [12] Barry du Toit Risk, theory, reflection: Limitations of the stochastic model of uncertainty in financial risk analysis Riskworx June 2004