Metascience in the context of Reproducibility

Research is creative and systematic work undertaken to increase the stock of knowledge. It involves the collection, organization, and analysis of evidence to increase understanding of a topic, characterized by a particular attentiveness to controlling sources of bias and error. These activities are characterized by accounting and controlling for biases. A research project may be an expansion of past work in the field. To test the validity of instruments, procedures, or experiments, research may replicate elements of prior projects or the project as a whole.

The primary purposes of basic research (as opposed to applied research) are documentation, discovery, interpretation, and the research and development (R&D) of methods and systems for the advancement of human knowledge. Approaches to research depend on epistemologies, which vary considerably both within and between humanities and sciences. There are several forms of research: scientific, humanities, artistic, economic, social, business, marketing, practitioner research, life, technological, etc. The scientific study of research practices is known as meta-research.

Evidence-based practice is the idea that occupational practices ought to be based on scientific evidence. The movement towards evidence-based practices attempts to encourage and, in some instances, require professionals and other decision-makers to pay more attention to evidence to inform their decision-making. The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research. The proposal has been controversial, with some arguing that results may not specialize to individuals as well as traditional practices.

Evidence-based practices have been gaining ground since the introduction of evidence-based medicine and have spread to the allied health professions, education, management, law, public policy, architecture, and other fields. In light of studies showing problems in scientific research (such as the replication crisis), there is also a movement to apply evidence-based practices in scientific research itself. Research into the evidence-based practice of science is called metascience.

A metatheory or meta-theory is a theory whose subject matter is another theory. Analyses or descriptions of an existing theory would be considered meta-theories. For mathematics and mathematical logic, a metatheory is a mathematical theory about another mathematical theory. Meta-theoretical investigations are part of the philosophy of science. The topic of metascience is an attempt to use scientific knowledge to improve the practice of science itself.

The study of metatheory became widespread during the 20th century after its application to various topics, including scientific linguistics and its concept of metalanguage.

Scientometrics is a subfield of informetrics that studies quantitative aspects of scholarly literature. Major research issues include the measurement of the impact of research papers and academic journals, the understanding of scientific citations, and the use of such measurements in policy and management contexts.

In practice there is a substantial overlap between scientometrics and other scientific fields such as information systems, information science, science of science policy, sociology of science, and metascience. Critics have argued that overreliance on scientometrics has created a system of perverse incentives, producing a publish or perish environment that leads to low-quality research.

Meta-analysis is a method of synthesis of quantitative data from multiple independent studies addressing a common research question. An important part of this method involves computing a combined effect size across all of the studies. As such, this statistical approach involves extracting effect sizes and variance measures from various studies. By combining these effect sizes the statistical power is improved and can resolve uncertainties or discrepancies found in individual studies. Meta-analyses are integral in supporting research grant proposals, shaping treatment guidelines, and influencing health policies. They are also pivotal in summarizing existing research to guide future studies, thereby cementing their role as a fundamental methodology in metascience. Meta-analyses are often, but not always, important components of a systematic review.

In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings in favor of positive results. The study of publication bias is an important topic in metascience.

Despite similar quality of execution and design, papers with statistically significant results are three times more likely to be published than those with null results. This unduly motivates researchers to manipulate their practices to ensure statistically significant results, such as by data dredging.

John P. A. Ioannidis (/ˌiːəˈniːdɪs/ EE-ə-NEE-diss; Greek: Ιωάννης Ιωαννίδης, pronounced [i.oˈanis i.oaˈniðis]; born August 21, 1965) is a Greek-American physician-scientist, writer and Stanford University professor who has made contributions to evidence-based medicine, epidemiology, and clinical research. Ioannidis studies scientific research itself – in other words, meta-research – primarily in clinical medicine and the social sciences.

He has served on the editorial board of over twenty scientific journals including Journal of the American Medical Association (JAMA), Journal of the National Cancer Institute (JNCI) and The Lancet.

In null-hypothesis significance testing, the p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct. A very small p-value means that such an extreme observed outcome would be very unlikely under the null hypothesis. Even though reporting p-values of statistical tests is common practice in academic publications of many quantitative fields, misinterpretation and misuse of p-values is widespread and has been a major topic in mathematics and metascience.

In 2016, the American Statistical Association (ASA) made a formal statement that "p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone" and that "a p-value, or statistical significance, does not measure the size of an effect or the importance of a result" or "evidence regarding a model or hypothesis". That said, a 2019 task force by ASA has issued a statement on statistical significance and replicability, concluding with: "p-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data".

Scientometrics is a subfield of informetrics that studies quantitative aspects of scholarly literature. Major research issues in scientometrics include the measurement of the impact of research papers and academic journals, the understanding of scientific citations, and the use of such measurements in policy and management contexts.

In practice, there is a substantial overlap between scientometrics and other scientific fields, including information systems, information science, the science of science policy, the sociology of science, and metascience. Critics have argued that overreliance on scientometrics has created a system of perverse incentives, producing a publish or perish environment that leads to low-quality research.