Are we getting smarter? Publication bias in estimating the Flynn effect

• Název práce v češtině: Are we getting smarter? Publication bias in estimating the Flynn effect
• Název v anglickém jazyce: Are we getting smarter? Publication bias in estimating the Flynn effect
• Akademický rok vypsání: 2020/2021
• Typ práce: diplomová práce
• Jazyk práce: angličtina
• Ústav: Institut ekonomických studií (23-IES)
• Vedoucí / školitel: doc. PhDr. Zuzana Havránková, Ph.D.
• Řešitel: skrytý - zadáno vedoucím/školitelem
• Datum přihlášení: 29.01.2021
• Datum zadání: 29.01.2021

Seznam odborné literatury

van Aert, R. C. & M. van Assen (2021). Correcting for publication bias in a meta-analysis with the p-uniform* method. Working paper, Tilburg University & Utrecht University, available online at osf.io/preprints/metaarxiv/zqjr9/download (accessed on Feb 1, 2021).
Amini, S. M. & C. F. Parmeter (2012): Comparison of model averaging techniques: Assessing growth determinants. Journal of Applied Econometrics 27(5): pp. 870–876.
Dutton, E., van der Linden, D., & Lynn, R. (2016). The negative Flynn Effect: A systematic literature review. Intelligence, 59, 163-169.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629–634.
Eicher, T.S., C. Papageorgiou, & A.E. Raftery (2011). Default priors and predictive performance in Bayesian model averaging, with application to growth determinants. Journal of Applied Econometrics, 26(1), 30-55.
Elliott, G., Kudrin, N., & Wuthrich, K. (2021). Detecting p-hacking. Available online at arxiv.org/pdf/1906.06711 (accessed on Feb 2, 2021, under review in Econometrica).
Flynn, J. R. (1984). The mean IQ of Americans: Massive gains 1932 to 1978. Psychological Bulletin, 95(1), 29–51.
Flynn, J. R., & Weiss, L. G. (2007). American IQ gains from 1932 to 2002: The WISC subtests and educational progress. International Journal of Testing, 7(2), 209-224.
Flynn, J. R. (2009). What Is Intelligence? Beyond the Flynn Effect. Cambridge: Cambridge University Press.
Grinblatt, M., Keloharju, M., & Linnainmaa, J. (2011). IQ and stock market participation. The Journal of Finance, 66(6), 2121-2164.
George, E. I. (2010). Dilution priors: Compensating for model space redundancy. In Borrowing Strength: Theory Powering Applications–A Festschrift for Lawrence D. Brown (pp. 158-165). Institute of Mathematical Statistics.
Hafer, R. W. (2017). New estimates on the relationship between IQ, economic growth and welfare. Intelligence, 61, 92–101.
Ioannidis, J. P., Stanley, T.D., & Doucouliagos, H. (2017). The Power of Bias in Economics Research. Economic Journal 127(605), 236–265.
Jones, G., & Schneider, W. J. (2006). Intelligence, human capital and economic growth: A Bayesian averaging of classical estimates (BACE) approach. Journal of Economic Growth, 11, 71–93.
Laciga, J., & Cígler, H. (2017). The Flynn effect in the Czech Republic. Intelligence, 61, 7-10.
Matousek, J., Havranek, T., & Irsova, Z. (2021). Individual Discount Rates: A Meta-Analysis of Experimental Evidence. Experimental Economics, forthcoming.
Meisenberg, G. (2012). National IQ and economic outcomes. Personality and Individual Differences, 53(2), 103-107.
Pietschnig, J., & Voracek, M. (2015). One century of global IQ gains: A formal meta-analysis of the Flynn effect (1909–2013). Perspectives on Psychological Science, 10(3), 282-306.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A Key to The File Drawer. Journal of Experimental Psychology: General, 143(2), 534-547.
Stanley, T. D. (2005). Beyond Publication Bias. Journal of Economic Surveys 19(3), 309–345.
Stanley, T. D., Jarrell, S.B., & Doucouliagos, H. (2010). Could It Be Better to Discard 90 % of the Data? A Statistical Paradox. The American Statistician 64, 70–77.
Teasdale, T. W., & Owen, D. R. (2005). A long-term rise and recent decline in intelligence test performance: The Flynn Effect in reverse. Personality and Individual Differences, 39(4), 837-843.
Trahan, L. H., Stuebing, K. K., Fletcher, J. M., & Hiscock, M. (2014). The Flynn effect: a meta-analysis. Psychological bulletin, 140(5), 1332-1360.
Zagorsky, J. L. (2007). Do you have to be smart to be rich? The impact of IQ on wealth, income and financial distress. Intelligence, 35(5), 489-501.

Předběžná náplň práce v anglickém jazyce

Motivation

Flynn effect refers to the rise in standardized intelligence scores observed over time. Although the measurement of the effect falls under the scope of psychologists, consequences of the effect are highly relevant for any economist. As Jones & Schneider (2006, p. 21) put it, the country-level IQ “alone can explain a substantial fraction of cross-country differences in living standards.” Zagorsky (2007), for example, shows that a one-point increase in person´s IQ comes with some $400 boost in person´s income per year. Hafer (2017) suggests that such a one-point increase in average IQ is associated with a 4% increase in country´s welfare growth. Seemingly, policies that accelerate the Flynn effect could make a country richer.

But how large is the Flynn effect and what exactly drives it? The seminal work of Flynn (1984) estimates this rise at over three points in IQ gains per decade and explains it by environmental factors such as medical care, nutrition, and education. Many commentators second this conclusion (Teasdale & Owen, 2005, Laciga & Cigler, 2017, among others) but some recent studies show a reverse phenomenon, comprehensively surveyed in Dutton et al. (2016): despite a superb medical care, good nutrition, and quality education, the IQ scores stagnate or even decline. Two recent meta-analyses, Trahan et al. (2014, focused on English-speaking countries) and Pietschnig & Voracek (2015) tried to shed more light on the variation behind the scores. Their detailed inquiries predominantly agree with Flynn (1984): being healthy, rich, and smart does correlate, and we seem to be getting smarter (or getting better in abstract thinking), indeed. The two studies, however, do not account for publication bias which has been shown to dramatically exaggerate the reported estimates in many literatures (see Ioannidis et al., 2017).

In my work, I plan to build on and complement the meta-study of Pietschnig & Voracek (2015). I will update their dataset with new studies and additional variables that capture the publication heterogeneity behind the primary research. Most importantly, I will collect the representative of statistical uncertainty behind the estimates of the Flynn effect (standard errors, p-values, or confidence intervals) and search for the presence of publication selection using the newest techniques from psychology and economics research. If my hypothesis about the presence of publication bias holds, the true value of the Flynn effect could be much smaller than usually thought and the reported wealth effect accompanying the IQ change could be just a product of a spurious correlation.

Hypotheses

1. Publication bias exaggerates the Flynn effect reported in the literature.
2. After correction for publication bias, the relationship between the Flynn effect and country´s level of development (or wealth) does not hold.
3. The Flynn effect decreases in time.

Methodology

I will further update the meta-analysis of Pietschnig & Voracek (2015) with more recent data and control for publication bias (among other controls for heterogeneity).

The Flynn effect is estimated as a mean difference score (mean comparison) and is reported with some measure of error. I will translate the effect and its error into comparable metrics (say Flynn effect per year with the respective standard errors) and use them to test whether the publication selection bias presents a problem in this literature. To begin with, I will construct a funnel plot due to Egger et. al. (1997). This visual test will be, however, also tested rigorously using Funnel Asymmetry Test by Stanley (2005). I will accompany my findings with several non-linear models, such as Top10 method (Stanley et. al., 2010), weighted average of adequately powered (Ioannidis et. al., 2017), a selection model as suggested by Andrews and Kasy (2019), the stem-based method (Furukawa, 2019) and the endogenous-kink model (Bom and Rachinger, 2019).

All the tests mentioned above assume that the effect and its standard error are exogenous. If there is, however, a “mechanical” relationship between the estimates and their standard errors in the absence of publication bias, these tests might not hold. There are not many ways to relax this exogeneity assumption. One of the simplest and possibly most natural ones comes in shape of the instrumental variable estimation, where the number of observations is used as an instrument for the standard error (Matousek et al., 2021). But even more recently, the psychologists and economists developed tests that do not need to assume any relationship between the effect and its standard error (which is also an estimate by itself). One such test is the p-uniform* method due to van Aert & van Assen (2021) which assumes certain distributions about the p-values in case the publication bias is present (building on the famous p-curve by Simonhson et al., 2014). The second test by Elliot et al. (2021) also builds also on the work of Simonhson et al. (2014) but adds several restrictions for p-values based on t-tests. Note that these tests are brand new and, to my knowledge, have not been used in a published (applied) meta-analysis, yet.

The story of publication bias is central to my thesis, but I will accompany my findings with the analysis of heterogeneity in the literature on Flynn effect. After all, it could be that I am wrong on the presence of publication bias or it could be that certain methods or subgroups of studies are driving the publication bias. I will construct several control variables to capture the basic design of primary studies (in addition to those coded by Pietschnig & Voracek, 2015). To deal with the model uncertainty (inherent in meta-analysis due to large number of explanatory variables), the model averaging approach will be used. I plan on employing the Bayesian model averaging (BMA) method for my baseline model (possibly with the unit information prior suggested by Eicher et al., 2011, combined with the dilution model prior by George, 2010, which treats for collinearity in the data). As a robustness check, I will use the Frequentist model averaging method (Amini & Parmeter, 2012).

In case my results are strong, I will construct a best practice estimate of Flynn effect and give it a monetary value following Zagorsky (2007) or Hafer (2017).

Expected Contribution

My contribution will be two-fold: first, building on Pietschnig & Voracek (2015) I will test for the presence of publication bias in the literature estimating the Flynn effect and search for potential channels that could drive this bias. To do so, I will use the novel methods that test for the bias but do not have to rely on any relationship between the effect and its standard error (additionally to the standard funnel asymmetry tests). Second, I will analyse the remaining heterogeneity as Pietschnig & Voracek (2015) but will account for the model uncertainty in a different way. This analysis will help me to construct a synthetic estimate of the effect corrected for the detected biases and evaluate the effect in money terms.

Outline

1. Introduction
- Motivation, contribution, main findings
2. About the Flynn effect
- Concise background on the effect supported by major academic references
- How is the effect estimated
3. Dataset
- Search query, inclusion criteria for the studies selection
- Final table of studies
- Basic summary statistics and what does it tell us, prima-facie patterns in data?
4. Is publication bias present?
- What is it, how it can be tested, problem of the exogeneity assumption between the effect and its standard error
- Linear and non-linear techniques, novel tests relaxing the relationship between the effect and its standard error
5. What else drives the Flynn effect estimates?
- Identification of major sources of heterogeneity besides the ones pointed out in Pietschnig & Voracek (2015), including the publication bias (if any)
- Best-practice estimate is derived based on the result of the BMA analysis and the results are discussed in detail (this part might be given a new section if needed)
6. Conclusion