Homework and academic achievement: A meta-analytic review of research

Bas, G.; Senturk, C.; Cigerci, F. M.

Issues in Educational Research, 2017, Vol 27(1), 31-50
[ Contents Vol 27 ] [ IIER Home ]

Homework and academic achievement: A meta-analytic review of research

Gökhan Baş
Ömer Halisdemir University, Turkey
Cihad Şentürk and Fatih Mehmet Ciğerci
Bilecik Şeyh Edebali University, Turkey

The main purpose of this study was to determine the effect of homework assignments on students' academic achievement. This meta-analysis sought an answer to the research question: "What kind of effect does homework assignment have on students' academic achievement levels?" In this research, meta-analysis was adopted to determine the effect of homework assignments on students' academic achievement. The effect sizes of the studies included in the meta-analysis were compared with regard to their methodological characteristics (research design, sample size, and publication bias) and substantive characteristics (course type, grade level, duration of implementation, instructional level, socioeconomic status, and setting). At the end of the research, it was revealed that homework assignments had a small effect size (d = 0.229) on students' academic achievement levels. Lastly, it was seen that there was not a significant difference with regard to the effect sizes of the studies with respect to all variables, except the course type variable in the research.

Introduction

The effectiveness of doing homework has long been a controversial issue amongst educators, who have been researching the topic for 75 years (Cooper, Robinson & Patall, 2006; Corno, 2000; Trautwein, Köller, Schmitz & Baumert, 2002). Homework can be defined as "tasks assigned to students by school teachers to be carried out during non-school hours" (Cooper, 1989, p. 7). However, the definition of homework can be varied depending on the following features: purpose, degree of choice (voluntary or mandatory), amount, skill or subject area, deadline for completion (long- or short-term), and degree of individualisation and social context (completed independently or with other students in pairs or in groups) (Cooper, 2007; Coutts, 2004). The purpose of homework refers to whether homework is assigned to students for instructional and/or non-instructional purposes. However, it is very rare for homework to be assigned just for one purpose. While some homework assignments are given for instructional purposes, such as providing students with the chance of reviewing or practising the material that has already been presented in the class, some may fulfil a school's mandates. The degree of choice refers to whether a homework assignment is voluntary or mandatory. The amount of homework assignments is related to the frequency and the length and this feature of homework assignments can affect and change deadlines for completion. Depending on the amount, it may take a short or long time for students to do set homework assignments. The degree of individualisation refers to whether homework is assigned to individual students, to groups of students, or to the whole class. The social context of homework means that while some homework is assigned to students to complete independently, some may require the assistance of other persons, such as parents, and some may require the involvement of groups of students working cooperatively.

On the other hand, instead of focusing on just one purpose, teachers usually give their students assignments for several purposes. These purposes can be classified into instructional and non-instructional purposes. Among the common instructional purposes of homework are:

To provide students with opportunities for practising, reviewing or reinforcing the material already presented in the class, and determining whether students have understood the lesson and/or have obtained desired skills (Becker & Epstein, 1982; Brewster & Fager, 2000; Cooper, 2007; Pytel, 2007).
To introduce new material to be presented by the teacher and help students to get the benefit while the new material is introduced in the class (Muhlenbruck, Cooper, Nye & Lindsay, 2000; Pytel, 2007). This type of homework can be called preparation assignments.
To provide students with an opportunity to apply and integrate previously learned skills to new situations and/or other interest areas (Cooper, Robinson & Patall, 2006; Hancock, 2001; Lee & Pruitt, 1979; Shellard & Turner, 2004).
To create occasions for students to use different resources like the Internet, library, reference books, etc. (Brewster & Fager, 2000).
To enable students to use their own skills and abilities to produce creative and individualised products (Corno, 2000; Horowitz, 2005; Nuzum, 1998).

Apart from instructional purposes, homework can be assigned for non-instructional purposes:

To establish and improve communication between parents and children about homework's importance for learning (Balli, Demo & Wedman, 1998; Cooper, Robinson & Patall, 2006; Van Voorhis, 2003).
To help students acquire responsibility, self-confidence and self-discipline (Brewster & Fager, 2000; Corno, 2000; Epstein & Van Voorhis, 2001; Hetherington, 2005).

Homework has always been a controversial educational issue and a continuously debated topic (Cooper, Robinson & Patall, 2006; Trautwein, Köller, Schmitz & Baumert, 2002). At the beginning of the 20th century, there existed an anti-homework movement, especially in the USA, and educators believed that homework was far from enhancing students' learning and it was a burden for students. However, the cold war and the launch of Sputnik satellite by Russians in the 1950s reversed the negative attitudes on homework. Americans became concerned about the fact that their education system lacked rigour; therefore, assigning more homework was seen as a tool to accelerate students' acquisition of knowledge and to raise globally competitive students. For these reasons, the US Government put the National Defence Education Act (NDEA) into action in 1957, aiming to promote science and maths curricula. However, in the mid-1960s throughout the 1970s, learning theories questioned the value of homework and it was claimed that homework assignments brought excessive pressure on students. However, due to concerns about the country losing its economic competitiveness, the education system forced teachers to assign more homework assignments in the 1980s and 1990s, which also gained support from the majority of the public. Today, there is still an ongoing debate on the value and effects of homework (Brewster & Fager, 2000; Buell, 2004; Gill & Schlossman, 2000; Gill & Schlossman, 2004 Vatterott, 2009).

Purpose of the study

The purpose of this study is to examine the effect size of homework assignments given to students on their academic achievement levels in school. This study is believed to be important for policymakers, curriculum developers and teachers who direct teaching and learning processes in the classroom.

Methodology

Research design

This study used the meta-analysis method proposed by Glass, McGaw and Smith (1981). Meta-analysis can be defined as a method of statistically analysing quantitative data obtained from many studies, which are independent from each other and reaching a general conclusion about their results (Glass, 1976; Hunter & Schmidt, 2004; Lipsey & Wilson, 2001). Meta-analysis can also be defined as the analysis of other analyses (Lyons, 2003). Meta-analytic procedures require a number of steps: (a) locating all possible studies, (b) coding the studies for salient features and calculating effect sizes, and (c) carrying out statistical analyses of the effect sizes and interpreting the data acquired (Höffler & Leutner, 2007).

Data sources

The studies used in this meta-analysis came from two main sources. While the previous studies constituted the first data source, the references of these studies were examined carefully in order to reach potential studies focusing on homework and academic achievement, whereas, the second data source of the research came from the studies in relation with a literature review between 2000 and 2015 years. Several education databases such as ERIC, EBSCO, Psyc INFO, Sociological Abstracts, ULAKBIM of the Turkish Scientific and Technological Research Committee, Dissertations Abstracts International, National Dissertations Centre of the Turkish Higher Education Council, ProQuest, SSCI, SCI, etc., as well as web-based repositories such as Google Scholar were searched by using such keywords as "homework and achievement", "homework and academic achievement", "achievement and homework", "achievement and homework", "effect of homework", "homework and performance", "out of school activities and achievement", etc. As a result of this extensive search, 88 non-duplicate potential studies were identified.

Criteria for inclusion

To be included in this review of research, the studies obtained from the related literature had to meet the following criteria.

Studies in the international and national literature which took homework and academic achievement into account were included.
Studies involving students from primary school through university level were included.
Studies comparing students in experimental groups using homework with those in control groups using traditional or other methods of teaching were included.
Studies from all countries were included in the research, but these reports had to be in English or in Turkish.
Studies with experimental and control groups were included in the meta-analysis. However, studies without control groups were not.
The studies had to supply both pre-test and post-test data.
The studies had to give all the necessary statistics and the data such as sample size, mean, and standard deviation values.
Each study included in the research had to be conducted over at least three weeks.
The studies had to have taken place from 2000 to the present.

Coding procedure

In order to examine the effect of homework on academic achievement, the studies included in the meta-analysis first had to be coded. Therefore, a formal coding form for the current meta-analysis was developed and some methodological and substantive characteristics were included in this form. These characteristics encompassed ten broad distinctions: (a) research design (randomised experimental and randomised quasi-experimental), (b) sample size (N ≤ 30 = small, N < 30 = large), (c) publication bias (published and unpublished studies) in regard of the methodological characteristics; (d) course type (science, mathematics, and chemistry courses), (e) grade level (1-4, 5-8, 9 +), (f) duration of implementation (short = 3-10 week, long = 11 + week), (g) instructional level (primary school, middle school, high school, university), (h) setting (urban, rural), and (i) socioeconomic status (high, low, mixed) in terms of the substantive characteristics.

In meta-analysis studies a coding procedure is suggested (see Card, 2012). In this regard, the studies included in this meta-analysis were coded independently by two experts in educational sciences. To find their inter-rater agreement reliability the Kappa statistic proposed by Cohen (1960) was adopted. As a result of the Kappa statistics performed, the inter-rater agreement reliability of the data of the included studies was found out to be high (Kappa = .981, p < .001, 95% GA). This result shows an almost perfect inter-rater agreement for the research (see Landis & Koch, 1977).

Calculating average effect sizes

In this study, for the analysis of the data, the meta-analysis method of "procedure effectiveness" was used. The purpose of this method is to compare the effect sizes of independent variable data used in more than one study by transforming the data into a common unit of measurement (Yıldız, 2002). In the meta-analysis of procedure effectiveness, standardised effect sizes, referred to as Cohen's d or Hedges's g, are used (Başol-Göçmen, 2004; Borenstein, Hedges, Higgins & Rothstein, 2009; Hartung, Knapp & Sinha, 2008; Hedges & Olkin, 1985). The common point of these formulas is that all of them were developed for studies and designed in a way to involve a group (Yõldõrõm, 2014). The effect sizes of d and g are calculated by dividing the total standard deviation by the difference between the experimental and control group mean scores (Borenstein et al., 2009). In meta-analysis studies, calculation of the effect size is fairly important to obtain accurate findings with standard deviations and to interpret these findings (Lipsey & Wilson, 2001). In this study, for the calculation of the effect size, Hedges's g was used, and the significance level for the statistical analyses was taken as 95% (Borenstein et al., 2009; Hedges & Olkin, 1985). Furthermore, for the interpretation of the effect sizes obtained as a result of meta-analysis, coefficient classification is taken into account (Hartung, Knapp & Sinha, 2008). In the current study, while interpreting the effect sizes, the effect size classification put forward by Cohen (1992) was adopted. According to Cohen's (1992) classification of effect size coefficient, the effect size is "small" for values between 0.20 and 0.50; "medium" for values between 0.50 and 0.80, and "large" for values of 0.80 or higher.

Results

General characteristics of the studies

In this part of the research, the general characteristics of the meta-analysis findings of the studies in relation with the effect of homework on students' academic achievement were given. The studies, which took the effect of homework on students' academic achievement into account, were combined into effect sizes with standard error and variance in the current meta-analysis.

When the studies included in the current meta-analysis, which took the effect of homework on students' academic achievement into account, were examined it was seen that 27% (n = 3) of these were published journal articles, 55% (n = 6), of them were master's theses, and 18% (n = 2) of them were doctorate dissertations. Of these studies, 64% (n = 7) were carried out at elementary school level, 18% (n = 2) were conducted at high school level, and 18% (n = 2) were carried out at university level.

Seven studies from a total of 11 studies had positive effect sizes, whereas four studies had a negative effect size in the meta-analysis. Thus, it may be suggested that 64% of the studies were positive, except 36% of them indicated homework did not benefit students' academic achievement. Therefore, it may be claimed that an estimated effect size found to be as positive means that the performance is in favour of the experimental group, whereas an estimated effect size found to be as negative means that the performance is in favour of the control group (Wolf, 1986). So, most of the studies were understood to show that homework assignments were effective in the academic achievement of students. Also, it was understood that while the largest effect size was found by Özben (2006), whereas the smallest one was found by Hyde (2008). One study found a large effect size, three studies found medium, three studies found small, and four studies found unimportant effect size, according to the classifications suggested by Cohen (1992).

Overall effect sizes of the studies

A total of 11 qualifying studies were included in the analysis with a total sample size of 862 students from elementary schools (n = 323), high schools (n = 287), and universities (n = 252). As a result of the comparisons based on a total of 11 studies conducted between 2000 and 2015, the overall weighted effect size was d = 0.229 (95% CI = -0.116 Ð 0.573). The Q value indicated that the distribution of effect sizes in this collection of studies was heterogeneous, Q(10) = 59.376, p < .001 (see Table 1). In other words, the variance of the effect sizes of the studies included in the research is larger than can be explained by simple sampling error so that a random effects model was adopted (see Borenstein et al., 2009; Hedges & Olkin, 1985). Some methodological characteristics (e.g., research design, sample size, and publication bias), as well as some substantive characteristics (e.g., course type, grade level, implementation duration, instructional level, setting, and socio-economic status) were used in the current research in order to explain this variance.

Table 1: Overall effect sizes of the studies

Studies ES SE Vari-
ance 95% CI Test of mean Test of hetero-
geneity in ES

Lower Upper Z-value P-value Q-val df (Q) P-val

1. Kaplan (2006) 0.727 0.241 0.058 0.254 1.199 3.015 0.003 59.376 10 0.000

2. Özben (2006) 1.164 0.314 0.099 0.548 1.779 3.703 0.000

3. Atlı (2012) 0.078 0.303 0.092 -0.516 0.671 0.256 0.798

4. Kapıkıran and Kıran (1999) 0.091 0.367 0.135 -0.629 0.810 0.247 0.805

5. Brewer (2009) 0.194 0.168 0.028 -0.136 0.523 1.153 0.249

6. Keck (2011) -0.126 0.271 0.073 -0.657 0.405 -0.646 0.642

7. Bertsos (2005) 0.124 0.203 0.041 -0.521 0.273 -0.611 0.541

8. Gebru (2012) 0.803 0.200 0.040 0.412 1.194 4.021 0.000

9. Özcan and Erktin (2015) -0.112 0.303 0.092 -0.706 0.482 -0.371 0.711

10. Hyde (2008) -1.275 0.341 0.116 -1.944 -0.657 -3.738 0.000

11. Al-Naqbi (2014) 0.807 0.150 0.022 0.514 1.100 5.397 0.000

Fixed 0.346 0.069 0.005 0.210 0.481 4.988 0.000

Random 0.229 0.176 0.031 -0.116 0.573 1.303 0.193

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
* p < .005

Methodological characteristics of the studies

The methodological characteristics of the studies included in the study were determined as research design, sample size, year of publication, and publication bias in the research. The characteristics of the studies, which were taken into account in this research, were analysed under sub-headings below.

Research design
There were two main characteristics of research design in this review: randomised experimental and randomised quasi-experimental. Randomised experimental research design (n = 2) included students, assigned randomly to conditions of both the experimental and the control groups. Whereas, randomised quasi-experimental research design (n = 9) included students at class level by choosing the experimental and the control groups randomly for the study. The average effect size for randomised experimental studies was d = -0.125 (95% CI = -0.443 to 0.194) and d = 0.450 (95% CI = 0.300 to 0.601) for randomised quasi-experimental studies, respectively (see Table 2).

Table 2: Effect sizes of the studies by research design

Research
design k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. RE 2 -0.125 0.162 0.026 -0.443 0.194 -0.767 0.443

2. RQE 9 0.450 0.077 0.006 0.300 0.601 5.879 0.000

Total Q_B

2.843 1 0.092

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES; RE = randomised experimental research design; RQE = randomised quasi-experimental research design.
* p > .005

The results of studies using randomised experimental research design were seen not to be significantly different from those that adopted randomised quasi-experimental research designs, QB(1) = 2.843, ns. Therefore, this finding indicates that academic achievement scores do not change depending on research design.

Sample size
The studies were divided into two categories of sample size, small (N ≤ 30, n = 6) and large (N > 30, n = 5). According to the analysis conducted in the research, the average effect size for small sample sizes was d = -0.009 (95% CI = -0.264 to 0.245) and d = 0.498 (95% CI = 0.335 to 0.661) for large sample sizes, respectively (see Table 3).

Table 3: Effect sizes of the studies by sample size

Sample
size k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. Small 6 -0.009 0.130 0.017 -0.264 0.245 -0.073 0.942

2. Large 5 0.498 0.083 0.007 0.335 0.661 5.994 0.000

Total Q_B

1.967 1 0.161

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
* p > .005

A significant difference between studies with small sample sizes and the ones with larger sample sizes, was not found, QB (1) = 1.967, ns. Thus, it can be said that academic achievement scores do not change depending on sample sizes.

Publication bias
Two measures were performed in the research-classical fail-safe N analysis to reduce the average effect size to insignificant levels which is needed to increase the p-value for the meta-analysis to above .05 (Rosenthal, 1979), as well as Orwin's fail-safe N test to decide the values of criterion for a trivial log odd's ratio and mean log odds ratio in missing studies (Orwin, 1983) - in order to determine the publication bias between published (journal articles) and unpublished (master's or doctoral dissertations) sources. In this study, the classical fail-safe N analysis showed that a total of 3100 studies with null results would be required to bring the overall effect size to trivial level at .01 (Table 4).

Table 4: Results of classical fail-safe N

Z-value for observed studies
P-value for observed studies
Alpha
Tails
Z for alpha
Number of observed studies
Number of missing studies that would bring p-value to > alpha 3.801
0.00
0.05
2.00
1.95
11
3100

Also, the Orwin's fail-safe N test, which estimates the number of missing null studies that would be required to bring the average effect size to trivial level at .01, indicated that the number of missing null studies to bring the existing overall average effect sizes to .01 was found to be 503 (Table 5).

Table 5: Results of Orwin's fail-safe N

Standardised difference in means in observed studies
Criterion for a 'trivial' standardised difference means
Mean standardised difference in means in missing studies
Number of missing studies needed to bring standardised difference in means under 0.01 0.34
0.00
0.00
503

In this study, the mean effect size for the published studies (n = 3) involved was d = 0.572 (95% CI = 0.323 to 0.822), whereas the average effect size for the unpublished studies (n = 8) was d = 0.254 (95% CI = 0.090 to 0.419) (Table 6).

Table 6: Effect sizes of the studies by publication bias

Publication
type k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. Published 3 0.572 0.127 0.016 0.323 0.822 4.501 0.000

2. Unpublished 8 0.254 0.084 0.007 0.090 0.419 3.034 0.002

Total Q_B

0.082 1 0.774

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
* p > .005

According to the analysis, there was no significant difference between the effect sizes of the published and the unpublished studies in the research, QB (1) = 0.082, ns. Therefore, it may be suggested that publication bias could not account for the significant positive effects seen across all studies, which revealed that no publication bias was observed in the current research.

Substantive characteristics of the studies

The substantive characteristics of the studies included in the study were determined as course type, year of publication, grade level, implementation duration, instructional level, setting, and socio-economic status in the research. These characteristics of the studies, which were taken into account in this research, were analysed under sub-headings below.

Course type
There were three main course types in this review: science (n = 3), mathematics (n = 5) and chemistry (n = 2). The average effect size for studies involving science courses was d = 0.657 (95% CI = 0.162 to 0.026), mathematics courses d = -0.084 (95% CI = -0.731 to 0.141), and chemistry courses d = 0.806 (95% CI = 0.571 to 1.040) (Table 7).

The studies involving science and chemistry courses were seen to be significantly different from the studies involving mathematics courses, QB (2) = 14.320, p < .001. This finding indicates that academic achievement scores may depend on the course types using homework in or out of school processes. Besides, the effect size of science and chemistry courses was seen to be medium, whereas the effect size for mathematics courses was found to be quite low.

Table 7: Effect sizes of the studies by course type

Course
type k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. SC 3 0.657 0.162 0.026 0.340 0.974 4.066 0.000

2. MT 5 -0.084 0.115 0.013 -0.309 0.141 -0.731 0.465

3. CH 2 0.806 0.120 0.014 0.571 1.040 6.730 0.000

4. BI(a) 1 - - - - - - -

Total Q_B

14.320 2 0.001

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES; SC = science course; MT = mathematics course; CH = chemistry course;
(a) BI = biology course (there was only one study of a biology course; it was removed from the research for methodological reasons).
* p < .005

Grade level
There were three main characteristics in relation to grade level: 1-4 classes (n = 4), 5-8 classes (n = 4), and 9 and above classes (n = 2). The mean effect size for the studies conducted in classes 1-4 was d = 0.206 (95% CI = -0.066 to 0.478), in classes 5-8 was d = 0.412 (95% CI = 0.140 to 0.684), and d = 0.479 (95% CI = 0.243 to 0.715) for studies carried out in classes 9 and above (Table 8).

Table 8: Effect sizes of the studies by grade level

Grade
level k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. 1-4 4 0.206 0.139 0.019 -0.066 0.478 1.484 0.138

2. 5-8 4 0.412 0.139 0.019 0.140 0.684 2.966 0.003

3. 9 + 2 0.479 0.120 0.014 0.243 0.715 3.980 0.000

Total Q_B

0.757 2 0.685

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
(a) One study (Brewer, 2009) was removed from this part of the research because it included all classes, instead of focusing on a specific grade level.
* p > .005

Homework and academic achievement at the three grade levels were seen not to be significantly different, QB (2) = 0.757, ns. This finding indicates that academic achievement scores do not change depending on the grade levels using the homework in or out of school processes. In all grade spans, the absolute difference between the effect sizes was quite small. However, even though no significant differences between grade levels were found in this research, academic achievement scores of students tended to rise as the grade levels went up. So, it may be concluded that homework works well in upper grade levels, such as 5-8 and 9 and above, rather than in lower grade levels such as 1-4.

Duration of implementation
Concerning duration of implementation, the studies in this meta-analysis were divided into two categories, short (1-10 weeks, N = 7) and long (11 + weeks, N = 4). The average effect size for short implementation duration was d = 0.437 (95% CI = 0.253 to 0.622) and d = 0.238 (95% CI = 0.037 to 0.438) for long implementation duration (Table 9).

Table 9: Effect sizes of the studies by duration of implementation

Duration k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. Short 7 0.437 0.094 0.009 0.253 0.622 4.640 0.000

2. Long 4 0.238 0.102 0.010 0.037 0.438 2.327 0.020

Total Q_B

0.005 1 0.942

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
* p > .005

A significant difference was not found between studies with short implementation duration and those with long implementation duration, QB (1) = 0.005, ns. Academic achievement scores do not change depending on the duration of implementation. In both implementation durations, the absolute difference between the effect sizes was seen to be small. However, although no significant difference between implementation durations was found, it was determined that academic achievement scores of students were higher in short implementation durations compared with the longer ones.

Instructional level
There were three main instructional levels found in this review, elementary school, high school and university. The mean effect size for the studies conducted in elementary schools was d = 0.151 (95% CI = -0.069 to 0.372), in high schools d = 0.479 (95% CI = 0.243 to 0.715), and in universities d = 0.446 (95% CI = 0.194 to 0.699) (Table 10).

Table 10: Effect sizes of the studies by instructional level

Instruct.
level k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. ElemS 7 0.151 0.113 0.013 -0.069 0.372 1.343 0.179

2. HS 2 0.479 0.120 0.014 0.243 0.715 3.980 0.000

3. Uni 2 0.446 0.129 0.017 0.194 0.699 3.472 0.001

Total Q_B

0.968 2 0.616

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
ElemS = elementary school; HS = high school; Uni = university.
* p > .005

Studies comparing homework and academic achievement at the three instructional levels were not to be significantly different from each other, QB (2) = 0.968, ns. This finding indicates that academic achievement scores do not change depending on the instructional levels using homework for in or out of school processes. However, although no significant difference was found between instructional levels, it was determined that academic achievement scores of students were higher in high schools as well as universities, rather than in elementary schools.

Socioeconomic status
There were two main socioeconomic status (SES) level in this review, low level (n = 3) and mixed level (n = 7). There was only one study focusing on the effect of homework on academic achievement conducted in a high SES school, so that this study was removed from the research for methodological reasons. The average effect size for studies involving low SES was d = 0.357 (95% CI = 0.011 to 0.704) and d = 0.381 (95% CI = 0.216 to 0.546) for studies involving mixed SES (Table 11).

Table 11: Effect sizes of the studies by socioeconomic status

SES k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. Low 3 0.357 0.177 0.031 0.011 0.704 2.019 0.043

2. Mixed 7 0.381 0.084 0.007 0.216 0.546 4.525 0.000

3. High(a) 1 - - - - - - -

Total Q_B

0.194 1 0.659

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES;
(a) There was only one study in high SES; it was removed from the research for methodological reasons.
* p > .005

Studies comparing homework and academic achievement with respect to SES were not found to be significantly different, QB (1) = 0.194, ns. This indicates that academic achievement scores do not change depending on the SES level using homework for in or out of school processes. However, it was determined that academic achievement scores of students were higher in mixed SES schools, compared with students in low SES schools.

Setting
There were two main types of school setting in this review, schools in rural and urban districts/provinces. The mean effect size for the studies conducted in rural schools was d = 0.315 (95% CI = -0.018 to 0.648) and in urban schools was d = 0.352 (95% CI = 0.203 to 0.500) (Table 12).

Table 12: Effect sizes of the studies by school setting

School
setting k ES SE Variance 95% CI Test of mean Test of heterogeneity in ES

Lower Upper Z-value P-value Q-value df (Q) P-value

1. Rural 3 0.315 0.170 0.029 -0.018 0.648 1.855 0.064

2. Urban 8 0.352 0.076 0.006 0.203 0.500 4.635 0.000

Total Q_B

0.163 1 0.686

Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES.
* p > .005

Studies comparing homework and academic achievement in regard to school setting were seen to be not significantly different from each other, QB (1) = 0.163, ns. This indicates that academic achievement scores do not change depending on the school setting using homework in or out of school processes. However, academic achievement scores of students were higher in urban schools, compared to rural schools.

Discussion

The aim of this study was to use the meta-analysis method to examine research on the effects of homework on students' academic success. Research studies were categorised under general characteristics (overall effect sizes), methodological characteristics (research design, sample size and publication bias) and substantive characteristics (course type, grade level, duration of implementation, instructional level, socioeconomic status, and setting). In examining publication bias (Rothstein, Sutton & Borenstein, 2005), only empirical researches with experimental and control groups were included in the study. The results of meta-analysis revealed that 64% of the research studies selected had findings reflecting the positive effects of homework on students' academic success. Although there are many research studies showing the positive effects of homework on academic success (e.g., Dodson, 2014; Hein & Wimer, 2007; Glazer & Williams, 2001; Grodner & Rupp, 2013; Gustafsson, 2013; Pelletier & Normore, 2007; Planchard, Daniel, Maroo, Mishra & McLean, 2015; Su, Huang, Yang, Ding & Hsieh, 2015; Trautwein, Köller, Schmitz & Baumert, 2002; Voorhis, 2003), some of them found that homework does not always affect academic success in general, but may have partial effects on academic success in certain courses, exams or classroom grades (e.g., Hong, Peng & Rowell, 2009; Murillo & Martinez-Garrido, 2014). Others showed that homework does not influence academic success at all (e.g., Booth, 2010; Kapıkıran & ı, 1999).

Since this research has made use of various samples consisting of different features, this study seeks to benefit from this knowledge effectively, to comment on it and to find the common effect size of the effects of homework on students' academic success. The studies examining the effects of homework on students' academic success between 2006 and 2014 were compiled with a meta-analysis method. Meta-analysis results revealed that, according to the fixed effects model, the effect size of the research studies contains heterogeneous features. Thus, a random effects model was used to determine the effects of homework on students' academic success. According to this model, the effect size value of the current study is d = 0.229. This value is positive and low in level according to Cohen's (1992) effect size categorisation. Cooper and Valentine (2001) reached similar outcomes by finding d = 0.21 in the overall general effect in their meta-analysis. Additionally, in their meta-synthesis Cooper, Robinson and Patall (2006) found a positive relationship between homework and academic success, which implies a similarity with this study.

The research works examined in this study were recognised under the titles of methodological characteristics, research design, sample size and publication bias. The research design title included randomised experimental research and randomised quasi-experimental research. This study's results showed no significant differences between randomised experimental research and randomised quasi-experimental research designs, QB (1) = 2.843. In their meta-analysis, Cooper, Robinson and Patall (2006) examined research work that used random and equivalent assigned groups and did not discover a meaningful difference between the groups. That is, the effects of homework on academic success did not differentiate according to the research design and results. Considering the sample size factors, there are not any meaningful differences between small sample sizes and large sample sizes, QB (1) = 1.967. In the light of this information, it can be said that the effects of homework on academic success does not differentiate in terms of sample size. Using quantitative synthesis of research, some other studies found that students doing homework are more successful than those not doing homework, whereas studies making use of homogeneity analyses revealed negative relationships. This difference among various research outcomes cannot be explained as due only to sampling uncertainty (Cooper & Valentine, 2001). The analysis between published and unpublished studies also resulted in no meaningful differences, QB (1) = 0.082. Since there are various factors affecting homework's role in students' academic success, the research design and methods for future studies in should be carefully chosen, considering all possible factors in the process.

The substantive characteristics in this study were considered under the titles of course type, grade level, duration of implementation, instructional level, socioeconomic status and setting. In course type, there were three science courses, five mathematics courses, two chemistry courses and one biology course (the last was removed from the research for methodological reasons). The effect sizes for these courses are science d = 0.657, chemistry d = 0.806 and mathematics d = -0.084. The effect of homework given in science and chemistry courses is significantly different from that given in mathematics courses, QB (2) = 14.320, p < .001. Considering these results, according to Cohen, Manion and Morrison's (2007) effect size classification, homework given in the chemistry courses has positive and quite influential contributions to academic achievement. Homework given in science courses has positive and strong effects on academic success. On the other hand, homework given in mathematics courses has negative and low level effects on academic success. This difference can stem from the qualitative differences in the given homework.

For grade level, the researcher included four studies in the grades 1 to 4 range, four studies in grades 5 to 8, and two studies from higher classes. This research did not differentiate between homework's effects on academic success in terms of grade levels. So, it can be stated that homework given in various grade levels does make a difference in terms of its effect on students' academic success. Although there are no differences among grade levels, as the grade levels rise, the given homework appears to increase student's academic success. Cooper (1989), Cooper and Valentine (2001) and Cooper, Robinson and Patall (2006) all shared this same observation in their work. A possible explanation for this phenomenon is that as the grade level rises, students become older, become more responsible, acquire higher levels of awareness, improve in other developmental domains, increase their knowledge, and can prepare more elegant and qualified homework as their skills such as problem solving, critical thinking, cooperative learning, attention and concentration improve (see Bempechat, 2004; Hoover-Demspey et al., 2001; Muhlenbruck, Cooper, Nye, & Lindsay, 2000). Additionally, in upper grades students tend to improve skills such as individual studying and self-learning, which again contributes to academic success.

Research studies were compared in terms of their empirical implementation durations, which found no meaningful differences, QB (1) = 0.005. However, the effects of homework on students' academic success were found to be higher in short-term studies than in long-term studies. Following the meta- analysis, the effects of homework on students' academic success were also examined in terms of instructional level, though no meaningful differences were revealed in this variable. Although there was no difference in the instructional level variable, homework given in high school and university levels increases students' academic success more than the homework given during primary school.

Similarly, there were not any significant findings suggesting that socio-economic status plays a role in homework's effects on students' academic success, QB (1) = 0.194. Although there is no meaningful difference among the studies considering the effect size, in the schools where socio-economic status were generally mixed, the given homework increased students' academic success more than in schools where the socio-economic level is low or high. Some of the research was conducted in rural areas while some were in urban areas. Concerning the geographical areas where the research were conducted, no meaningful differences were found, QB (1) = 0.163. Despite there being no significant differences among studies about type of area and homework's effects on students' academic success, the homework given in urban schools seemed to increase students' academic success points more than that given in rural schools. The reason for this might be that students at urban schools have more and varied opportunities than the students at rural schools.

There are some scientifically unsound designs in the research studying the effects of homework on students' academic level. In some studies, many factors were neglected, including quantity/quality of homework, duration, access to parental or peer help, guidance services to parents and students, feedback, access to resources and technological support, teacher qualifications, time students spend on homework, and pre-knowledge of students about research methods, which can have an effect on study findings. Bryan and Nelson (1994) revealed in their studies that students find homework boring and develop a negative attitude towards courses because of it. Hence, such variables as the roles of teachers and parents, the quality, quantity and duration of homework, the appropriateness for students' developmental level, guidance and feedback to students, are to be taken into consideration when estimating homework's effects on academic success.

Acknowledgment

An initial presentation of this research was given at the VI International Congress on Research in Education, held in the Faculty of Education, Recep Tayyip Erdoğan University, Rize, Turkey.

References

Studies marked with asterisk (*) were included in the meta-analysis

*Al-Naqbi, A. K. (2014). The effects of instructional homework technique on chemistry achievement of the United Arab Emirates male and female tenth graders. International Journal for Research in Education, 35, 1-27. https://www.researchgate.net/publication/
265602747_The_Effects_of_Instructional_Homework_Technique_on_Chemistry_Achievement_of_the_
United_Arab_Emirates_Male_and_Female_Tenth_Graders

*Atlı, S. (2012). Effect of homework given in 4th grade science and technology course on students' concept learning, academic achievement, and attitudes towards homework. Unpublished master's thesis, Niğde University, Niğde, Turkey.

Balli, S. J., Demo, D. H. & Wedman, J. F. (1998). Family involvement with children's homework: An intervention in the middle grades. Family Relations, 47(2), 149-157. http://www.jstor.org/stable/585619

Başol-Göçmen, G. (2004). A general revision of meta-analysis. Sakarya Üniversitesi Eğitim Fakültesis Dergisi, 7, 186-192.

Becker, H. J. & Epstein, J. L. (1982). Parent involvement: A survey of teacher practices. The Elementary School Journal, 83(2), 85-102. http://www.journals.uchicago.edu/doi/abs/10.1086/461297?journalCode=esj

Bempechat, J. (2004). The motivational benefits of homework: A social-cognitive perspective. Theory into Practice, 43(3), 189-196. https://muse.jhu.edu/article/171674/pdf

*Bertsos, G. F. (2005). Differentiating biology homework to enhance academic achievement. Unpublished master's thesis, Eastern Michigan University, Ypsilanti, Michigan, USA. http://commons.emich.edu/theses/126/

Booth, G. I. (2010). The effects of homework assessment on student motivation and achievement. Unpublished master's thesis, Central Washington University, Washington, USA.

Borenstein, M., Hedges, L. V., Higgins, J. P. T. & Rothstein, H. R. (2009). Introduction to meta-analysis. New York: John Wiley and Sons.

*Brewer, D. S. (2009). The effects of online homework on achievement and self-efficacy of college algebra students. Unpublished doctoral dissertation, Utah State University, Logan, Utah, USA. http://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=1414&context=etd

Brewster, C. & Fager, J. (2000). Increasing student engagement and motivation: From time-on-task to homework. Northwest Regional Educational Laboratory. http://educationnorthwest.org/webfm_send/452

Bryan, T. & Nelson, C. (1994). Doing homework: Perspectives of elementary and junior high school students. Journal of Learning Disabilities, 27(8), 488-499. http://dx.doi.org/10.1177/002221949402700804

Buell, J. (2004). Closing the book on homework: Enhancing public education and freeing family time. Philadelphia, PA: Temple University Press.

Card, N. A. (2012). Applied meta-analysis for social science research. New York: The Guilford Press.

Cohen, L., Manion, L. & Morrison, K. (2007). Research methods in education (6th ed.). New York: Taylor & Francis.

Cohen, J. (1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98-101. http://dx.doi.org/10.1111/1467-8721.ep10768783

Cooper, H. (2007). The battle over homework: Common ground for administrators, teachers, and parents. Thousand Oaks, CA: Corwin Press.

Cooper, H., Robinson, J. C. & Patall, E. A. (2006). Does homework improve academic achievement? A synthesis of research, 1987-2003. Review of Educational Research, 76(1), 1-62. http://dx.doi.org/10.3102/00346543076001001

Cooper, H. & Valentine, J. C. (2001). Using research to answer practical questions about homework. Educational Psychologist, 36(3), 143-153. http://dx.doi.org/10.1207/S15326985EP3603_1

Cooper, H. (1989). Homework. New York: Longman.

Corno, L. (2000). Looking at homework differently. The Elementary School Journal, 100(5), 529-548. http://dx.doi.org/10.1086/499654

Coutts, P. M. (2004). Meanings of homework and implications for practice. Theory into Practice, 43(3), 182-188. http://dx.doi.org/10.1207/s15430421tip4303_3

Dodson, J. R. (2014). The impact of online homework on class productivity. Science Education International, 25(4), 354-371. http://files.eric.ed.gov/fulltext/EJ1052920.pdf

Epstein, J. L. & Van Voorhis, F. L. (2001). More than minutes: Teachers' roles in designing homework. Educational Psychologist, 36(3), 181-193. http://dx.doi.org/10.1207/S15326985EP3603_4

*Gebru, M. T. (2012). The effects of clickers and online homework on students' achievement in general chemistry. Unpublished doctoral dissertation, Middle Tennessee State University, Murfreesboro, TN, USA. http://jewlscholar.mtsu.edu/bitstream/handle/mtsu/3872/3514922.pdf?sequence=1

Gill, B. P. & Schlossman, S. L. (2004). Villain or savior? The American discourse on homework, 1850-2003. Theory into Practice, 43(3), 174-181. http://www.history.cmu.edu/docs/schlossman/Villiain-or-Savior.pdf

Gill, B. P. & Schlossman, S. L. (2000). The lost cause of homework reform. American Journal of Education, 109(1), 27-62. http://www.jstor.org/stable/1085422

Glass, G. V., McGaw, B. & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: SAGE.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. http://dx.doi.org/10.3102/0013189X005010003

Glazer, T. N. & Williams, S. (2001). Averting the homework crisis. Educational Leadership, 58(7), 43-45. http://www.ascd.org/publications/educational-leadership/apr01/vol58/num07/Averting-the-Homework-Crisis.aspx

Gustafsson, J. E. (2013). Causal inference in educational effectiveness research: A comparison of three methods to investigate effects of homework on student achievement. School Effectiveness and School Improvement, 24(3), 275-295. http://dx.doi.org/10.1080/09243453.2013.806334

Grodner, A. & Rupp, N. G. (2013). The role of homework in student learning outcomes: Evidence from a field experiment. The Journal of Economic Education, 44(2), 93-109. http://dx.doi.org/10.1080/00220485.2013.770334

Hancock, J. (2001). Homework: A literature review. Occasional Paper No. 37, Center for Research and Evaluation, University of Maine, Orono, ME. http://libraries.maine.edu/cre/37/No37.htm

Hartung, J., Knapp, G. & Sinha, B. K. (2008). Statistical meta-analysis with applications. New York: Wiley.

Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.

Hein, D. L. & Wimer, S. L. (2007). Improving homework completion and motivation of middle school students through behavior modification, graphing and parent communication. Unpublished master's thesis, Saint Xavier University, Chicago, USA. http://eric.ed.gov/?id=ED498932

Hong, E., Peng, Y. & Rowell, L. L. (2009). Homework self-regulation: Grade, gender, and achievement-level differences. Learning and Individual Differences, 19(2), 269-276. http://dx.doi.org/10.1016/j.lindif.2008.11.009

Hoover-Dempsey, K. V., Battiato, A. C., Walker, J. M. T., Reed, R. P., DeJong, J. M. & Jones, K. P. (2001). Parental involvement in homework. Educational Psychologist, 36(3), 195-209. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.456.9721&rep=rep1&type=pdf

Horowitz, S. H. (2005). Research roundup. National Center for Learning Disabilities. Retrieved from http://www.ncld.org/content/view/577 [not found 9 Nov 2016]

Höffler, T. N. & Leutner, D. (2007). Instructional animation versus static pictures: A meta-analysis. Learning and Instruction, 17(6), 722-738. http://dx.doi.org/10.1016/j.learninstruc.2007.09.013

Hunter, J. E. & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: SAGE.

*Hyde, M. (2008). The effects of math homework on student achievement in the fourth grade. Unpublished master's thesis, Kennesaw State University, Kennesaw, USA. https://commons.kennesaw.edu/gpc/sites/commons.kennesaw.edu.gpc/files/Paper%20Hyde_0.pdf

*Kapıkıran, Ş. & Kıran, H. (1999). The effect of homework on student's academic success. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 5, 54-60.

*Kaplan, B. (2006). The effect of homework to success of students and learning concepts in unit of "electricity which directs our life style". Unpublished master's thesis, Marmara University, Istanbul, Turkey.

*Keck, A. E. (2011). Differentiating homework and its effects on achievement: A case study of two classes. Unpublished master's thesis, LaGrange College, Lagrange, Georgia, USA. http://home.lagrange.edu/educate/Advanced%20Programs/M.Ed.%20Defense%20Assessment/Keck%20final%20final.docx

Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. http://www.jstor.org/stable/2529310

Lee, J. F. & Pruitt, K. W. (1979). Homework assignments: Class games or teaching tools? Clearing House, 53(1), 31-35. http://dx.doi.org/10.1080/00098655.1979.9957112

Lipsey, M. & Wilson, D. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.

Lyons, L. C. (2003). Meta-analysis: Methods for accumulating results across research domains. http://lyonsmorris.com/MetAa/

Muhlenbruck, L., Cooper, H., Nye, B. & Lindsay, J. J. (2000). Homework and achievement: Explaining the different strengths of relation at the elementary and secondary school levels. Social Psychology of Education, 3(4), 295-317. http://dx.doi.org/10.1023/A:1009680513901

Murillo, F. J. & Martinez-Garrido, C. (2014). Homework and primary-school students' academic achievement in Latin America. International Review of Education, 60(5), 661-681. http://dx.doi.org/10.1007/s11159-014-9440-2

Nuzum, M. (1998). Creating success. Instructor, 108(3), 86-91. http://eric.ed.gov/?id=EJ576389

Orwin, R. G. (1983). A fail-safe N for effect size in meta analysis. Journal of Educational Statistics, 8(2), 157-159. http://www.jstor.org/stable/1164923

*Özben, B. G. (2006). Effect of homework studies in elementary second level science course on students achievement. Unpublished master's thesis, Gazi University, Ankara, Turkey.

*Özcan, Z. Ç. & Erktin, E. (2015). Enhancing mathematics achievement of elementary school students through homework assignments enriched with metacognitive questions. Eurasia Journal of Mathematics, Science & Technology Education, 11(6), 1415-1427. http://www.iserjournals.com/journals/eurasia/articles/10.12973/eurasia.2015.1402a

Pelletier, R. & Normore, A. H. (2007). The predictive power of homework assignments on student achievement in mathematics. In S. M. Nielsen & M. S. Plakhotnik (Eds.), Proceedings of the Sixth Annual College of Education Research Conference: Urban and International Education Section (pp. 84-89). Miami: Florida International University. http://digitalcommons.fiu.edu/cgi/viewcontent.cgi?article=1266&context=sferc

Planchard, M., Daniel, K. L., Maroo, J., Mishra, J. & McLean, T. (2015). Homework, motivation, and academic achievement in a college genetics course. Bioscene: Journal of College Biology Teaching, 41(2), 11-18. http://files.eric.ed.gov/fulltext/EJ1086528.pdf

Pytel, B. (2007). Homework - What research says. [not found 5 Jan 2017] http://educationalissues.suite101.com/article.cfm/homework_what_research_says

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. http://psycnet.apa.org/doi/10.1037/0033-2909.86.3.638

Rothstein, H. R., Sutton, A. J. & Borenstein, M. (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. West Sussex, England: Wiley.

Shellard, E. G. & Turner, J. R. (2004). Homework: Research and best practice. Arlington, VA: ERS Focus on Educational Research Service.

Su, A. Y. S., Huang, C. S. J., Yang, S. J. H., Ding, T. J. & Hsieh, Y. Z. (2015). Effects of annotations and homework on learning achievement: An empirical study of Scratch programming pedagogy. Educational Technology & Society, 18(4), 331-343. http://www.ifets.info/journals/18_4/25.pdf

Trautwein, U., Köller, O., Schmitz, B. & Baumert, J. (2002). Do homework assignments enhance achievement? A multilevel analysis in 7th-grade mathematics. Contemporary Educational Psychology, 27(1), 26-50. http://dx.doi.org/10.1006/ceps.2001.1084

Van Voorhis, F. L. (2003). Interactive homework in middle school: Effects on family involvement and science achievement. The Journal of Educational Research, 96(6), 323-338. http://dx.doi.org/10.1080/00220670309596616

Vatterott, C. (2009). Rethinking homework: Best practices that support diverse needs. Alexandria, VA: Association for Supervision and Curriculum Development.

Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Thousand Oaks, CA: SAGE.

Yıldırım, N. (2014). Meta-analysis. In M. Metin (Ed.), Scientific research methods in education: Theory into practice (pp. 138-159). Ankara: Pegem Akademi.

Authors: Dr Gökhan Baş is an assistant professor at the Faculty of Education, Ömer Halisdemir University, Niğde, Turkey. His research interests include curriculum and instruction, teaching-learning process, and educational measurement and evaluation.
Email: gokhanbas51@gmail.com

Cihad Şentürk is a lecturer at the School of Health Services, Bilecik Şeyh Edebali University, Bilecik, Turkey. His research interests include curriculum and instruction, teaching-learning process, and primary education.
Email: cihadsenturk@gmail.com

Dr Fatih Mehmet Ciğerci is an assistant professor at the School of Health Services, Bilecik Şeyh Edebali University, Bilecik, Turkey. His research interests include primary education and qualitative research.
Email: fatihcigerci@gmail.com

Please cite as: Baş, G., Şentürk, C. & Ciğerci, F. M. (2017). Homework and academic achievement: A meta-analytic review of research. Issues in Educational Research, 27(1), 31-50. http://www.iier.org.au/iier27/bas.html

[ PDF version of this article ] [ Contents Vol 27 ] [ IIER Home ]
This URL: http://www.iier.org.au/iier27/bas.html
Created 15 Jan 2017. Last revision: 15 Jan 2017.
HTML: Roger Atkinson [rjatkinson@bigpond.com]

Studies		ES	SE	Vari- ance	95% CI		Test of mean		Test of hetero- geneity in ES
Studies		ES	SE	Vari- ance	Lower	Upper	Z-value	P-value	Q-val	df (Q)	P-val
1.	Kaplan (2006)	0.727	0.241	0.058	0.254	1.199	3.015	0.003	59.376	10	0.000
2.	Özben (2006)	1.164	0.314	0.099	0.548	1.779	3.703	0.000
3.	Atlı (2012)	0.078	0.303	0.092	-0.516	0.671	0.256	0.798
4.	Kapıkıran and Kıran (1999)	0.091	0.367	0.135	-0.629	0.810	0.247	0.805
5.	Brewer (2009)	0.194	0.168	0.028	-0.136	0.523	1.153	0.249
6.	Keck (2011)	-0.126	0.271	0.073	-0.657	0.405	-0.646	0.642
7.	Bertsos (2005)	0.124	0.203	0.041	-0.521	0.273	-0.611	0.541
8.	Gebru (2012)	0.803	0.200	0.040	0.412	1.194	4.021	0.000
9.	Özcan and Erktin (2015)	-0.112	0.303	0.092	-0.706	0.482	-0.371	0.711
10.	Hyde (2008)	-1.275	0.341	0.116	-1.944	-0.657	-3.738	0.000
11.	Al-Naqbi (2014)	0.807	0.150	0.022	0.514	1.100	5.397	0.000
Fixed		0.346	0.069	0.005	0.210	0.481	4.988	0.000
Random		0.229	0.176	0.031	-0.116	0.573	1.303	0.193
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. * p < .005

Research design	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Research design	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. RE	2	-0.125	0.162	0.026	-0.443	0.194	-0.767	0.443
2. RQE	9	0.450	0.077	0.006	0.300	0.601	5.879	0.000
Total Q_B									2.843	1	0.092
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES; RE = randomised experimental research design; RQE = randomised quasi-experimental research design. * p > .005

Sample size	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Sample size	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. Small	6	-0.009	0.130	0.017	-0.264	0.245	-0.073	0.942
2. Large	5	0.498	0.083	0.007	0.335	0.661	5.994	0.000
Total Q_B									1.967	1	0.161
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. * p > .005

Publication type	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Publication type	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. Published	3	0.572	0.127	0.016	0.323	0.822	4.501	0.000
2. Unpublished	8	0.254	0.084	0.007	0.090	0.419	3.034	0.002
Total Q_B									0.082	1	0.774
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. * p > .005

Course type	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Course type	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. SC	3	0.657	0.162	0.026	0.340	0.974	4.066	0.000
2. MT	5	-0.084	0.115	0.013	-0.309	0.141	-0.731	0.465
3. CH	2	0.806	0.120	0.014	0.571	1.040	6.730	0.000
4. BI(a)	1	-	-	-	-	-	-	-
Total Q_B									14.320	2	0.001
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES; SC = science course; MT = mathematics course; CH = chemistry course; (a) BI = biology course (there was only one study of a biology course; it was removed from the research for methodological reasons). * p < .005

Grade level	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Grade level	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. 1-4	4	0.206	0.139	0.019	-0.066	0.478	1.484	0.138
2. 5-8	4	0.412	0.139	0.019	0.140	0.684	2.966	0.003
3. 9 +	2	0.479	0.120	0.014	0.243	0.715	3.980	0.000
Total Q_B									0.757	2	0.685
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. (a) One study (Brewer, 2009) was removed from this part of the research because it included all classes, instead of focusing on a specific grade level. * p > .005

Duration	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Duration	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. Short	7	0.437	0.094	0.009	0.253	0.622	4.640	0.000
2. Long	4	0.238	0.102	0.010	0.037	0.438	2.327	0.020
Total Q_B									0.005	1	0.942
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. * p > .005

Instruct. level	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
Instruct. level	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. ElemS	7	0.151	0.113	0.013	-0.069	0.372	1.343	0.179
2. HS	2	0.479	0.120	0.014	0.243	0.715	3.980	0.000
3. Uni	2	0.446	0.129	0.017	0.194	0.699	3.472	0.001
Total Q_B									0.968	2	0.616
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. ElemS = elementary school; HS = high school; Uni = university. * p > .005

School setting	k	ES	SE	Variance	95% CI		Test of mean		Test of heterogeneity in ES
School setting	k	ES	SE	Variance	Lower	Upper	Z-value	P-value	Q-value	df (Q)	P-value
1. Rural	3	0.315	0.170	0.029	-0.018	0.648	1.855	0.064
2. Urban	8	0.352	0.076	0.006	0.203	0.500	4.635	0.000
Total Q_B									0.163	1	0.686
Notes. k = number of effect sizes; ES = effect sizes; SE = standard error; CI = confidence of interval for the average value of ES. * p > .005