Critical Appraisal Statistical Inference – Article by Tamir (1988)

Critical appraisal statistical inference - article by Tamir (1988)

Write a critical appraisal of the author's use of statistical inference and more general issues of presentation in the article by Tamir (1988).

Introduction

Academic achievement throughout school has a major impact on the opportunities which are available to individuals with regards to further education and once they leave the educational system. Improving levels of educational attainment in the UK is a consistent component of government agenda, with a particular focus on lessening inequalities across the population (Cassen & Kingdon, 2007, online). Understanding factors which influence academic achievement is therefore clearly important in the design of effective educational policy. It has been relatively well established that there are gender differences in general educational attainment. Although this is less marked in the UK than in many other countries, it is a general trend across Europe and internationally (Buchmann et al., 2008, p. 319; Eurydice, 2009, p. 74). There have been numerous studies which have sought to understand whether these differences apply within specific subjects. One example of such a study was conducted by Tamir et al. (1988, p. 128), and aimed to investigate whether there were differences in achievement and experiences between boys and girls in high school science. This essay presents a critical analysis of the study with regards to the statistical analysis performed and the conclusions inferred. The first section presents a brief overview of the study, followed by a more in-depth discussion of the statistical inference used.

Study Overview

The study sought to evaluate gender differences in science achievement and attitudes towards science among Israeli 12th grade students. The sample included both those majoring in science, be that physics, chemistry or biology, and those not majoring in science. Data was collected using both surveys and objective testing instruments. Overall, there were 10 different dimensions of high school science experiences which were investigated within the study including achievement, attitudes towards science both for their own personal use and for society, preferences with regards to science and learning, and their learning and study experiences. Socioeconomic status was also considered as an additional explanatory variable. The authors concluded on the basis of their study that physics was predominantly chosen by boys while biology was chosen by more girls. The study also indicated that boys had a preference towards science subjects, had more positive attitudes towards science, viewed themselves as higher achievers in this area, and were more inclined towards scientific careers. The study also indicated that boys were higher achievers in physics and earth sciences than girls and that non-science majors were relatively scientifically illiterate.

Academic Achievement

All participants were administered a general science test. Those studying one of the three individual majors were also administered a separate test based on this major. In addition, understanding of science was measured using the 20-item Understanding of Science Measure (SUM). The use of objective testing is important in achieving valid measures of student academic achievement. Although the two are linked, internal and external factors may also interfere so that self-concept may not always match actual performance; this could therefore skew results (Caprara et al., 2011, p. 78). Similar problems could also be seen with regards to self-reported attitudes (Alexander & Winne, 2006, p. 330). It is also important that both were measured, as there is evidence that objectively measured academic achievement and subjective attitudes may not directly correlate (See & Khoo, 2011, p. 180). Data were analysed using SPSS. It is not made entirely clear which statistical tests were used to analyse the data, but the presence of a column marked t in the results tables would appear to indicate that the students' t test was used. This would have been an appropriate test to use for comparison of two samples, in this case taking boys and girls as separate samples and analyzing for a difference with respect to other variables. However, it is possible that some of the assumptions of this test may have been violated. First, the test assumes that all observations are independent (Gliner & Morgan, 2009, p. 219). However there is evidence that peer influence may be a major factor determining attitudes towards science, which could indicate that these observations are not independent. Teacher influence is also important, so there could be trends within each class (Talton & Simpson, 1985, p. 19; George, 2006, p. 571). Also, the test relies on all observations coming from a normal distribution (Gliner & Morgan, 2009, p. 219). A sample size of 2 153 pupils was used. This was generated using random sampling from a sampling frame of 68 different high schools and 137 12th Grade classes. The sample was stratified according to the major studied by the pupil, although there were different numbers of each included: 926 in biology, 484 in physics and 249 in chemistry. This was based on inclusion of 68 biology classes, 39 physics classes and 18 chemistry classes. As a control group there were also 404 non-science majors included, from 26 different classes. As a large sample size was used, it would be expected that the distribution would approach normal, therefore this assumption may be upheld (Underhill & Bradfield, 2004, p. 8). However the assumption that the variance of the dependent variables is approximately equal could be called into question given the unequal sizes of the different groups (Gliner & Morgan, 2009, p. 219). The authors inferred that boys excel in physics and application. This was based on the results for these areas within the general science test (3M), as well as in the tool specifically measuring achievement in physics (3P). Although all study participants were reportedly administered the 3M general science test, this is inconsistent with the results presented, which include a total of only 1590 participants. The number of boys taking the physics 3P test was also much larger than the number of girls, possibly leading to questioning of the assumptions underlying the t test (Gliner & Morgan, 2009, p. 219). The authors also concluded that the girls outperformed boys on the biology test 3B. Here too, the number of girls was much larger than the number of boys in the sample. This also appeared to be inconsistent with the results of the general science test 3M. On this same basis, the authors concluded that there was no difference in chemistry achievement for those majoring in one of the subjects. There was no set of results presented for the general science test broken down according to major studied. This means that it is difficult to determine whether this is the case, or whether this conclusion could also be applied to those not majoring in science. There was no evidence of confounding, as there was no individual area in which girls significantly outperformed boys. This same trend was also seen in the non-science participants. The authors also concluded that non-science students are scientifically illiterate. Although this could be judged to be true when comparing the results achieved by participants in the 3M and 3N tests, it is not clear whether these two tests were different. In addition, there were a much larger number of participants completing the 3M than the 3N test.

Attitudes Towards Science

Understanding the attitude of pupils towards science is also important as this is linked to achievement (Koballa & Crawley, 1985, p. 222). Inference regarding major subject choice was that girls had a preference for biology and non-science, while boys had a preference for physics. This was appropriate, based simply on comparison of proportions studying each subject. Attitudes towards science were measured using a 40-item scale (3ATT). Scientific or cognitive preference was measured using a 20-item Cognitive Preference Inventory (PREF). One issue with the use of this type of instrument is that there needs to be some assessment to ensure that there is consistency across items in measuring a given construct (See & Khoo, 2011, p. 181). The authors concluded that boys had more positive attitude towards science than girls. This was justified according to analysis of the specific questions regarding science being important to both themselves and to society. However given that the attitude scale (3ATT) had 40 items, this represents only a small number of the questions to which responses were generated. Similar could also be said of conclusions drawn on attitudes toward science learning, attitude toward learning and school, interest in biology and inquiry experiences, and self-perception of achievement in science and math. As with attitudes toward science, the tables presented by the authors provided only a small snapshot of the completed survey instruments. Therefore it is difficult to determine whether the responses provided to other items were also consistent with the inferred conclusions. The authors concluded that there were no differences between the preferences of boys and girls with regard to cognitive modes. However there were no results presented from which to understand whether there could have been small differences. There was also no analysis according to stratification of participants in terms of major. The authors concluded that there were gender differences with regard to science career orientations. They also concluded that there was an almost perfect match between the subjects which the pupils chose to major in and their future career aspirations. Both of these conclusions were justified by the data presented and the results of the t test analysis.

Other Explanatory Factors

It would be argued that the major omission from this study was the failure to relate the different variables on which data were collected. This could have provided a more detailed picture of the gender differences and could have provided a more realistic picture of the interaction between different factors in determining pupils' choices with regards to science education. Importantly it could also have provided more information on whether differences observed were all due to gender individually or whether other factors contributed more towards differences, but were perhaps themselves influenced by gender. This would be expected on the basis of numerous other research studies which have indicated that there is a complex interplay of factors determining achievement in science, including internal and external factors (Wolf & Fraser, 2008, p. 321). One of the most important findings reported by the authors may relate to the differences in socioeconomic status between boys and girls in the study sample, which has been shown to be an important influence on academic achievement by other authors (Sirin, 2005, p. 417). Their results indicated that boys had a higher socioeconomic status on average. They suggested that this implied more girls of lower socioeconomic status elected to major in science in high school. However it is not possible to infer causality from this type of statistical test. It could instead be that more boys of higher socioeconomic status choose to major in science, or a combination of the two. This would appear to be more consistent with their conclusion that boys with lower socioeconomic status tend to select vocational courses. It doesn't explain what girls of higher socioeconomic status choose to major in. It could also be merely a coincidental finding. Importantly, this inequality in socioeconomic status could also be a potential explanatory factor for some of the other differences between boys and girls reported in the study. The authors concluded that this was not the case with respect to analysis of the data on physics achievement. Analysis of covariance when SES was maintained at a constant level demonstrated that there remained significant differences between boys and girls in this respect. However it may have been more pertinent to perform multivariate regression analysis on this data instead. There was unfortunately no inclusion of this variable in the models used to analyse the differences in other factors.

Conclusions

Overall, the statistical inference of the authors was accurate in identifying simple trends in the data. However more complex statistical analysis, such as the use of multiple regression modeling could have better elucidated the relationships between the variables explored. There is some consideration given by the authors as to the implications of their findings, but this type of analysis could have provided results more relevant to practice. There is also little discussion of the generalizability of the findings to the wider population. Yet this may be an important consideration if the results were to be used to shape educational policy and programme design. It would be suggested that the results could only be partially generalizable, as the study sample would not reflect the same type of distribution of characteristics of individuals in other countries. For example as the study was conducted in Israel all those included in the sample were Jewish.

References

Alexander, P.A. & Winne, P.H. (2006) Handbook of Educational Psychology – 2nd Edition. Mahwah, NJ: Lawrence Erlbaum Associates, p. 330. Buchmann, C., DiPrete, T.A. & McDaniel, A. (2008) Gender inequalities in education. Annual Review of Sociology, 34, 319-337. Caprara, G.V., Vecchione, M., Alessandri, G., Gerbino, M. & Barbaranelli, C. (2011) The contribution of personality traits and self-efficacy beliefs to academic achievement: A longitudinal study. British Journal of Educational Psychology, 81(1), 78-96. Cassen, R. & Kingdon, G. (2007) Tackling Low Educational Achievement. Joseph Rowntree Foundation. Available [online] from: https://www.jrf.org.uk/publications/tackling-low-educational-achievement [Accessed 02/12/2011]. Eurydice (2009) Gender Differences in Educational Outcomes. Brussels: Education, Audiovisual and Culture Executive Agency. Available [online] from: https://eacea.ec.europa.eu/education/eurydice/documents/thematic_reports/120EN.pdf [Accessed 02/12/2011]. George, R. (2006) A cross-domain analysis of change in students' attitudes toward science and attitudes about the utility of science. International Journal of Science Education, 28(6), 571-589. Gliner, J.A. & Morgan, G.A. (2009) Research Methods in Applied Settings. Mahwah, NJ: Lawrence Erlbaum Associates, p. 219. Koballa, T.R. & Crawley, F.E. (1985) The influence of attitude on science teaching and learning. School Science and Mathematics, 85(3), 222-232. See, Y.H.M. & Khoo, B.L.Z. (2011) Tailoring information to change attitudes: A meta-structural approach. In I.M. Saleh & M.S. Khine (Eds.) Attitude Research in Science Education. USA: Information Age Publishing, pp. 177-198. Sirin, S.R. (2005) Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417-453. Talton, E.L. & Simpson, R.D. (1985) Relationships between peer and individual attitudes toward science among adolescent students. Science Education, 69(1), 19-24. Tamir, P. (1988) Gender differences in high school science in Israel. British Educational Research Journal, 14(2), 127-140. Underhill, L. & Bradfield, D. (2004) IntroSTAT – 2nd Edition. South Africa: Creda Communications, p. 8. Wolf, S.J. & Fraser, B.J. (2008) Learning environment, attitudes and achievement among middle-school science students using inquiry-based laboratory activities. Research in Science Education, 38(3), 321-341.

Cite this page