Improving Student Comprehension through Collecting and Interpreting Relevant Primary Data

Lisa Giddings

Associate Professor of Economics

University of Wisconsin-La Crosse

1725 State Street

La Crosse, WI 54601

Giddings.lisa@uwlax.edu

http://www.uwlax.edu/faculty/giddings

Office: 608-785-5297

Fax: 608-785-8549

DRAFT: Please do not quote without author’s permission.

Submitted to the American Economist, December 2005

Title: Improving Student Comprehension through Collecting and Interpreting Relevant Primary Data

Abstract: The purpose of this study was to examine the effects of having students collect relevant primary data for later coursework. Analysis was conducted over three semesters of a 300-level general education economics course at the University of Wisconsin-La Crosse. Pre- and post-tests were administered on students in treatment and control groups in order to test the efficacy of collecting primary data as a pedagogical tool. It was found that collecting primary data that is later used in homework assignments and papers significantly improved student exit exams in two of the three semesters analyzed.

I. Introduction

Economics courses can fail for many reasons. Students, for example, may not grasp the theory, understand the graphs or the algebra, or care about the material itself and, therefore, not invest time or energy into learning. W. Lee Hansen, Michael Salemi, and John J. Siegfried (2002: 464) argue that introductory economics courses fail “because [they do] not teach students how to apply economics to their personal, professional, and public lives… students never master the basics.” One way to get students invested in the material, to master the basics, and to see how it applies to their own personal lives, is through the collection and interpretation of primary data.

This research explores the effects of using primary data in the classroom as a pedagogical tool to better engage students with the course material. An experiment was conducted over the course of three semesters comparing a treatment and a control section of a course entitled “Women in the U.S. Economy,” a 300-level general education economics course taught at the University of Wisconsin-La Crosse. Students in the treatment group interviewed couples using a survey adopted from Phillip Blumstein and Pepper Schwartz’ American Couples (1983). Class data was then compiled and students were required to use the data to explore issues discussed in the course through homework and papers. Members of the control group analyzed the data but did not take part in collecting the data. Pre- and post-tests were employed to measure changes in student understanding of the major concepts developed in the course. 

Results of ordinary least square regression analyses show a significant difference in learning outcomes between the control and the treatment group. This evidence indicates that gathering the primary data improves student comprehension of some of the basic concepts and ideas covered in the course. I believe that this is because the act of gathering the primary data serves to engage students in the learning process and, as a result, gain a greater understanding of the basic concepts of the course. Because the control group did not actually gather the data, they did not see the material in a personal way and apply concepts to their own lives. This evidence suggests that that actually collecting the survey data significantly improved the treatment group’s overall understanding of the basic building blocks of the course. It is believed that this, then will aid in student comprehension of the higher-level and applied concepts studied in the course.

The rest of the paper is organized as follows. Section II provides some background on the course. Section III provides a literature review of the use of primary data as a pedagogical tool. Section IV discusses the experiment. Section V presents the data and the methodology used. Section VI provides results. This is followed by a conclusion.

II. Background: The Course

            “Women in the US Economy” is a general education course in the department of economics at the University of Wisconsin-La Crosse. The course has no prerequisites, despite being a 300-level writing-intensive course, and there are many incentives for business students to take the class.  It fulfills one of their general education requirements, one of their writing-intensive upper-level course requirements, is listed in the college of business and is an economics course (for majors and minors). Plus, I suspect, students expect it will be easy. As a result, the department often offers the course (sometimes twice per semester to as many as 80 students) and, interestingly, nearly 50 percent of the students enrolled in the course are men.

            The course covers three major aspects of issues related to women and the United States economy: the labor market, the household, and poverty. It combines historical information with statistics and economic theory. One goal of the course is to improve student’s ability to think critically; to recognize that few issues are black or white, and to be able to use logic as well as identify reputable sources to defend whatever position they decide is most correct. The issues presented in the course lend themselves to exercises in critical thinking. For example, while we can measure a wage gap between men and women, its size is subject to measurement issues, and its cause is likely due to many factors including differences in schooling and preferences between men and women as well as, possibly, discrimination. Furthermore, prominent scholars disagree about the issue, providing fodder for good classroom discussion.

Like Hansen, Salemi, and Siegfried’s (2002: 464) description of introductory economics courses, however, this course failed. Despite providing students with current, accessible articles and data, they simply did not believe the information presented. Additionally, students did not seem to find the issues presented in the class relevant, important, or even interesting.

My first response was to provide them with recent newspaper articles (including both local, national and international information) reinforcing the class material in addition to the textbook. When this did not work, I presented data from the Census Bureau. Students would do the homework and interpret their graphs correctly, but still, in class discussions, their opinions remained the same: men and women are essentially equal. In discussion, they would articulate that there is no difference between men’s and women’s wages or occupations. I could not even get to the critical thinking portion of the material, for example, exploring whether any observed inequality is a result of women’s choices or discrimination. Even if they conceded that inequality existed, they were convinced that it did so in the distant past and was, therefore, irrelevant.

            This problem is different than the usual run-of-the-mill undergraduate laziness, lack of time spent studying, or over-commitment that generally leads to an unproductive classroom. It is not that students were not doing the work. Also, it is not necessarily a matter of learning. My hypothesis is that their disbelief in the issues presented is related to four distinct factors that all relate to their level of engagement with the material. First, the students have not yet experienced most of the issues that I present to them. In their experience women do not get paid less (they all work for less than $8/hour); women do not experience occupational segregation (they all work at bars or restaurants, or at the university); there is no obvious “old boys network” working to advance men and discriminate against women (the bar and restaurant owners certainly want to hire women and they do not work in jobs with hierarchies that affect them); neither men nor women do housework (either in their apartments or their dorms); none of them have children so they have not had to make choices about careers versus family; no one is in poverty; and their university is 99 percent white, so affirmative action is something they read about only in Newsweek. They simply cannot relate to the material.[i]

            Second, from their perspective, the information is an historical relic. Most of their mothers worked while they were young and so the phenomenon of women moving into the labor market does not represent a critical juncture in their history. Third, they do not make the connection between the study of economics and public policy. And fourth, they do not connect themselves with public policy. In summary, the students do not believe the material because they have not experienced the issues and were not engaged with it.

III. Primary Data as a Pedagogical Tool

            Engaging students through collecting and interacting with primary data is essentially an extension of active learning or “problem based learning,” and is certainly not new. In fact, getting students to perform experiments and collect primary data is such a common technique in the sciences, it has not been explicitly published as a pedagogical tool.

A problem-based learning (PBL) curriculum “provides authentic experiences that foster active learning, support knowledge construction, and naturally integrate school learning and real life” (ASCD, 2005). It is claimed to be a practical strategy for fostering deeper, critical, active-learning strategies (Ramsden, 1992). PBL also provides a stimulus for learning (Boud and Feletti, 1991). The use of primary data in the classroom can support problem based learning by providing students with hands-on evidence with which to explore complex problems. Peter Milbury and Brett Silva (1998) developed activities that drew upon “authentic, primary sources” from the Library of Congress American Memory Web site. Their goal was to actively engage students in research and critical thinking. According to Milbury and Silva, the primary sources helped students develop good habits when working with a large amount of information about “fuzzy” problems with no “preconceived” or “textbook” solutions. Furthermore, students could not simply “cut and paste” information from the Internet because they were required to use the data at hand.

Carol G. Johnston, Richard H. James, Jenny N. Lye, and Ian M. McDonald (2000) evaluated a project that incorporated problem-based learning in an economics classroom. They found that the project generated a “positive student reaction . . . and evidence of increased student preparation for tutorials” (p. 13). The authors did not, however, find any measurable improvement in exam scores.

Several studies have documented the explicit effects of using primary data in the classroom. In a course titled “Early United States History,” Susan Leighow Meo (2000) had undergraduate students read a series of primary sources and analyze them in a journal. One result of this interaction with primary sources was to “allow students to see history as an ongoing process of constructing the past, rather than a fixed body of knowledge.” According to Meo, the use of primary sources improved the students’ ability to identify different authors’ perspectives and evaluate different sources’ credibility. Additionally, she claims that teaching using primary sources modeled the pedagogical strategies for future teachers in the classroom.

Wynell B. Schamel (1998) used primary source documents in a secondary school history course. By using primary sources, Schamel claims that students learned that the record of historical facts reflects the “personal, social, political, or economics views of the participants who created the sources” and that their own biases come into play upon their review. A benefit of using the primary sources collection, according to Schamel, is the development of broad cognitive and analytical skills. Additionally, she claims that teaching using primary sources modeled the pedagogical strategies for future teachers in the classroom.

IV. The Experiment

This experiment follows the typical strategy of economic education research by pretesting students, applying an innovative pedagogical strategy to a treatment group, and posttesting the students in order to see of the treated group learned differently than the control group. Using this logic, the learning process can be thought of as similar to a production process in which the “output is increased economic aptitude” (Salemi and Tauchen, 1980: 42). In this model the inputs to the learning process are student aptitude at the beginning of the course, the time a student spends studying, and the characteristics of the learning environment including the treatment itself.

Having identified a problem in my course relating to student engagement rather than to effort, I designed a cross-sectional household survey that students could administer themselves and then use the aggregate class data in homework assignments and papers. It is hypothesized that actually gathering primary data engages the students in a way that develops an understanding of the issues in a much more effective way. Gathering the data is a significant step toward engaging with the material, not simply analyzing data given to you. The students had already worked with much more accurate data from the Census – data with a much larger and random sample – but were still not engaged in the material or the issues. It is believed that when students go out to real people and ask them real questions, and everyone else in the course is doing the same, that the issues become real to them.

The survey is based on the concepts presented during the course as well as questions posed by Blumstein and Schwartz (1983).[ii] It includes basic demographic information such as race, income, and education level, as well as information on who performs domestic duties and a series of questions on participants’ opinions on various issues. It is ideal for exploring the issues presented in the course and, ironically, exaggerates the data provided to them. For example, the class surveys regularly result in wage gaps that are larger than the national average.

This assignment is given to the treatment group on the first day of class and students were expected to turn in their results within the first week of classes. They are required to choose a couple that is currently cohabitating, not as roommates, but not necessarily legally married. Once the interviews were conducted, the surveys were compiled and made the entire data set available to the class. Students were then required to use the data set in several homework assignments.[iii]

This assignment took a relatively small amount of time and was performed entirely outside of the classroom. Each survey could be conducted over the phone and took approximately 10 minutes to complete. While the members of the control group had no corresponding outside activity, it is expected that the act of gathering the data itself rather than the time spent gathering the survey data will serve to engage the students in the treatment group as compared to the control group.

V. Data and Methodology

The experiment was conducted over the course of three semesters. During each of the three semesters, the two sections of the course were taught back-to-back on Tuesday and Thursday afternoons. In order to control for selectivity bias, one section each semester was randomly selected to conduct the survey. This section was designated the “treatment group.”

In order to evaluate the effects of gathering the primary data, pretests and posttests were administered to six sections of the course during three separate semesters.[iv] Pretests were administered during the first week of classes and posttests were administered during the last week of classes. The test contains 31 questions related to the class material, including both quantitative and qualitative, open-ended components. The questions were designed to measure student understanding of the basic definitions and concepts discussed in the class and attempted to avoid economic jargon. The instrument does not capture deep understanding or the ability to apply knowledge gained from the course. However, as stated, in order to get to higher learning of the material, one must first grasp the basic concepts. Furthermore, I had identified that students in the course were not getting these basic concepts. For this study, only the quantitative answers were evaluated. A statistical algorithm was employed to grade the quantitative questions of the entry and exit exams in order to assure consistent grading.

It should be noted that neither the pretest nor the posttest counted toward the student’s grade. While it is possible that not attaching a grade to the test would provide students little motivation to do well on the posttest, it is also possible that attaching a grade to the test would discourage students in the pretest, or encourage students to cheat in order to improve their score. In any case, both grading and not grading the exams introduces perverse incentives and therefore biases in the resulting data. Furthermore, all sections were treated in the exact same way, so any bias is consistent across control and treatment groups as well as across semesters.

            It was expected that the scores on the entry exam would not differ significantly between the treatment section and the control section, and that the scores would improve for both the treatment and the control group. I hypothesized that because members of the treatment groups were more engaged with the material, their posttest scores would be significantly higher than that of the control group on average.

            In order to control for differences in student aptitude, this study relied on other relevant data rather than use the pretest as a proxy (Salemi and Tauchen, 1980). The variables used to control for student aptitude included students’ ages, their cumulative grade point averages, in which college they were enrolled, their major, and their grade for the course as well as ACT scores in English, math, and their comprehensive ACT score. A dummy variable indicating whether the courses were in the fall or the spring semesters is included to control for any differences that the season or the semester itself might introduce. Additionally, dummy variables indicating whether the students took the course as an elective or as a requirement are included. Table 1 provides a list of the variables and their descriptions. Table 2 presents the means and standard deviations of the variables broken down by the treatment/control groups and semester.

Table 1 Variable Definitions

Table 2. Table of Means, Standard Deviations are in Parentheses

Source: author’s calculations.

            Table 3 provides the results from difference of means tests for the entry test and exit test results for the entire sample as well as broken down by semesters. This evidence shows that there is no significant difference between the treatment and the control groups on the pretest in any of the semesters examined. There is, however, a statistically significant difference between the treatment and control groups in the entire sample in the results of the posttest. On average, members of the treatment group scored 3.725 points higher than those in the control group on the posttest. This difference was significant at the 0.05 level. Based on the results presented in this table, differences in mean scores between the treatment and control groups were significant in semesters 1 and 3, however, they were not significantly different in the second semester.

Table 3. Difference of Means Tests: Treatment Versus Control, standard deviations in parentheses under means and 2-tailed significance tests in parentheses under t-tests

            In order to test for the effects of gathering the primary data on the student’s posttest scores while controlling for the other inputs into the learning model, including student aptitude, ordinary least squares regressions were run with posttest scores as dependent variables. Equation 1 represents the model:

            First, the dummy variable representing the treatment was included in the regression. This represents the effect of the environmental factor in the classroom on the learning outcome. All environmental factors other than the treatment are held constant. It is expected that the treatment will have a significant positive effect on the Posttest. A dummy variable representing the fall semester is included to control for any differences in the semester in which students took the course. A dummy variable for the sex of the student was included. Research suggests that males and females differ in their performances in economics exams. Mary L. Williams, Charles Waldauer and Vijaya G. Duggal (1992) found some evidence that males outperformed females on essay exams in principles courses. They also found no difference in overall quantitative skills between men and women. Benjamin Greene (1997) also found no evidence that females are better at more verbal evaluations of economic knowledge. Because this study was based on quantitative questions, it was not expected that the variable “Sex” will have any significant effect on either the entry or the exit scores.

            Age may have had a positive effect on both the entry and the exit scores as it may proxy for year in school or capture some measure of maturity or even possibly seriousness on behalf of the student in terms of study habits. Also included were dummy variables indicating whether a student was an economics major, and if they were not enrolled in the college of business. It is expected that economics majors and those enrolled in the college of business would perform better than students who were not economics majors or who were not enrolled in the college of business.

            To control for student aptitude, several proxy measures were included in the regression analysis. These included the student’s cumulative grade point average,[v]^,[vi] the final grade in the course, and their English, math, and composite ACT scores. Whether the student takes the course as an elective or a requirement is expected that students that took the course as an elective would be more likely to perform better than those that took the course as a requirement. Information on any student input such as time spent studying was not available.

VI. Results

            Table 3 presents the results of standard ordinary least square regression. The standard errors listed in all of the tables are corrected for heteroskedasticity of unknown form (White, 1980). The results presented in Table 3 indicate that, looking at the entire sample overall, the treatment had a positive and significant effect on the posttest results. The effect is larger in Semester 1 and Semester 3, however the level of significance drops in both of the individual semesters. The treatment does not have a significant effect on the posttest results in Semester 2.

Table 3: Ordinary Least Squares Regression Results, Dependent Variable: Posttest t-statistics are in parentheses.

Source: author’s calculations.

            Looking at the entire sample, age, being an economics major, and the comprehensive ACT score had a significant positive effect on the posttest outcome. Age improves the posttest score by less than one half of a point whereas having an economics major improves students’ posttest scores by more than three points, and the ACTComp improves posttest scores by over one point. One could infer that students that have declared economics as a major have greater aptitude toward the subject or are inclined to enjoy the material more than non-majors and therefore performed better on the posttest. Interestingly, ACTEng had a significant negative effect on the posttest score.

            Although these variables did not affect the posttest results significantly, whether or not the course was taken in the fall semester, being a woman and being a non-business major resulted in lower posttest outcomes on average in the overall sample. Furthermore, one’s grade in the course, while insignificant, was negatively related to the posttest results. This is surprising. While the posttest measures only basic concepts, it is expected that if students master the basic concepts, they will do better at higher-level thinking and more applied exams over the material. This evidence indicates that while those individuals in the treatment group may have significantly improved their understanding of the basic concepts, they could not extend this improvement to the more higher-level and applied concepts covered in the course and measured in the final grade.[vii]

            Focusing on the individual semester results, ACTEng had a negative, significant effect on the posttest score in Semester 1. On average, those in the treatment group in Semester 1 increased their posttest scores by over four points. Surprisingly, the ACTMth score had a negative significant effect on the posttest score in Semester 2 and, as reported, the treatment had no significant effect on the posttest scores in this semester. In Semester 3, as reported, the treatment had a positive, significant effect. On average, those in the treatment group in Semester 3 increased their posttest results by over four points. Women did significantly better on the posttest exam in Semester 3 than did men, scoring on average 4.35 points better. Age was also significant in the third semester. On average, each additional year resulted in nearly a point on the posttest results. Having declared an economics major significantly improved the posttest results as was the case in the overall sample. The model in Semester 3 was perhaps the best out of the three, having explained over 44 per cent of the variance in the posttest results.          

VII. Conclusions

            This experiment set out to test the effects using primary data as a pedagogical tool. Specifically, this experiment tested the effects of having students gather and use primary data as compared to a control group that only used but did not actually collect the primary data. It was hypothesized that the act of carrying out the survey would engage students in the course material and significantly improve their learning outcomes.

            These results indicate that there is evidence that gathering primary data positively affects students’ outcome on the posttest score measuring their understanding of the basic concepts of the course. This may be an indication of greater engagement with the course material and, as a result, a better comprehension of the ideas presented in the class. Even after controlling for other factors that might affect one’s ability to perform on the post-test, the treatment of actually gathering the survey data improves student scores on the post-test. This experiment, however, was limited to only measuring the effect of gathering the data. I suspect that using a survey to explore the issues of the course affected all of the students, not just the ones who gathered the data.       

            In addition to becoming more engaged with the material, however, there are several other spillover benefits from using primary data as a pedagogical tool. As stated in the review of the literature, both gathering and using survey data models the research strategies for future economists. First, students learn about data. They learn that data is rarely “clean” and, depending on how the researcher deals with these issues, they may get different results. Furthermore, students learn the importance of what questions are asked and how statistics can tell different stories. One student, for example, became interested in the difference between reporting the mean and the median when exploring the wage gap between men and women. Students also learn to use a statistical software package, in this case Excel or SPSS. Finally, and most unexpectedly, because the data changes every semester, students are unable to plagiarize their papers either from the Internet, or from students of previous semesters. As a result, using the data provides a built-in method to prevent such cheating. 

References

Association for Supervision and Curriculum Development (ASCD). 2005. “The Definition of Problem-Based Learning.” Website last visited, May 19, 2005. http://www.ascd.org/portal/site/ascd/menuitem.a4befa0de1b8d1fddeb3ffdb62108a0c/

Blumstein, Philip and Pepper Schwartz. 1983. American Couples: Money, Work, Sex. New York: William Morrow and Company, Inc.

Boud, D., and G. Feletti. 1991. The Challenge of Problem-Based Learning. London: Kogan Page.

Bransford, John D., Ann L. Brown, and Rodney R.  Cocking. 2000. How People Learn: Brain, Mind Experience, and School. National Academy Press.

Gardner, Howard. 1991. The Unschooled Mind: How Children Think and How Schools Should Teach. New York: Basic Books.

Greene, Benjamin. 1997. “Verbal Abilities, Gender, and the Introductory Economics Course: A New Look at an Old Assumption.” Journal of Economic Education. Vol. 28(1): 13-30.

Hansen, W. Lee, Michael Salemi and John J. Siegfried. 2002. “Use It or Lose It: Teaching Literacy in the Economics Principles Course.” American Economic Review, Vol. 92(2): 463-72.

Harbury, C. D. and R. Szreter. 1970. “The Value of Prior Experience of Economics for University Students.” Journal of Economic Education. Vol. 2(1): 56-62.

Johnston, Carol G., Richard H. James, Jenny N. Lye, and Ian M. McDonald. 2000. “An Evaluation of Collaborative Problem Solving for Learning Economics.” Journal of Economic Education. Winter: 13-29.

Leighow Meo, Susan. 2000. “’In Their Own Eyes’: Using Journals with Primary Sources With College Students.”

Milbury, Peter and Brett Silva. 1998. “Problem-Based Learning, Primary Sources, and Information Literacy.” Multimedia Schools. Vol. 5(4): 40-45.

Poole, William. 2004. “Free Trade: Why Are Economists and Noneconomists So Far Apart?” Federal Reserve Bank of St. Louis Review. Sept/Oct: 1 – 6.

Ramsden, P. 1992. Learning To Teach in Higher Education. London. Routledge.

Salemi, Michael K. and Goerge E. Tauchen. 1980. “Guessing and the Error Structure of Learning Models.” The American Economic Review, Vol. 70(2): 41-46.

Schamel, Wynell B. 1998. Teaching with Documents: Using Primary Sources from the National Archives. Volume 2.

White, Halbert L. 1980. “A Heteroscedasticity-consistent covariance matrix estimator and direct test for heteroscedasticity.” Econometrica, Vol. 48(4): 817-838.

Williams, Mary L., Charles Waldauer and Vijaya G. Duggal. 1992. “Gender Differences in Economic Knowledge: An Extension of the Analysis.” Journal of Economic Education, Vol. 23(3): 219-231.