The General Education Assessment Working Group

Final Report and Recommendations

September 26, 2005

Working Group Members

Sandy Grunwald, Working Group Chair Summer 2004 & 2005, Faculty in Chemistry

Mike Abler, Member Summer 2004 & 2005, Faculty in Biology

Gwen Achenreiner, Member Summer 2004 & 2005, Faculty in Marketing

Eric Kraemer, Member Summer 2005, Faculty in Philosophy

David Reineke, Member Summer 2005, Statistical Consulting Center

Bruce Riley, Member Summer 2005, Faculty in Mathematics

Rachelle Toupence, Member Summer 2004, Faculty in Recreation and Therapeutic Management

Brad Seebach, Member Summer 2004 & 2005, Faculty in Biology

Introduction

            The General Education Assessment Working Group was appointed by the General Education Committee to develop, pilot test, and implement assessment instruments which assess student learning at UW-La Crosse in the broad areas of critical thinking, global perspective and science knowledge.

Working Group Activities and Background Information

           In Summer 2004 the group developed two different assessment instruments designed to assess student learning using the outcomes generated by the General Education Committee (4/28/03).  These outcomes were based in the broad categories of analytic and problem-solving skills; quantitative reasoning skills; content knowledge in science; and global perspective.  Both assessment instruments were pilot tested in Fall 2004 to approximately 30 students, revised, and rubrics were developed to score the student responses.  In Spring 2005 these instruments, along with an assessment instrument previously developed by the General Education Assessment Committee focused exclusively on global perspective, were administered to a larger cohort of UWL students with varying demographics.  Finally in Summer 2005 the working group reviewers scored the student responses.  These scores were statistically analyzed to determine any correlation between score and student demographic.  A complete report on each instrument, including correlation results, can be found in Appendix A, Appendix B and Appendix C. 

Along with these assessment instruments another working group developed an instrument focused on assessing student learning in written communication, responsible citizenship and integration of knowledge.  The results of that assessment instrument are not included in this report. 

Summary of Results

A.                  Global Perspective

            All three assessment instruments that were administered in Spring 2005 assessed student’s global perspective, specifically the student’s ability to “explain the rationales for cultural behaviors different from one’s own”.  The mean score for Instrument #1 was 2.12 (competent) on a scale of 1 (naïve) to 4 (sophisticated).  The mean score for Instrument #2 was 0.67 (scale 0 to 2) and for Instrument #3 was 1.08 (scale 0 to 2).  It was hypothesized that the student’s ability would increase with completion of courses having global awareness as a focus, increased age, more years in college, and participation in a study abroad experience (see Table 1 summary).

            The expected correlation was observed between student ability and number of global awareness courses in 1 of 3 assessment instruments.  Though no statistical difference was observed in the other 2 assessment instruments the data did show a trend in these 2 instruments whereby the mean student score increased with greater number of global awareness courses.  

            No statistical difference was observed between global perspective student ability and participation in a study abroad experience; however, in 2 of 3 instruments the data showed a slight trend of a higher score for students having some global experience versus students who had no global experience.  

            The expected correlation was observed between student ability and age in 1 of 3 assessment instruments.  Though no statistical difference was observed in the other 2 assessment instruments the data showed a trend in 1 of the instruments whereby the mean student score increased with age.

            The expected correlation was observed between student ability and year in college in 1 of 3 assessment instruments.  Though no statistical difference was observed in the other 2 assessment instruments the data showed a trend in 1 of the instruments whereby the mean student score increased with year in college.

            The students were also asked at the end of the demographic information “Have you had any experiences which you felt were helpful in interpreting the story/article which you read and answered questions on?  If so, please describe them below.”  There was a correlation between a Yes response and increased student ability in 2 of 3 instruments.  Though no statistical difference was observed in the other assessment instrument the data showed a trend whereby the mean student score was higher for students answering Yes to this question.  Further analysis on what type of experiences were reported and student ability needs to be undertaken.

            Furthermore the student’s ability was correlated with “what college is your academic major”.  Students associated with the College of Business Administration showed a statistically significant lower mean global perspective score in 2 of 3 instruments.  Though no statistical difference was observed in the other instrument the data showed a trend whereby the mean student score was lower for students in CBA than the other colleges.  However, upon further analysis it was determined that of the students in CBA who were administered the instruments, they were unproportionately freshmen/sophomore level vs. junior/senior level.  This discrepancy in student demographics likely accounts for the results observed. 

Table 1:  Summary of Correlations/Trends with Global Perspective Student Ability

*Instrument #1 is the Global Perspective Assessment Instrument summarized in Appendix A; Instrument #2 is the Genetically Modified Foods Instrument summarized in Appendix B; Instrument #3 is the Medicine Without Doctors/HIV Instrument summarized in Appendix C

B.      Critical Thinking – Analytic Reasoning

            Two of the three assessment instruments that were administered in Spring 2005 assessed the critical thinking ability of students specifically the student’s ability to “formulate and support ideas with sufficient reasoning, evidence and persuasive appeals, and proper attribution”.  The mean overall student score was consistent between the 2 assessment instruments at 0.98 and 0.99 (scale 0 to 2).  It was hypothesized that there would be a correlation between student’s ability and increased age and more years in college. Also correlation of the student’s ability with “what college is your academic major” was analyzed. 

            No statistical correlation was observed between student ability and increased age; however, in 1 instrument the data showed a slight trend of a higher score for students from age 18 to age 22.   

            No statistical correlation was observed between student ability and year in college. Though no statistical difference was observed in the 2 assessment instruments, the data did show a trend in these 2 instruments whereby the mean student score was higher for seniors than for freshmen.

            Furthermore there were inconsistent correlations between student ability and college of their academic major between the 2 assessment instruments.  Thus no conclusions can be made regarding this correlation.  

      Critical Thinking – Quantitative Reasoning (interpreting tables and graphs)

            Two of the three assessment instruments that were administered in Spring 2005 assessed the critical thinking ability of students specifically the student’s ability to “use mathematical and logical methods to solve problems” in the form of interpreting tables and graphs.  In one of the instruments students were asked a question whereby they had to interpret a basic table to obtain the correct data (Q5, Appendix C).  In the other instrument students were asked a question whereby they had to interpret a basic pie chart (Q7, Appendix B).  Students who were successful in this basic quantitative reasoning were 87% and 85%, respectively, showing consistency between the two assessment instruments.  There was no statistical correlation between student ability and any of the student demographic information.

            In one of the instruments students were asked a question whereby they had to interpret a more complex bar graph (Q6, Appendix B).  Only 40% of the students were successful in their ability to perform this task, leaving 60% unsuccessful.  Again there was no statistical correlation between student ability and any of the student demographic information.

C.                  Science Content Knowledge

Two of the three assessment instruments that were administered in Spring 2005 assessed the science content knowledge of students.  This was done by asking questions that addressed science knowledge that should be acquired through in awareness of issues addressed in the popular press.  There were also a few that addressed science content specific to the main focus of the assessment instrument.  The mean student score for the various science questions varied dramatically between the different questions.  For example the mean score was 1.73 on a 0 to 2 point scale for a question that asked how the AIDS virus could be acquired and the mean score was 0.83 on a 0 to 2 point scale for a question that asked if antibiotics should be used to treat HIV/AIDS.  Both of these questions are deemed as important science knowledge needed to understand issues that are addressed in the popular press. 

            There was not an overall consistent correlation of mean score and student demographics for the various science content questions; however, in several cases the following was seen:

• students in SAH performed better than students in other colleges
• seniors performed better than freshmen
• performance was higher for students having taken more courses with a science focus.   

(see Appendix B and C for details on statistically relevant correlations)

Brief Recommendations. 

The Working Group recommends:

1.       implementation of a process for administering assessment instruments to lower and upper level students, besides asking instructors to volunteer a class period.

2.       exploration of methods to increase student ability in interpreting data in graphs/tables that are beyond basic level.

3.       that the General Education Committee communicates the student learning outcome findings pertaining to global perspective to the UWL International Task Force.

4.       further analysis of the correlation between global perspective learning and “other experience” be undertaken to determine what experiences enhanced student’s global perspective ability.

5.       that the General Education Committee use the results of these assessments instruments to provide benchmarks of student learning and watch for changes over time. 

Appendix A

Analysis of Global Perspective Assessment Instrument

Section 1: Background – In Fall 2003 the General Education Committee established an Assessment Sub-Committee whose goal was to develop, pilot test and implement an assessment instrument to measure student learning in the area of global perspective, specifically if students are “able to explain the rationales for cultural behaviors different from one’s own”.  This has been identified as a student learning outcome for the general education program (approved outcome 9/12/05 Gen Ed Committee).  Members of this committee were Terry Beck (consultant), John Betton, Sandy Grunwald, and Jean Hindson.  This assessment instrument consists of a reading from the book "Ponds of Kalambayi" by Mike Tidwell (1990) pub Lyons and Burford and the student responses to 4 questions pertaining to that reading (see Global Perspective Instrument at the end of Appendix A). 

The assessment instrument was piloted in Fall 2003 to 17 modern language students.  The student’s responses were scored using a rubric that defined sophisticated, knowledgeable, competent or naïve responses (see end of Appendix A).  Before scoring, the reviewers standardized their responses by scoring and reviewing these scores to determine that all reviewers used consistent methods of analysis.  Every student response was scored by at least 2 reviewers. 

After this initial pilot the Assessment Sub-Committee deemed this global perspective assessment instrument worthy of being used for further assessment purposes; therefore, in the Spring of 2004 the instrument was administered to 67 students enrolled in upper level management courses.  The student’s responses were scored by 2 reviewers from the Assessment Sub-Committee and their scores was analyzed by the UWL Statistical Consulting Center versus various student demographics (age, study abroad experience, number of courses with a global awareness emphasis, and outside experience).   The results of this assessment can be found in the Results section of Appendix A.

In Spring 2005 this assessment instrument was administered on a larger scale to UWL students.  The students varied in age, year in college, college of their major, study abroad experience etc…  Again each student response was scored by 2 reviewers (Sandy Grunwald, chemistry and Bradley Seebach, biology) who were part of the General Education Assessment Summer Working Group.  They used the scoring matrix described above to evaluate student responses.  Results and analysis of this more extensive assessment can be found in the Results section of Appendix A.

Section 2: Results & Demographics– The following shows the results of the student responses to the global perspective assessment instrument.  Note that the results will be divided into 3 trials for the 3 different times that the instrument was administered. 

Trial 1 - Initial pilot group, 17 modern language students, Fall 2003

Mean Score – 2.14 which is slightly higher than competent

Trial 2 – 67 upper level management students, Spring 2004

Mean Score – 2.01 which is rated as competent

For the raw data see Sandra Grunwald, Chemistry or David Reineke, Mathematics

Analysis of results by David Reineke, Statistical Consulting Center:

Correlation coefficients were obtained for the relationship between the average global perspective score and age group (r = 0.203, P = 0.123) as well as for average global perspective score and the number of courses with international or other cultural content (r = 0.069, P = 0.598).  Neither relationship was found to be statistically significant at the 5% level of significance. Furthermore, due to the low number of cases where study abroad was indicated, no statistical relationship could be established with the average global perspective score.

However, a significant difference in average global perspective score was found between students who have had experiences with other cultures and students who have not. Using a two-sample t-test, it was determined that the average global perspective score for students who have had experiences with other cultures is significantly higher than for students who have not (P = 0.001).  In fact, with 95% confidence we can say that the average global perspective score is 0.42 to 1.47 units higher for such students.  The sample mean of the average score for the 11 students who have had experiences with other cultures is 2.7, roughly translated as “knowledgeable,” whereas the sample mean for the 49 students who did not indicate having some other cultural experience is 1.8, or “competent.”  (Note:  cases where the global perspective scores differed by more than 1 were excluded – however, the results would not have differed if they had been).

Trial 3 – 189 students with varying demographics, Spring 2005

Mean Score – 2.13 which is slightly higher than competent

For the raw data see Sandra Grunwald, Chemistry or David Reineke, Mathematics

Statistical Analysis and Conclusions by David Reineke, UWL Statistical Consulting Center:

Analysis of variance was used to determine if significant differences exist among the mean Global Perspective scores for age, year in college, college of academic major, and number of courses involving global awareness while t-tests for independent samples were used for the dichotomous variables of time spent studying abroad and whether or not the student had other helpful experiences.  Although the amount of time spent studying abroad had five categories, the small number of students in the study who have studied abroad made it necessary to collapse the variable into two categories: “None” and “Some.” 

In every case, the normality and equal variance assumptions were examined, and where appropriate, the nonparametric Kruskal-Wallis test or Mann-Whitney U-test was used instead of the analysis of variance or t-test, respectively.  A level of significance of 0.05 was used for each test.

Only two significant differences in mean Global Perspective score emerged.  Students in the College of Liberal Studies had a significantly greater mean Global Perspective score than those in the College of Business Administration (P = 0.010) and students who reported having an experience that was helpful in interpreting the story or article had a significantly greater mean Global Perspective score (P = 0.013) than those who did not report such an experience. Demographic and descriptive statistics are given in the tables that follow. 

1. Trial 3 Descriptive Statistics

The following tables contain descriptive statistics for the Global Perspective score according to each of the demographic variables: age, year in college, college of academic major, time spent studying abroad, number of courses with global awareness, and whether or not the student had other helpful experiences.  Asterisks with accompanying footnotes indicate significant differences.

Note: Student Score 1 = naïve, 2 = competent, 3=knowledgeable, 4=sophisticated