Provost’s Council; Faculty Senate Executive Committee; Academic Staff
Kathleen Enz Finken, Provost / Vice Chancellor for Academic Affairs; Robert
Faculty Assistant to the Provost
from: Teri Lyn Hinds, Director of Institutional Research
subject: Proposed Process for Requesting Email Addresses for Surveys
In response to several concerns arising due to increasing use of electronic surveys for student class and research projects, a working group comprised of representatives from the faculty in each College, staff in ITS, the Institutional Review Board, the University FERPA officer, Institutional Research and the Statistical Consulting Center is proposing a new procedure for requesting student email addresses for use in survey research projects. The new process will centralize requests and streamline many of the necessary approvals for the release of student contact information.
Process for Requesting Student Email Lists
1. Design the survey. In designing survey research, students should keep in mind that due to FERPA limitations, they may not be able to get access to some information about their individual sample members from the institution (e.g., race / ethnicity, gender, etc.); it may be best to plan to ask these as questions in the survey if analyses will need to include them. Aggregate sample demographics will be provided if requested for use in response analyses.
2. Determine an appropriate sample size. The type of analysis that will be conducted with the results of the survey will impact the necessary sample size. Students should work with their faculty advisors and are encouraged to contact the Statistical Consulting Center for assistance in determining an appropriate sample size for their research.
3. Determine an expected response rate. While the response rate will depend on many factors, including the nature of the questions asked, the length of the survey, and whether reminders or follow-up emails are sent, students can generally expect a response rate of between 20-40% for survey research at UW-L. As we gather additional data on response rates, we will be able to provide more specific information to guide researchers in determining expected response rates.
4. Complete the IRB approval process. From A Researcher’s Guide for Submission of Protocols (http://www.uwlax.edu/grants/compliance/IRB.htm):
All research involving human subjects must be reviewed by the IRB; however, research activities that fall under any of the following federally defined exemptible categories will qualify for an IRB expedited review, and may not be subject to further IRB requirements (e.g., annual reviews, informed consent requirements; however, it is strongly suggested that [implied] informed consent always be [obtained]). Investigators believing their research activities to be exempt must follow all procedures and specific requirements on pages 13-15 (Narrative Statement Guidelines and Guidelines for Submission of Protocols) of this guide.
Written consent need not be obtained for electronic survey research, however, participants should be informed at the beginning of the survey that they are participating in research and that they may choose not to complete the survey without penalty.
Classroom projects are not automatically exempt. Exempt status for each course (e.g., BUS 230) must be obtained prior to the release of email addresses. Exemptions are granted in perpetuity unless course content or procedures change substantially. Student projects that differ from the previously exempted course process must be reviewed by the IRB.
5. Complete the Email Addresses for Survey Research Request Form.
a. Requests for survey samples should be made by faculty or staff advisors or class instructors working with students.
b. The survey will branch to one of two forms, depending on whether the sample is being requested for research as defined above or for exempt class-based projects. Requests for non-class-based projects will be checked with IRB and the University FERPA officer to ensure that approval for the study has been received and the data requested is in compliance with University FERPA policies.
c. Assuming all approvals are in place, samples will either be sent to the faculty or staff advisor as an Excel file, or loaded directly into Qualtrics and made available to the faculty or staff advisor as a Panel.
6. Administer the survey. While multiple reminders will, to a point, increase response rate, researchers should consider the law of diminishing returns and exercise appropriate judgment in determining the number of contacts to send non-respondents.
7. Send response rate information to the Office of Institutional Research. In order to assist other’s wishing to conduct research using UW-L students, we will be collecting information on response rates for different surveys so that appropriate sample sizes can be more accurately identified. At the moment, the method for collecting response rate information has not been finalized, but is likely to consist of a simple web-based form asking for information about the sample size, the number of contacts made to recipients, the length of time the survey was open for responses, and the number of complete responses received.
In April 2008, the Office of Institutional Research was alerted to several issues relating to increasing use of electronic surveys for student and institutional surveying.
· Research proposals based on sampling UW-L students within certain protected categories (e.g., women, students of color, etc.), while probably legitimate research, present a conflict with our institutional responsibilities under the Family Educational Rights and Privacy Act (FERPA).
o UW-L’s FERPA guidelines restrict what variables can be used to create survey samples to what’s included in our institutional defined directory information, however this information is not currently conveyed to students conducting research.
o Further, there is confusion on campus regarding the different purposes of the Institutional Review Board and our obligations under FERPA.
§ The Institutional Review Board is primarily concerned with ensuring the protection of human subjects; our obligations under FERPA are primarily concerned with protecting the privacy of our students.
§ This confusion results in some students receiving IRB approval but being denied the release of their requested sample because it is based on a protected data element.
· Student researchers using campus subjects are increasingly requesting larger sample sizes, and many are simply surveying the entire student body. Aside from the pedagogical concern related to proper sample size calculations, this practice creates a workload concern for ITS.
o Student email accounts are restricted to 50 addresses per sent email; to override this limitation, student researchers were required to make an appointment with the ITS Support Center during which a student consultant assists them with sending a bulk email.
o To accomplish this, the campus email administrator also manually altered the restriction on the student account prior to the appointment and then manually re-instated it once the email has been sent.
· Larger-than-necessary sample sizes and increased survey volume may also lead to survey fatigue among the population. While we know that there has been a noticeable increase in the frequency of electronic surveys sent to our students, we have no data on how many surveys are being sent nor on how many times the average student is contacted for a survey in a given term or year. Several institutions are experiencing reduced response rates on all surveys, including large institutionally-sponsored and funded surveys used for assessment of student outcomes that they believe is related to the increased frequency of electronic surveys. We do not currently know whether we have seen declining response rates because we have no means of tracking either the quantity of surveys or their response rates.
A working group comprised of representatives from the faculty in each College, staff in ITS, the Institutional Review Board, the University FERPA officer, Institutional Research and the Statistical Consulting Center met to discuss these issues and work to develop a method of resolving them. The working group recognized that there were several process issues that needed to be addressed and that the appropriate approach for each was somewhat different. The primary issues that the group felt could be ameliorated were the workload issues in ITS, the lack of data regarding the volume of surveys conducted using UW-L students as subjects, the inappropriate inflation of survey samples, and the lack of information regarding response rates. The group met several times over the summer and fall of 2008 to create a plan for centralizing requests for student email addresses for survey research that would also supply sufficient information back to researchers regarding appropriate sample size and University policies related to IRB and FERPA. The process was piloted with two faculty members (Enilda Delgado, Sociology, and TJ Brooks, Economics) who use survey research as part of assigned coursework and the feedback from these instructors has helped further refine the process presented below.
Lack of Data Regarding the Volume of Surveys Conducted Using UW-L Students as Subjects
One of the biggest challenges to gathering information about how many surveys are conducted using UW-L students as subjects is that there are multiple points of entry a researcher can use to gain a list of student email addresses: through the Office of Records and Registration; through the ITS Support Center; and through Institutional Research. Therefore, in order to facilitate collecting data related to survey volume, the group agreed to create a single point of entry for researchers wishing to obtain lists of student email addresses for survey research. A web-based sample request form was created and revised based on feedback from the faculty involved in the pilot and members of the working group (http://www.uwlax.edu/its/helpdesk/enterprise_work_request.htm).
In addition to being stored in the survey database, all requests submitted using the form are emailed automatically to the Office of Institutional Research, who will follow up with the requester to ensure IRB approval has been secured, if necessary. Once IRB approval has been ascertained, the request will be forwarded to the University’s FERPA Officer in the Office of Records and Registration. Assuming that the request meets the FERPA guidelines, the FERPA Officer will forward the request to the ITS Support Center, where the sample will be created and sent to the requester. Questions at any point along the way will be addressed directly to the requester as soon as possible. This process ensures that all points of compliance with federal legislation governing both research using human subjects and the University’s release of student information are met prior to the release of student information.
While the use of a single point of entry will capture most, if not all, research-related surveys conducted on campus, there are some surveys conducted by the institution administration (e.g., related to the University mission, or services provided by an office or division) that are sent using ITS maintained lists of all students and all faculty. The email address request process will not capture this kind of survey activity, however, it is the hope of the members of the working group that such surveys will, in the future, either request a list of student email addresses using the request form or, at the very least, notify the Office of Institutional Research prior to sending surveys via these email groups so that they can be counted in the volume of survey activity.
Assistance in Selecting an Appropriate Sample Size
Appropriate sample size for any given research project will depend on a variety of factors including the type of analysis being conducted, the expected effect size, and desired confidence interval for estimating effects. As student research at UW-L should always be approached from the perspective of training future researchers, the group felt that the best approach to encourage appropriate sample size identification would be to provide resources and information to aid in making the decision. The survey form mentioned in the previous section includes the following statement designed to point students in the direction of the Statistical Consulting Center prior to conducing their research:
The emergence of on-line survey software has made it very easy to increase the size of research samples without increasing the cost of the research. Unfortunately, this can result in samples that are larger than necessary to answer the research question being investigated. Please request a sample size that is appropriate for your research question and intended analysis. If you would like assistance in determining an appropriate sample size for your research, please contact the Statistical Consulting Center prior to completing this form.
Information about the UW-L student population, as well as a link to a survey sample size calculator to be used for research intended to describe the larger population, was added after the pilot at the suggestion of the faculty involved in the pilot. The Office of Institutional Research will insert a link that will bring students to a table housed on the University Data section of the Provost’s website containing basic demographic data broken out by various groups for the current semester. A link to Creative Research System’s (CRS) sample size calculator (along with a statement that the link does not imply endorsement of CRS, but is instead provided as a service) was also added.
ITS Workload & Policies Regarding Student Email Accounts
In order to address the impact of student researchers sending out bulk emails, the working group agreed that the approach that would be most consistent with the policy for limiting the number of email addresses a student could send to at one time would be to create instructions for breaking samples into groups of 50 email addresses that could be copied and pasted into Outlook Web Access. Given that it would be unusual for a research survey sample to be larger than 1,000 addresses, this will mean that at most student researchers will need to send 20 emails to reach their entire sample. Given that prior to the availability of electronic survey software, researchers would have had to send invitations by postal mail or contact subjects by telephone, the group agreed that this would not be an undue hardship. Instructions were created by members of the ITS Support Center and are available online (Mailing to Large Groups for Students Using Outlook Web Access - http://www.uwlax.edu/itssupport/docs/mailingtolargegroups.htm).
With UW-L’s recent purchase and adoption of Qualtrics, which has the ability to manage survey panels, students may be able to send survey invitations through Qualtrics without hitting the 50 address limitation. The Qualtrics working group is investigating the use of panels more thoroughly and will advise students to use this option when appropriate.
Lack of Information on Expected response rates
Just as there has been no method in place to track the volume of survey activity on campus, there has been no central mechanism for collecting information related to response rates. As determination of appropriate sample size is, as has previously been mentioned, dependent on many factors, one of which is expected response rate, the working group felt that it would be of great use to campus researchers to coordinate the collection of response rates and provide information about expected response rates back to researchers as a way of further ensuring appropriate sample sizes.
The sample request form introduces a method of tracking how many surveys are conducted using UW-L students as subjects; it also creates an opportunity to follow up with researchers to collect information about their response rates. The Office of Institutional Research will therefore follow up with all researchers after the completion of their survey to gather information about the number of reminders or prompts that were sent to survey subjects, whether any incentives for response were offered, and the total number of responses received. This information will then be summarized and made available to campus researchers to further assist them in selecting an appropriate sample size for their research. These data can then also be tracked over time to monitor for a negative relationship between survey volume and response rates so that appropriate action can be explored if such a relationship is identified.
Possible Future Directions
Throughout the development of the above process, it was remarked by several people that a method of tracking when surveys are “in the field” on campus would likely also have an impact on response rate as well as possible survey fatigue. While this supposition is likely true, given current time and resource restraints, such a project is infeasible at this time. The need has, however, been noted and may be revisited if it becomes more critical and/or additional resources are identified.