Survey Design and SoTL

Things to consider before drafting a survey

What are some general concepts I am interested in? (e.g., how useful were parts of the course? How much do students feel they learned from the course as a whole? What suggestions do students have about improving the course?)
What variables can I use to measure these concepts?
What are the specific questions which help me get at these concepts (e.g., how useful were the lectures? How useful was assignment 1?)
What demographic questions can I ask that help me address my research questions (e.g., gpa, expected grade, gender, major)?
What quantitative or qualitative analyses can I use to answer the questions that I have?

Open ended (i.e., fill in the blank)

Closed ended

Multiple choice (e.g., indicate your major) – Easy to gather data for variables when only one answer is possible
Checklist (e.g., select all pre-requisite courses you have taken) – Easy to gather large amounts of data especially for multiple grouping variables when each answer can be present/absent
Rating scale – easy to count for small scales and easy to convert to numbers for quantitative analysis
- e.g., 1=strongly disagree; 2=disagree; 3=neither agree/disagree; 4=agree; 5=strongly agree
- Keep category wordings as “evenly spaced”, balanced, and non-overlapping as possible
- If absolutes (e.g., Always, Never) are used are these likely or meaningful?
- 4-7 choices are most typical
- Choices should generally be qualitative or both numerical and qualitative (e.g., 1=strongly disagree, 2=disagree… etc.)
- Is a neutral response choice (odd number of categories) desired or is “forced choice” preferred (even number of categories)
- Consider if a do not know/not applicable option should be included

Best Practices in Creating Questions

Relate to variables/concepts you are interested in and analyses you plan to use
Are clear and easy to interpret by intended reader. Have definitions and/or clarifications if required
Answers can be recalled easily by participants (e.g., didn’t happen too long ago, isn’t too hard to estimate)
Word questions positively (e.g., how useful was the assignment?) rather than negatively (e.g., how useless was the assignment?)
Questions should be concise
Not leading or biasing (do not “suggest” an answer to the question, word as neutrally as possible)
Not “double barrelled” (e.g., “do you like science and math?”)
When reviewing your questions, ask the following:
- Have all concepts of interest been represented by different questions?
- Are all general concepts sufficiently covered by multiple questions (at least 3-5) to assess them?
- Can the literature help inform question wording and/or variable choice(s)?
- Does each question have a clear purpose?
- Do I have an idea of how this question will be used in the analysis and why I’m asking it?

Even if it is “validated” not all validated scales are created equal and modification of an instrument will require further work and reassessment
Questions to consider regarding the validity of an instrument:
- Where was it published? Has it been improved over time?
- How frequently is it used/cited?
- What methods were used to validate the instrument? How rigorous was this and how large/generalizable was the population used?
How well-suited is this instrument for your study?
- How closely does the construct in the instrument match your interest?
- Has it been tested on a similar population to mine?
- What adaptations might be needed for my own research?

(Note: changing an instrument will “break” its validation)

1=Disagree, 2=Somewhat Disagree, 3=Neither Agree Nor Disagree, 4=Somewhat Agree, 5=Agree, Not Applicable or Do Not Know

1=Almost Never, 2=Seldom, 3=Sometimes, 4=Often, 5=Very Often, Not Applicable or Do Not Know

1=Very Poor, 2=Poor, 3=Satisfactory, 4=Good, 5=Very Good, Not Applicable or Do Not Know

1=Very Little, 2=A Little, 3=Somewhat, 4=A Lot, 5=Quite a Lot, Not Applicable or Do Not Know

1=Mostly Not Useful, 2=Not Very Useful, 3=Somewhat Useful, 4=Very Useful, 5=Extremely Useful, Not Applicable or Do Not Know