Survey Design and SoTL

The following information is available as a PDF

Things to consider before drafting a survey

  • What are some general concepts I am interested in? (e.g., how useful were parts of the course? How much do students feel they learned from the course as a whole? What suggestions do students have about improving the course?)
  • What variables can I use to measure these concepts?
  • What are the specific questions which help me get at these concepts (e.g., how useful were the lectures? How useful was assignment 1?)
  • What demographic questions can I ask that help me address my research questions (e.g., gpa, expected grade, gender, major)?
  • What quantitative or qualitative analyses can I use to answer the questions that I have?

Survey Question Types

Open ended (i.e., fill in the blank)

  • Short answer
  • Paragraph
  • List

Closed ended

  • Multiple choice (e.g., indicate your major) – Easy to gather data for variables when only one answer is possible
  • Checklist (e.g., select all pre-requisite courses you have taken) – Easy to gather large amounts of data especially for multiple grouping variables when each answer can be present/absent
  • Rating scale – easy to count for small scales and easy to convert to numbers for quantitative analysis
    • e.g., 1=strongly disagree; 2=disagree; 3=neither agree/disagree; 4=agree; 5=strongly agree
    • Keep category wordings as “evenly spaced”, balanced, and non-overlapping as possible
    • If absolutes (e.g., Always, Never) are used are these likely or meaningful?
    • 4-7 choices are most typical
    • Choices should generally be qualitative or both numerical and qualitative (e.g., 1=strongly disagree, 2=disagree… etc.)
    • Is a neutral response choice (odd number of categories) desired or is “forced choice” preferred (even number of categories)
    • Consider if a do not know/not applicable option should be included

Best Practices in Creating Questions

  • Relate to variables/concepts you are interested in and analyses you plan to use
  • Are clear and easy to interpret by intended reader. Have definitions and/or clarifications if required
  • Answers can be recalled easily by participants (e.g., didn’t happen too long ago, isn’t too hard to estimate)
  • Word questions positively (e.g., how useful was the assignment?) rather than negatively (e.g., how useless was the assignment?)
  • Questions should be concise
  • Not leading or biasing (do not “suggest” an answer to the question, word as neutrally as possible)
  • Not “double barrelled” (e.g., “do you like science and math?”)
  • When reviewing your questions, ask the following:
    • Have all concepts of interest been represented by different questions?
    • Are all general concepts sufficiently covered by multiple questions (at least 3-5) to assess them?
    • Can the literature help inform question wording and/or variable choice(s)?
    • Does each question have a clear purpose?
    • Do I have an idea of how this question will be used in the analysis and why I’m asking it?

Using Validated Instruments

  • Even if it is “validated” not all validated scales are created equal and modification of an instrument will require further work and reassessment
  • Questions to consider regarding the validity of an instrument:
    • Where was it published? Has it been improved over time?
    • How frequently is it used/cited?
    • What methods were used to validate the instrument? How rigorous was this and how large/generalizable was the population used?
  • How well-suited is this instrument for your study?
    • How closely does the construct in the instrument match your interest?
    • Has it been tested on a similar population to mine?
    • What adaptations might be needed for my own research?

(Note: changing an instrument will “break” its validation)

Example Scales

1=Disagree, 2=Somewhat Disagree, 3=Neither Agree Nor Disagree, 4=Somewhat Agree, 5=Agree, Not Applicable or Do Not Know

1=Almost Never, 2=Seldom, 3=Sometimes, 4=Often, 5=Very Often, Not Applicable or Do Not Know

1=Very Poor, 2=Poor, 3=Satisfactory, 4=Good, 5=Very Good, Not Applicable or Do Not Know

1=Very Little, 2=A Little, 3=Somewhat, 4=A Lot, 5=Quite a Lot, Not Applicable or Do Not Know

1=Mostly Not Useful, 2=Not Very Useful, 3=Somewhat Useful, 4=Very Useful, 5=Extremely Useful, Not Applicable or Do Not Know

Further Reading

For more in-depth information on scale development and survey design, we recommend the following resources:

DeVellis, R. F. (2003). Scale Development: Theory and Applications (Vol. 26). Thousand Oaks, CA: Sage Publications.

Krosnick, J. A., and Presser, S. (2010) Question and Questionnaire Design. In P.V. Marsden, and J.D. Wright (Ed.), Handbook of Survey Research (2nd ed, pp. 263-313). Bingley, WA, UK: Emerald Group Publishing Limited.