EDITING AND CRAFTING GOOD SURVEY QUESTIONS AND QUESTIONNAIRES: Many survey researchers will tell you that writing survey questions and crafting a survey questionnaire is often the most challenging and most important task in designing a survey research project. The challenge comes from the fact that it is difficult to write questions that avoid potential bias or ambiguity. The importance of the questionnaire can be overlooked with studies that involve a complex survey design, yet still a poor survey instrument will doom even the best designed survey research study. I am currently putting together a short book based on my current research on questionnaire design. This document will be based on my own experiences combined with what I have learned from the other researchers who have published work around the methodology of writing good survey question.
IMPROVING THE ANNUAL SURVEY OF REFUGEES: Since the 1980s, the Office of Refugee Resettlement (ORR) has conducted the Annual Survey of Refugees (ASR), which collects information on refugees during their first five years after arrival in the U.S. The ASR is the only scientifically collected source of national data on refugees’ progress toward self-sufficiency and integration. ORR uses the ASR results alongside other information sources to fulfill its Congressionally mandated reporting requirement following the Refugee Act of 1980. Tim is the ASR technical leader for this project responsible for improving the survey and sample design.
WHO HAS TROUBLE REPORTING PRIOR DAY EVENTS: Research has shown that surveys that require respondents to recall events can be subject to fairly high measurement error. Recall error tends be less problematic for highly salient events and events that have recently occurred. However, there is less information on whether some respondents are prone to having problems with answering questions that involve recalling an event. This research uses data from the American Driving Study where people are asked to report the length of driving trips that they took yesterday. We analyze of over 16,000 reported driving trips from data collected from 7,913 respondents who reported having been the driver for at least one driving trip on the day before they were interviewed (yesterday). For this analysis, we are concerned with two types of recall problems; 1) the inability of the respondent to provide an estimate of either the length or duration of a driving trip; 2) providing an estimate of miles driven that is inconsistent given the duration and purpose of the trip. This research finds difference in the characteristics of respondents who are more likely to have problems reporting their prior day driving behavior. But, the main finding is that longer trips were harder to report on and have a bigger impact on key survey estimates. As part of this research we document at important considerations that survey practitioners should keep in mind when designing surveys that include recall questions. This research has led to a recent publication in Survey Practice.
METHODOLOGY RESEARCH ON ESTIMATING ARTS ATTENDANCE: This research involves two separate issues:
First reconciling difference between Census Bureau and Academic/Commercial arts participation estimates. The Survey of Public Participation in the Arts (SPPA) was established in 1982 to provide reliable benchmarks of Americans’ participation in the arts in the context of other uses of their free time. When the 1997 SPPA was conducted by a commercial survey firm (Westat), rather than the US Bureau of the Census, the arts participation estimates were notably higher than in the previous three surveys conducted by Census in 1982, 1985 and 1987 (as well as subsequent SPPA surveys conducted by Census in 2002 and 2008). For each arts activity, the Census figures were up to 13 percentage points lower than those from Westat and other surveys. This article explores several explanations for these higher figures, not just internal factors (response rates, activity intercorrelations, survey introduction, etc.) but external estimates in relation to academic/commercial surveys conducted by the General Social Survey (GSS) and the Harris organization.
Second proxy versus self-reporting participation. The research on this issue includes a the published article Can Your Spouse Accurately Report Your Activities - An Examination of Proxy Reporting From the 2008 Survey of Public Participation in the Arts: This article uses information collected on the 2008 Survey of Public Participation in the Arts (SPPA) which is sponsored by the National Endowment for the Arts and was conducted as a supplement to the May 2008 Current Population Survey (CPS). A big challenge in being a supplement to the CPS is that respondent selection procedures for SPPA supplement differ from the CPS procedures. CPS is administered to any person 16 or older who is able to report employment information on all persons 16 years or older in the household. While the SPPA collects information on all adults 18 or older and it is felt that many questions on the survey require self-reports rather than proxy reports. In 2002, the Census Bureau interviewers tried to complete the SPPA supplement with all persons 18 or older, but after 4 call attempts they accepted proxy reports. To make the SPPA a better fit for the CPS protocol, rather than attempt to interview all adults in the household , the 2008 SPPA accepted proxy responses for spouses or partners (for many questions). This change in design makes it much easier to measure the impact of proxy reports given that they were collected by design rather than out of necessity. This research explores the extent to which proxy reporting may have resulted in over or under reporting participation. And when there are differences should you adjust your estimate? Of particular interest in the published paper were comparisons between husbands reporting about the wives activities and vice a versa.
CO-CHAIRING THE AAPOR TRANSPARENCY INIATITAITIVE: AAPOR’s Transparency Initiative is designed to promote methodological disclosure through a proactive, educational approach that helps survey organizations in developing simple and efficient means for routinely disclosing the research methods associated with their publicly released studies. The Transparency Initiative is an approach to the goal of an open science of survey research by acknowledging those organizations that pledge to practice transparency in their reporting of survey-based research findings. In doing so, AAPOR makes no judgment about the approach, quality or rigor of the methods being disclosed.
WHAT TO DO WHEN INTERVIEWING THE PERSON YOU WOULD LIKE TO SPEAK WITH IS NOT FEASIBLE OF PRACTICAL: Why should you consider accepting a proxy response? In most situations you would most likely want to question your targeted respondent directly, however many surveys allow a knowledgeable person to answer on behalf of others. There are three main reasons why surveys accept proxy responses. First there is proxy out of necessity because the individuals that you would like to interview cannon be reached or are unable to respond on their own behalf because of they may be either too young or too old to respond or cannot respond due to a physical or mental condition. Some surveys will exclude these individuals, while other surveys will accept a proxy response because excluding them would lead to bias estimates. Second there is proxy to increase the efficiency of a survey. This occurs when you accept proxy responses for people capable of providing their own information in order to save time and money. Third there is proxy to improve the quality of the data for studies in which it is believed that proxy information would be as good as or perhaps even better than the information you would obtain from a direct interview. A very common situation of proxy by design is getting information about child from a parent. We do know that in general proxy responses have the potential to be less accurate, but less is known about the degree of proxy reporting inaccuracies. This research is aimed at learning how to decide what kinds of information can be reliably collected via proxy, who makes a good proxy respondent, and how to tailor the questions to improve the accuracy of the proxy responses. Can Your Spouse Accurately Report Your Activities? An Examination of Proxy Reporting is a journal article that is based on some of this research.
UPDATING GRAND SLAM STATISTICS ON TENNIS PLAYERS: The Grand Slam tennis program is a statistical model that allows you to simulate matches between former winners of grand slam tournaments (Australian Open, French Open, Wimbledon, and the U.S. Open). I am intermittently researching an updating the accuracy of the statistics used in the model and when appropriate I will add new players. You are welcome to download a free shareware copy of the "Grand Slam Tennis" program from this web site.
Timothy Triplett's Home Page: