| Recruitment Design | Invited HHs | Recruited HHs | Completed HHs | Recruit Rate | Conversion Rate | Response Rate |
|---|---|---|---|---|---|---|
| Traditional recruitment | 10,250 | 423 | 227 | 4.10% | 54% | 2.20% |
| All-in-one recruitment | 10,250 | 374 | 180 | 3.70% | 48% | 1.80% |
| Total | 20,500 | 797 | 407 | 3.90% | 51% | 2.00% |
| Recruited households are those who complete the initial signup survey. Conversion rate is the percentage of recruited households who complete the travel diary. | ||||||
Survey Sampling
This section provides the sampling plan methodology for the three waves of the Travel Behavior Inventory. Below is a brief overview of the goals, methods and outcomes of each survey wave.
Overview by wave
2018 pre-test
Before sending invitations for the first wave of the main study (2019), a pre-test was conducted to test survey administration processes, designs, instruments (rMove™ for Web and Smartphone), and resulting data to ensure that all platforms were optimized for the main survey effort. The pre-test tested two different study designs, traditional
rMove™ and rMove™ All-in-One
(AIO). The traditional design had all respondents recruit online, and get assigned to rMove™ if eligible, while AIO had respondents complete both the recruitment survey and travel diary on rMove™. During the pre-test, 50% of invited households were instructed to recruit using the traditional design and 50% of invited households were instructed to recruit using the AIO study design.
After completing the pre-test the consultant team and the project review team analyzed the study methodology and the pre-test results. As shown in Table 1, these results indicated that the AIO study design yielded a lower recruit rate and a lower response rate than the traditional study design.
In addition to testing effectiveness of study designs, the pre-test provided insight on response rates for hard-to-reach populations within the study region. A second finding from the pre-test showed that younger and lower-income households completed at a higher rate in the All-in-One study design. These populations are typically considered hard-to-reach groups in household travel surveys, so this finding indicated a promising potential use case for the AIO study design.
2019, Wave 1
All data collected for the 2019 survey were collected through address-based sampling methods (ABS).
Based upon the pre-test findings, the project review team decided to split the sample of the main study such that 85% of participants were assigned to the traditional study design and 15% of participants were assigned to the AIO study design. Typically hard-to-survey populations received AIO invitations.
While representation in the sample was consistent with that observed in other household travel surveys, the unweighted response from hard-to-survey households was still lower than desired. The results of 2019 data collection resulted in recommendations later implemented in 2021, including:
- Increased differential incentives for households typically hard to survey or recruit.
- Increased the proportion of hard to survey households invited to participate through ABS.
- Revised the survey methodology to allow households in which all adults own a qualifying smartphone to choose whether they prefer to report travel using their smartphone, online, or through a call center.
- Implemented non-probability (convenience) sampling methods to increase the participation from hard-to-survey groups.
Read more in the Survey Methodology Report for 2019.
2021, Wave 2
The 2021 survey faced several key challenges including the COVID-19 pandemic, declining trust in government, and postal delays. Despite these historic challenges, the 2021 survey surpassed the overall survey participation target of 7,500 households.
The 2021 survey departed from the 2019 survey design in utilizing an opt-in approach where households where all adults had a smartphone could choose to report their travel by smartphone, online, or through the call center, rather than being assigned to participate by smartphone based on smartphone ownership. This resulted in 41% of participants completing the survey by smartphone, 53% online, and 6% by call center1.
The 2021 survey also included a substantial set of data collected using supplemental non-probability methods to increase the proportion of hard-to-survey households in the final dataset. ABS efforts yielded 90% of the complete households, while supplemental sampling efforts yielded 10% of the complete households. Non-probability sampling methods included outreach through community-based organizations and leveraging Metro Transit’s Transit Assistance Program (TAP) email and text lists.
RSG and the Met Council worked closely with a public outreach firm, NewPublica to coordinate an effort to invite community-based organization (CBO) members to participate in the Travel Behavior Inventory with a focus on CBOs that are primarily composed of Black, Indigenous, people of color (BIPOC) community members that were underrepresented in the 2019 survey. NewPublica coordinated with CBOs to determine the best means to invite CBO members and facilitate members invitation to participate in the survey. The CBO sample were offered higher differential incentives.
Working closely with Metro Transit, RSG invited Metro Transit customers to participate in the Travel Behavior Inventory. The goal of this method was to leverage a reasonably low-cost method to try to improve response among certified low-income populations who are known to be hard-to-survey and for whom there is some overlap with underrepresented races and ethnicities. Of specific interest was the population of riders who have been certified as low-income, receive a form of transit subsidy, and for whom an email address or mobile phone number is available. For additional details see the Survey Methodology Report for 2021 Appendix.
2023, Wave 3
As in 2021, the 2023 survey employed an opt-in approach to recruiting households to the smartphone-based travel diary. This resulted in 48% of participants completing the survey by smartphone, 39% online, and 12% by call center.
The 2023 sample plan aimed to improve recruitment of demographic groups that were underrepresented in 2019 and 2021. RSG implemented a combination of ABS and non-probability sample methods (community-based organization outreach). Due to the lower quality of the CBO outreach survey data, these records were not weighted in the final dataset. For additional details see the Outreach Report for 2023.
Sampling Methods
Sampling Goals
The 2019 Travel Behavior Inventory aimed to sample 7,500 households, which equates to a 0.20% sample rate according to data from the 2016 ACS 5-year estimates. Beyond achieving the overall sample target, the survey also aimed to ensure that the sample was representative across key demographics and behaviors, as discussed below.
Sampling Frame
The Travel Behavior Inventory region is comprised of the seven-county Twin Cities metropolitan area, nine adjoining ring counties in Minnesota, and three bordering counties in Wisconsin. RSG used ABS to select a random sample of addresses from all residential addresses in the study area. Using this method, all households within each defined area have an equal chance of selection for the sample. The sampled addresses were purchased from Marketing Systems Group (MSG), which maintains the Computer Delivery Sequence file from the U.S.
When purchasing the addresses, RSG also purchased the estimated household income for the list of addresses. Typically, MSG provides an estimated income for about 85-90% of the total addresses at a cost of $0.01 per address. The estimated household income data was used to aid address selection.
Address-Based Sampling
Sample Segmentation
RSG stratified the sample using census Block Group (BG) data from the most recently available 2012–2016 American Community Survey 5-year estimates (ACS). The most detailed way to stratify the sample is to use census BGs, which are the smallest geography for which most census and ACS tables are publicly available. Each BG generally contains between 600 and 3,000 people. According to this ACS data, the study region contains 1.4 million households and 3.6 million persons. Group Quarters, excluded from the sample frame, are a relatively small segment of the population at 2%.
Sampling planning methodologies employed in the Travel Behavior Inventory mitigated non-response bias and other known biases to meet study targets.
For the 2019 survey, RSG coordinated with the Met Council to determine four key groups to sample. Based upon the Met Council’s interest in proportional representation from low income, rural, and limited English-speaking households for the main study, RSG proposed the following mutually exclusive and collectively exhaustive sample segments:
- Core-Urban BGs: Comprised of the BGs in the Twin Cities seven-county metropolitan area that do not qualify for the hard-to-reach segment below and which are designated as Urban in the Thrive MSP 2040 Community Designations.
- Core-Rural BGs: Comprised of the BGs in the Twin Cities seven-county metropolitan area that do not qualify for the hard-to-reach segment below and which are designated as Rural in the Thrive MSP 2040 Community Designations.
- Rural Ring BGs: Comprised of the BGs in the twelve ring counties surrounding the seven-county metropolitan area that do not qualify for the hard-to-reach segment below.
- Hard to reach BGs: Comprised of BGs in the nineteen-county study region that were in the 90th percentile of BGs with the highest percentage of households with annual incomes below $25,000 and/or the 90th percentile of BGs with the highest percentage of limited English-speaking households.
Starting in 2021, RSG proposed a more specific focus on sampling residents who are Hispanic and/or Black, Indigenous, and people of color (BIPOC). Sample segments for the 2021 and 2023 surveys built upon the 2018-2019 Travel Behavior Inventory, but dropped the Hard-to-Reach
segment in favor of sub-segmenting the Core Urban geography into five groups, making the final segments:
- Core-Urban BGs – Group 1: Comprised of the BGs in the Twin Cities seven-county metropolitan area which are designated as Urban in the Thrive MSP 2040 Community Designations and whose population is at least 80% Hispanic and/or BIPOC.
- Core-Urban BGs – Group 2: Comprised of the BGs in the Twin Cities seven-county metropolitan area which are designated as Urban in the Thrive MSP 2040 Community Designations and whose population is 60%-80% Hispanic and/or BIPOC.
- Core-Urban BGs – Group 3: Comprised of the BGs in the Twin Cities seven-county metropolitan area which are designated as Urban in the Thrive MSP 2040 Community Designations and whose population is 40%-60% Hispanic and/or BIPOC.
- Core-Urban BGs – Group 4: Comprised of the BGs in the Twin Cities seven-county metropolitan area which are designated as Urban in the Thrive MSP 2040 Community Designations and whose population is 20%-40% Hispanic and/or BIPOC.
- Core-Urban BGS – Group 5: Comprised of the BGs in the Twin Cities seven-county metropolitan area which are designated as Urban in the Thrive MSP 2040 Community Designations and whose population is less than 20% Hispanic and/or BIPOC.
- Core-Rural BGs: Comprised of the BGs in the Twin Cities seven-county metropolitan area which are designated as Rural in the Thrive MSP 2040 Community Designations.
- Rural Ring BGs: Comprised of the BGs in the twelve ring counties surrounding the seven-county metropolitan area.