Advantages of surveys

The great advantage of survey methods is economy of scale. They’re a quick, cost-effective way of gathering large amounts of information and generating usable knowledge, particularly on attitudes and behaviour, including:

• Investigating perceptions and opinions across large numbers of people.
• Discovering common patterns within groups.
• Comparing differences between groups.
• Gathering baseline data to assess subsequent trends or changes.
• Enabling generalisations, when sample size permits.

This versatility makes surveys a useful method throughout the design process, from identifying relevant issues at the outset to getting feedback on options, through to post-completion evaluation. They’re a familiar format for most participants, and it’s reasonably simple to create and run a survey with the user-friendly software available. Packages normally include calculation and analysis features, so statistical ability isn’t required either.

Disadvantages of surveys

Surveys, methodologically speaking, aren’t quite as straightforward to design as they may seem. It’s very easy for the questions, answer options or structure to unwittingly skew the results. I note these pitfalls not to dissuade anyone from running surveys but as a reminder that flawed surveys yield flawed information, which no one wants.

Firstly, opportunities for bias abound (see the boxed text at the end of section 6.3). Surveys rely on self-report, asking respondents to tick the box that best describes their behaviour or attitudes. However, respondents may also give what they think are correct or desirable answers; they may interpret questions in unforeseen ways, or they may under-or over-state their attitudes or behaviour. They might find the questionnaire dull and stop considering their answers carefully, or it might not include the questions or answer options they want. But their responses must still be accepted at face value.

Secondly, can the complexity of human experience and behaviour really be captured in simple answers to simple questions? A survey works fine for uncontroversial or uncomplicated topics, but not when nuanced or in-depth information is needed.

And I’m afraid there’s more. Do the findings represent the views of the silent majority who didn’t complete the questionnaire? Surveys can reflect what people usually do or prefer, but are less effective at finding out why. Behaviour and feelings may also change over time, so a dataset is a snapshot of a specific moment but has a limited shelf life, especially so in a design context where attitudes towards a proposal will often change as it develops. Don’t let this put you off using surveys; they can yield highly valuable data, but careful construction and testing beforehand are essential.

Sampling

The sampling strategy, in research terms, is the method of choosing participants for a project out of all the potential participants. In spatial design, the nature of the proposal and its context can determine the sampling strategy. Sometimes it’s possible to engage with all those potentially affected by a development – every household in a street, for example – so the strategy might aim for a 100% sample. But when this isn’t practicable, how to decide who participates? For a public realm proposal, the sampling strategy might focus on maximising the participation of specific groups or achieving a minimum number of responses. Is it likely that survey participants’ responses might vary according to demographic factors, postcode area and so on? If so, survey enough people to yield usable data and allow valid comparisons between groups. And if the initial survey doesn’t receive many responses from certain groups, get more responses from people in those groups until there’s enough to work with. Be prepared to make direct contact and offer active encouragement to marginalised groups to ensure they have their say. Remember that for survey results to have any reliability, groups whose views may differ from the majority must be included, and the extent of those differences understood. (This might mean a follow-up meeting or focus group, for example.)

Figure 6.1 Suggested survey process

Piloting

The piloting phase is perhaps the most critical part of the survey process. Don’t even think about launching a questionnaire without testing it first on colleagues, friends, community contacts and anyone with a few minutes to spare.

Piloting has three purposes:

1. Identifying technical issues in completing the questionnaire.
2. Checking whether the questions and answers are clear, and follow a logical structure.
3. Noting the time needed for completion (normally longer than expected!)

Keep fine-tuning the questions and re-testing until no further improvements can be made. If there are questions with an ‘Other, please specify’ option, check for frequently occurring responses there that could be added to the multiple-choice options; also check whether free-text responses will be analysable (a word limit or different question wording might be necessary, for instance). Before going live, double-check that printed questionnaires are identical to the final online version, to ensure consistency when these responses are input manually.

Test analysis

Run some test reports on the pilot responses and/or some dummy data, and make sure that data appears in the correct format and fields, that it can be sorted, filtered and analysed to the required level of detail, and does everything else needed. Finally, ask the pilot respondents for honest feedback on the experience of completing the survey. Was it repetitive? Was it engaging? Were the answer options adequate? Did they feel there were any leading questions?

And on the subject of bias, the pilot phase is the last chance to decide whether the questions are actually going to generate the information required. Is anything missing? Are the issues relevant to the people who will be responding? Does the survey reflect their concerns and interests, or that of the design team’s? (Hint: it should be theirs, as far as possible.)

Good structure

A logical questionnaire flow helps respondents comprehend questions more readily and answer more easily. Start with a short introduction explaining the survey’s purpose, who it’s for, approximate completion time (noted at the pilot stage) and guidance on completion (if questions can be skipped, or if it can be saved to finish later, for instance). The opening questions should be quick and easy for respondents to answer and put them at ease, and usually ask for factual details like age, gender, location or other necessary personal information.

Use the research strategy questions to develop the body of the questionnaire. Identify the main areas to cover, resisting the temptation to include as many subjects as possible, and refine these into no more than five topics. Then adopt the ‘funnelling’ technique to arrange the topics, putting the more straightforward ones at the start and proceeding to the more challenging. Briefly introduce the topic at the start of each section, as though changing the subject in conversation. Some questions may need routing (for example ‘If “No”, please go to Question 4’), but do this only when really necessary, to avoid potential confusion. Close the questionnaire with an ‘Any other comments’ field, some final thanks, information about next steps and further opportunities for participation, and contact details in case of any queries. Figure 6.2 illustrates the suggested structure.

Figure 6.2 Suggested questionnaire structure

Checking the data

There are some checks that are worth performing before beginning any analytical work.

Firstly, look at the response rate. How does the number of potential respondents (if known) compare with the number of completed questionnaires? The number of completed responses compared to total number of survey website visitors can be also interesting. There’s no standard definition of a ‘good’ response rate; it depends partly on the nature of the community surveyed. A low rate might still be representative in a more homogenous neighbourhood, whereas a higher rate will be preferable somewhere that a diverse range of views is likely.

Secondly, in a larger scale survey, check how far responses reflect local demographics, if possible. Are any groups especially over- or under-represented? If so, this needs to be borne in mind at the reporting stage, and it might be necessary to invite some groups directly to submit their views for more representative findings. The demographic breakdown of respondents will never precisely mirror the local population, but try and identify any major imbalances before drawing conclusions from the data.

Finally, check to see how many respondents dropped out of the questionnaire, and where; it reflects on the questionnaire design. A high drop-out rate can indicate that a questionnaire was too long, boring, difficult to understand or didn’t ask relevant questions. If participation isn’t pitched at the right level or capturing people’s interest, this should be addressed before the process goes much further.

Analysing the data

Queries and reports can be run now. Any survey software will offer some analytical functionality, but it may be preferable to import the data into a spreadsheet to work on. There will be a mass of numerical data, and if open-ended questions were included, there will be text-based answers to analyse as well. Starting with the number crunching, keep this aspect of the analysis simple, unless a stats expert is available. Identify the obvious headline figures, such as strong majority views or trends, for instance, and also issues where opinion is clearly divided. Where there are no strong signals from the responses, it’s enough to note that the data is inconclusive.

If a good cross-section of the community has responded in sufficient numbers, the data may allow comparison between groups: looking at responses according to postcode or demographics, for instance. Small samples don’t allow valid comparisons, and neither do responses from groups of fewer than ten within the dataset. But if the data permits, check whether responses from specific groups differ much from the whole and report any significant variations; it’s important that the majority view isn’t perceived as representative of the whole community. Remember that comparisons between percentage figures need careful wording to avoid misunderstandings. For example, if 30% of respondents want Option A and 60% want Option B, this is often mistakenly expressed as ‘30% more people preferred Option B to Option A’, when in fact 100% more – twice as many – preferred Option B. Also remember to be clear that figures refer to the percentage of those who responded to that question, which may or may not be many of the total respondents, or a representative sample overall; see ‘How to lie (unintentionally) with statistics’ at section 9.2 for a real-world example.

Analysing free-text responses requires a different approach. If answers to the ‘Other’ option in multiple-choice questions are short and simple, it might be possible to treat them as quantitative data that can be roughly grouped and counted (an exception to my general advice not to render qualitative data as statistics). Longer and more detailed responses to open-ended questions require the most work. It may be that the research objectives can provide a structure for categorising these initially. It’s likely more detailed work will be needed, however, depending on the volume and length of responses. Methodical approaches are set out in the ‘Quick guide to analysing qualitative data’ on p. 60, which looks for themes and connections in the responses, grouping and regrouping the data in different ways to make sense of it as a whole. In analysing free-text responses, be open to any unexpected issues that seem significant and note them in the analysis even if they weren’t in the initial strategy.

The design team, respondents and the client will often find a one-page summary of the headline figures helpful, covering findings on key issues and other significant results. Graphs, charts and infographics make research results more intelligible to many people, so include plenty of illustrations. Be transparent in reporting the results and represent the whole range of views, not just the majority or those who support the proposal. The purpose is to inform the design and work with the public response, so be open about the feedback received. Check the ‘Quick guide to reporting research’ on p. 136 for more guidance on presenting findings and messages.