What it takes to carry out a survey

Modern society makes huge demands for information. Information has to be timely, relevant and reliable. Statistical information is used for various purposes, including the formulation of policies and decisions by businesses. Surveys are one source of statistical information.

The target population refers to the entire group from which a researcher would like to obtain information. The term survey refers to the collection of information from individuals, households or businesses. It includes both sample-based surveys, where information is gathered from a representative group within a population, or census, where all members of a determined population are interviewed. Normally, a survey is associated with gathering information from a sample.

So, for example, the government carries out a survey to evaluate the impact of existing legislation or policies. An importer examines the market potential of a new product. Different techniques can be used to capture the information, mainly mail surveys, telephone interviewing, face-to-face interviews, internet-based questionnaires and e-mail.

Sample selection is a very important stage in the business process of carrying out a survey. There are numerous recorded instances of bad sample designs. Perhaps the most classical example is the case of the survey which was carried out by The Literary Digest in the US back in 1936. At the time, Franklin Roosevelt had been President for one term. The Literary Digest predicted that Alf Landon would win the election by 43 per cent to 57 per cent. The Digest mailed over 10 million questionnaires to names drawn from lists of automobile and telephone owners. Some 2.3 million people answered the questionnaire.

In the meantime, George Gallup sampled only 50,000 people and predicted that Mr Roosevelt would win. Mr Gallup's prediction was ridiculed as naïve. But Mr Roosevelt won with 62 per cent of the vote.

The Literary Digest had made two serious mistakes. The list of names it had used was biased in favour of those who owned a car or telephone. It is important to note that in the Depression period many did not own a car or telephone. Moreover, The Literary Digest depended on voluntary response, a bias resulting in favour of those who were unhappy with Mr Roosevelt as these people were more likely to respond. The magazine went bankrupt not much later, this fact being associated by many to this result.

In a well-designed survey, the sample is selected scientifically and not haphazardly or only from people who volunteer to participate. Importantly, the lists from where samples are selected need to include every individual in the population in order to ensure that each member of the population has a known chance of being selected. Only this method ensures that results of the survey be projected on the population.

The sample size depends on many factors, mainly accuracy and resources that are available. There is no simple rule to follow, although mathematical formulae exist by which the sample size can be determined. Moreover, the level of detail needed in the results has also got to be considered as there needs to be enough representation among sub-samples that will be analysed separately.

Many are sceptical of the results of a sample survey. They question the fact that the results, based on a sample of a few hundreds or thousands, be generalised on the population. However, a properly designed survey consisting of a sample of even a few hundreds or thousands can reflect the characteristics of the population. This can be shown both practically and theoretically.

Once this point is appreciated, one will understand the benefits of carrying out sample surveys rather than exhaustive surveys. Sample surveys are less expensive, faster, easier to control and need considerably less resources to carry out when compared with censuses. However, often samples are selected from lists where information on every individual, household or business is first gathered.

By their nature, surveys give rise to sampling errors. Different samples drawn from the same population will give different results, these differences arising due to sampling errors. On the other hand, censuses do not have sampling error as every unit in the population is selected.

Moreover, detailed analysis on sub-groups may be impossible due to insufficient sample sizes of sub-samples.

Surveys have various purposes, including socio-economic research and market research. Professionals from the social, economic, psychological, health and other domains carry out surveys to measure income, expenditure, attitudes, implications of health problems, effects on the quality of life and numerous other factors.

Results derived from surveys reflect socio-economic dynamics and are used to direct policies aimed at improving people's lives. Companies use surveys to measure productivity, measure quality and focus more on customers by understanding their needs.

The following are some of the most popular techniques by which data is captured.

Mail surveys: Normally paper questionnaires are sent to respondents which they fill in and return. In order to increase response, business reply envelopes are usually attached with the questionnaires.

Telephone surveys: The use of telephone interviewing is becoming more popular. Most telephone surveys carried out by the National Statistics Office are conducted by means of computer assisted telephone interviewing (CATI). With CATI, questions appear on the computer screen and interviewers enter the information which is provided by respondents directly in a database.

The screens are programmed to appear systematically so interviewers cannot skip questions or ask them out of context. If the respondent provides a reply which is outside an accepted range, the system will indicate this and respondents will be asked to correct or confirm their reply.

Face-to-face interviewing: Usually interviewers call at the respondents' homes or workplace to interview them. This technique is normally used in surveys which are considerably long and complex in nature. Traditionally, data was captured using paper questionnaires. Nowadays, the NSO is switching to computer assisted personal interviewing (CAPI) where, similarly to CATI, data is captured by means of computers.

Internet-based surveys: The advent of the internet led to organisations which specialise in survey research to experiment with this technology. Given the limited diffusion of the internet among private households in Malta, the NSO is at present only using this technology to capture data from businesses. Monthly information is submitted via the internet by most accommodation establishments on the number of arrivals and nights spent. Other establishments which do not have access to the internet send the information either by fax or postal mail.

Organisations which carry out surveys should ensure that the data which is supplied by respondents is kept confidential and used only for research or statistical purposes. Interviewers should also be required to take an oath of confidentiality before they start working with a survey organisation. In Malta, the Statistics Authority Act XXIV of 2000, while obliging respondents to provide the information requested by the NSO, safeguards their confidentiality.

The scepticism I referred to earlier may also have been fuelled by some low quality surveys that are carried out, both locally and abroad. Nowadays it is fashionable to attach the term scientific with many surveys that are published. However, quite a number of these surveys fail to include even basic methodological information like response rate, etc.

Interviewers are sometimes instructed to select participants themselves randomly without following any systematic procedure, e.g. choosing every nth person, etc. This practice introduces bias and distorts results. At times individuals who do not accept to participate in the survey are substituted, distorting the probability of selection in the process.

Even worse, non-respondents are sometimes substituted with others who are known to the researchers or interviewer, this practice creating bias. Some of the samples are selected from lists that are not representative of the population, for example, the telephone directory.

These researchers fail to qualify that a significant number of individuals opt not to appear in the telephone directory, hence the representation of surveys which are based on samples drawn from the telephone directory is questionable. In these circumstances, one cannot but question the reliability of such surveys.

In reality, the "researchers" who carry out such research are doing a disservice to the survey research profession because people would associate all surveys with such work.

There are also concerns that call-in TV inquiries can be used to generalise results on the population. The outcome of such inquiries cannot even be used to generalise results on the viewers of the programme for the simple reason that participants are not selected scientifically.

Mr Bonnici is manager, research and methodology unit, National Statistics Office. The unit specialises in carrying out survey research among individuals, households and businesses. It is also responsible for developing survey processes, including project management, sample and questionnaire design.