About Survey Instruments: A Brief Introduction
About Survey Instruments: A Brief Introduction
For students, and those less experienced, what follows is intended
to explain what surveys do and to help one understand when and how
to use them. It is not a comprehensive tutorial, but rather an
introduction with pointers to more detailed sources. It also includes a
bibliography on materials about surveys, their construction, and their
use.
Epistemologically, surveys provide one way of obtaining and
validating knowledge. Consequently, it is important that the context
for this type of IS research be considered. Hence, this section also
provides a summary that explains when survey research is appropriate and the steps required
to use the survey approach correctly.
First, one should understand what a survey is actually used for. A survey is a way of going from observations to theory validation. As part of a panel discussion (Newsted, Chin, Ngwenyama, and Lee, 1996), Chin presented an exhibit that nicely outlines how this is done:
Observed responses
become data on single questions.
These questions are
aggregated into scales.
Appropriate numerical
formulas are applied to these numbers.
The results of these
formulas lead to conceptual representations of what has been measured.
These concepts are the constructs of interest to researchers. The usual objective for Information Systems researchers using this approach is to determine the relationship of these constructs as a way of making sense of behavior surrounding and involving IS.
The methodology used to study these relationships is also an important aspect of survey research. Techniques range from the reporting of simple scale means, the use of analysis of variance of results in different conditions, through regression analysis and the analysis of paths between constructs, to the use of "second generation" techniques such as LISREL and PLS, which analyze measurement models and structural models simultaneously.
Trochim's (1998) web pages show how some of the simpler of these approaches work. It has links dealing with:
How to select basic
statistics.
A useful glossary of
statistical terms.
Links to a variety of
other statistical sites.
Samples of applying
various techniques to the area of web page evaluation.
Chin's (1995) website provides pointers toward more advanced approaches using structural equation modeling (SEM). Among other things it discusses:
The need for a
covariance matrix to show the relations among constructs and variables.
Model specification
and graphical ways of representing a model.
Steps involved in
using structural equation models.
A link describing the
Partial Least Squares Approach (PLS) to SEM.
An extensive set of
references using the PLS approach
In a panel discussion at the 1996 International Conference on Information Systems, Newsted, Chin, Ngwenyama, and Lee (1996) raised the question whether surveys had outlived their usefulness. The panelists concluded that surveys are appropriate in certain conditions and less useful in others. But this is more complicated than just saying surveys should be used only in objective or positivist research and not in more subjective or interpretivist research. Burrel and Morgan (1979) provide a detailed distinction between these approaches.
In the '96 panel, Allen Lee indicated that in positivist research, surveys are particularly useful in determining the actual values of variables under study, and the strengths of relationships among them. In an interpretivist context surveys are appropriate as a complement to other forms of data or observations. They can serve as a way to add to one's knowledge through "triangulation" as one of several methods. Thus it is important to realize that while surveys are typically used in quantitative research, they can also help qualitative researchers as well. The panel gives further details. In particular one should review the notes below the slides in Lee's presentation as they discuss the inter-relations between these types of research and describe situations where these approaches and orientations fit together. For those interested in the qualitative approach, Myers provides an excellent site offering considerable detail on this type of research.
Overall the survey approach can be seen to have the following strengths and weaknesses:
After having decided to employ survey research it is advisable to search the literature for measures of your constructs of interest. This process can be expedited by reviewing our list of constructs or using the search engine. If you are unable to find an appropriate instrument with an associated methodology, it may be necessary to develop one. To do so requires a theoretical foundation in the area of study and an understanding of the survey process. But, even when an existing instrument is being used, an appropriate theory and a correct process are necessary.
A carefully constructed theory is a precursor to the actual use of an instrument. Theories propose constructs and their expected relations. They guide the investigation of these relations as they attempt to help one understand behavior and identify regularities in it. However, this is also a two-way street: instrument development can often refine theories as well. As Straub (1989) indicates, "Attention to instrument issues ... brings greater clarity to the formulation and interpretation of research questions. In the process of validating an instrument, the researcher is engaged, in a very real sense, in a reality check. He or she finds out in relatively short order how well conceptualization of problems and solutions matches with actual experience of practitioners." (p. 148).
A number of exemplary survey-based studies describe instrument development within a theoretical context. In the following two studies researchers have taken particular care to consider theories and to identify the steps in instrument creation and validation:
With a theoretical foundation in place, the activities of the survey process can be considered. Grover (1997) provides a detailed check list to be followed in the development and use of an instrument. Significant among these steps are:
determination of the unit of analysis (e.g., the individual, group, or
organization)
creation and use of multi-item scales
pre-testing and use of pilot data
assessment of both
construct and
content validity
assessment of
reliability
random sampling from a defined sample frame
determination of an appropriate response rate and evaluation of
nonresponse bias
assessment of whether significant correlations imply real causal
relations
determination of statistical power of the final analysis.
Dillman (1978) offers a number of very practical tips to facilitate the surveying process and increase response rates. Salant and Dillman (1994) provide a very good update to Dillman's initial classic.
The following references provide more detail on the theories and processes involved in surveying. These are divided into sources specific to IS as well general references on surveying. A broad understanding of survey methodology as well as knowledge about specific IS issues are useful in using the survey approach.
General References on Survey Research
IS-Specific References on Survey Research
Since surveying involves a number of detailed and sometimes tedious steps, it is reasonable to ask if parts of it can be automated (especially since much of the analysis of survey data has been automated). The answer is a tentative "yes." However, care must be taken to ensure that computerized surveying leads to the same results as traditional, noncomputerized approaches using paper and pencil.
Bratton and Newsted (1995) summarize much of the work in this area and show that differences in results can be due as much to different interfaces as to differences in techniques. Interestingly, fewer differences seem to be found in newer work (Liefeld, 1992; Potosky and Bobko, 1997), as compared to studies before 1990 (e.g. Liefeld, 1988; Kiesler and Sproull, 1986). Recently, however, Webster and Compeau (1996) have found that, while there may not be differences in means between computerized and noncomputerized surveying, there may be differences in correlations with other variables such as those involving training performance. While there appears to be a growing acceptance of automated survey methodologies, there is a need for caution in the use of computerized techniques.
Automated surveying can be done using the Internet. Mueller (1997) provides a good example of how surveys can be done on the Web. He uses an instrument that is part of his on-going research on computer experience.
There are a number of commercial companies that provide computerized surveying for a fee. While we have not tested these products, they appear to have useful capabilities -- perhaps, especially for those wishing to do commercial surveys in IS.
QuestionMark
Quiz Factory
Survey Said
ScreenSurvey from the National Research Council of Canada
Surveyor Manager (in French) créez, diffusez et évaluez
des questionnaires.
Jeffrey Dodgson's
List of Survey Software
ISWorld Net Navigation Map
The ISWorld Research Home Page
About Survey Instruments: A Brief Introduction [where you are now].