Tradeoffs

The last topic I will discuss in the Research Design section is the tradeoffs. One of the most important activities when designing a research approach is understanding the tradeoffs that we have to make. For instance, issues like sampling (including procedures and size), the precision of measures, and the number of variables are common.

Sampling procedures should be determined to obtain the sample that represents the population that the researcher wants to study. But if you consider availability and cost for samples, the sample my hurt external validity in terms of sample representativeness.

• Ideal: Ensure that you have access to a large population, from which you can take truly randomized samples. If using a convenience sample, ensure sample relevance by making sure the characteristics of the population meet the boundary conditions of the theory.
• Tradeoff:  Access is most often determined by relationships, grants, and blind luck, which force the researcher to compromise on the above ideals.

Random sampling procedure will help to make our sample representative of the population that we want to study. Very costly and is sometimes impossible.  Convenient samples are much more easily accessible and are at relatively low cost.  However, using a convenience sample (e.g., undergraduates) may hurt sample representativeness.

Large sample size increases statistical power and reduces the possibility of Type II errors. But overpowered samples can be wasteful in terms of unnecessary effort, time, and resources spent, and may be oversensitive to trivial or irrelevant "significant" findings (Mone, et al., 1996)

• Ideal:  As big of a sample size as possible, to limit type II error, and to ensure power.  
• Tradeoff:  Overpowered studies, will waste time and resources by using overly large samples.

Large number of measures increases the information about reliability of validity (increases construct validity), but it may decrease a participant’s motivation, and may increase the participant’s awareness of the test/manipulation. Implementing a small number of measures saves time and costs but will provide little information about reliability and validity.

• Ideal: One should typically strive for using multiple measures to increase the validity of findings.
• Tradeoff: May lead to the measurement of a different though possibly related construct. 

Trying to increase the precision of measures reduces error variance, but then may become meaningless measures for our research. Trying to increase the meaningfulness of measures tends to result in less precise measures.  In short, low generalizability → low error variance, while high generalizability → high error variance.

• Ideal: To improve measurement, researchers will often try to develop a new instrument.
• Tradeoff:  A huge investment in time and effort and no normative data based.

Increase in the strength of measures enables us to detect significant effect on the dependent variables, but it makes reactive treatment and thus the problems of demand characteristics and evaluation apprehension will occur. Decrease the strength of measures decrease the participants' awareness of the manipulation and thus increase external validity, but it may cause range restriction, small effect size, and Type II error will increases.

Strength of manipulation: Strengthening the manipulation increases effect size and enables the researcher to detect the significant effect of the manipulation. But that increase can lead to the participant’s awareness of treatment and can cause apprehension evaluation.  : In field settings, the range of independent variables that the researcher want to manipulate can be large, and the variance of dependent variable will be large. However, it is difficult to attribute the large difference in dependent variables to the manipulation because of a lot of noise. On the other hand, in laboratory settings, the researcher can manipulate variables more precisely than filed settings. But the range and strength of the variables tend to be small, and the effects to the dependent variable will also be small.

• Ideal: You want to maximize systematic (experimental) variance.  Design, plan and conduct research so that the experimental conditions are as different as possible.  Similarly you want to control extraneous systematic variance, and minimize error variance.
• Tradeoff:  When everything is constant, you can always show an effect. Lose generalizability.

A large number of variables can make a model more comprehensive, but it will also increase complexity and the difficulty of analysis.  It becomes difficult to make causal inferences both logically and statistically. Highly sophisticated statistical procedures must be used when there are many independent/dependent variables.  Also, effects of some independent variables may be small.  A small number of variables will enable the researcher to focus on each specific variable more and in turn will make the model easier to analyze. Tradeoff is between external (more variables) and internal (fewer variables) validity.

• Ideal: Each dependent variable is the presumed effect of one or more independent variables as an antecedent.  When operationally defined they are observable and measurable.
• Tradeoff:  Variables are difficult to define and measure. Problems of generalizability related to paper-people studies.  Problems also encountered with self-report measures, due to differences between behavior intention and actual behavior

Conclusion
There is no perfect research. The choice of design depends on what type of information the researcher wants. Therefore, it is imperative to ask the right questions ahead of time. This will lead to a more effective design.  Also, it is often preferable to use more than one design if possible (triangulation), as this will give us more useful information.

(Adapted from group and course notes)

(Flashcards and other resources here)