Tradeoffs

The last topic I will discuss in the Research Design section is the tradeoffs. One of the most important activities when designing a research approach is understanding the tradeoffs that we have to make. For instance, issues like sampling (including procedures and size), the precision of measures, and the number of variables are common.

Sampling procedures should be determined to obtain the sample that represents the population that the researcher wants to study. But if you consider availability and cost for samples, the sample my hurt external validity in terms of sample representativeness.

• Ideal: Ensure that you have access to a large population, from which you can take truly randomized samples. If using a convenience sample, ensure sample relevance by making sure the characteristics of the population meet the boundary conditions of the theory.
• Tradeoff:  Access is most often determined by relationships, grants, and blind luck, which force the researcher to compromise on the above ideals.

Random sampling procedure will help to make our sample representative of the population that we want to study. Very costly and is sometimes impossible.  Convenient samples are much more easily accessible and are at relatively low cost.  However, using a convenience sample (e.g., undergraduates) may hurt sample representativeness.

Large sample size increases statistical power and reduces the possibility of Type II errors. But overpowered samples can be wasteful in terms of unnecessary effort, time, and resources spent, and may be oversensitive to trivial or irrelevant "significant" findings (Mone, et al., 1996)

• Ideal:  As big of a sample size as possible, to limit type II error, and to ensure power.  
• Tradeoff:  Overpowered studies, will waste time and resources by using overly large samples.

Large number of measures increases the information about reliability of validity (increases construct validity), but it may decrease a participant’s motivation, and may increase the participant’s awareness of the test/manipulation. Implementing a small number of measures saves time and costs but will provide little information about reliability and validity.

• Ideal: One should typically strive for using multiple measures to increase the validity of findings.
• Tradeoff: May lead to the measurement of a different though possibly related construct. 

Trying to increase the precision of measures reduces error variance, but then may become meaningless measures for our research. Trying to increase the meaningfulness of measures tends to result in less precise measures.  In short, low generalizability → low error variance, while high generalizability → high error variance.

• Ideal: To improve measurement, researchers will often try to develop a new instrument.
• Tradeoff:  A huge investment in time and effort and no normative data based.

Increase in the strength of measures enables us to detect significant effect on the dependent variables, but it makes reactive treatment and thus the problems of demand characteristics and evaluation apprehension will occur. Decrease the strength of measures decrease the participants' awareness of the manipulation and thus increase external validity, but it may cause range restriction, small effect size, and Type II error will increases.

Strength of manipulation: Strengthening the manipulation increases effect size and enables the researcher to detect the significant effect of the manipulation. But that increase can lead to the participant’s awareness of treatment and can cause apprehension evaluation.  : In field settings, the range of independent variables that the researcher want to manipulate can be large, and the variance of dependent variable will be large. However, it is difficult to attribute the large difference in dependent variables to the manipulation because of a lot of noise. On the other hand, in laboratory settings, the researcher can manipulate variables more precisely than filed settings. But the range and strength of the variables tend to be small, and the effects to the dependent variable will also be small.

• Ideal: You want to maximize systematic (experimental) variance.  Design, plan and conduct research so that the experimental conditions are as different as possible.  Similarly you want to control extraneous systematic variance, and minimize error variance.
• Tradeoff:  When everything is constant, you can always show an effect. Lose generalizability.

A large number of variables can make a model more comprehensive, but it will also increase complexity and the difficulty of analysis.  It becomes difficult to make causal inferences both logically and statistically. Highly sophisticated statistical procedures must be used when there are many independent/dependent variables.  Also, effects of some independent variables may be small.  A small number of variables will enable the researcher to focus on each specific variable more and in turn will make the model easier to analyze. Tradeoff is between external (more variables) and internal (fewer variables) validity.

• Ideal: Each dependent variable is the presumed effect of one or more independent variables as an antecedent.  When operationally defined they are observable and measurable.
• Tradeoff:  Variables are difficult to define and measure. Problems of generalizability related to paper-people studies.  Problems also encountered with self-report measures, due to differences between behavior intention and actual behavior

Conclusion
There is no perfect research. The choice of design depends on what type of information the researcher wants. Therefore, it is imperative to ask the right questions ahead of time. This will lead to a more effective design.  Also, it is often preferable to use more than one design if possible (triangulation), as this will give us more useful information.

(Adapted from group and course notes)

(Flashcards and other resources here)

Method Variance

I will briefly discuss method variance, as I have mentioned it as a potential problem in previous posts. It is generally understood that research focusing on macro-level questions (such as strategy) rarely use laboratory experiments, and rarely have the problem of common method variance.

However, each time a research question emerges and the research design is created, it is important to remember and know the whole set of approaches available.

Method variance refers to variance that is attributable to the measurement method rather than to the construct of interest (Podsakoff, Mackensie, Lee, and Podsakoff 2003).

Method variance represents a major problem in research design since it can have a substantial impact on the observed relationships between predictor and criterion variables in organizational and behavioral research. Method variance:

• Is one of the main sources of measurement error.
• Threatens the validity of the conclusions about the relationships between measures 
• Provides an alternative explanation for the observed relationships between measures of different constructs that is independent of the one hypothesized
• Yields potentially misleading conclusions

In order to address this problem there are procedure and statistical remedies.

Procedure remedies

• Obtain measures of the predictor and criterion variables from different sources.
• Temporal, proximal, psychological, or methodological separation of measurement
• Protecting respondent anonymity and reducing evaluation apprehension.
• Counterbalancing question order.

Statistical Remedies

• Herman’s single-factor test
• Partial correlation procedures designed to control for method biases.
• Controlling for the effects of a directly measured latent methods factor 
• Controlling for the effects of a single unmeasured latent method factor 
• Use of multiple-method factors to control method variance
• Correlated uniqueness model 

(Adapted from group and course notes)

(Flashcards and other resources here)

Context or Setting

This discussion is an extension on the Lab, Field, and Survey post.

The place where we conduct research on organizations can have a significant impact not only on how we conduct research but also on the results of our studies and investigations. This means paying attention to methodological fit, and how we match the type of research question we wish to explore with the type of setting to use for it.

The first step to consider, is what the current state of theory is at.  Next, we will select a specific setting, which will have implications for the amount of control we have over extraneous variables.

• Nascent Theory Research
• Here,  little or no previous theory exists. Researchers do not know what issues might emerge from the data so they avoid hypothesizing specific relationships between variables.
• Intermediate Theory Research
• This draws from prior work to propose new constructs and/or provisional theoretical relationships.  Careful analysis of both quantitative and qualitative data increase confidence that the explanations are more plausible than alternatives.
• Mature Theory Research
• Encompasses precise models, supported by extensive research on a set of related questions in varied settings.  Research is elegant, complex and logically rigorous.  Questions tend to focus on elaborating, clarifying or challenging specific aspects of existing theories.

Research performed in organizations (i.e., in the field) differs from research performed in a lab. Some experiments can only be performed in the field, while others will benefit greatly from being studied in a lab where distractions are minimal and there is little chance of subjects discussing the experiment.

As Bouchard points out in his 1976 paper, field settings differ from lab settings in 3 major ways:

• Boundary factors
• Intensity, e.g. firings, layoffs, demotions, transfers in the workplace, etc.
• Range, e.g. group size, span of control, time span, even physical spaces
• Duration and frequency – ability to cross response system threshold. 
• Structural factors
• Natural time constants – only field studies allow us to study natural temporal structures
• Natural units – there is “intrinsic order” in every field environment that we can study
• Complexity 
• Factors that broaden type of questions asked (not unique to the field)
• Setting effects – field settings are very complex with a large number of forces at work in any given situation; they are dynamic and a relationship found at time T1 may not be found at time T2. 
• Representativeness – while lab studies are fairly strong on representativeness of subjects, they are weak on representativeness of treatments. 

The instruments we use for conducting research in organizations will depend not only on the object of our study, but also on what is feasible within an organization. Some organizations may be completely open to an experimental setting, whereas others may only be willing to provide retrospective documentation and yet others will only be willing to administer surveys and questionnaires without participating in anything else.

• Qualitative/Survey Research
• Tools include Interviews and Participant Observation, as well as various Unobtrusive Measures (archival data, physical evidence).
• Field Experiments
• You can achieve better analysis using selection bias models and propensity score analysis. Useful also is the Ecological Momentary Assessment (EMA), which captures momentary behaviors and tracks them over time.
• Laboratory Experiment
• Different types of tools could include Standard Laboratory Experiments, Free Simulations and Experimental Simulations. 

Depending on what is available within an organization, the budget, and project objectives researchers will decide what instrument is better to use. Bouchard specifies 5 main methods to use in a field setting:

1. Interview – there are different types of interviews and the type we use will depend on our research objectives as well as the organization. 
2. Questionnaires – it’s a good idea to involve some of the respondents in the construction of the questionnaire (influential and most competent is a huge plus). 
3. Participant observation – based on theory that an interpretation of an event may be correct only when it is a composite of 2 points of view, inside and outside. 
4. Systematic observation – includes self-observation (self-reports, diaries, checklists), analysis of verbal material.
5. Unobtrusive measures, i.e. archives, physical traces, simple observation, etc. 

Overall, field research is considered the most realistic and “is where the generality, applicability, and utility of psychological knowledge are put to the test” (Bouchard 1976). Field research can provide invaluable insights that lab studies will not be able to cover. However, we need to remember that field research is often more difficult, ambiguous, and costly and may not be generalizable.  Methodological choices can enhance or diminish the ability to address particular research questions.  The appropriate choices will result in fit through the logical pairing between methods and the state of theory development when a study is conducted.

(Adapted from group and course notes)

(Flashcards and other resources here)

Qualitative Techniques

Qualitative techniques for data collection are suited for the exploratory studies when little is know about the phenomena. That is, these techniques are well suited for the early stage of the five-step logical path for the programmatic research (McGrath, 1964).  However, these techniques are not suited for the testing theories or making causal inferences, as they do not allow for manipulation or control of variables. As Lee et al (1999) suggest, qualitative techniques are well suited for the purpose of description, interpretation, and explanation but are not suited for issues of prevalence, generalizability, and calibration.

Some qualitative techniques for gathering data were discussed in the Lee, Mitchell, and Sablynski 1999 paper:

1. Observation:
• Relatively passive and nonintrusive, mainly to acquaint the researcher with the site and its members. Used early in a qualitative study. Two variants of this technique: take an organizational training course or to function as an actual employee.
• Direct, Systematic Behavioral Observation
• Use this method when the research question deals with overt behavior
• Involve explicit, systematic procedures for observing, recording, and categorizing behavior
• Choice of behavioral units to record is critical
• Considerable training required to achieve reliability in coding behavior
• Can either be a “fly on the wall” or become an active participant
2. Access archival records 
• Also passive and nonintrusive. Although “archival records” may not be a study’s main source of data, they can effectively confirm, supplement, or elaborate upon one’s more primary information. Potential problems include reactivity of measures, reliability of data, and construct validity of measures such as documents and records.
3. Interviews
• More active and intrusive, used most frequently. Vary in duration, formality, number of people interviewed at one time, and how data are recorded. Researchers should decide ahead of time re duration, formality, number of people interviewed at one time, and how data are recorded AND be able to explain these decisions during peer review process.  One potential con: the observed and/or the examined people are almost always aware that they are being monitored. 
4. Questionnaires 
• More active and intrusive. Because questionnaires reduce spontaneity, inhibit free-flowing speech, and constrain ones’ manner, they provide more supplemental (rather than primary) data, similar to archival records. Nevertheless, questionnaires can “orient” the respondent and get everyone “on the same page.”

If interview and questionnaires are employed, Sackett & Larson discussed issues related to measures:

• Rely on self-reports or reports about another person or entity
• Interviews and questionnaires are conceptually similar; choice between is based on situation-specific, pragmatic considerations
• Can be used as a substitute for direct observation (demographics), as a tool to assess internal states such as attitudes (because these are not directly observable, or as a measure of perceptions.

Qualitative techniques offer the following strengths:

• Frees researchers from geographic concerns
• Saves time and money in comparison to observation

However, it also has the following weaknesses

• Ambiguities about direction of causality
• Problems with common method variance

Qualitative techniques are not popular. Pratt 2008 and Hannah & Lautsch 2011 addressed why scholars do not employ qualitative techniques:

• Uncertainty about how to conduct good qualitative research, even among qualitative researchers
• Lack of consensus in evaluating qualitative research, which makes it hard to publish: 
• Overly high standards for qualitative research (higher than for quantitative research) → harder to get accepted for publication. Requirements re theory development are much higher compared to quant papers. 
• Inappropriate standards, i.e., quantitative standards are inappropriately applied to qualitative research. 

(Adapted from group and course notes)

(Flashcards and other resources here)

Lab, Field, and Survey

In this post I will summarize lab, field, and surveys. The most important things to remember are the measures that are used and the inferences each approach allows us to make.

Lab Experiment

• Requires IVs and DVs, pre and post testing, experimental and control groups.
• Does not recreate reality, but studies variables in highly controlled, generic/created situation with stimulus/manipulation controlled by the experimenter.

Field Experiment:

• Similar to the lab experiment, it manipulates something to see effect on something else. Less control.

Survey

• Doesn’t include a manipulation and the goal is to leave setting and participants just as they were found. 
• Looks at natural variance instead of stimulating variance.

The measures used for each approach are:

• Field and Lab Experiments
• Dependent variable is real behavior.
• Uses ANOVA, t-tests, etc.
• Looks at mean differences.
• Surveys
• Dependent variable is measured through self-reports.  
• Correlational and often measured at same time.  
• Common method variance issues.

The key aspect to keep in mind when designing a study is understanding how each approach allows us to make different inferences. Ideally, there will be triangulation - using multiple approaches.

• Lab and Field Experiments
• Causal: treatment X caused outcome Y
• Able to do this because of issue of control (rule out extraneous influences on Y). 
• Field allows for time.
• Surveys:
• Correlational
• Non-causal
• Associational inferences
• X varies with Y

It is good to keep in mind the pros and cons for each approach.

Laboratory

• PROS
• Random assignment
• Control
• Precision
• Causal inferences
• Can plan
• CONS
• Ethical limitations
• Generalizability
• Short-lived (brevity)
• \Weaker manipulations than “real-life” counterparts
• May miss critical boundary conditions
• Artificial
• Demand characteristics
• Evaluation apprehension
• Experimenter expectancy

Field Experiments

• PROS
• Random assignment (possibly)
• Manipulation in real setting
• Allows control for causal relationships
• May encourage application of results
• Longer time frame
• Context is meaningful to participants
• Subjects less likely to be aware in experiment
• CONS
• Hard to do
• Hard to get true control groups
• Hard to control outside influences/confounds (therefore, stimulus less impactful)
• Difficult to control independent variable (therefore, hard to draw “cause-effect” relationships)
• Cost/time
• Ethical considerations in selecting control group
• Demand characteristics
• Evaluation apprehension
• Experimenter expectancy

Surveys

• PROS
• Natural setting (perhaps more ext. validity)
• More realistic (more believable)
• Describes the population
• CONS
• Hard to control
• May not be as generalizable as we think
• Difficult to replicate
• Not causal
• More bias (therefore, less reliable)
• Cross-sectional (measure everything at once—may lead to common method variance)

(Adapted from group and course notes)

(Flashcards and other resources here)

Research Design and Time

In the previous post, I detailed a few different types of research designs. In this post I will talk about an important element in research design: Time.

According to Mitchell and James, time is treated as a commodity that can be broken into meaningful segments or blocks. It flows evenly and consistently, it’s precise and quantifiable, and is ordinal.

Time is an important variable to consider in theory because theory concerns about causal relationship between x and y and time is a marker. In relationship between X and Y, there are time lag, duration, rate of change, reciprocal causality and non-linear cyclical and oscillating issues. Thus, time in theory is significant. By the same token, time is important when designing a research study because the measurement of when x and y occur and when they are measured would make significant differences in the result. If we ignore time in the theory-building phase, it would make bad theory. If we ignore time in the designing phase, it may lead to bad design and misleading results.

Five major ways in which theory informs method with respect to time:

1. First, we need to know the time lag between X and Y. How long after X occurs does Y occur? 
2. Second, X and Y have durations. Not all variables occur instantaneously. 
3. Third, X and Y may change over time. We need to know the rate of change. 
4. Fourth, in some cases we have dynamic relationships in which X and Y both change. The rate of change for both variables should be known, as well as how the X,Y relationship changes.
5. Finally, in some cases we have reciprocal causation: X causes Y and Y causes X. This situation requires an understanding of two sets of lags, durations, and possibly rates.

There are several ways to consider time into a research. First, we can think of the time of the measurement. If we know the time relation between variables of interest, we can consider time easily. However, we do not know it most of the time because theory rarely specifies time issues. By posing some questions, we can address this issue. When should I start measure X (start time issue)? How stable X is when measuring (stable issue)? When does Y appear after X (lag issue)?

Second, we can address time issue in our study considering the frequency of measurement. Y may change over time systematically, but not particularly complex. To assess rate of change we need multiple assessments through theoretical consideration of intra individual change, inter individual change and contextual change.

Lastly, we want to examine the issue of measure stability. Changes in the assessment of a variable over time can be due to random error, systematic error or systematic change. For this issue, test-retest assessment can provide information on the stability of X and Y and time-series (numerous observations on a small number of subjects) designs assess sources of error over time.

Some ways to think about time include:

• Construct Mitchell’s Moderation by Causal Cycle curve: illustrated periods of time in which X,Y interact
• Equilibration period: period it takes X to affect Y and for Y to reach state of constancy of state of equilibration
• Equilibrium period: when Y reaches state of equilibration, Y is said to enter an equilibrium type condition. The scores of Y may continue to change in this period, but the changes are small and constancy is resumed fairly quickly. This is when Y should be measured
• Entropic period: changes in Y are completely uncertain with respect to given set of measurements on Y. This is final state of causal cycle. This is when you need to stop measuring Y. 
• Draw diagram for relationships between X,Y in order to get an idea of issues such as lags, change and reciprocal causality
• Consider amount or rate of change. How much does X effect Y? Is this rate constant? If not, do you have to measure Y multiple times?
• Specify time in your research design before you begin
• The Frequency of Measurement--As Kelly and McGrath (1988) point out, we need at least three assessments to look at a curvilinear relationship; four for oscillation; and perhaps more for rhythms, spirals, and cycles.
• Stability (reliability)--Test-retest assessments (if variables are assessed during a steady-state or equilibrium period) can provide information on the stability (reliability) of X and Y, and time-series designs, as we mentioned earlier, also assess sources of error over time.

(Adapted from group and course notes)

(Flashcards and other resources here)

Research Designs

In the Causality post, I discussed the Solomon four group design. This design addressed some concerns on internal and external validity.

In this post, I will discuss other research designs, their advantages and disadvantages.

Nonequivalent Control Group Design

Advantages

• Controls for the main effects of history, maturation, testing, instrumentation, selection and maturation.

Disadvantages

• We do not test for the interaction between selection and maturation.  
• The problem of regression.

Counterbalanced Design

Advantages

• This design has internal validity on all the individual points.

Disadvantages

• There are systematic selection factors involved in the natural assemblage of the groups, as well as the effects associated with specific sequences of treatments.

Multiple Time-Series

Advantages

• An extended version of the Time Series design and the Nonequivalent Control Group design, this model controls for all internal sources of validity.  

Disadvantages

• The only disadvantage is that we are not controlling for potential interaction of testing and selection with X.

Time Series

Advantages

• Maturation is well controlled for (unless it is very abrupt), instrumentation is also well controlled for assuming we keep the same measures.  
• Regression, selection and mortality effects may also be ruled out.

Disadvantages

• Failure to control history is the biggest weakness.

(Adapted from group and course notes)

(Flashcards and other resources here)

Causality

A crucial part of research design is the ability to establish causality. In order to make a statement about causality, three conditions are necessary:

• the cause precedes the effect,
• the cause and effect covary, and
• there is no plausible alternative explanation for the covariation.

In order to make these inferences, Campbell and Stanley (1963) discuss that the Solomon four-group design is best suited to make statements of causality.

The true design 4 (pretest-posttest control design) can maintain internal validity. History, maturation, and testing are controlled. Regression is controlled in so far as mean differences are concerned. Selection is ruled out through randomization. It can tell whether mortality and instrumentation offer plausible rival hypotheses. The Solomon four-group design not only increases external validity by controlling the main effect of testing and interaction of testing and X, but it also replicates the effect of X. Among the pre-experimental designs, the one shot design is least suited for these inferences. This design totally lacks control and is of no scientific value.  The process of comparison is required.

Correlation does not indicate causality. Correlation simply implies that the mean difference between groups show that the two groups are related on some attributes.

• Correlational data helps to disconfirm existence of causal relationships.
• Weak correlationional design: Consider the case in which two units of analysis (two groups for comparison) are being observed outside the laboratory setting. One group gets treated with X, the other doesn’t. However, these two groups differ on many other attributes other than the presence/absence of X. Each of these other attributes could create difference in Os and each therefore provides a plausible rival hypothesis that X had an effect.
• Stronger correlational design that could point to more causal relationships: You still have groups for comparison, but X is a naturally occurring event that varies with one group more so than it does with the others. The key is that X is not artificially implanted, but is naturally part of one group and not a part of the other. Example: heavy smoking and lung cancer.

A correlational study must meet the first, three of the kinds of validity (internal validity, statistical conclusion validity, and construct validity) (Mitchell, 1985). Then, based on Popper (1959, 1963)'s falsification orientation, causal hypotheses with correlation zero will be disconfirmed, otherwise the hypotheses survive and will be examined further by cross-validation or other methodologies.

Furthermore, in a correlational study, Campbell & Stanley (1963) suggest that if the correlation is zero, the hypothesis on causality can be disconfirmed, otherwise the hypothesis survives and the researcher can examine it further through the use of other settings.

The causal interpretation of a simple or a partial correlation depends upon both the presence of a compatible plausible causal hypothesis and the absence of plausible rival hypotheses to explain the correlation upon other grounds.  Any third variable that could affect the signaling frequency of both pairs of drivers in a similar fashion becomes a plausible rival hypothesis.

(Adapted from group and course notes)

(Flashcards and other resources here)

Internal and External Validity

In the previous post, I discussed the four types of validity.

Here, I will discuss validity in terms of research design. As we create and design the way we are going to answer our research question, there are threats to internal validity and external validity that must be considered.

Internal validity seeks to isolate the cause and effect relationship. In other words, it is focused on making sure that the manipulation (independent variable) has some effect on the dependent variable. This sheds light on the true relationship between the variables. The primary threats to internal validity are:

• History 
• Maturation
• Testing
• Instrumentation
• Statistical regression
• Selection
• Experimental mortality
• Selection-maturation interaction

On the other hand, external validity is focused on the ability to generalize beyond the original study across times, settings, measurements, and persons. Threats include:

• Reactive or interaction effect of testing.
• Interaction effects of selection biases and the experimental variable.
• Reactive effects of experimental arrangements.
• Multiple-treatment interference.

I will discuss different designs in a future post (laboratory, survey, field). But it is relevant to note how some of these research designs might have internal/external validity problems. 

In laboratory studies, for example, it is possible to isolate the variables of interest to determine if there is a relationship between the independent and dependent variables. This means it is high on internal validity. However, because the laboratory is a highly controlled environment, it is difficult to generalize to conditions outside the lab. So, in a lab study it is important to create a balance between internal and external validity. The greater internal validity, the more controlled the lab experiment is, and so the more difficult it is to generalize.

Field studies are the opposite. When introducing a manipulation, it is difficult to show that it was the manipulation and not something else that had an effect on the dependent variable. The external validity in a field study has fewer challenges to overcome than a lab study. It is possible, with careful research design, to generalize to settings beyond the to the original study. 

Depending on the goal, internal or external validity may be more important. If the main goal is only to determine the true relationship between the independent and dependent variables, external validity is not necessarily important. If the main goal is to apply the findings to the world beyond the lab then external validity is important. Ultimately, research goals are interested in both, and thus a balance between internal and external validity must be found. It is possible to have a research design that uses a laboratory study to determine the relationship of two variables, and follow up with a field study to test it. 

(Adapted from group and course notes)

(Flashcards and other resources here)

Hypothesis Testing

There are a few standard steps in hypothesis testing:

1. State the hypothesis in general terms
• Exploratory approaches.
• Intentional search approaches.
• Extending-coupling approaches.
2. Operationalize the hypothesis:
• What will be measures/observed? (Dependent variables) 
• What will be manipulated?  (Independent variables) 
• How are these tied to the hypothesis? 
3. What methods will be employed?
• How will you test your hypothesis? 
• How will the relation between your independent and dependent variables be examined?
4. What results do you anticipate?
• How will the results provide evidence for your hypothesis

It is also important to note that a hypothesis is never really proved or disproved. You provide evidence either in favor of the hypothesis or evidence that casts doubt on the hypothesis. Thus, empirical evidence never proves the hypothesis but rather “establishes” the hypothesis for acceptance. As such, the hypothesis is never proven as a logical consequence of empirical evidence. A core tenet of the scientific process and theory building is falsifiability.

Furthermore, statistical tests aid in making inference from the observed sample to unobserved population. That is, they let you evaluate the population indirectly.  One can only go from the denial of S (sample) to the denial of P (population); and not from the assertion of S to the assertion of P. Statistical tests also allow making statements of reliability of the measure.

(Adapted from group and course notes)

(Flashcards and other resources here)

Validity

The most common definition of validity is typified by the question: are we measuring what we think we are measuring. There are four types of validity: face, content, criterion, and construct.

First is construct validity. It focuses on measuring concepts that are not directly observable and that we try to infer. For instance intelligence, anxiety or attitude. Rather than focusing on the tests themselves, the focus is on the meaning of the tests and its factors. Construct validity, then, is preoccupied with not just validating the test, but the theory, theoretical constructs, and scientific empirical inquiry (1). In addition, both convergence and discriminability are required.

• Convergence means that the evidence gathered all indicates the same thing (1). 
• Discriminability means that it is possible to point out which measurements are not related to the construct. 
• This is the most important form of validity because it connects measures with theories. 

Criterion validity compares tests scores with one or more external variables or criteria that is known to measure the attribute of interest. There are two types of criterion validity: predictive and concurrent.

• Predictive validity focuses on future performance. 
• Concurrent is at the same time. 
• The information criterion validity provides is useful for new tests. However, its biggest challenge is the criterion used in the comparison.
• The information that these next two measures provide is mostly obvious. They require judgment and appearance. 

Content validity focuses on the question: is the content of this measure representative of the universe of content of the property being measured.  A test with high content validity would, in theory, be a representative sample of the universe of content. This type of validity relies on judgment.  Some universes of content are easier to judge than others. For the obvious ones, content validity is assumed. For instance, a test of arithmetic to determine weather students can do additions.

Last is face validity, which focuses on looking to see if the measure obviously and clearly measures the intended measure. Measures are difficult to justify using face validity. However, it is the most basic and without it the rest of the validity methods would not work. For example, if we intent to measure a person’s ability to run, but we measure the number of pillows they own, face validity would indicate that this may not be an obviously valid measure.

(Adapted from group and course notes)

(Flashcards and other resources here)

Reliability

Reliability is the consistency or stability of a measuring instrument.  Kerlinger defined it as "the proportion of the ‘true’ variance to the total obtained variance of the data yielded by a measuring instrument.”  Operationally, this translates to the proportion of error variance to the total variance yielded by a measuring instrument subtracted from 1.00 (the index of 1.00 indicating perfect reliability).

Schwab defined reliability as referring to the ratio of “true” to total variance in a set of parallel measurements obtained on an individual. While Mitchell discussed reliability as the measure of correlation between maximally similar items and assesses the random or choice error.

It is also useful to think of reliability as:
VO = VT + Ve   (Variance Observed = True Variance + error)

The four methods for assessing reliability are:

1. Kuder-Richarson-21 (KR-21) is considered the most conservative estimate for reliability of an instrument with binary scoring or where the response scale is dichotomous (e.g., true or false).  The Kuder-Richardson formulas are special cases of Cronbach’s coefficient alpha.  Cronbach’s alpha is the most frequently used metric because it is a) conservative, b) easier and less time-consuming than test-retest or parallel forms, and c) does not involve cutting out data as in the split-half test. 
2. Split-half: This involves splitting the items into two halves, with the goal of obtaining two equal or equivalent halves.  Each person will have two half-scores; the responses of both halves are compared (possibly using a Pearson correlation) to ensure internal consistency.  This is fairly conservative because it will underestimate the true reliability, since it is only the correlation of two halves of the test.  Splitting the sample may be difficult, and the assessment of reliability now depends upon fewer items.
3. Parallel forms: This involves creating two measures that are equivalent, but not identical.  This can be very time-consuming for the researcher.  Each person would be subjected to measurements by both instruments.  The score of these two measures is then compared for consistency.  This measure is less conservative because there is the chance for fatigue or boredom when respondents must complete two measurement indices.  (Test A and Test B).
4. Test-retest: Used to measure the stability of a measure over time.  This involves administering the same measurement instrument to the same group of people on two different occasions.   This measure is less conservative when there is high attrition if the organism being measured goes through dramatic developmental changes from time 1 to time 2.  Not a good way of computing the reliability coefficient if attrition is high or the organisms being measured will be going through a dramatic developmental change

(Adapted from group and course notes)

(Flashcards and other resources here)

Measurement

Kerlinger and others have discussed measurement bias and measurement development.

Measurement biases involve systematic error that can occur in collecting relevant data. Common measurement biases include:

• Instrument bias. Instrument bias occurs when calibration errors lead to inaccurate measurements being recorded, e.g., an unbalanced weight scale. (questionnaires, company records)
• Insensitive measure bias. Insensitive measure bias occurs when the measurement tool(s) used are not sensitive enough to detect what might be important differences in the variable of interest. (questionnaires)
• Expectation bias. Expectation bias occurs in the absence of masking or blinding, when observers may err in measuring data toward the expected outcome. This bias usually favors the treatment group. (behavioral observations)
• Recall or memory bias. Recall or memory bias can be a problem if outcomes being measured require that subjects recall past events. Often a person recalls positive events more than negative ones. Alternatively, certain subjects may be questioned more vigorously than others, thereby improving their recollections. (questionnaires)
• Attention bias. Attention bias occurs because people who are part of a study are usually aware of their involvement, and as a result of the attention received may give more favorable responses or perform better than people who are unaware of the study’s intent. (behavioral observations, questionnaires)
• Verification or work-up bias. Verification or work-up bias is associated mainly with test validation studies. In these cases, if the sample used to assess a measurement tool (e.g., diagnostic test) is restricted only to who have the condition of factor being measured, the sensitivity of the measure can be overestimated.

To develop a new measure in a field, the following process is suggested:

1. Specify domain of construct
• Extensive literature review to define the exact construct I want to measure or evaluate.  
• Measuring this construct would involve developing a scale to generate a degree of presence or absence of that construct or items making up that construct. 
• In the instance of survey questions the measure may be only positive numbers (unipolar) with different degrees of the same attribute in mind, or positive and negative numbers on the scale, which conveys more bipolar dimensions.
2. Empirically determine the extent to which items measure that domain
• The levels of the scales need to be appropriate of so that differences between measures be interpretable as quantitative differences in the property measured. 
• Questions would be closed response to reduce variability in the responses and reduce ambiguity. From the initial sampling of respondents, the classification of respondents into categories and employing a factor analysis to verify construct validity. 
• Each item on the questionnaire needs to address a single issue and measure. 
• Each of the questions would then be evaluated for correlation to make sure that multiple questions were not measuring the same construct to simplify the measure.  
• Construct Validity would then be verified with other measures of the same construct to be similar (Convergent Validity) and Correlations between measures of the same construct (discriminant validity).
3. Examine the extent to which the measure produces results that are predictable from the theoretical hypotheses.
• Half of this initial sample would be utilized for exploratory factor analysis and after the measures and groupings were defined, the other half of the sample would be compared and measured with Confirmatory Factor Analysis (Cronbach’s Alpha) to verify the constructs we were measuring fit the data appropriately. 
• This replication of the data will help assess the consistency, reliability, and validity of the measure.

(Adapted from group and course notes)

(Flashcards and other resources here)

Theory Evaluation

The cycle of theory building approach as discussed in the previous post allow us to see the big picture of theory building process. This way of theory building allows us to integrate the dichotomies in business academia between field-based research and large-sample data analysis; theoretical vs. applied research; between deductive and inductive theory building. Hence, it leads us to develop a good research questions.

There are a few ways to evaluate theory.

• Falsifiability: must be possible to refute empirically.
• Utility: refers to the usefulness of theoretical systems.
• Explain: means establishing meaning of constructs, variables, and linkages
• Prediction: means testing meaning by comparing it to empirical evidence and theory provides mechanism for predicting beyond chance

Corley and Gioia (2011) discuss theory contribution in terms of different quadrants:

• Incremental Insight – Is the contribution “significant”? Do we get closer to the truth?
• Revelatory Insight* – Does it surprise us?
• Scientific Utility* – Does it improve the conceptual rigor or assist in forming testable predictions?
• Practical Utility – Can the theory be applied to real-world problems? 

Note (*): Papers  further NE in the diagram to the left are considered as providing greater contribution than papers to their SW.

(Adapted from group and course notes)

(Flashcards and other resources here)

Theory Building

There are different ways in which one can build a theory. However, a few things must be kept in mind:

1. Completeness
• Are all relevant factors included?
2. Parsimony
• Should some factors be deleted because they provide little value?
3. Relationships
• Why are you selecting the various factors and why do they relate? 
• Gets at assumptions of author and why people should care (underlying economic, social dynamics, etc.).

Christensen (2001) suggested a synthesized model of theory building throughout a range of fields. The model consists of four stages that repeat. This cycle of theory building includes both deductive and inductive modes.

1. Observing phenomena, and carefully describing and recording those observations
2. Classifying the phenomena into categories of similar things: The aim is to simplify and organize the world in ways that highlight the most meaningful differences amongst phenomena
3. Building theories that explain the behavior of the phenomena (describing what causes what, why, and under what circumstances)
4. Using theory to predict what they will observe when they go out and observe more phenomena under various conditions for more accurate description, revising a classification scheme and/or articulating a new statement of what causes what under what circumstances
5. Once researchers have defined a set of categories that are collectively exhaustive and mutually exclusive, then the theory built becomes a paradigm. 

Christensen also talks about the discovery of anomalous phenomena is the pivotal element in the process of building improved theory because the anomaly observation creates reliable mechanism in classification. Furthermore, the anomalies may lead to a toppling of a reigning paradigm. 

(Adapted from group and course notes)

(Flashcards and other resources here)

Theory

According to Babbie, theory is:

• A systematic sets of interrelated statements intended to explain some aspect of social life (they attempt to explain what we see). 
• It is tied to observable events and makes predictions about empirical findings. 
• And it is a series of structures linking constructs to action, eventually linked to behavior. 

One way to think about theory is to think about modeling.

Whetten (1989) suggests the Modeling-as-theorizing methodology for theory development that uses graphical modeling logic conventions. He suggests that “A theory is a collection of assertions, both verbal and symbolic that identifies WHAT variables are important for what reasons, specifies HOW they are interrelated and WHY, and identifies the CONDITIONS under which they should be related or not related.”

Here is some more explanations about “WHAT, HOW, WHY, and the CONDITIONS.”

1. ‘Whats’-as-constructs: What are the elements of my conceptualization?
• Brainstorming constructs using post-it notes, PIN. 
• Assessing complementarity or compatibility of the constructs by considering the scope of the concepts and the coherence of the constructs.
2. ‘Hows’-as-relationships: the specification of relationships between constructs is the key difference between a theory and a list:
• Placing the core construct in the center.
• Aligning constructs horizontally in terms of sequence.
• Creating vertical dimension on the core sequence of the model for moderating constructs. 
• Using arrows, directions, line thickness to indicate explicit theoretically relevant relationships in conceptualization.
3. ‘Whys’-as-conceptual assumptions: The conceptual assumptions underlying a theory can be thought of as 'second order explanations' the implicit whys underlying an explicit answer to a specific why question.
• Considering various typologies in the field and how various conceptual assumptions can pose a threat to coherence.
4. ‘When/where/who’-as-contextual assumptions: specify the contextual boundaries, or conditions, that circumscribe a set of theoretical propositions.

(Adapted from group and course notes)

(Flashcards and other resources here)

Paradigms

The word Paradigm is meant to be the fundamental frames of reference that underlie social theories and inquiry and provide ways of observing so that we are able to understand concepts and constructs. They are usually difficult to recognize since they are taken for granted and seem to be the way things “are”. Eventually, when the shortcoming of a paradigm becomes obvious, a new one usually emerges to supplant the old.

In Social Sciences, paradigms are rarely discarded, but are built upon. They lose or gain popularity, but each gives insights that others lack and ignores aspects of social life that others reveal. They are really either more or less useful.  Paradigms can help research by providing a solid framework to build upon. In a well-developed field, there usually is methodological standards and are constantly maintained and these are heavily tested. Paradigms can hinder research because there might not be consensus on the underlying methods, and because of the various numbers of ways research is being conducted, it is hard to test previous theories or other preceding works.

In developing a new paradigm, one of the most important components of a new paradigm or advancement of knowledge is consensus. According to Pfeffer (1993), there needs to be a minimum level of consensus about research questions and methods or fields cannot expect to produce knowledge in a cumulative, developmental process. One cannot build a new paradigm if there is not agreement on fundamentals and time will be wasted debating principles. Pfeffer says that disagreement needs to be distinguished over:

• substantial research questions that are considered important 
• ways that relevant variables should be measured and modeled 
• methods used to collect and analyze relevant data 
• the theoretical models of behavior used to guide the measurement process and the 
• the rules for determining the approach to each of these four domains that are more or less fruitful 

There needs to be this agreement in a field of study before a new paradigm can be developed. As time goes on and there is consensus on over fundamental goals there will be less contesting new research processes and methods, since these will be more standardized.

Fields in social science that very diverse and pull from multiple disciplines can be useful as long as the diversity can be resolved to a certain extent. The field needs to encourage resolution and agreement over methods and ideas. One way of doing this would be to develop consensus among academics or people in positions of power at annual meetings to discuss various views. With the unification or standardization by elite members of the field consensus could start to be established and new paradigms could start to take shape.

(Adapted from group and course notes)

(Flashcards and other resources here)

Philosophy of Science: Alternative Approaches

Knowledge has traditionally been produced from the positivistic research tradition (Sandberg 2005). This research paradigm asserts that the rules governing social life can be studied through rational and objective scientific approaches (Babbie 2010). Rational scientific theory is grounded on the assumptions discussed in the previous post. In addition, it assumes that research objects and subjects can be separated, that there is such thing as an objective reality, and that language can mirror that reality (Sandberg 2005).

Alternative approaches to science have not gained wide acceptance. While Sandberg argues that in the past three decades focus on interpretive approach has increased in use, these papers still represent a vast minority of published research. The primary challenge is claiming that the knowledge they produce is valid and generalizable given that they reject the idea of objective reality. Sandberg further discusses several strategies to achieve validity and reliability. However, the established research community continues to expect rigorous measures of reliability, validity, and generalizability based on positivist traditions. Alternative approaches include: interpretive, postmodern, and hermeneutics.

Interpretive Approaches
In recent decades, however, interpretive approaches have emerged from the ideas of philosophical phenomenology (Sandberg 2005). The interpretive research strategy rejects positivist assumptions:

• Subject and object are seen as constituting a single, inseparable relation. 
• Advocates of this approach argue that there is no objective reality:
• Our lived experiences are always related to our descriptions of reality. 
• Language is not a mirror of reality:
• Since our descriptions of reality are colored by our specific historical, cultural, and ideological view of reality, language cannot objectively mirror reality.

Postmodern Approaches
In addition, to interpretive approaches, the postmodern approach has also challenged rational science. This approach revolts against taken-for-granted assumptions, challenges conventional wisdom and transgresses established boundaries with bold conjectures and innovative methods.  It claims that:

• The traditional scientific research approach in the organizational sciences has neglected intuition or subjective experience in research reports. 
• Researchers in the field assume too readily that their data represent the truth about an objectively measured world.

The postmodern approach advocates for an inclusive scientific approach that breaks down disciplinary boundaries, challenges conventional wisdom, and gives voice to viewpoints that have not been given much attention (i.e. feminist ideas).

Hermeneutics
And finally hermeneutics is a philosophy of understanding/interpreting texts. However, some of the assumptions and techniques can be applied to organizational science that goes beyond the rational science approach. Researchers should aim to understand more fully the organizational practices, institutions, economics and social structures, and cultural factors that influence the phenomena being studied (understand the phenomena in these contexts). Hermeneutics is an iterative approach, with continuous critique being the goal. The hermeneutic circle emphasizes the significance of context for purposes of interpretation.

(Adapted from group and course notes)

(Flashcards and other resources here)

Philosophy of Science

The scientific approach is best described as a process. Science is best described not by its output, but by its method (e.g., process).  What unites all scientific inquiry is its process of inquiry.  Science is a systematic process of developing theoretical structure, testing internal consistency, and empirically testing hypotheses.  

The process starts with doubt and a struggle to formulate a question.  The researcher studies the literature, solidifies the question, and develops a hypothesis.  Empirical implications are deduced and tested.  The original problem is then clarified and the hypothesis evaluated.  This new information feeds back into the beginning of the loop, the hypothesis is reworked or expanded or changed entirely, and everything happens again.  Perhaps most importantly, the process of science is self-correcting. 

Science is a process that develops in a circular pattern between theory and data.  Knowledge must be logically valid and empirically verified.   Scientists engage in the research process in order to evaluate a theory against these criteria.  There are seven principle stages (Nachmias and Nachmias 1987): 

This is a self-correcting cycle that continually refines and advances knowledge.

As a caveat: most science doesn’t directly follow this neat process.  In addition, a single research piece will rarely complete the whole cycle.  Finally, the above process is implemented to systematically and empirically test refutable theories and hypothesis, while controlling for confounding variables.

On a meta scale, an argument can be made that paradigm development is an overarching part of this process, with normal science and revolutionary science cycling back and forth (Kuhn).

It is also important to mention alternative ways to acquire knowledge. These include:

• Authoritarian Mode: those who are socially or politically defined as knowledge producers define truths. Examples include shamans, kings, archbishops, communists, etc.  This truth machine can only be altered by overwhelming evidence of the truths being false, or by political/social change.
• Mystical Mode: Knowledge is obtained supernaturally by mystical leaders (prophets, gods, etc) or by revelation. 
• Rationalistic Mode: the total of reality can be deduced from strict adherence to the forms and rules of logic.  This school of philosophy was founded by Aristotle and championed by Kant.  Pure math is an example of this.  Its proofs are true and exist independent from reality. 
• Common Sense: this can include both “a priori knowledge” (that something is self evident) as well as knowledge from “tenacity” (holding firmly to a truth that you know to be true because you are holding firmly to it).

Kerlinger (1986) defines scientific research as a “systematic, controlled, empirical, and critical investigation of natural phenomena guided by theory and hypotheses about the presumed relations among such phenomena.”    There are 6 assumptions that facilitate scientific research, which also sometimes constrain its findings. These are sometimes called epistemology – or how we know (Dewey).

1. Nature is orderly – Things are not completely random.  There is a pattern that can be understood in events and in people. 
2. We can know nature – The main application of this is that we can know ourselves, since we are part of nature.  There is nothing special about people.
3. Knowledge is superior to ignorance – Knowledge has both utilitarian and intrinsic benefits.  This notion that people should always pursue truth is at odds with “closed systems” of absolute truth.  
4. All natural phenomena have natural causes – science examines normal cause and effect relationships, not supernatural phenomena or miracles.  Science is agnostic to those.
5. Nothing is self-evident – tradition, common sense, and beliefs are not sufficient.  There must be objective evidence. 
6. Knowledge is derived from the acquisition of experience – instead of being based on pure reason, knowledge is gained from empirical tests of reality. 

The scientific process hinges on these assumptions.  Interpretive approaches criticize this process because they argue that many phenomena are not orderly, replicable, or consistent.  Critics allege that without consistency the scientific process breaks down to some extent.  However, if these assumptions hold, then the scientific process accumulates a body of reliable knowledge that helps us to explain, predict, and understand empirical phenomena.  If someone believes that these assumptions do not hold, or if they do not agree with the nature of the scientific process, then they are free to develop an alternative approach.  No reasonable alternative approach has been proposed to date.

(Adapted from group and course notes)

(Flashcards and other resources here)

Research Design

The reason Research Design is so important is that it ensures that the evidence collected can be used to answer the research question as "unambiguously as possible." Science in general and social science in particular have been subjected to increasing criticism with respects to practicality, significance, application and meaning.  A strong research design can help address some of this criticism.

To that end, there are three themes that were covered in class.

1. Background
• This covers the scientific approach and different perspectives on the philosophy of science.
2. Building Blocks
•  This includes theory construction, measurement, reliability, validity, variables, and hypothesis generation.
3. Strategies
• This covers different strategies for doing research including laboratory studies, field experiments, surveys, and qualitative research.

These topics in combination will help inform a researcher on the major decisions when designing a plan of study.