Research and Statistical Support

MODULE 9

Factor Analysis with Maximum Likelihood Extraction in SPSS.

Before we begin with the analysis; let's take a moment to address and hopefully clarify one of the most confusing and misarticulated issues in statistical teaching and practice literature.

First, Principal Components Analysis (PCA) is a variable reduction technique which maximizes the amount of variance accounted for in the observed variables by a smaller group of variables called COMPONENTS. As an example, consider the following situation. Let's say, we have 500 questions on a survey we designed to measure persistence. We want to reduce the number of questions so that it does not take someone 3 hours to complete the survey. It would be appropriate to use PCA to reduce the number of questions by identifying and removing redundant questions. For instance, if question 122 and question 356 are virtually identical (i.e. they ask the exact same thing but in different ways), then one of them is not necessary. The PCA process allows us to reduce the number of questions or variables down to their PRINCIPAL COMPONENTS.

PCA is commonly, but very confusingly, called exploratory factor analysis (EFA). The use of the word factor in EFA is inappropriate and confusing because we are really interested in COMPONENTS, not factors. This issue is made more confusing by some software packages (e.g. PASW/SPSS & SAS) which list or use PCA under the heading factor analysis.

Second, Factor Analysis (FA) is typically used to confirm the latent factor structure for a group of measured variables. Latent factors are unobserved variables which typically can not be directly measured; but, they are assumed to cause the scores we observe on the measured or indicator variables. FA is a model based technique. It is concerned with modeling the relationships between measured variables, latent factors, and error.

As stated in O'Rourke, Hatcher, and Stepanski (2005): "Both (PCA & FA) are methods that can be used to identify groups of observed variables that tend to hang together empirically. Both procedures can also be performed with the SAS FACTOR procedure and they generally tend to provide similar results. Nonetheless, there are some important conceptual differences between principal component analysis and factor analysis that should be understood at the outset. Perhaps the most important deals with the assumption of an underlying causal structure. Factor analysis assumes that the covariation in the observed variables is due to the presence of one or more latent variables (factors) that exert causal influence on these observed variables" (p. 436).

Final thoughts. Both PCA and FA can be used as exploratory analysis. But; PCA is predominantly used in an exploratory fashion and almost never used in a confirmatory fashion. FA can be used in an exploratory fashion, but most of the time it is used in a confirmatory fashion because it is concerned with modeling factor structure. The choice of which is used should be driven by the goals of the analyst. If you are interested in reducing the observed variables down to their principal components while maximizing the variance accounted for in the variables by the components, then you should be using PCA. If you are concerned with modeling the latent factors (and their relationships) which cause the scores on your observed variables, then you should be using FA.

O'Rourke, N., Hatcher, L., & Stepanski, E.J. (2005). A step-by-step approach to using SAS for univariate and multivariate statistics, Second Edition. Cary, NC: SAS Institute Inc.

Factor Analysis

The following covers a few of the SPSS procedures for conducting factor analysis with maximum likelihood extraction. For the duration of this tutorial we will be using the ExampleData4.sav file.

FA 1. Begin by clicking on Analyze, Dimension Reduction, Factor...

Next, highlight all the variables of interest (y1 - y15) and use the top arrow button to move them to the Variables: box. Then click the Descriptives button and select the following. Then click the Continue button.

Next, click on the Extraction button. In the Method drop-down menu, choose Maximum likelihood. Then select Unrotated factor solution and Scree plot. Notice the extraction is based on factors with eigenvalues greater than 1 (by default). There are a number of perspectives on determining the number of factors to extract and what criteria to use for extraction. Originally, eigenvalues greater than 1 was generally accepted. However, more recently Zwick and Velicer (1986) have suggested, Horn’s (1965) parallel analysis tends to be more precise in determining the number of reliable components or factors. Unfortunately, Parallel Analysis is not available in SPSS. Therefore, a review of the parallel analysis engine (Patil, Singh, Mishra, & Donavan, 2007) is strongly recommended. Next, click the Continue button.

Next, click on Scores and select Save as Variables. This will create new variables (1 per extracted factor) which will allow us to evaluate which type of rotation strategy is appropriate in subsequent factor analysis.

Next, click the OK button.

The output should be similar to that displayed below.

The Descriptive Statistics table simply provides mean, standard deviation, and number of observations for each variable included in the analysis.

The Correlation Matrix table provides correlation coefficients and p-values for each pair of variables included in the analysis. A close inspection of these correlations can offer insights into the factor structure.

The next table is used as to test assumptions; essentially, the Kaiser-Meyer-Olking (KMO) statistic should be greater than 0.600 and the Bartlett's test should be significant (e.g. p < .05). KMO is used for assessing sampling adequacy and evaluates the correlations and partial correlations to determine if the data are likely to coalesce on factors (i.e. some items highly correlated, some not). The Bartlett's test evaluates whether or not our correlation matrix is an identity matrix (1 on the diagonal & 0 on the off-diagonal). Here, it indicates that our correlation matrix (of items) is not an identity matrix--we can verify this by looking at the correlation matrix. The off-diagonal values of our correlation matrix are NOT zeros, therefore the matrix is NOT an identity matrix.

A communality (h²) is the sum of the squared factor loadings and represents the amount of variance in that variable accounted for by all the factors. For example, all five extracted factors account for 33.4% of the variance in variable y1 (h² = .334).

The next table displays the amount of variance accounted for in the variables' or items' variance-covariance matrix by each of the factors and cumulatively by all the factors. Here we see that all 5 extracted factors (those with an eigenvalue greater than 1) account for 32.209% of the variance in the items' variance-covariance matrix.

The scree plot graphically displays the information in the previous table; the factors' eigenvalues.

The next table displays each variable's loading on each factor. We notice from the output, we have two items (y14 & y15) which do not load on the first factor (always the strongest without rotation) but create their own retained factor (also with eigenvalue greater than 1). We know a factor should have, as a minimum, 3 items/variables; but let's reserve deletion of items until we can discover whether or not our factors are related.

Finally, we have the goodness-of-fit table; which gives an indication of how well our 5 factors reproduce the variables' or items' variance-covariance matrix. Here, the test shows that the reproduced matrix is NOT significantly different from the observed matrix -- which is what we would hope to find; indicating good fit.

Next, we will use the saved factor scores to determine if our factors are correlated, which would suggest an oblique rotation strategy would be appropriate. If the factors are not correlated, then an orthogonal rotation strategy is appropriate. Keep in mind, if literature and theory suggest the factors are related then you may use an oblique rotation, regardless of what you find in your data.

In the Data Window, click on Analyze, Correlated, Bivariate...

Next, highlight and use the arrow button to move all the REGR factor scores to the Variables: box. Then click the OK button.

The output should display the correlation matrix for the 5 factor scores as below.

Here, we see that none of the factor scores are related, which suggests the factors themselves are not related -- which indicates we should use an orthogonal rotation in subsequent factor analysis.

FA 2. VARIMAX rotation imposed. Next, we re-run the FA specifying 5 factors to be retained. We will also specify the VARIMAX rotation strategy, which is a form of orthogonal rotation.

Begin by clicking on Analyze, Dimension Reduction, Factor...

Next, you should see that the previous run is still specified; variables y1 through y15. Next click on Descriptives...and select the following; we no longer need the univariate descriptives, the correlation matrix, or the KMO and Bartlett's tests. Then click the Continue button. Next, click on the Extraction... button. We no longer need the scree plot; but we do need to change the number of  factors to extract. We know from the first run, there were 5 factors with eigenvalues greater than one, so we select 5 factors to extract. Then click the Continue button.

Next, click on Rotation... and select Varimax. Then click the Continue button. Then click on the Scores... button and remove the selection for Save as Variables. Then click the Continue button. Then click the OK button.

The first 3 tables in the output should be identical to what is displayed above from FA 1; accept, now we have two new tables at the bottom of the output.

The rotated factor matrix table shows which items/variables load on which factors after rotation. We see that the rotation cleaned up the interpretation by eliminating the global first factor. This provides a clear depiction of our factor structure (marked with red ellipses).

Again, the Factor Transformation Matrix simply displays the factor correlation matrix prior to and after rotation.

FA 3. Finally, we can eliminate the two items (y14 & y15) which (a) by themselves create a factor (factors should have more than 2 items or variables) and (b) do not load on the un-rotated or initial factor 1. Again, click on Analyze, Dimension Reduction, then Factor...

Again, you'll notice the previous run is still specified, however we need to remove the y14 and y15 variables. Next, click on Extraction... and change the number of factors to extract from 5 to 4. Then click the Continue button and then click the OK button.

The output should be similar to what is displayed below.

The communalities are lower than we would prefer (generally would like to see at least 0.450).

The four extracted (and rotated) factors account for 35.483 % of the variance in the items' variance-covariance matrix.

The Factor Matrix table displays factor loadings for each item (prior to rotation).

The goodness-of-fit table; which gives an indication of how well our 5 factors reproduce the variables' or items' variance-covariance matrix. Here, the test shows that the reproduced matrix is NOT significantly different from the observed matrix -- which is what we would hope to find; indicating good fit.

The Rotated Factor Matrix displays the loadings for each item on each rotated factor, again clearly showing the factor structure.

And again, the Factor Transformation Matrix displays the correlations among the factors prior to and after rotation.

As a general conclusion, we can say we have four factors accounting for 35.483 % of the variance in our 13 items. In the Rotated Factor Matrix table we see clear factor structure displayed; meaning, each item loads predominantly on one factor. For instance, the first four items load virtually exclusively on Factor 1. Furthermore, if we look at the communalities we see that all the items displayed a communality of 0.30 or greater, with one exception. The exception is y4, which is a little lower than we would like and given that Factor 1 has three other items which load substantially on it, we may choose to remove item y4 from further analysis or measurement in the future.

o = T + e

obs = True + err

It is important to note that traditional factor analysis assumes the Classical Test Theory of measurement, which states that observed scores (obs) are a result of true scores (True) and error (err). Therefore, in most factor model diagrams, the arrows point to the observed variables (note, in the diagram below the coefficients are not present) reflecting the assumption that the true value of the factor and the error combines to result in the observed score(s).

REFERENCES / RESOURCES

Horn, J. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179 – 185.

O'Rourke, N., Hatcher, L., & Stepanski, E.J. (2005). A step-by-step approach to using SAS for univariate and multivariate statistics, Second Edition. Cary, NC: SAS Institute Inc.

Patil, V. H., Singh, S. N., Mishra, S., & Donavan, D. T. (2007). Parallel Analysis Engine to Aid Determining Number of Factors to Retain [Computer software]. Retrieved 08/23/2009 from http://ires.ku.edu/~smishra/parallelengine.htm

Zwick, W. R., & Velicer, W. F. (1986). Factors influencing five rules for determing the number of components to retain. Psychological Bulletin, 99, 432 – 442.