Exploratory factor analysis on situational judgment data after controling for scenario effect
Article Sidebar
Published:
Nov 30, 2013
Main Article Content
Abstract
We demonstrated how to perform exploratory factor analysis on situational judgment data of creative personality. Situational judgment test(SJT) has a long history of 80 years and has settled down in industrial/organizational settings for the last 20 years. However, we have not overcome the problem of estimating construct because item scores in SJT contain method effect generated by the scenarios as well as response scores to the substantive questions. We applied Asparouhov and Muthen’s(2009) logic of explorary structural equation modeling(ESEM) to open an approach to estimating common factor structures after controling for the method effect generated by scenarios. Using ESEM, it is possible to specify measurement error correlations in the frame of structural equation modeling and to use exploratory approach to factor analysis on the remaining part of data. As a result we could estimate a four-factor structure on data of 40 items with eight scenarios measuring creative personality of college students.
Metrics
Metrics Loading ...
Article Details
How to Cite
Namgung, J., Lee, S., & Kim, H. (2013). Exploratory factor analysis on situational judgment data after controling for scenario effect. Korean Journal of Industrial and Organizational Psychology, 26(4), 599–624. https://doi.org/10.24230/kjiop.v26i4.599-624
Section
Empirical Articles
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
강민우, 이순묵, 윤창영. (2005). 지시문과 채점방식에 따른 상황판단검사의 타당도 비교. 한국심리학회지: 산업 및 조직, 18(3), 547- 565.
김효선 (2013). 상황판단형 창의성격검사 제작 및 타당화. 성균관대학교 석사학위 청구 논문.
이상철, 이순묵, 조영일. (2003). 지필형 상황판단검사에 대한 비평적 고찰. 한국심리학회지: 산업 및 조직, 16(3), 129-154.
이순묵 (2004). 상황역량 측정에서 상황에 대한 두 관점: 측정오차인가, 해석되어야 할 환경요인인가? 한국심리학회지: 산업 및 조직, 17(2), 243-264.
이순묵 (1995). 요인분석 I. 서울: 학지사.
이순묵, 윤창영, 김영록. (2005). MTMM 자료분석에 대한 비평적 고찰. 한국심리학회지: 산업 및 조직, 18(3), 455-480.
AERA, APA, & NCME (1999). Standards for Educational and Psychological Testing. Washington, DC: Authors.
Asparouhov, T. & Muthen, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397-438.
Becker, T. E. & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625-641.
Bovaird, J. A. & Koziol, N. A. (2012). Measurement models for ordered-categorical indicators. In Hoyle, R. H. (Ed.) Handbook of Structural Equation Modeling. NY: Guilford Press.
Browne, M. W. (1979). The maximum-likelihood solution in inter-battery factor analysis. British Journal of Mathematical and Statistical Psychology, 32, 75-86.
Browne, M. W. (1980). Factor analysis of multiple batteries by maximum likelihood. British Journal of Mathematical and Statistical Psychology, 33, 184-199.
Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36, 111-150.
Browne, M. W. & Tateneni, K. (1997). Noniterative Estimation for the Multiple Battery Factor Analysis Model. Behaviometrika, 24(1), 3-18.
Campbell, D. T. & Fiske, D. W. (1959). Convergent and Discriminant Validation by Multitrait-Multimethod Matrix. Psychological Bulletin, 56, 81-105.
Chan, D. & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159.
Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
Conway, J. M. (1998). Estimation and uses of the proportion of method variance for multitrait- multimethod data. Organizational Research Methods, 1, 209-222.
Gorsuch, R. L. (1983). Factor Analysis (2nd Ed.). Hillsdale, NJ: LEA.
Hunter, J. E. (1983). A causal analysis of cognitive ability, job knowledge, job performance. Journal of Vocational Behavior, 29, 340-362.
James, L. R., Demaree, R. G., Mulaik, S. A., & Ladd, R. T. (1992). Validity generalization in the context of situational models. Journal of Applied Psychology, 77, 3-14.
Landy, F. J. & Shankster, L. J. (1994). Personnel selection and placement. Annual Review of Psychology, 45, 261-296.
Lee, G., Brennan, R. L., & Frisbie, D. A. (2000). Incorporating the testlet concept in test score analyses. Educational Measurement: Issues and Practice, 19, 9-15.
Lievens, F. & Conway, J. M (2001). Dimension and Exercise Variance in Assessment Center Scores: A large-scale evaluation of multitrait- multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
Lowry, P. E. (1997). The assessment center process: New directions.Journal of Social Behavior and Personality, 12, 53-62.
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490-504.
Marsh, H. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and few solutions. Applied Psychological Measurement, 13, 335-361.
Marsh, H. W. & Bailey, M. (1991). Confirmatory factor analyses of multitrait-multimethod data: comparison of the behavior of alternative models. Applied Psychological Measurement, 15, 47-70.
McDaniel, M. A., Hartman, N. S., & Grubb III, W. L. (2003). Situational Judgment Tests, Knowledge, Behavioral Tendency. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology. Orlando, April, 2003.
McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., & Campion, M. A. (2001). Use of Situational Judgment Tests to Predict Job Performance: A Clarification of the Literature. Journal of Applied Psychology, 86, 730-740.
McDaniel, M. A., Yost, A. P., Ludwick, M. H., Hense, R. C., & Hartman, N. S. (2004). Incremental validity of a situational judgment test. Paper presented at the 19th Annual Conference of the Society for Industrial and Organizational Psychology. Chicago. April, 2004.
Montanelli, R. G. Jr. & Humphreys, L. G. (1976). Latent Roots of Random Data Correlation Matrices with Squared Multiple Correlation on the Diagonal: A Monte Carlo Study. Psychometrika, 41, 341-347.
Moss, F. A. (1926). Do you know how to get along with people? Why some people get ahead in the world while others do not. Scientific American, 135, 26-27.
Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An Alternative Selection Procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
Mulaik, S. A. (2010). Foundations of Factor Analysis. New York, NJ: Chapman & Hall/CRC.
Mullins, M. E. & Schmitt, N. (1998). Situational judgment testing: Will the real constructs please present themselves? Paper presented at the 13th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
Robertson, I., Gratton, L., & Sharpley, D. (1989). The psychometric properties of managerial assessment centres: Dimensions into exercises won’t go. Journal of Occupational Psychology, 60, 187-195.
Smith, K. C. & McDaniel, M. A. (1998). Criterion and construct validity evidence for a situational judgment measure. Paper presented at the 13th Annual Convention of the Society for Industrial and Organizational Psychology, Dallas, TX.
Sternberg, R. J., Conway, B. E., Ketron, J. L., & Bernstein, M. (1981). People's Conceptions of Intelligence. Journal of Personality and Social Psychology, 41, 37-55
Wagner, R. K. & Sternberg, R. J. (1985). Practical intelligence in real-world pursuits: The role of tacit knowledge. Journal of Personality and Social Psychology, 49(2), 436-458.
Wagner, R. K. & Sternberg, R. J. (1993). TKIM: The commonsense manager, user manual. New York: Hartcourt Brace Jovanovich, Inc.
Weekley, J. A. & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.
Weekley, J. A. & Jones, C. (1999). Further Studies of Situational Tests. Personnel Psychology, 52, 679-700.
김효선 (2013). 상황판단형 창의성격검사 제작 및 타당화. 성균관대학교 석사학위 청구 논문.
이상철, 이순묵, 조영일. (2003). 지필형 상황판단검사에 대한 비평적 고찰. 한국심리학회지: 산업 및 조직, 16(3), 129-154.
이순묵 (2004). 상황역량 측정에서 상황에 대한 두 관점: 측정오차인가, 해석되어야 할 환경요인인가? 한국심리학회지: 산업 및 조직, 17(2), 243-264.
이순묵 (1995). 요인분석 I. 서울: 학지사.
이순묵, 윤창영, 김영록. (2005). MTMM 자료분석에 대한 비평적 고찰. 한국심리학회지: 산업 및 조직, 18(3), 455-480.
AERA, APA, & NCME (1999). Standards for Educational and Psychological Testing. Washington, DC: Authors.
Asparouhov, T. & Muthen, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397-438.
Becker, T. E. & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625-641.
Bovaird, J. A. & Koziol, N. A. (2012). Measurement models for ordered-categorical indicators. In Hoyle, R. H. (Ed.) Handbook of Structural Equation Modeling. NY: Guilford Press.
Browne, M. W. (1979). The maximum-likelihood solution in inter-battery factor analysis. British Journal of Mathematical and Statistical Psychology, 32, 75-86.
Browne, M. W. (1980). Factor analysis of multiple batteries by maximum likelihood. British Journal of Mathematical and Statistical Psychology, 33, 184-199.
Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36, 111-150.
Browne, M. W. & Tateneni, K. (1997). Noniterative Estimation for the Multiple Battery Factor Analysis Model. Behaviometrika, 24(1), 3-18.
Campbell, D. T. & Fiske, D. W. (1959). Convergent and Discriminant Validation by Multitrait-Multimethod Matrix. Psychological Bulletin, 56, 81-105.
Chan, D. & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159.
Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
Conway, J. M. (1998). Estimation and uses of the proportion of method variance for multitrait- multimethod data. Organizational Research Methods, 1, 209-222.
Gorsuch, R. L. (1983). Factor Analysis (2nd Ed.). Hillsdale, NJ: LEA.
Hunter, J. E. (1983). A causal analysis of cognitive ability, job knowledge, job performance. Journal of Vocational Behavior, 29, 340-362.
James, L. R., Demaree, R. G., Mulaik, S. A., & Ladd, R. T. (1992). Validity generalization in the context of situational models. Journal of Applied Psychology, 77, 3-14.
Landy, F. J. & Shankster, L. J. (1994). Personnel selection and placement. Annual Review of Psychology, 45, 261-296.
Lee, G., Brennan, R. L., & Frisbie, D. A. (2000). Incorporating the testlet concept in test score analyses. Educational Measurement: Issues and Practice, 19, 9-15.
Lievens, F. & Conway, J. M (2001). Dimension and Exercise Variance in Assessment Center Scores: A large-scale evaluation of multitrait- multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
Lowry, P. E. (1997). The assessment center process: New directions.Journal of Social Behavior and Personality, 12, 53-62.
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490-504.
Marsh, H. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and few solutions. Applied Psychological Measurement, 13, 335-361.
Marsh, H. W. & Bailey, M. (1991). Confirmatory factor analyses of multitrait-multimethod data: comparison of the behavior of alternative models. Applied Psychological Measurement, 15, 47-70.
McDaniel, M. A., Hartman, N. S., & Grubb III, W. L. (2003). Situational Judgment Tests, Knowledge, Behavioral Tendency. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology. Orlando, April, 2003.
McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., & Campion, M. A. (2001). Use of Situational Judgment Tests to Predict Job Performance: A Clarification of the Literature. Journal of Applied Psychology, 86, 730-740.
McDaniel, M. A., Yost, A. P., Ludwick, M. H., Hense, R. C., & Hartman, N. S. (2004). Incremental validity of a situational judgment test. Paper presented at the 19th Annual Conference of the Society for Industrial and Organizational Psychology. Chicago. April, 2004.
Montanelli, R. G. Jr. & Humphreys, L. G. (1976). Latent Roots of Random Data Correlation Matrices with Squared Multiple Correlation on the Diagonal: A Monte Carlo Study. Psychometrika, 41, 341-347.
Moss, F. A. (1926). Do you know how to get along with people? Why some people get ahead in the world while others do not. Scientific American, 135, 26-27.
Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An Alternative Selection Procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
Mulaik, S. A. (2010). Foundations of Factor Analysis. New York, NJ: Chapman & Hall/CRC.
Mullins, M. E. & Schmitt, N. (1998). Situational judgment testing: Will the real constructs please present themselves? Paper presented at the 13th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
Robertson, I., Gratton, L., & Sharpley, D. (1989). The psychometric properties of managerial assessment centres: Dimensions into exercises won’t go. Journal of Occupational Psychology, 60, 187-195.
Smith, K. C. & McDaniel, M. A. (1998). Criterion and construct validity evidence for a situational judgment measure. Paper presented at the 13th Annual Convention of the Society for Industrial and Organizational Psychology, Dallas, TX.
Sternberg, R. J., Conway, B. E., Ketron, J. L., & Bernstein, M. (1981). People's Conceptions of Intelligence. Journal of Personality and Social Psychology, 41, 37-55
Wagner, R. K. & Sternberg, R. J. (1985). Practical intelligence in real-world pursuits: The role of tacit knowledge. Journal of Personality and Social Psychology, 49(2), 436-458.
Wagner, R. K. & Sternberg, R. J. (1993). TKIM: The commonsense manager, user manual. New York: Hartcourt Brace Jovanovich, Inc.
Weekley, J. A. & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.
Weekley, J. A. & Jones, C. (1999). Further Studies of Situational Tests. Personnel Psychology, 52, 679-700.