Comparison of validities for scoring keys and scoring algorithms in situational judgment test The influence of faking
Article Sidebar
Published:
Feb 28, 2011
Main Article Content
Abstract
The purpose of the present study was to examine the fakability of the situational judgment test. Specifically, the study was focused on the following questions; (1) whether participants are able to fake their answers on the situational judgment test in the real situation of selection, (2) whether faking influences the criterion-related validity of the situational judgment test and its incremental validity over cognitive and personality tests, and (3) whether the combination of different scoring key(SME consensus, average in response, and empirical keying) and different scoring algorithm(scenario, Best-Worst, and Pick most) has influence on the degree of fakability as well as both criterion-related validity and incremental validity of the situational judgment test. 110 students who applied to the leadership program were considered the faking group, while 129 students of B department at A university were considered the honest group. The members of both groups completed a cognitive test, a personality questionnaire and a situational judgment test. Only for the situational judgment tests, each group was asked to respond as instructed. Another group of 78 students of A university participated in the survey to develop two scoring key(empirical, average in response keying). SME consensus key was developed by 9 SMEs(5 undergraduate students with leadership and good GPA, 4 graduate students). And then 9 situational judgment scores were produced independently. Results indicated that the all scores of students in the faking group were significantly higher than those of students in the honest group. Furthermore, criterion-related validity of the situational judgement test in the honest group was higher than that of the faking group for both task performance and contextual performance. While faking had negative effects on the criterion-related validity for both criteria of performance, incremental validity of the situational judgement test in the honest group was higher than that of the faking group only for the contextual criteria. Finally, the limitation and future direction of the present study were discussed.
Metrics
Metrics Loading ...
Article Details
How to Cite
Kim, E., Han, Y., & Kim, M. (2011). Comparison of validities for scoring keys and scoring algorithms in situational judgment test: The influence of faking. Korean Journal of Industrial and Organizational Psychology, 24(1), 231–255. https://doi.org/10.24230/kjiop.v24i1.231-255
Section
Empirical Articles
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
강민우, 윤창영, 이순묵 (2005). 지시문과 채점방식에 따른 상황판단검사의 타당도 비교. 한국심리학회지: 산업 및 조직, 18(3), 547-565.
김명소, 이헌주 (2006). 성격검사의 형식이 응답왜곡에 미치는 효과. 한국심리학회지: 산업 및 조직, 19(3), 371-393.
박동건, 전인식 (2001). 전기자료(biodata) 문항의 가중치 부여 체계간의 타당도 연구: 분석집단 크기에 따른 비교연구. 한국심리학회지: 산업 및 조직, 14(1), 101-113.
연합뉴스 (2005). 신입사원 채용 시 인, 적성 검사 강화. 9월 26일자.
이순묵 (2003). 지필형 상황판단검사에 대한 비평적 고찰: 한국 심리학회지: 산업 및 조직, 16(3), 129-154.
유태용, 이기범, Ashton, M. C. (2004). 한국판 HEXACO 성격검사의 구성타당화 연구. 한국심리학회지: 사회 및 성격, 18(3), 61-75.
Arthur, W., & Valido, A. J. (2008). The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. Journal of Applied Psychology, 93, 435-442.
Bergman, M. E., Drasgow, F., Donovan, M. A., Henning, J. B., & Juraska, S. (2006). Scoring situational judgement tests: Once you get the data, Your trouble begin. International Journal of Selection and assessment, 14, 223-235.
Bess, T. L., & Mullins, M. E (2002). Exploring a dimensionality of situational judgment: task and contextual knowledge. Paper presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Canada.
Chan, d., & Schmitt, N (2005). Situatioanl judgment Tests. In A. Evers, N. Anderson, & O. Smit-Voskuijl(Eds.) The blackwell handbook of Personnel selection(pp.219-242). The blackwell Publisher.
Clevenger, J., Pereira, G. M., Wiechmann, D., Schmidt-Harvey, V. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
Clevenger, J. p, & Haaland D. E. (2000). Examining the relationship beetween job knowledge and situational judgment performance. Paper presented at the 15th Annual Conference of the Society of Industrial and Organizational Psychology. New Orleans. April.
Cureton, E. E. (1950). Validity, reliability, and baloney. Educational Measurement, 5, 115-124.
Douglas, E. F., McDaniel, M. A., & Snell, A. F. (1996). The validity of non-cognitive measure decays when applicants fake. Paper presented at the annual conference of the Academy of Management, Cincinnati, OH.
Dwight, S. A. (1999). An assessment of the difference of warning applicants not to fake. Unpublished doctoral dissertation, State University of New York at Albany.
Ellingson, J. E., Smith, D. B., & Sackett, P. R. (2001). Investigating the Influence of social desirability on personal factor structure. Journal of Applied Psychology, 86, 122-133.
Hogan, J. B. (1994). Empirical keying of background data measures. In G.S. Stokes, M.D. Mumford and W. A. Owens(Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance predictions(pp.69-107). Palo Aloto: Consulting Psychology Press.
Hooper, A. C., Cullen, M. J., & Sackett, P. R. (2006). “Operational threat to the use of SJTs: faking, coaching, and retesting issue”, in Weekley, J. A., & Ployhart, R. E.(Eds), Situational Judgment Tests: Theory, Measurement and Application, Lawrence Erlbaum Associates, Mahwah, NJ, pp.205-232.
Hough, L. M. (1998). Effects of intentional distortion in personality measurement and evaluation of suggested palliatives. Human Performance, 11, 209-244.
Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Detrerminats, detection, and amelioration of adverse impact in personnel selection procedure: Issue, evidence, and lessons learned. Internal Journal of Selection and assessment, 9, 152-194.
Juraska, S. E., & Drasgow, F. (2001). Faking situational judgment: A tests of Conflicts Resolution Skills Assessment. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego. CA.
Knapp, D. J., Campbell, C. H., Borman, W. C.,Pulakos, E. D., & Hanson, M. A. (2001). Performance assessment for a population of jobs. In J. P. Campbell & D. J. Knapp(Eds), Exploring the limits in personnel selection and classfication. Mahwah, N. J: Lawrence Erlbaum Associates.
Latham., G. P., & Saari, L. M. (1984). Do people do what they say? Further studies of the situational interview. Journal of Applied Psychology, 69, 569-573.
Latham., G. P., Saari, L. M., Pursell, E. D., & Campion, M. A. (1980). The situational interview. Journal of applied psychology, 65, 422-427.
Lievens, F., Peeters, H & Schollaert, F. (2008). Situational judgment tests: a review of recent research. Personnel Review. 37. 426-411.
McDaniel, L. A, Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgement test to predict job performance: A clarification of the literature. Journal of Applied psychology, 86, 812-821.
McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & GrubⅢ, W. L. (2003). Situational Judgments, response instructions, and validity: A meta analysis. Personnel psychology, 60, 63-91.
McDaniel., M. A & Nguyen, N. T. (2001). Situational judgment tests: a review of practice and construct assessed. International Journal of Selection and assessement, 9, 103-113.
McFarland, L. A., & Ryan, A. M. (2000). Variance in faking across non-cognitive measure: Effects on faking behavior and test measurement properties. Journal of Personality Assessment, 78l. 348-369.
Motowidlo, S. J. & Tippins, N. (1993). Further studies of low-fidelity simulation in the form of a situational inventory. Journal of Occupational and Organizational Psychology, 66, 337-344.
Motowidlo. S. J., & Dunnet, M.D & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied psychology, 75, 640-647.
Motowidlo. S. J., Hanson. M. A., & Craft. J.L (1997). Applied measurement method in industrial Psychology. In D. I., Whetzel & G. R. Wheaton(Eds). Applied measurement method in industrial psychology(pp.241-260). Palo Alto. CA: Davies-Black
Mumford, T. V., & Owens, W. A. (1987). Methodology review: Principles, procedures, and findings in the application of background data measures. Applied Psychological Measurement, 11, 1-31.
Ones, D. S., & Viswesvaran, C. (1998). The effects of social desirability and faking on personality and integrity assessment for personnel selection. Human Performance, 11, 245-269.
Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. j., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-208.
Paullin, C., & Hanson, M. A. (2001). Comparing the validity of rationally-derived and empirically-derived scoring keys for situational judgment inventory. Paper presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA
Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college student's success: the influence of faking. Educational and psychological Measurement. 65, 70-89.
Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11, 1-16.
Ployhart, R. E., & Ryan, A. M. (1998). The relative importance of procedure and distributive justice in determining applicant's reactions. Journal of applied psychology, 83, 3-16.
Ployhart, R. E., Schnider, & Schmitt, N. (2005). Organizational staffing: Contemporary practice and theory. Mahwah, NJ: Lawrence Erlbaum Associates.
Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752.
Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241-258.
Reynolds, D. H., Winter, J. L., & Scott, D. R. (1999). Development, validation and translation of a Professional-level situational judgment Inventory items. Invited presentation to College Board, New York.
Schmidt, D. B., & Wolf, P. P. (2003). Susceptibility of SJTs to applicant faking: An examination of applicant and incumbent samples, Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
Shuang-Yueh Pui. (2007). Situational Judgement Tests: A Measurement of Judgement?. The degree of Master of Arts in Industrial and Organizational Psychology the Graduate College of Bowling Green State University.
Smith, K. C., & McDaniel, M. A. (1998). Criterion and construct validity evidence for a situational judgment measure. Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Inc., Dallas, TX.
Smith, C. A.,. Organ, D W & Near, J (1983). Organizational Citizenship Behavior: Its Nature and Antecedents. Journal of Applied Psychology, 68, 653-663.
Wall Street Journal. (2009). Adding Personality to the College Admissions Mix. 8월 20일자
Waugh, G. (2002). Selecting response options and item for situational judgment test. Paper presented as part of following symposium - Understanding and predicting Performance in future jobs. 17th Annual conference of society for industrial and organizational psychology, Tronto.
Weekley, J. A., & Jones, C. (1997). Wideo-based situational testing. Personnel psychology, 50, 24-49.
Weekley, J. A., & Ployhart, R. E. (2005). Situational judgment: Antecedents and relation ship with performance. Human Performance, 18, 81-104.
Weekley, J. A., & Ployhart, R. E. (2006). An introduction to situational judgment testing. In J. A, Weekley & R. E. Ployhart(Eds.) Situational judgment tests. Mahwah, NJ: Lawrence Erlbaum Associates.
Weekley, J. A., Ployharts, R. E., & Harold, C. (2003). Personality and situational judgment tests across applicant and incumbent setting: An examination of validity, measurement, and subgrouping differences. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
Weekley., J. a. & Gier, J. A. (1987). Reliability and validity of situational interview for a sales position. Journal of applied Psychology, 72, 484-487.
Zickar, M. J. (1997). Computer simulation of faking on a personality test. Paper Presented at the annual conference of the Society for Industrial and Organizational Psychology, St Louis, Mo.
Zickar, M. J., & Robie, C. (1999). Modeling faking good on personality items: An item-level analysis, Journal of Applied Psychology, 84, 551-563.
김명소, 이헌주 (2006). 성격검사의 형식이 응답왜곡에 미치는 효과. 한국심리학회지: 산업 및 조직, 19(3), 371-393.
박동건, 전인식 (2001). 전기자료(biodata) 문항의 가중치 부여 체계간의 타당도 연구: 분석집단 크기에 따른 비교연구. 한국심리학회지: 산업 및 조직, 14(1), 101-113.
연합뉴스 (2005). 신입사원 채용 시 인, 적성 검사 강화. 9월 26일자.
이순묵 (2003). 지필형 상황판단검사에 대한 비평적 고찰: 한국 심리학회지: 산업 및 조직, 16(3), 129-154.
유태용, 이기범, Ashton, M. C. (2004). 한국판 HEXACO 성격검사의 구성타당화 연구. 한국심리학회지: 사회 및 성격, 18(3), 61-75.
Arthur, W., & Valido, A. J. (2008). The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. Journal of Applied Psychology, 93, 435-442.
Bergman, M. E., Drasgow, F., Donovan, M. A., Henning, J. B., & Juraska, S. (2006). Scoring situational judgement tests: Once you get the data, Your trouble begin. International Journal of Selection and assessment, 14, 223-235.
Bess, T. L., & Mullins, M. E (2002). Exploring a dimensionality of situational judgment: task and contextual knowledge. Paper presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Canada.
Chan, d., & Schmitt, N (2005). Situatioanl judgment Tests. In A. Evers, N. Anderson, & O. Smit-Voskuijl(Eds.) The blackwell handbook of Personnel selection(pp.219-242). The blackwell Publisher.
Clevenger, J., Pereira, G. M., Wiechmann, D., Schmidt-Harvey, V. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.
Clevenger, J. p, & Haaland D. E. (2000). Examining the relationship beetween job knowledge and situational judgment performance. Paper presented at the 15th Annual Conference of the Society of Industrial and Organizational Psychology. New Orleans. April.
Cureton, E. E. (1950). Validity, reliability, and baloney. Educational Measurement, 5, 115-124.
Douglas, E. F., McDaniel, M. A., & Snell, A. F. (1996). The validity of non-cognitive measure decays when applicants fake. Paper presented at the annual conference of the Academy of Management, Cincinnati, OH.
Dwight, S. A. (1999). An assessment of the difference of warning applicants not to fake. Unpublished doctoral dissertation, State University of New York at Albany.
Ellingson, J. E., Smith, D. B., & Sackett, P. R. (2001). Investigating the Influence of social desirability on personal factor structure. Journal of Applied Psychology, 86, 122-133.
Hogan, J. B. (1994). Empirical keying of background data measures. In G.S. Stokes, M.D. Mumford and W. A. Owens(Eds), Biodata handbook: Theory, research, and use of biographical information in selection and performance predictions(pp.69-107). Palo Aloto: Consulting Psychology Press.
Hooper, A. C., Cullen, M. J., & Sackett, P. R. (2006). “Operational threat to the use of SJTs: faking, coaching, and retesting issue”, in Weekley, J. A., & Ployhart, R. E.(Eds), Situational Judgment Tests: Theory, Measurement and Application, Lawrence Erlbaum Associates, Mahwah, NJ, pp.205-232.
Hough, L. M. (1998). Effects of intentional distortion in personality measurement and evaluation of suggested palliatives. Human Performance, 11, 209-244.
Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Detrerminats, detection, and amelioration of adverse impact in personnel selection procedure: Issue, evidence, and lessons learned. Internal Journal of Selection and assessment, 9, 152-194.
Juraska, S. E., & Drasgow, F. (2001). Faking situational judgment: A tests of Conflicts Resolution Skills Assessment. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego. CA.
Knapp, D. J., Campbell, C. H., Borman, W. C.,Pulakos, E. D., & Hanson, M. A. (2001). Performance assessment for a population of jobs. In J. P. Campbell & D. J. Knapp(Eds), Exploring the limits in personnel selection and classfication. Mahwah, N. J: Lawrence Erlbaum Associates.
Latham., G. P., & Saari, L. M. (1984). Do people do what they say? Further studies of the situational interview. Journal of Applied Psychology, 69, 569-573.
Latham., G. P., Saari, L. M., Pursell, E. D., & Campion, M. A. (1980). The situational interview. Journal of applied psychology, 65, 422-427.
Lievens, F., Peeters, H & Schollaert, F. (2008). Situational judgment tests: a review of recent research. Personnel Review. 37. 426-411.
McDaniel, L. A, Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgement test to predict job performance: A clarification of the literature. Journal of Applied psychology, 86, 812-821.
McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & GrubⅢ, W. L. (2003). Situational Judgments, response instructions, and validity: A meta analysis. Personnel psychology, 60, 63-91.
McDaniel., M. A & Nguyen, N. T. (2001). Situational judgment tests: a review of practice and construct assessed. International Journal of Selection and assessement, 9, 103-113.
McFarland, L. A., & Ryan, A. M. (2000). Variance in faking across non-cognitive measure: Effects on faking behavior and test measurement properties. Journal of Personality Assessment, 78l. 348-369.
Motowidlo, S. J. & Tippins, N. (1993). Further studies of low-fidelity simulation in the form of a situational inventory. Journal of Occupational and Organizational Psychology, 66, 337-344.
Motowidlo. S. J., & Dunnet, M.D & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied psychology, 75, 640-647.
Motowidlo. S. J., Hanson. M. A., & Craft. J.L (1997). Applied measurement method in industrial Psychology. In D. I., Whetzel & G. R. Wheaton(Eds). Applied measurement method in industrial psychology(pp.241-260). Palo Alto. CA: Davies-Black
Mumford, T. V., & Owens, W. A. (1987). Methodology review: Principles, procedures, and findings in the application of background data measures. Applied Psychological Measurement, 11, 1-31.
Ones, D. S., & Viswesvaran, C. (1998). The effects of social desirability and faking on personality and integrity assessment for personnel selection. Human Performance, 11, 245-269.
Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. j., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-208.
Paullin, C., & Hanson, M. A. (2001). Comparing the validity of rationally-derived and empirically-derived scoring keys for situational judgment inventory. Paper presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA
Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college student's success: the influence of faking. Educational and psychological Measurement. 65, 70-89.
Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11, 1-16.
Ployhart, R. E., & Ryan, A. M. (1998). The relative importance of procedure and distributive justice in determining applicant's reactions. Journal of applied psychology, 83, 3-16.
Ployhart, R. E., Schnider, & Schmitt, N. (2005). Organizational staffing: Contemporary practice and theory. Mahwah, NJ: Lawrence Erlbaum Associates.
Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752.
Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241-258.
Reynolds, D. H., Winter, J. L., & Scott, D. R. (1999). Development, validation and translation of a Professional-level situational judgment Inventory items. Invited presentation to College Board, New York.
Schmidt, D. B., & Wolf, P. P. (2003). Susceptibility of SJTs to applicant faking: An examination of applicant and incumbent samples, Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
Shuang-Yueh Pui. (2007). Situational Judgement Tests: A Measurement of Judgement?. The degree of Master of Arts in Industrial and Organizational Psychology the Graduate College of Bowling Green State University.
Smith, K. C., & McDaniel, M. A. (1998). Criterion and construct validity evidence for a situational judgment measure. Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Inc., Dallas, TX.
Smith, C. A.,. Organ, D W & Near, J (1983). Organizational Citizenship Behavior: Its Nature and Antecedents. Journal of Applied Psychology, 68, 653-663.
Wall Street Journal. (2009). Adding Personality to the College Admissions Mix. 8월 20일자
Waugh, G. (2002). Selecting response options and item for situational judgment test. Paper presented as part of following symposium - Understanding and predicting Performance in future jobs. 17th Annual conference of society for industrial and organizational psychology, Tronto.
Weekley, J. A., & Jones, C. (1997). Wideo-based situational testing. Personnel psychology, 50, 24-49.
Weekley, J. A., & Ployhart, R. E. (2005). Situational judgment: Antecedents and relation ship with performance. Human Performance, 18, 81-104.
Weekley, J. A., & Ployhart, R. E. (2006). An introduction to situational judgment testing. In J. A, Weekley & R. E. Ployhart(Eds.) Situational judgment tests. Mahwah, NJ: Lawrence Erlbaum Associates.
Weekley, J. A., Ployharts, R. E., & Harold, C. (2003). Personality and situational judgment tests across applicant and incumbent setting: An examination of validity, measurement, and subgrouping differences. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
Weekley., J. a. & Gier, J. A. (1987). Reliability and validity of situational interview for a sales position. Journal of applied Psychology, 72, 484-487.
Zickar, M. J. (1997). Computer simulation of faking on a personality test. Paper Presented at the annual conference of the Society for Industrial and Organizational Psychology, St Louis, Mo.
Zickar, M. J., & Robie, C. (1999). Modeling faking good on personality items: An item-level analysis, Journal of Applied Psychology, 84, 551-563.