Asia pacific journal of information systems

The Korea Society of Management Information Systems (한국경영정보학회)
권호 표지
DOI QR코드

DOI QR Code

Validity of Language-Based Algorithms Trained on Supervisor Feedback Language for Predicting Interpersonal Fairness in Performance Feedback

  • Jisoo Ock (Department of Business Administration, Pusan National University) ;
  • Joyce S. Pang (Division of Psychology, Nanyang Technological University)
  • Received : 2023.05.26
  • Accepted : 2023.12.07
  • Published : 2023.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Previous research has shown that employees tend to react more positively to corrective feedback from supervisors to the extent they perceive that they were treated with empathy, respect, and concern towards fair interpersonal treatment in receiving the feedback information. Then, to facilitate effective supervisory feedback and coaching, it would be useful for organizations to monitor the contents of feedback exchanges between supervisors and employees to make sure that supervisors are providing performance feedback using languages that are more likely to be perceived as interpersonally fair. Computer-aided text analysis holds potential as a useful tool that organizations can use to efficiently monitor the quality of the feedback messages that supervisors provide to their employees. In the current study, we applied computer-aided text analysis (using closed-vocabulary text analysis) and machine learning to examine the validity of language-based algorithms trained on supervisor language in performance feedback situations for predicting human ratings of feedback interpersonal fairness. Results showed that language-based algorithms predicted feedback interpersonal fairness with reasonable level of accuracy. Our findings provide supportive evidence for the promise of using employee language data for managing (and improving) performance management in organizations.

Keywords

Acknowledgement

This work was supported by a 2-Year Research Grant of Pusan National University.

References

  1. Batson, C. D. (2011). Altruism in Humans. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195341065.001.0001
  2. Chapman, B. P., Weiss, A., and Duberstein, P. R. (2016). Statistical learning theory for high dimensional prediction: Application to criterion-keyed scale development. Psychological Methods, 21(4), 603-620. https://doi.org/10.1037/met0000088
  3. Chilazi, S., Bohnet, I., and Hauser, O. (2021). Achieving gender balance at all levels of your company. Harvard Business Review. Retrieved from https://hbr.org/2021/11/achieving-gender-balanceat-all-levels-of-your-company
  4. Cropanzano, R., Bowen, D. E., and Gilliland, S. W. (2007). The management of organizational justice. Academy of Management Perspectives, 21(4), 34-48. https://doi.org/10.5465/amp.2007.27895338
  5. Eichstaedt, J. C., Kern, M. L., Yaden, D. B., Schwartz, H. A., Giorgi, S., Park, G., Hagan, C. A., Tobolsky, V. A., Smith, L. K. Buffone, A., Iwry, J., Seligman, M. E. P., and Ungar, L. H. (2021). Closed- and open-vocabulary approaches to text analysis: A review, quantitative comparison, and recommendations. Psychological Methods, 26(4), 398-427. https://doi.org/10.1037/met0000349
  6. Ewenstein, B., Hancock, B., and Komm, A. (2016). Ahead of the curve: The future of performance management. McKinsey Quarterly. Retrieved from www.mckinsey.com/business-functions/organization/our-insights/ahead-of-the-curve-the-future-ofperformance-management
  7. Fedor, D. B., Eder, R. W., and Buckley, M. R. (1989). The contributory effects of supervisor intentions on subordinate feedback responses. Organizational Behavior and Human Deicision Process, 44(3), 396-414. https://doi.org/10.1016/0749-5978(89)90016-2
  8. Gallo, J. R., and Steelman, L. A. (2019). Using a training intervention to improve the feedback environment. In L. A. Steelman and J. R. Williams (Eds.), Feedback at work (pp. 163-174). Springer Nature. https://doi.org/10.1007/978-3-030-30915-2_9
  9. Gregory, J. B., Levy, P. E., and Jeffers, M. (2008). Development of a model of the feedback process within executive coaching. Consulting Psychology Journal: Practice and Research, 60(1), 42-56. https://doi.org/10.1037/1065-9293.60.1.42
  10. Hickman, L., Saef, R., Ng, V., Woo, S. E., Tay, L., and Bosch, N. (2021). Developing and evaluating language-based machine learning algorithms for inferring applicant personality in video interviews. Human Resource Management Journal. https://doi.org/10.1111/1748-8583.12356
  11. Jung, Y., and Suh, Y. (2019). Mining the voice of employees: A text mining approach to identifying and analyzing job satisfaction factors from online employee reviews. Decision Support Systems, 123, 113074. https://doi.org/10.1016/j.dss.2019.113074
  12. Kern, M. L., Park, G., Eichstaedt, J. C., Schwartz, H. A., Sap, M., Smith, L. K., and Ungar, L. H. (2016). Gaining insights from social media language: Methodologies and challenges. Psychological Methods, 21(4), 507-525. https://doi.org/10.1037/met0000091
  13. Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1-26. https://doi.org/10.18637/jss.v028.i05
  14. Kurz, R., and Bartram, D. (2002). Competency and individual performance: Modeling the world of work. In I. T. Robertson, M. Callinan, and D. Bartram (Eds.), Organizational effectiveness: The role of psychology (pp. 227-255). Wiley.
  15. London, M. (2003). Job feedback: Giving, seeking, and using feedback for performance improvement. Erlbaum.
  16. Leung, K., Su, S., and Morris, M. W. (2001). When is criticism not constructive? The roles of fairness perceptions and dispositional attributions in employee acceptance of critical supervisory feedback. Human Relations, 54(9), 1155-1187. https://doi.org/10.1177/0018726701549002
  17. Madera, J. M., Hebl, M. R., Dial, H., Martin, R., and Valian, V. (2019). Raising doubt in letters of recommendation for academia: Gender differences and their impact. Journal of Business and Psychology, 34(3), 287-303. https://doi.org/10.1007/s10869-018-9541-1
  18. Meyer, H. H. (1991). A solution to the performance appraisal feedback enigma. Academic of Management Executive, 5(1), 68-76. https://doi.org/10.5465/AME.1991.4274724
  19. McKenny, A. F., Aguinis, H., Short, J. C., and Anglin, A. H. (2018). What doesn't get measured does exist: Improving the accuracy of computer-aided text analysis. Journal of Management, 44(7), 2909-2933. https://doi.org/10.1177/0149206316657594
  20. Murphy, K. R., and Cleveland, J. N. (1995). Understanding Performance Appraisal: Social, Organizational and Goal-Oriented Perspectives. Sage.
  21. Nguyen, D. T., Gancarz, T., Ng, F., Dabbish, L. A., and Dow, S. P. (2017). Fruitful feedback: Positive affective language and source anonymity improve critique reception and work outcomes. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 1024-1034). ACM. https://doi.org/10.1145/2998181.2998319
  22. Ock, J., and Pang, J. S. (2022, April). Effect of Personality and Knowledge about Professional Conduct on Feedback Delivery [Poster]. Society for Industrial and Organizational Psychology Annual Conference, Seattle, WA, United States.
  23. O'Malley, A. L., and Gregory, J. B. (2011). Don't be such a downer: Using positive psychology to enhance the value of negative feedback. The Psychologist-Manager Journal, 14(4), 247-264. https://doi.org/10.1080/10887156.2011.621776
  24. Pang, J. S., and Ock, J. (2023). Supervisors' prosocial feedback delivery: Dispositional trait and motivational concerns. In J. Wood, J. Ramsay, K. Thirumaran, and E. Ng (Eds.), Managing People across Asia-Pacific: An Organizational Psychology Approach. Edward Elger Publishing.
  25. Pennebaker, J. W., Boyd, R. L., Jordan, K., and Blackburn, K. (2015). The Development and Psychometric Properties of LIWC2015. Austin, TX: University of Texas at Austin.
  26. Petryk, M., Rivera, M., Bhattacharya, S., Qiu, L., and Kumar, S. (2022). How network embeddedness affects real-time performance feedback: An empirical investigation. Information Systems Research, 33(4), 1467-1489. https://doi.org/10.1287/isre.2022.1110
  27. Pulakos, E. D., and O'Leary, R. S. (2011). Why is performance management broken? Industrial and Organizational Psychology, 4(2), 146-164. https://doi.org/10.1111/j.1754-9434.2011.01315.x
  28. Rivera, M., Qiu, L., Kumar, S., and Petrucci, T. (2021). Are traditional performance reviews outdated? An empirical analysis on continuous, real-time feedback in the workplace. Information Systems Research, 32(2), 517-540. https://doi.org/10.1287/isre.2020.0979
  29. Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., and Ungar, L. H. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS ONE, 8(9), e73791. https://doi.org/10.1371/journal.pone.0073791
  30. Sheets, T. L., Belwalkar, B. B., Toaddy, S. R., and McClure, T. K. (2019). Filling the I-O/technology void. In R. N. Landers (Ed.), The Cambridge Handbook of Technology and Employee Behavior (pp. 22-37). Cambridge University Press. https://doi.org/10.1017/9781108649636.003
  31. Speer, A. B. (2021). Scoring dimension-level job performance from narrative comments: Validity and generalizability when using natural language processing. Organizational Research Methods, 24(3), 572-594. https://doi.org/10.1177/1094428120930815
  32. Speer, A. B., Schwendeman, M. G., Reich, C. C., Tenbrink, A. P., and Siver, S. R. (2019). Investigating the construct validity of performance comments: Creation of the Great Eight narrative dictionary. Journal of Business and Psychology, 34(6), 747-767. https://doi.org/10.1007/s10869-018-9599-9
  33. Spence, J. R., and Keeping, L. (2011). Conscious rating distortion in performance appraisal: A review, commentary, and proposed framework for research. Human Resource Management Review, 21(2), 85-95. https://doi.org/10.1016/j.hrmr.2010.09.013
  34. Steelman, L. A., Levy, P. E., and Snell, A. F. (2004). The feedback environment scale: Construct definition, measurement, and validation. Educational and Psychological Measurement, 64(1), 165-184. https://doi.org/10.1177/0013164403258440
  35. Steelman, L. A., and Rutkowski, K. A. (2004). Moderators of employee reactions to negative feedback. Journal of Managerial Psychology, 19(1), 6-18. https://doi.org/10.1108/02683940410520637
  36. UN Women (2021). Snapshot of women's leadership in Asia and the Pacific. Retrieved from https://asiapacific.unwomen.org/en/news-and-events/in-focus/csw/snapshot-of-womens-leadership-in-asia-and-the-pacific#_ftn17
  37. VandeWalle, D., Ganesan, S., Challagalla, G. N., and Brown, S. P. (2000). An integrated model of feedback-seeking behavior: Disposition, context, and cognition. Journal of Applied Psychology, 85(6), 996-1003. https://doi.org/10.1037/0021-9010.85.6.996
  38. Whitaker, B. G., Dahling, J. J., and Levy, P. (2007). The development of a feedback environment and role clarity model of job performance. Journal of Management, 33(4), 570-591. https://doi.org/10.1177/0149206306297581
  39. Young, S. F., Richard, E. M., Moukarzel, R. G., Steelman, L. A., and Gentry, W. A. (2017). How empathetic concern helps leaders in providing negative feedback: A two-study examination. Journal of Occupational and Organizational Psychology, 90(4), 535-558. https://doi.org/10.1111/joop.12184
  40. Yukl, G. (2002). Leadership in Organizations. Prentice Hall.