Selection of Reliable and Valid Surgeon Performance Measures

William Thomas Cecil

doi:doi:10.11648/j.ajmse.20200505.12

| Peer-Reviewed

Selection of Reliable and Valid Surgeon Performance Measures

William Thomas Cecil

Published in American Journal of Management Science and Engineering (Volume 5, Issue 5)

Received: 26 October 2020 Accepted: 3 November 2020 Published: 11 November 2020

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.

Published in	American Journal of Management Science and Engineering (Volume 5, Issue 5)
DOI	10.11648/j.ajmse.20200505.12
Page(s)	62-69
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2020. Published by Science Publishing Group

Keywords

Multilevel Mixed-Effects Modeling, Risk Adjustment for Clinical Outcomes, Reliability, Validity

References

[1]	Yi, S. G., N. P. Wray, S. L. Jones, B. L. Bass, J. Nishioka, S. Brann, and C. M. Ashton. 2013. "Surgeon-Specific Performance Reports in General Surgery: An Observational Study of Initial Implementation and Adoption." Journal of the American College of Surgeons 217 (4): 636-647.E631.
[2]	Allen, M., and O. Pierce. 2015. "Making the Cut: Why Choosing the Right Surgeon Matters Even More than You Know." ProPublica, July 13, 2015.
[3]	Birkmeyer, N. J., and J. D. Birkmeyer. 2006. "Strategies for Improving Surgical Quality--Should Payers Reward Excellence or Effort?" New England Journal of Medicine 354 (8): 864-870.
[4]	Cooper, W. O., O. Guillamondegui, O. J. Hines, C. S. Hultman, R. R. Kelz, P. Shen, D. A. Spain, J. F. Sweeney, I. N. Moore, J. Hopkins, I. R. Horowitz, R. M. Howerton, J. W. Meredith, N. O. Spell, P. Sullivan, H. J. Domenico, J. W. Pichert, T. F. Catron, L. E. Webb, R. R. Dmochowski, J. Karrass, and G. B. Hickson. 2017. "Use of Unsolicited Patient Observations to Identify Surgeons with Increased Risk for Postoperative Complications." JAMA Surgery 152 (6): 522-529.
[5]	Stephens, M. R., W. G. Lewis, A. E. Brewster, I. Lord, G. R. Blackshaw, I. Hodzovic, G. V. Thomas, S. A. Roberts, T. D. Crosby, C. Gent, M. C. Allison, and K. Shute. 2006. "Multidisciplinary Team Management is Associated with Improved Outcomes after Surgery for Esophageal Cancer." Diseases of the Esophagus 19 (3): 164-171.
[6]	Nurok, M., N. Sadovnikoff, and B. Gewertz. 2016. "Contemporary Multidisciplinary Care--Who Is the Captain of the Ship, and Does It Matter?" JAMA Surgery 151 (4): 309-310.
[7]	Statement on Physician-Led Team-Based Surgical Care. 2016. "American College of Surgeons." [accessed on August 11, 2017]. Available at: https://www.facs.org/about-acs/statements/86-team-based.
[8]	Hall, B. L., K. M. Huffman, B. H. Hamilton, J. L. Paruch, L. Zhou, K. E. Richards, M. E. Cohen, and C. Y. Ko. 2015. "Profiling Individual Surgeon Performance Using Information from a High-Quality Clinical Registry: Opportunities and Limitations." Journal of the American College of Surgeons 221 (5): 901-913.
[9]	Shih, T., A. I. Cole, P. M. Al-Attar, A. Chakrabarti, H. A. Fardous, P. F. Helvie, M. T. Kemp, C. Lee, E. Shtull-Leber, D. A. Campbell, M. J. Englesbe. 2015. "Reliability of Surgeon-Specific Reporting of Complications after Colectomy." Annals of surgery 261 (5): 920-925.
[10]	Iezzoni, L. I. 2003. Risk Adjustment for Measuring Health Care Outcomes, Third Edition. Chicago, Illinois: Health Administration Press.
[11]	Adams, J. L. 2009. The Reliability of Provider Profiling: A Tutorial. Santa Monica: RAND Corporation.
[12]	Adams, J. L., A. Mehrotra, J. W. Thomas, and E. A. McGlynn. 2010. "Physician Cost Profiling — Reliability and Risk of Misclassification." New England Journal of Medicine 362 (11): 1014-1021.
[13]	Rabe-Hesketh, S., and A. Skrondal. 2012. Multilevel and Longitudinal Modeling Using Stata, Volumes I and II, Third Edition. College Station, TX: Stata Press. Stata Corp LP.
[14]	Guillamondegui, O. D., O. L. Gunter, L. Hines, B. J. Martin, W. Gibson, P. C. Clarke, W. T. Cecil, and J. B. Cofer. 2012. "Using the National Surgical Quality Improvement Program and the Tennessee Surgical Quality Collaborative to Improve Surgical Outcomes." Journal of the American College of Surgeons 214 (4): 709-714.
[15]	Rotondo, M. F., C. Cribari, and R. S. Smith. 2014. Resources for Optimal Care of the Injured Patient. Chicago, Illinois.: Committee on Trauma, American College of Surgeons.
[16]	Stone, A. B., M. C. Grant, C. Pio Roda, D. Hobson, T. Pawlik, C. L. Wu, and E. C. Wick. 2016. "Implementation Costs of an Enhanced Recovery after Surgery Program in the United States: A Financial Model and Sensitivity Analysis Based on Experiences at a Quaternary Academic Medical Center." Journal of the American College of Surgeons 222 (3): 219-225.
[17]	Hopper, A. N., M. H. Jamison, and W. G. Lewis. 2007. "Learning Curves in Surgical Practice." Postgraduate Medical Journal 83 (986): 777-779.
[18]	Jaffer, A., B. Bednarz, B. Challacombe, and S. Sriprasad. 2009. "The assessment of surgical competency in the UK." International Journal of Surgery 7 (1): 12-15.
[19]	Khan, N., H. Abboudi, M. S. Khan, P. Dasgupta, and K. Ahmed. 2014. "Measuring the surgical 'learning curve': methods, variables and competency." BJU International 113 (3): 504-508.

Cite This Article

Plain Text BibTeX RIS

APA Style

William Thomas Cecil. (2020). Selection of Reliable and Valid Surgeon Performance Measures. American Journal of Management Science and Engineering, 5(5), 62-69. https://doi.org/10.11648/j.ajmse.20200505.12

Copy | Download

ACS Style

William Thomas Cecil. Selection of Reliable and Valid Surgeon Performance Measures. Am. J. Manag. Sci. Eng. 2020, 5(5), 62-69. doi: 10.11648/j.ajmse.20200505.12

Copy | Download

AMA Style

William Thomas Cecil. Selection of Reliable and Valid Surgeon Performance Measures. Am J Manag Sci Eng. 2020;5(5):62-69. doi: 10.11648/j.ajmse.20200505.12

Copy | Download

@article{10.11648/j.ajmse.20200505.12,
  author = {William Thomas Cecil},
  title = {Selection of Reliable and Valid Surgeon Performance Measures},
  journal = {American Journal of Management Science and Engineering},
  volume = {5},
  number = {5},
  pages = {62-69},
  doi = {10.11648/j.ajmse.20200505.12},
  url = {https://doi.org/10.11648/j.ajmse.20200505.12},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajmse.20200505.12},
  abstract = {Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.},
 year = {2020}
}

Copy | Download

TY  - JOUR
T1  - Selection of Reliable and Valid Surgeon Performance Measures
AU  - William Thomas Cecil
Y1  - 2020/11/11
PY  - 2020
N1  - https://doi.org/10.11648/j.ajmse.20200505.12
DO  - 10.11648/j.ajmse.20200505.12
T2  - American Journal of Management Science and Engineering
JF  - American Journal of Management Science and Engineering
JO  - American Journal of Management Science and Engineering
SP  - 62
EP  - 69
PB  - Science Publishing Group
SN  - 2575-1379
UR  - https://doi.org/10.11648/j.ajmse.20200505.12
AB  - Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.
VL  - 5
IS  - 5
ER  -

Copy | Download

Author Information

William Thomas Cecil

Independent Consultant, Knoxville, Tennessee, the United States

Download PDF

Sections

Plain Text BibTeX RIS

APA Style

William Thomas Cecil. (2020). Selection of Reliable and Valid Surgeon Performance Measures. American Journal of Management Science and Engineering, 5(5), 62-69. https://doi.org/10.11648/j.ajmse.20200505.12

Copy | Download

ACS Style

William Thomas Cecil. Selection of Reliable and Valid Surgeon Performance Measures. Am. J. Manag. Sci. Eng. 2020, 5(5), 62-69. doi: 10.11648/j.ajmse.20200505.12

Copy | Download

AMA Style

William Thomas Cecil. Selection of Reliable and Valid Surgeon Performance Measures. Am J Manag Sci Eng. 2020;5(5):62-69. doi: 10.11648/j.ajmse.20200505.12

Copy | Download

@article{10.11648/j.ajmse.20200505.12,
  author = {William Thomas Cecil},
  title = {Selection of Reliable and Valid Surgeon Performance Measures},
  journal = {American Journal of Management Science and Engineering},
  volume = {5},
  number = {5},
  pages = {62-69},
  doi = {10.11648/j.ajmse.20200505.12},
  url = {https://doi.org/10.11648/j.ajmse.20200505.12},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajmse.20200505.12},
  abstract = {Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.},
 year = {2020}
}

Copy | Download

TY  - JOUR
T1  - Selection of Reliable and Valid Surgeon Performance Measures
AU  - William Thomas Cecil
Y1  - 2020/11/11
PY  - 2020
N1  - https://doi.org/10.11648/j.ajmse.20200505.12
DO  - 10.11648/j.ajmse.20200505.12
T2  - American Journal of Management Science and Engineering
JF  - American Journal of Management Science and Engineering
JO  - American Journal of Management Science and Engineering
SP  - 62
EP  - 69
PB  - Science Publishing Group
SN  - 2575-1379
UR  - https://doi.org/10.11648/j.ajmse.20200505.12
AB  - Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.
VL  - 5
IS  - 5
ER  -

Copy | Download