Received date: March 15, 2018; Accepted date: March 19, 2018; Published date: March 22, 2018
Citation: Wilcox RE, Lawson KA, Acosta WR (2018) Contribution of Cognitive and Non-Cognitive Admissions Measures to Pharm.D. Applicant Admission. J Pharma Pract Edu Vol 1:10.
Copyright: © 2018 Wilcox RE, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Background: Recent literature reviews have highlighted the need for more quantitative measures of non-cognitive behavior that, coupled with cognitive measures, can inform a holistic Pharmacy admissions process to train expert clinical practitioners in patient-centered care in an inter-professional collaborative practice environment. The present investigation was a retrospective analysis of cognitive (science GPAs, PCAT composite scores) and non-cognitive (professional development ratings (RATE), and multiple mini-interview scores (MMI)) variables to determine how these metrics could contribute to an effective admissions process in a Pharm.D. program.
Methods: Analyses were conducted with de-identified data from applicants to a Pharm.D program during the 2015 and 2016 application cycles. Professional ratings (RATE) were developed by the college’s admissions committee and had been used effectively since 2010. MMI scenarios were initially obtained from a set developed by ProFitHR and then adapted for the US health care system and pharmacy by a scenario implementation team that was part of the MMI task force. A two-part screening system was utilized in which step 1 included GPA, PCAT and RATE to determine those individuals invited for interviews and step 2 included these plus the MMI score in a holistic review process. A final composite z-score, calculated as the arithmetic mean of the z-scores for GPA, PCAT, RATE and MMI was calculated and used to rank the interviewed applicants. This ranking was used as a check during admissions meetings and as a way to determine the order in which applications were discussed. Data were obtained from a total of 235 applicants who were interviewed in 2015 and 207 applicants who were interviewed in 2016. Data were analyzed by logistic regression. Exempt status was approved by the IRB.
Results: All four admissions variables showed significant overlap in relation to offers of admission along with discrimination between applicants receiving offers and those not receiving offers. There was less overlap for the MMI scores between applicants receiving offers and those not receiving offers. Logistic regression demonstrated that GPA, PCAT, RATE and MMI were all important predictors of admission.
Conclusions: Together, the results suggest that a 2-step admissions process consisting of a step 1 using cognitive and non-cognitive admissions measures refined by a step 2 with more advanced non-cognitive measures yields a balance of predictors to be used by admissions committees.
Pharmacy admissions; GPA; PCAT; Multiple miniinterview (MMI); Professional development ratings; Success predictors
It is vital to society to have pharmacists who excel in their practice. Also, the cost and effort required to train Pharm.D. students have increased substantially as the role of the profession in medication therapy management has become more widely appreciated. As with other health professions, Pharm.D. programs have moved more to holistic admissions evaluations in an attempt to capture both cognitive and non-cognitive skills necessary for professional success. Elsewhere, we have reviewed the manner in which health care professions including pharmacy have attempted to meet the goal of admitting students who will be successful practitioners .
It is well recognized that traditional academic markers such as GPA and achievement test scores can predict success in the classroom, but not in experiential training or clinical practice [2-4]. In contrast, systematic ratings of professional development and innovative interview methods such as the “admissions OSCE” (Objective Structured Clinical Exams)- Multiple Mini-Interview (MMI) do seem well suited to predict performance in experiential training . However, there is a marked disparity in the difficulty in obtaining and evaluating cognitive measures such as GPA and PCAT–readily available from the application and the MMI, which must be implemented on site to individual applicants. Thus, 2-step admissions processes, consisting of an initial screening using information gleaned from the application (step 1) followed by a more in depth, on-site interview process (step 2) .
At our institution, we have utilized holistic admissions criteria for a number of years, using such a 2-step process. Initially, applicants are screened for GPA and PCAT scores and rated on professional development using a 9-item scale (RATE). Subsequently, those applicants considered to be a potentially good fit were historically invited for standard interviews or, most recently, for multiple-mini-interviews (MMI). The present retrospective analysis determined the contribution of cognitive and non-cognitive measures to the eventual decision to admit the student to the Pharm.D. program.
As highlighted above, the admissions committee at our institution has used a 2-step admissions procedure for several years as part of a holistic review. Initially, pre-requisite GPA and PCAT scores are used for the cognitive measures. Non-cognitive information is gleaned from the full application, including letters of recommendation and summarized (RATE) via a 9-item scale with equal ratings for experience in pharmacy practice, teaching, research, student organizations, community service, overcoming hardship, essay quality, plus information from two letters of recommendation (Table 1). GPA, composite PCAT score and RATE are converted to Z-scores and the arithmetic mean of GPA, PCAT and RATE calculated-thus weighting cognitive measures twice as heavily as non-cognitive measures. This metric provides a guide to reviewers as they determine from examination of the complete application those individuals who will be invited for interviews. Following the interview process, the MMI score is determined and its Z-score calculated. Then, the average for GPA, PCAT, RATE and MMI is determined-weighing cognitive and non-cognitive measures equally and presented to the admissions committee for a holistic evaluation.
|Field Name||Field Info*||Scoring Range|
|PHR||Pharmacy work experience||1 to 5|
|ORG||Organizational experience||1 to 5|
|SVC||Service and volunteer experience||1 to 5|
|TEACH||Teaching, tutoring, mentoring experience||1 to 5|
|RESEARCH||Research experience||1 to 5|
|OVERCOMING HARDSHIP||Overcoming hardship; these include but are not limited to the following: single parent, socioeconomic status of family, first generation attending college, overcoming adversity, resident of an underserved area of the state or an area of Texas with a health professions shortage, race and ethnicity, and cultural background.||1 to 5|
|LOR1||Rate letter of recommendation #1||1 to 5|
|LOR2||Rate letter of recommendation #2||1 to 5|
|ESSAY||Rate essay quality including “Why Pharmacy?”||1 to 5|
|All members of Pharm D Admissions Committee need to be able to enter data into the above fields.|
|*For each of the field shown below please provide a single score in the range of 1-5 [1 min, 5 max]. Both quantity and quality of the experience should contribute to the score. Thus, someone who has worked 2 years as a CPhT and exhibited strong leadership by training other techs should receive a higher score than someone who had merely worked for 4 years. 1=bottom 20% of applicants, 2=21-40% of applicants, 3=41-60% of applicants, 4=61-80% of applicants, 5=top 81-99% of applicants.|
Table 1: Professional development rating (Rate)
Following final review by the admissions committee, offers in 2015 were extended to 150 applicants, a position on the wait list to 36 applicants, and deny to 49 applicants. Offers in 2016 were extended to 129 applicants, a position on the wait list to 28 applicants, and deny to 50 applicants.
Since 2015, the College has used for the interview process the MMI (8 stations with scenarios). The scenarios represent attributes selected by the College faculty members as being important to success in a Pharm.D. program and in pharmacy practice. The attributes are: 1) problem solving/critical thinking/ intellectual capacity; 2) compassion/commitment to care; 3) respect towards others; 4) integrity/ethical reasoning; 5) communication skills; 6) organizational efficiency; 7) relationship-building skills; 8) motivation to become a pharmacist; and 9) leadership skills.
Each scenario was scored for 3 or more attributes and all attributes were represented in the set of scenarios, some several times. Attributes within scenarios were scored from 1–10 via a rating scale with a grading rubric and labels for scores (1-2, poor; 3-4, fair; 5-6, good; 7-8, very good; 9-10, excellent). Three of the scenarios were interactive between an actor and the applicant or between two applicants. Although there were eight active scenarios, stations 7 and 8, involving model toy construction, were each scored by two evaluators, yielding a total of 10 evaluator scores. In the 2014-2015 cycle, the number of attributes assessed yielded a total MMI score of 300. In the 2015-2016 admissions cycle the number of attributes for the new set of station queries yielded a total MMI score of 330.
Scenarios were initially obtained from a set developed by ProFitHR and then adapted for the US health care system and pharmacy by a scenario implementation team that was part of the MMI task force. This task force met periodically for approximately three years to assist the College faculty members in evaluating Pharm D attributes and then assessing the background of the MMI literature, logistics, scenarios, resources, and data analysis. Scenarios used in 2015 were: 1) heart attack, 2) difficult patient, 3) beneficial activities for pharmacy students, 4) accidental death, x) rest station (not scored), 5) medical management team, 6) accidental vaccination (with actor), 7) model toy, construct according to instructions from another applicant, and 8) model toy, provide instructions for another applicant to make the toy. Stations 7 and 8 were each scored by two evaluators. Scenarios information for 2016 included: 1) reiki (healing touch), 2) delusions and treatment compliance, 3) pharmacy school advice, 4) conflict resolution, 5) toy received, 6) toy sender, 7) brain damage, 8) test preparation, x) rest station (not scored). Data were analyzed by logistic regression.
Descriptive Information: All admissions committees have the difficult task of making the optimal match between a set of applicants and the academic program to which these individuals apply. For competitive programs, there is an additional level of difficulty for the admissions committee, namely that the majority of the applicants are quite strong so that the matching process becomes, perhaps, even more of a challenge. Below, we summarize the information on cognitive (GPA and PCAT) plus non-cognitive (RATE and MMI) for the applicants in the 2015 and 2016 admissions cycles. Overall, these represented strong applicant pools with considerable overlap between those students who were given offers and those who did not receive an offer.
Figure 1 shows data on science GPA, PCAT composite percentile, professional development rating (RATE), and MMI score for the 2014-2015 and 2015-2016 admissions cycles. Tables 2 and 3 show admissions variable data for individual applicants in the 2014-2015 and 2015-2016 admission cycles respectively.
|Variable*||Decision||N||Mean||Std. Error||95% Confidence Interval for Mean|
|Lower Bound||Upper Bound|
|*SCIGPA=science GPA; COMPOS=PCAT composite percentile; RATE=professional development rating as average and with a max of 5; MMI=score on MMI with max=300; N=initial offer.|
Table 2: Admissions variables for 2014-2015 admissions cycle.
|Variable*||Decision||N||Mean||Std. Error||95% Confidence Interval for Mean|
|Lower Bound||Upper Bound|
|*SCIGPA=science GPA; COMPOS=PCAT composite percentile; RATE=professional development rating as average and with a max of 5; MMI=score on MMI with max=330; N=initial offer.|
Table 3: Admissions variables for the 2015-2016 admissions cycle.
Figure 1C and 1D: Panels C-D show the applicant data for the 2014-2015 v 2015-2016 admissions cycles as upper and lower portions of the panel, respectively. (C) Professional Development Rating scored from 1-5 with 5 maximum, (D) MMI score with a maximum of 300 for 2014-2015 cycle and 330 for 2015-2016 cycle.
It is evident that overall the applicant pools for both cycles were strong academically:
Science GPA was roughly 3.5 for applicants given offers and 3.2 for those not given offers;
PCAT composite percentile was approximately 85% for applicants given offers and 73% for those not given offers.
The tables also show that there was more separation between applicants given offers and those not given offers on the noncognitive measures:
Professional development rating (RATE) was 3.2 (out of 5) for applicants given offers and 2.6 for those not given offers.
MMI scores were higher in applicants given offers (210 out of 300 for 2014-2015 and 266 out of 330 for 2015-2016) versus applicants not given offers (171 out of 300 for 2014-2015 and 231 out of 330 for 2015-2016).
The major focus of the present paper was to determine the extent to which adding a non-cognitive admissions measure (MMI) as a second step in a 2-step admissions process improved the prediction of admissions decisions. Here, step-1 consisted of a combination of 2 cognitive measures (science GPA and the PCAT composite percentile score) plus 1 non-cognitive measure (RATE, representing an arithmetic average of scores in 9-areas of professional development).
Tables 4 and 5 present the results of the logistic regression analyses for the 2014-2015 and 2015-2016 admission cycles, respectively. Crucial to the interpretation of logistic regression is the value of the odds ratio, which is the exponential of B (exp(B)). This is an indicator of the change in odds resulting from a unit change in the predictor . If the value is greater than 1, then it indicates that as the predictor increases, the odds of the outcome occurring increase. Conversely, a value less than 1 indicates that as the predictor increases, the odds of the outcome occurring decrease.
|B||S.E.||Wald||df||Sig.||Exp(B)||95% C.I. for EXP(B)|
|*Regression was conducted on z-scores for science GPA (SCIGPA), PCAT composite percentile(COMPOS),professional development rating (RATE, maximum 5), and MMI score (Maximum 300). Offer=decision 1 while deny=decision 0.|
Table 4: 2014-2015 Admissions Cycle. Logistic Regression Analysis*.
|B||S.E.||Wald||df||Sig.||Exp (B)||95% C.I. for EXP (B)|
|*Regression was conducted on z-scores for science GPA (SCIGPA), PCAT composite percentile (COMPOS),professional development rating (RATE, maximum 5), and MMI score (Maximum 330). Offer=decision 1 while deny=decision 0.|
Table 5: 2015-2016 Admissions Cycle. Logistic Regression Analysis*.
For applicants in the 2014-2015 admissions cycle, unit increases in professional development rating (RATE) yielded a 5.99-fold increase in the likelihood of being given an offer and for MMI score, a 13.26-fold increase. For applicants in the 2015-2016 admissions cycle, unit increases in professional development rating (RATE) yielded a 1.61-fold increase (not significant) in the likelihood of being given an offer and for MMI score, a 13.23-fold increase. Thus, both cognitive and noncognitive measures contributed significantly to the likelihood of an applicant being given an offer. However, it should be noted that all predictors were significant in the 2014-2015 admissions cycle, but science GPA and professional development ratings were not significant in the 2015-2016 admissions cycle.
From its inception in 2004 to the present, the MMI has been recognized as a tool with great potential for evaluating non-cognitive attributes in health professions [7,8]. This metric has been used to predict performance of applicants in a variety of professional programs [9-28]. The MMI has been shown to be cost-effective , flexible in terms of format [16-18,28-35] and independent of personality and emotional intelligence measures [36-39]. The present results have extended this work to a systematic evaluation of the relationship among cognitive (GPA and PCAT) and non-cognitive (professional development ratings, RATE and MMI) to predict admission and to discern the interrelationships among the predictor variables for a 2-step admissions process.
For the two admissions cycles studied, we found that professional development rating (RATE) and MMI score contributed significantly to the likelihood of the applicant receiving an offer. For the 2014-2015 cycle, science GPA and PCAT composite percentile score also contributed meaningfully to holistically derived offers. We speculate that this may be due to the fact that the 2014-2015 applicant pool was particularly strong in both GPA (3.54 for offer) and PCAT (87.4% for offer), whereas the 2015-2016 applicant pool was slightly less strong in these cognitive measures (3.41 GPA and 84.5% for offers, respectively). In any case, the substantial overlap between sets of applicants receiving and not receiving offers but with invitations to interview, would tend to minimize the statistical significance of these two variables.
The incorporation of the MMI into a 2-step admissions process (initial screen via GPA, PCAT and RATE) followed by interview (MMI) was highly effective for both admission cycles studied. These findings are consistent with and extend the current literature [9-12]. Furthermore, the MMI was well behaved in terms of consistency across evaluators and stations (scenarios), suggesting that the MMI was a valid measure in this study. These results are also consonant with findings in the field [11-40]. Our results are consistent with, and extend the current literature on the use of the MMI as a predictor of admission into health professions programs [5,7-9,12-14,20-43]. Significantly, our results demonstrate the potential value of professional development ratings (RATE) as important “pre-screening” tools for interview invitations using the MMI. As expected, based on the strong applicant ratings of professional development (RATE) in step 1 of the admissions process, most of the applicants performed well on the MMI. In fact, the vast majority of the applicants performed consistently on the MMI with a very few performing in an especially robust manner and a few performing poorly.
Our analyses demonstrated that there are significant differences in each of the admissions variables of GPA, PCAT, RATE and MMI between the offer-status groups. The fact that all four admission-predictor variables (GPA, PCAT, RATE and MMI) contributed significantly to an offer of admission is consistent with the hypothesis that academic and professional development measurements are useful in informing the admissions committee. In particular, our findings with both RATE and MMI extend the published literature via the explicit incorporation of professional development ratings and the demonstration of the relationship between RATE and MMI score [5-7,14]. Our results illustrate the importance of the possible independent predictive value of professional development assessments and their value as a screening tool for invitations to the MMI. Thus, our results appear consistent with the literature that the MMI assesses a different dimension than do the didactic admission measures but extend the literature in showing the overlap of professional development ratings with other measures [24,25,33,44].
The authors gratefully acknowledge the extraordinary efforts of the Pharm.D. Student Interview Team Chairs: Angelica Cuenco, Isaac Perales, Lillian Tran, & Victoria Vaughn in making the MMI possible.
All Published work is licensed under a Creative Commons Attribution 4.0 International License
Copyright © 2019 All rights reserved. iMedPub LTD Last revised : February 19, 2019