Attributes That Matter: Beyond the Usual in College Admission and Success
January 16-18, 2013
Westin Bonaventure, 404 South Figueroa Street, Los Angeles, CA
For the Conference blog and photos, click on the dates below.
Keynote Address: Assessing Non-cognitive Variables: Issues and Applications
Dr. William Sedlacek, Emeritus Professor of Education at the University of Maryland, College Park
Dr. Sedlacek began the morning with an examination of noncognitive variables. He described some of the measurement reasons why we need to look at something else. Specifially, he suggests that we have the statistical problem of restriction of range. We don’t really know how to measure attributes any better. We also have the issue of grade inflation. Some places have almost everyone getting 4.0s on a 4-point system. Even taking more courses in a certain area doesn’t reveal new information- at a certain point, you just top out.
We need to examine a range of things that can be considered. Dr. Sedlacek referenced Sternberg’s work on Intelligence types. Sternberg suggest there re three types of intelligences: Componential- refers to the interpreting information in a well defined and unchanging context (standardized tests), experiential- interpret information in changing contexts (not able to measure with standardized tests), and contextual (ability to adapt to a changing environment and negotiate the system, again, not measured by standardized tests.
Sedlacek decribed additional noncognitive variables, including Self-Concept, Realistic Self-Appraisal, and Handling System, suggesting that these can be measured and developed. Self-Concept means that you think well about yourself and see yourself in higher education. If you don’t have this self-concept, then you will struggle in the higher education environment. Realistic Self-Appraisal suggests that you understand your strengths and weaknesses. A good program lets a program express both what they can and can’t do, this can be measured, and then you can support the areas that need development. Finally, Sedlacek elaborated on the notion of Handling the System/Racism. This suggests that the person can work the system and take an assertive approach to dealing with existing wrongs. You can see that if you were to try to design a test, it would not be practical to try to assess these elements.
Other dimensions include Leadership (in both traditional and non-traditional groups), Long-Range Goals (the longer the better, but it is contextually based- long-range to one may look different to another, but the point is having a plan), Strong Support Person (someone who pushes the student back toward the system), Community (loners have more difficulty), and Non-Traditional Learning (what have you learned from your experiences outside of the academy?).
The advantages of noncognitive variables include: it is a research based approach, there are multiple ways to assess, it is retention related, considers diversity, tested legally, revised to fit situation, no cost, student development, community building, and supported by high school counselors.
Dr. Sedlacek understands the difficulty of taking a new approach, but is open to the challenge and criticism. He quoted Elbert Hubert, “To avoid criticism, say nothing, do nothing, be nothing” and suggested that the criticism can be countered and the ideas should be discussed. He also described several key legal cases that support the use of noncognitive variables and highlighted the recognition of Oregon State as an exemplary program as evidence of support for the approach. After providing resources and references, Dr. Sedlacek left the participants with a challenge, "What don't you try this?" and cast an invitation to all to implement these measurements as a step toward societal change.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Morning Session 1- Non-cognitive Variables in Action
Jon Boeckenstedt, Associate VP of Enrollment Management and Marketing, DePaul University
Noah Buckley, Director of Admissions, Oregon State University
Noah Buckley began the session by describing how Oregon State has implemented noncognitive variables in their process and the related research and results. Their tool, the “Insight Resume” was developed with the support of Dr. Sedlacek. It gives the school a way to consider the noncognitive skills that relate to college success in the admissions process. The university launched the process in 2004. The tool includes 6 questions, that align with Sedlacek’s work and assess a students experience and competency in: leadership/group contributions, knowledge in a field/creativity, dealing with adversity, community service, handling systemic change/discrimination, and goals/task commitment. The questions have been the same since the inception of the program.
Each question is scored from 1(low) to 3 (high) by an Insight Resume reader and is blind of all student information. Scores and responses are used to inform some admissions decisions, general scholarship selection process, and at times, “red flag” responses are addressed.
A 2009 analysis by the university aimed to measure the correlation of the Insight Resume scores to retention and graduation and to predict such outcomes. They examined the 2004-2007 cohorts with gender, ethnicity, and Pell status as variables. The results demonstrated that the 2004 cohort had higher odds of retention and graduation with higher IR scores. The 2005 cohort results demonstrated that African Americans and Native Americans showed increased odds of retention with increased IR scores. The 2006 and 2007 cohorts showed no evidence of a relationship between IR scores and graduation and retention. Thus, the conclusion was “In no case does the inclusion of IR scores offer substantive improvements in predications.” GPA was actually a better predictor.
Buckley then moved the focus to today’s applicants and how they are scored and how the results are used in practice. The admissions excepts are scored twice with the goal of finding “the diamond in the rough”. High achieving students who are eligible for scholarships are also scored twice and that is then fed into an equation whereby students are selected for scholarship. Everyone else is scored once. These “red flag” students have demonstrated that they do not have the views and perspectives that would make them a fit on campus. The university has denied admission based on responses to these questions.
In the future, the university realizes they must ask themselves if they should continue to use the IR as a filter. They see the opportunity to use the identification of non-cognitive deficiencies as an opportunity to provide support to build competency through courses for students who are low in various components of the assessment. Other opportunities to support students include early alert monitoring (providing a support system), advisor flagging and training, intentional linking to existing programs and services and new courses to support the development of deficient areas.
Next, Jon Boeckenstedt, presented DePaul University’s use of noncognitive variables. He highlighted the title of the conference and emphasized the notion of “Beyond the Usual”, noting that understanding the context of what we can and can’t predict is important to understand. He asked, “What really is the nature of college admissions?” and described how, over the years, it has been considered a Gate-Keeper, Labor Market Sorter, a “cog in one of the greatest inequality producing machines this country has known” (Brooks). Today, Boeckenstedt suggests, we aim to consider college admissions as a process to identify students who have the potential to be successful on a campus. Currently, however, grade point average and test scores are the measures we use.
Boeckenstedt quoted Einstein (below) and noted that there are just such a wide range of factors that contribute to student success, that it is very difficult to develop a formulas.
“When the number of factors coming into play in a phenomenological complex is too large scientific method in most cases fails. One need only think of the weather, in which case the prediction even for a few days ahead is impossible.”
From there, he moved the focus to the practices in play at DePaul. He described the university as the most mission-driven university he has worked with, highlighting demographics that demonstrate the undergraduate diversity. He notes, however, that their movement toward noncognitive variables was not to develop diversity, but rather a reaction to their previous use of used untested and unproven measures of admissions. They realized that some things meant even more in terms of student success than class rank, GPA, and test scores.
The DIAMOND project at DePaul was developed to explore and understand the importance of non-cognitive variables in student success and retention, and extract the characteristics deemed important. Boeckenstedt noted, however, that even if they had a “perfect predictor” of year 1 GPA, the persistence beyond the second year couldn’t be accurately predicted.
The DIAMOND project included many of the same types of questions as the University of Oregon and was scored in the same way. The results so far demonstrate that the DIAMOND scores are less correlated with income and ethnicity- which is a good thing. They are also not correlated with the ACT. Some students show low ACT scores and high DIAMOND scores- these are the students that will experience the biggest bump in the process. Students with higher than average DIAMOND scores have greater retention whether the ACT is above or below average- so there is some promise in the approach, but they still find that high school GPA is still the most significant predictor.
In conclusion, Boeckenstedt emphasized that there is no silver bullet when explaining and predicting human behavior. It is possible to believe the in the power of the variable while realizing that the instruments to measure it may not be as robust as we like.
~~~~~~~~~~~~~~~~~~~~~
Morning Session 2- Attributes of Good Students and Good Professionals
Moderator: Patrick Kyllonen, Senior Research Director of the Center for Academic and Workforce Readiness and Success at Educational Testing Service (ETS) in Princeton, NJ.
Panelists:
Steve Kappler is the Assistant Vice President and Head of Postsecondary Strategy for Career and College Readiness at ACT
Sheldon Zedeck is Professor of the Graduate School in the Department of Psychology at the University of California at Berkeley
Kyllonen launched the session by framing the key concepts to consider during the panel discussion. He highlighted the title of the conference, Attributes that Matter, and noted that this title begs the question, “What does matter?” To begin to answer this question, Kyllonen pointed to a recent study, Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. This study pointed to three key areas that are needed to be successful: Cognitive, Interpersonal, and Intrapersonal skills. Kyllonen identified the uses, challenges, and approaches related to assessing non-cognitive variables. More specifically, he outlined:
Uses: college admissions, college success, work success
Challenges; noncognitive assessment, “fakability”/coaching, cross-cultural comparability, test-taking motivation, applicant reaction
Approaches: Self-ratings, others’ ratings, situational judgment tests, forced-choice assessments, behavioral interviews, an actionable feedback.
With these in mind, the first panelist, Steve Kappler shared perspectives related the ACT scores, suggesting that there is much more information that can be gained from the results than a simple number. According to Kappler, the scores can actually tell you what a student can do and they can tell a student what they need to work on. He emphasized three domains of college and career readiness that ACT targets for identification and intervention opportunities: Academic Readiness, Academic Behavioral Readiness, and Career and Educational Planning.
From an academic readiness perspective, ACT surveys universities to determine what students need to know and be able to do to be successful in their institutions and surveys high schools to understand how their preparation or curriculum aligns. Kappler noted that excessive grade inflation is an obstacle, noting that 25% of students self-reported in 2012 that they had a GPA of 3.75 or above. This self-reporting is in conflict with the bell curve that ACT scores indicate. Thus, assessments can be used to provide more accurate data. Using the state of Illinois as an example, Kappler pointed to the relationship between ACT benchmarks and student enrollment, re-enrollment, and non-enrollment in college and the predictive validity of ACT composite scores in terms of degree completion within 6 years. He emphasized that there is quite a bit of information in the ACT results that can be used for predictive purposes.
In terms of career and educational planning, Kappler described ACT measurements of expressed vs. measured interest, and academic ability as compared with interests. These are of particular importance, as they point to intervention opportunities, as students can bolster weak areas if they have an interested in a certain area but do not have the academic background to support that path.
Kappler next described Academic Behavior Domains: Motivation, Social Engagement, and Self-Regulation. An analysis of student data found that the more motivated and engaged students were, the higher the GPA. Lower self-regulation related to lower GPA. And from a social engagement perspective, the fewer extracurricular activities a student participated in during high school, the more likely they would drop out.
The next panelist, Sheldon Zedeck, described the research at UC Berkeley on predicting lawyer effectiveness with non-cognitive measures and the use in such measures in law admissions decisions. His talk contrasted the predictive ability of tools that are used for admission to law school as compared to their ability to predict success as a lawyer. Berkeley was very interested in identifying lawyer on-the-job effectiveness factors and developing evaluation scales to predict such effectiveness. To reach these goals, they conducted a “job analysis” of lawyering and effectiveness dimensions. Next, they identified other predictors, non-cognitive variables, to explain lawyering effectiveness. Their process included hypothesizing predictors, developing/selecting tests, administering the tests, collecting performance evaluations, and then establishing test-performance statistical relationships. The predictors included UGPA, LSAT, and INDEX.
Zedeck next described the results of their examination. 26 effectiveness factors were identified across the practice of law. They were curious to know if the LSAT could predict effectiveness as a lawyer and thus developed performance evaluation scales for each of the factors. The 26 factors were organized into 8 categories including: intellectual and cognitive, research and information gathering, communications, planning and organizing, conflict resolution, client and business relationships- entrepreneurship, working with others (other attorneys), and character (passion and engagement, diligence, etc.). The scales became the criterion by which they analyzed the predictive capabilities of the LSAT.
As they developed new predictors, they looked at Snyder’s self-monitoring assessment, carver’s optimism scales, emotional intelligence and a biographical inventory (past experiences) approach, accomplishment and experience records, situational judgment, and a moral responsibility scale. Zedeck continued to describe the many evaluation tools they used to examine the 26 factors, including those already developed and some that were developed in-house.
After identifying the many tools, they began their validity study with 1100 respondents. They found that their new suite of tests did not do better than LSAT in predicting law school performance and there were interesting findings in terms of their ability to predict effectiveness on the job. The LSAT predicted 12 of the 26 dimensions, but 8 were negative. The INDEX predicted 9 of the 26 performance dimensions and 4 were negative. Thus, for some domains, the better you did on the LSAT, the less effective you would be in the workforce in that specific domain. Zedeck found that their personality assessments showed less differences between minorities and Caucasians- and thus the use of these tools supports the identification of a more diverse student body.
Zedeck suggests that this type of process can also be done with undergrads. He did not promote abandoning the cognitive assessments, but to use them within the battery of tools used to identify students with potential for success.
Master Class - Beyond Grutter, Gratz, and Fisher: Legal and Educational Implications of Considerations of Race and Other Options in College Admission
Art Coleman, Managing partner and co-founder of Education Counsel LLC.
In his presentation, Coleman returned to a central issue associated with admissions, merit, and the policy and legal implications when race or ethnicity enter the picture.
Effective Policy Development
Coleman began with an overview of effective policy development, noting key spheres of influence on policy development and implementation. These spheres include stakeholder support, research and experience, and law. According to Coleman, one of the myths that is frequently out there is that we have a great disconnect between what law demands and good education requires. He suggests that we should be looking at the question of one fundamental alignment, noting that a university can have the best, legally-based, articulate, policy ever, but without stakeholder support, it will not work. This is not about taking care of the legal questions or figuring out the research foundations- but rather we must be transparent about what is going on behind closed doors in the admissions office. Additionally, he warned that when there are volatile issues, such as race and ethnicity, you must have a compelling story to tell. It is important to pay attention to the laws, but you must also pay attention to the court of public opinion, or the “stakeholder” sphere of influence, and be able to communicate your story in a way that can be understood.
Educational Benefits of Diversity
The “benefits of educational diversity” perspective is primarily the frame by which universities ground their discussions. From the College Board to Medical colleges, etc., it has been suggested that racial diversity is essential for us to succeed. The theory of action recognizes that a diverse learning environment, with an appropriate pedagogical focus, leads to enhanced teaching and learning, and improved outcomes. Students in such environments are better prepared for productive lives in the workplace and society. Coleman suggested that “educationally sound and legally defensible race/ethnicity-conscious practices are the product of a well-designed, institutionally-aligned and integrated process that connects means to ends.”
This is the theory that the University of Michigan, for example, was framed on. This is the framework on which one would identify goals and objectives, and strategies to meet each of those. If you cannot gage the success toward you objectives and goals, you are not in a legally defensible position. This theory is the way we tend to talk about admissions among legal, admissions, and institutional leaders. Coleman suggests, however, that we are missing the central issue, “What is merit?”
Missing in Action: Merit
According to Coleman, we tend to stay within our comfort zone and talk about process. Although there is nothing wrong with that, those who challenge the lawfulness of a university’s approaches tend to focus not on the process but the substance- what are you considering? A simple story: test scores, plus grades, equals merit, can be easily tackled. If it can be proven that someone with lower scores and grades was admitted above someone with higher grades and scores, the university is in trouble. Coleman noted that he finds it remarkable how we are not capturing the intelligence and power behind telling the story. The question of merit, who is admitted, and the composition of the class, is not that simple. The question of merit must be tied to what the university needs, the mission, and what will contribute to the overall needs and experience of the institution.
Coleman then reviewed cases from the perspective of admissions polices and legal outcomes:
1978 Regents of UC vs. Bakke. UC defined merit as anyone over a 2.5, stating clearly that “we will interview no one with a GPA below 2.5”. UC admitted a set-aside group of 16 out of 100 minority students who had GPAs that went as low as 2.11. The university in this case did not stand by the very theory of what merit was for them in practice.
Grutter v. Bollinger (2003). The admissions process at the University of Michigan includes: 1) individual review of all applications, with a focus on academic criteria, 2) likely contribution to intellectual/social life of the institution, and 3) contribution to diversity which can enrich the education of all. The court upheld the admissions policy.
Fisher v. University of Texas (2013). The UT-Austin admissions policy includes a 90% allotment of seats to Texas residents. 81% of entering class was admitted under the top 10% law. The remaining Texas residents compete for admission based on academic and person achievement indices (PAI). Under PAI, SES, Family status, standardized test score as compared to the high school average, and race can be considered.
The implications for action, according to the AAMC, suggest that authentic holistic review should be multi-faceted, involving many student qualities and characteristics (that may include race and ethnicity). So how do we communicate the multiple strands that are part of the admissions process? Coleman suggests that the AAMC model is an effective start.
Coleman stated if there is one key take-away, the one that is as profound as any to think about in conversations about race and ethnicity, is that race is more than just the “check the box”. Coleman suggested that we reflect on the consequences of a court stating that race and ethnicity can never be considered- what does that mean? How is it possible to be blind to race? Coleman concluded that we must be more sophisticated about how we think and talk about race and ethnicity.
Key Questions (summarized):
Fisher Overview
Coleman transitioned into an overview of the fisher case with two key questions of relevance: Necessity? And What’s the Impact? The court basically said that if you are using race as a factor of admissions to try to get to some goal, you must show demonstrative evidence that the consideration of race helps you achieve your goal. In the case of Michigan, they succeeded in demonstrating the need for consideration of race. In the Fisher case, Texas suggests that the modest consideration of race does have an impact on the university that is supported by evidence. The plaintiff’s position was that the policies did not have enough of a substantial change in terms of the campus to justify that consideration of race- no matter how limited it is in the equation. The question, then, is what is critical mass? How does that vary based on location? Texas had 20% of minority students, suggesting that race is not necessary. Texas, however, suggests that because they have not achieved critical mass at the classroom level, they have not met their goal.
With this in mind, we wonder what will the future hold? And what it should hold?
~~~~~~~~~~~~~~~~~~~~~~~~
Keynote Dinner Address
David Coleman, President and CEO of the College Board
Coleman launched his discussion with an explanation of his decision to participate in the CERPP annual conference, noting that it is the last long distance trip that he will take in the next 6 months. He selected this conference to participate because of Dr. Jerry Lucido’s work in helping traditionally underserved students in accessing college. Coleman acknowledged Dr. Lucido’s passion for the notion that dreams must be preserved and described Lucido as someone who continues to find opportunities to preserve the American dream. Thus, he was motivated to join Lucido’s efforts and participate by delivering the dinner address at the conference.
After a standing ovation for Jerry’s work, Coleman shifted his presentation toward his interest with admissions officers. He highlighted their complex work that involves real decisions and noted that they are truly educators. The moment a student engages in college is life-hanging, and thus, the consequences of the work of admissions officers is monumental.
In regards to his role as president of the College Board, Coleman defined recommitting the organization to the ideals of equity and access as one of his primary goals. Additionally, he stated that the mission of the College Board can and must change- it is not enough to connect kids to college, they must complete.
Noting Duckworth’s work on the idea of grit and self-control and Ericsson’s work on deliberate practice, Coleman described himself as interested in the practices that make us excellent. He suggested that people tend to hide their practice, so it looks easy, but most things are by design. He also noted that the ability to work alone is also an important factor- you must be able to work alone to have deliberate practice and you must be able to revise your work.
Coleman moved his focus to testing instruments, including the SAT and ACT, remarking that they do not necessarily add predictive value. There are several hypotheses to be considered:
Our shared challenge is to get more verifiable and consistent data. We must ask:
What are the kinds of firmer non-cognitive data or patterns that we can collect?
Are they resilient? Did a student receive a D and then rally and ultimately do well. Why?
The College Board will invest resources in the next few years in a search for kids who can go to college, but don’t. Each year 40,000 students do well (top quartile on the SATs), but do not go to college- a notion that Coleman finds unconscionable. Equally upsetting are the lowest income kids who, at the moment of college application, do not apply. Coleman holds the College Board accountable for shoring this up. He said it is our shared work to find the kids who are forgotten.
Coleman described the College Board as having a deep desire to talk to institutions and educators to learn how we can work together. He said he wants to be in a conversation with institutions to create an exam that is what institutions need and asks for. CB will improve, not hurt, differentiation at the high ends. They will get data faster than they did before. In partnership, the College Board and educational institutions can work toward setting the conditions for college opportunity and success for all students.