Politics, Technology, and Language

If thought corrupts language, language can also corrupt thought — George Orwell

Lies, damn lies, and college rankings

Posted by metaphorical on 13 March 2007

“In other words, in the absence of real data, they will make up a number.”

We know there’s lies, damn lies, and statistics. Sarah Lawrence College is tired of the lies, and they’re not going to participate any more. Naturally, they’re going to be punished. It just stands to reason.

In this scenario, the lying statistics are the SAT scores used by the U.S. News & World Report college rankings. It seems that more and more colleges aren’t using SAT scores in their determinations of which applicants to accept. But in order to not upset their U.S. News ranking, they are making SATs “optional.”

The result is a statistical advantage of the “damn lies” sort. As Inside Higher Ed puts it,

When applicants learn that a college is SAT-optional, it doesn’t take an 800 math score to figure out the statistically wise strategy. If your scores are at or above reported averages, submit them. Otherwise, don’t. Not surprisingly then, many colleges that go SAT-optional experience both a surge in applications and an increase in their SAT averages … and their U.S. News rankings go up.

Sarah Lawrence decided it didn’t want any part of that. In an op-ed piece in Sunday’s Washington Post, the school’s president, Michele Tolela Myers, explained:

Two years ago, we at Sarah Lawrence College decided to stop using SAT scores in our admission process. We didn’t make them optional, as some schools do. We simply told our prospective students not to bother sending them. We determined that the best predictors of success at Sarah Lawrence are high school grades in rigorous college-prep courses, teachers’ recommendations and extensive writing samples. We are a writing-intensive school, and the information produced by SAT scores added little to our ability to predict how a student would do at our college; it did, however, do much to bias admission in favor of those who could afford expensive coaching sessions.

The school is doing just as well as it ever did; incoming students measure up just fine against earlier ones. “Their average high school grades, high school ranks and grades in Advanced Placement courses have not changed,” Myers wrote.

But this principled decision has put us in jeopardy. I was recently informed by the director of data research at U.S. News, the person at the magazine who has a lot to say about how the rankings are computed, that absent students’ SAT scores, the magazine will calculate the college’s ranking by assuming an arbitrary average SAT score of one standard deviation (roughly 200 points) below the average score of our peer group.

In other words, in the absence of real data, they will make up a number. He made clear to me that he believes that schools that do not use SAT scores in their admission process are admitting less capable students and therefore should lose points on their selectivity index. Our experience, of course, tells us otherwise.

One option is to just stop playing the U.S. News game entirely. Sarah Lawrence has considered that. But there’s a stiff price to be paid. Myers says,

several faculty members and deans suggested that perhaps it was time to stop playing ranking roulette and opt out of the survey. A few colleges explore this option each year, but most don’t follow through (Reed College is one of the few exceptions I know of), because, like unilateral disarmament, unilateral withdrawal from the U.S. News ranking system is dangerous.

I don’t know if you’ve seen USN&WR lately. It’s as skinny, compared to its heyday, as the abbreviation is compared to spelling out its whole name. The college ranking gig is one of the few things it has going for itself. According to a Folio article back in 1993, in 1992, “the ad-page battle of the newsweeklies, an annual contest between Time and Newsweek, has yielded a surprise winner – U.S. News & World Report.” By 2004, it was “the struggling bottom player among the big traditional news weeklies.”

I should mention that my alma mater, SUNY’s College at Geneseo, is a winner at the U.S. News game of statistical roulette. Over the course of the 1980s, Geneseo steadily rose so that now it consistently ranks among the top mid-sized colleges, private or public. It happens to have done it the hard way, by creating a tough, humanities and science based core curriculum that was a a magnet that attracted serious, smart students and repelled those who are not.

But no matter how well U.S. News has done by Geneseo, you have to admire Sarah Lawrence for preferring no stats to lying stats. Naturally, no honest deed goes unpunished. Hopefully, though, U.S. News will get theirs, as word spreads that its numbers are thin as its ad pages.

13 Responses to “Lies, damn lies, and college rankings”

  1. Blue Athena said

    There was a splash in the news back when Reed pulled out — and all the fake calculations and lies in the rankings were pretty well exposed then. But the awareness fades fast and every news media in the country still picks up these figures and repeats them. The result is colleges that will do anything to manipulate their place in the rankings — at a hefty cost to real education.

    Good for Sarah Lawrence. I’m not too sure about their decision to ignore SATs, but any part screws with USN&WR and their fake data is good in my book.

  2. Athena, the points Myers made about the SATs – not writing oriented (even with the writing test), and measures students disparately because of the prep services, seem valid concerns. What’s the case in their favor?

  3. Blue Athena said

    Similar concerns can be raised with grades. They vary based on the level of grade inflation and the difficulty of the courses–not remotely standardized in the US. Rankings have the same problems, being manipulated by the type of courses taken and the method used for assigning grades to different difficulty levels.

    Further, heavy reliance on grades is one of the prime forces behind rampant grade inflation….something that has made grades ever more meaningless.

    As an interesting note, Reed relied more heavily on standardized tests and examples of specialized work, and found grades a very bad predictor of the kind of success they were looking for.

    I don’t think either is “right”. A combination of factors is likely a better measure. And perhaps having a variety of schools which all look at different factors is what we want in a diverse society. Something that cannot be measured and scaled by a simplistic system like the USN&WR.

  4. Blue Athena said

    While there is some validity to the desire to avoid total reliance on SATs (they do not tell us everything by any means) I think the backlash is another example of the how-soon-we-forget syndrome. The tests were actually introduced to counter discrimination. They were meant to replace a system that judged one based on parentage, money and the known (and very real) difference in sending school quality. They were supposed to level the playing field.

    And these tests weren’t perfect — and in many cases introduced new inaccuracies and discriminations. But to pretend that there was or is a perfect world in which standardized testing doesn’t play a part is fantasy.

    And do we actually have anyone but a Sarah Lawrence official to say that the students really do as well now? Do we have comparisons of graduate school admissions by field? Do we have the GRE scores they will be forced to take if they want to attend most graduate schools? Do we have any measurable gauge of success other than their own grading system? Keep in mind that there are a number of other conceivable reasons for SL’s decision than the one they announce publicly. It’s not fair to go into these possible reasons without hard data or officials to defend their claims, but I have no trouble imagining them myself.

    Schools that make SATs optional like to say the students do just as well who didn’t submit scores, but then they go on to point out that those students steer towards different majors than those who did dubmit scores. Is it really that different fields require different standards, or are we just creating an underclass of academic fields that now contain fewer top students and are drawing a weaker set? Are we dumbing down certain fields by setting the standards lower?

    Let me reiterate that I do not think highest level SAT scores are the end-all measure. The very highest levels do not correlate to top success, and the same is true, as I’ve mentioned before, of very top level foreign service exam scores. Perhaps (to throw out a theory) there is some correlation to mild autism-spectrum disorders and very high scores. Or maybe it’s a weakness in an overly academic mindset. I don’t know. But this doesn’t negate the fact that there is a correlation with success at most levels, and that standardized tests offer some of the only protection against fraudulent grading and prejudicial decision making.

    I liken the anti-standardized test move to the “new” idea of community policing. This is being paraded all over the country as the latest idea, and isn’t it oh-so-wonderful. Have people really forgotten why we eliminated community policing? Are we now prepared to ensure the same mistakes don’t occur again? I hope so. And I hope it isn’t the same community policing now that we had in the days of wide-spread corruption, just as I hope the errors of the past won’t be reproduced in an attempt overthrow all standardized academic measures in order to achieve some mythical “right formula” for selecting university students.

  5. digglahhh said

    Wow, several things to mention here.

    First and foremost, most students (and especially parents) would be well served to pay substantially less mind to the college rankings than they do. The course material is largely the same (Plato is Plato) and you can find brilliant and inspiring professors everywhere. I’m very much convinced that what you pay for in the “elite” schools is really nothing more than greater networking opportunities.

    In the recent past I dealt with some unstable employment situations (high SAT scores don’t correlate to success, remember). One option I had was to teach for Kaplan or to freelance as a private SAT tutor. I explored those options a bit, but with ideological trepidation. Expensive one-on-one tutoring classes are, in my view, tantamount to sociological cheating. Damn if I’m going to help some investment banker’s son get into an Ivy at the cost of one fewer seat for the kid down my street. I could help the kid down the street too, even for free, but I was weighing this option as an employment opportunity.

    And I fully agree with Blue Athena, and I’ll take it one step further. Not only are grades largely meaningless. Undergraduate college degrees are too! The degree is the new diploma, at the expense of six-figure debt you can earn the security of not having to work at McDonalds, though maybe you’ll wind up earning 25K working an entry level job at their corporate offices.

    The manipulation of this data is only as relevant as the amount of weight you gave to the idea of ranking schools in the first place. Prospective students should be much more concerned with selecting a school that fits them as a person than the allure of some contrived scholastic ranking.

  6. Blue Athena said

    digglahhh, I agree with most of what you say. Those who rely on rankings over their own research end up at the wrong schools, and grades at undergraduate institutions have generally been inflated every bit as badly as high school grades. I do, however, disagree on the argument that a class is a class wherever you take it. Yes, there certainly are inspiring teachers everywhere. The worst professor I ever had was from MIT while one of the best was from a second tier state college.

    But the students are not the same at all schools, and that has a big impact on the classes. I took classes as an undergrad at 5 different institutions, where the students ranged from devoted to learning to total slackers. One college — at which I picked up a few classes after I graduated — was so pathetic in its student body that I considered it a serious waste of taxpayer money (and that was the school with my favorite professor).

    When you have lazy, unmotivated or weak students in classes the pace will be slower and the quantity learned will be less. You can still learn at such a place, and faculty will be very happy to have you there. But it isn’t the same.

    So, yes, by all means research and pick the best school for you or your children based on their needs, interests and skills. But keep in mind that they aren’t all the same, even if the differences aren’t realistically rankable on some simplistic and largely dishonest single scale.

  7. I was going to post something very similar to Athena’s last comment. Besides motivation, there’s diversity, of all sorts. Your average Iowa farmboy and -girl needs more than a bit of exposure to urban cultures, those of Europe and the developing world, and so on. And when I picked the New School (in New York) for my M.F.A., it was in part because there were a lot of students in their 30s and 40s and even some in their 50s. It makes a huge difference.

    In addition, Plato isn’t Plato. Even though great teachers can be found everywhere, they don’t teach great classes without some support from a strong department. There’s less point to having the best Plato teacher in a 3-state radius if he or she only teaches it once every few years. And he or she probably is going to get snatched up by a stronger more attractive school (and often the largest attraction is a good student body, so this point blends back into the first).

    When it comes to choosing schools, some people flourish living at home, many just keep hanging out with their high school friends. Some do well at a big Division I school, others get lost in the crowd. And so on.

    Speaking of which, schools correspondingly need a diversity of entry points. As a person who got into college almost entirely on the strength of his SATs, I’d be saddened to the point of despondency if no schools used them. I don’t think that’s what’s on the table here. Some schools will find them to be good predictors of academic success, others, such as Sarah Lawrence apparently, less so. I suspect that some schools might find the very top SAT scores to be a good predictor, but middling-good scores not to be, or to be a harbinger of success only in conjunction with other factors (e.g., good evaluations from alumni interviews).

    And whatever you think about SATs, the one unconscionable thing to do is to report only the better scores to U.S. News, which is what some schools are in effect doing. Sarah Lawrence is refusing to, and that’s an honorable thing, and that was the point of the post.

  8. Blue Athena said

    And a good point. The most honorable thing I can imagine is for no school to report anything to US News. It’s sad that so many schools have been so weak for so long on this front.

  9. digglahhh said

    Points taken.

    I guess I took for granted the diversity of my non-elite alma mater.

    I probably also took for granted, as I usually do in these types of discussions, my motivation to learn and educate myself independent of any formal schooling system. Basically, I’m less influenced by my surrounding than most others. But at the same time, many of the weak students and understaffed/funded departments are partly manifestations of turning higher education into a commodity. Because we bastardize noble, necessary, blue collar professions that are not reliant on high standards of scholastic aptitude, we see general dumbing down of higher education and a proliferation of disinterested and weak students in college classrooms.

    One more thing, several of the best professors I had at Queens College made a conscious decision to teach at a place like that. I know because I had discussions with them on this very subject. As somebody who is considering the professorial path myself, I would not want to teach at an “elite” institution, precisely because I seek to rectify the disparities between the elites and the non-elites that have been mentioned. I am not alone in this view.

  10. fisher investment

    Hi. Thanks for the good read.

  11. Kit said

    As a Sarah Lawrence student who is the son of a Sarah Lawrence Alum, I have to say that I don’t think SLC will suffer from not accepting SAT scores.

    While it’s true that they consider grades, the thing I think former President Myers forgot to mention is the fact that SLC’s admissions process also required applicants to write a few papers and submit one from a class with the teacher’s remarks on it. In my year and then some that I’ve been here, I’ve yet to take “an examination”. But I’ve written plenty of papers, and knowing my writing skills helped the admissions department to know that I could handle the Sarah Lawrence experience.

    In terms of our own grading system, we do focus primarily on detailed evaluations. In our 15-person max seminars, we see our teachers one-on-one every other week for conference related to our independent study project. Based on this close contact, they are able to write us detailed evaluations (both in the middle of the semester and at the end of the semester). These evaluations are much more useful in helping the teachers write us letters of recommendation later than a simple “A-” would be. Futhermore, the school fosters a learning environment that’s absolutely phenomenal, and I say this having transferred in from another college.

    Basically, what I’m saying is that Sarah Lawrence has a very unique pedagogy based on John Dewey’s educational theories, and just like SLC refuses to judge students against each other (which is not to imply we’re not competitive, but we don’t compete with each other, just with ourselves!), SLC shouldn’t be judged against other colleges, just itself.

  12. I’m a graduate student at the U.Minnesota’s School of Public Health. I was invited last year to speak with another grad student to a group of undergrads interested in the field, which is *very* diverse. After the general information, they weren’t particularly interested in my program. They were asking basic questions about graduate school.

    I told them they shouldn’t get too hung up on their grades. A stunned look appeared across the room. I informed them, to their disbelief, that in 10 years, no one is going to care if they got an A+ or a C in their classes. Their future employers are going to care what they did more than where they went or what their GPA was. I’ve hired people. Their GPA wasn’t the first thing I looked at; it wasn’t even the 3rd or 4th.

    I attended West Virginia University for my undergraduate work. Why? It was in-state and I got financial aid there. Sure, it’s a big school. My lower chemistry classes had well over 100 people (if not 200) in them. But past that, I got a good, solid education. How would it compare to other schools? Not sure. But at 40 I had a great job with good money and a good professional reputation. What more would I want? A “Brand Name” diploma? That’s the only thing these ‘ranking’ systems are for: Brand Name marketing.

  13. clairedeplume said

    I care how well professionals learn their craft. Who would you rather have operating on your brain, or defending in your court, or designing the tall buildings where you work – a ‘C’ or an ‘A+’ achiever who not only knows the basics, but can actually think & apply that learned knowledge? Few advancements are discovered by people with reasonable memories and an ability to get by.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: