I am currently marking my way through 70 exam scripts, for a couple of the introductory English Literature modules at my university. This blog post is the confession of this examiner, perhaps a bit risky if you find out who I really am, but nevertheless I hope worth making public as a way of demystifying a process between the end of the exam and the publication of marks that students do not often see or even understand.

If I remember my own, not-too-distant days correctly, students might want to imagine that examiners treat their scripts as sacred objects. A student has attended numerous lectures and read numerous books over the year, poured over revision notes long into the night, and then spent a few hours hunched over a desk in some dismal hall, frantically trying to pour out knowledge in the hopes that that brief exam will do justice to all the hours of work put in over the previous year. With this much invested on a few sheets of paper, surely examiners deal with them reverently, in a darkened room, with the white paper subject to the glare of an anglepoise lamp, as the examiner interrogates and teases that script to give up its worthy marks?

The reality is somewhat different. Naturally, I look after exams with the utmost care, and mark them as conscientiously as I can. However, there are certain unavoidable practicalities of marking, and of human psychology, that mitigate against any such pure, religious process described above.

The big practical issue is time. With a large number of scripts to be marked in a brief period, it is simply not possible to spend hours on each one. It would be nice if I could read an essay carefully, and then go for a walk, take a shower, and massage my temples as I try to weigh up whether to give it 66 percent or a 67. But that does not - it cannot - happen. Even marking a qualitative, essay-based subject like English, having read an essay I tend to place my mark quickly and instinctively. At my university, we work from very detailed guidelines that explain the characteristics that should be present in an essay for it to merit a First, 2:1, 2:2 or lower, with each band sub-divided into two, for example, a high 2:1 (65 to 69 percent) or a low 2:1 (60 to 64 percent). It is very rare that I ponder deeply what percentage to give an essay. Essays usually fall easily into a band, and the pressure of having perhaps a week to mark 50 scripts leaves me little time to deliberate at length whether it needs a 64 or a 63.

People often grumble that an essay-based exam cannot be marked as objectively and as fairly as something like mathematics, with a right or wrong answer. Certainly the personality of the marker may have an effect on a percentage point here or there. But on the whole it is always surprising from my examiner's perspective how easily papers drop into one of these assigned bands. The moral for university students, then, is not to lose sleep over percentages. It is the band that says everything about what sort of student you are, even if you are frustratingly just on the borderline. In many ways, a 69 percent is the most horrible mark an examiner has to give - and in the last few days I have been heard shouting at papers, because I was frustrated that a good student was not quite there, and could see that with a little nudge and feedback the student could go on to improve in subsequent essays. But my 69s are below that glass ceiling not because a few tiny details were overlooked by me, the examiner, not because I was tired, or because my football team had just lost, but because it read, argued, reasoned, discussed, evidenced in ways which said 2:1.

With this caveat about the band being everything, I will admit to some of the other factors that an examiner faces that may well lead to small variations in marks.

Imagine this scenario. I have just read two First-class essays. The third essay I mark is going to have to do something impressive not to look weaker in comparison (for those of a mathematical bent, this is an effect called regression to the mean). Perhaps I will dock it a few more marks than I might have done if marking it in isolation, because it compares worse against the previous efforts. But in the alternative scenario, marked after two solid but not particularly remarkable 2:1 essays, perhaps suddenly essay three looks better than that localised average. I know that I must be guilty, at times, of marking relative to other essays, rather than against the single standard of the mark scheme.

Luckily, there are a few ways to negate this effect. One of the most hotly debated is that fad of the 1990s, the bell curve. Perhaps I get a run of three weak essays before lunch, and then suddenly give three Firsts after lunch. Is it that I am in a better mood after my break? Is it that I have remembered those three earlier, average essays, so that those that come later are bound to look more positively in their light? I do get anxious when runs of unusually high or low results happen - as they have done this year - and that is why I find the bell curve a useful check. I may perceive that my marks are being affected by local circumstances, but taking a larger sample of my marks, I can see that they have fallen out in a normal distribution. Usually, there is a statistically good range, with a smattering of 2:2s and Firsts, and the majority bunching around the mid 2:1.



The reason that the bell curve, or normal distribtion, comes in for debate is that it is tempting to mark for the curve, rather than to construct the curve on the basis of marks. Out of ten essays I have given three 66s. Better make the next one a 59 or 71 just to smooth out the graph. This is a real risk for the individual examiner, whilst institutionally it may be tempting to adjust marks across the board to create a smooth curve with its apex at the point the university suspects most candidates should be at. In my institution, with most students coming with excellent A-levels, we would expect more high 2:1s and Firsts than another institution with a lower achieving intake, so our marks tend to have a peak around the high 60s.

Now I do not know - or have reason to believe - that my own institution does any sort of retrospective adjustment to bump our averages higher than the national baseline for English Literature degrees, but if they did the problem would be clear. Just as I get funny moments marking when there have been no Firsts for ages then three come along at once, an institution could quite feasibly have consecutive year groups which seem to achieve comparable marks, until one year is comprised of an unusually bright or slightly less well-performing group. By shoving that bell curve to fit expectations based on previous experience, the institution is engaging in a sort of social engineering, making results fit students, rather than the other way around, so that the unusually bright or underperforming group is down or upgraded unfairly. This is precisely the sort of complaint about "grade inflation" long levelled at A-Levels and GCSEs, and increasingly at universities. But as an examiner, I can sympathise with the faith in statistics and the normal distribution, because it offers subjects like English an objective foundation for marking, helping to cancel out those personal factors that do come into play, no matter how hard one tries to contain them.

The bell curve aside, students need to remember that the mark they get is not dependent on the individual examiner because other, less controversial, controls are there to restrict the impact any one examiner can have. I have admitted that time, my mood, marking an essay relative to previous results, the effect of statistics, all can affect what percentage an essay achieves, even though I would hope that these would not affect which broader band an exam falls into. But once they leave my hands, exams are filtered through layers of double-marking, moderation by other examiners from within the institution, oversight by external examiners outside of the university, anonymous exam codes, board meetings, appeals procedures, publicly displayed marks so that it is possible to see how each year's exams compare to previous ones and, finally, individual students can request copies of their exam papers and examiner's comments under the Data Protection Act. These controls too ensure that, when the best-willed examiner gets a mark an entire band out, it should be an isolated incident.

However, this last control - allowing students to see and hence to interrogate their own papers - is also controversial. My own university does not exactly make public the fact that students have a legal right to see their scripts after they have been marked. Personally, I think this right should become an expectation among students, who are still often fearful of approaching departments with what seem like trivial requests. The National Union of Students has a policy that feedback should be provided on exams, and have issued stickers for students to put on exam papers stating that "Exam Feedback Helps Me Learn." From an examiner's perspective, although in many cases it is not possible to indicate specific places where students might improve (again, partly because time pressure makes it impossible to write detailed comments), there are many papers about which I do note specific stylistic issues that could be quite easily addressed. Making these comments, though, seems like shouting into the wind, if students are never going to get the opportunity to see them. Having gone to the effort to mark a script as an examiner, why not at least allow students to get as much from your work as possible?

Besides the adminstrative burden, the reason universities are reluctant to provide exam feedback is, I suspect, from a fear of litigation or of students picking examiners up on every point to gain even more marks. Even if the fear of litigation is a little hyperbolic, the idea of student's challenging their papers may affect the exam process unduly. Those students prepared to go through the technical process of questioning their results may end up with better marks than those who are mostly concerned with studying their subject for the pleasure of it, and who are not so end-focused, and who simply accept the results given to them and look to the following year. In a system where exams are always open to challenge, results might become partly determined by a student's ability to work the system, rather than their ability in any given subject. On the other hand, is this issue not precisely the problem with exams overall, that not only are they testing knowledge but they are also testing one's ability to sit exams and to have good "exam technique" in the first place? Allowing students to interrogate and receive feedback on their own marks at the end of the process only mirrors the effect that happens in that artificial period called "exam season" at the start of it. At this time of year students who may have done less work all year sit down to cram and prepare model answers just to pass the three essay questions on an exam, whilst students who have conscientiously studied broadly throughout the year continue in their model approach to their subject in a way that does not always help them to focus on the specialised nature of an exam. As an examiner, I usually have a pretty good hunch which students have prepared to pass a few questions on the exam, and which have enjoyed studying their course as a whole, but it is a very difficult thing to prove, and it is not possible to adjust marks based on a hunch.

From my examiner's perspective, then, encouraging students to seek the written feedback from their exams would be a positive step, because it would add a qualitative report to the process, allowing those students who have worked well throughout the year even if not reflected in the pure exam percentage to seek guidance on how to improve. These sorts of students are more likely to incorporate these comments into their more holistic approach to the subject (such as their desire to write well), than those who simply aim to pass the exam as a technical challenge, and so hopefully some sort of levelling might be achieved.

If you are a student reading this post, then, I hope you feel some sense of schadenfreude. If you have been sat there feeling fed up about the fact that you have to work through exams which seem a disproportionate measure compared to the way you have worked throughout the year, it is worth knowing that this examiner at least feels the same way about marking the exams. It may be slightly disturbing that I have drawn attention to the human frailties of the marking process, but on the other hand I hope too students appreciate firstly that it is bands, not single percentages, that are the most important indicator of ability, and secondly appreciate the lengths institutions go to in order to mitigate against any widespread effect marks can be consistently misjudged, even though probably every examiner misplaces a percentage point here or there, and even occasionally gets a band wrong.

The trouble is, exams remain the most efficient system we have for testing even qualitative subjects like English. The good news is that even though there may be candidates who can work the exam system to their unrepresentative benefit, and even though examiners of essay-based subjects may be unable, as ordinary human beings, to mark every essay to its perfectly deserved percentage, on the whole, the system, tumultuous though it is during the early days of Summer, does work.

Labels: essay marking, exams, Literature, University Life

I have not posted anything here for a while. This is not due to my unwillingness to comment on Tony Blair's retirement, the climate change bill, the bad science of Panorama's Wi-Fi "investigation," or the hilarious science of the newly opened Creation Museum in Kentucky. Rather, it's got much to do with the large pile of exam papers that have been sitting on my desk for the past couple of weeks. This time last year, I commented on the postgraduate perspective on undergraduate exams, lamenting the communal hush brought by exams upon the lively university activity as well as remembering that they provide for one of the great British moments of the communal moan.

This year, that general moan has felt a little more prosaic, as I have heard it not amongst the students but amongst staff who have to mark the papers. Merrily responding to an email asking if I would like to mark some exams this year, my smile dropped as I was landed with 70 plus scripts. Marking these has been a frustrating experience, time consuming, often repetitive, but - conscious of the responsibility of marking summative as opposed to formative work - I have had to focus closely on the task.

Marking criteria in a subject such as English are notoriously problemmatic. Whilst the rubric has obviously been carefully considered, how is one to judge the difference between "well focused work" (65% to 69%) and "relevant work" (60% to 64%)? The mark schemes can only be taken up to a point, from where intuition takes over, the sense of a First as opposed to Two-One class work; this indefinable difference leaves high Two-One students seeing through a notorious glass barrier between a 69% and 70%. I have some sympathy with the government's plans to standardise degree classifications, which at the moment are not comparable across different universities or subjects, making it very difficult for employers (who may not be aware of the divide between a First and a Two-One, or of the difference between the University of Polytechnic and the University of Redbrick) to compare candidates.

And yet, having covered so many scripts, the glass barrier seems to me to be a valid one, and there is a qualitative difference between top and good work, one which cannot accurately be reflected in the quantitative difference of a single percentage point. Further, marking by a combination of rubric and experience does appear to work, at least according to the systems of double marking, moderation, external examining and the distribution curves against which we are judged. My grades passed their moderation, though with some slight modification in precise percentages in the first category, and the tally chart of grades I have been keeping has turned out to form the tell-tale bell of the normal distribution, centred around the high two-one.

More positively from a personal point of view has been the opportunity to get the sense of a year group, and a year's work (something I can't obtain by teaching a few tutorials a year to a few groups). As script after script pursues similar lines of argument, and presents comparable pieces of evidence, and similar historical, social and philosophical understanding, I realise that teaching does actually work: lectures have been attended, information has been absorbed, knowledge gained. Even when formal teaching comprises the minor part (about six hours) of the undergraduate week, it has a huge impact on a student's cumulative education.

However, the recognition of this leaves me frustrated that a further opportunity to educate students is not being pursued. The greatest frustration of marking has been my inability to follow up those marks with individual advice about how they might be improved. A student (not one of mine) last week remarked that she had never attended any of the one-to-one essay handback consultations with her tutors. I remarked that, regardless of whether they wanted to go or not, it is slightly unfair on the lecturer not to attend, since if they are anything like me, the greatest satisfaction is filling an essay with red pen, but then being able to tell the student precisely which aspects of their work were really positive, and how they can build on them. Seeing their subsequent essays, in which they have adopted this advice, gives a massive boost to the teacher. Teaching the really bright students, those who come with a unique and advanced writing style anyway, is rewarding, but I'm not sure how much "teaching" they actually benefit from; teaching those with potential not yet fulfilled, and bringing it out through contact with them, is by far the best aspect of the job.

Yet it is an aspect for which there is not as yet a replicable system in relation to the end of year exams. How frustrating it is to mark a paper in which one answer attains a good First, whilst the other two answers are solid Two-Ones! I am confident that, because the students don't see the breakdown of their marks, that Two-One candidate might go away from the board on which results are published believing themselves to be sitting comfortably in that latter category, whereas I know that, if they were told that they were capable of the very highest work, and shown the evidence of this on the papers, then the prophetic fallacy might kick in. At the bottom of our marking forms is a reminder that under new data protection laws, students can ask to see the forms, but since I doubt many are pro-active, we should make it available to them from the beginning. Our feedback may only take the form of a couple of sentences, and clearly there are not the resources to have face-to-face meetings with students, but to see that First on the page, nestling there amongst the expected results, would be, one hopes, a significant incentive for the second and third years, when marks count towards their final degree classification.

There is a great danger that the First word is something only whispered to a select few, adding to the mystique of the glass barrier. I know this was something I encountered, and even by the end of my degree I was still unsure precisely what constituted top work, and even whether I deserved it. This really is a culture of (a word often used wrongly) elitism, because every student coming to the top universities with good A-level grades should be capable of striving for the ultimate result, though for a number of reasons they may not reach it (the formal degree is only a part of a university education). So today, with almost every student, I bring the word into public discourse, saying openly which parts are First Class responses, demanding that if they get a Two-One on their early essays they should be aiming to achive the grade above by the end. Inexperienced (and possibly naive), I cannot know what impact this actually has. But it's a shame to put so much effort into marking, only to have students discouraged from making efforts for top marks.

Labels: essay marking, exams, Postgraduate Diary

Although I did not have any exams last year, my girlfriend did, and so this is the first spring in a long time when the black stresses of revision have not buzzed either around our kitchen table or in my head, thick and horrible as the swarms of flies that emerge at this time of year. You would think, then, that sitting on the other side of the table in the exam hall as an invigilator would be a pleasant experience, a refreshing dose of shadenfreude. However, sitting exposed at the front, scanning the ranks of hunched heads before me, I think I experience something like a mild case of PTSD. As the spiel plays out of my own mouth - "read the rubric," "don't forget to fill in your exam code," "you have one hour remaining," "fifteen minutes remaining" - I step back momentarily to the other side and my heart races slightly, my palms sweat and, I swear, the skin on the fingers that pinch my pen starts to toughen to blisters.

In reality, though, I have nothing more stressful to do than to hand out spare paper, escort people to the toilet, catch up on my reading or, indeed, write this blog entry. I am also in the position of being able to reflect on the sheer waste of it all. Over the next three weeks, the stationery juggernaut rumbles through academia: mountains of paper are despatched to accumulate in (striking) lecturers' offices; gallons of ink are poured on blank pages; treasury tags, graph paper, forms, lists stand piled on the invigilators' desks; thousands of randomly generated exam codes are etched on minds for a few weeks, then wiped forever. Wasted above all seem the thousands of hours that between them these talented intellects, sportspeople, artists, fundraisers, hell raisers, have spent in the flickering flourescence of the library away from these activities, all for the sake of three hours crunched beneath a small desk, desperately writing the last, dying pages of their university careers on which the value of the whole of the previous three or four years rests.

In the day's of witch hunting there was a classic catch-22 test, brilliantly parodied in Monty Python's Quest for the Holy Grail). In it the accused witch was tossed into a pond with feet and hands bound. If the she floated, the judges believed God had rejected her, proving she was evil. If she sank, however, this showed God's acceptance of her; of course, this also often lead to drowning. This analogy captures what I believe to be the unfair judgements intrinsic to end-of-year exams. Those who have worked steadily and well throughout the year might underperform, whilst those who have not committed time and effort in their studies might spontaneously produce a brilliant script. The judgement of the exams might sink you no matter how good you are.

So why bother? Students hate them; examiners loathe marking them; the only person who benefits is me, getting paid for three hours of "staying vigilant." However, surely Spring would not feel the same without the ritual coming-around of exams. There is something about knowing that this rite of passage, with its standard rubrics and its protocals and its silences, is being undertaken in the same way in every lecture room in this university, in universities up and down the country, for thousands of students. Like England being knocked out of a major sporting event and the Summer hosepipe ban, this is one of our great British moments of the communal moan.

Nevertheless, having made that optimistic point, as I must stop writing now to give the fifteen minute warning, I glad to be watching rather than doing. The "post" that signifies where I am in my university career is a welcome reminder that these events are, for me, finished.

Labels: exams, Postgraduate Diary