Posts

MONTHLY BLOG 105, Researchers, Do Your Ideas Have Impact? A Critique of Short-Term Impact Assessments

If citing, please kindly acknowledge copyright © Penelope J. Corfield (2019)

Clenched Fist
© Victor-Portal-Fist (2019)

 Researchers, do your ideas have impact? Does your work produce ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’? Since 2014, that question has been addressed to all research-active UK academics during the assessments for the Research Excellence Framework (REF), which is the new ‘improved’ name for the older Research Assessment Exercise (RAE).1

From its first proposal, however, and long before implementation, the Impact Agenda has proved controversial.2 Each academic is asked to produce for assessment, within a specified timespan (usually seven years), four items of published research. These contributions may be long or short, major or minor. But, in the unlovely terminology of the assessment world, each one is termed a ‘Unit of Output’ and is marked separately. Then the results can be tallied for each researcher, for each Department or Faculty, and for each University. The process is mechanistic, putting the delivery of quantity ahead of quality. And now the REF’s whistle demands demonstrable civic ‘impact’ as well.

These changes add to the complexities of an already intricate and unduly time-consuming assessment process. But ‘Impact’ certainly sounds great. It’s punchy, powerful: Pow! When hearing criticisms of this requirement, people are prone to protest: ‘But surely you want your research to have impact?’ To which the answer is clearly ‘Yes’. No-one wants to be irrelevant and ignored.

However, much depends upon the definition of impact – and whether it is appropriate to expect measurable impact from each individual Unit of Output. Counting/assessing each individual tree is a methodology that will serve only to obscure sight of the entire forest. And will hamper its future growth.

In some cases, to be sure, immediate impact can be readily demonstrated. A historian working on a popular topic can display new results in a special exhibition, assuming that provision is made for the time and organisational effort required. Attendance figures can then be tallied and appreciative visitors’ comments logged. (Fortunately, people who make an effort to attend an exhibition usually reply ‘Yes’ when asked ‘Did you learn something new?’). Bingo. The virtuous circle is closed: new research → an innovative exhibition → gratified and informed members of the public → relieved University administrators → happy politicians and voters.

Yet not all research topics are suitable to generate, within the timespan of the research assessment cycle, the exhibitions, TV programmes, radio interviews, Twitterstorms, applied welfare programmes, environmental improvements, or any of the other multifarious means of bringing the subject to public attention and benefit.

The current approach focuses upon the short-term and upon the first applications of knowledge rather than upon the long-term and the often indirect slow-fuse combustion effects of innovative research. It fails to register that new ideas do not automatically have instant success. Some of the greatest innovations take time – sometimes a very long time – to become appreciated even by fellow researchers, let alone by the general public. Moreover, in many research fields, there has to be scope for ‘trial and error’. Short-term failures are part of the price of innovation for ultimate long-term gain. Unsurprisingly, therefore, the history of science and technology contains many examples of wrong turnings and mistakes, along the pathways to improvement.3

An Einstein, challenging the research fundamentals of his subject, would get short shrift in today’s assessment world. It took 15 years between the first publication of his paper on Special Relativity in 1905 and the wider scientific acceptance of his theory, once his predictions were confirmed experimentally. And it has taken another hundred years for the full scientific and cultural applications of the core concept to become both applied and absorbed.4 But even then, some of Einstein’s later ideas, in search of a Unified Field Theory to embrace analytically all the fundamental forces of nature, have not (yet) been accepted by his fellow scientists.5 Even a towering genius can err.

Knowledge is a fluid and ever-debated resource which has many different applications over time. Applied subjects (such as engineering; medicine; architecture; public health) are much more likely to have detectable and direct ‘impact’, although those fields also require time for development. ‘Pure’ or theoretical subjects (like mathematics), meanwhile, are more likely to achieve their effects indirectly. Yet technology and the sciences – let alone many other aspects of life – could not thrive without the calculative powers of mathematics, as the unspoken language of science. Moreover, it is not unknown for advances in ‘pure’ mathematics, which have no apparent immediate use, to become crucial many years subsequently. (An example is the role of abstract Number Theory for the later development of both cryptography and digital computing).6

Hence the Impact Agenda is alarmingly short-termist in its formulation. It is liable to discourage blue skies innovation and originality, in the haste to produce the required volume of output with proven impact.

It is also fundamentally wrong that the assessment formula precludes the contribution of research to teaching and vice versa. Historically, the proud boast of the Universities has been the integral link between both those activities. Academics are not just transmitting current knowhow to the next generation of students but they (with the stimulus and often the direct cooperation of their students) are simultaneously working to expand, refine, debate, develop and apply the entire corpus of knowledge itself. Moreover, they are undertaking these processes within an international framework of shared endeavour. This comment does not imply, by the way, that all knowledge is originally derived from academics. It comes indeed from multiple human resources, the unlearned as well as learned. Yet increasingly it is the research Universities which play a leading role in collecting, systematising, testing, critiquing, applying, developing and advancing the entire corpus of human knowledge, which provides the essential firepower for today’s economies and societies.7

These considerations make the current Impact Agenda all the more disappointing. It ignores the combined impact of research upon teaching, and vice versa. It privileges ‘applied’ over ‘pure’ knowledge. It prefers instant contributions over long-term development. It discourages innovation, sharing and cooperation. And it entirely ignores the international context of knowledge development and its transmission. Instead, it encourages researchers to break down their output into bite-sized chunks; to be risk-averse; to try for crowd-pleasers; and to feel harried and unloved, as all sectors of the educational world are supposed to compete endlessly against one another.

No one gains from warped assessment systems. Instead, everyone loses, as civic trust is eroded. Accountability is an entirely ‘good thing’. But only when done intelligently and without discouraging innovation. ‘Trial and error’ contains the possibility of error, for the greater good. So the quest for instant and local impact should not be overdone. True impact entails a degree of adventure, which should be figured into the system. To repeat a dictum which is commonly attributed to Einstein (because it summarises his known viewpoint), original research requires an element of uncertainty: ‘If we knew what it was we were doing, it would not be called “research”, would it?’8

ENDNOTES:

1 See The Research Excellence Framework: Diversity, Collaboration, Impact Criteria, and Preparing for Open Access (Westminster, 2019); and historical context in https://en.wikipedia.org/wiki/Research_Assessment_Exercise.

2 See e.g. B.R. Martin, ‘The Research Excellence Framework and the “Impact Agenda”: Are We Creating a Frankenstein Monster?’ Research Evaluation, 20 (Sept. 2011), pp. 247-54; and other contributions in same issue.

3 S. Firestein, Failure: Why Science is So Successful (Oxford, 2015); [History of Science Congress Papers], Failed Innovations: Symposium (1992).

4 See P.C.W. Davies, About Time: Einstein’s Unfinished Revolution (New York, 1995); L.P. Williams (ed.), Relativity Theory: Its Origins and Impact on Modern Thought (Chichester, 1968); C. Christodoulides, The Special Theory of Relativity: Foundations, Theory, Verification, Applications (2016).

5 F. Finster and others (eds), Quantum Field Theory and Gravity: Conceptual and Mathematical Advances in the Search for a Unified Framework (Basel, 2012).

6 M.R. Schroeder, Number Theory in Science and Communications: With Applications in Cryptography, Physics, Biology, Digital Information and Computing (Berlin, 2008).

7 J. Mokyr, The Gifts of Athena: Historical Origins of the Knowledge Economy (Princeton, 2002); A. Valero and J. van Reenen, ‘The Economic Impact of Universities: Evidence from Across the Globe’ (CEP Discussion Paper No. 1444, 2016), in Vox: https://voxeu.org/article/how-universities-boost-economic-growth

8 For the common attribution and its uncertainty, see [D. Hirshman], ‘Adventures in Fact-Checking: Einstein Quote Edition’, https://asociologist.com/2010/09/04/adventures-in-fact-checking-einstein-quote-edition/

For further discussion, see

To read other discussion-points, please click here

To download Monthly Blog 105 please click here

MONTHLY BLOG 81, RESPONDING TO ANONYMOUS ACADEMIC ASSESSMENTS

If citing, please kindly acknowledge copyright © Penelope J. Corfield (2017)

(*) This BLOG follows its matching BLOG/80 (Aug. 2017)
on ‘Writing Anonymous Academic Assessments’

2017-08 No3 handshake diagram

The first arrival of anonymous assessments of one’s own research is almost invariably annoying. There’s something about the format which gives the author-less verdict a quality of Olympian majesty. And, even if the verdict is favourable, there’s a lurking feeling that one is a mere minnow, being condescended to by a remote and all-wise deity. Ouch!

However, after recovering from one’s initial fury, it’s best to rally and to view the whole exercise as a free consultation. Instead of rushing into print, and getting a stinker of a review, the stinker is delivered in the form of an anonymous assessment before publication. The anonymous critic is, in fact, the best friend, lurking in disguise.

As well as writing constant assessments, academics also read one anothers’ work in typescript. But, as researchers say, ‘good criticism is hard to get’. Many friends just respond loyally: ‘Darling, it’s wonderful; but there’s a typo on page 33’. Such a reaction is not much use. In the case of an anonymous assessment, by contrast, someone has gone to a lot of trouble to identify all your faults. And, what’s more, to give you a chance of remedying them before publication.

On balance, I would say that 80% of all the anonymous advice, which I’ve received over the years, has been invaluable. Another 10% is comparatively trivial, meaning either that the assessor has been sleeping on the job or (rarely) that there’s nothing major to criticise or discuss. But 10% of responses are positively unhelpful, either through being too crushing – or simply irrelevant.

One example of off-the-wall and unusable reflections concerned my editorial introduction to a book of essays entitled: Language, History & Class (1991).1 The anonymous assessor said firmly that I was wrong; and offered, at some length, his/her own philosophical alternative historical/linguistic theory as a variant. In one way, it was a very generous piece of writing. But, on the other hand, it was entirely wasted. I couldn’t use the alternative view, because I disagreed with it – and anyway, it wouldn’t be either right or politic to take someone else’s original thesis as my own, whether I agreed or disagreed. Something in my text had apparently rapped the assessor’s intellectual funny-bone, causing him/her to get distracted into inventing a new theory rather than reviewing a book proposal. The alternative approach was so off-the-wall that I never saw it appear anywhere in print. It was an intellectual kite that never flew.

2017-09 No2 kite in trees

Generally, however, after the first moment of silent fury at reading the anonymous assessor, I buckle down and enjoy the chance to revise in the light of a really in-depth analysis. Often, rewriting helps to strengthen my arguments, giving me a chance to rebut criticisms explicitly. And, simultaneously, the rewrite allows scope for clarification, if ideas were poorly or incompletely expressed first time round. Sometimes points have been made out of their logical order and need reshuffling. And finally, I sometimes (not too often!) change my mind, in the light of criticisms; and the process of rewriting allows me to push my argument into new directions.

In reporting subsequently to the publishers or editors, who have commissioned the anonymous assessment, there is one golden rule. The criticisms do not have to be adopted wholesale. But they must be acknowledged, not simply dismissed. I remember one former PhD student, when editing her first essay for a learned journal, miserably wondering whether she had to ditch her entire argument, in the light of a critical assessment. I was horrified at the prospect. Of course, she had to stand by her new interpretation. (She did). The essay would appear under her name and must therefore represent her considered views. An adverse anonymous assessment does not have the status of a royal command. Instead, the hostile cross-fire gives authors a chance, pre-publication, to decide whether to strengthen or to adapt their arguments.

Then it’s up to editors to decide. Usually they appreciate the chance to get new views into print, with the prospect of opening up further debates. But editors do like to be reassured that the revisionist piece has been submitted knowingly, with a full awareness of the potential controversies to follow, and that the study is well argued and substantiated. In comparatively rare cases, when challenging new views are rejected by one journal, there’s a reasonable chance that the ‘new look’ can find a home elsewhere. Since historical research relies upon debate and disagreement, it’s not such a big deal to find one (temporarily) prevalent view coming up for critique and/or complete refutation.

Only in very rare cases are anonymous assessors unduly harsh or vitriolic. I’ve had plenty of negative responses myself but never anything without some constructive aim or intention. One hostile case, however, occurred in response to a former student of mine who had written an excellent essay on the social history of nineteenth-century Sussex. Some element of the argument had apparently infuriated the anonymous assessor. He/she basically argued that the essay should not have been written. There was nothing constructive upon which the author could build. Fortunately in this case, the journal editor had asked for two anonymous assessments. The second was much more positive, enabling my former student to revise the essay into a stylish contribution. However, I advised her to write to the editor, explaining calmly that she had considered the negative assessment carefully before disregarding it. The fact that the angry assessor’s report had mis-named ‘Sussex’ throughout as ‘Suffolk’ suggested that the tirade was not based upon a very close reading. The editor took this strong hint on board; and the revised essay successfully appeared in print.2

These examples indicate the intricacies of peer review and the publication process. They are socially imbedded – and far from purely impartial. But they strive for an interactive collegial process, which seeks to iron out individual rancour or prejudice. Personally, I take anonymous academic assessments of my embryonic work as seriously as I expect my own anonymous academic assessments to be taken by the anonymous recipients. The veil of secrecy strives to make the exchange of ideas a ‘pure’ intellectual exercise, without the formal courtesies and pleasantries. (Actually, if one wants, it’s usually possible to make a stab at identifying the critics, using one’s research-honed powers. But in my experience, that’s an unproductive distraction).

Scholars who are published in peer-reviewed outlets are thus in constant dialogue (or, preferably, ‘plurilogue’),3 not just generally with their peers, and patchily with their precursors in earlier generations but specifically with their specially recruited anonymised critics. Wrestling with obdurate drafts is often exasperating and lonely work, as Hogarth knew – as seen in a detail from his Distrest Poet (c.1736) below. Yet scholarly authors don’t work in isolation. A tribe of anonymous academic critics, friendly readers, and interventionist editors/publishers are looking over their shoulders. So it’s best to bite the bullet; to revise coolly; and then to publish and be damned/whatever.

2017-09 No3 Hogarth's distressed_poet

1 P.J. Corfield, ‘Historians and Language’, in P.J. Corfield (ed.), Language, History and Class (Oxford, 1991), pp. 1-29; slightly amended text also transl. into Greek for publication in Histor, 12 (May 2001), pp. 5-43.

2 A. Warner, ‘Finding the Aristocracy: A Case Study of Rural Sussex, 1780-1880’, Southern History, 35 (2013), pp. 98-126.

3 For this usage, see P.J. Corfield, ‘Does the Study of History ‘Progress’? And does Plurilogue Help?’ BLOG/61 (Jan. 2016), in www.penelopejcorfield.com/monthly-blogs.

For further discussion, see

To read other discussion-points, please click here

To download Monthly Blog 81 please click here

MONTHLY BLOG 60, WRITING THROUGH A BIG RESEARCH PROJECT, NOT WRITING UP

If citing, please kindly acknowledge copyright © Penelope J. Corfield (2015)

My heart sinks when I hear someone declare gaily: ‘I’ve done all the research; now all I have to do is write it up’.1 So what’s so wrong with that? It sounds so straightforward. First research, then sit down and write. Then, bingo, big party with lots of happy friends and relieved research supervisor.

But undertaking a big project in the Humanities or Social Sciences doesn’t and shouldn’t work like that.2 So my heart sinks on behalf of any researcher who declares ‘All I have to do is write it up’, because he or she has been wasting a lot of time, under the impression that they have been working hard. Far from being close to the end of a big project, they have hardly begun.

Why so? There are both practical and intellectual reasons for ‘writing through’ a big research project, rather than ‘writing up’ at the end. For a start, stringing words and paragraphs together to construct a book-length study takes a lot of time. The exercise entails ordering a miscellany of thoughts into a satisfactory sequence, marshalling a huge amount of documented detail to expound the sustained argument, and then punching home a set of original conclusions. It’s an arduous art, not an automatic procedure.
2015-12 No1 Hogarth's distressed_poet

Hogarth’s Distrest Poet (1741) expresses the agonies of composition, as he sits in a poky garret, poor and dishevelled, with abandoned drafts at his feet.

Writing and research in the Humanities and Social Sciences should thus proceed in tandem. These tasks between them provide the necessary legs which enable a project to advance. No supervised researcher should be without a target deadline for a forthcoming report or interim paper, which collectively function as prototype chapters. That rule applies from the onset, starting with a written review of the research questions, or bibliographical overview, or primary source search – or however the project is launched. Without ‘writing through’, researchers do not really appreciate what they have found or what they are arguing. Certainly there will be much redrafting and revision, as the research progresses. That’s all part of the process.

But grappling with ideas to turn them into a sustained account in written words is not just a medium for communication. It’s a mechanism for cogitation itself. Just as spoken language crystallises instinctive feelings into expressed thoughts, so the process of turning thoughts into written form advances, clarifies and extends their meaning to form a considered analysis. A book can say much more than a speech, because it’s longer and more complexly structured than even the longest speech. Writing through continually means thinking through properly.

Incidentally, what about prose style? The answer is: suit yourself. Match your personality. Obviously, suit the subject-matter too. Snappy dictums are good value. I enjoy them myself. They punch an argument home. But non-stop bullet-points are wearing. Ideas are unduly compressed. Readers can be stunned. The big argument goes missing. Writing short sentences is fun. Brevity challenges the mind. I could go on. And on. One gets a second wind. But content is also required. Otherwise, vacuity is revealed. And exhaustion threatens. So arguments need building. One point after another. There may be an exception. Sometimes they prove the rule. Sometimes, however, not. It depends upon the evidence. Everything needs evaluation. Points are sometimes obvious. Yet there’s room for subtlety. Don’t succumb to the obvious. Meanings multiply. Take your time. Think things through. Test arguments against data. There’s always a rival case. But what’s the final conclusion? Surely, it’s clear enough. Think kindly of your readers. Employ authorial diversity. Meaning what exactly? [162 words in 39 sentences, none longer than five words]

Alternatively, the full and unmitigated case for long, intricate, sinuous, thoughtful yet controlled sentences, winding their way gracefully and inexorably across vast tracts of crisp, white paper can be made not only in terms of academic pretentiousness – always the last resort of the petty-minded – but also in terms of intellectual expansiveness and mental ‘stretch’, with a capacity to reflect and inflect even the most subtle nuances of thought, although it should certainly be remembered that, without some authorial control or indeed domination in the form of a final full-stop, the impatient reader – eager to follow the by-ways yet equally anxious to seize the cardinal point – can find a numbing, not to say crushing, sense of boredom beginning to overtake the responsive mind, as it struggles to remember the opening gambit, let alone the many intermediate staging posts, as the overall argument staggers and reels towards what I can only describe, with some difficulty, as the ultimate conclusion or final verdict: The End! [162 words in one sentence, also fun to write].3

In other words, my stylistic advice is to vary the mix of sentence lengths. A combination of an Ernest-Hemingway-style brevity with an Edward Gibbonian luxuriance allows points to be fully developed, but also summarised pithily.

Thus, in order to develop a sustained case within a major research project, my organisational advice is to ‘write through’ throughout. That’s the only real way to germinate, sustain, develop, understand innerly and simultaneously communicate a big overarching picture, complete with supporting arguments and data. Oh, and my final point? Let’s banish the dreadful phrase ‘writing up’. It means bodging.
2015-12 No2 Writing

A snappy dictum from the American journalist and writer William Zinsser (1922-2015).

1 This BLOG is a companion-piece to PJC BLOG/59, ‘Supervising a Big Research Project to Finish Well and on Time: Three Framework Rules’ (Nov. 2015). Also relevant is PJC BLOG/34 ‘Coping with Writer’s Block’ (Oct. 2013).

2 In the Sciences, the model is somewhat different, according to the differential weight given to experimental research processes/outcomes and to written output.

3 My puny effort barely registers in the smallest foothills of lengthy sentences in the English language, one celebrated example being Molly Bloom’s soliloquy as finale to James Joyce’s Ulysses (1922), reportedly in a sentence of over 4,000 words.

4 Hemingway is commonly cited as the maestro of pithiness. Yet the playwright Samuel Beckett also shares the honours in the brevity stakes, writing in sharp contradistinction to his friend and fellow-Irishman James Joyce.

For further discussion, see

To read other discussion-points, please click here

To download Monthly Blog 60 please click here