KEVIN BECK

Australian Research Quality Frameworks

The control of publicly funded research by interested peer groups, authored: 2005.
Another free to use public interest web site on the
Mosaic Portal Network


Created by

Kevin Beck




MOSAIC




ACADEMIC
Academic Research






Critique of a draft presented to the Minister for consideration and approval, by an Expert Advisory Panel, by Kevin R Beck 2005. Sources are attributed and full documentation can be found at the indicated web sites in the text.


A Recommended framework for Research Quality Assessment in Australia

The Minister for Education, Science, and Training in Australia commissioned a panel of eminent experts to develop a proposal for a Research Quality Framework. The committee published its preferred model in September 2005, and called for public comment.

If the purpose of the framework is to provide a process to assess research solely on behalf of the universities themselves, then the proposed form of peer review, inherent in the recommended Research Quality Framework is adequate. If the assessments are undertaken in order that the body, and its members, can be seen to have discharged their obligations under the legislative Act, again the proposed system, of peer dominated determination, is adequate.

If, conversely, the assessment is intended to serve government or taxpayer, in terms of tapping the innovation and talent of all aspirant researchers, including those in the private sector, forging relationships and synergies, then it is inadequate.

Chris Topley, the Deputy Vice Chancellor Research and Development at the London Metropolitan University commenting on the United Kingdom proposals raises points that are applicable to the Australian recommendation. "The proposed methodology measures the goodness and badness of research by a set of standards devised by those being measured (university academics); and the judgements are made by panels largely consisting of those being measured (university academics). The outcome is that good research (of the type that academics choose to define as good research) is well funded, while bad research (again as defined by academics) is not. If the fundamental purpose of research assessment is to promulgate this somewhat introspective view of good and bad research, then the present system is not only adequate, it is all but ideal. Most metrics systems, which perhaps theoretically could get around this question of introspection, in fact come up against the same difficulty. Citations are largely cited by academics; g rant applications are largely judged by academics; drafts of papers are largely refereed by academics, and so on.

Academics are not necessarily the best qualified to judge whether they are giving good value in their work to the government or the taxpayer/voter. Nor, rationally, are they necessarily in the best position to judge how funds should be used in support of research, unless one is prepared to accept that the production of good research, as defined by those same academics, is the best use of funds. There is a significant and serious danger of a self-fulfilling prophesy in such an argument, and many believe that this is precisely the position in which UK Higher Education now finds itself".(Chris Topley, Deputy V-C Research and Development, Research Strategy Committee, London Metropolitan University, 28 November 2002).

The Australian Minister for Education and Training the Honourable Brendan Nelson MP's says in a media statement,

"The features of assessment described in the Paper reflect a desire to identify high quality and high impact research in an international context, while also recognising the research priorities for Australia in the future."

The Minister's Foreword in the Australian Research Quality Framework draft says inter alia:

"While being the best in Australia is an achievement, Australia's future depends on a collective effort by the Australian Government, industry and the community in identifying areas of international research excellence in Australia. Many of the features outlined in this paper, including how the assessment mechanism will be structured, the membership of assessment panels and the kind of assessment criteria and guidelines required for the RQF reflect this desire to focus on excellence in research quality and impact".

How is a collective effort to be achieved when the Research Quality Framework put forward for publicly funded research is limited to universities and other publicly owned entities such as the CSIRO and (public - private incorporated) Cooperative Based Research Centres?

Privately funded research entities appear to be excluded.

Scope

According to the proposed model before the Minister, the RQF will include all publicly funded research conducted within Australia's university sector and Publicly Funded Research Agencies.

How are private sector and not for profit philanthropic research facilities to be drawn into the Australian framework of research excellence implied in the Minister's statements? Is it assumed that the bulk of research monies go to universities?

The panel recommending the model to the government defines the quality and impact criteria that are to be the primary objective of the framework. Quality and Impact

"The quality of research including its intrinsic merit and academic impact - academic impact relates to the recognition of the originality of research by peers and its impact on the development of the same or related discipline areas; and its broader impact or use. That is the extent to which research is successfully applied - broader impact or usefulness relates to the recognition by qualified end-users that quality research has been successfully applied".

It is not clear who the "qualified end users" might be? The prominence of "recognition of academic impact by peers" is a feature of the preferred model and this is not surprising given the constituent make up of the Advisory Committee. The notion of peer review is deeply embedded in academic systems of all kinds and is generally vigorously defended by academics. It seems absolutely right that 'peers' should indeed be those that judge the merit of anything that is itself wholly academic in nature, since no others are better qualified. Thus, as long as the assessment of research is viewed as an exercise in judging the merit of research from a wholly academic standpoint, there ought to be no serious challenge to the peer review system. However, if it is the government's intention that public research should serve e a public or commercial good, outside of the narrow academia utilisation then peer review and control is likely to stymie this objective.

The Australian government policy statement, "Backing Australia's Ability - Building Our Future Through Science and Innovation," includes a Message from the Prime Minister, John Howard, which answers the question of intention.

"The Australian Government's goal is for Australia to build a world-class innovation system. This ambitious agenda depends on effective partnerships between governments at all levels, researchers and business, to share the substantial financial investment necessary and to ensure that ideas move smoothly from generation to end use.

Mapping Australian Science and Innovation, released in late 2003, and a range of policy reviews and programme evaluations have provided the factual basis for policies, which will assist Australia to achieve even higher levels of performance. These policies will expand our capacity to deliver tangible results from our science and innovation system for advancing Australia's international competitiveness and productivity and creating jobs.

This Government is the first to develop National Research Priorities to align Australia's research effort with the community's economic, social and environmental goals. Through Backing Australia's Ability - Building Our Future through Science and Innovation, we are further encouraging Australia's research and business communities to work together in pursuit of those goals".

This implies something well beyond a system of dominated peer review solely for serving university academic research. The Prime Minister talks of a policy that involves collaboration, transparency, and broad assessment of what is valuable.

Underlying principles

The committee states that the proposed RQF will be underpinned by the following principles:

Transparency: process and results are clear and open to stakeholders, including the use of reliable/repeatable measures/metrics;

Acceptability: broad acceptance of the approach and measures;

Effectiveness: the applied model achieves the purposes of valid and accurate assessment, and avoids a high cost of implementation and the imposition of a high administrative burden; and

Encourages Positive Behaviours: improving the quality and impact of research and further developing and supporting a vibrant research culture in Australia.

These are considered motherhood statements in that they do not define how these principles will "develop and support a vibrant research culture" beyond the academic walls of the participating institutions.

Transparency may be obvious to those who work in the higher education sector or are closely associated with it. However, to the general community (taxpayer) and to the private sector and others seeking to win research grants from the available pots of money, transparency has not been a hallmark of the operation of the assessment of research within the Departments of the Australian government and the decision makers (academics) who serve on the assessment committees. Further Australian universities, and their obsession with peer structures, are not known for their transparency in the operation and decision-making within their Universities. Peer methodologies are precisely what they imply: hierarchical, and egotistical, in design and operation.

The positive behaviours goal is an outworking of the peer pressure in institutions to "shape" the behaviour of academic researchers and to ensure compliance. Mavericks (often the creators of real innovation) are reigned in, brought to heel or perhaps ostracised and excluded.

In 2000, the Australian government allocated $50,000,000 (over a number of years) to CSIRO for R&D in light metals. The CSIRO work was to be utilised in the commercial application of the Australian Magnesium Corporation production of magnesium, alloys, and later thin sheet metal. There was much debate, recorded in Senate Estimate hearings and departmental and parliamentary records of correspondence on the "exclusivity" of this funding and its application. The record of the Senate can be found in Estimates hearings of January - February 2002 and in correspondence to then Ministers Senator Nick Minchin and Peter McGauran as well as other parliamentarians. The papers in the Senate clearly show that the "peer mechanism' and procedures regulated by the government were not followed and that the R&D Board did not receive quality advice and assessment in acco5rdance with those regulations.

External critics (to the committee of expert peers) argued that the process used by the government, through a Research and Development Board, of assessing and funding research was not transparent. They argued against all the expert advice that the AMC project would not be viable and that CSIRO would not meet its flagship objective. The peer process for research decisions was confronted. Rather than look at whether there was any substance the membership of the academic - industry expert group, continued to maintain their line to the government, placing their eggs (so to speak) in one very expensive basket, over which they had unchallengeable control.

In the years 2000 - 2003 the writer of this article travelled in Australia, Europe and the United Kingdom, in the USA and Canada, meeting with senior researchers in public and private sectors, in universities and private research facilities including the Australian Cooperative Research Centre established for this application development in Queensland. Also meeting with manufacturers and end users in the automobile industry in Detroit USA and Stuttgart Germany and with senior scientists and managers in Biliton London and BHP, Melbourne. The consensus of all of these meetings particularly internationally was that the development of a magnesium industry on the scale published in the Australian governments Light Metals Agenda would not eventuate. The writer, up until 2004, continually conveyed this view to the Australian government. Face to face, meetings with Minister Peter McGauran again took place. He arranged a meeting with the relevant manager of what is called a "flagship" programme within CSIRO, to no avail. In defence of the politicians, it must be said that it is a leap of faith to cast aside the advice of experts. All the experts said that the Light Metals Agenda, and government initiatives and the flagship magnesium projects and research were solid. Ford invested $40,000,000 in return for sales take outs of 50% of the production. The market agreed and shareholders invested about $1.4 billion, with $150,000,000 coming from the Queensland government. The federal government even proposed, but did not finally give, a $200,000,000 loan to the magnesium industry.

In 2004, the Australian Magnesium Corporation project to manufacture metal for use by Ford and others worldwide, collapsed.

What is not clear to the writer is how the group advising the government and engaged in the research could have got is so wrong when so many international sources available to the writer were giving contrary views. The only conclusion reached is that the primary objective of the parties was to carry out the research and to build a production plant. Beyond that, there seemed little recognition, or knowledge of, market and competition potential and realities.


It may be that the "peer process" in science, commercial and other spheres of society and economy are self reinforcing.

Questions as to what CSIRO was doing and whether R&D is still being undertaken on, that project in concert with what is left of AMC goes unanswered by the government and CSIRO. Where is the transparency in that application of public monies into a publicly funded research entity, CSIRO? Where is the effectiveness of the investment demonstrable and its application to innovation in Australia? The public interest and broader tests implied in the Prime Minister's statements above were not served. In 2005 the Minister for Education, Science and Training, the Honourable Brendan Nelson abolished the Research and Development Board.

Whilst the above example is not precisely relevant to the determination of broader university academic research funding (even though CSIRO is included in the framework and the CRC included a university in Queensland) it does demonstrate that that the major purported strength, and influence, of the 'expert' approach is its familiarity and security, trusting well tried experts. Its major weakness is its tendency to become self-defining in that what the experts think is good is what is funded. Views to the contrary are dismissed if they are not considered expert and the critic/s is, are, not considered expert.

There is no doubt that the proposed Research Quality Framework will be a useful political tool for relying upon "experts" and thus ceding ultimate accountability to a third party. However, it is beyond reasonableness to suggest that it measures anything of transparent value to the world outside academe and university management, government beyond how much money has been spent. It is unlikely that the public, or competing interests and enquiring minds will ever be told on precisely what, in detail, it has been spent. Where is the process, defined ihe framework of publishing what units of funds are spent on what elements of research within what disciplines of Australia's publicly funded universities? Will the researchers point to an article in a peer-reviewed journal? One assumes that researchers do report to the entity that grants them the funds. However there is a record of avoiding disclosure of the expenditure if questioned in any depth or detail. The committee appears to be recommending that reportin g will be on the basis of metrics, which is, the common method used in Australia. Metrics has the danger of promoting certain shaping and behavioural modification particularly in relation to disclosure and reporting and the time spent on these activities as against actual research.

"Measuring research performance entirely on metrics is a flawed approach. Such a system cannot measure the future potential, or even attempt to do so. Virtually all metrics are tedious and administratively expensive to compile, and most metrics systems tend to 'channel' researchers into pursuing the metrics for their own sake, rather than doing what they are best at, in their own best way. Available metrics can perhaps be combined to provide an accurate picture of the strengths that those metrics purport to measure, retrospectively. They cannot be prospective, and, because they are 'metric', they frequently undervalue such matters as ethos, environment, strategic vision, breadth, connectedness to other disciplines etc. They are also particularly unsuited to research in the Arts and Humanities. As to the behavioural effects of metrics; there is no question that behaviours would change so as to pursue the rewards. Strengths of such a system, apart from simplicity, are few; the much-vaunted objectivity is s eriously open to question, and the data simplicity is a myth. Weaknesses are many, including the unfortunate behavioural effects and the danger of reductio ad absurdum".(Topley, op cit).

The proposed Quality framework is a model to ensure maintenance of control of the direction of, and the participants in, Australia's publicly funded research sector by a very specific group of interests and participants. It is about getting block funding which is then allocated according to peer ranking, prestige and perhaps even patronage. This is evidenced by the following extracts from the recommendation to the Minister.

"Panels representing a broad discipline area should consist of approximately 12-15 members each, and include:

Academic peers and other expert reviewers

International reviewers will comprise approximately 50% of the membership for the RQF Assessment Panels.

Panel Chairs will apply guidelines to determine their membership, particularly with reference to the international membership, according to the discipline/s needs".

The framework embeds ranking, exclusivity, and patronage of particular researchers and the establishment of a system that allows university heads to decide (through bock grants) who will get the money and for what.

"Development of the RQF: Progress to-date on agreed features

Drawing on the evidence from overseas and Australian research quality assessment exercises; feedback received in response to the RQF Issues and Advanced Approaches Papers; and outcomes from the detailed work of three EAG Working Groups, the EAG agreed that the RQF should have the following features:

It should enable comparisons to be made of the research excellence between universities and PFRAs. However, funding pools for universities and PFRAs should be kept separate.

Should focus on research quality whose value would be related to its impact and would include assessment of its impact.

Institutions should decide, against a set of guidelines, which researchers would be assessed in the RQF.

Research groupings will be the subject of assessment in the RQF. Research groupings may be either identified Research Groups whose work may cut across disciplinary boundaries, or aggregations of individuals who do not necessarily work together but whose work fits within a common broad Research Fields, Courses and Disciplines (RFCD) classification.

Nominated research groupings' research outputs should be placed into context through statements from institutions.

Assessment panels should be established for the RQF; and these panels should be structured on broad-disciplinary lines based on aggregations of the ABS RFCD classifications. Panels should have explicit procedures for assessment of cross disciplinary research.

Representation on assessment panels should consist of a majority of academic peer-reviewers with additional research end user reviewers, as appropriate.

The RQF should have a graded points system/rating scale eg 0 to 4, 1 to 5, etc. which is sensitive to discipline differences.

It should be structured in a way, which enables assessments of quality to be provided at multiple levels, for example, by broad discipline area, Faculty or other structural unit up to institutional-level. There is to be no reporting at the level of individual researchers.

Funding allocated to institutions on the basis of the RQF should take the form of block grants for which institutions have discretion to determine their internal distribution."

The extracts appear to support a proposition that this framework enshrines institutional systemic power and control, limiting participation to "academic peers", "graded lecturers" of a certain status and "control mechanisms of "grouping" and "creating aliases for researchers by grouping", particularly for those people whose status is not yet recognisable to the academic world. This places the research under the broad, and non-transparent umbrella, of the university of entity, carrying out the research. The framework limits participation.

"Eligibility for assessment in the RQF

The draft recommendation inter alia, says that: "eligible academic staff are to be nominated by universities for assessment should hold university Level B and above positions. Academic research staff whose positions are funded through individual competitive research grants may also be nominated for assessment, for example postdoctoral fellows holding Level A positions. Individual researchers will be nominated by universities as members of a 'research grouping' A research grouping can be either an aggregation of individual researchers who do not necessarily work together but whose work fits within a common broad RFCD classification, or an identified Research Group in which individual researchers are working together on research that may cut across disciplinary boundaries."

The recommended framework appears to contravene the Government's desire to open up research to an innovative broader collaboration between creative individuals, industry, community, and institutions. The suggested framework is also about volume and frequency, a part of the peer status process of universities.


Evidence portfolios for assessment

"Universities will provide evidence portfolios of individual researcher's outputs within nominated research groupings. To be included for assessment, two conditions must be met. Each individual researcher included in a research grouping must have produced at least four research outputs over the assessment period."

The Minister for Education, Science and Training in a number of his policy speeches has raised the issue of educational excellence and prestige not necessarily needing to be tied to the number of papers a researcher has had published. Whilst the Minister has indicated that this is collaboration with industry, subtly the university peers seek to ensure minimisation of the impact of external parties upon their deliberations and decisions. Here is a reference to limiting the scope of industry consultation and formal linkages.

"The evidence of research impact should be verifiable. Consequently, and as in the assessment of research quality, the expert advisory group (EAG) agreed that research impact should be retrospective and demonstrable rather than prospective and potential. It should not focus on mechanisms and structures designed to increase the likelihood of impact (such as industry consultation or formal linkage arrangements), but rather on the outcomes of such arrangements".

The rating system recommended in the framework appears to be a mechanism to ensure control limiting assessment criteria within a narrow peer group of the relevant disciplines of the academic world.

"The EAG recommends separate criterion-referenced ratings for research quality and research impact. It recommends that research quality should be rated on a five point scale. There is clear acceptance that the top ratings will be based on notions of "international excellence" - outstanding research achievements on an international scale".

Who decide what the five-point scale is and who will be rated according to it? A number of existing Australian, and international models, were examined by the EAG. One can note a consistency between members of the academia regardless of location or type. All reinforce the "prestige of being published" and the merit of international recognition. None actually require that the research be demonstrated to be able to, or to have in retrospect, have delivered a public good as the primary consideration. Rather the research's objective is to add to the body of collective knowledge.


Rating Descriptor
5
The majority of research outputs were considered to be in at least the top 20% of research in its field internationally, with a significant percentage (>25%) in the top 10%. There was evidence of high international peer esteem and significant impact on the international academic community.


If the only purpose of assessing research was to measure, in hindsight, how well it supported the economy, and the deliverable benefits to Australian citizens, one wouldn't seek to know which were the best publications attached to each academic author in international journals, what the peers thought and how the international academic community was impacted.

One might seek information as to how the research impacted the number of contracts won by commercial or non-profit entities using the research and actually delivered, the technology transferred, knowledge transferred, and the clear contribution to humanity in some form or other. Such data may not be objective, but they are closer to the purpose than the descriptive table above, proposed by the EAG.

In reality, one expects that the assessment might be for multiple purposes, which ought to lead to the citing of different kinds of objective data, not the very narrow interpretation and shell in the box above.

The designers of the recommended Australian Research Quality Framework have chosen a nu,ber of Australian and international examples of rating systems that might be used. They all reinforce the "peer control." With great respect to the expert advisory group members and the academic world, the funds for research come from the community at large. The greater numbers of taxpayers are predominantly laypersons, as are the politicians they elect, and they may not find the descriptors all that illuminating. The question is how will members of parliament; particularly the cabinet, approving and funding the framework decipher the jargon below, which is meaningful, and measurable, only to the closeted world of academia.


Possible assessment scales and descriptors

New Zealand Quality Categories and Criteria

Peer Esteem Assessment Criteria


Rating
Descriptor
7/6
The evidence portfolio would be expected to demonstrate that the staff member has attracted world-class recognition through her or his research. This could be reflected by some or all of the following: the receipt of prestigious prizes, or fellowships of leading learned societies/academies or prestigious institutions, or special status with professional or academic societies, or editorship, membership of editorial panels or referees of top-ranked journals, or awards for research as well as invited attendance, or examination of PhDs and presentation at prestigious academic and industry conferences/events. An ability to attract overseas/top research students and scholars as well as to mentor his/her own students into postdoctoral and other fellowships, scholarships and positions in centres of research excellence could be demonstrated in the evidence portfolio. A consistent record of favourable citations of research should combine with strong evidence of positive research reviews and contribution to knowledge in the discipline (including overseas where relevant), and movement into creative practice.
5/4
The evidence portfolio shows that the staff member, through their research, is recognised within New Zealand or elsewhere and is esteemed beyond the researcher's own institution. The evidence portfolio demonstrates peer esteem by providing evidence of some or all of the following: the receipt of prizes, membership of a professional society or similar with restricted or elected membership or honours or special status with professional or academic societies, editorship or membership(s) of editorial panels of reputable journals within New Zealand or elsewhere, research fellowships of esteemed institutions, reviewing of journal submissions and book proposals, PhD examination or invitations for keynote addresses for conferences/events that are at a middle level of excellence. A consistent record of research citation and positive reviews of specific research outputs and/or overall contribution to research knowledge in a discipline or substantive area of knowledge or practice can be expected. The evidence portfolio could demonstrate graduate students moving into research scholarships or postdoctoral fellowships or junior lectureships in departments with good research ratings.
3/2
The evidence portfolio demonstrates a developing recognition among peers of the staff member's research contribution and developing rigour in the application of research techniques. This may be evidenced through attracting awards and invitations to present research to informed audiences, within and possibly beyond the applicant's immediate institution, as well as positive reviews and citations, or being asked to referee research outputs. Where the staff member has an involvement primarily in commissioned research outputs, reference to letters of commendation or other evidence of esteem by commissioning agents could be expected.
1
Minimal evidence of peer esteem generated through research activities
0
No evidence of peer esteem generated through research activities.




New Zealand Quality Categories and Criteria
Overall Quality Category

Rating Descriptor
Quality Category A
To be assigned an "A" for his/hers EP it would normally be expected that the staff member has, during the assessment period in question, produced research outputs of a world-class standard, established a high level of peer recognition and esteem within the relevant subject area of his/her research, and made a significant contribution to the New Zealand and/or international research environment.
Quality Category B
To be assigned an "B" for his/hers EP it would normally be expected that the staff member has, during the assessment period in question, produced research outputs of a high quality, acquired recognition by peers for her/his research at a national level, and made a contribution to the research environment beyond her/his institution and/or significant contribution within her/his institution.
Quality Category C
To be assigned an "C" for his/hers EP it would normally be expected that the staff member has, during the assessment period in question, produced a reasonable quantity of quality-assured research outputs, acquired some peer recognition for her/his research, and made a contribution to the research environment within her/his institution.
Quality Category C (NE)
To be assigned an "C (NE)" for his/hers EP a new or emerging researcher would normally be expected, during the assessment period in question, to have produced a reasonable platform of research, as evidenced by having a) completed her/his doctorate or equivalent qualification, AND b) produced at least two quality -assured research outputs' OR c) produced research output equivalent to a) and b).
Quality Category R or R (NE)
An "R" or "R(NE)" will be assigned to an evidence portfolio that does not demonstrate the quality standard required for a "C" Quality category of higher or, in the case of a new or emerging researcher, the standard required for a "C (NE)" Quality Category or higher


Australian Research Council criteria and guidelines to assessors


Rating
Descriptor
90 - 100
Outstanding: Of the highest merit, at the forefront of international research in the field. Less than 2% of applications should receive scores in this band.
85 - 90
Excellent: Strongly competitive at international levels. Less than 20% of all applications should receive scores in this band.
80 - 85
Very Good: An interesting sound, compelling proposal. Approximately 30% of all applications should receive scores in this band. Approximately 50% of all applications will have scores above 80.
75 - 80
Good: A sound research proposal, but lacks a compelling element. Approximately 30% of all applications are likely to fall into this band.
70 - 75
Fair: the proposal has potential, but requires development to be supportable. Up to 20% of all applications are likely to have a score below 75.
0 - 70
Flawed: the proposal has one or more fatal flaws.


Source: ARC 'Discovery Projects, Linkage Projects: Assessor Handbook'


CSIRO Science Assessment Review. A variety of information will be assessed by the review panel, including research work, and information about the division including research plans, and significant outcomes. There are two categories of benchmarks used

Research community position


Rating
Descriptor
Benchmark
Sustained scientific leader- well recognised in the international research community for this.
Strong
Able to set and sustain new scientific/technical directions within the international research community.
Favourable
Able to maintain a good position in the international research community "pack": not a scientific leader except in developing niches (not mainstream areas).
Tenable
Not able to set or sustain independent scientific/technical direction - a sense of being continually a follower.
Weak
Declining quality of scientific/technical directions output compared with other research groups. Often a short term "fire-fighting" focus.
0 - 70
Flawed: the proposal has one or more fatal flaws.




Industry/community Position:
Rating
Descriptor
Benchmark
Research results used to set the pace and direction of technically-based commercial, environmental, community or policy development - recognised in industry or community for this.
Strong
Research results able to be used by organisations to distinguish themselves form peers or competitors.
Favourable
Research results able to be used by organisations to improve their position relative to peers or competitors.
Tenable
Research results able to be used by organisations to maintain, but not improve, their position relative to peers or competitors. Research results not able to be used to differentiate organisations from their peers or competition.
Weak
. Research results not able to be used by organisations to even maintain their position relative to peers or competitors.
0 - 70
Flawed: the proposal has one or more fatal flaws.


Source: Department of Education Science and Training, (2005) 'RQF Issues Paper', pg 57




Australian Nuclear Science and Testing Organisation Ratings Scale
Rating
Descriptor
World Class
Sustained leadership and accomplishment exemplified by outputs that influence highly original work that ranks with the best of its kind. Within the top 5% of the field internationally.
Very Significant
Making a significant contribution to the field. Within the top 10% of the field internationally.
Significant
Demonstrating a platform of research output that has generated substantial new ideas, interpretations or critical findings and makes an internationally recognised contribution to existing paradigms and practises, within the top 25% of the field.
Contributing
Demonstrating a platform of research activity (or developing research activity) and output that is based on a sound/justifiable methodology, and makes a contribution to the research within the discipline and/or to applied knowledge.
Limited
The research outputs will have been assessed as having either limited or no significant impact, contribute little or no additional understanding or insight in the discipline or field, and/or is considered to be lacking in appropriate application of theory and/or methods.


Source: RQF Submission No. RQF010165


UK Research Assessment Exercise,2001
Rating
Descriptor
5*
Quality that equates to attainable levels of international excellence in more than half of the research activity submitted and attainable levels of national excellence in the remainder.
5
Quality that equates to attainable levels of international excellence in up to half of the research activity submitted and attainable levels of national excellence in the remainder.
4
Quality that equates to attainable levels of national excellence in virtually all of the research activity submitted, showing some evidence of international excellence
3a
Quality that equates to attainable levels of national excellence in over two-thirds of the research activity submitted, possibly showing some evidence of international excellence.
3b
Quality that equates to attainable levels of national excellence in more than half of the research activity submitted.
2
Quality that equates to attainable levels of national excellence in up to half of the research activity submitted.
1
Quality that equates to attainable levels of national excellence in none, or virtually none, of the research activity submitted.


Source: Roberts G., (2003) 'Joint Consultation on the review of Review of Research Assessment', HEFCE pg 79



UK RAE 2008 Quality Profile
Rating
Descriptor
Four star
Quality that is world-leading in terms of significance and rigour.
Three Star
Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence
Two star
Quality that is recognised internationally in terms of originality, significance and rigour.
One star
Quality that is recognised nationally in terms of originality, significance and rigour.
Unclassified
Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purpose of this assessment.


Source: RAE 2008 (2005) 'Guidance to panels' HEFCE pg25


By comparison the UK and the Australian Research Council frameworks do not contain the same heavy focus on peer ranking of the New Zealand, Australian CSIRO and ANSTO models. The United Kingdom model, on the face of it, appears to be more highly transparent in its ratings and objectives and to recognise that the research, rather than the person's peer status, is the prime focus.

The 'value for money' notion (espoused in the Australian and United Kingdom governments' policy approaches) is an extremely important one. However it is one that needs to be balanced carefully against the question of absolute quality using some fathomable 'fitness for purpose' definition, whether it be commercial or simply learning for learning's sake to separate man from the rest of the animal kingdom. The rating systems structures are hierarchical. They do not take into account the proposition that some at the lower level of the structure might actually outperform (in one or more significant social, economic and commercial contribution terms) and also in terms of value for money, those in the upper levels that are more prestigious and costly.



Topley (op. cit) says of the UK rating system, "in a relative sense, we imagine it would not be difficult to demonstrate that some 3a rated departments out-perform some 5* departments when value for money alone is considered, simply because they cost so little. Indeed, in a rather trivial way, all 3b rated departments that produce output out-perform 5* departments in 'value for money' simply because they cost nothing! Clearly, some absolute standards of quality are also necessary. Unfortunately, the current system purports to supply the latter (but see several comments elsewhere) but makes no attempt to measure the former. Thus what is important is not how one might establish the 'baseline ratings' - there are several ways that this could be done. Rather, the key lies in what expectations one can define and reasonably impose upon any given baseline rating, in order in the future to measure whether they are, or are not, out performing the 'value for money' targets. This could be achieved through self-assessment and self-declared target setting, judged and calibrated by expert review".

The question is who should undertake an expert review within the context of "value for money" and contribution to Australia's national interest? Should it be the academics that directly benefit, as a group, and who are conflicted or is it a separate and independent body made up of a cross section of the community, who take advice and make decisions that are written and challengeable, prior to the actual grant of monies? The tendency is to limit democratic participation in favour of the "expert".


"Peer review system is flawed, scientists say", January 18 2003, The Guardian Newspaper in the United Kingdom.

A report from an international collaboration of scientists argues that he time-honoured system of peer review, which has existed in some form for at least 200 years, is possibly bunk. Tom Jefferson, of the Cochrane Collaboration Methods Group, said: "If peer review were a new medicine, it would never get a licence. "We have found little empirical evidence to support the use of peer review as a mechanism to ensure the quality of research reporting, and there is even more depressing evidence about its value in deciding what should be funded." The study focused on biomedical research, but there "was no reason to assume that the inefficiency of this system would not pertain across other scientific disciplines". Dr Jefferson's team scrutinised 135 studies designed to assess the evidence that peer review was an effective method of deciding what should be published. We had great difficulty in finding any real hard evidence of the system's effectiveness, which is disappointing, as peer review is the cornerstone of editorial policies worldwide," he said. "Scientists compete with each other for space to publish in the most prestigious and most widely read journals, space is allocated by editors, and peer review plays a big part in the process. "Publishing is the key to advancement and research riches. Nobel prizes have hinged on peer review, yet it may be seriously flawed. The problem is compounded because scientists can't agree about how the quality of peer review should be measured." Dr Jefferson's team is calling for a large, well-funded program of research on the effects of peer review. But does their work cut the mustard? Asked whether it was peer reviewed, Dr Jefferson said: "Yes, and it was done through collaboration rather than in the adversarial way."

PEER REVIEW is not perfect, and when it is done sloppily, journals publish research that is flawed. Even when peer review is rigorous, flawed research sometimes gets into the literature.

Critics of peer review claim that it is inherently conservative and biased towards conventional work, and discourages innovators or original thinkers. Some even go further by contending that the refereeing process is corrupt, subjective, and clouded by professional jealousy, which prevents good research from seeing the light of day. Unfortunately, examples do exist where the peer review process has failed. We can only guess how Jenner felt 200 years ago, when his account of the first use of a vaccination against smallpox was rejected by the referees. We tend to forget that just because all scientific work is peer reviewed before publication, it does not mean that peer review is itself
a scientific activity in the sense that it is precise, and free from bias and subjective judgement. It is therefore surprising that we have such an uncritical faith in the peer review process. Does agreement between two referees tell us anything more than that both referees share the same set of prejudices?

The traditional methods of scholarly publishing and peer review do not live up to
the needs of efficient communication and quality assurance in today's rapidly developing and highly diverse world of science. A large proportion of scientific publications are careless, useless, or false. Furthermore, they inhibit scholarly communication and scientific progress. This view may sound provocative but unfortunately it is not exaggerated (Pöschl, 2004a,b; and references therein).