Privacy Statement, Uses and Motivation
|
Privacy Statement, Uses and Motivation |
MOSAIC PORTAL SITE MAP
 | The specific category pages of the Mosaic Portal includes the latest news about your work, profession, hobby or interests and lots of other relevant stuff? Eliminate the tiresome irrelevancies of a general web search by using the specific categories of the Mosaic Portal and let the Mosaic Portal bring it to you |
A deep extended index of subjects and subtopics where the research, and search, has been done to find sites with news, articles and resources about your work, study, family, lifestyle, interests, hobbies, entertainment, beliefs, and more
|
|
IMPORTANT
every portal page uses scripting to update content.
So turn your active content on.
|
|
|
|
A Recommended framework for Research Quality Assessment in Australia The Minister for Education, Science, and Training in Australia commissioned a panel of eminent experts to develop a proposal for a Research Quality Framework. The committee published its preferred model in September 2005, and called for public comment. If the purpose of the framework is to provide a process to assess research solely on behalf of the universities themselves, then the proposed form of peer review, inherent in the recommended Research Quality Framework is adequate. If the assessments are undertaken in order that the body, and its members, can be seen to have discharged their obligations under the legislative Act, again the proposed system, of peer dominated determination, is adequate. If, conversely, the assessment is intended to serve government or taxpayer, in terms of tapping the innovation and talent of all aspirant researchers, including those in the private sector, forging relationships and synergies, then it is inadequate. Chris Topley, the Deputy Vice Chancellor Research and Development at the London Metropolitan University commenting on the United Kingdom proposals raises points that are applicable to the Australian recommendation. "The proposed methodology measures the goodness and badness of research by a set of standards devised by those being measured (university academics); and the judgements are made by panels largely consisting of those being measured (university academics). The outcome is that good research (of the type that academics choose to define as good research) is well funded, while bad research (again as defined by academics) is not. If the fundamental purpose of research assessment is to promulgate this somewhat introspective view of good and bad research, then the present system is not only adequate, it is all but ideal. Most metrics systems, which perhaps theoretically could get around this question of introspection, in fact come up against the same difficulty. Citations are largely cited by academics; g rant applications are largely judged by academics; drafts of papers are largely refereed by academics, and so on. Academics are not necessarily the best qualified to judge whether they are giving good value in their work to the government or the taxpayer/voter. Nor, rationally, are they necessarily in the best position to judge how funds should be used in support of research, unless one is prepared to accept that the production of good research, as defined by those same academics, is the best use of funds. There is a significant and serious danger of a self-fulfilling prophesy in such an argument, and many believe that this is precisely the position in which UK Higher Education now finds itself".(Chris Topley, Deputy V-C Research and Development, Research Strategy Committee, London Metropolitan University, 28 November 2002). The Australian Minister for Education and Training the Honourable Brendan Nelson MP's says in a media statement, "The features of assessment described in the Paper reflect a desire to identify high quality and high impact research in an international context, while also recognising the research priorities for Australia in the future." The Minister's Foreword in the Australian Research Quality Framework draft says inter alia: "While being the best in Australia is an achievement, Australia's future depends on a collective effort by the Australian Government, industry and the community in identifying areas of international research excellence in Australia. Many of the features outlined in this paper, including how the assessment mechanism will be structured, the membership of assessment panels and the kind of assessment criteria and guidelines required for the RQF reflect this desire to focus on excellence in research quality and impact". How is a collective effort to be achieved when the Research Quality Framework put forward for publicly funded research is limited to universities and other publicly owned entities such as the CSIRO and (public - private incorporated) Cooperative Based Research Centres? Privately funded research entities appear to be excluded. Scope According to the proposed model before the Minister, the RQF will include all publicly funded research conducted within Australia's university sector and Publicly Funded Research Agencies. How are private sector and not for profit philanthropic research facilities to be drawn into the Australian framework of research excellence implied in the Minister's statements? Is it assumed that the bulk of research monies go to universities? The panel recommending the model to the government defines the quality and impact criteria that are to be the primary objective of the framework. Quality and Impact "The quality of research including its intrinsic merit and academic impact - academic impact relates to the recognition of the originality of research by peers and its impact on the development of the same or related discipline areas; and its broader impact or use. That is the extent to which research is successfully applied - broader impact or usefulness relates to the recognition by qualified end-users that quality research has been successfully applied". It is not clear who the "qualified end users" might be? The prominence of "recognition of academic impact by peers" is a feature of the preferred model and this is not surprising given the constituent make up of the Advisory Committee. The notion of peer review is deeply embedded in academic systems of all kinds and is generally vigorously defended by academics. It seems absolutely right that 'peers' should indeed be those that judge the merit of anything that is itself wholly academic in nature, since no others are better qualified. Thus, as long as the assessment of research is viewed as an exercise in judging the merit of research from a wholly academic standpoint, there ought to be no serious challenge to the peer review system. However, if it is the government's intention that public research should serve e a public or commercial good, outside of the narrow academia utilisation then peer review and control is likely to stymie this objective. The Australian government policy statement, "Backing Australia's Ability - Building Our Future Through Science and Innovation," includes a Message from the Prime Minister, John Howard, which answers the question of intention. "The Australian Government's goal is for Australia to build a world-class innovation system. This ambitious agenda depends on effective partnerships between governments at all levels, researchers and business, to share the substantial financial investment necessary and to ensure that ideas move smoothly from generation to end use. Mapping Australian Science and Innovation, released in late 2003, and a range of policy reviews and programme evaluations have provided the factual basis for policies, which will assist Australia to achieve even higher levels of performance. These policies will expand our capacity to deliver tangible results from our science and innovation system for advancing Australia's international competitiveness and productivity and creating jobs. This Government is the first to develop National Research Priorities to align Australia's research effort with the community's economic, social and environmental goals. Through Backing Australia's Ability - Building Our Future through Science and Innovation, we are further encouraging Australia's research and business communities to work together in pursuit of those goals". This implies something well beyond a system of dominated peer review solely for serving university academic research. The Prime Minister talks of a policy that involves collaboration, transparency, and broad assessment of what is valuable. Underlying principles The committee states that the proposed RQF will be underpinned by the following principles: Transparency: process and results are clear and open to stakeholders, including the use of reliable/repeatable measures/metrics; Acceptability: broad acceptance of the approach and measures; Effectiveness: the applied model achieves the purposes of valid and accurate assessment, and avoids a high cost of implementation and the imposition of a high administrative burden; and Encourages Positive Behaviours: improving the quality and impact of research and further developing and supporting a vibrant research culture in Australia. These are considered motherhood statements in that they do not define how these principles will "develop and support a vibrant research culture" beyond the academic walls of the participating institutions. Transparency may be obvious to those who work in the higher education sector or are closely associated with it. However, to the general community (taxpayer) and to the private sector and others seeking to win research grants from the available pots of money, transparency has not been a hallmark of the operation of the assessment of research within the Departments of the Australian government and the decision makers (academics) who serve on the assessment committees. Further Australian universities, and their obsession with peer structures, are not known for their transparency in the operation and decision-making within their Universities. Peer methodologies are precisely what they imply: hierarchical, and egotistical, in design and operation. The positive behaviours goal is an outworking of the peer pressure in institutions to "shape" the behaviour of academic researchers and to ensure compliance. Mavericks (often the creators of real innovation) are reigned in, brought to heel or perhaps ostracised and excluded. In 2000, the Australian government allocated $50,000,000 (over a number of years) to CSIRO for R&D in light metals. The CSIRO work was to be utilised in the commercial application of the Australian Magnesium Corporation production of magnesium, alloys, and later thin sheet metal. There was much debate, recorded in Senate Estimate hearings and departmental and parliamentary records of correspondence on the "exclusivity" of this funding and its application. The record of the Senate can be found in Estimates hearings of January - February 2002 and in correspondence to then Ministers Senator Nick Minchin and Peter McGauran as well as other parliamentarians. The papers in the Senate clearly show that the "peer mechanism' and procedures regulated by the government were not followed and that the R&D Board did not receive quality advice and assessment in acco5rdance with those regulations. External critics (to the committee of expert peers) argued that the process used by the government, through a Research and Development Board, of assessing and funding research was not transparent. They argued against all the expert advice that the AMC project would not be viable and that CSIRO would not meet its flagship objective. The peer process for research decisions was confronted. Rather than look at whether there was any substance the membership of the academic - industry expert group, continued to maintain their line to the government, placing their eggs (so to speak) in one very expensive basket, over which they had unchallengeable control. In the years 2000 - 2003 the writer of this article travelled in Australia, Europe and the United Kingdom, in the USA and Canada, meeting with senior researchers in public and private sectors, in universities and private research facilities including the Australian Cooperative Research Centre established for this application development in Queensland. Also meeting with manufacturers and end users in the automobile industry in Detroit USA and Stuttgart Germany and with senior scientists and managers in Biliton London and BHP, Melbourne. The consensus of all of these meetings particularly internationally was that the development of a magnesium industry on the scale published in the Australian governments Light Metals Agenda would not eventuate. The writer, up until 2004, continually conveyed this view to the Australian government. Face to face, meetings with Minister Peter McGauran again took place. He arranged a meeting with the relevant manager of what is called a "flagship" programme within CSIRO, to no avail. In defence of the politicians, it must be said that it is a leap of faith to cast aside the advice of experts. All the experts said that the Light Metals Agenda, and government initiatives and the flagship magnesium projects and research were solid. Ford invested $40,000,000 in return for sales take outs of 50% of the production. The market agreed and shareholders invested about $1.4 billion, with $150,000,000 coming from the Queensland government. The federal government even proposed, but did not finally give, a $200,000,000 loan to the magnesium industry. In 2004, the Australian Magnesium Corporation project to manufacture metal for use by Ford and others worldwide, collapsed. What is not clear to the writer is how the group advising the government and engaged in the research could have got is so wrong when so many international sources available to the writer were giving contrary views. The only conclusion reached is that the primary objective of the parties was to carry out the research and to build a production plant. Beyond that, there seemed little recognition, or knowledge of, market and competition potential and realities. It may be that the "peer process" in science, commercial and other spheres of society and economy are self reinforcing. Questions as to what CSIRO was doing and whether R&D is still being undertaken on, that project in concert with what is left of AMC goes unanswered by the government and CSIRO. Where is the transparency in that application of public monies into a publicly funded research entity, CSIRO? Where is the effectiveness of the investment demonstrable and its application to innovation in Australia? The public interest and broader tests implied in the Prime Minister's statements above were not served. In 2005 the Minister for Education, Science and Training, the Honourable Brendan Nelson abolished the Research and Development Board. Whilst the above example is not precisely relevant to the determination of broader university academic research funding (even though CSIRO is included in the framework and the CRC included a university in Queensland) it does demonstrate that that the major purported strength, and influence, of the 'expert' approach is its familiarity and security, trusting well tried experts. Its major weakness is its tendency to become self-defining in that what the experts think is good is what is funded. Views to the contrary are dismissed if they are not considered expert and the critic/s is, are, not considered expert. There is no doubt that the proposed Research Quality Framework will be a useful political tool for relying upon "experts" and thus ceding ultimate accountability to a third party. However, it is beyond reasonableness to suggest that it measures anything of transparent value to the world outside academe and university management, government beyond how much money has been spent. It is unlikely that the public, or competing interests and enquiring minds will ever be told on precisely what, in detail, it has been spent. Where is the process, defined ihe framework of publishing what units of funds are spent on what elements of research within what disciplines of Australia's publicly funded universities? Will the researchers point to an article in a peer-reviewed journal? One assumes that researchers do report to the entity that grants them the funds. However there is a record of avoiding disclosure of the expenditure if questioned in any depth or detail. The committee appears to be recommending that reportin g will be on the basis of metrics, which is, the common method used in Australia. Metrics has the danger of promoting certain shaping and behavioural modification particularly in relation to disclosure and reporting and the time spent on these activities as against actual research. "Measuring research performance entirely on metrics is a flawed approach. Such a system cannot measure the future potential, or even attempt to do so. Virtually all metrics are tedious and administratively expensive to compile, and most metrics systems tend to 'channel' researchers into pursuing the metrics for their own sake, rather than doing what they are best at, in their own best way. Available metrics can perhaps be combined to provide an accurate picture of the strengths that those metrics purport to measure, retrospectively. They cannot be prospective, and, because they are 'metric', they frequently undervalue such matters as ethos, environment, strategic vision, breadth, connectedness to other disciplines etc. They are also particularly unsuited to research in the Arts and Humanities. As to the behavioural effects of metrics; there is no question that behaviours would change so as to pursue the rewards. Strengths of such a system, apart from simplicity, are few; the much-vaunted objectivity is s eriously open to question, and the data simplicity is a myth. Weaknesses are many, including the unfortunate behavioural effects and the danger of reductio ad absurdum".(Topley, op cit). The proposed Quality framework is a model to ensure maintenance of control of the direction of, and the participants in, Australia's publicly funded research sector by a very specific group of interests and participants. It is about getting block funding which is then allocated according to peer ranking, prestige and perhaps even patronage. This is evidenced by the following extracts from the recommendation to the Minister. "Panels representing a broad discipline area should consist of approximately 12-15 members each, and include: Academic peers and other expert reviewers International reviewers will comprise approximately 50% of the membership for the RQF Assessment Panels. Panel Chairs will apply guidelines to determine their membership, particularly with reference to the international membership, according to the discipline/s needs". The framework embeds ranking, exclusivity, and patronage of particular researchers and the establishment of a system that allows university heads to decide (through bock grants) who will get the money and for what. "Development of the RQF: Progress to-date on agreed features Drawing on the evidence from overseas and Australian research quality assessment exercises; feedback received in response to the RQF Issues and Advanced Approaches Papers; and outcomes from the detailed work of three EAG Working Groups, the EAG agreed that the RQF should have the following features: It should enable comparisons to be made of the research excellence between universities and PFRAs. However, funding pools for universities and PFRAs should be kept separate. Should focus on research quality whose value would be related to its impact and would include assessment of its impact. Institutions should decide, against a set of guidelines, which researchers would be assessed in the RQF. Research groupings will be the subject of assessment in the RQF. Research groupings may be either identified Research Groups whose work may cut across disciplinary boundaries, or aggregations of individuals who do not necessarily work together but whose work fits within a common broad Research Fields, Courses and Disciplines (RFCD) classification. Nominated research groupings' research outputs should be placed into context through statements from institutions. Assessment panels should be established for the RQF; and these panels should be structured on broad-disciplinary lines based on aggregations of the ABS RFCD classifications. Panels should have explicit procedures for assessment of cross disciplinary research. Representation on assessment panels should consist of a majority of academic peer-reviewers with additional research end user reviewers, as appropriate. The RQF should have a graded points system/rating scale eg 0 to 4, 1 to 5, etc. which is sensitive to discipline differences. It should be structured in a way, which enables assessments of quality to be provided at multiple levels, for example, by broad discipline area, Faculty or other structural unit up to institutional-level. There is to be no reporting at the level of individual researchers. Funding allocated to institutions on the basis of the RQF should take the form of block grants for which institutions have discretion to determine their internal distribution." The extracts appear to support a proposition that this framework enshrines institutional systemic power and control, limiting participation to "academic peers", "graded lecturers" of a certain status and "control mechanisms of "grouping" and "creating aliases for researchers by grouping", particularly for those people whose status is not yet recognisable to the academic world. This places the research under the broad, and non-transparent umbrella, of the university of entity, carrying out the research. The framework limits participation. "Eligibility for assessment in the RQF The draft recommendation inter alia, says that: "eligible academic staff are to be nominated by universities for assessment should hold university Level B and above positions. Academic research staff whose positions are funded through individual competitive research grants may also be nominated for assessment, for example postdoctoral fellows holding Level A positions. Individual researchers will be nominated by universities as members of a 'research grouping' A research grouping can be either an aggregation of individual researchers who do not necessarily work together but whose work fits within a common broad RFCD classification, or an identified Research Group in which individual researchers are working together on research that may cut across disciplinary boundaries." The recommended framework appears to contravene the Government's desire to open up research to an innovative broader collaboration between creative individuals, industry, community, and institutions. The suggested framework is also about volume and frequency, a part of the peer status process of universities. Evidence portfolios for assessment "Universities will provide evidence portfolios of individual researcher's outputs within nominated research groupings. To be included for assessment, two conditions must be met. Each individual researcher included in a research grouping must have produced at least four research outputs over the assessment period." The Minister for Education, Science and Training in a number of his policy speeches has raised the issue of educational excellence and prestige not necessarily needing to be tied to the number of papers a researcher has had published. Whilst the Minister has indicated that this is collaboration with industry, subtly the university peers seek to ensure minimisation of the impact of external parties upon their deliberations and decisions. Here is a reference to limiting the scope of industry consultation and formal linkages. "The evidence of research impact should be verifiable. Consequently, and as in the assessment of research quality, the expert advisory group (EAG) agreed that research impact should be retrospective and demonstrable rather than prospective and potential. It should not focus on mechanisms and structures designed to increase the likelihood of impact (such as industry consultation or formal linkage arrangements), but rather on the outcomes of such arrangements". The rating system recommended in the framework appears to be a mechanism to ensure control limiting assessment criteria within a narrow peer group of the relevant disciplines of the academic world. "The EAG recommends separate criterion-referenced ratings for research quality and research impact. It recommends that research quality should be rated on a five point scale. There is clear acceptance that the top ratings will be based on notions of "international excellence" - outstanding research achievements on an international scale". Who decide what the five-point scale is and who will be rated according to it? A number of existing Australian, and international models, were examined by the EAG. One can note a consistency between members of the academia regardless of location or type. All reinforce the "prestige of being published" and the merit of international recognition. None actually require that the research be demonstrated to be able to, or to have in retrospect, have delivered a public good as the primary consideration. Rather the research's objective is to add to the body of collective knowledge. |
| Rating | Descriptor |
| The majority of research outputs were considered to be in at least the top 20% of research in its field internationally, with a significant percentage (>25%) in the top 10%. There was evidence of high international peer esteem and significant impact on the international academic community. |
|
If the only purpose of assessing research was to measure, in hindsight,
how well it supported the economy, and the deliverable benefits to Australian citizens, one wouldn't seek to know which were the best publications attached to each academic author in international journals, what the peers thought and how the international academic community was impacted. One might seek information as to how the research impacted the number of contracts won by commercial or non-profit entities using the research and actually delivered, the technology transferred, knowledge transferred, and the clear contribution to humanity in some form or other. Such data may not be objective, but they are closer to the purpose than the descriptive table above, proposed by the EAG. In reality, one expects that the assessment might be for multiple purposes, which ought to lead to the citing of different kinds of objective data, not the very narrow interpretation and shell in the box above. The designers of the recommended Australian Research Quality Framework have chosen a nu,ber of Australian and international examples of rating systems that might be used. They all reinforce the "peer control." With great respect to the expert advisory group members and the academic world, the funds for research come from the community at large. The greater numbers of taxpayers are predominantly laypersons, as are the politicians they elect, and they may not find the descriptors all that illuminating. The question is how will members of parliament; particularly the cabinet, approving and funding the framework decipher the jargon below, which is meaningful, and measurable, only to the closeted world of academia. |
|
Possible assessment scales and descriptors New Zealand Quality Categories and Criteria Peer Esteem Assessment Criteria |
|
|
|
| The evidence portfolio would be expected to demonstrate that the staff member has attracted world-class recognition through her or his research. This could be reflected by some or all of the following: the receipt of prestigious prizes, or fellowships of leading learned societies/academies or prestigious institutions, or special status with professional or academic societies, or editorship, membership of editorial panels or referees of top-ranked journals, or awards for research as well as invited attendance, or examination of PhDs and presentation at prestigious academic and industry conferences/events. An ability to attract overseas/top research students and scholars as well as to mentor his/her own students into postdoctoral and other fellowships, scholarships and positions in centres of research excellence could be demonstrated in the evidence portfolio. A consistent record of favourable citations of research should combine with strong evidence of positive research reviews and contribution to knowledge in the discipline (including overseas where relevant), and movement into creative practice. | |
| The evidence portfolio shows that the staff member, through their research, is recognised within New Zealand or elsewhere and is esteemed beyond the researcher's own institution. The evidence portfolio demonstrates peer esteem by providing evidence of some or all of the following: the receipt of prizes, membership of a professional society or similar with restricted or elected membership or honours or special status with professional or academic societies, editorship or membership(s) of editorial panels of reputable journals within New Zealand or elsewhere, research fellowships of esteemed institutions, reviewing of journal submissions and book proposals, PhD examination or invitations for keynote addresses for conferences/events that are at a middle level of excellence. A consistent record of research citation and positive reviews of specific research outputs and/or overall contribution to research knowledge in a discipline or substantive area of knowledge or practice can be expected. The evidence portfolio could demonstrate graduate students moving into research scholarships or postdoctoral fellowships or junior lectureships in departments with good research ratings. | |
| The evidence portfolio demonstrates a developing recognition among peers of the staff member's research contribution and developing rigour in the application of research techniques. This may be evidenced through attracting awards and invitations to present research to informed audiences, within and possibly beyond the applicant's immediate institution, as well as positive reviews and citations, or being asked to referee research outputs. Where the staff member has an involvement primarily in commissioned research outputs, reference to letters of commendation or other evidence of esteem by commissioning agents could be expected. | |
| Minimal evidence of peer esteem generated through research activities | |
| No evidence of peer esteem generated through research activities. |
|
New Zealand Quality Categories and Criteria Overall Quality Category |
| Rating | Descriptor |
| To be assigned an "A" for his/hers EP it would normally be expected that the staff member has, during the assessment period in question, produced research outputs of a world-class standard, established a high level of peer recognition and esteem within the relevant subject area of his/her research, and made a significant contribution to the New Zealand and/or international research environment. | |
| To be assigned an "B" for his/hers EP it would normally be expected that the staff member has, during the assessment period in question, produced research outputs of a high quality, acquired recognition by peers for her/his research at a national level, and made a contribution to the research environment beyond her/his institution and/or significant contribution within her/his institution. | |
| To be assigned an "C" for his/hers EP it would normally be expected that the staff member has, during the assessment period in question, produced a reasonable quantity of quality-assured research outputs, acquired some peer recognition for her/his research, and made a contribution to the research environment within her/his institution. | |
| To be assigned an "C (NE)" for his/hers EP a new or emerging researcher would normally be expected, during the assessment period in question, to have produced a reasonable platform of research, as evidenced by having a) completed her/his doctorate or equivalent qualification, AND b) produced at least two quality -assured research outputs' OR c) produced research output equivalent to a) and b). | |
| An "R" or "R(NE)" will be assigned to an evidence portfolio that does not demonstrate the quality standard required for a "C" Quality category of higher or, in the case of a new or emerging researcher, the standard required for a "C (NE)" Quality Category or higher |
|
Australian Research Council criteria and guidelines to assessors |
|
|
|
| Outstanding: Of the highest merit, at the forefront of international research in the field. Less than 2% of applications should receive scores in this band. | |
| Excellent: Strongly competitive at international levels. Less than 20% of all applications should receive scores in this band. | |
| Very Good: An interesting sound, compelling proposal. Approximately 30% of all applications should receive scores in this band. Approximately 50% of all applications will have scores above 80. | |
| Good: A sound research proposal, but lacks a compelling element. Approximately 30% of all applications are likely to fall into this band. | |
| Fair: the proposal has potential, but requires development to be supportable. Up to 20% of all applications are likely to have a score below 75. | |
| Flawed: the proposal has one or more fatal flaws. |
|
Source: ARC 'Discovery Projects, Linkage Projects: Assessor Handbook' CSIRO Science Assessment Review. A variety of information will be assessed by the review panel, including research work, and information about the division including research plans, and significant outcomes. There are two categories of benchmarks used Research community position |
|
