Engineering

EVALUATOR S GUIDE BOOK

Categories
Published
of 33
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Description
ROMANIAN RESEARCH ASSESSEMENT EXERCISE (RRAE) EVALUATOR S GUIDE BOOK Domain Electrical engineering Content 1. Introduction The Evaluation Methodology The Electronic Evaluation File...10
Transcript
ROMANIAN RESEARCH ASSESSEMENT EXERCISE (RRAE) EVALUATOR S GUIDE BOOK Domain Electrical engineering Content 1. Introduction The Evaluation Methodology The Electronic Evaluation File Evaluator s work guidelines...11 Stage I: online Stage II: in panel Panel Coordinator s work guidelines...13 Stage I: online Stage II: On-site Visits and Panel Meeting Stage III: Finalizing the evaluation process and the classification of the universities on the domain Bibliography...14 Annex 1. Particularization of the quality levels meaning according to the specific methodology of the domain Electrical engineering...15 Annex 2. Panel Meetings Schedule...23 Annex 3. Templates for Reports generated during RRAE...24 Evaluation form...24 On-site evaluation form...32 Panel report...35 Final ranking report...43 Annex 4. Glossary of terms used in RRAE 1. Introduction ROMANIAN RESEARCH ASSESSMENT EXERCISE (RRAE) is the essential component of the Doctorate in Universities of Excellence Research Assessment and Support for Scientific Publishing strategic project, a project financed by The Sectorial Operational Programme for the Development of Human Resources (SOPDHR) and undertaken by The Executive Agency for Higher Education, Research, Development and Innovation Funding (EAHERDIF) between The Exercise s objective is the quantitative and qualitative evaluation, with regards to specific scientific domains and international standards, of the scientific research conducted in Romanian universities. This assessment is based on The General Methodology [1], which is further divided by domains in The specific guide books of the evaluation domains [2], developed in the project, through the considerable involvement of the national academic community and with the endorsement of an international panel of experts. At the end of The Exercise for each of the forty two domains of scientific research identified in the project, a hierarchy of Romanian universities will result based on the obtained research performances. RRAE results will permit formulation of legislative proposals regarding research financing to be strongly connected to the achieved performances and to the perspective of sustainable development of the universities. This perspective will stimulate the competition between universities, the participation of Romanian researchers in the international networks of research, and the increase of visibility and academic prestige in a global context. At the same time, The Exercise itself proposes to identify the universities whose potential and strategic programme can generate excellence, thus backing the realization of the Excellency Universities in Romania programme, a programme elaborated in the frame of the present project [3]. The Assessment Electronic Platform (Romanian acronym SISEC) is the informatics infrastructure, developed in the project. SISEC follows three main objectives: (i). to mediate the introduction by universities of the information on which the evaluation is based (Data gathering module); (ii). to provide informatics support for the qualitative evaluation in a peer-review system (Evaluation module); (iii). to generate reports on the scientific research conducted in the universities in formats requested by various stakeholders (Reporting module). 3 The platform will allow the periodical monitoring of research results and the identification of excellence groups whose financial support will contribute to the increase of visibility of Romanian scientific research. As the qualitative evaluation will also be conducted by foreign experts, all the descriptive fields from SISEC will be filled out in English by researchers and domain coordinators from the universities. The structure of the actual Evaluator's Guide Book is the following: in Chapter 2 (Evaluation Methodology) the evaluated domains, the four criteria and the accompanying descriptors are shown, next to the duties of participants in the evaluation process; in Chapter 3 (Electronic Evaluation File) the structure of The electronic file generated by SISEC on the basis of the information introduced by the universities and subjected to both evaluations: quantitative (performed automatically) and qualitative (through peer-review activity of the Romanian and foreign evaluation experts), is shown; in Chapter 4 (Evaluator s work Guidelines) the steps followed by the evaluator to access SISEC and to perform the qualitative evaluation of the files which are automatically allocated by the electronic platform, are shown in detail; in Chapter 5 (Panel Coordinator s Work Guidelines) the responsibilities and the stages followed by the panel coordinator in RRAE are shown. Further, after the Bibliography section, the document contains a detailing of the significance of quality levels specific to each domain, a particularization undertaken by the Evaluation panels for each domain in the process of elaborating the Evaluation Methodology (Annex 1), the scheduling of meetings of the Evaluation panels (Annex 2), Templates for Reports generated during RRAE (Annex 3) and, finally, explanations of the terms used in RRAE Glossary of terms (Annex 4). 4 2. The Evaluation Methodology The elaboration of evaluation methodology of scientific research in Romanian universities, on the basis of which the current Research Assessment Exercise takes place, was conducted between December 2008 and April 2010 and is presented in detail on the Web page of the project [1 and 2]. The methodology is based on a peer-review evaluation process, with national and foreign evaluators, being an evaluation by research domains. The taxonomy of the specialty domains is intended to provide a referential framework to The Exercise, highlighting affinities and connections between domains, structured in Domains groups and which must be treated consistently by the evaluators. The 42 domains used in RRAE are listed below: Group I - Natural Sciences 1. Mathematics 2. Informatics 3. Physics 4. Chemistry 5. Geology and geography Group II- Engineering Sciences 6. Civil engineering and installations 7. Mechanical engineering and mechatronics 8. Aerospatiale engineering 9. Transportation 10. Chemical engineering 11. Materials science 12. Oil, gas and mines 13. Industrial engineering 14. Electrical engineering 15. Energetics 16. Electronics and telecommunications 17. System engineering 18. Computers and information technology 19. Biotechnologies, food security and engineering 20. Environmental sciences Group III - Social and Economic Sciences 21. Law and administrative sciences 22. Economic sciences 23. Military sciences, security and information 24. Political sciences and international relations 25. Communication and media 26. Sociology, anthropology and social assistance 27. Psychology 28. Education science 29. Sports Group IV - Human Sciences 30. Philosophy 31. History 32. Theology and religious studies 33. Philology Group V - Arts and Architecture 34. Cinematography and performing arts 35. Music 36. Visual arts 37. Architecture and urbanism Group VI Life Sciences 38. Biology 39. Agriculture and forestry 40. Veterinary medicine and zootechny 41. Medicine 42. Pharmacology 5 In the current Romanian Research Assessment Exercise the following four criteria are used [1]: Criterion I: The results obtained in the activity of scientific research; Criterion II: The environment of scientific research; Criterion III: The prestige in the academic community; Criterion IV: Financial resources brought for the scientific research. Each of these four criteria has a specific weight and a set of descriptors, as follows. CRITERIA I. The results obtained in the activity of scientific research(scientific Output) (60 70 %) Maximum 3 indicators DESCRIPTORS Articles: Publications rated Web of Science; Published in proceedings of scientific events; Magazines from international data base. Scientific books of author and chapter books Translations Patents Copyrights Protected Achievements Socio-Economic Products (products and/or innovative services with an socioeconomic impact which can be demonstrated) II. The environment of scientific research (Research Environment) (10 30 %) Maximum 4 indicators PhD Advisors Organization of Scientific Events Youth Research Program (Mechanisms to attract young researchers) Research infrastructure Access to scientific literature Edited volumes Edited translations III. The prestige in the academic community(academic Recognition) (5 15 %) Maximum 3 indicators Invited Professor Invited Lectures Citations and reviews of author s creations Member of Romanian Academy, of branch academies and foreign academies IV. Financial resources brought for the scientific research (Research Contracts) Funds drawn for research from national and international contracts. (5 10 %) 1 indicator The quantifying of descriptors is done by a set of formulas elaborated by the project team together with the 42 domain coordinators, on the basis of the national and international experience in research 6 evaluation. The mathematical relations which lead to the numerical values associated to each o evaluation criterion are specified in The detailed presentation of the evaluation formulas which are going to be used in Romanian Research Assessment Exercise [4]. The indicators which resulted so forth contain two distinct levels of evaluation: a quantitative and a qualitative one. The quantitative evaluation is automatically performed by the informatics platform, on the basis of the registered information, while the qualitative evaluation is conducted by the evaluators, on the basis of the documents subjected to analysis. The evaluation procedure from a certain research domain follows, successively, the next stages: 1. The qualitative and quantitative evaluation of files by each criterion is done, every file receiving (based on evaluation formulas) four numerical values, each for every criterion. 2. The files are ranked by each criterion, thus resulting in four hierarchies, each for every evaluation criterion. 3. The files from the top of the four hierarchies will receive a maximum number of points (according to [1]). These values are indicated in the table below. 4. The other files receive a number of points which is proportional to the numerical value received for that criterion. 5. The total number of points of university file (for the evaluated domain) is calculated by adding the number of points received for the four criteria. 6. Finally, the ranking of files by a research domain is done depending on the total number of points received. Natural Sciences Engineering Sciences Social and Economic Sciences Humanist Sciences Arts and Architecture Life Sciences Criterion I 70 points 65 points 60 points 60 points 60 points 60 points Criterion II 10 points 15 points 15 points 15 points 30 points 20 points Criterion III 10 points 10 points 15 points 15 points 5 points 10 points Criterion IV 10 points 10 points 10 points 10 points 5 points 10 points The maximum points corresponding to the four criteria for each group of domains established in The General Methodology of Evaluation [1]. By researchers, according to this guide, we understand the didactic staff and the researchers employed by the university as of 31-st of December Each researcher will have an individual account in Assessment Electronic Platform (Romanian acronym SISEC), through which she/he will introduce all the information subject to evaluation. 7 Attending PhD students who don t have an employment contract with the university will have an individual account in Assessment Electronic Platform (Romanian acronym SISEC) by which they will introduce all the information subject to evaluation. Criterion I, II and III imply both a quantitative evaluation (automatically achieved by SISEC) and a qualitative one, performed by the evaluators through the framing of the elements subjected to the qualitative evaluation by quality levels. Criterion IV contains just the quantitative component. The information necessary for the evaluation by the criteria I and III are obtained from the individual data introduced in SISEC by the researchers, while the information needed for the evaluation of the Criterion IV are introduced by the domain coordinator from the university. At the criterion II the information required by the PhD Advisors descriptor are introduced by the researchers (if they have the position of a doctorate supervisor), the rest of the information being introduced by the domain coordinator from the university. RRAE calls for two categories of actors involved in the process: the universities, represented by rectors, domain coordinators, researchers and attending PhD students (they don t have an employment contract with the university); the national and foreign evaluators. At the university level, the rector selects from the list of the forty two domains subjected to evaluation the domains which are covered in the university. The rector will also establish the domain coordinators and the list of researchers and post-graduates for each research domain; he/she will introduce and will validate through the platform the data required at the university level and in the end he/she will validate all the elements which will be entered into the evaluation process. On the basis of the lists with the identification data of the researchers and the PhD students and of the list with the identification data of the domain coordinators, established by the rectors, the accounts for the domain coordinators, researchers and postgraduates will be generated. The domain coordinator is the person appointed by the rector as a responsible for the research evaluation for a certain domain covered by the respective university. He/ She introduces the information for criteria II (Research Environment) and IV (Research Contracts), debugs the error messages (SISEC) and validates the data introduced by the researchers in the respective domain. The researchers and the PhD students will register in SISEC the scientific production (Criterion I) and each researcher who has a non-null scientific production on a certain descriptor has to introduce in SISEC, for the qualitative evaluation, 10% (but not less than one element) from the scientific production accompanying that descriptor (for example, articles, books and so on). The researchers will register in SISEC 8 the accompanying information to the prestige in the academic community (Criterion III), also. The researchers who have the right to supervise doctorates will introduce in Criterion II (Research Environment) the information afferent to PhD Advisors descriptor; the rest of the information specific to The Criterion II shall be introduced by the domain coordinator. The evaluators have access, through SISEC, to the integral electronic files allocated for the evaluation, thus having the possibility of a complete picture of the achieved performances in scientific research for the universities, by the evaluated domains. The national and foreign evaluators will analyze from the qualitative point of view a part of the information registered by the universities with the electronic platform as follows: For Criterion I, the qualitative evaluation is done only for 10% of the scientific production. For every element subjected to the qualitative analysis in the frame of Criterion I (articles, books etc.), the evaluators will choose explicitly one of the four quality levels described below. For criteria II and III all the information introduced in SISEC by the universities will be qualitatively evaluated. The evaluators will indicate just the number of elements (for example, scientific events, edited volumes and so on) accompanying each quality level. According to General methodology of evaluation [1], in the frame of RRAE, the evaluators will choose for each element subjected to qualitative evaluation one of the following four quality levels: top international; international; national; local. The detailed description of these levels of quality is specific to each evaluation domain and it is presented in Annex 1 of the current Guide Book. 9 3. The Electronic Evaluation File The electronic evaluation file (later called Evaluation file or just File) is associated to a domain of science from a certain university and it contains the entire information provided by the University for (quantitative and qualitative) evaluation in RRAE, by the respective domain. Beside the elements subjected to the evaluation, the evaluation file also contains the general information specific to the university and the evaluated domain. The electronic file is organized in three sections: Section I: General data which contain the following information: I.1 General data about the university introduced by Rector; I.2 General data about the evaluated domain introduced by the domain coordinator (number of researchers, number of PhD students, specific information domain available for the domain coordinator); I.3 Information about the number of elements uploaded to SISEC for each Criterion and Descriptor. Section II: Data for qualitative evaluation, where will be presented to the evaluator the elements subjected to the evaluation, in an interactive way, on each descriptor as follows: at Criterion I for each descriptor 10% of the scientific production will be posted (the selection of the elements being done by the researchers); at Criteria II and III all data will be posted. Every element subjected to the qualitative evaluation, together with the afferent information (text domains, *. pdf files, etc.), will be accompanied by a drop-down list, through which the evaluator will select one of the quality levels (top international, international, national or local) and by a text area, where the evaluator will provide the arguments for his decision. Section III: Integral data, which allows the evaluator to visualize the entire information referring to the evaluated domain, introduced for the domain by the researchers, domain coordinators and university, for all four evaluation criteria. The electronic platform (SISEC) ensures a functionality by which the electronic file can be saved in a *.pdf file format), a file which can be stored on a removable storage (e.g. DVD) or can be printed. This Electronic file will be validated by the domain coordinator from the university and in the end by the Rector before the assessment process begins. 10 4. Evaluator s work guidelines The evaluators are prestigious researchers from Romania and abroad, selected after a large consultation of the academic community. They will analyze and assign one of the four quality levels defined in General methodology of evaluation [1], the information registered by the universities in SISEC and destined for the qualitative evaluation. These represent 10% of the total of information from Criterion I. They will also evaluate the integral information uploaded at Criteria II and III. In RRAE, the evaluators will perform their activity, successively following these stages: Stage I: online a. The evaluator accesses SISEC using the identification data (username, password) received by from the electronic platform administrator; b. See The Evaluator Guide Book specific to the domain. This guide is available in English, in his/her SISEC work space. When he/she has doubts/questions regarding the unfolding of the evaluation procedure, the evaluator can consult the Panel Coordinator; c. In his/her work space, he/she identifies the electronic files which were automatically allocated to him/her by the electronic system for the qualitative evaluation. For each file, if conflicts of interests are not detected, he/she will accept the file for the evaluation, selecting the proper option from SISEC. If not, he/she refuses the file evaluation and he/she will receive another file for the evaluation, from the Evaluation panel coordinator. d. The proper qualitative evaluation of each file: i. The evaluator will examine th
Search
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks