ISSN Print 2500–1094
ISSN Online 2542–1204
Bulletin of RSMU
BIOMEDICAL JOURNAL OF PIROGOV UNIVERSITY (MOSCOW, RUSSIA)
REVIEW
Comparative analysis of modern approaches to the performance assessment of scientific medical organizations in Russia and abroad
About authors
1 Ministry of Healthcare of the Russian Federation, Moscow, Russia
2 Higher School of Healthcare Management,The First Sechenov Moscow State Medical University, Moscow, Russia
3 Russian Society of Regenerative Medicine, Medical Rehabilitation, Physiotherapists and Balneologists, Moscow, Russia
Correspondence should be addressed: Anna Aniskevich
Rakhmanovskii per., d. 3, Moscow, Russia, 127994, GSP-4; ur.xednay@hciveksina.anna
Received: 2016-02-20
Accepted: 2016-02-24
Published online: 2017-01-05
|
A modern approach to assessing the performance of medical research institutions in Russia’s health sector is mostly quantitative in nature and features a large number of indicators. On one hand, such comprehensive analysis facilitates collection of miscellaneous data about organizations and certain areas of their activities. On the other hand, however, it complicates regular monitoring and receipt of objective results. Use of numerical scales involves calculation of the total average indicator based on which further comparative analysis is to be conducted. Combination of diverse factors, inability to identify the most important ones and complexity in clear interpretation of values increase the likelihood of errors in assessing research institutions while applying a quantitative approach. The assessment results may be incomplete or not sufficiently reliable to address the issue of further prospects of the research institution [1].
According to information on the activities of research institutions carrying out research, developmental and technological activities for the purpose of monitoring and evaluation, approved by Order No 162 of the Russian Ministry of Education and Science on March 5, 2014, the following are the key performance indicators:
- effectiveness and relevance of scientific research;
- human resource development;
- integration into the global scientific space, dissemination of scientific knowledge and enhancement of the prestige of science;
- resourcing the activities of research institution.
Effectiveness and relevance of research institutions
The effectiveness and relevance of the activities of medical research institutions are enough for evaluation of the quantitative and qualitative characteristics of research activities of the institutions. For a comprehensive evaluation of this sector, resource indicators — funding, personnel and logistics — are also factored in.
Application of mainly quantitative methods for performance analysis of research institutions has led to active development of scientometric systems [2, 3, 4]. Presently, Russia is ranked 15–18th in the world in terms of number of scientific publications. However, in terms of number of cited published works, the country does not make it in the list of 20 leading countries [5].
Scientometrics are actively used in Russia and abroad as a reliable tool for evaluating scientific organizations and communities. Domestic scientometrics has been used by Russian Science Citation Index since 2005 [2, 6]. The main indicators used to measure the impact of research include publication activity of scientists in Russian and foreign scientific journals [6, 7, 8]. The number of publications and number of cited works included in global databases Web of Science and Scopus, as well as national information-analytical system Russian Science Citation Index are mainly analyzed [9]. Quantitative analysis of publication activity is a simple, yet reliable and intuitive method for determining the effectiveness of a research organization. Qualitative evaluation methods are characterized by accuracy, timeliness, representativeness and accessibility [3, 4].
Most industrialized nations, including Russia, use quantitative assessment of indicators [10]. For example, the UK uses three main criteria for analysis. The first of these involves the study of the newness, importance and degree of development of scientific subjects. The second examines the magnitude of the research results [11]. The third investigates the competitiveness of a research institution [12]. Examination procedure uses double-blind method [13]. The effectiveness of the activities of national universities in the UK is assessed via the Research Assessment Exercise program once every 4 years. The final rating data serve as the basis for further financing. A similar analysis in the preparation of economic decisions is also applied in the United States [14, 15].
In the US, the main method for evaluation of the effectiveness and safety of the technologies created are systematic literature reviews, whose analysis is carried out by organizations specially created for these purposes. The second evaluation method is cost-effectiveness analysis in which the costs of achieving additional year of life adjusted for quality is calculated. Based on the values obtained, decisions on further financing are taken. Clinical and economic analysis method is less widely used [16].
Medical science is the most cited in the global scientific community. The most famous scientometric database — Web of Science and Scopus, high-demand library fund — the U. S. National Library of Medicine [16, 17].
National databases are widely used by non-English speaking scientific communities. For example, the Chinese Social Sciences Citation Index covers most of the country’s journals. At the same time, they try to increase the share of publications in European and American journals. Similar projects have been implemented in Taiwan (Taiwan Humanities Citation Index) and in Japan (Citation Database for Japanese Papers) [18].
Among scientometric indicators in foreign countries, the impact factor of publications, which is annually calculated at the Institute for Scientific Information (ISI) and published in the Journal Citation Reports is used. Since this criterion has been used in the West since the 70s of the 20th century, this impact factor is quite high for many European and American journals, unlike Russian journals [19]. For example, the 2015 impact factor of one of the most prestigious medical periodicals The New England Journal of Medicine is 55.87 [20]. European and American researchers try to publish in journals with very high impact factors to increase prestige and promote career development [16].
The resource estimate of the effectiveness and relevance of scientific research is important, but today it is a vulnerable area for many Russian medical and scientific institutions. Often, research institutions operate a poor assessment system for evaluating the prospects of patents and intellectual right management [21, 22]. Only in recent years that effective measures aimed at creating small innovative enterprises that can make productive use of patented innovative products and services were applied. To upgrade this direction, a mechanism for state guarantee of procurement of innovative products and formation of technology transfer centers was created [5].
In most economically developed countries in the world, intellectual property is commercialized [23]. The US is characterized by transfer of property rights to intellectual products or services created through state support to the private sector due to the fact that the state by law could not be the owners of such products or services. In the EU, UK and Japan, the government owns certain rights to intellectual activity and is actively involved in commercialization of research products [24].
The financial impact of scientific research in Russia is evaluated using two indicators: sources of income of the research institution and type of activity. From the side of the government, financial support is provided from federal budget as part of Russian government’s programs “Healthcare Development” for 2013–2020 and “Development of Science and Technology” for 2013–2020, and in accordance with government directives and support grants [16]. At the same time, government financial support for research institutions decreases and the share of private capital increases as the final result comes nearer. Russia is ranked 9th by level of research funding in the world [5, 25]. Improving public funding with increase in the cost of medical science is necessary, but it is not enough for modernization of this sector.
Most foreign countries actively search for the most effective mechanisms for funding of medical research institutions. The most common are grants and state support, as well as sponsorship from research funds, councils and business organizations [26]. The importance of innovation for the public sector is growing. The US is characterized by funding of both fundamental and applied research. In most EU countries, the government provides effective resourcing; private sector support is weak and irregular [27]. At the same time, there is joint performance assessment of both research institutions and agencies responsible for research development and funding [28]. Quantitative analysis approaches should be complemented with qualitative analysis approaches, particularly the criteria of efficiency and safety of innovative medical technologies [16].
Human resource development
Among all the performance criteria of institutions, evaluation of personnel potential in Russian medical research institutions receives the least number of indicators (only four). Data obtained in the end is insufficient for system analysis. However, the majority of performance indicators of any research institution and its competitiveness depend precisely on the human resource capacity of that institution.
By number of scientists, Russia is ranked 4th behind China, USA and Japan [5]. In recent years, a new theory accounts for a shift in approach relating staff to costs. This theory considers the staff as the most important resource of the effectiveness of any institution.
Qualitative and quantitative assessment indicators of human resource capacity reflect the degree of implementation of research programs, the effectiveness of the institution’s structure and use of human resources, increase in productivity and quality of research. Using unrelated criteria with different analysis significance makes it difficult to obtain an objective final assessment of the entire institution [29]. Lack of strictly formal assessment of personnel potential of research medical institutions and databases, as well as qualitative factors affecting the result of activities pose challenges for the objective assessment of the sector [30]. Human potential assessment in foreign countries depends primarily on the status and quality of research activities of the scientist and research institution in the international community [31].
The US National Science Foundation applies scientific management to modernization of the staffing sector. They use lifelong learning, professional development motivation, manifestation of leadership skills and creative potential of employees [5, 32].
Enhancing the prestige of Russian science
Evaluation of indicators of integration into the global scientific space, dissemination of scientific knowledge and enhancement of the prestige of science for Russia is extremely urgent thanks to the obvious need to modernize this sector. Russians have recently been coming up with the idea of prestige of scientific work and academic status [33]. Socio-economic instability has led to sharp decline in the reputation of scientists. Owing to low wages, only about 9 % of Russians regard research profession as prestigious [5]. The ongoing loss of staff by many research institutions due to economic reasons reduces the efficiency of research and the overall level of institutions. This, of course, is reflected in the assessment of this sector. Translational barriers [34], unattractiveness of investment in medical science and lack of competitiveness of intellectual production when compared with economically developed countries are only a small list of problems to be addressed.
In most economically developed countries worldwide, public opinion polls clearly demonstrate the opposite results.
According to statistics, about 51 % of respondents in the US believe that a career as a scientist is highly prestigious, 25 % said it is very prestigious, and 20 % consider it as prestigious [5]. International experience shows that high social status of the research elite reflects the socio-economic level of the country and the pursued information policy [33].
In the European Union, issues of protection of rights of not only those involved in clinical trials, but also the animals that take part in the experiments, as well as the problem of changes in the legislation are traditionally important for the medical research community. Some European and American scientific communities are in favor of public access to information on patients who participated in clinical trials with the aim of further analysis and receipt of reliable data for further diagnosis and treatment of a wide range of diseases, especially cancer. The European scientific elite, with support from one of the leading research organizations in Germany — Max Planck Society — is in favor of open access to scientific publications for any person in accordance with the Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities. To draw attention to these issues, research societies and institutions, which are widely represented on the Internet and receive enough response from a non-indifferent population, are being created [35, 36, 37].
Resourcing the activities of a research institution
To ensure effective monitoring of Russian research and medical institutions, indicators responsible for resourcing of the activities of research institutions are among the most important [33]. In recent years, Russia has taken a number of measures to modernize her research institutions, which, in contrast to Western countries, has led to a reduction in the number of researchers in the research sector [38].
Mainly quantitative indicators, which are apparently formal in nature, are used to analyze the sector. The number of young scientists under the age of 39 is one of the consistently low indicators in many research institutions. Sociological studies showed that there is little interest among young people in research careers due to low pay, poor prestige of the profession, lack of research funding, poor social conditions and increasing bureaucracy [39]. The average number of young researchers does not exceed 25 % of the entire number of researchers [5].
Intangible assets in the modern Russian scientific institution are becoming an important criterion of the effectiveness of the organization. They are accounted for in the balance sheet as non-current assets. Exclusive intellectual rights of an institution contribute to monopolization of the right to use such rights, including to receive income from transfer of the rights to the industrial sector. Presence of innovative intangible assets allows to pay royalties, while adding the cost into the cost of the assets [40]. However, the actual use of these assets is small compared with other Russian science sectors. Patented technologies are often not used in production.
The salary of researchers is also an important factor in the effectiveness of the activities of scientific institutions and its final evaluation [41]. In most economically developed countries of the world, there is increased research funding, thereby attracting more researchers [33].
According to the US National Science Foundation, 60–65 % of funds are spent on salaries of researchers, graduate and undergraduate students, 12 % goes to the purchase of new equipment, 11 % is spent on education, about 6 % on technology transfer and only 1–3 % on administration [16, 42].
CONCLUSION
In order to effectively implement the Strategy Of Development Of Medical Science In The Russian Federation For The Period Till 2025, approved by Order No 2580-r of the Government of the Russian Federation on 28 December 2012 [5], there is need to develop better performance criteria for evaluation of research medical institutions.
According to some researchers [43], mainly qualitative indicators resulting from examination should be used to evaluate the effectiveness of research institutions. However, most experts recommend building a model that factors in constant monitoring, assessment algorithms, creation and maintenance of a personalized database of employees’ register [20]. Such an analytical model will form the final assessment of research institutions and will contribute to decision-making with regards to prospects. Therefore, it should include a systematic approach and have data accumulation, processing and storage functions [44]. Based on the above, at this stage of developing methods for research evaluation, the task of developing an objective approach that would obtain final performance indicators for Russian research health institutions is extremely important.