Με ρωτάνε συχνά πώς αντιμετωπίζουν οι αμερικανοί πανεπιστημιακοί την αξιολόγηση, η οποία τώρα γίνεται προσπάθεια να εισαχθεί στην Ελλάδα. Το παρακάτω κείμενο του συναδέλφου μου στο CCNY Michael Green είναι μια καλή απάντηση. Απευθύνεται σε Τούρκους πανεπιστημιακούς και τους εξηγεί την αξία της αξιολόγησης. Σημειωτέον ότι ο Michael ασχολείται ενεργά με τον συνδικαλισμό (σήμερα είναι πρόεδρος του CCNY chapter του συνδικάτου μας). Παρατηρήστε την ποιότητα λόγου και επιχειρημάτων και κάντε σύγκριση με τον λόγο της ΠΟΣΔΕΠ.

SOME SUGGESTIONS AS TO POSSIBLE EVALUATION/PRIORITIZATION SCHEMES FOR TURKISH SCIENCE

Evaluation: It is common, and generally required, to have academic departments, parts of national laboratories, and other scientific organizations evaluated periodically by external teams of examiners, at least in the United States. University programs undergo accreditation by professional organizations (e.g., the American Chemical Society evaluates departments to decide whether to allow these departments to “certify” their graduates). Laboratories at the National Institutes of Health are evaluated to determine what steps can be taken to improve their effectiveness and productivity. The process is often more friendly than not, in that the evaluators often know, and support, the work of the colleagues whose laboratories they are evaluating. Typically, the most useful reports are framed in terms of what could improve the department or laboratory being evaluated, rather than in terms of what the members are doing wrong. Academic departments are often evaluated by groups of outside experts, with a report given to the department, and the college (or other relevant) administration, suggesting ways in which the department can be improved. If the administration provides the suggested resources, and the department is able to follow the recommendations of the committee, the result can be a considerable improvement in the quality of the department. Often the evaluation itself provides necessary and useful pressure on an administration to provide the resources.

Sometimes, in other countries, a ministry of science (or equivalent) attempts to evaluate the productivity of the entire scientific establishment of that country, with a view to allocating the resources available in the most effective manner. This is typically done in countries that are scientifically advanced, although not in the U.S. Unfortunately such national evaluations often produce a result that is more divisive than valuable.

Turkey is not a scientifically advanced country yet, but appears to be in an intermediate position, in the following sense: although it is not yet generally competitive internationally, it has groups, and individual scientific workers, who could be; some groups already are. They could be far more effective if they had the resources that are available to scientific workers in other countries. As it is some of the best scientific workers leave for other countries and do not return. It also appears that collaborations among scientific workers are less common than in other countries, especially the most scientifically advanced countries. However, the country has limited resources (actually, every country has limited resources, but the limitations are more severe in Turkey than in most of Europe, Japan, or the U.S.). Nevertheless, it should be possible for those groups in Turkey that could compete to do so if they had only somewhat more resources than they now have. How can they make the case for getting the resources, given what is available in Turkey?

In the U.S. where limitations, even if less severe, are still real, typically groups form that can make the collective claim that the resources that are allocated to them will be fully utilized, even though no individual member of the group could make such a claim. On a larger scale, there are national and regional facilities that are designed to be used by groups of perhaps hundreds of workers. These include moderate sized facilities such as synchrotron light sources, large scale structural biology centers, particle accelerators of less than international scale, and supercomputer centers, among others. These tend to cost amounts in the tens of millions of dollars, up to perhaps hundreds of millions. Much larger facilities are multi-national or global (e.g., CERN, the international Tokamak scheduled to be built in Cadarache, France, and large scale astronomical telescopes).

For Turkey, much smaller facilities, on the scale of hundreds of thousands of dollars to a few million, are likely to be involved. These would be of tremendous importance in that they would make available instruments, such as NMRs that are the everyday bread-and-butter of standard research (in experimental chemistry for NMRs, for example); this particular example is an instrument taken for granted in every serious research chemistry department in the U.S., Europe, or Japan, but is unavailable to most Turkish researchers, preventing these researchers from working at a level comparable to those in their field in these countries. The limitations are set by the resources, not the ability of the research workers.

There is no end to the resources that researchers can ask for, of course. When a group in one of these other countries proposes that a facility be built, the claims are vetted by outside experts who are able to evaluate the relative merits of proposals (as more proposals always come in than can be funded).

Also, in deciding the facilities to be funded, it is not sufficient to evaluate a group, and decide on its competence relative to other groups; it is also necessary to decide what type of work is most important.

Priorities for large facilities and general direction of research are set by independent groups of scientists who are expert in at least closely related fields. This can be done at many levels. For resources that must be allocated on a national scale in the U.S., as for example major telescopes or space projects, a national committee is picked from the most senior practitioners in the field. The committee sets national priorities for some years to come, and the exercise is repeated as needed (perhaps every 5 to 10 years). The entire community (say, of astronomers) then backs the major project priorities (as a community: this does not mean every single astronomer supports the committee decision, of course), and Congress tends to give very strong credence to these priorities, when determining funding. Small science is affected more by the interests of individual investigators, who continue to apply for individual grants. Even here, some fields are encouraged far more than others. Also, for purposes of deciding what is large scale, what is small scale, it is necessary to take context into account; things that are small scale elsewhere may be large scale in Turkey.

The range of resources available in Turkey is smaller, so items on the scale of $1 million are large enough to require prioritization here, while in the U. S. such an item would be assigned either to a department, a single large group, or a handful of smaller groups in a collaboration. Nevertheless, the same general idea applies.

We have, therefore, identified two things as useful: evaluation of the work of the various laboratories, and prioritization of national research projects.

Evaluation:
a) Local evaluations: Evaluation allows determination of the needs, strengths and weaknesses of the group being evaluated. It requires considerable effort both on the part of the evaluators and those being evaluated. Generally, a report on the activities of the department, or laboratory, is prepared before the evaluation, and supplied to the evaluators weeks in advance; thus there is less time spent on establishing background facts. In evaluating individual departments, the evaluation is rather detailed, in that individual workers are likely to have their work evaluated. Here we are concerned with evaluation of research activity. However, it may be that not all teaching faculty members are interested in a major research career; only those who wish to compete as researchers need to be included. Others will make their career largely as teachers. The emphasis has to be on means of facilitation of research on the part of those who are interested.

Hostile criticism is essentially never productive. For one thing, it may be hard to know the reasons for failure on the part of a particular researcher. Very often the researcher will do better when conditions are better. Sometimes the researcher’s ideas are routine, and in this case the researcher is likely to deserve a lower level of funding.

The report of the evaluation committee can be used by members of the department or other group being evaluated to back up requests for resources, to learn what experts think of their efforts, and to help decide how to move forward productively. Great caution should be used by administrators in using the report to withdraw resources from those given lower ratings in the evaluation report; however, if the report is to have any use, some degree of reallocation is inevitable. The spirit in which administrators approach this delicate task will have a great deal to do with whether the entire process is seen to be useful by the affected community, thus with the degree of cooperation which is to be expected. If the administrators show lack of sensitivity, negative consequences, with attendant loss of productivity, can result. It may also be that the report suggests changes in administrative procedures, and changes in these may also have to be considered; administrative techniques are a part of any evaluation.

b) National evaluations: A different level of examination is required on a national scale. It may make sense to have a more experienced committee to evaluate the organization of science, and the level and means of allocation of resources for research. Outside (international) evaluators may be useful, but Turkish scientists with a record of success on an international scale must be the primary evaluators. The composition of the committee will have to be handled with care. Much more important is the cooperation of the members of the community being evaluated. It appears that the members of the communities involved may not be aware of the ways in which their interests coincide, and, in the context of the practice of science in Turkey, the value of collaboration as a community in obtaining resources. Such possibilities may be more apparent to those who bring a fresh point of view to examining current practice. The evaluation needs to point to investments that would be most likely to be of the most benefit to the overall progress of science in each field being evaluated. It may be that the various fields of science could be evaluated sequentially; attempting to do the entire scientific establishment of Turkey in one year, say, may be impossible. It seems likely that the result of the evaluation may be a recommendation for shared facilities (the NMR facility at Gebze might be an example, as well as an example of why it is not sufficient for departments in other cities; NMR is a routine, everyday, tool to chemists, and having to send samples to a facility in a different city is simply not practical. Ankara, in particular, would need a facility of its own, even if it has to be shared among several universities). It would also be necessary to state a set of conditions for the use of these facilities, and a means to insure that researchers were able to make use of the facilities when they have a need for them. Recommendations may include the formation of collaborations, or the means, or incentives, for stimulating their formation (e.g., grants specifically available to collaborating groups of scientists). This is up to the evaluating committee. Some facilities may really have to be national facilities—for example, a national computer center can be used remotely from anywhere in the country (or anywhere else), and is most useful on the largest possible scale. Perhaps a synchrotron light source might be a national facility. For cases in which the facilities are laboratories rather than computers, it may be found useful to include ways to have researchers move to central facilities from time to time to take advantage of the instruments there. The scale of facilities is something else that could be considered.

Evaluation also has the function of suggesting abandoning certain lines of work, and this can obviously cause pain. Researchers in a disfavored field clearly have the right to make the strongest case they can to the committee for their own efforts.

Prioritization:
There are two questions: who will do the prioritization, and what criteria should be used. The two are to some extent connected. In each field, a committee for that field would be needed. The needs of different fields will to some extent dictate what is possible and reasonable. As very large facilities are not on the agenda, those needing such facilities will have to look into some sort of national subscription to multinational facilities, as is apparently already the case with CERN. For fields in which Turkey can be competitive on a national scale (e.g., chemistry, laboratory scale physics—say physics of condensed phases—computational chemistry, other forms of chemistry, and some forms of biology, including bioinformatics) prioritization on a national scale is possible.

Perhaps it would make sense to have those who are interested in competing internationally identify themselves, and then form a prioritization committee (with perhaps some selection, or election, among the members, if there are too many for an efficient committee). It need not be the case that the highest priorities go to the internationally “hottest” current fields. However, there should be some sense of connection to what is considered important, or likely to be important in the foreseeable future, in the larger international community. This is a point on which the committee may wish to consult international experts. Although it may be impossible to have all the expertise come from Turkey, it is necessary to identify the most useful fields for Turkey to invest its limited scientific funds; thus it may be necessary to have the primary input from Turkish researchers. It is conceivable, for example, that environmental science, analytical chemistry, and natural products chemistry, might be assigned a more significant role in Turkey than in the U. S. If the experts were taken from outside, the priorities of Turkish science might be distorted in favor of generic priorities of science in general. On the other hand, as Turkey moves forward, it will make sense to have more effort devoted to cutting edge work in those fields that are scientifically most promising overall. Probably it will be found to be a mistake to excessively neglect fields not directly important to the Turkish economy, although this must be left for the committees, and the Turkish government through its funding agencies, to determine. One particular field may be of special interest: embryonic stem cell research is acceptable in Turkey. Therefore, Turkey may wish to compete here, as other, larger, scientific communities are restricted by political considerations in their ability to work in this area. On the other hand, this field is actively pursued in a number of countries that are scientifically strong (compared to Turkey), like China and South Korea.
Conclusion: If Turkey is to compete in science generally and thus to benefit Turkey itself, it will have to direct its resources in a careful manner. This will require prioritization of fields, evaluation of departments and institutions, and cooperation of researchers to share resources.

Michael E. Green
Professor of Chemistry
City College of New York
160 Convent Avenue
New York NY 10031

Advertisements