Is the Distribution of Research Grants Sustainable? An Empirical Study of Grant Assessment

Abstract

Financing of basic research is an important task in supporting research activities and development of dynamically advancing interdisciplinary fields of science. A significant challenge in this aspect is the correct distribution of limited finances sustainably. In this paper, we present an empirical study related to National Science Centre (NSC), which is the main government agency in Poland. NSC funds projects in Arts, Humanities and Social Sciences, Life Sciences and Physical Sciences and Engineering. In this work, we analyse three primary funding schemes of NSC, which are called PRELUDIUM, SONATA and OPUS. Each of theses programms is asigned to another group of scientists from beginners to experts. Projects’ data concerning PRELUDIUM, SONATA and OPUS schemes are collected from NSC projects database (only completed projects) and proccessed for further investigation. Effectiveness and sustainability of projects implemented in scientific fields are analysed concerning criteria such as the total number of book publications, papers, amount of grants and IF points. The results obtained are presented regarding scientific disciplines and panels. Prevailingly, the PRELUDIUM scheme exhibits better results in the majority of criteria and panels.

1. Introduction

The system of financing science in Poland has changed in recent years, mainly due to the reform introduced in 2010 [1,2]. Instead of one central institution, which is involved in financing research (formerly the Ministry of Science and Higher Education), supporting the research is dealt with by two statutorily created government agencies, the National Science Centre and the National Centre for Research and Development, sharing financing basic and applied research. Creators of the Polish reform while working on regulations quite openly declared their inspiration with solutions functioning all over the world.

From the beginning of its activity, the NSC has assumed that there is no such area that is beyond its interest, except only the scope of state defence and security, which results from statutory solutions. It must of course be remembered that in the system of financing science there is a second executive agency, namely the National Centre for Research and Development, covering applied research. Significant areas of financing, including the National Humanities Development Program or financing of large research infrastructure, remain under the responsibility of the Ministry of Science and Education. Funds for science are also allocated by the Foundation for Polish Science, as well as- far too small the economy [3].

In Europe, research funding is provided by the independent European Research Council (ERC). The European Research Council, unlike the Polish National Science Centre, deals with the financing of any type of scientific research, regardless of whether it is basic or applied [4]. The ERC offers mainly two types of programs under which scientists can apply for money for research: Starting Grants and Advanced Grants. Under the former, funding of up to €2 million is available for scientists who have obtained a doctorate in the period from 2 to 12 years before entering the competition. Like the National Science Centre, the ERC does not apply here the age criterion: funding may be applied for persons of all ages, provided that the criterion of time elapsed since obtaining the PhD degree. The Advanced Grants program, on the other hand, finances projects conducted by experienced researchers. The most important criterion for the evaluation of grant applications by ERC experts is excellence, or excellence achieved through competition. Research proposals submitted in the framework of ERC competitions should be innovative, addressing relevant issues and providing an opportunity for breakthrough discoveries with well-planned research projects.

The National Science Foundation is an U.S. government agency dedicated exclusively to financing basic research. The National Science Foundation offers dozens of grant programs for scientists at every level of development. Substantive evaluation of applications is carried out in the NSF in a similar way as in the ERC and in the Polish National Science Centre. The applications are evaluated by teams of experts [5].

On the basis of the two examples quoted above, it can be seen that regardless of the type of research financed or the range of disciplines in which the funds can be applied for, the basic basis for the submission of the best known models of financing science is always the same. Grant applications are to receive the best, and which projects are the best decide the scientists themselves. [6].

Bibliometrics is a tool that allows a rational and strictly measurable assessment of the published scientific output. It can be defined as a set of methods mathematical and statistical methods used to assess the impact of individual scientific publications, journals, as well as authors. These indicators are used to assess the quality of the scientific journal, the parametric evaluation of scientific institutions, as well as to evaluate the scientific output of the employee. The following measures are most popular in Poland [7]:

• Citation Index-different kinds of indicators showing how many times and by whom a given publication has been quoted. The most valuable publications are those quoted only by foreign authors.
• Impact Factor-referred to as the “measure of impact” of scientific journals. So there’s the prestige index of a given journal. Often used also as a so-called “prestige indicator”. The total IF for assessing the scientific output of individuals.
• Hirsch Index-assessed against the scientific output of the individual researcher as being capable of publishing well-quoted scientific papers. Defined as the number “h” of scientific papers, each of which is quoted “h” times.
• The Ministry of Science and Higher Education score-an indicator to evaluate both scientific entities and the entire output of a single researcher. It can be used both quantitatively as well as qualitatively. Often required in the process of doctoral studies and obtaining habilitation.

With an increasing number of applications for grants, research projects, consortia and large research networks, more and more time is taken up by researchers to evaluate and review written proposals. by other researchers. Instead of the traditional quality assessment of studies using the classic peer review method, which is quite expensive and time-consuming, there is more and more often used hasty analyses based on bibliometric data. Differences in the nature of bibliometric data concerning representatives of different fields of science [8] may be important in the work of interdisciplinary panels of the National Science Centre or the Foundation for Polish Science. During the work of the panel it would be optimal to use only factual expert opinions and refrain from direct comparison of bibliometric indicators. Otherwise, when bibliometric data is considered as a supporting argument, the differences in publication and citation in different fields of science should be taken into account. Bibliometric indices depend on the specificities of individual scientific disciplines. Often, there are differences even within a single discipline. In order to be able to interpret the bibliometric indicators characterising a particular researcher’s work prudently, a profound knowledge of a particular field of knowledge is necessary. When comparing such data for two scientists working in the same specialty, it is worthwhile to compare the data scaled per researcher’s head, e.g. the number of quotations divided by the average number of authors of one work of each scientist [9].

The scoring of the scientific journals of the Ministry of Science and Higher Education (list A) was based solely on the Impact Factor (IF) of the Web of Science database [10]. Journals are classified within a given science based on the Journal Citation Report (JCR) from 2010. However, experience shows that the IF factor is not an optimal measure of the journals’ value. One reason is the uneven level of Philadelphia list journals and the fact that citations are counted the same regardless of the quality of the journal in which they appear. It seems that a much better indicator than IF of the quality of a scientific journal is the Eigenfactor Article Infuence Score (AIS). It is based on an idea used in an Internet search engine Google, which assesses the importance of the parties online with information on the subject [11,12]. Google’s ingenious algorithm is to create a graph showing the connection between all the sites on the Internet, construct an appropriate connection matrix and find its dominant own vector, whose components characterize the value of the site.

It is also important to mention the CiteScore (CS) factor, which is competitive with the Impact Factor (IF). CiteScore (CS) is calculated for magazines that are indexed in the Scopus database, which is owned by Elsevier [13]. Five basic types of peer-reviewed texts that are indexed to Scopus are used to calculate CiteScore, i.e., scientific articles, reviews, conference reports, book chapters, and articles that are set date descriptions [14]. In the case of CiteScore, it is calculated for a given year by dividing the number of quotations for the last four years with publications published at that time by the number of those publications. For the Impact Factor (IF), 2 years are taken into account. They also differ in databases and the number of indexed journals.

The SCImago Journal Rank (SJR) and SNIP (Source Normalized Impact Paper) are also among the most popular factors [15]. The first of these SCImago Journal Rank (SJR) is also like the CiteScore factor an alternative to the Impact Factor (IF) [16,17]. It was created by SCImago, a team of researchers from the Spanish National Scientific Council and Spanish universities. The SJR is calculated from the indexed journals in the Scopus database. The factor is the quotient of the average number of citations for the last three years with the number of documents indexed over the same three-year period, where the value of citations is changed by the SJR index of the journals from which the citations originate [18].

SNIP measures the contextual impact of a citation by assigning a value to it depending on the total number of cites in a given field of knowledge. The number of articles quoting a given journal is taken into account in relation to the number of items in the lists of used literature (references) of these articles [19]. These references form the context of the citation. In those disciplines where quotations are not numerous and the list of literature contains only a few items, a single citation becomes more valuable. The opposite is true in areas with very high citation rates, where lists of literature containing more than 300 items are quite common. Then a single citation loses its value [20].

Bibliometrics can therefore only serve as an auxiliary function for the initial selection of proposals. Often the correlation between the number of quotations of an article, IF of the journal in which it was published, and its scientific value is insignificant. It is often better to take into account the actual number of citations, and to use a more objective AIS to evaluate the journal. [21].

1.1. National Science Centre

NSC (National Science Centre) is a state executive agency based in Cracow (Poland), that was established on the basis of the Act of 30 April 2010 about the National Science Centre [22]. It was officially opened on 4 March 2011. The National Science Centre has been established to finance basic research, i.e., research aimed at seeking answers to fundamental questions about of man and the surrounding reality, as well as acquiring knowledge about the foundations of phenomena and observed facts. The NSC also covers the costs of research related to culture and learning about the past in its broadest sense [23]. The objective of the NSC is to support research activities in basic research, i.e., empirical or theoretical work aimed primarily at acquiring new knowledge of the underlying foundations of phenomena and observable facts, without any direct commercial application in mind. NSC finances, among others, research projects in the field of basic research. The tasks of the NSC are following [22]:

• Financing of fundamental research carried out in the form of:
  • research projects, including financing the purchase or manufacture of scientific and research equipment necessary for the implementation of these projects,
  • research projects not subject to co-financing from foreign funds within the framework of international programs or initiatives announced in bilateral or multilateral cooperation or research projects carried out with the use of large international research facilities by Polish research teams,
  • research projects carried out by young scientists within the meaning of Article 360(2) of the Act of 20 July 2018-Law on higher education and science, including those aimed at creating a unique scientific workshop or establishing a new scientific team,
  • doctoral scholarships and internships after obtaining the academic PhD degree (doctor’s degree),
  • research projects for experienced researchers at the frontier of research, including interdisciplinary research, relevant to scientific advancement, beyond the state of the art, which may result in scientific discoveries;
• Financing of research activities other than those referred to in point 1, which do not fall within the scope of the research funded by the National Centre for Research and Development;
• Supervision over the implementation of scientific research;
• International cooperation in financing fundamental research activities;
• Disseminating information on competitions announced by the Centre to the scientific community;
• Inspiring and monitoring fundamental research funding from outside of the state budget;
• Performance of other tasks commissioned by the Minister, important for the development of basic research, including the development of research programmes important for national culture, while ensuring financial resources for the implementation of these tasks;
• Financing of the National Science Centre prize awarded in accordance with the regulations defined by the Council and approved by the Minister.

Every 3 months NSC announces at least one of the series of competitions for research projects, that also include financing the purchase or manufacture of scientific and research equipment necessary to carry out these projects. Types of competitions include PRELUDIUM, SONATA and OPUS [22].

1.2. The OPUS Competition

The OPUS Competition, organized by the National Centre of Science, is a competition for research projects for all scientists, regardless of their degree, and includes the possibility of financing the purchase or manufacture of scientific and technical equipment necessary for the implementation of these projects. Research projects may be implemented individually by a project manager or by a research team consisting of a project manager and any number of contractors. Projects are assessed by the following criteria: the scientific achievements of the project leader, the scientific level of the planned activities, the innovative character of the project and its impact on the development of the scientific discipline, as well as the budget allocated to the project. The duration of the project participating in the competition should be 12, 24 or 36 months, and the cost of purchasing a single scientific research apparatus cannot exceed 500,000 PLN for ST and NZ and 150,000 PLN for HS. Applications to take part in the competition are accepted twice a year (March–June and September–December).

1.3. The PRELUDIUM Competition

The PRELUDIUM Competition, organized by the National Centre of Science, is a competition for research projects designed for people who start their scientific careers, i.e., do not have a PhD degree. The applicant does not have to be a student of doctoral studies, and the grant does not have to be connected with the prepared doctoral thesis. Research projects may be implemented by up to 3 contractors, among which there may be only one independent research worker (promoter or scientific tutor). Projects are evaluated by the following criteria: the scientific achievements of the project leader and the supervisor, the scientific level of the planned activities, the innovative character of the project and its impact on the scientific career development of the project leader. The duration of the project participating in the competition should be 12, 24 or 36 months, and the cost of funding may not exceed 50,000, 100,000 or 150,000, depending on the duration of the project and the expenditure on equipment may not exceed 30% of the amount of funds for the implementation of the project applied for. The beneficiary of these funds may not be a scientific guardian or a promoter, and obtaining a doctoral degree is not tantamount to completing the project. Applications to take part in the competition are accepted twice a year (March–June and September–December).

The PRELUDIUM deserves special attention because it is especially addressed to young researchers and therefore greatly facilitates their scientific career development.

1.4. The SONATA Competition

The SONATA Competition, organized by the National Science Centre, is a competition for research projects, designed for people with a PhD degree, obtained up to 5 years before the year of application. The aim of the competition is to support the project manager in conducting innovative scientific research of basic character using modern equipment and/or original methodological solutions. Within a research project there can be only one independent research workers with a habilitation degree or a scientific title, provided that they are not employed in the unit that implements the project. The projects are evaluated by following criteria: the scientific achievements of the project leader, the scientific level of the planned activities, the innovative character of the project and its impact on the development of the scientific discipline and the involvement of the entity in which the project will be implemented. The duration of the project participating in the competition should be 12, 24 or 36 months, and the total cost of equipment purchase may not exceed 500,000 PLN for ST and NZ and 150,000 PLN for HS. There are no restrictions imposed as to the upper limit of funding for one project. Applications to take part in the competition are accepted twice a year (March–June and September–December).

1.5. Domains and Panels for Qualification and Evaluation of Projects

The Council of the NSC adopted as the basis for the process of qualification and evaluation of research projects the division into 25 panels (disciplines or groups of disciplines), thematically covering the entire area of research, in three science groups [22]:

• HS-Humanities, Social Sciences and the Arts (6 panels: HS1–HS6);
• ST-Science and Technology (10 panels: ST1–ST10);
• NZ-Life Sciences (9 panels: NZ1–NZ9).

Most panels in each group combine several disciplines [24]. The names of the panels and the areas they cover are presented in the

Table 1. Panels in ST-Science and Technology.

Symbol	Panel Name	Disciplines
ST1	Mathematical sciences	all mathematics, theoretics and applied mathematics, as well as the mathematics fundamentals of science, physics and statistics
ST2	Basic components of matter	elementary, nuclear, plasma, atomic, molecular, gas and optical particle physics
ST3	Physics of condensed phase	structure, electronic properties, fluids, nano-science, biological physics
ST4	Chemistry	physical/chemical chemistry, theoretical chemistry, analytical chemistry, inorganic chemistry, organic chemistry, method development
ST5	Materials	obtaining materials, relations between structure and properties, advanced and functional materials with assumed properties, (macro)molecular architecture, material engineering
ST6	IT and information technology	information technologies and systems, information technology, computing, scientific calculations, intelligent systems
ST7	Systems and telecommunication engineering	electronics, telecommunication, optoelectronics
ST8	Process and production engineering	modelling, design, control, construction structures and processes, material engineering, energy systems
ST9	Astronomy and space research	astrophysics, astrochemistry, astrobiology, solar system, planetary systems, star, galactic and extra galactic astronomy, space research, instruments
ST10	Earth sciences	geological sciences, atmospheric and climate sciences, geochemistry, geodesy, geo-ecology, geophysics, physical geography, geo-informatics, planetary geology, soil science, mining, chemical and physical oceanology, environmental protection

,

Table 2. Panels of HS-Humanities, Social Sciences and Arts.

Symbol	Panel Name	Disciplines
HS1	Fundamental questions about the nature of man and the surrounding reality	philosophy, cognitive science, religious studies, theology
HS2	Culture and cultural creation	literature studies, linguistics, cultural studies, library studies, art studies, architecture
HS3	Knowledge of the past	history, archaeology, ethnology, ethnology, cultural anthropology
HS4	Unit, institutions, markets	economics, finance, management, demography, social and economic geography, urban planning
HS5	Law, policy sciences, public policies	law, policy sciences, regional and social policy
HS6	Man and social life	psychology, pedagogy, sociology

and

Table 3. Panels of NZ-Life Sciences.

Symbol	Panel Name	Disciplines
NZ1	Basic life processes at the molecular level	molecular biology, structural biology, biotechnology
NZ2	Genetics, genomics	molecular genetics, genomics, proteomics, bioinformatics, systemic biology, molecular epidemiology
NZ3	Biology at the cellular level	cell biology, development and ageing biology, neuroscience
NZ4	Biology at the level of tissues, organs and organisms	structure and function of systems, organs and organisms of humans and animals, experimental medicine, basics of nervous system diseases
NZ5	Non-communicable diseases of humans and animals	causes, mechanisms, recognition and treatment of diseases, poisonings and injuries (except for diseases of the nervous system)
NZ6	Immunology and infectious diseases in humans and animals	immunity, immunological diseases, immunotherapy, infectious and invasive diseases, microbiology, transplantology, allergology
NZ7	Medicines science and public health	epidemiology, civilisation diseases and social environmental threats to human and animal health, medical and veterinary public health protection, occupational medicine, drug science
NZ8	Basics of knowledge about life at the environmental level	evolutionary biology, population biology, environmental biology, systematics
NZ9	Basics of applied life sciences	agriculture, forestry, horticulture, fisheries, nutrition and food, environmental biotechnology

.

The aim of this paper was to compare of the profit generated by the projects within the different panels for PRELUDIUM, OPUS and SONATA.

2. Data and Methods

The plan of this research includes the following stages:

• Data collection for project implementation.
  • Selection of types of NSC competitions for research.
  • Collection of data on projects implemented within the framework of selected competitions.
  • Correction of erroneous data and their preparation for analysis.
• Carrying out tests.
  • Examining the effectiveness of projects implemented within the framework of specific calls for proposals and scientific fields.
  • Recording the obtained results in the form of tables and column charts.
• Comparison of results.
• Conclusions.

2.1. Dataset

In this analysis, the Polish dataset from the NSC’s project database is used, that is available on website of NSC [22]. Data of projects for competition PRELUDIUM, SONATA and OPUS for science group ST, NZ and HS was collected.

Following data was obtained for each project:

• Type of panel;
• Granted amount;
• Total amount spent on the purchase of necessary apparature;
• Total IF (Impact Factor) sum;
• Total number of articles with IF;
• Total number of all articles;
• Total number of conference publications;
• Total number of book publications;
• Number of performers;
• Date of start of the project and duration of the project in months.

Impact Factor is the most recognizable factor in the international scientific community. It is a measure of the frequency with which an average article from a given journal was quoted in a given year. Impact Factor helps to assess the relevance of a journal, especially when compared to other journals in the same field, because IF strongly prefers English language magazines and non-humanistic sciences. To calculate IF for a journal for a given year, it is necessary to count the number of quotes from articles in that journal that have appeared in that year, and then divide this value by the sum of the articles published in that journal in the previous two years.

Impact Factor was introduced in the 1950s and 1960s by Eugene Garfield, assistant librarian at Johns Hopkins University, to assist in the selection of journal subscriptions [25,26,27]. E. Garfield established the Institute of Scientific Information (ISI) in 1960, the first company to benefit from providing access to scientific information, initially by publishing weekly “Current Contents” and the annual “Science Citation Index”. In 1992, the Thomson Group purchased ISI and created Web of Science, a Web of Science networked database integration. In 2012, Reuters merged with Thomson Reuters to form Thomson Reuters, a global corporation. IF was calculated and published by Thomson Reuters, that indexed more than 11,000 journals every year and published “Thomson Reuters Journal Citation Reports”. Impact Factor is currently being established by Thomson Scientific (Institute of Scientific Information in Philadelphia). This information is given by the “Journal Citation Reports” (JCR) database, that provides information about the world’s leading journals [28,29].

2.2. Limitations of the Study

Information was collected on 4597 projects that had been started in the 2011–2016 period and had been finished in the 2012–2018 period. Only the data of the projects with a settled status was used. The data was collected on 2 August 2019.

Table 4. Number of projects in particular competitions and panels.

Science Group	Panel	PRELUDIUM	Sonata	Opus	Total
ST	ST1	50	20	64
	ST2	45	14	55
	ST3	42	5	68
	ST4	85	20	70
	ST5	200	76	223
	ST6	61	22	46
	ST7	72	32	103
	ST8	138	35	128
	ST9	27	10	44
	ST10	70	16	95
Total ST		790	250	896	1936
NZ	NZ1	50	9	57
	NZ2	47	11	36
	NZ3	48	9	51
	NZ4	65	14	81
	NZ5	53	19	58
	NZ6	24	4	36
	NZ7	69	18	70
	NZ8	51	10	69
	NZ9	94	25	105
Total NZ		501	119	563	1183
HS	HS1	44	15	59
	HS2	118	36	171
	HS3	96	28	115
	HS4	99	64	236
	HS5	67	31	91
	HS6	85	41	82
Total HS		509	215	754	1478
Total		1800	584	2213	4597

presents the number of projects that were financed under the competitions: PRELUDIUM, SONATA and OPUS, by area: ST, NZ and HS and their panels. The largest number of projects, as many as 236 were financed in the HS4 panel for the OPUS competition, and the smallest number, i.e., only 4 were financed in the NZ6 panel for the SONATA competition. The table also presents the individual amounts in relation to panels and types of competitions. In total, the largest number of projects-896-were submitted for the OPUS competition in the ST field, and the smallest-119 for the SONATA competition in the NZ field.

2.3. Data Preprocessing

To complete the data collection task, the script written in Python language was used, which enabled automatic saving of the data to xlsx file. A preliminary analysis of the data revealed potentially wrong IF values for 254 projects (5.53% of all projects analysed). These values were verified from the ”Journal Citation Reports” [30] (access date 28 September 2019). The most common errors found were:

• attributing the IF to articles that did not have it,
• omitting the dot in the IF value,
• entering a journal’s scores instead of its IF,
• giving the IF from the wrong year.

The collected and corrected data for the planned research in the form of summary statements obtained using pivot tables in Microsoft Excel spreadsheet are presented in the following tables.

Table 5. Data of projects of ST panel.

Competition	Panel	Projects	Granted Amount [PLN]	Apparat. Cost [PLN]	Min. Amount [PLN]	Max. Amount [PLN]	Total IF	Articles with IF	Articles	Conf. Publ.	Book publ.	Duration [Months]
PRELUDIUM	ST1	50	3,168,930	159,752	29,050	98,690	114.132	116	123	6	4	1230
	ST2	45	3,226,177	282,100	24,600	126,200	506.956	127	133	39	1	997
	ST3	42	4,950,084	694,578	32,750	395,670	449.871	159	162	32	2	1077
	ST4	85	9,098,363	1,685,610	40,150	257,510	833.445	224	237	60	15	1966
	ST5	200	24,096,436	3,095,475	41,600	360,000	1643.36	454	480	248	17	5101
	ST6	61	6,065,010	343,260	42,000	319,800	181.04	89	117	161	14	1581
	ST7	72	8,780,607	915,963	26,000	276,180	259.789	127	186	201	20	1813
	ST8	138	17,219,416	2,327,388	39,000	386,984	465.517	247	421	273	61	3392
	ST9	27	2,268,451	129,900	34,400	156,840	291.581	64	64	37	0	694
	ST10	70	8,223,837	644,330	20,150	29,7290	275.958	126	153	134	41	1897
Total		790	8,7097,311	10,278,356	20,150	395,670	5021.649	1733	2076	1191	175	19,748
Sonata	ST1	20	2,738,977	103,600	38,920	374,400	110.038	107	114	6	2	684
	ST2	14	4,021,878	665,376	60,720	794,160	479.756	143	145	33	1	516
	ST3	5	2,240,591	503,600	179,200	800,000	129.766	36	37	12	6	186
	ST4	20	7,320,582	2,436,480	120,300	644,400	386.556	102	104	32	5	708
	ST5	76	37,558,364	9,558,070	173,400	1,037,900	1350.465	395	425	251	18	2942
	ST6	22	7,733,554	832,000	22,800	798,000	71.518	52	86	138	20	803
	ST7	32	19,465,646	3,286,000	292,700	1,000,000	427.106	224	260	206	35	1290
	ST8	35	15,667,184	2,698,562	143,880	910,000	275.262	139	227	173	19	1296
	ST9	10	4,059,839	593,500	171,600	996,140	314.068	66	68	26	0	414
	ST10	16	5,521,296	648,200	124,080	660,950	111.46	56	68	74	10	578
Total		250	106,327,911	21,325,388	22,800	1,037,900	3655.995	1320	1534	951	116	9417
Opus	ST1	64	15,688,744	567,950	51,900	785,000	569.698	592	652	20	48	2120
	ST2	55	17,859,102	2,884,200	44,000	1,031,969	2592.668	701	739	200	7	1791
	ST3	68	28,219,245	6,835,600	81,700	1,235,000	1628.566	559	578	140	9	2240
	ST4	70	31,674,241	9,677,916	95,500	1,703,000	1355.928	395	410	137	39	2309
	ST5	223	11,8752,348	30,667,319	63,595	1,429,900	4940.395	1417	1472	930	59	7772
	ST6	46	20,392,344	1,076,786	65,320	1,264,230	440.412	244	337	444	91	1506
	ST7	103	55,751,812	8,466,150	88,000	1,138,300	1041.434	563	786	784	160	3357
	ST8	128	61,414,425	11,558,020	97,500	1,049,500	1001.193	562	872	544	120	4190
	ST9	44	20,019,410	2,584,600	56,940	994,770	2206.793	449	461	127	5	1479
	ST10	95	28,343,785	5,389,058	42,450	913,550	731.319	349	452	321	104	2983
Total		896	398,115,456	79,707,599	42,450	1,703,000	16,508.406	5831	6759	3647	642	29,747

,

Table 6. Data of projects of NZ panel.

Competition	Panel	Projects	Granted Amount [PLN]	Apparat. Cost [PLN]	Min. Amount [PLN]	Max. Amount [PLN]	Total IF	Articles with IF	Articles	Conf. Publ.	Book publ.	Duration [Months]
PRELUDIUM	NZ1	50	5,826,556	341,000	48,100	275,600	281.382	87	93	50	1	1220
	NZ2	47	6,084,492	462,620	36,140	393,900	373.702	82	85	34	5	1153
	NZ3	48	5,915,166	200,000	49,200	369,500	324.328	77	82	59	5	1210
	NZ4	65	8,556,019	679,967	45,500	325,920	319.522	111	119	135	4	1719
	NZ5	53	8,250,189	609,707	29,640	1,122,400	305.982	110	117	99	5	1526
	NZ6	24	3,765,687	322,000	49,400	458,750	150.133	55	58	28	4	650
	NZ7	69	8,685,251	828,376	45,175	435,125	352.788	143	152	149	8	1762
	NZ8	51	6,760,563	765,665	40,000	247,100	323.204	134	146	43	1	1437
	NZ9	94	12,975,469	1,550,445	47,999	758,745	481.73	211	241	235	15	2369
Total		501	66,819,392	5,759,780	29,640	1,122,400	2912.771	1010	1093	832	48	13,046
Sonata	NZ1	9	5,802,290	577,000	295,000	1,000,000	138.344	27	29	8	1	378
	NZ2	11	6,103,230	1,313,500	266,500	1,000,000	130.465	40	41	14	6	402
	NZ3	9	4,994,444	724,600	296,500	1,000,000	97.506	22	23	25	3	372
	NZ4	14	6,127,177	944,600	150,000	1,000,000	124.263	45	51	62	3	558
	NZ5	19	8,148,560	871,250	94,000	681,874	182.267	64	79	86	3	705
	NZ6	4	2,233,612	106,500	364,000	999,212	30.384	11	13	8	4	174
	NZ7	18	8,613,968	1,478,700	196,115	1,000,000	185.617	78	89	82	2	685
	NZ8	10	4,649,376	955,428	101,210	1,000,000	106.935	31	35	17	0	408
	NZ9	25	11,935,845	1,849,800	199,500	850,000	235.132	111	124	175	19	960
Total		119	58,608,502	8,821,378	94,000	1,000,000	1230.913	429	484	477	41	4642
Opus	NZ1	57	30,026,462	5,362,482	174,850	997,120	1152.692	290	297	112	7	2028
	NZ2	36	19,579,355	1,808,982	199,810	1,262,170	620.522	126	139	85	4	1250
	NZ3	51	27,719,798	3,693,673	225,000	984,000	555.675	144	163	155	15	1788
	NZ4	81	36,327,282	5,156,882	45,500	1,020,000	784.83	292	316	270	7	2784
	NZ5	58	26,405,448	2,577,136	131,900	1,277,080	586.529	217	247	158	3	1978
	NZ6	36	15,381,731	1,975,000	100,000	807,400	507.561	144	154	115	6	1230
	NZ7	70	32,454,016	5,800,642	61,800	988,600	773.94	281	310	241	7	2434
	NZ8	69	23,445,669	4,033,838	89,180	892,900	628.409	277	314	153	17	2349
	NZ9	105	44,377,322	7,921,468	123,682	984,000	858.272	359	429	378	32	3662
Total		563	255,717,083	38,330,103	45,500	1,277,080	6468.43	2130	2369	1667	98	19,503

and

Table 7. Data of projects of HS panel.

Competition	Panel	Projects	Granted Amount [PLN]	Apparat. Cost [PLN]	Min. Amount [PLN]	Max. Amount [PLN]	Total IF	Articles with IF	Articles	Conf. Publ.	Book publ.	Duration [Months]
PRELUDIUM	HS1	44	2,072,074	61,030	8375	112,245	7.173	9	110	14	51	987
	HS2	118	8,179,151	483,023	15,640	262,300	0.968	2	241	93	184	2914
	HS3	96	7,944,432	203,446	11,100	238,850	20.95	18	211	71	94	2526
	HS4	99	6,431,007	332,884	20,000	160,900	45.084	41	335	89	118	2246
	HS5	67	3,851,214	135,070	13,440	145,440	6.704	5	135	33	97	1433
	HS6	85	6,163,280	351,804	23,020	199,960	117.611	68	143	11	65	2066
Total		509	34,641,158	1,567,257	8375	262,300	198.49	143	1175	311	609	12,172
Sonata	HS1	15	2,340,957	29,400	33,000	648,200	27.667	18	88	10	39	528
	HS2	36	7,113,523	530,718	34,200	808,250	9.668	5	130	46	130	1266
	HS3	28	6,006,401	157,178	47,930	471,600	31.938	22	83	38	48	969
	HS4	64	1,1065,601	731,064	41,800	564,820	82.05	73	425	128	140	2300
	HS5	31	2,587,334	110,492	41,600	191,183	8.408	8	92	19	75	1105
	HS6	41	9,400,351	943,365	61,500	559,352	274.01	127	213	11	61	1560
Total		215	38,514,167	2,502,217	33,000	808,250	433.741	253	1031	252	493	7728
Opus	HS1	59	5,702,253	137,394	14,760	313,125	33.042	44	175	37	167	1596
	HS2	171	29,021,486	2,089,985	20,000	900,366	56.631	54	475	166	491	5209
	HS3	115	21,048,633	1,193,780	18,850	554,600	28.575	23	273	97	239	3618
	HS4	236	42,527,803	1,620,186	26,400	634,120	200.382	190	1350	377	565	6092
	HS5	91	13,052,889	205,000	29,940	435,600	10.215	9	298	52	269	2449
	HS6	82	22,859,934	1,793,505	18,250	893,750	326.723	160	333	42	102	2578
Total		754	134,212,998	7,039,850	14,760	900,366	655.568	480	2904	771	1833	21,542

show the parameters used for ST, NZ and HS respectively, broken down by type of competition: PRELUDIUM, SONATA and OPUS, and panel types and their total value. Meaning of the abbreviations used in headline of these tables is included in

Table 8. Meaning of the abbreviations used in the three tables above.

	Abbreviation	Meaning
1	Competition	Type of competition
2	Panel	Name of panel
3	Projects	Total number of projects per panel
4	Granted amount	Total granted amount of projects per panel
5	Apparat. Cost	Total cost of apparature of projects per panel
6	Min. Amount	Minimum granted amount of project per panel
7	Max. Amount	Maximum granted amount of project per panel
8	Total IF	Total sum of IF points of projects per panel
9	Articles with IF	Total number of articles with IF points of projects per panel
10	Articles	Total number of all articles of projects per panel
11	Conf. Publ.	Total number of conference publications of projects per panel
12	Book publ.	Total number of book publications of projects per panel
13	Duration (months)	Total number of months of duration of projects per panel

. Two parameters used deserve special attention-granted amount and total IF (number of Impact Factor), which were the basis for calculating all coefficients calculated in the research.

The effectiveness of selected grant programs was assessed by comparing 12 profit criteria and 3 cost criteria within each scientific field and panel against the 12 profit criteria and 3 cost criteria set. The chosen criteria to evaluate the effectiveness of the projects are as follows:

Profit criteria:

• IF points per 1 million Polish zloty (PLN) of the grant amount allocated to projects;
• Number of conference publications per 1 million PLN of the grant amount allocated to projects;
• Number of papers per 1 million PLN of the grant amount allocated to projects;
• Number of book publications per 1 million PLN of the grant amount allocated to projects;
• Number of papers with IF per 1 million PLN of the grant amount allocated to projects;
• Number of projects per 1 million PLN of the grant amount allocated to projects.
• IF points per 1 million PLN spent on apparature;
• Number of conference publications per 1 million PLN spent on apparature;
• Number of papers per 1 million PLN spent on apparature;
• Number of book publications per 1 million PLN spent on apparature;
• Number of papers with IF per 1 million PLN spent on apparature;
• Number of projects per 1 million PLN spent on apparature.

The first criterion applied by us, i.e., the total IF value of publications produced with the support of research grants per 1 million PLN of the value of these grants (IF / 1 million PLN), is a measure of the evaluation of the effectiveness of research projects called Wroclaw Effectiveness Indicator for Grants (WEIG). WEIG measures the hypothetical IF value of publications related to an investment of 1 million PLN in a given grant type, researcher-manager or field of science. This measure makes it possible to assess changes in effectiveness over time, compare individual grants, grant managers, research units, as well as areas and disciplines, including defined panels used by the National Science Centre.

The publicly available WEIG values for individual panels and competitions, or even for individual completed projects, may be important information for all project managers to enable them to place their own project against other similar projects (for example, in the same panel and competition type). WEIG can be used to compare between types of competitions as well as in an international context, i.e., the effectiveness of European grants, the effectiveness of Polish research compared to research supported by foreign agencies such as NSF (USA), DFG (Germany), EPSRC (UK), GACR (Czech Republic) or the EU. It is even possible to compare WEIG indicators for individual NSC grant panels with ERC (European Research Council) grants, as both agencies use the same panels. The WEIG could also be used to assess the implementation of completed projects, which already has an impact on the assessment of subsequent applications from a given manager [31].

Cost criteria:

• Cost of equipment per project;
• Duration of the project in months;
• Grant amount allocated to the project.

3. Results

The results obtained are organized in three sections:

• Analysis of profit criteria per 1 million PLN of the grant amount allocated to projects;
• Analysis of profit criteria per 1 million PLN of the grant amount spent on the purchase or manufacture of equipment necessary for the implementation of projects;
• Analysis of cost criteria for implemented projects.

The division into the above sections enables a broad analysis thanks to three different, independent views of the problem. This way of analysis allows for a reliable assessment of sustainability in the scope of the competitions and fields in question, because we can only talk about sustainability if it is proportionate in many important aspects. The sustainability assessment algorithm for scientific projects presented in this article is easy, intuitive and applicable also taking into account other criteria that will be relevant in this case.

3.1. Analysis of Profit Criteria Per 1 Million PLN of the Grant Amount Allocated to Projects

Indicators, the values of which are obtained by calculating what proportion of a certain amount of funding is obtained or worked out, are one of the more popular and authoritative ways of assessing effectiveness and development in a given field. They make it possible to objectively assess the relationship between the specified values, to assess the economic situation in the case in question and to anticipate future changes. Thanks to the assessment of the size of the profit criteria for projects per 1 million PLN allocated by the NSC for projects, it is possible to determine which projects generate the highest profits, i.e., the most and the least effective. After comparing the obtained values, we have the opportunity to assess which projects have made the best use of the received co-financing amount. The review of these results makes it possible to determine whether the degree of funding for projects under various competitions and fields is balanced, or whether there are situations where projects representing specific fields of science are insufficiently funded compared to other fields of science. The importance of such a cross-cutting assessment of the economic performance of projects is high, as catching certain distortions in this area and intervening early may prevent negligence in unduly undervalued scientific fields and types of competitions. This, in turn, may contribute to increasing the efficiency and productivity of research teams representing given fields of science and groups of scientists, which has a positive impact on the development of science, innovation and economy in a given country.

We have selected 6 profit criteria for this section of research. These are listed below.

Criteria analysed:

• IF points;
• Number of papers with IF points;
• Number of papers;
• Number of conference publications;
• Number of book publications;
• Number of projects;

3.1.1. Methods

The values of these criteria and the amounts of project financing were summarised in the framework of individual competitions and panels, and then the values of these criteria were calculated per 1 million PLN of the total NSC financing. Data on these criteria are available on the NSC website, for each project that has received funding. These criteria are a good indication of whether the financing of the projects under consideration is sustainable. These are values that are of high importance for scientists and their teams, universities and, consequently, for the evaluation of science in a country. The resulting indicator values result directly from the original project data. Despite its simplicity, this method has advantages in the form of faithful representation of the actual state of affairs of the studied area. The data in this case are not excessively processed and thus are not distorted, and the presented research results are objective and exclude the possibility of data manipulation in order to obtain the desired results.

In order to calculate the values of all of the above mentioned indicators, we used the Equation (1)

$$V_1=\frac{S_t·F_s}{F_t}$$

(1)

where:

$V_1$ means the value of the calculated indicator;

$S_t$ means aggregate sum of a given parameter within a specific competition and panel;

$F_s$ means 1 million PLN;

$F_t$ means aggregate amount of total NSC funding within a specific competition and panel in PLN.

3.1.2. Results

The results to be analysed in the form of average value of IF points, number of articles from IF, all articles, conference and book publications and implemented projects per 1 million PLN of the grant amount allocated to projects were obtained using a script in Python language.

The results obtained, that make it possible to illustrate and compare the effectiveness of grant programmes, are presented graphically in the form of column charts and discussed in detail in Appendix A.1.

In the case of the total IF points per 1 million PLN, PRELUDIUM gained an advantage over other competitions within 9 out of 10 evaluated panels belonging to the ST domain and within all panels from the NZ domain. This trend was not observed in the HS field. In general, the highest scores were observed in ST, then in NZ, whereas in HS the results were much lower.

In the case of total number of papers with IF per 1 million PLN, ST PRELUDIUM achieved the highest score in 9 out of 10 panels, for NZ in all panels, which again proves the domination of projects implemented under PRELUDIUM in ST and NZ fields. For HS also in the case of this parameter we do not observe such regularity.

For the parameter total number of all paper per 1 million PLN within ST the highest results were achieved by PRELUDIUM in 8 out of 10 panels, and in NZ in all panels. For this parameter this trend was repeated in 4 out of 6 HS panels.

For the parameter total number of conference publications per 1 million PLN, the highest results were most often achieved by PRELUDIUM, because in ST in 9 out of 10 panels, in NZ in 7 out of 9 panels and in HS in 5 out of 6 panels.

In the case of the parameter total number of book publications for 1 million PLN, the higher observed trend of PRELUDIUM domination did not repeat itself in the ST and NZ domains, whereas PRELUDIUM domination was observed in 4 out of 6 HS panels. For ST PRELUDIUM it is dominated by half of the tested panels, and for NZ by 3 out of 9 panels.

The different distribution of areas where PRELUDIUM dominates for this parameter and the lower results for ST and NZ than for the other parameters and HS for this parameter are due to the fact that the result of projects in HS, which is a humanities field, is often books. In the case of ST and NZ, much more often project results are included in conference articles and publications.

In the case of the last parameter evaluated in this section of research, i.e., the number of projects per 1 million PLN, we can see the undisputed, clearly prevailing dominance of PRELUDIUM in all panels of all fields.

It can therefore be concluded that, in order to balance the level of funding of individual projects, it would be appropriate to allocate more money to the funding of projects under the PRELUDIUM competition, especially in the ST and NZ panels.

The high values testify to the high efficiency of the projects in the selected competition and panel compared to other competitions and projects, as well as their high potential to generate good results in the future. Therefore, they prove that the funding for these projects has been too low so far and suggest the need to allocate more funds to them in the future.

The obtained results are presented in following

Table 9. Results as average values for individual profit criteria per 1 million PLN allocated to projects.

		IF Points	Papers with IF	Papers	Conference Publications	Book Publications	Projects
HS	PRELUDIUM	5.68	4.26	36.58	8.54	18.84	15.72
	SONATA	9.72	5.88	27.72	6.19	15.18	6.38
	OPUS	4.82	3.80	21.53	5.25	15.99	6.3
NZ	PRELUDIUM	44.93	14.99	16.11	11.22	0.69	7.51
	SONATA	20.58	6.75	7.60	6.75	0.69	1.99
	OPUS	26.33	8.39	9.28	6.34	0.37	2.2
ST	PRELUDIUM	69.24	23.86	27.04	13.52	1.74	10.25
	SONATA	45.24	16.86	18.77	8.61	1.22	2.99
	OPUS	51.62	18.72	21.26	9.20	1.87	2.54

and

Table 10. Ranking of results of profit criteria for individual competitions within the HS, NZ and ST domains per 1 million PLN grant amount allocated to projects.

		IF Points	Papers with IF	Papers	Conference Publications	Book Publications	Projects
HS	PRELUDIUM	2	2	1	1	1	1
	SONATA	1	1	2	2	3	2
	OPUS	3	3	3	3	2	3
NZ	PRELUDIUM	1	1	1	1	1	1
	SONATA	3	3	3	2	1	3
	OPUS	2	2	2	3	2	2
ST	PRELUDIUM	1	1	1	1	2	1
	SONATA	3	3	3	3	3	2
	OPUS	2	2	2	2	1	3

.

In the case of the average IF value of 1 million PLN per HS project, SONATA scored the highest score, followed by PRELUDIUM and OPUS in last place. In the case of NZ, PRELUDIUM came first, OPUS came second and SONATA third. Within ST, PRELUDIUM came first, OPUS came second and SONATA third.

In the case of of total number of papers with IF per one million PLN within HS, the highest score was achieved by SONATA, followed by PRELUDIUM, with OPUS taking last place. In case of NZ, PRELUDIUM came first, OPUS came second and SONATA third. Within ST, PRELUDIUM came first, OPUS came second and SONATA third.

In the case of the total number of all papers per one million PLN for the HS project, PRELUDIUM scored the highest score, followed by SONATA, and OPUS took last place. In case of NZ, PRELUDIUM came first, OPUS came second and SONATA third. Within ST, PRELUDIUM came first, OPUS came second, and SONATA came third.

In the case of the total number of conference publications per one million PLN per HS project, PRELUDIUM scored the highest score, followed by SONATA and OPUS in last place. In case of NZ, PRELUDIUM came first, SONATA came second and OPUS third. Within ST, PRELUDIUM came first, followed by SONATA in the second OPUS, and SONATA in the third OPUS.

In the case of the total number of book publications per one million PLN per HS project, PRELUDIUM scored the highest score, followed by OPUS and SONATA in last place. In case of NZ, PRELUDIUM and SONATA came first, and OPUS was last. Within ST, OPUS took first place, PRELUDIUM took second place, and SONATA took third place.

In the case of the total number of project publications per one million PLN per project within HS, PRELUDIUM scored the highest score, followed by SONATA and OPUS in last place. In case of NZ, PRELUDIUM came first, OPUS came second and SONATA third. Within ST, PRELUDIUM came first, SONATA came second, and OPUS came third.

Projects financed by the Preludium program achieved the highest result for the number of papers, conference and book publications and projects for the HS domain. PRELUDIUM projects scored highest rank for all the criteria examined for NZ and for all the ST criteria except the number of books published. For the HS domain, most of the tested PRELUDIUM criteria achieved better values than other contests.

3.2. Analysis of Profit Criteria Per 1 Million PLN of the Grant Amount Spent on the Purchase or Manufacture of Equipment Necessary for the Implementation of Projects

In this section we have included the results of research on the efficiency and sustainable development of projects in individual competitions and panels in relation to the amount of funding from the NSC for the purchase of equipment and apparatus necessary to implement the project.

Criteria analysed:

• IF points;
• Number of papers with IF points;
• Number of papers;
• Number of conference publications;
• Number of book publications;
• Number of projects;

These are therefore the same data as in Section 3.1, but recalculated in such a way as to examine the sustainability of the project funding in terms of the amount needed to purchase equipment for the project. As in Section 3.1, this is the profit criterion. The higher its value, the more advantageous the result is.

3.2.1. Methods

In order to calculate the values of all of the above mentioned indicators, we used the Equation (2)

$$V_2=\frac{S_t·P_s}{P_t}$$

(2)

where:

$V_2$ means the value of the calculated indicator;

$S_t$ means aggregate sum of a given parameter within a specific competition and panel;

$P_s$ means 1 million PLN;

$P_t$ means aggregate amount of NSC funding spent on the purchase of equipment necessary to implement projects within a specific competition and panel in PLN.

The high values prove the high profitability of the projects, since they achieve high values of the examined parameters at low cost of the necessary equipment. In this case, it is worth considering increasing the co-financing of such projects in order to ensure balance of financial resources in comparison with other projects and to contribute to achieving even more favourable results in the future. Providing technical facilities in the form of more and more modern equipment contributes to increasing the capabilities of scientists by accelerating their work and using modern techniques. In turn, the low results in this section of research indicate that certain projects are not cost-effective. In such cases, it is advisable to evaluate them in order to make changes to improve the results in the future. The assessment of the sustainability of the financing necessary for the implementation of equipment projects should for this section take particular account of the area to which the projects belong, as the need for technical facilities depends on its specificity. This demand is lower for the Humanities, Social Sciences and Arts (HS panels) and much higher for Science and Technology (ST panels) and Life Sciences (NZ panels).

3.2.2. Results

The results obtained, that make it possible to illustrate and compare the effectiveness of grant programmes, are presented graphically in the form of column charts and discussed in detail in Appendix A.2.

In case of the parameter total IF points per 1 million PLN apparature cost, for ST the highest score in 9 out of 10 panels was PRELUDIUM, for NZ PRELUDIUM it dominates in terms of the highest score in all panels. This trend is not observed in the case of HS panels.

In case of total number of papers with IF per 1 million PLN apparature cost for ST PRELUDIUM wins in 9 out of 10 panels, for ST in all panels and for HS in 4 out of 6 panels.

The results for the parameter total number of conference publications per 1 million PLN apparature cost are very similar, in 9 out of 10 ST panels, in all NZ panels and in 5 out of 6 HS panels PRELUDIUM wins.

For the parameter total nuber of book publications per 1 million PLN apparature cost in 6 out of 10 ST and NZ panels and in 4 out of 6 HS panels PRELUDIUM wins.

In case of the last parameter we examined in the current section, i.e., number of projects per 1 million PLN apparature cost in all fields and panels, PRELUDIUM wins the leading result.

The results to be analysed in the form of average value of IF points, number of articles from IF, all articles, conference and book publications and projects per 1 million PLN of the grant amount allocated for the equipment necessary to implement the projects were obtained using a script in Python language. The obtained results are presented in following

Table 11. Results as average values of individual profit criteria per 1 million PLN spent on apparature.

		IF Points	Papers with IF	Papers	Conference Publications	Book Publications	Projects
HS	PRELUDIUM	123.65	98.93	958.46	218.98	489.34	412.03
	SONATA	273.54	178.09	901.00	171.21	468.64	194.62
	OPUS	107.87	102.62	700.37	156.62	561.40	207.14
NZ	PRELUDIUM	650.15	201.73	215.90	150.07	9.54	106.20
	SONATA	162.7	53.02	60.42	51.82	7.36	16.30
	OPUS	193.64	59.88	66.32	45.44	2.51	15.91
ST	PRELUDIUM	786.85	287.71	322.53	163.62	19.83	128.01
	SONATA	335.99	178.22	196.50	62.14	9.38	33.09
	OPUS	404.96	203.44	232.26	83.02	22.92	25.68

and

Table 12. Ranking of results for individual profit criteria within the HS, NZ and ST domains per 1 million PLN spent on apparature.

		IF Points	Papers with IF	Papers	Conference Publications	Book Publications	Projects
HS	PRELUDIUM	2	3	1	1	2	1
	SONATA	1	1	2	2	3	3
	OPUS	3	2	3	3	1	2
NZ	PRELUDIUM	1	1	1	1	1	1
	SONATA	3	3	3	2	2	2
	OPUS	2	2	2	3	3	3
ST	PRELUDIUM	1	1	1	1	2	1
	SONATA	3	3	3	3	3	2
	OPUS	2	2	2	2	1	3

.

Analyzing the ranking results of average IF value per 1 million PLN spent on apparatus in the case of HS, we can conclude that SONATA obtained the highest score, followed by PRELUDIUM and OPUS in the last place. In the case of NZ, PRELUDIUM placed the first position, OPUS was the second and SONATA third. Within ST, PRELUDIUM had the best result again when OPUS came second, and SONATA took the last place in the ranking.

When it comes to a total number of papers with IF value per 1 million PLN spent on apparatus in the example of HS, we can conclude that SONATA was placed on the top, when OPUS and PRELUDIUM were in worse positions. In the case of NZ, PRELUDIUM proved to be better assessed than OPUS and SONATA in the last place. Analyzing the ST domain, the situation in the ranking is the same as in the case of NZ with PRELUDIUM as the best choice.

When it comes to a total number of papers per 1 million PLN spent on apparatus in the example of HS, PRELUDIUM was placed in the first place, SONATA was the second and OPUS closes the ranking. In the case of NZ, PRELUDIUM proved to be better than OPUS and SONATA. In ST domain, PRELUDIUM scored the best result before OPUS and SONATA.

In the case of the total number of conference publications per one million PLN apparatus cost, analyzing HS and NZ domains, the ranking presents PRELUDIUM as the best one, before SONATA and OPUS in the third place. In the case of ST, PRELUDIUM was the best again, OPUS took the second place, and SONATA was the least attractive.

Analyzing the ranking results of total number of book publications per one million PLN spent on apparatus in the case of HS, we can conclude that OPUS obtained the highest score, followed by PRELUDIUM and SONATA in the last place. In the case of NZ, PRELUDIUM placed the first position, SONATA was the second, and OPUS closed the ranking as the last one. Within ST, OPUS had the best result again when PRELUDIUM came second, and SONATA took the last place in the ranking.

The last analyzed criterion is the number of projects per 1 million PLN cost of apparatus. In the case of HS, PRELUDIUM proved to be better assessed than OPUS and SONATA as the worst option. In the case of NZ, PRELUDIUM obtained the best result in the ranking, the second was SONATA, and the last place came to OPUS. In ST domain, PRELUDIUM dominated second-placed SONATA and third-placed OPUS.

Projects financed by the PRELUDIUM programme scored highest in terms of number of articles, conference publications and projects in the area of HS. PRELUDIUM projects achieved highest rank for all the criteria examined for NZ area and for all the criteria except the number of books published in the ST domain.

3.3. Analysis of Cost Criteria for Implemented Projects

For research to be authoritative and reliable, it is essential that it is multidirectional and that many different aspects of the problem are considered. For this reason, Section 3.3 has been devoted by us to analysing the effectiveness of the research against cost criteria. In this group of studies, the low value of the surveyed indicator shows high efficiency and productivity. In the list below we present 3 cost criteria that we decided to include in our research.

Criteria analysed:

• Cost of equipment per project;
• Duration of the project in months;
• Grant amount allocated to the projects;

3.3.1. Methods

In order to calculate the values of all of the above mentioned indicators, we used the Equation (3)

$$V_3=\frac{S_t}{P_t}$$

(3)

where:

$V_3$ means the value of the calculated indicator;

$S_t$ means aggregate sum of a given parameter within a specific competition and panel;

$P_t$ means aggregate amount of projects within a specific competition and panel.

The resulting value of calculated indicators is the average value of a given parameter within all analyzed projects. Calculating indicators using this method is a good solution for our study, because we have a very large number of projects, there are no data gaps, they were verified by us at an earlier stage for correctness, so they do not require further processing. Thanks to this, we avoid distorting the values, and thus the results obtained and the final results of the study present the actual situation.

The first cost criterion, i.e., the cost of equipment necessary for the implementation of projects, should be considered individually within each scientific field, due to different needs for equipment facilities for different fields. However, if the results in some competitions are lower than in others, this is a signal of a low level of their co–financing, which is contrary to the assumptions of sustainable project financing.

The second cost criterion, namely duration of the project is an important indicator for the evaluation of the project implementation, as the results of the project are more up–to–date and useful, without unnecessarily prolonging the project duration. This in turn contributes to higher values for the profit criteria indicators.

Low values of the third parameter examined by us in this section, which belongs to the group of cost criteria, i.e., the amount of funding allocated to the implementation of projects, especially when observing much higher values for projects from other competitions and fields, indicate their insufficient funding and should encourage us to take steps to balance the funding, especially if discrepancies appear within different competitions. Differences within domains may be due to their specificities and different needs, and therefore for different fields of science the results of this study should be interpreted individually.

3.3.2. Results

The results for analysis in the form of average values of equipment cost per project, project duration in months and the total grant amount allocated to the projects were obtained using a script in Python language.

The results obtained, that make it possible to illustrate and compare the effectiveness of grant programmes, are presented graphically in the form of column charts and discussed in detail in Appendix A.3.

For the parameter total cost of apparature per one project in all domains and panels, PRELUDIUM scored best. The value of this cost criterion stands out in terms of much lower values for PRELUDIUM, compared to other competitions.

For the parameter duration in months per one project the situation is identical, PRELUDIUM wins in all fields and panels. In case of the parameter grant amount per one project the trend of obtained results is identical, PRELUDIUM wins here too.

The clear domination of the best results for PRELUDIUM for the cost criteria we evaluated proves the high efficiency and profitability of projects financed under this competition.

The obtained results are presented graphically in

Table 13. Results as average values for cost individual criteria.

		Apparature Cost	Duration in Months	Amount
HS	PRELUDIUM	2852.83	23.64	65,685.26
	SONATA	10,051.95	35.77	175,635.87
	OPUS	9320.25	28.86	175,302.80
NZ	PRELUDIUM	11,080.43	26.15	134,432.20
	SONATA	72,851.98	39.73	511,147.94
	OPUS	66,275.90	34.63	463,473.37
ST	PRELUDIUM	11,053.95	24.95	102,811.00
	SONATA	71,848.49	37.37	389,108.75
	OPUS	74,898.42	32.96	418,758.13

and

Table 14. Ranking of results of cost criteria for individual competitions within the HS, NZ and ST domains.

		Apparature Cost	Duration in Months	Amount
HS	PRELUDIUM	1	1	1
	SONATA	3	3	3
	OPUS	2	2	2
NZ	PRELUDIUM	1	1	1
	SONATA	3	3	3
	OPUS	2	2	2
ST	PRELUDIUM	1	1	1
	SONATA	2	3	2
	OPUS	3	2	3

.

Analyzing the ranking results of the total cost of apparatus in PLN per one project in the case of HS, we can conclude that PRELUDIUM obtained the highest score, followed by OPUS and SONATA in the last place. In the case of NZ, PRELUDIUM has placed the first position again, OPUS was the second and SONATA third. Within ST, PRELUDIUM had the best result when SONATA came second, and OPUS took the last place in the ranking.

When it comes to the duration in months per one project in the example of HS, we can conclude that PRELUDIUM was placed on the top, when OPUS and PRELUDIUM were respectively second and third. In the case of NZ, PRELUDIUM proved to be better assessed than OPUS and SONATA in the last place. Analyzing the ST domain, the situation in the ranking is the same as in the case of HS and NZ with PRELUDIUM as the best choice, before OPUS and SONATA.

The last analyzed criterion is the grant amount per project. In the case of HS, PRELUDIUM proved to be better assessed than OPUS and SONATA as the worst option. In the case of NZ, PRELUDIUM obtained the best result in the ranking, the second was OPUS, and the last place came to SONATA. Within ST, PRELUDIUM reached the best score again when SONATA came second, and OPUS took the last place in the ranking.

In the case of cost criteria, PRELUDIUM projects were ranked first in all criteria for all the scientific disciplines analysed by us.

4. Discussion

In general, it is difficult to compare different scientific panels. The differences between the panels may result not only from the differences in the effectiveness of conducting grants, but also from the average IF values of journals in a given field. Undertaken analysis is an example of the use of typical statistics and the creation of new measures to compare complex objects that are part of a dynamically developing interdisciplinary field of data science [31]. Our research has shown a very high effectiveness of PRELUDIUM projects in most of the criteria analyzed in comparison to the other two competitions included in this research called SONATA and OPUS. This fact applies to profit and cost criteria. The most recognizable criterion of effectiveness in the international scientific community is the Impact Factor. In this respect, projects financed under the PRELUDIUM achieved the optimal results for the scientific fields of NZ and ST, and for the HS domain-the second place. This demonstrates the outstanding advantage of PRELUDIUM in terms of IF criterion, especially as the results achieved by projects in all competitions within NZ and ST are much higher than those achieved within HS. This results from the adopted measure of the effect-the IF impact coefficient, which prefers English language journals and non-humanistic sciences [31]. In addition, the PRELUDIUM has the advantage that significantly facilitates the career development of early-stage researchers, as it is aimed at researchers without a PhD degree. In this study, PRELUDIUM projects account for 39.31% of all analysed projects in the three competitions. Supporting young scientists by NSC is therefore a decision that brings benefits to individuals, universities and Polish science. NSC has announced on its website that it is committed to supporting young scientists by allocating no less than 20% of the funds at its disposal to support the development of early-stage researchers.

Use of a Python script to automate the task of collecting a large dataset has saved time and streamlined work needed. It provides great opportunities to detect the relationships between the data, as well as to make them visible and easier to analyze. However, data collected automatically should always be verified for correctness, because the sources may contain errors such as those described in the previous part of this paper. By knowing the correct ranges of values that IF values for articles can take, the wrong data for projects, which accounted for 5.53% of all projects, was corrected.

5. Conclusions

The conclusion is that PRELUDIUM projects yield very outstanding results in comparison with the other two competitions for the vast majority of the assessed criteria within the scientific fields of HS, NZ and ST. There is no sustainability in the individual panels. The leading PRELUDIUM program almost everywhere, is slightly underfunded. That’s why we can also assume that every penny is spent by PRELUDIUM grants most effectively.

After analysing the results, it can be concluded that the results for the HS area are slightly different from those for the ST and NZ areas. First of all, for most of the parameters assessed, the performance of ST and NZ areas is much higher than that of HS, which shows that NSC-funded projects in ST and NZ areas generate higher returns than projects in HS areas. It also means that the different scientific fields differ and are best evaluated separately. It also indicates that there are different funding needs and opportunities within the different domains and panels, so it is appropriate that monitoring and funding of projects within the different domains and panels should be planned, considered and implemented on a case-by-case basis. Analysing the results obtained, it can be concluded that this is the case, since the values of the last cost criterion we examined, i.e., grant amount per one project, are much higher for ST and NZ than for HS, so the financing is tailored to different needs and possibilities from one area to another. In the case of this criterion, it is noted that the funding values are in all areas much lower for PRELUDIUM than for SONATA and OPUS (in most cases, projects financed under PRELUDIUM receive more than 50% less funding than for SONATA and OPUS. In order to ensure sustainable development opportunities for young scientists implementing scientific projects under PRELUDIUM, it seems appropriate to consider increasing the level of their co-financing, especially because these projects achieve very favourable results compared to other programmes in terms of the vast majority of parameters we analyse.

However, deeper reflections require further analysis. The task of evaluating the effectiveness of grant programs is difficult. Still, the results obtained allow making assumptions that in further works, one can build an objective model that will indicate a generalised ranking, taking into account the specificity of fields and disciplines. This is a significant challenge that could enable more sustainable financing of researchers and better efficiency of public spending. Future work will focus on building a tool for qualitative evaluation of projects with the use of simple bibliometric data. For this purpose, an appropriate method of multi-criteria decision support should be first chosen, what is not trivial research task [32].