Impact of Changes to Procedures on the Evaluation of the Effectiveness of Forms of Professional Activation in Poland

Abstract

Active labour market policy is connected with the necessity to account for the funds allocated for it. The conducted analysis forms a part of research on the evaluation of the effects of changes introduced by legal regulations. The aim of this research is to assess the impact of changes to the methodology of calculating on values of the cost and employment effectiveness of basic forms of economic activation in Poland. They were introduced in 2015 in connection with ongoing discussion regarding the effectiveness of the evaluation methods used. The Ministry of Economic Development, Labour and Technology is currently responsible for activating the unemployed in Poland, and funds come from the Labour Fund. The analysis used is the regression discontinuity design. This analysis showed that significant changes occurred only in the slope of the regression line for cost effectiveness after 2015 for both procedures of its calculation. This shows that the new, introduced methods of calculating effectiveness were cosmetic in nature and did not significantly affect their values. A good recommendation for improving the method of evaluating forms of economic activation of the unemployed could be to extend the time of required employment.

1. Introduction

One of the most important economic and social problems in the modern world is the phenomenon of unemployment. It has a negative impact both on the economic growth of countries and on the functioning of households. Therefore, various tools are used to mitigate the effects of unemployment. Their aim is to limit its duration and prevent the depreciation of human capital. Within the framework of active labour market policy, governments apply various forms of support and activation of people who remain unemployed. Most often, public funds are used for this purpose. In order to confirm the validity and effectiveness of the use of such measures, evaluation studies are conducted. Due to the social aspect, it is important to employ as many job seekers as possible. From the point of view of institutions financing active support programmes, it is necessary to evaluate the spending of funds. In the case of unemployment, research may concern many problems, such as: the identification of groups of people at risk of long-term unemployment, assessment of the impact of implemented programmes on getting people out of unemployment, and monitoring the spending of funds allocated for these purposes. These analyses are largely based on the research work of James Heckman (Heckman et al. 1999), winner of the Nobel Prize in Economics. Research on the effectiveness of labour market policies in different countries focuses on different aspects. In the European Union, employment effects are mostly studied, as the main objective is to avoid the high costs of long-term unemployment. In the United States, the income effects of active employment programmes are mainly analysed. Labour market research indicates that the amount and duration of unemployment benefits have a positive effect on time spent looking for employment (Moffitt 1985; Katz and Meyer 1990; Meyer 1990; Hunt 1995; Card and Levine 2000; Hahn et al. 2001; Lalive 2007; Bieszk-Stolorz and Markowicz 2015). Overly protective state social policies may foster the occurrence of long-term unemployment. A better impact on the process of getting out of unemployment is given by the implementation of various programmes increasing professional activity. Such activities are also carried out in Poland. However, there is a problem with the evaluation of the actions taken. The catalogue of basic forms of activation is updated from time to time. In 2015, following comments from the bodies controlling the labour offices in Poland, a major modification was introduced in the way of calculating the applied measures.

The aim of this research is to assess the impact of changes, since 2015, to the methodology of counting the level of cost and employment effectiveness of basic forms of economic activation in Poland. The choice of Poland is dictated by the fact that it represents post-communist countries that have undergone systemic transformation. It does not have as much experience of a market economy, and therefore of unemployment, as the countries of Western Europe. It is not a welfare state due to its lower level of economic development compared to the highly developed countries of Western Europe. It is therefore interesting to see how the new regulations improve the situation.

2. Conceptual Background

Many methods for measuring the impact of interventions are described and applied in the literature. Among them, the Regression Discontinuity Design (RDD) is a popular model. It was first introduced in the social sciences by Thistlethwaite and Campbell (1960). They analysed the influence of awards on the future achievements of students (career aspirations, registration for postgraduate studies, academic achievements, etc.). Their study was conducted through an award that was based on an observed test score. This award, the National Merit Award, was presented to students who achieved a specified minimum score. Differences in the success of students who scored above and below this score were attributed to the impact of the award. The prestige of the award was found to have an effect on improved test scores, making it more likely to be received, but the hypothesis that the award received influenced students’ subsequent attitudes and career plans was not confirmed.

Although the regression discontinuity design has been known for several decades, its use in research has been quite rare (Cook 2008). It began to be used increasingly in socio-economic analysis from the late 1990s onwards. Since then, there have been many articles in the fields of education, the labour market, forensics, environmental protection, and health care, on the economic impact of regulatory change (Lee and Munk 2008; Lee and Lemieux 2010; Dong et al. 2019; Lotsu et al. 2019; Wang et al. 2019, 2021; Ogunmodede et al. 2020; Song et al. 2020; Zhao et al. 2021).

An example of the application of the RDD model can be found in the assessment of the labour market impact of a tax introduced in France in 1987, and modified in 1992, called the Delalande tax (Behaghel et al. 2008). It was a tax paid by employers who dismissed workers aged 50 and over. In fact, it became the cause of the frequent dismissal of workers before the age of 50 and a general reluctance to hire older workers. In 1992, a change was introduced to provide tax exemptions for companies employing people aged 50 and over. After this change, the transition rate from unemployment to employment for older workers increased significantly. The use of the RDD model also made it possible to assess the transition rate at age 50. It turned out that the legal act from 1987 resulted in rates for those under 50 and those aged 50 and over that were not significantly different, although younger workers found work slightly faster. Legislative changes in 1992 resulted in a significant change. It turned out that the chances of finding a job for unemployed older people increased significantly compared to younger ones.

Many articles on the application of the RDD model in economics have been written by Lalive. In 1988, the Austrian government introduced an extension of the duration of unemployment benefits from 30 to 209 weeks. This applied to unemployed people aged 50 and over, living in certain areas of Austria for at least 6 months, and previously working. Such a large change in benefit collection increased the duration of unemployment, especially for women, due to the proximity to retirement (Lalive 2007). Lalive’s study used an RDD model in a sharp form with an eligibility threshold concerning age and distance from the border of the risk regions.

Brügger et al. (2009) hypothesised that cultural differences may influence the magnitude of unemployment. The analysis they conducted was based on a comparison of communities in Switzerland separated by a linguistic and mental barrier between a part of the country with French, Italian and Rhaeto-Romanic languages and a German-speaking part. Public employment services operate in the same way throughout Switzerland. However, people living in the Latin-speaking part (the contractual border area) took about 18% longer to look for a job than their neighbours and were about 40% less likely to find a job. The authors showed that in the search for work, cultural differences are economically as important as the duration of benefits or changes in their amounts.

Sharp discontinuous regression models with multiple age thresholds were used by Schmieder et al. (2012) to study the effect of potential benefit duration on unemployment duration in Germany between 1980 and 2008. These authors emphasise that the study was based on a liquidity-constrained job search model and the assumption that the German unemployment benefit system is ideal and should therefore be used for this type of study. They took as discontinuity thresholds the age thresholds determining the potential duration of benefit receipt (42, 44, and 49 years). They confirmed the hypothesis that the duration of benefit receipt extended the job search period significantly. The study also showed a discontinuity in the duration of benefit collection according to the age of the unemployed person. These authors hypothesised that companies took into account the more favourable age thresholds for extending the duration of benefit collection when an employee was made redundant.

Markowicz and Bieszk-Stolorz (2016) analysed the impact of the implementation of a programme increasing professional effectiveness on the duration of unemployment. The premise of the programme was to equalise the labour market opportunities of young people under 30 years of age by adapting their professional qualifications to labour market requirements. The use of a regression discontinuity design allowed the identification of the impact of the programme increasing professional effectiveness on the duration of unemployment, i.e., the assessment of the effectiveness of the programme. Two groups of unemployed people were compared: those included and those not included in the programme. Despite a decrease in the average duration of unemployment in both groups, the positive effect of the impact of the implemented project in the threshold of 30 years was statistically significant.

Decramer and Vanormelingen (2016) analysed the effects of an investment subsidy programme for small- and medium-sized enterprises in Flanders for the period 2004–2009. The subsidies were awarded according to a ranking system that favoured young, growing, and productive firms with a strong cash flow, granting subsidies to the highest scoring firms until the depletion of funds. They confirmed there was a positive effect on firm-level investment, employment, output, and productivity for the firms that were granted the subsidy, but only for small firms. This effect was small in relation to the cost of the subsidy.

Cadil et al. (2018) studied the impact of public R&D support to private enterprises in Czechia on their economic sustainability. They showed that policies supporting the private sector led to higher innovation activity, but did not lead to higher value added and productivity of the supported entities.

Cansino et al. (2018) studied the Labour Market Insertion Contract, which was an Active Labour Market Policy (ALMP) tool introduced in Spain. It targeted young people who had difficulty entering the labour market, and was introduced to reduce the unemployment rate. A microeconomic analysis was carried out based on causal statistical inference by using propensity score matching and kernel and radial estimators. The analysis identified that the employability of people who were recipients of an insertion contract was lower in the short and medium term than that of individuals who had signed a temporary employment contract. This research contributes to the literature by evaluating whether there is empirical evidence to support a political decision to revoke an ALMP.

A very big problem of the contemporary labour market is the phenomenon of NEETs. This is the term used for young people not in employment, education, or training. Ruesga-Benito et al. (2018) examined the impact of socio-economic variables related to the level of national welfare and social protection, as well as the risk of poverty for and the social exclusion of young NEETs. The highest proportion of such individuals was in Greece and Italy. The main findings pointed to the need to strengthen social protection as a key factor to avoid the discouragement of young people, thus reducing the likelihood of their becoming NEETs. Such measures should be broad-based and cover economic and social policies, including health and education.

Castaño et al. (2019) applied counterfactual analysis to evaluate the way in which European Rural Development Programmes (RDPs) were impacted between 2007 and 2013. They concluded that there was no adequate counterfactual method to evaluate ex-post socio-economic impacts as required by the evaluation standard. Indeed, the limiting factors were the availability of data, certain methodological assumptions, and constraints on evaluation resources such as expertise, time, budget.

3. Forms of Activation of the Unemployed in Poland and Methods of Assessing Their Effectiveness

After Poland’s accession to the European Union, the scope of actions related to the professional activation of the unemployed, addressed to groups of people in a difficult situation in the labour market, increased. In 2004, the Act on Promotion of Employment and Labour Market Institutions (Journal of Laws 2016, No. 99, item 1001) was introduced. Its objective was to adjust Polish labour market policy to the European Employment Strategy. The act divided actions activating the unemployed into labour market services and instruments. Labour market services included assistance in seeking work. It consisted of: job placement, vocational counselling, professional information, assistance in active job seeking, and the organisation of training. Professional activation of the unemployed was connected with the introduction of such instruments as: intervention works, public works, professional preparation in the workplace, internships, and funds for creating jobs. Since 2004, dozens of modifications have been made to the act. These modifications aimed to make the act more effective and to ensure that it adapted to changing needs and socio-economic conditions. Thanks to this act, there has been an expansion of access to labour market services, a promotion of lifelong learning programmes, and a modification of the system of unemployment benefits, as well as the implementation of a system of incentives for starting economic activity and creating new jobs.

The Ministry of Economic Development, Labour and Technology is currently responsible for the activation of the unemployed in Poland. Its name changed several times during the analysed period. In Poland, the activation of the unemployed is directly handled by voivodeship and poviat labour offices. Their task is to obtain funds and organise activation measures. Resources come mainly from the Labour Fund, to which contributions are paid by employees and companies. Other resources come from the EU projects.

The level of the registered unemployment rate in Poland fluctuated between 2004 and 2019. In 2004 it was equal to 19% and in the following years it decreased to 9.5% (2008). Then, as a result of the global financial crisis, it rose to 13.4% (2012). It remained at the same level in 2013 and then decreased to 5.2% in 2019. The number of registered unemployed changed in a similar way (Figure 1).

In order to counteract unemployment, various forms of support for the unemployed are implemented in labour offices. The catalogue of basic forms of professional activation is determined by the minister in charge of labour. In 2005–2019 this catalogue was subject to changes. In 2005, it included 7 forms: training, employment within the framework of intervention works, employment within the framework of public works, internships at employers, preparation of adults for occupation in the workplace, granting the unemployed one-time resources to subsidise starting economic activity, and granting employers refunds for costs of equipment or additional equipment for the unemployed sent to these workplaces. In 2006–2009, an eighth form was added: performing socially useful work. In 2010–2013, there were 7 forms—preparation for occupation in the workplace was removed from the catalogue. In 2014–2019, the catalogue included 6 forms—socially useful works was removed. Due to their auxiliary nature or insignificant share in the expenditures incurred, in the years 2005–2019 some forms of support were not included (e.g., activation supplement, scholarships for the period of continuing education, refund of travel and accommodation costs, refund of social insurance premiums for farmers dismissed from work, training loans). In the analysed period, the largest number of people benefited from trainings and traineeships.

Figure 1 presents the number of unemployed people participating in activation programmes in 2005–2019. In 2005–2010, this number grew, and was much higher than in subsequent years. In the analysed period, on average, slightly more than 92% of people that had started participation in selected forms of economic activation had completed it, but only about 60% of them found employment. In 2011, there was a clear decrease in the number of the unemployed participating in programmes. This is due to the fact that a large number of people started participating in activation forms in the previous year and finished in the following year. Poland was placed under the excessive deficit procedure and, in line with the Ecofin Council recommendation (of 7 July 2009), was required to correct its general government deficit below 3% of GDP by 2012. Being subject to this procedure entailed a reduction in spending by the Labour Fund, which is part of this sector.

Numerous labour market studies show that the impact of characteristics of the unemployed on the risk of entering unemployment and re-entering employment varies greatly. Bieszk-Stolorz (2020) showed that age and education influenced the risk of multiple registrations in Polish labour offices, while gender and seniority had no significant impact. In a similar way, characteristics of the unemployed influenced the risk of first returning to the office. However, they did not influence subsequent registrations. Researchers often draw attention to the need for flexible family-friendly policies that would allow women to enter the labour market after maternity. Landmesser (2014) in her study showed that the exit rate from unemployment for women was lower than for men. The gender gap narrowed after about age 40. The transition rate to non-participation was higher for women than for men, but this difference also disappeared with age. The unemployment phenomenon also varies by the age of the unemployed. Guijarro (2018) analysed the effectiveness of active labour market initiatives in the Valencia region, one of the areas most affected by unemployment in Spain in 2016. He showed that those most at risk of unemployment were those aged 55 and over and women. In the case of women, only those with higher education had a high probability of finding a job. A similar study of the labour market in Poland was conducted by Bieszk-Stolorz and Dmytrów (2018). Using classification trees and survival analysis methods, they showed that among the long-term unemployed, women aged 60+ had the greatest problems returning to work, while young people with tertiary education had the least problems.

Unemployment in Poland is strongly spatially differentiated (Woźniak-Jęchorek 2015; Murawska 2016; Tatarczak and Boichuk 2018; Bieszk-Stolorz and Dmytrów 2019, 2020). The Polish labour market is strongly distorted by annual fluctuations of other macroeconomic economic factors (Hadas-Dyduch et al. 2016). There are a number of problems limiting or preventing the effective activation of vulnerable people in the labour market. The most important of these is the low flexibility of the European Social Fund. In addition to increasing flexibility, it is also recommended to individualise projects, which would contribute to an effective transition from the passive to the active labour market.

To evaluate the effectiveness of the applied forms of economic activation in Poland, two basic measures are used: employment effectiveness and cost effectiveness. The very concept of effectiveness is the subject of many discussions and analyses. First of all, it is a tool for measuring the effectiveness of management, as it makes it possible to determine to what extent set goals are achieved. It describes the relation of achieved effects to incurred expenditures. In economic theory, economic effectiveness is considered in two aspects. One of them is connected with the pursuit of effectiveness maximisation, i.e., achieving the highest possible result from a given amount of outlays. The second one results from striving to save resources, i.e., the achievement of a selected level of the result, while minimising the outlays (costs). In the case of the assessment of the effectiveness of public sector functioning, actions are taken in the direction of the assessment of the effectiveness of the use of state policy instruments, in particular, the assessment of the legitimacy of the further use of a given instrument and the indication of more effective solutions. On the labour market, support programmes for the unemployed are monitored. From the social point of view, it is important to employ the maximum number of job seekers. On the other hand, from the point of view of institutions financing active support programmes, it is important to reliably assess the expenditure of funds. In order to evaluate the effectiveness of implemented programmes, the cost of participation in a form of activation, and the employment and cost effectiveness ratios are determined. These values are determined for basic forms of economic activation.

Employment effectiveness (re-employment rate) is an indicator that measures the probability of finding a job after participating in a programme. It is determined by the formula:

$$EE=\frac{number of persons employed during or after end of participation}{number of persons that ended their participation}·100\%$$

(1)

Cost effectiveness (cost of re-employment) assesses the cost of getting an unemployed person into employment. It is expressed by the formula:

$$CE=\frac{amount of expenditure from the Labour Fund}{number of persons indicated as employed}$$

(2)

The participation cost in a form of activation determines the average cost of activating one person. It is calculated by means of the formula:

$$PC=\frac{amount of expenditure from the Labour Fund}{number of persons who participated in a given form}$$

(3)

In 2015, there was a change in the methodology for counting effectiveness. This change concerned two aspects: the definition of the end of participation in activation and the definition of employment. Before 2015, the date of completion of participation in activation was the date of completion of participation in a given form of support (trainings, traineeships) or the date of completion of financing from the Labour Fund (public works, intervention works, socially useful works, refunds for equipping or retrofitting a workstation). The change to the definition of employment is related to the change to the conditions after fulfilment of which an activated person can be considered employed. Before 2015, a person was considered employed if, after completing participation in a particular form of activation, they did not re-register with a poviat labour office within 3 months. It could happen that some people lost their jobs and did not re-register, and in the calculations they were considered as employed. In our research, this method for defining the end of participation and employment will be called the old procedure.

The provisions of the Act of 20 April 2004 on employment promotion and labour market institutions (Journal of Laws 2016, item 645, as amended) defined the date of completion of activation as the moment when the obligation period ends. In the case of refunding the costs of providing additional equipment or equipping a workstation, this is the period during which the employer is obliged to maintain the workstation created or equipped with the funds from the Labour Fund. Since 2015, it has also been required that an unemployed person, after participating in activation, works for at least 30 days over a period of 3 months and on this account (with few exceptions) social insurance contributions are paid to the Social Insurance Institution (Polish abbreviation ZUS). This method of defining the end of participation and employment will be called the new procedure.

These 2015 changes were of particular importance with regards to subsidising business activity and refunding the costs of equipping or retrofitting a workstation. Both these forms of activation are connected with a one-time payment of funds. At the same time, the duration of these forms is long. The evaluation of employment effects after three months from the payment of funds, i.e., in the period of obligatory running business activity, resulted in employment effectiveness for both forms being 100%, while cost effectiveness was equal to the average amount of subsidies or refunds. The modifications made were supposed to improve data quality.

Persons participating in a given form of activation are those for whom expenditures from the Labour Fund were made in a given year, regardless of the duration of activation or the multiple use of the same form of activation.

Persons who completed participation in a given form of activation are persons who in the examined year completed participation in a given form of economic activation financed from the Labour Fund, including persons who completed activation that commenced in previous years.

People who have been employed are those who, during or within 3 months after completing participation in a given form of activation, took up a job and kept it for at least 30 days, according to data obtained from the Social Insurance Institution.

In the analysed period, the highest cost effectiveness was characterised by subsidies for starting economic activity and refunding costs of equipping or retrofitting a workstation. On the other hand, the highest employment effectiveness was achieved by subsidies for starting business activity and intervention works. The greatest number of people participated in internships organised by labour offices. In Poland, groups of people most at risk of unemployment are identified. These are the unemployed in a special situation in the labour market. They include:

• the unemployed under 30 years of age (with a separate group under 25 years of age);
• the long-term unemployed;
• the unemployed aged over 50;
• the unemployed using social welfare benefits;
• the unemployed having at least one child under 6 years of age or at least one disabled child under 18 years of age;
• the disabled unemployed;
• caregivers of a disabled person.

These people have priority in being referred to special programmes.

This study will compare the effectiveness determined after the change from 2015 (new procedure) to the effectiveness determined on the basis of the old regulations (old procedure). Figure 2 presents the cost of participation, cost and employment effectiveness, and the activation programmes in Poland in 2005–2019 determined on the basis of the new and old procedures. In Figure 2 and Figure 3, values representing cost effectiveness and cost of participation are presented in USD. We used the exchange rate as of 23 May 2021 (1 USD = 3.69 PLN). After the introduction of the new way of defining people who have taken up employment since 2015, the cost of participation has not changed, but the value of both forms of effectiveness has changed. Still, the cost incurred per person employed (cost effectiveness) was much higher than the cost per person starting to participate in the programme.

In the years 2006–2010 a large increase in cost effectiveness (from about PLN 6.5 thousand/USD 1.76 thousand to about PLN 13 thousand/USD 3.52 thousand) was accompanied by a relatively constant employment effectiveness at the level of about 55% (Figure 2). In 2011 there was a significant drop in cost effectiveness—compared to the previous year—from about PLN 13 thousand (USD 3.52 thousand) to about PLN 9 thousand (USD 2.44 thousand). This was related to the correction of the general government deficit described earlier. Then, until 2015, both types of effectiveness showed an upward trend, and after 2015, employment effectiveness continued to increase with a decreasing value of cost effectiveness. While the increase in cost effectiveness should be considered as a negative phenomenon (from the point of view of public finances), it should be noted that at the same time the absolute difference between this effectiveness and the cost of participation decreased. All this indicates a better use of funds for the economic activation of the unemployed in 2015–2019.

Criticism of the applied measures of effectiveness concerns mainly the manner of determining the number of people employed after completing participation in the selected form of activation. The analysis conducted by the Ministry of the Family, Labour and Social Policy concerned a short period, i.e., three months. The Supreme Audit Office (Polish abbreviation NIK) conducted an audit concerning the durability of employment after two years from the beginning of the activation process. In the opinion of the Supreme Audit Office, employment effectiveness indicators set by poviat employment offices were overestimated. Due to taking into account forms of activity with one hundred percent effectiveness assumed in advance, the phenomenon of cream skimming (cherry picking) may occur. According to the methodology adopted by the NIK, employment can be considered permanent if after 2 years (or more) a person does not return to the list of the unemployed. The survey conducted according to this principle indicated that the least effective forms of activation were public works and traineeships. The permanence of employment remained at a high, almost unchanged level in the case of grants for business activity. It was shown that the greatest chances for finding a permanent job were given by new forms of support: employment vouchers and settlement vouchers. A total of 72% of people who used these vouchers did not return to the register of the unemployed after two years from being granted these forms of support. Unfortunately, these instruments were used to a very small extent. Active labour market policies in Poland have a positive effect on the labour market, consisting only in a temporary reduction in unemployment.

4. Research Methodology

In order to ensure that the assessment of the effectiveness of different programmes against unemployment is the same in the Member States benefiting from EU funds, the European Commission for the assessment of their effectiveness proposed counterfactual methods, which were included in the publication Design and Commissioning of Counterfactual Impact Evaluations. A Practical Guidance for ESF Managing Authorities (Morris et al. 2013). The aim of the methods contained therein is, inter alia, the assessment of the average impact effect related to the analysis of cause-and-effect relationships between phenomena (Landmesser 2011). The basic element of the models is a random variable describing two states: impact (Y₁) and no impact (Y₀). The relationship between the observed outcome and hypothetical outcomes can be presented as follows:

$$Y=d{Y}_1+\left(1-d\right)Y_{ }\mathrm{for}d\in \left\{0, 1\right\}$$

(4)

The value of the variable Y is modelled as a conditional expected value, with a known realisation of the vector of observed characteristics X. The average treatment effect (ATE) is given by the formula:

$$ATE=E(Y_1 | X)-E(Y_0 | X)$$

(5)

Popular evaluation research methods in this area include the regression discontinuity design (RDD) model.

The literature mentions two types of such models: sharp design and fuzzy design (Trochim 1984; Hahn et al. 2001). The sharp form can be used when the cut-off point (threshold) completely identifies the experimental group. If this is not the case and the individuals still have to fulfil an additional condition, then the fuzzy form of the model should be used. In the case of the conducted study, the sharp form of the model was used due to the fixed single condition.

On each side (left—l, right—r) of the cut-off point X₀, parameters are estimated for two separate regression functions (Lee and Lemieux 2010):

$$Y={\alpha }_i+f_i\left(X-X_0\right)+\epsilon \mathrm{for}i\in \left\{l,r\right\}$$

(6)

Model (6) can be written as follows:

$$Y={\alpha }_l+\tau D+f\left(X-X_0\right)+\epsilon \mathrm{for}\tau ={\alpha }_r-{\alpha }_l$$

(7)

where:

$$f\left(X-X_0\right)=f_l\left(X-X_0\right)+D\left[f_r\left(X-X_0\right)-f_l\left(X-X_0\right)\right]$$

(8)

D is a binary variable whose form depends on the position of the experimental group relative to a defined eligibility threshold $X_0$.

If the variable D is defined as follows:

$$D=\{\begin{array}{c}0\mathrm{for}X<X_0\\ 1\mathrm{for}X\ge X_0\end{array}$$

(9)

then the experimental group is defined by the condition X ≥ $X_0$ and the control group by X < $X_0$.

If the functions f_l and f_r are linear, they take the form:

$$f_i\left(X-X_0\right)={\beta }_i\left(X-X_0\right)\mathrm{for}i\in \left\{l,r\right\}$$

(10)

a model (7):

$$Y={\alpha }_l+\tau D+{\beta }_l\left(X-X_0\right)+\beta D\left(X-X_0\right)+\epsilon$$

(11)

where:

$\beta ={\beta }_r-{\beta }_l$.

$Y$—dependent variable;

$X$—independent variable;

$X_0$—threshold;

$\tau$—average impact effect at threshold $X_0$;

$D$—artificial dichotomous variable.

In model (12) the parameter τ determines the change in the value of the variable Y when passing through the threshold. If it is positive, there was an increase in the value at the threshold; if negative, a decrease in the value of the dependent variable.

Lee and Lemieux (2010) found that one of the main advantages of using a regression discontinuity design is that the results can be represented by simple graphs, which greatly increases the transparency of the method.

The following hypothesis H was proposed: the introduced changes to the method of calculating cost and employment effectiveness will not significantly affect their values. One of the counterfactual methods was used in the analysis: the model of regression discontinuity design. The study was based on statistical data published by the Ministry of Family, Labour and Social Policy in Poland (MRPiT 2021).

5. Empirical Results

In 2015, the methodology for calculating the employment effectiveness for the basic forms of professional activation changed and the conditions under which an activated person can be considered employed changed. This resulted in a change in the value of both forms of effectiveness, while the cost of participation of the unemployed in the forms of professional activation did not change. The differences concern the values of both forms of effectiveness in 2015–2019.

Since the change in the calculation of effectiveness occurred from 2015, it was decided to take this year as the threshold in the regression discontinuity design model. The model parameters were estimated in two subgroups. The first consisted of the actual effectiveness values, i.e., according to the new way of calculating them (new procedure). The second subgroup consisted of both forms of effectiveness calculated according to the old rules (old procedure). Then, using a regression discontinuity design model, it was assessed whether and to what extent the change in the way both forms of effectiveness were calculated significantly affected their change in the threshold.

Results relating to cost effectiveness are presented in PLN and USD (according to the previously applied exchange rate 1 USD = 3.69 PLN).

In the first stage of the study, cost effectiveness models were estimated. For reasons of monotonicity in the sub-periods, data from 2011–2019 was considered (Figure 2). Subsequent years were denoted $X_i$ for i = 1, 2, …, 9. In this case, the eligibility threshold (2015) has a value of $X_0=X_5=5$. To estimate the difference in cost-effectiveness $Y_i$ in the threshold the model was used:

$${\widehat{Y}}_i={\alpha }_l+\tau D+{\beta }_l\left(X_i-X_0\right)+\beta D\left(X_i-X_0\right)$$

(12)

where:

Y_i—cost effectiveness;

X_i—year number;

$\tau$—average impact of the change on cost-effectiveness at threshold $X_0$;

$X_0=5$.

Artificial variable $D$ is defined as follows:

$$D=\{\begin{array}{c}0 \mathrm{for} 1\le X_i<5\\ 1 \mathrm{for} 5\le X_i\le 9\end{array}$$

(13)

The results of the estimation of both models are shown in

Table 1. Results of parameter estimation of regression discontinuity design models (cost effectiveness).

Parameter	Parameter’s Estimator PLN (USD)	Standard Error PLN (USD)	p-Value	Parameter’s Estimator PLN (USD)	Standard Error PLN (USD)	p-Value
	New Procedure R2 = 0.8488			Old Procedure R2 = 0.8015
${\alpha }_l$	12,219.64 (3311.56)	658.76 (178.53)	0.0000	12,219.64 (3311.56)	606.77 (164.44)	0.0000
$\tau$	367.64 (99.63)	779.45 (211.23)	0.6570	−245.80 (66.61)	717.95 (194.57)	0.7460
${\beta }_l$	689.96 (186.98)	240.54 (65.19)	0.0351	689.96 (186.98)	221.56 (60.04)	0.0264
$\beta$	−1358.77 (368.23)	294.61 (79.84)	0.0058	−1042.06 (282.40)	271.36 (73.54)	0.0121

Source: own elaboration.

.

In the case of cost effectiveness calculated according to the new procedure, a model was obtained:

$$\begin{gathered} {\widehat{Y}}_{i\mathrm{PLN}}=\mathrm{12,219.64}+367.64D+689.96\left(X_i-5\right)-1358.77D\left(X_i-5\right) \\ {\widehat{Y}}_{i\mathrm{USD}}=3311.56+99.63D+186.98\left(X_i-5\right)-368.23D\left(X_i-5\right) \end{gathered}$$

(14)

Model (14) after transformations takes the form:

$$\begin{gathered} {\widehat{Y}}_{i\mathrm{PLN}}=8769.84+689.96X_i \mathrm{for}D=0 \\ {\widehat{Y}}_{i\mathrm{USD}}=2376.65+186.98X_i \mathrm{for}D=0 \end{gathered}$$

(15)

$$\begin{gathered} {\widehat{Y}}_{i\mathrm{PLN}}=16,931.33-668.81X_i\mathrm{for}D=1 \\ {\widehat{Y}}_{i\mathrm{USD}}=4588.44+181.25X_i \mathrm{for}D=1 \end{gathered}$$

(16)

If the old counting procedure is left in place, the discontinuous regression model has the form:

$$\begin{gathered} {\widehat{Y}}_{i\mathrm{PLN}}=12,219.64-245.80D+689.96\left(X_i-5\right)-1042.06D\left(X_i-5\right) \\ {\widehat{Y}}_{i\mathrm{USD}}=3311.56-66.61D+186.98\left(X_i-5\right)-282.40D\left(X_i-5\right) \end{gathered}$$

(17)

Model (17) after transformations takes the form:

$$\begin{gathered} {\widehat{Y}}_{i\mathrm{PLN}}=8769.84+689.96X_i\mathrm{for}D=0 \\ {\widehat{Y}}_{i\mathrm{USD}}=2376.65+186.98X_i \mathrm{for}D=0 \end{gathered}$$

(18)

$$\begin{gathered} {\widehat{Y}}_{i\mathrm{PLN}}=13,734.34-352.10X_i\mathrm{for}D=1 \\ {\widehat{Y}}_{i\mathrm{USD}}=3722.04+95.42X_i \mathrm{for}D=1 \end{gathered}$$

(19)

Equations (15) and (18) are the same in that they describe a linear model of cost effectiveness before 2015.

Equations (16) and (19) describe linear models of cost effectiveness from 2015 determined by the new and old counting procedures, respectively.

Regression discontinuity design models for cost effectiveness are shown in Figure 3.

The estimates presented in Table 1 lead to the following conclusions:

• The change in counting did not result in a significant jump in the cost effectiveness at the threshold, i.e., in 2015 (no significance of the parameter $\tau$ at the variable $D$);
• All the obtained models have significant slope coefficients. The signs standing next to these parameters indicate that, before 2015, cost effectiveness was increasing in subsequent years (positive value), and since 2015, for both counting procedures, it has been decreasing (negative value);
• The change in counting led to a significant change in the slope angle of the regression line from 2015 (significance of parameter $\beta$). This parameter was negative and decreased, which means that cost effectiveness calculated according to the new procedure was decreasing faster than according to the old procedure.

The second stage of the research consisted of estimating the parameters of the employment effectiveness models. Due to monotonicity, data from 2009–2019 were included in the sub-periods (Figure 2). Subsequent years were denoted as $X_i$ for i = 1, 2, …, 8, 9, 11. In this case, the threshold (2015) has the value $X_0=X_7=7$. To estimate the difference in employment effectiveness $Y_i$ in the threshold $X_0=7$  (2015), the model (12) was used. In this case, the artificial variable $D$ is defined as follows:

$$D=\{\begin{array}{c}0 \mathrm{for} 1\le X_i<7\\ 1 \mathrm{for} 7\le X_i\le 11\end{array}$$

(20)

Results of the estimation of both models are shown in

Table 2. Results of the estimation of parameters of the regression discontinuity design and linear regression models (employment effectiveness).

Parameter	Parameter’s Estimator	Standard Error	p-Value	Parameter’s Estimator	Standard Error	p-Value
	New Procedure			Old Procedure
Regression discontinuity design
${\alpha }_l$	0.7545	0.0278	0.0000	0.7545	0.0311	0.0000
$\tau$	0.0088	0.0362	0.8155	0.0448	0.0404	0.3046
${\beta }_l$	0.0424	0.0071	0.0006	0.0424	0.0080	0.0011
$\beta$	−0.0169	0.0118	0.1970	−0.0317	0.0132	0.0480
fit	R2 = 0.9615			R2 = 0.9529
Linear regression models
α	0.4762	0.0194	0.0000	0.4819	0.0269	0.0000
β	0.0374	0.0029	0.0000	0.0370	0.0040	0.0000
fit	R2 = 0.9500			R2 = 0.9062

Source: own elaboration.

.

Parameters $\tau$ and $\beta$ of both models are statistically insignificant (at $p=0.05$). These are the parameters standing by the variables associated with the artificial variable $D$. It was decided to remove this variable completely from model (12) and estimate an ordinary linear regression model:

$${\widehat{Y}}_i=\alpha +\beta X_i$$

(21)

where $\alpha ,\beta$ are the model parameters.

In this case, all parameters are statistically significant. The following two models were obtained:

$$\mathrm{For}\mathrm{the}\mathrm{model}\mathrm{according}\mathrm{to}\mathrm{the}\mathrm{new}\mathrm{procedure}:{\widehat{Y}}_i=0.4762+0.0374X_i$$

(22)

$$\mathrm{For}\mathrm{the}\mathrm{model}\mathrm{according}\mathrm{to}\mathrm{the}\mathrm{old}\mathrm{procedure}:{\widehat{Y}}_i=0.4819+0.0370X_i$$

(23)

The results of the estimation of the model parameters (21) for the new and old procedures are presented in Table 2, and the regression line plots are shown in Figure 4.

To investigate whether the change in the procedure for counting employment effectiveness after 2015 affected the rate of their growth, a test for equality of the two slope coefficients of the regression functions described by Equations (23) and (24) was performed. Student’s t-test was applied. The calculated t-value was 0.094. The p-value was 0.926, so there were no grounds for rejecting the null hypothesis that the growth rates of employment effectiveness calculated by both procedures did not differ significantly from each other.

The estimates presented in Table 2 lead to the following conclusions:

• The changes to the calculation of effectiveness introduced since 2015 did not result in a significant jump in the cost value at the threshold, i.e., in 2015 (no significance of the $\tau$ parameter at the variable $D_i)$, and also did not cause a significant change in the slope of the regression line since 2015 (no significance of the parameter $\beta$);
• The values of employment effectiveness have increased since 2009, both for the new and old calculation procedures (positive values of β parameters). However, the increase in the case of the new procedure is slightly higher (higher value of the slope coefficient of the simple regression);
• The change in the procedure for calculating employment effectiveness caused practically no differences in its growth in the period under study (the Student’s t-test did not show the significance of differences between the respective slope coefficients of the regression function).

6. Discussion

The conducted study confirms the hypothesis formulated earlier. The introduced changes to the method of calculating cost and employment effectiveness did not significantly influence their values. Remarks of controlling bodies concerning the method of calculating cost and employment effectiveness influenced the decision for a different definition of employment and termination of participation in activation. However, these changes were cosmetic in nature. Testing the effectiveness of the applied programmes of professional activation of the unemployed is not an easy task. The comments in the NIK report (NIK 2014) indicate reservations about the definition of people taking up employment after completing participation in programmes. The two-year period of employment suggested in this report as an effect of the programme seems to be correct. On the other hand, some evaluation studies require settlement of the programme in a shorter period, i.e., one year. Changes to the definition of termination of participation in activation and the definition of employment introduced since 2015 have solved this problem only partially. This is confirmed by the results of the conducted study. The new procedure for calculating effectiveness slightly lowered the value of employment effectiveness. This change was statistically insignificant. As a result of the new procedure, the variability of cost effectiveness increased slightly. This change was also statistically insignificant. The only significant change was the change in the rate of decline of cost effectiveness after changing the method of its calculation (it was slightly faster).

This study confirms the results of previous research conducted by authors (Bieszk-Stolorz and Dmytrów 2021). They analysed the impact of changes to calculation procedures on cost and employment effectiveness in Poland for voivodeships. The voivodeships were clustered using the k-means method for both the new and old calculation procedures. The impact of changes introduced since 2015 on clustering results was analysed. Differences were examined using a simple similarity measure. During the analysed period, the change in the procedure for determining performance measures generally did not result in large differences in clustering results. The exception was the year 2016. In 2018 the clustering results were identical. The voivodeships in which more funds were allocated to activation (there were high values of cost effectiveness) generally used these funds to a lesser extent (values of employment effectiveness were lower), and vice versa.

The analysis refers to Poland, but it can be applied to other economic or social aspects, also in other countries. This is an example in which certain regulatory changes did not have a significant impact on the formation of indicators determining the effectiveness of measures aimed at activating the unemployed.

Studies on the impact of changing legislation or policies on socio-economic phenomena lead to different conclusions. Sometimes, the impact significantly alters the process under analysis. An example is the study by Lotsu et al. (2019), who showed that the power factor correction policy implemented by Ghana’s government is effective. Similarly, Wang et al. (2019) showed that in northern China, air quality has gradually improved due to a series of heating policy changes implemented by the central government in recent years. Wang et al. (2021) analysed the correlation between the policy stringency index and the number of deaths from COVID-19 in the world. They showed that policy control measures had a significant effect on epidemic control. However, changes introduced do not always have a significant effect on the phenomenon analysed. An example is the study by Stancanelli (2012), which aimed to assess the impact of the introduction of an unfavourable pension policy in France in 1993 on the economic behaviour of spouses. The analysis showed that the introduced reform did not significantly affect the probability of retirement for married women, while it significantly reduced the probability of retirement at retirement age for married men.

This study used data from labour office registers. Full monitoring of people long after they have left the register is a difficult task. This is because it would require costly surveys. As highlighted by d’Hombres and Santangelo (2019), research approaches based on administrative data are the most common. In Europe, these data are most often used for counterfactual assessments of the impact of active labour market policies. These authors point out that counterfactual impact evaluations based on administrative data usually measure the impact of active labour market policies on several outcome variables, estimate heterogeneous effects across different subgroups, and examine short- and long-term effects. In contrast, survey-based analyses tend to be less comprehensive. Rettore and Trivellato (2019) point to similar advantages of this type of research. Factors limiting the study of the impact of policies on unemployment are also pointed out by Pesquera Alonso et al. (2021). In their study, they assessed the impact of the Youth Guarantee policy launched in 2013 by the European Union. It aimed to address high levels of youth unemployment and NEET rates. In their findings, they revealed the importance of time in policy implementation, differences and similarities between countries, and hidden problems in data collection that led to errors and misleading results. The authors concluded that it was too early to assess the usefulness of the policy and recommended improving the data collection process.

7. Conclusions

Several general conclusions can be drawn from the study:

• The changes to the method of calculation of both types of effectiveness did not significantly affect their values;
• Only the rate of change of cost effectiveness changed (although to a small extent);
• The obtained results confirmed the conclusions of earlier studies using other methods.

This analysis forms a part of the research on impact assessment of changes introduced by legal regulations. In this particular case, the change of calculation method was supposed to improve the value of applied performance measures. Despite changes to the definitions of individual elements, there was no significant change in the values of these measures. A good solution would be to better define people employed after participating in forms of professional activation. However, this would require better and longer monitoring of the employment history of the de-registered unemployed.

The study shows that the introduction of new regulations to improve the information value of indicators should be monitored and adapted to changing conditions. This is particularly important in the case of the unemployed. It is necessary to find activation measures best adapted to market needs. Therefore, in Poland, the catalogue of these measures and the way of calculating effectiveness indicators should be subject to changes.

Another remark is related to the method itself. The results obtained as a result of applying the regression discontinuity design model are worth presenting in graphical form. Graphs show especially well the analysed changes or their absence. The sharp form of the model presented in the paper is suitable for the analysis of socio-economic phenomena for which there has been a change of conditions or a change of the method of their measurement. This change must occur at a precisely defined moment (eligibility threshold). It can be the date of introduction of a new regulation/law or the date of a spectacular event which influences the analysed socio-economic phenomenon. The results obtained by applying a regression discontinuity design model make it possible to assess whether these changes have had a significant impact on the further course of the analysed phenomenon and whether there has been a significant change in its eligibility threshold. The sharp form of the model used in the presented study is useful in the analysis of phenomena that may be disturbed by changes in legislation or as a result of the introduction of new social programmes. Such changes usually take effect at a precise moment in time. An additional advantage of this method is its recommendation by the European Commission for evaluation studies.

Our direction for future research in this area will involve an analysis of the impact of protective measures in the labour market (due to the COVID-19 pandemic) on the effectiveness of forms of professional activation.