Background: Efficiency measurement by means of data envelopment analysis (DEA) in the non-profit sector has focused on the so-called Stage I of non-profit organisations, namely, fundraising efforts (which are the most influential determinants of raising funds in order to increase the amount of contributions). However, for the so-called Stage II of non-profit organisations, namely, spending the achieved resources to program services delivery, DEA studies are very scarce. In attempting to address this research gap and to the best of our knowledge, this investigation is the first study that applies DEA to the assessment of international cooperation schemata. Consequently, we offer a significant contribution to the literature by overcoming the limitations of other techniques used to assess the efficiency and providing new insight into the efficiency of targeted different international cooperation schemata (ICS) in international cooperation development projects.

Aim: The purpose of this study is to evaluate and compare the efficiency of the ICS of developmental projects funded by the Spanish Agency for International Cooperation for Development.

Setting: Our setting is composed of different international cooperation projects funded with different schemata by the Spanish Agency for International Cooperation for Development between 2002 and 2006 in two African countries that are top priority targets of Spanish international aid: Morocco, and Mozambique.

Methods: Using a sample of 48 international cooperation projects carried out in two African countries considered priorities in the Spanish Cooperation Master Plan, we analyse project efficiency using DEA.

Results: The findings suggest that some schemata are more efficient than others when applied to international cooperation projects (ICS). Specifically, we find that permanent open-call subsidies are more efficient than non-governmental development organisation subsidies.

Conclusion: Measures for evaluating international aid projects with respect to efficiency are problematic. The DEA method provides an ex-post meausure of efficiency that allows for the measurement in a specific and objetive way of the results achieved by each project and to propose corrective actions for the future. The comparison among ICS provides an opportunity to identify the conditions under which an ICs may achieve greater efficiency.

Introduction

International cooperation has grown significantly in recent years, in fact, in 2009 the Organisation for Economic Co-operation and Development’s Development Assistance Committee (DAC) provided 85 874 million euros in official development assistance (ODA), of which 6% corresponded to Spanish ODA (MAEC 2009). The increase in the volume of aid and other resources towards achieving the United Nations’ eight millennium development goals by 2015 has been accompanied by a growing demand for effectiveness in development assistance (OCDE 2005:1). Countries that participated in the Second High Level Forum on Aid Development Effectiveness in Paris in 2005 recognised and committed to increasing aid effectiveness to achieve the millennium development goals.

Bilateral ODA from DAC countries, especially in the case of Spain, represents more than 70% of total aid since 2000. African countries are specially considered in the destination of this specific aid. In fact, two areas are priorities for Spanish international cooperation: north of Africa (Maghreb, Middle and Near East: Algeria, Morocco, Mauritania, Tunisia) and sub-Saharan countries (sub-Saharan Africa: Mozambique, Angola, Namibia, Senegal and Cape Verde). In the period 2002–2006, countries belonging to North Africa received approximately 15% of bilateral ODA per year on average and those from sub-Saharan Africa received approximately 10% of bilateral ODA per year on average. Although different international cooperation schemata (ICS) are used to run these bilateral aid projects, all ICS are implemented either by non-governmental development organisations (NGDOs) or by the Spanish Agency for International Cooperation for Development (AECID). In each case, the primary focus is on effectiveness as the essential principle.

However, in a world of scarce resources–especially in the international cooperation sector–effectively achieving goals is not sufficient. What really matters is efficiency: that is, controlling the level of resources used to achieve a certain goal (Banerjee 2008; Duflo & Kremer 2003). Increasing efficiency implies that a project is able to carry out more activities using the same amount of resources, which allows the project to extend the effect of the assigned resources for development as well as to reduce the pressure on the founder organisations and entities. In addition, the trend of national governments towards outsourcing international development aid makes efficiency imperative for non-profit organisations, especially NGDOs.

Accounting measures of efficiency in the non-profit sector, such as programme spending, price, technical efficiency and allocative efficiency, are based on ratios. These ratio measures only show the average percentage of the available resources devoted to the mission of the non-profit organisation or to the goals of the projects implemented by a non-profit organisation, but they do not provide information about the level of the achievement of the objectives or the level of resources applied, that is to say, they indicate how much of the donors’ contributions goes to the cause but do not take into account the impact of the non-profit’s spending.

To address this limitation, DEA methodology is proposed to analyse the relative efficiency in the non-profit sector. This technique estimates the relative efficiency objectively and numerically, taking into account several inputs and outputs without assuming weights and functional form of the production function, which makes it especially suited for non-profit organisations and, consequently, for the evaluation of international cooperation projects.

This methodology has already been used in earlier efficiency studies in the non-profit sector, but these studies have focused on the so-called Stage I of non-profit organisations, namely fundraising efforts (which are the most influential determinants of raising funds in order to increase the amount of contributions). However, for the so-called Stage II of non-profit organisations, namely spending the achieved resources to programme services delivery, DEA studies are very scarce. In attempting to address this research gap, this article assesses how efficiently the aid funds are actually spent by different Spanish aid instruments for the intended purposes and to compare different instruments in this regard. Specifically, this study, applying DEA, evaluates and compares the efficiency of a set of Spanish international cooperation projects funded with different schemata by the AECID between 2002 and 2006 in two African countries that are top priority targets of Spanish international aid: Morocco and Mozambique. To the best of our knowledge, this investigation is the first study that applies DEA to the assessment of ICS. Consequently, we offer a significant contribution to the literature by overcoming the limitations of other techniques used to assess the efficiency and providing new insight into the efficiency of targeted ICS in international cooperation development projects.

The remainder of this article is organised as follows. ‘Implementation of Spanish ODA’ section explains the various ICS applied to implement the ODA. ‘The efficiency issue’ section presents the different measures that have been used to assess aid efficiency and the measurements resulting from applying DEA. The section also discusses the appropriateness of applying DEA methodology to this study. The ‘Method’ section describes the sample, information sources and variables used to measure efficiency. The ‘Results’ section provides the main results, and the ‘Discussion and conclusions’ section offers conclusions, discusses the main limitations of the study and provides some lines for further research.

Implementation of Spanish ODA

International cooperation projects are funded using different ICS from ODA. The Organisation for Economic Co-operation and Development (OECD 2008) defines ODA as resources transferred to recipient countries on the DAC¹ list as well as to multilateral institutions for development. This assistance is provided by international agencies to promote the development and welfare of recipient countries. In addition, to be characterised as ODA, transfers must be concessional² in character. Bilateral or multilateral ODA to recipient countries can be provided by different sorts of agencies (i.e. state or local governments or by their executive agencies). We focus on non-reimbursable and bilateral ODA schemata funded by the AECID. Permanent open-call subsidies (POC) and NGDO subsidies (NS) are the kinds of schemata most employed by the AECID.

POC subsidies fund international cooperation projects of a particular initiative of any non-profit organisation. Call resolutions of the current Spanish Cooperation Master Plan and the International Cooperation Annual Plan state the priorities for granting these subsidies. In the selection of applicants, the AECID considers the project’s application to the sectorial and geographical priorities of Spanish cooperation; its complementarities with other projects related to cooperation for development; the content, relevance and quality of the project; and the appropriateness and viability of the project.

According to the Spanish Cooperation Master Plan 2005–2008, NGDO funding is held in accordance with the objectives and priorities of the Spanish international cooperation policy. Consequently, subsidies to NGDOs are in line with the principles of cooperation, complementarities, quality and effectiveness of aid. AECID can fund NGDO projects by means of two schemata of cooperation: actions and programmes. Firstly, cooperation for development actions has a single development goal, is carried out in one country and targets a predefined population; in addition, their effects should extend beyond the execution period. Secondly, programmes have a multiyear basis and pursue an overall development goal shared by both the NGDO and the AECID. They can involve actions in more than one country and in more than one sector. They have a maximum duration of 4 years, which can be extended for two additional years. In addition, only NGDOs identified as ‘qualified’ by the AECID (based on criteria such as years of experience, financial strength, transparency, accountability and human resources, among others) can access these cooperation instruments.

Spanish cooperation establishes geographical priorities for aid. These priorities are based on several factors: development indicators, the presence and articulation of Spanish cooperation in the country, the possible partnership framework in the country, the potential of the country as a development partner and the relative position of Spanish cooperation regarding other donors³. Given these priorities, beneficiaries of assistance can be broken down into three subgroups: (1) countries with a wide association, (2) countries with a focused association and (3) countries associated for consolidating development achievements.

Countries with a wide association include the least developed countries, with lower-middle income. In these countries, large volumes of ODA can be channelled through a long-term framework and the application of different kinds of ICS. Countries with a focused association also include least developed countries with lower-middle income but in which the cooperation programme does not allow a long-term or established framework association. The development impact in countries in this group increases when the aid is focused on a particular aspect identified along with the partner country. Therefore, a more selective use of instruments is recommended. Finally, some countries associate to consolidate development achievements, that is, the aim of aid in these countries is to strengthen inclusive public policies, promote South–South cooperation, encourage triangular cooperation and provide global public goods.

The efficiency issue

To assess the efficiency of ICS, we must first define efficiency. The concept of efficiency widely refers to the relation between the level of objectives achieved and the amount of resources applied (Martín, Martín & Hernangómez 2007). The first problem relative to this definition is as follows: given the relation between objectives and resources, how can we determine whether it is efficient? Or, in other words, how do we establish an appropriate standard of efficiency? A second problem lies in the objective’s multiple dimensions (outputs) and the multiple resources used (inputs), which requires that we assess all dimensions simultaneously and that we assign weights to all organisational factors (Hernangómez et al. 2009).

To evaluate the efficiency of ICS, we must determine the goal. In this sense, the understanding of the framework within which they operate is required. That is, ICS are the means by which organisations act to improve the quality of life in developing countries. The schemata are tailored to address a specific need (or set of needs) in the receptor country. Consequently, if the objective of the ICS is related to the question of what they are trying to achieve, the answer is to improve the quality of life and welfare for people in developing countries, an objective that is shared by the NGDO subsector globally. Therefore, the objective of the ICS is to improve the welfare of as many people as possible by meeting the specific need for which they were created.

Extending this thinking to the institutions that manage ICS, the notion of efficiency suggests that if NGDOs were marketable, an investor would find more value in organisations with the ability to generate greater prosperity or bring benefit to a greater number of people using the least possible resources. That is, the buyer would choose the organisation that manages more efficiently the chosen instrument of cooperation. Therefore, to evaluate the efficiency of ICS, we extrapolate the arguments used to assess the efficiency with which NGDOs operate.

Assessing efficiency

Assessing the efficiency of non-profits organisations such as NGDOs and their instruments – in this case, cooperation for development projects – involves certain difficulties because the objective is often more difficult to identify and measure than the objective of a firm⁴. In addition, because projects have multiple dimensions (project quality, relevance to the needs they serve, sustainability, impact, etc.), the production function can be difficult to determine.

Proposed methods for evaluating the efficiency of NGDOs are commonly based on accounting measures, mainly ratio analysis, and include price, programme ratio, technical efficiency and allocative efficiency. Price is defined as the cost, for a donor, of purchasing a dollar of output for an organisation’s beneficiaries. Without accounting for tax deduction, price is measured as the inverse of the percentage of expenditure on projects (Callen 1994; Posnett & Sandler 1989; Tinkelman 1998; Weisbrod & Dominguez 1986). Some studies use programme ratio represented as the percentage of project expenditures to total expenditures to measure efficiency (Baber, Roberts & Visvanathan 2001; Roberts, Smith & Taranto 2004). Baber et al. (2001) argue that this measure may indicate the fundraising strategy of the non-profit organisation. Allocative efficiency, a very similar and also widely used measure (Andrés-Alonso, Azofra-Palenzuela & Romero-Merino 2010; Andrés-Alonso, Martín-Cruz & Romero-Merino 2006; Callen & Falk 1993; Hernangómez et al. 2009), calculates the percentage of project expenditures to total revenue of the entity. Technical efficiency or administrative expense ratio (Andrés et al. 2006, 2010; Callen & Falk 1993; Greenlee & Brown 1999; Hernangómez et al. 2009) is defined as the percentage of administrative expenses on total expenditures.

However, these measures do not correspond to the concept of efficiency as previously defined – that is, the relation between the objectives achieved and the resources used – because they address neither the level of achievement of the objectives nor the level of resources applied. They only provide information on the extent to which a non-profit organisation devotes the available resources to its mission, which corresponds to the definition of efficiency established by Parsons (2003); that is, these definitions commonly show the average percentage of each donation that reaches the beneficiaries of the organisation. Despite their limitations, these indicators – especially technical and allocative efficiency – are widely used by both donors (Hyndman 1991; Khumawala & Gordon 1997) in their donation decisions as well as by the organisations themselves to show an image of transparency and fulfilment of their mission in hopes of generating greater fundraising. The use of these indicators is based mainly on its simplicity, easiness of obtaining information and the possibility to make uniform comparisons among entities.

In comparison with accounting methods, frontier analysis is becoming more and more important. Frontier analyses use the construction of efficient frontiers as a means to measure the relative efficiency of a group of organisations. The notion of the existence of a maximum achievable level underlies this methodology. Efficient frontiers can be calculated using either parametric or non-parametric techniques. Parametric techniques require the specification of a functional form of the production function to estimate (i.e. regress) the optimum relation between outputs and inputs. These techniques are very sensitive to the specification of the production function. When the functional form is not correctly determined, results are significantly biased. Thus, the efficient frontier method is only recommended if a specification of the production function close to the underlying production technology can be obtained.

Non-parametric methods use DEA to determine the efficient frontier based on a global efficiency indicator that relates objectives (outputs) and resources used to obtain these objectives (inputs). Therefore, DEA allows us to measure the relative efficiency of homogeneous decision-making units (DMU) while including a large number of variables and relations (constraints). This method is particularly suitable for the non-profit sector for two reasons. Firstly, DEA is especially adaptable to the features of non-profit organisations, in which the pricing of products and services is difficult, production function is not easily defined and multiple inputs and outputs are at play. Secondly, DEA establishes an objective indicator of efficiency for groups of organisations that do not have aggregate indicators and considers the multiple simultaneous objectives that non-profits try to achieve (Hernangómez et al. 2009).

Data envelopment analysis

Developed by Charnes, Cooper and Rhodes (1978), data envelopment analysis (DEA) consists of a linear programming technique that evaluates the relative efficiency of different DMU. The method identifies the efficient⁵ units and builds a frontier shaped by the linear combinations between these units. The efficiency of other units is measured by their distance from the estimated efficient frontier.

The analysis is formulated as a mathematical optimisation problem in which the estimated variables are the efficiency indices – defined as the weighted sums of outputs and weighted sums of inputs ratio – and the corresponding weights. It is a non-parametric technique because it does not specify a functional form between the maximum achievable outputs and required inputs. Thus, the application of this methodology makes it possible to obtain an efficiency indicator for each DMU that varies between zero and one (with one as the indicator of those DMU located in the frontier). In addition, the weights are endogenously determined, with their values corresponding to the maximum achievable efficiency. Therefore, this methodology allows us to compare the efficiency level of different DMUs, assists in identifying best practice and objectively determines the productive improvement and overcomes the problems associated with accounting and parametric frontier methods.

Several DEA models exist according to different assumptions about production technology and the related restrictions incorporated. Figure 1 provides the mathematical formulation of the basic model, assuming constant returns to scale for output and input orientation. Where ϕ₀ and φ₀ represent the efficiency indicators for each orientation, y_rj and x_ij represent the amount of output r and input i of DMU j, ur₀ and vi₀ represent the weights of output r and input i. Finally, ε is a small enough-positive value which the weights cannot fall (Basso & Funari 2004).

As previously mentioned, multiple outputs and inputs can be included in the analysis. However, as the number of included variables increase, the discriminating power of the DEA⁶ decreases. In addition, no tests can determine the statistical significance of the included variables or evaluate the goodness of fit. Because of these considerations, the general recommendation is that the number of inputs and outputs should not exceed one-third of the sample size (Banker, Charnes & Cooper 1984; Banker et al. 1989; McMillan & Datta 1998). Moreover, as DEA estimates relative efficiency, the DMU included in the analysis must be homogeneous because of the sensitiveness of this technique to extreme values. That means that they must apply the same production technology and operate under the similar institutional framework.

A final consideration is related to the choice of analysis orientation (output vs. input). This option depends on the nature of the problem and is related to the ability of organisations to control variables. If the analysed units are restricted in the management of inputs, output orientation is chosen, and vice versa. In the output-oriented approach, a unit is defined as efficient when no other unit can produce a higher level of output using a lower level of input. Likewise, for an input-oriented approach, a unit is defined as efficient when no other unit can reduce the input amount (keeping the proportion) and get the same amount of output.

In sum, we apply DEA to assess the efficiency of ICS. This analysis allows us to measure efficiency – defined by the relation between the objectives achieved and the resources used – and to include multiple inputs and outputs without defining the functional form of the production function. Furthermore, DEA makes possible valid comparisons among projects, and the available data meet the methodology requirements.

Applying DEA to evaluate efficiency of the non-profit sector

We are motivated to apply DEA to the non-profit sector to evaluate the efficiency of non-profits to produce the goods and services provided to beneficiaries with the donated resources simultaneously considering their multiple objectives. This methodology is particularly applicable because it allows us to identify the input reduction necessary to achieve efficiency (McMillan & Datta 1998) as well as the levels of inefficiency of different units (Charnes et al. 1994; Farrel 1957; Farrell & Fieldhouse 1962) and their causes.

Research on the study of efficiency in the non-profit sector, especially in the areas of health (Al-Shammari 1999; García et al. 1999; Hofmarcher, Paterson & Riedel 2002; Magnussen 1996; Valdmanis 1992; Watcharasriroj & Tang 2004) and education (Avkiran 2001; Banker, Janakiraman & Natarajan 2004; Johnes & Johnes 1995; Korkonen, Tainio & Wallenius 2001; McMillan & Datta 1998; Ruggiero 1999), is extensive. Yet, in the field of international cooperation for development, the literature is still scant. However, several prior studies have applied DEA to measure the relative efficiency of the NGDOs (Hernangómez et al. 2006, 2009; Martín, Martín & Hernangómez 2005, 2007), the efficiency of the two stages of the production process of NGDOs (García & Marcuello 2007; Golden et al. 2012; Marcuello 1999) and the efficiency of international cooperation for development projects (Martín, Martín & Gámez 2012).

Although a functional form of the production function is not necessary, DEA does require that the relevant outputs and inputs of the evaluated activity be established. The most common approach to making this determination is the development of a list of outputs and inputs, identified either from observation of evaluated units or from a literature review. From this list, the most suitable outputs and inputs are selected, considering the preservation of DEA discriminating power (García & Marcuello 2007; Hernangómez et al. 2006, 2009; Marcuello 1999; Martín et al. 2005, 2007).

Output determination requires the definition of the objectives to be achieved. Regarding the mission of the NGDOs, two main objectives have been considered, such as the eradication of poverty in the non–developed countries and education for development in Northern societies. In line with these objectives, prior empirical studies that have applied DEA to the field of international cooperation have used project expenditures (Golden et al. 2012; Marcuello 1999), total funds raised (García & Marcuello 2007; Golden et al. 2012) and the number of projects (Hernangómez et al. 2006, 2009; Martín et al. 2005, 2007) as outputs.

Previous research has measured inputs as operating costs, donations and grants (Golden et al. 2012; Marcuello 1999), number of volunteers (García & Marcuello 2007; Marcuello 1999), administrative expenses (García & Marcuello 2007), income, number of employees (García & Marcuello 2007; Hernangómez et al. 2006, 2009; Martín et al. 2005, 2007) and the age of the organisation (Hernangómez et al. 2006, 2009; Martín et al. 2005, 2007). Some measures have been used both as inputs and outputs, given the two stages identified in NGDOs functioning: activities aimed at fundraising and the application of the obtained resources to perform projects.

Method

Sample and information sources

The objective of this study is to analyse the efficiency of the ICS for Spanish ODA instruments applied in two African countries with a wide association (i.e. a long-term framework and the application of different kinds of ICS are possible), Morocco and Mozambique. These countries are considered with top priority for Spanish cooperation aid. The study period extends from 2002 to 2006. This term of 5 years includes the latest available data at the collection moment (June–December 2009) and is long enough to draw conclusions from the analysis. Because of the lack of a database, information was obtained at the headquarters of the AECID using information reports from NGDOs, AECID technicians and the audits of performed projects as well as in-depth interviews with technicians of the analysed countries.

Table 1 shows figures of bilateral ODA received by the two sample countries between 2002 and 2006. Aid to these countries grew over the whole period, especially towards the end when new policies of aid for development that aimed at reaching a rate of 0.7% of the gross domestic product were enacted. The lowest percentage of aid to these countries is 3.3% in 2003, and the highest percentage of aid is 6.1% in 2004. Our sample consists of 48 ICS, of which 18 are POC subsidies and 30 are NGDO subsidies. Furthermore, of a total of 48 ICS, 30 correspond to Morocco and 18 to Mozambique.

The 48 ICS analysed are projects related to the millennium development goals focused on community development (health, democratic governance, gender, education, rural development, economic growth and employment), infrastructure and agriculture. Instruments focused on community development represent 90% of the total ICS, whereas those focused on infrastructure and agriculture represent 6% and 4%, respectively. So our sample is composed of quite homogeneous instruments because the inputs employed to perform most of them are very similar (intensive in human factor and low capital-intensive). This way, we feel confident about the homogeneity of the sample and think that DEA application should not be a serious concern.

No database collects information about the ICS management. Therefore, we were required to build a new database with information provided by the AECID. In so doing, we needed to solve some problems. Firstly, no protocol exists for monitoring these instruments. As a result, quantitative information is not always available for all projects, and it is not possible to obtain all the items identified for all sample projects. In addition, the period of execution is commonly extended and delays occur. Consequently, the sample includes projects currently underway and projects for which information gaps exist.

Secondly, AECID has an established set of accounting items for budgeting ICS (see Appendix 1). In the database used to perform DEA, we set a correspondence between those items and the items associated with the National Public Accountancy Plan (NPAP). Thus, identified inputs correspond to the NPAP items.

Variables

In the section titled ‘Applying DEA to evaluate efficiency of the non-profit sector’, we presented the main variables used to measure the efficiency of NGDOs as well as their chief objectives and the main resources used in achieving them. However, we are not trying to assess the efficiency of the actual NGDO missions but rather the efficiency of the specific instruments used to achieve these missions. Therefore, we must conceptualise the objective of these instruments to define the outputs as well as the main resources to determine the inputs to be included in the analysis.

Martín et al. (2012) argue that the objective of international cooperation projects performed by NGDOs is the maximisation of social welfare of the population or area where the project is being carried out. In this line, we define the objective of the ICS as meeting the specific need for which they were created for many people as possible. Then, we establish a list of dimensions that make this objective operative and for which data are available. Our list consists of the number of activities, beneficiaries, areas, families, and organisations and duration in months⁷.

When determining the dimensions to measure the output, we considered the availability of information and level of importance that AECID technicians gave to these dimensions. Our goal is to select dimensions with information available for as many instruments as possible so that we can make significant DEA comparisons. In addition, we asked 15 AECID technicians, who are responsible for ex-ante evaluation, monitoring, supervision and ex-post evaluation of the different instruments, to rate the importance of the chosen dimensions on a scale from least important to most important. Table 2 shows the results. Unfortunately, the two dimensions most valued by the AECID technicians have the lowest level of information; therefore, data availability is the primary criterion for the selection of outputs, which dictate that we focus on duration, activities and areas served.

We select the following as the main inputs to measure the efficiency of ICS: purchases of raw materials, staffing costs (salaries), communications and other services, services from independent professionals, purchases of other supplies, repairs and maintenance, leasing charges and fees, and subsistence allowance. In selecting the inputs for the efficiency analysis, we assume that those inputs used by most instruments are the ones that endow a higher discriminating power to DEA because they allow for a more homogeneous comparison. Table 3 shows that most often used inputs are purchases of raw materials, staffing costs, and communications and other services – all of which are used by over 90% of the instruments included in the sample and together represent 91% of total executed expenditures.

In sum, the production function (the base scenario) with which we assess the efficiency of ICS includes purchases, wages and salaries, and communications and other services as inputs and activities performed, and areas served as outputs. We opt for an output orientation to perform DEA because we assume that, given a certain resource level (i.e. available budget), each instrument will attempt to maximise results (i.e. to reach the largest number of individuals or areas, perform the most activities, etc.).

Results

In this section, we present the results of the DEA analysis performed by means of the previously discussed outputs and inputs. The sample consists of 48 instruments, of which 18 are POC subsidies and 30 are NGDO subsidies. Firstly, we assess the efficiency of the two types of ICS. Then, we analyse the efficiency with which these instruments are implemented in each of the two priority African countries for Spanish ODA included in our sample (general model). These results are subjected to a sensitivity analysis, in the first place by removing the extreme cases (the most efficient and the least efficient instruments) and secondly by performing the DEA analyses with different inputs specification, to assess the effects from inclusion or exclusion of certain cost categories. Finally, we complement our analysis with two additional models (alternative models 1 and 2) to measure the consistency of results and to check the robustness of the analysis. Table 4 shows the inputs and outputs for the three models. Appendix 2 reports the results for the two alternative models.

Table 5, Panel A, provides a summary of results of the general model. The results show that 12 instruments are efficient, which represents 25% of the total observations included in the sample. In terms of efficiency by type of instrument, five POC subsidies (27.7%) and seven NGDO subsidies (23.3%) are efficient. The average efficiency for each type of instrument – 0.70 for POC subsidies and 0.65 for NGDO subsidies – is in line with values obtained by prior empirical studies that apply DEA to the cooperation for development sector (Hernangómez et al. 2006, 2009; Martín et al. 2005, 2007).

Although these results allow us to compare the efficiency of different instruments, we continue our analysis by applying our DEA model to a narrower scope. Specifically, we consider independently each of the two countries included in the sample in Panel B of Table 5. This examination allows us to make comparisons within a single environment and thus avoid complications and differences caused by differing contexts. In addition, we can identify any pattern of significant differences across countries for particular instruments. Panel B shows the percentage of efficient instruments and the average efficiency in the performance of analysed instruments for each country. Morocco achieves results both in the percentage of efficient instruments as well as the average efficiency in the performance of analysed instruments that are clearly superior to Mozambique. That is, the results suggest that the different instruments of cooperation for development are implemented with greater efficiency in Morocco.

In Panel C, we further narrow our focus as we now analyse the ICS for Morocco alone, because, with a sample of 30 ICS, it is the only country that fulfils the requirements imposed by DEA and thus allows us to perform the analysis. As shown in Panel C of Table 5, three POC subsidies (27.4%) and five NGDO subsidies (26.3%) are efficient – in both cases, higher percentages than those obtained for the total sample, especially for POC. The average values are also higher than those achieved for the whole sample and show again that POC subsidies is the instrument that achieves the highest levels of efficiency (0.77).

When we estimate the frontier by removing the most efficient (12) and the least efficient (12) instruments, results (see Table 6) are quite similar regarding those obtained with the whole sample; again POC subsidies are more efficient than NGDO subsidies. Average efficiency and percentage of efficient instruments are a bit higher than initial results, but it must be considered that a small sample size reduces the discriminating power of DEA so that the number of efficient units and the average efficiency tend to increase. These results show no significant influence of extreme values on the efficient frontier.

We have also estimated the general model (outputs: activities performed and areas served) with different combinations of inputs, relative to the original model. Purchases, staffing costs, and communications and other services are the cost categories considered because they represent a substantial amount of the instruments’ budget (at least 10% of the total expenditure) and are included in almost all instruments (over 90%). The results of the analyses performed with the abovementioned combinations of inputs (see Tables 7–9) are in line with the results of the initial analysis.

Table 7 shows the results of the analysis performed with salaries and purchases as inputs, the most important factors regarding their percentage over total expenses (45% and 34%, respectively). Average efficiency is almost identical relative to the base scenario (0.67 vs. 0.66), and the percentage of efficient instruments is quite similar (27.1% vs. 25%). POC subsidies efficiency is higher than NGDO subsidies efficiency both in the whole sample and in the subsample including only Morocco’s instruments. In addition, the average efficiency of the instruments performed in Morocco is higher than the average efficiency of instruments executed in Mozambique. In general, these results give support to those obtained in the base scenario of the general model.

Table 8 shows the results of the analysis performed with purchases and other services as inputs. We observe a similar tendency relative to the base scenario: POC subsidies are more efficient than NGDO subsidies both in the whole sample and in the subsample including only Morocco’s instruments, and the average efficiency of the instruments performed in Morocco is higher than the average efficiency of the instruments developed in Mozambique. However, in this analysis, the average efficiency of the whole sample and the percentage of efficient instruments are lower than in the base scenario. These findings pose that salaries as input have a considerable influence on DEA results, which is understandable given its importance over total expenses (34%).

Table 9 shows the results of the analysis performed with salaries and other services as inputs. Again, we observe a similar tendency relative to the base scenario: POC subsidies are more efficient than NGDO subsidies both in the whole sample and in the subsample including only Morocco’s instruments, and the average efficiency of the instruments developed in Morocco is higher than the average efficiency of instruments performed in Mozambique. Average efficiency of the whole sample and percentage of the efficient instruments are a bit lower than in the base scenario. Again, these findings show that purchases as input have a considerable influence on DEA results, which is understandable given its importance over total expenses (45%).

We have performed two additional analyses to check the consistency and robustness of our results, identified as alternative model 1 and alternative model 2 (see Appendix 2). Alternative model 1 includes activities performed, areas served and institutions involved as outputs and purchases, wages and salaries, and communications and other services as inputs, whereas alternative model 2 includes activities performed, areas served and number of beneficiaries as outputs and the three aforementioned inputs. The results of these analyses show that among the instruments examined, again POC subsidies are more efficient than the NGDO subsidies and Morocco outperforms Mozambique regarding the level of efficiency achieved in the execution of the ICS. Average efficiency is higher in DEA alternative models than in the base scenario of DEA general model (0.74 and 0.76 vs. 0.66). Likewise, the percentage of efficient instruments is higher in DEA alternative models than in the base scenario of DEA general model (29.8 and 36.4 vs. 25.0), which is understandable taking into account DEA properties: The discriminating power of the DEA decreases both when the number of included variables increases and when sample size decreases, resulting in better average efficiency and a higher percentage of efficient units. Therefore, these findings give support to the results obtained in the base case scenario of the general model.

Discussion and conclusion

As we pose throughout this study, measures for evaluating international aid projects with respect to efficiency are problematic because there is no evident bottom line, and the measurement of performance, unlike firms, cannot be done in terms of profit or profitability alone. Accounting measures, mainly ratio analysis, do not capture efficiency of output and are subject to measurement error through valid or invalid accounting manipulations (Golden et al. 2012). In a multiple input and output environment as is the case of international aid projects, DEA is an appropriate method because it provides a measurement that can simultaneously identify inefficient projects and also produces information regarding sources of inefficiencies, what inputs are being over-utilised and by how much to make the project efficiency and also identifies a set of efficient project against which to benchmark. Thus, DEA analysis allows for the comparison of international aid projects efficiency across different ICS responding the donors’ concern of what types of instruments best allocate the donated resources to the dedicated cause.

Our results show that among the instruments examined within the sample countries, POC subsidies are more efficient than the NGDO subsidies. These findings are substantiated by the results from both alternative models, reported in Appendix 2. Furthermore, when we conduct the analysis exclusively for Morocco – the country in which the greatest number of ICS is put into effect among our sample countries – the result holds. Therefore, we may infer that this result is considerably independent from the particular environment in which the ICS are performed.

With respect to the efficiency achieved in each country, assessed through the general model and all its different specifications, Morocco achieves a higher level of average efficiency in the execution of the ICS. This result is confirmed by both alternative model 1 and alternative model 2, which qualifies Morocco as the most efficient country. Considering the results of all models, Morocco appears to be the most efficient country, but it is required to take into account that efficiency can vary depending on the model specification (inputs and outputs employed to assess the efficiency). Findings should be interpreted cautiously given that the relative position of countries in terms of efficiency could be variable and it is not usual that a country would be absolutely more efficient than the rest.

Several implications for the AECID (which is the founder of the analysed instruments) can be derived from these findings. The use of this ex-post measure of efficiency allows for the measurement in a specific and objective way of the results achieved by each project and to propose corrective actions for the future. Likewise, the comparison among ICS provides an opportunity to identify the conditions under which an ICS may achieve greater efficiency. However, DEA is not without drawbacks, for example, its sensitiveness to model specification, so a critical decision is the determination of inputs and outputs because the results could be substantially different if the model is estimated with a different combination of inputs and outputs.

Considering the pros and cons of DEA, the assessment of international aid projects needs to be complemented with other types of analyses to obtain consistent and reliable results. A solid in-depth qualitative assessment of projects analysing dimensions such as their relevance, effectiveness, sustainability and impact (variables recommended by the DAC to assess project performance) is crucial to the success of the projects. Although DEA can contribute towards shedding some light on assessing the efficiency of international aid projects, more research is needed to address this important issue.

This study has several limitations. Firstly, the main limitation is the sample size. A greater number of observations and even variety of ICS would be desirable; however, it is extremely difficult to obtain systematic information of inputs and outputs of the ICS because of the lack of a common procedure and computerised collection of information by agencies (in this case, the AECID). Secondly, DEA requires that units included in the analysis must be homogeneous because of the sensitiveness of this technique to extreme values. Although we consider our sample homogenous enough, we are aware that it is always advisable to gather a more homogeneous sample. Thirdly, instruments’ data do not allow us to perform DEA analysis with the most relevant dimensions of the output, as they were ranked by the AECID experts. Fourthly, DEA results are sensitive to model specification, so it is very important to include relevant inputs and outputs. Future research may measure the impact of the ICS to improve the quality of inputs and outputs used to characterise the production function. Finally, the analysis period corresponds to a time horizon of 5 years. It would be interesting to extend this period and to perform an analysis by year to assess whether the composition of the selection committee may influence the choice of more or less efficient projects.

Acknowledgements

The authors would like to thank the anonymous reviewers for their constructive comments, as they have proved very helpful towards improving the manuscript by providing further work or additional clarification when required. Financial support has been received from the AECID (08-CAP2-0770).

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

References

Ahsan, K. & Gunawan, I., 2010, ‘Analysis of cost and schedule performance of international development projects’, International Journal of Project Management 28(1), 68–78.

Al-Shammari, M., 1999, ‘A multi-criteria data envelopment analysis model for measuring the productive efficiency of hospitals’, International Journal of Operations and Production Management 19(9), 879–890.

Andrés-Alonso, P., Martín-Cruz, N. & Romero-Merino, E., 2006, ‘The governance of nonprofit organizations: Empirical evidence from nongovernmental development organizations in Spain’, Nonprofit and Voluntary Sector Quarterly 35(4), 588–604.

Andrés-Alonso, P., Azofra-Palenzuela, V. & Romero-Merino, E., 2010, ‘Beyond the disciplinary role of governance: How boards add value to Spanish foundations’, British Journal of Management 21(1), 100–114.

Avkiran, N., 2001, ‘Investigating technical and scale efficiencies of Australian universities through data envelopment analysis’, Socio-Economic Planning Sciences 35(1), 57–80.

Baber, W.R., Roberts, A.A. & Visvanathan, G., 2001, ‘Charitable organizations’ strategies and program-spending ratios’, Accounting Horizons 15(4), 329–343.

Banerjee, A.V., 2008, ‘Big answers for big questions: The presumption of growth policy’, Brookings Conference: What Works in Development? Thinking Big and Thinking Small, Washington, May.

Banker, R.D., Charnes, A. & Cooper, W.W., 1984, ‘Some models for estimating technical and scale efficiencies in data envelopment analysis’, Management Science 30(9), 1078–1092.

Banker, R.D., Charnes, A., Cooper, W.W., Swarts, J. & Thomas, D.A., 1989, ‘An introduction to data envelopment analysis with some of their models and its uses’, Research in Governmental and Nonprofit Accounting 5, 125–163.

Banker, R.D., Janakiraman, S. & Natarajan, R., 2004, ‘Analysis of trends in technical and allocative efficiency: An application to Texas public school districts’, European Journal of Operational Research 145(2), 477–491.

Basso, A. & Funari, S., 2004, ‘A quantitative approach to evaluate the relative efficiency of museums’, Journal of Cultural Economics 28(3), 195–216.

Callen, J.L., 1994, ‘Money donations, volunteering and organizational efficiency’, Journal of Productivity Analysis 5(3), 215–228.

Callen, J. & Falk, H., 1993, ‘Agency and efficiency in nonprofit organizations: The case of “specific health focus” charities’, The Accounting Review 68(1), 48–65.

Charnes, A., Cooper, W. & Rhodes, E., 1978, ‘Measuring the efficiency of decisions making units’, European Journal of Operational Research 2(6), 429–444.

Charnes, A., Cooper, W., Lewin, A. & Seiford, L. (eds.), 1994, Data envelopment analysis. Theory, methodology and applications, Kluwer Academic, Boston, MA.

Diallo, A. & Thuillier, D., 2004, ‘The success dimensions of international development projects: The perceptions of African project coordinators’, International Journal of Project Management 22(1), 19–31. https://doi.org/10.1016/S0263-7863(03)00008-5

Duflo, E. & Kremer, M., 2003, ‘Use of randomization in the evaluation of development effectiveness’, World Bank Operations Evaluation Department (OED) Conference on Evaluation and Development Effectiveness, Washington, July.

Farrell, M.J., 1957, ‘The measurement of productive efficiency’, Journal of the Royal Statistical Society 120(3), 253–290. https://doi.org/10.2307/2343100

Farrell, M.J. & Fieldhouse, M., 1962, ‘Estimating efficient production functions under increasing returns to scale’, Journal of the Royal Statistical Society 125(2), 252–267. https://doi.org/10.2307/2982329

García, L. & Marcuello, C., 2007, ‘Eficiencia y captación de fondos en las organizaciones no gubernamentales para el desarrollo (Efficiency and fund-raising in non-governmental development organizations)’, CIRIEC-España, Revista de Economía Pública, Social y Cooperativa 58, 221–249.

García, F., Marcuello, C., Serrano, D. & Urbina, O., 1999, ‘Evaluation of efficiency in primary health care centres: An application of data envelopment analysis’, Financial, Accountability and Management 15(1), 67–83. https://doi.org/10.1111/1468-0408.00074

Golden, L., Brockett, P., Betak, J., Smith, K. & Cooper, w., 2012, ‘Efficiency metrics for non-profit marketing/fundraising and service provision – a DEA analysis’, Journal of Management and Marketing Research 9(1), 1–25.

Greenlee, J. & Brown, K., 1999, ‘The impact of accounting information on contributions to charitable organizations’, Research in Accounting Regulation 13, 111–125.

Hernangómez, J., Martín, N. & Martín, V., 2006, ‘La relevancia del objetivo en la medida de la eficiencia. Un análisis para las ONGD españolas desde la Teoría del Comportamiento (Aim relevance in assessing efficiency. Analysing Spanish NGDO applying behavioral theory)’, Boletín Económico del ICE 2884, 17–30.

Hernangómez, J., Martín, V. & Martín, N., 2009, Implicaciones de la organización interna sobre la eficiencia. La alicación de la teoría de la agencia y la metodología DEA a las ONGD españolas (Implications of internal organization on efficiency. The use of agency theory and DEA methodology to Spanish NGDO). Cuadernos de Economía y Dirección de la Empresa, 40, 17–49. https://doi.org/10.1016/S1138-5758(09)70041-6

Hofmarcher, M., Paterson, I. & Riedel, M., 2002, ‘Measuring hospital efficiency in Austria: A DEA approach’, Health Care Management Science 5(1), 7–14. https://doi.org/10.1023/A:1013292801100

Hyndman, N., 1991, ‘Contributions to charities. A comparison of their information needs and the perceptions of such by the providers of information’, Financial Accountability and Management 7(2), 69–82. https://doi.org/10.1111/j.1468-0408.1991.tb00126.x

Johnes, J. & Johnes, G., 1995, ‘Research funding and performance in U.K. university departments of economics: A frontier analysis’, Economics of Education Review 14(3), 301–314. https://doi.org/10.1016/0272-7757(95)00008-8

Korhonen, P., Tainio, R. & Wallenius, J., 2001, ‘Value efficiency analysis of academic research’, European Journal of Operational Research 130(1), 121–132. https://doi.org/10.1016/S0377-2217(00)00050-3

Khumawala, S.B. & Gordon, T.P., 1997, ‘Bridging the credibility of GAAP: Individual donors and the new accounting standards for nonprofit organizations’, Accounting Horizons 11(3), 45–68.

MAEC, 2009, ‘Seguimiento del PACI’, Ministerio de Asuntos Exteriores y Cooperación (Spanish Ministry of Foreign Affairs and Cooperation), viewed 21 May 2011, from http://www.maec.es/es/MenuPpal/CooperacionInternacional/Publicacionesydocumentacion/Documents/Seguimiento%20PACI%2009.pdf

Magnussen, J., 1996, ‘Efficiency measurement and the operationalization of hospital production’, Health Services Research 31(1), 21–37.

Marcuello, C., 1999, ‘Análisis de la conducta y eficiencia de las organizaciones no gubernamentales para el desarrollo españolas (Efficiency and behavior analysis of Spanish nongovernmental development organizations)’, Información Comercial Española 778, 181–196.

Martín, V., Martín, N. & Hernangómez, J., 2005, ‘La valoración de los resultados de las entidades sin fines de lucro: Una aplicación en el análisis de la eficiencia de la arquitectura organizativa de las ONGD Españolas (Performance appraisal of nonprofit organizations: Analysing the efficiency of Spanish NGDO organizational architecture)’, Revista AECA 71, 34–40.

Martín, N., Martín, V. & Hernangómez, J., 2007, ‘El deleite de la eficiencia (The Delight of Efficiency)’, Universia Business Review 14, 56–67.

Martín, N., Martín, V. & Gámez, C., 2012, ‘La eficiencia y el “misreporting” contable en las ONGD Españolas. Análisis de los proyectos de cooperación internacional para el desarrollo (Efficiency and accounting “misreporting” in Spanish NGDOs: An analysis of international development cooperation projects)’, Academia, Revista Latinoamericana de Administración 51(4), 1–14.

Mcmillan, M. & Datta, D., 1998, ‘The relative efficiencies of Canadian universities: A DEA perspective’, Canadian Public Policy 24(4), 485–511. https://doi.org/10.2307/3552021

OCDE, 2005, ‘The Paris declaration on aid effectiveness and the Accra agenda for action’, Organisation for Economic Co-operación and Development, March, viewed 20 May 2011, from http://www.oecd.org/development/aideffectiveness/34580968.pdf

OECD, 2008, ‘Is it ODA? Factsheet’, Organisation for Economic Co-operation and Development, November, viewed 22 May 2011, from http://www.oecd.org/dataoecd/21/21/34086975.pdf

Parsons, L.M., 2003, ‘Is accounting information from nonprofit organizations useful to donors? A review of charitable giving and value-relevance’, Journal of Accounting Literature 22, 104–129.

Posnett, J. & Sandler, T., 1989, ‘Demand for charity donations in private non-profit markets: The case of the U.K.’, Journal of Public Economics 40(2), 187–200. https://doi.org/10.1016/0047-2727(89)90002-9

Roberts, A.A., Smith, P. & Taranto, K., 2004, ‘Marginal spending and efficiency in charities’, Proceedings of the Annual American Accounting Association Meeting, Orlando, FL.

Ruggiero, J., 1999, ‘Non-parametric analysis of educational costs’, European Journal of Operational Research 119(3), 605–612. https://doi.org/10.1016/S0377-2217(98)00351-8

Tinkelman, D., 1998, ‘Differences in sensitivity of financial statement users to joint cost allocations: The case of non-profit organizations’, Journal of Accounting, Auditing and Finance 13(4), 377–393. https://doi.org/10.1177/0148558X9801300401

Valdmanis, V., 1992, ‘Sensitivity analysis for DEA models. An empirical example using public vs. NFP hospitals’, Journal of Public Economics 48(2), 185–205. https://doi.org/10.1016/0047-2727(92)90026-C

Watcharasriroj, B. & Tang, J., 2004, ‘The effects of size and information technology on hospital efficiency’, Journal of Hich Technology Management Research 15(1), 1–16.

Weisbrod, B.A. & Dominguez, N.D., 1986, ‘Demand for collective goods in private nonprofit markets: Can fundraising expenditures help overcome free-rider behavior?’, Journal of Public Economics 30(1), 83–96. https://doi.org/10.1016/0047-2727(86)90078-2

Appendix 1

Accounting items established by Spanish Agency for International Cooperation for Development (AECID)

Appendix 2

Footnotes

1. All low- and middle-income countries with the exception of G8 members, EU members and countries with an admission date for entry into the EU.

2. At least 25% of aid must be a contribution, and interest rates on loans must be below market levels.

3. Plan Director de la Cooperación Española 2010. Same criteria are applied from 2010.

4. Instead of efficiency, recent studies have focused on the outcomes or success of international cooperation projects (Ahsan & Gunawan 2010; Diallo & Thuillier 2004).

5. Pareto efficiency criterion is used: A unit is efficient if no others obtain higher levels of one output without producing less of another output and without increasing the use of any inputs. Or, if no unit produces the same amount of outputs using less of some input without increasing the use of others (Charnes et al. 1978).

6. A greater number of variables relative to sample size increase the likelihood to be efficient in any of the variables for the analysed units.

7. The time spent in a manufacturing process is taken as an input and, in the comparison of processes, efficiency is rated by the amount of time spent at the same cost: A more efficient process requires less time but equal cost. However, we consider real time by which duration is considered an output. That is, because a longer contract period extends the project’s capacity to benefit the target population, more time at the same cost equals greater efficiency. Therefore, a real-time analysis assumes that between two projects with equal cost, a donor will prefer to finance the one that requires more time to perform.

Original Research

Efficiency of international cooperation schemata in African countries: A comparative analysis using a data envelopment analysis approach

Victor Martin-Perez, Natalia Martin-Cruz

Abstract

Introduction

Implementation of Spanish ODA

The efficiency issue

Assessing efficiency

Data envelopment analysis

Applying DEA to evaluate efficiency of the non-profit sector

Method

Sample and information sources

Variables

Results

Discussion and conclusion

Acknowledgements

Competing interests

Author’s contributions

References

Appendix 1

Accounting items established by Spanish Agency for International Cooperation for Development (AECID)

Appendix 2

Footnotes