Article published in French in Travail et Emploi, n° 135, juillet-septembre 2013.
- 1 Article 39 of the French Constitution. The Organic law (or enabling Act) No 2009-403 of 15 April 2 (...)
- 2 “Parliament votes the laws. It controls the actions of the Government. It evaluates public policie (...)
1In a democratic system citizens may demand accountability for the use of public finances via their representatives. This process involves enacting rules, evaluating their scope, and resorting to an argumentative apparatus in which statistical data occupy a central place. Figures are used ex ante to provide reasons to legislate, by identifying situations whose evolution calls for changes to applicable rules. Figures are also used as the basis of predictions about the effects of new rules on a particular situation. Ex post, statistical evaluations aim to measure the impact of rules, defined according to their use, and to assess their effectiveness, which is judged in terms of the ability of a rule to meet stated objectives. Ex ante assessments involve prior studies. These may concern the rationalisation of budget choices (Lévy-Lambert, Guillaume, 1971), and impact studies which have been made mandatory under the French Constitution since 2009, in the passing of parliamentary Bills.1 Ex post evaluations concern the vast field of public policy assessment (Viveret, 1990), which is officially vested in the French Parliament under Article 24 of the Constitution.2 Evaluation techniques have developed above all in the “preparation” of norms and standards, appearing as of the 1960s, with laws and rules that are very explicitly presented as “experimental”. Such evaluation was given a constitutional basis in Law No 2003-276 of 28 March 2003, which concerns the devolved organization of the French Republic, and which authorises normative experimentation both nationally and locally. This innovation opened the door to another type of experimentation, namely random experimentation, which draws on social science methodologies, and which is also applied to international policies on poverty (Allègre, 2008; Gomel, Serverin, 2009).
2It is now a little more than five years since random social experimentation has been used in France, within the perimeter of social policy reform. It is therefore legitimate to review the meaning and scope of this kind of experimentation. To this end, three questions may be raised:
1/ In terms of legal science, what is the role of this form of experimentation in the perimeter of normative experimentation?
2/ From a scientific point of view, what are the lessons of this experimentation for social experimentation applied to human behaviour?
3/ From a socio-political perspective, what has the contribution of experimentation been to the implementation of reforms?
Lastly, this article will examine the ethical and scientific demands which are inherent in all forms of experimentation relating to human behaviour.
3If we look at the chronology of experimentation in public policy in France, it can be seen that the Administration (government) preceded researchers in being concerned about anticipating the effects of reforms. Making such practices constitutional has opened the way to normative experimentation at the national and local level. Random social experimentation has emerged locally where social policies actually develop.
4Testing laws begins with the idea that political decisions strive for positive effects through reform, but are also concerned about negative effects. From this point of view, administrative experimentation already has a long history.
- 3 Law No 75–17 of 17 January 1975 introduced abortion (IVG), and Law No 88-1088 of 1 December 1988 i (...)
- 4 Until 1979 for the IVG Law (Art. 2), and until 30 June 1992 for the RMI Law (Art. 52).
5The term “experimentation” was not used in France’s laws on abortion (Interruption volontaire de grossesse – IVG) and Minimum Income Support (Revenu minimum d’insertion – RMI), but they were in fact the first examples to use testing.3 The IVG Law on abortion suspended criminal law for a period of five years, while the RMI Law provided minimum income support for three years.4 Once these terms ended, the respective governments introduced new Bills to perpetuate the laws.
6Testing laws also arose in waivers and derogations to the rules of ordinary law affecting people in the same situation, and which therefore bypassed the principle of equality before the law.
7Successive interventions by France’s Constitutional Court on such measures have recognised their experimental nature as a justification for breaching the principle of equality. In a ruling pronounced on 28 July 1993, the Court specified that:
“It is even open to Parliament to provide for the possibility of conducting experiments that involve derogations to the rules defined above, to allow it subsequently to adopt new rules, in the wake of results, which are appropriate to changes in the missions of the establishments in question. However, lawmakers have to define precisely the nature and scope of the experiments, the situations in which they may be conducted, the conditions and procedures according to which they are evaluated that may lead them to be maintained, modified, generalised or dropped.”
- 5 Conseil constitutionnel (Constitutionnal Council), decision 6 November 1996, No 96-383 DC.
8A second ruling on 6 November 1996 concerned the information and consultation of employees in certain companies. It reiterated these demands in situations in which social partners obtained derogations to ordinary law on collective bargaining agreements, due to the experimental nature of a measure, provided that the Government reported on experiments to Parliament within five years.5
9These legislative practices have been supplemented by executive measures, involving the enactment of standards that are limited in time and space.
- 6 Conseil d’État (Council of State), Section, 13 October 1967, No 64778, Rec. CE., p. 365; Revue de (...)
- 7 Conseil d’État, 21 February 1968, No 68615 et seq., Ordre des avocats près la cour d’appel de Pari (...)
- 8 Conseil d’État, Assemblée générale, section TP, 24 June 1993, avis No 353605.
10Thus, the principle of equality before the law was not deemed to have been breached in the case of the progressive application of health controls in food companies, due to a lack of veterinarians.6 Similarly, a decree in 1968 creating a new judge only in certain courts or courts of appeal was not considered as violating the principle of equality before the law in so far as such limitations were provisional.7 Lastly, an opinion given by France’s Council of State on 24 June 1993 accepted different tariff measures applied to users of railway lines by the SNCF (the French railway company). Even though the users had similar characteristics, tariff differences were accepted because they were limited to an experiment lasting one year, which was to lead to the definition of criteria used in a new tariff regime.8
11The validity of legal measures and regulations therefore depends on the existence of an assessment being undertaken by an authority which is enacting standards and the assessment must be followed up by a report.
12After 40 years of testing legislation of all types, it was no longer possible to continue with such methods without giving them a constitutional foundation. In its ruling of 17 January 2002, the Constitutional Council considered
- 9 Conseil constitutionnel, 17 January 2002, No 2001-454, statut de la Corse, note J.-E. Schoettl : (...)
“that by opening to Parliament, albeit experimentally and based on a derogation limited in time, the possibility of authorising local government in Corsica to take measures falling within the field of the law, the law in question actually intervenes in areas which belong to the jurisdiction of the Constitution”.9
- 10 “On the basis of the fourth paragraph of Article 72 of the Constitution, the law authorises local (...)
13Following this decision, Constitutional Law No 2003-246 of 28 March 2003, relating to the devolved organization of the Republic (known to some as “Act II of France’s devolution”: Janin, 2005), authorised normative experimentation both nationally and locally. On the one hand, it introduced Article 37-1 into the Constitution which states that: “laws and regulations may include measures of an experimental nature which are limited in time”. On the other hand, it added to Article 72 of the Constitution, which relates to local government, a fourth paragraph allowing local authorities the possibility to “derogate from legislative and regulatory measures which govern the exercise of their competencies, on an experimental basis for limited purposes and duration”. Applying this latter measure, the Organic Law No 2003-704 of 1 August 2003 of the General Code of Local Government added a chapter specifically dedicated to experiments, for which the Organic Law provides a general framework.10
14Observers have noted that including experimentation in the Constitution has led neither to much controversy nor to debate (Stahl, 2010). This relative silence suggests there is a kind of consensus on the fact that, following 40 years of practice, there is an interest in testing norms and standards before generalising them. Yet by setting out the virtues of “normative prudence”, constitutional texts have now opened the door to experimental methods which have never been discussed before.
15Since the revision of the Constitution came into force in 2003, article 37-1 has been invoked most, mainly in administrative (i.e. governmental) reasoning. Social experimentation for its part is based on Article 72.
16Following the tradition of texts relating to testing, experimentation under article 37-1 has been largely geared to administrative action, from a technical and budgetary point of view.
- 11 Conseil constitutionnel, 12 August 2004, No 2004-503 DC: “article 37-1 of the Constitution, which (...)
17The first use of constitutional authorisation occurred with the Law of 13 August 2004 concerning “local liberties and responsibilities” (Janin, 2005). This Law expanded the field of experimentation with respect to the transfer and delegation of responsibilities and competencies, be they for the management of financing or the exercise of competencies. When it examined an appeal against this text, the Constitutional Council drew on this new foundation to reaffirm the criteria for experimentation which had been forged in previous decisions.11
18In another decision of 13 December 2007, the Constitutional Council states that
- 12 Conseil constitutionnel, 13 December 2007, No 2007-558 DC – the Law financing public health insura (...)
“article 37-1 of the Constitution provides that the Social Security Financing Law includes experimental provisions […], but that the Council would have to censor provisions that […] are actually not about experimental application, but applications limited in time”.12
19This position expresses an essential idea, namely that experimental measures can only function if they are accompanied by an evaluation that looks at all the consequences of the reform.
- 13 Sections 37-1 and 38 of the Constitution were covered by Ordinance No 2006-433 of 13 April 2006, “ (...)
20Other reforms pursuing economic or social or environmental purposes are expensive and so need to go through an experimental “airlock”. There are many examples.13 The latest example concerns the development of citizen assessors in criminal cases, by Law No 2011-939 of 10 August 2011. This resulted in a (negative) report issued by two judges of France’s Supreme court (Cour de cassation) (Salvat, Boccon-Gibod, 2013).
21In all these areas, experimentation concerns norms and standards themselves, and its inclusion in all norms affecting social, economic, and budgetary policies. The task of experimentation falls on the Administration itself, without recourse to external research. Above all, no scientific hypotheses are tested. Thus, for example, it was not planned in the experiment on citizen assessors to test the “severity” of non-magistrates, even though this had been one of the reasons for the reform. Only the consequences of the measure were to be assessed. In short, experimentation amounts to a sort of “penitence clause” to avoid undertaking future measures that are expensive, controversial, or which present serious risks of being distorted.
22Random experimentation does not have its roots in article 37-1, but in Article 72 of the Constitution. For the first time, the objective of the experiment was not to check the quality of a norm, the feasibility of reform or even its cost. Instead, experimentation was to measure the effects on the behaviour of the persons subject to the experiment.
23Social experimentation with random assignment has a long international history. But it began in France with the report of the Families, Vulnerability and Poverty Committee (Commission Familles, vulnérabilité, pauvreté, 2005), chaired by Martin Hirsch. It suggested the government use “experimentation programmes that it [the government] would define itself but would like to test on part of the national territory” (Gomel, Serverin, 2009, 2011). Lawmakers took up this advice in two stages. First with Article 142 of the Finance Law for 2007 of 21 December 2006:
“On an experimental basis, to improve the conditions of financial incentives for persons returning to work and to simplify access to assisted employment contracts, the Departments [i.e. French counties] mentioned by the decree provided for in Article LO 1113-2 of the General Code on Local Authorities are authorised to adopt all of the exceptions to the provisions of the Labour Code and Code for Social Action and Families in favour of the beneficiaries of minimum income, for a period of three years from the date of publication of the decree […]”
24On 4 May 2007, two Departments (Eure and Côte d’Or), which had previously declared themselves as volunteers on 1 February and 23 March 2007, were authorised by decree to conduct experiments. These related firstly to improving the conditions of financial incentives to return to employment; and secondly to simplifying access to subsidised work contracts. On 20 June 2007, the General Council of Eure adopted a resolution setting out the derogation rules for experimentation in the Department. Without waiting for the end of this first experiment, the Parliament adopted a substantial measure affecting its design in the summer of 2007, by moving to a second stage of legislation. Articles 18-23 of the Law of 21 August 2007 called “TEPA” (the law in favour of work, employment and real disposable income) opened up the experimentation of an “Active Solidarity Income” (Revenu de solidarité active – RSA) to all volunteering Departments. Section 22 of the Law requires the government to file an evaluation report before any generalisation of the RSA. The rapporteur of the Bill generalising the RSA lauded the approach taken by this assessment:
- 14 Daubresse M.-P. (rapporteur) (2008), « Rapport fait au nom de la commission des affaires culturel (...)
“An evaluation committee was set up under the chairmanship of M. François Bourguignon, established from the beginning of the experimentation, and not a posteriori as is too often the case, and an evaluation protocol was drafted, based on comparisons with control areas with experimental areas”.14
25The model of randomised experimentation is thus covered by the constitutional framework authorising local derogations to rules, for simple methods of evaluation. Its influence has grown with the approach of “social experimentation”, which is presented in France as a major innovation that can resolve complex social issues otherwise endlessly debated inconclusively (Commission Famille, vulnérabilité, pauvreté, 2005). But it is far from having replaced traditional administrative experiments. Only these can anticipate the effects of reforms in all their dimensions, without succumbing to the pressure for change. It is here that the two forms of assessment diverge. Administrative experimentation is motivated by doubts over norms and standards : it seeks to anticipate the negative consequences of reform decisions. Random testing is guided by hopes concerning new norms and standards : in this case proposed reforms are assumed to have positive effects.
26Economics as a science is familiar with the idea of experimentation. In mainstream neo-classical economics, experimentation is one of the “empirical dimensions” of testing microeconomic models, by simulating agents’ real behaviour in a laboratory environment. This field has grown considerably since its beginnings in the immediate post-war period, leading finally to the award of the 2002 Nobel prize in economics to Vernon L. Smith “for having made laboratory experimentation an instrument of empirical economic analysis” (Biais, Rullière, 2003). However, while this form of experimentation does partly help with decision-making, particularly in terms of economic policy, its primary goal is knowledge : its aim is to test the validity of economic theories, or to explain the observed regularities.
27Experimentation takes on another form when it is used not merely to provide “validated” theoretical models for policy-makers, but when it aims to test the effectiveness of a given policy instrument in achieving a specific result. The process then falls within the field of experimental science, and requires the use of a robust method to create test groups. Random assignment, theorised by Ronald Fischer in 1935, is the best means for conducting this type of research. It has the merit of rendering the various samples thus constituted identical, as “randomisers” have systematically pointed out. Precision only depends on sample size:
- 15 Ministère des Sports, de la Jeunesse, de l’Éducation populaire et de la Vie associative, Fonds d’e (...)
“Random draws, of a test group and a control group within a same population, ensure the comparability of the two groups: on average, the population characteristics of each of these groups are not statistically different. The larger the sample, the more these characteristics are likely to be similar. This property applies to the observable characteristics (gender, age, academic achievement, educational level, etc.), but also those which are not : characteristics which are specific to individuals and difficult to measure, such as motivation, self-confidence, etc.”15
28Strict adherence to random assignment is a safeguard against selection biases that disrupt the value and precision of estimates made from the data of an experiment. The areas studied by Fischer concerned experimental science: the random assignment here is a matter of procedure, of method, as is the quality of the collection of experimental data. When an experiment is conducted according to the rules of the art, the difference in values for the criterion tested between the treated and untreated groups is indeed due to the specific effects of the criterion itself.
29It is tempting to consider various social policies as “treatments” applied to populations whose behaviour policy-makers want to change. Compared to other methods for proving the effectiveness of reform, “randomisers” hold out the promise of greater scientific certainty regarding the effects of a measure on specific behaviours. However, once the experiments are conducted in the real world and not in the laboratory, the evidence is more difficult to establish. As we shall see, several examples of experiments conducted in France in recent years show that their proponents have not attached due attention to the consequences of their immersion in the real world concerning the robustness of the results. In the case of the RSA, it seems that initially assessors of the experiment were only observers who had no control over the conduct of operations at any time. In the experiment on the placement of unemployed persons, the assessors were not able to control random assignment in the operation. Finally, in the testing of the “Parents’ schoolbag” (la Mallette des parents) project, which aims to provide parents with better information about how their children’s schooling and education operate, it was not possible to isolate statistically the specific effect of the measure.
30The results of the experiments concerning RSA income support conducted in 33 Departments on incentives to return to work generated controversy in parliamentary discussions in the autumn 2008 (Gomel, Serverin, 2009). These results were also discussed by some economists (Cahuc, Zylberberg, 2009). Despite the very short duration of the experiment (five months), experimenters provided discussions with a very weighty “finding”, namely that the rate of return to employment had increased by 30 %. A few months later, when results were available for a slightly longer period, the effect of the RSA turned out to be three times lower. Whatever the actual level, this percentage identified did not have the meaning which lawmakers attributed to it. But it only had a purely statistical sense, summed up by the Scientific Evaluation Committee :
“The rate of entry into employment of recipients of the RMI in the experimental areas is on average higher than in the control areas, but the gap varies quite widely between Departments and over time.”
(Comité d’évaluation des expérimentations, 2009, p. 12.)
31Nevertheless, the difference between the experimental and control areas, which was 30 % on average in the first five months, was no more than 9 % after 15 months, at the usual 5 % threshold of significance (Comité d’évaluation des expérimentations, 2009, p. 13) :
“[...] The average monthly rate of job entry in control areas is 3.1 %. The difference between experimental and control areas zones is 0.28 percentage points, or an extra 9 % of job entry rates in areas experimenting the RSA. […] The probability of being wrong in asserting that the effect of the experimental RSA on the return to work is greater than zero is 12 %. This is not a high value, but nevertheless leaves limited room for uncertainty.”
32This was not the only weakness of the experiment. When the policy came into force, a massive phenomenon of non-take-up of its new component –the RSA-activité, aimed at the working poor– cast doubt on the relevance of a model geared only to financial incentives (Serverin, 2012).
33Apart from discussion of the results, two methodological issues raise questions.
34First, the assessors were not able to determine the method of fixing the test and control areas, which is particularly disturbing given the importance the experimenters give to random assignment. In fact the law gave local authorities the freedom to choose the concrete modalities of the experimentation. In particular, the Regional Councils selected the zones for experimentation. These were often the most disadvantaged socially, while Councils showed no concern for finding a control zone for comparisons. The team responsible for selecting the control areas was not able to match each zone tested with a control area. It was not therefore possible to compare the rate of return to employment for each Department conducting the RSA experiment with the effects of the RMI, which would have provided information about variations in the RSA experiments. As the treated and control populations were comparable on a general level, the only calculable outcomes about the return to work related to the average difference between the entire population treated and entire control population.
- 16 Another survey conducted by the Crédoc (Research Centre on the Study and Monitoring of Living Cond (...)
35Second, qualitative research monitoring the experimentation has shown that information about the persons concerned in the areas experimenting the RSA was not better controlled for either. Yet it was precisely here the incentive effects are to be found, which the proponents of experimentation were looking for. A monographic study conducted during the experiment in five Departments (Loncle et al., 2008) showed that communication with target audiences was “prudent”, not to say opaque, whereas persons concerned should have had access to complete information. The objectives themselves were not limited to the resumption of activity, but could have been broader (changing the action of administrators accompanying the policy, ensuring predictability of income, etc.). They would then have shown greater continuity with the reorganizations already occurring as part of the devolution processes of the RMI.16
36In reality, the conditions of the experiment were only controlled at the margin. This should have prompted caution on behalf of the assessors, and at the very least, it should have led them to refuse to provide any results whatsoever concerning the “incentive” nature of the measure.
37A large randomised experiment was conducted in 2007 by France’s national unemployment insurance fund and the public network of jobcentres (respectively Unédic and ANPE) on enhanced individual support and accompaniment for jobseekers. This experiment testified to the changes concerning the role of random chance in experiments, the reasons for such shifts and their consequences for results.
38The evaluation of the effectiveness of two procedures for the enhanced accompaniment of jobseekers was a first in France, in terms of the exceptional means mobilised (the evaluation covered altogether more than 200,000 jobseekers) and its methodological quality. On the basis of a random draw, three groups were created :
“those who were offered POP (Private Operator Placement) accompaniment ; persons offered ‘company destination’ or CVE accompaniment (Cap vers l’entreprise), and persons provided with the traditional enhanced accompaniment provided by French jobcentres”.
(ANPE, Dares, Unédic, 2008, p. 4)
39The authors stressed that the random draw ensured the comparability of the three groups (p. 4):
“The same proportion of young people, women, etc. (this can be checked) but also the same proportion of motivated, dynamic jobseekers, etc. (even if we cannot check this, as it is not really possible to measure motivation or energy, it is sure to be so if the groups are large enough and are drawn randomly). Therefore, if we observe differences in jobseekers return to work between these three groups, we know that these differences are due to the fact that these persons were not offered the same type of support, and that only.”
- 17 Before the merger of the ANPE-Assedic (the payment centres of the Unédic unemployment insurance fu (...)
- 18 It is only really possible to compare POP support with standard jobseeker support on the one hand, (...)
40Despite this presentation, everyone remembers that the experiment pointed to the superiority of the public operator (the CVE programme) over the private operators, while the experiment occurred at a time when the ANPE and the Unédic were competing with private operators to obtain the market for jobseeker placement.17 Several factors explain this misunderstanding, though a direct comparison between the CVE and the POP was not on the agenda of the experiment.18 It seems there were even hesitations concerning the objectives of the study within the final report itself (Behaghel et al., 2009).
41On the one hand, the report recalls the initial objective (p. IV):
“This protocol, of ‘controlled experimentation’, guarantees that the different situations in the labour market that can be observed between the groups after several months can only result from the benefits of the programmes. Thus the value-added of CVE and POP programmes is measured with precision and certainty”.
42Rates for exiting unemployment at three, six and twelve months of compensated flows without enhanced accompaniment are given: 12% at three months, 23% at six months, and 37% at twelve months. “After 12 months, CVE support raises the exit rate from unemployment from 37 % to 44 %” and “the effects of POP […] are generally weaker and later” (p. V).
43On the other hand, the report emphasizes the importance of direct comparison between the two operators. For this, observation was restricted to common geographic areas to ensure that the “economic conditions and target populations were then identical” (p. V).
“In these areas, the effects of both programmes are stronger, but also more mixed: CVE increased the exit rates by 11 points from the third month, while over this horizon the effect of POP was still not significant. At 12 months, CVE had an impact of 8.5 points and POPs had an impact of 6.4 points” (p. V).
44From one part of the report to another, the objective of the assessment changed, from examining the behaviour of jobseekers to looking at the operators. This shift was due to the conditions in which the experiment was conducted, and more specifically due to the direct intervention of project operators in the final assignment of jobseekers to the groups tested. Indeed, according to the study, only 47% of jobseekers randomly assigned to a POP were actually supported by the POP, with the support rate of CVE projects being 43%. At the same time, only a small proportion of non-entrants into projects –less than 20%– reflected the refusal of jobseekers themselves from participating in a project. In most cases, jobseekers were rejected from participating after selection was directly organized by operators, on a case-by-case basis. The latter tried to keep only those jobseekers most likely, in their view, to benefit from the enhanced accompaniment. The greater selectivity of the CVE scheme could be explained by a better knowledge of jobseekers and so explain the greater success of enhanced accompaniment. This issue is discussed at the very end of the summary of the final evaluation report. It is pointed out as a new item to be addressed in future analyses. But it actually revives questions about the ability of random assignment, which was the main innovation of the method, to help identify the direct effects of the programmes in explaining differences in the exit rates from unemployment into getting a job. As the authors of the report themselves finally state,
“it is plausible that differences in outcomes [of the POPs] compared to the CVE programmes may be interpreted by the incentives given to the different actors involved” (p. VIII).
45Despite the very significant adjustments to the random assignment in the implementation of the experiment, the final report continued to focus, in its presentation of the experimental protocol, on the importance of (pp. 2-3) :
“The value-added of a programme is defined as the difference in the situation of individuals benefiting from the programme compared to what would have been their situation if they had not benefited from it […]. To re-create this hypothetical situation (i.e. the counter-factual), assessment generally refers to a control group. […] Constructing an appropriate control group is difficult. […] It is nevertheless possible to ensure, in the construction, that we can obtain a group of beneficiaries and a control group which are thoroughly comparable. This can be done using random selection from the same population. The differences in paths observed following the programme’s implementation can then be attributed transparently and robustly to the programme, and to it alone. This was the principle of the experimental protocol applied to the evaluation of the POP and CVE programmes…”
46The final presentation, once the experiment was conducted, used exactly the same words to publicise the assessment (ANPE, Dares, Unédic, 2008), despite the “uncertainties” faced during the implementation.
47The assessors neglected the impact the selection of operators had on the scope of the enhanced accompaniment, and were only concerned about the voluntary participation (compliance) of jobseekers:
“The assessment only provides information on the average impact of enhanced support when participation is voluntary, and only if 50% of persons to whom the programme was offered actually participated : the average effect on all jobseekers would potentially be different if participation were compulsory and if 100% of jobseekers thus participated” (p. 14).
- 19 The acceptance of jobseekers randomly assigned to reinforced accompaniment under CVE or POP is onl (...)
48What is ultimately calculated is the average impact of support on persons accompanied (Parienté, 2008). The conditions of the effective entry into the support programmes of the experiment were taken as given.19 Yet these conditions modified the very essence of the “support effect”.
49The “Parents’ schoolbag” project implemented in 2008-2009, within the Youth Experimentation Fund (Fonds d’expérimentation pour la jeunesse – FEJ), held out the same prospects of improving problematic social situations (Avvisati et al., 2011). It was evaluated by a team from the Paris School of Economics. Funding for this experiment was justified by the fact that the involvement of parents in middle school and its effects on pupils’ behaviour is a crucial question which has long been debated. And “yet nothing was ever really tried to shed light on the question rigorously” (p. 5). For the first time in France, an experimental trial in this area would aid in “judging the effectiveness of a policy” and “the appropriateness of generalising” the policy (p. 5). This would be achieved thanks to an experimental protocol aiming at “a rigorous and transparent assessment” (p. 5) of the programme. According to the investigators, “as a random draw ensures that there is no systematic difference […], the differences that can be observed […] can be definitely attributed to a single cause: the benefit of the programme” (p. 5).
50The trial took place in 40 middle schools belonging to the regional education Academy of Créteil (to the south-east of central Paris) (pp. 3-4).
“Earlier this year, about a hundred classes were randomly selected. Their parents were then invited by the principal to attend a series of briefings on the functioning of the middle school and to discuss how best to help children and to interact with teachers. At the end of the school year, when comparing these parents to other classes in the first year of middle school (initially similar but not randomly selected), they were characterised by being significantly more involved in their children’s schooling. […] In particular, the data show that this increased involvement resulted in a significant improvement of children’s behaviour.”
51This improvement was measured using a series of indicators of student behaviour relating to absenteeism, sanctions, “distinctions” and to school life scores.
“Whether it is absenteeism or a composite ‘quality of behaviour’ score summarising three other variables, the test groups’ advantage over the control groups is about 10% of a standard deviation. Given their magnitude, these differences are clearly greater than chance effects in the investigation” (p. 15).
52Moreover, the experiments have shown that “the virtuous effect of these openness policies towards parents extends beyond the usual circle of those families who are most attentive to the tuition of their children, and affects families and pupils who are most detached from the school system (by ripple effects in classrooms)” (p. 3).
53But how is this improvement to be explained? For the experimenters, there was no doubt that this was indeed a result of the parental involvement programme. However, it is far from certain that the participating population was actually more involved in middle-school life. The results of the experiment show an equivalent rate of non-responses by the parents of test groups and control groups in the final assessment, whereas one would have expected greater participation of the former. This is a weakness in the methodology which was recognised by the experimenters, but which they suggested is limited in scope:
“If the response rates were very different […], then comparing the responses obtained in test classes and control classes could no longer be interpreted as reflecting the sole effect of eligibility for the experimental programme” (p. 10).
54But even if non-significant, the fact that the rate of parent responses in test classes is still slightly lower than for the control classes is troublesome, for an experiment that specifically addresses parental involvement and its effect on children’s behaviour.
55As we can see, it is not so easy in a “natural” experimental framework to ensure the validity of results, from an internal point of view (in terms of the variables used). This is even truer from an external point of view, because of the impossibility of being certain that other causal variables do not intervene. Yet it is the external validity that has to do with generalisability : “are these results valid also for the broader population for which the policy or treatment is being considered ?” (Rodrik, 2008, p. 16).
56The scrapping of an experiment is not easy even if Article LO 1113-6 of Local Government Code provides for this. The assessors of another experimental programme of the Academy of Créteil to tackle truancy did indeed manifest such reticence. This programme made the headlines and received a negative review in April 2010, at the end of its pilot scheme, immediately followed by justifications (Behaghel, Gurgand, 2010, p. 16). This review “does not mean the measures had no effects –only an impact assessment could demonstrate this– but that the conditions of assessment were not met”. Accordingly, the conditions of the scheme’s social acceptance mentioned in the review in fact constituted two additional limits relating to the legitimacy of an experimentation in social matters: first, issues to be evaluated should be consensual (though randomised experimentation is presented in canonical way as determining issues that divide public opinion when there is a lack of evidence); and second, projects must be defined without prior knowledge that they will be funded as part of an evaluation. This seems to contradict the whole principle for responding to the call for projects by the Youth Experimentation Fund. The assessors added another type of condition (p. 2):
“from a scientific point of view, it is clear that the conditions for an objective assessment –that actors do not change their behaviour in view of assessment– were not met during the pilot project. There is no reason to think that these conditions were met in 2010-2011”.
57Knowing when and how to give up a project is indeed a challenge facing social experimenters who are officially involved in reforms.
- 20 Guide méthodologique pour l’évaluation des expérimentations sociales à l’intention des porteurs de (...)
58One of the remarkable features of randomised social experiments is their vocation to be generalised. A guide on the methodology of the evaluation of social experiments (Le Guide méthodologique pour l’évaluation des expérimentations sociales)20, published for project leaders in 2009, in fact set this as the primary purpose of projects (p. 2):
“This is a social policy innovation which is being initially launched on a small scale, given uncertainties about its effects and implementation under conditions that allow evaluation of the results, with a view to future generalisation, if the results are convincing”.
59In its first evaluation report, the Scientific Council of the experimentation fund argued that the assessment of its action “would be judged in particular by its ability to inspire public decisions” (Ministère de l’Éducation nationale, de la Jeunesse et de la Vie associative, 2011, p. 6).
60If we take the promises of social experimenters seriously, then we should soon see reforms emerge that are entirely constructed on the basis of experimental results. It is precisely this criterion of operationality that has dominated financial investment in such experiments for the last five years. It seems that today there is a growing awareness concerning the realism of these goals, and that there is a return to a more traditional approach to evaluation.
61Soon after having been promoted by the Hirsch Commission, random experiments found their place in the budgets of central government, in the form of two dedicated funds: the Innovation and Social Experimentation Fund (Fonds d’innovation et d’expérimentation sociale – FIES), established in 2006 ; and the fund supporting experimentation for young people, better known under the name of the Youth Experimentation Fund (Fonds d’expérimentation pour la jeunesse – FEJ), created in 2008.
- 21 The Interministerial Mission on Annual Projects for Performance (2012) (« Programme 304. Lutte co (...)
62Since the 2006 Finance Law, the FIES has provided support for the development of experiments in anticipation and in support of public policies favouring solidarity and social cohesion. Experimentation is included in the Mission: “Solidarity, integration and equal opportunities”, in the Programme 304 of “Fighting poverty : active solidarity income and social experiments”, under Action No 2 called “Social experiments and other experiments in social and social economy matters”.21 This action is presented as an embodiment of
“Resolution No 13 of the Families, vulnerability, poverty report of 2005, which aimed to make ‘boldness, innovation and experimentation’ keywords in public actions, based on the fact that action required for vulnerable families had to be customised and innovative” (p. 31).
- 22 A document for cross-cutting policies. The 2011 Finance Bill (2011) (Document de politique transve (...)
63Scientific backing is widely put to use: “selected by a panel including qualified personalities, experiments supported by the innovation fund and social experimentation are to be encouraged through calls for structured projects on topics listed above”.22
- 23 Created by Article 25 of the Law of 1 Deccember 2008, modified by the Law No 2010-1658 of the 29 D (...)
64As for the fund supporting experimentation for young people,23 it places experiments in the longer time horizon of research:
- 24 Decree No 2011-1603 of 21 November 2011.
“The fund supporting experimentation for young people, established under Article 25 of the Law of 1 December 2008 referred to above, aims to fund experimental programmes geared to promoting pupils’ success in schools, to contributing to equal opportunities and to improving the sustainable social and professional integration of young people under 25 years old. The fund may therefore finance spin-off experiments in new territories […].”24
65To its promoters, the fund’s objective is not to finance projects only for their own sake, but “to learn, in order to capitalise mobilised knowledge for the design of future public policies” (Ministère de l’Éducation nationale, de la Jeunesse et de la Vie associative, 2011, p. 7). Budgets and actions have followed. Since its creation, the Youth Experimentation Fund has launched 11 calls for projects, received over 1,500 experimental project proposals, and has brought together over 30 expert panels. More than 380 experiments (lasting an average of three years) have been selected throughout the country (mainland France and overseas territories) addressing extremely varied themes.25
66While the activities of the FIES have not been accompanied by dedicated indicators, the actions of the EFJ have had two. These were set out in the 2012 Finance Bill, in Programme 163 for “Youth and associative life” of the annual performance project of the “Sport, youth and community life” Mission. The former measures the share of projects which actually began six months after their selection. The latter indicator identifies “controlled experiments” within all experiments. This indicator is accompanied by a commentary which restates the principles and precepts of the methodological guide (p. 107) :
“[The indicator is] intended to support the development, via the EFJ, of a so-called ‘controlled’ experimental approach, based on the observation of a test group -benefitting from a particular policy/measure and a control group that does not benefit from it. These groups are constructed by selecting people using random draws. This approach is based on quantitative assessment, and has been practiced especially in Anglo-Saxon countries for decades. It still remains to be developed widely in France, especially in the field of public policy. Based on representative samples (from hundreds of individuals to thousands, or more), the approach provides strong demonstrative scope to the policies/measures assessed.”
67The emphasis given to methodology reached its high point here, virtually holding out the possibilities of an official science in the service of political action. This is clearly going a bit too far. A significant change of tone can be seen in the PAPs (Projets annuels de performance) of the 2013 Finance Bill, which leans towards an approach that is not so tied to scientific procedure.
68The change in presentation is particularly sensitive for the experiments included in the Programme 304 of “Fighting poverty: active solidarity income and social experiments”. From 2013 onwards, Action 12 for a “social and solidarity-based economy” of the programme has been dedicated exclusively to actions relating to the support and development of a social and solidarity-based economy. The actions relating to other experiments in social matters are included in Action 13 for “Other experimentations”, which has changed significantly in volume. The actions of Programme 304 are thus loosing experimental funding: of the €5,981,487 attributed to the former action for “Social experimentation and other experiences in social areas and social economy” in 2012, €5 million has now been transferred to Action 12 for a “Social and solidarity-based economy”. The remaining €981,947 is now only linked to experiments in a limited way:
- 26 Commitment authorisations (Autorisations d’engagement – AE) are the upper limit of expenditure whi (...)
- 27 The Interministerial Mission on Annual Projects for Performance (2013) (Mission interministérielle (...)
“The 2013 budget of €981,487, for which committed authorisations = payment appropriations,26 will support the development of experiments in anticipation and in support of public policies in favour of solidarity and social cohesion.
The funds of Action 13 will allow support to be obtained for the operation of the New Agency for Active Solidarity (Agence nouvelle des solidarités actives – ANSA). This is part of a multi-year convention currently operating, with the objective of contributing to the development of experiments, the pooling of good practices across geographical areas (particularly in terms of access to minimum benefits by claimants) and testing and evaluating of innovative projects in the field to tackle poverty, projects which are focused on preventing the breakdown of social bonds.
They must also enable the development of social engineering approaches, as part of experimental programmes to test the relevance, effectiveness, coherence and efficiency of public policies supporting social innovation on a limited scale (in four of France’s regions). The programme will aim to strengthen the tracking and support capacities of decentralised networks of social cohesion with respect to initiatives aimed at strengthening social bonds in geographical areas, and to create momentum in the field of social innovation.”27
69The aim of supporting local projects henceforth could not be stated more clearly, and this has more to do with administrative experimentation than science-based experimentation.
- 28 The Interministerial Mission on Annual Projects for Performance (2012) (« Programme 163. Jeunesse (...)
70In the PAP of the Sport, Youth and Associations Mission for 2013, the tone also changed concerning the presentation of the objectives of experimentation.28 To be sure, the goal was “to support and evaluate, in a specified manner, innovative measures that contribute to youth empowerment, in the context of implementing new public policies for young people” (p. 89). But the methodology is only mentioned in passing, recalling that “the external and scientific evaluation of these projects, if possible controlled, is an integral part of the selection conditions of funded projects” (p. 89). The focus is on the dissemination of the work, as a way of pushing experimenters to take on the function of disseminating information. A new indicator (indicator 4.1) is produced, whose purpose is to measure the dissemination of the results of supported experiments:
“to provide useful food for thought for policy-makers as part of the development of youth policies, the results of experiments supported under the FEJ must be available and accessible. Their posting on the government website (www.jeunes.gouv.fr) is therefore important” (p. 89).
71The indicator includes two measures:
“The share of the experiments which were the subject of a processed and published evaluation report / The total number of funded experiments ; The share of the final evaluation reports that are processed and published in the year / total number of reports expected in the year” (p. 89).
72In the new terminology, the aim is now to provide “food for thought”, and not to generate strong demonstrative reach. In this context, simple random experiments become a mere reservoir of ideas, and are no longer oriented towards specific reform projects.
- 29 The Interministerial Mission on Annual Projects for Performance (2013) (« Programme 137. Égalité (...)
73Finally, new experiments are emerging which are more administrative in their approach to testing standards. The Solidarity, Inclusion and Equal Opportunities Mission includes Programme 137 for “Equality between women and men”. In 2013, Action No 14 was introduced to “support actions for experimenting measures in favour of equality between women and men” (p. 126).29 With a budget of €6.3 million, this action involved the creation of a budgetary fund, on 1 January 2013, for “experimentation in favour of women’s rights and equality between women and men”. Its objective is to “implement support programmes and experiments and lay the foundations of new practices promoting professional equality and the effective protection of women against violence” (p. 111 and 126).
- 30 They will relate to “the development of agreements in companies and the improvement of their quali (...)
74These experiments are defined by their objectives, without specifying any method.30 There is no longer any question of using randomisation, only support for testing measures in the true administrative tradition.
- 31 Article L. 1221-7 of the Labour Code (modified by a Law of 22 March 2012) states that in “companie (...)
- 32 Behaghel L., Crépon B., and Le Barbanchon T. (2011), « CV anonyme : ce que dit l’évaluation », (...)
75This change could augur a redistribution of responsibilities in their respective fields : decision-makers decide, social scientists conduct scientific experiments, which they must validate according to the rules of the art. However, researchers involved in experimenting reforms are still subject to the risk of their findings being instrumentalised : for example, as with the researchers involved in an experiment on the use of anonymous Curriculum Vitae (CVs), prior to the publication of a government decree (Behaghel et al., 2011).31 By invoking the results of the study, the Commissioner for Diversity and Equal Opportunities, Yazid Sabeg, renounced publishing the decree that would have made anonymity mandatory for candidates applying to the companies concerned. However, the authors of the study had not given a conclusion “on the full effects of generalising anonymous CVs”, while they specified that the study did not allow “for the existence of discrimination in hiring to be tested” as they had stated a few months earlier, in a long letter to the press.32 Researchers therefore clearly face difficulties in controlling the interpretations of their results by their sponsors. But such differences in interpretation are hardly surprising : in fact they bear out the autonomy of political justification relative to scientific justification, and should lead researchers to look elsewhere to obtain the validation or their work, rather than in political decision-making.
76The major criticism that can be made concerning social experiments, in the French tradition, is that they have blurred the line between research and policy making (Gomel, Serverin, 2009). To restore the distinction, two dimensions should be considered : i) the ethical concerns in the conduct of research, and ii) the necessary control of experiments by scientific communities, to ensure independent research.
- 33 This specification was introduced by Law No 94-630 of 25 July 1994, and was upheld by the Law of 7 (...)
77Various kinds of ethical control may be exercised in experimental social research. Apart from the ethical commitments of researchers themselves with regard to their sponsors (public or private), these involve especially authorisation and control mechanisms that govern research on human beings, in particular in the area of biomedical research. Can these rules be applied to social experiments ? There is no simple answer to this. Rules are set out in legal codes relating to the ethics of experimental research focusing on human life (the human body and health), but not on human behaviour. A priori, the definition of biomedical research in Article L. 1121-1 of the Public Health Code (in the version derived from the Ordinance of 23 February 2010) does not seem suited to this type of research : “The research conducted and practiced on humans for the development of biological and medical knowledge is authorised under the conditions provided for in this book and is designated below as being ‘biomedical research’”. However, Article L. 1121-3, paragraph 4 of the Code also clearly provides an (indirect) reference to experiment on behaviour : “in the sciences of human behaviour, a qualified person, together with the investigator, may direct research”.33
- 34 Article L. 1121-3 includes a 7th paragraph stipulating that “non-interventional research may be ca (...)
78This simple statement has found a normative extension in Law No 2012-300 of 5 March 2012 on research involving human beings, following the Bill submitted to Parliament by Olivier Jardé (Member of Parliament), in 2009. Its author wanted to give a status to non-interventional studies (such as monitoring cohorts), to allow them to be reviewed by ethics committees. The law introduced in Article L. 1121-1 of the Public Health Code distinguishes between interventional research (which involves intervention on humans) and non-interventional research, “in which all acts are performed and products used in the usual way, without additional or unusual procedures for diagnosis, treatment or monitoring”.34
79This text puts experimental research under the supervision of committees protecting persons. No experimentation on people’s behaviour, claiming to be “scientific” will henceforth be able to escape presentation before ethical bodies. This therefore only excludes administrative-type experiments, relating to the organization of government bodies, including assessments which do not seek to be scientific but which are technical and administrative, and which are similar to impact studies (Gomel, Serverin, 2011).
- 35 Comets, Réflexions sur éthique et sciences du comportement humain, 23 February 2007, p. 23 and: “A (...)
- 36 Comets, Éthique de la recherche dans l’expérimentation sociale, 19 January 2010. Online http://www (...)
80The law thus also supports ethics, as has now been identified for several years by the CNRS Ethics Committee (Comité d’éthique du CNRS – Comets). In a first opinion published in 2007, the committee recommended particular caution concerning experiments on behaviour which may involve psychological risks when applied to vulnerable persons, such as “persons in social difficulty, immigrants, prisoners, drug addicts”.35 In 2010, a second opinion focused on the ethics of research in social experimentation, and also recommended the submission of projects to ethical principles, and their publication in journals specialised in the experimental sciences.36
81Independent research is not synonymous with research without control. Assessment and evaluation have always been included in the core institutional arrangements for public research, through two complementary processes : the control of expenditure, and control of relevance. The first is normally a responsibility of the public authority that allocates funds to research organizations ; the second is provided by what is commonly called the “scientific community”. These principles of assessment are found in a specific chapter of France’s Research Code : the “Evaluation and monitoring of research and technological development”. Article L.114-1 in particular states that
“research activities funded in whole or in part by public monies, undertaken by public or private operators, are assessed on the basis of objective criteria tailored to each of them and based on the best international practices. Among these criteria, the contributions to the development of scientific culture […] are taken into account”.
82These assessments are carried out in full transparency, as provided by Article L. 114-1-1 :
“The procedures and results of the evaluation of a research activity funded in whole or in part by public funds provided for in Article L.114-1 are made public under conditions ensuring respect for secrets protected by law and the confidentiality clauses in contracts with third parties […]”.
83While social experiments do clearly fall within the domain of scientific research, as claimed by their promoters, and if they are necessarily publicly funded, then they should follow two paths of scientific assessment : i) assessment by peers, through publications in scientific journals with referee committees ; and ii) the budgetary assessment of the funds they use, whose proportion should be measured relative to other public research funding. The first type of assessment clearly requires defining the disciplinary scope of studies (experimental psychology, social psychology, economics, statistics, etc.), in order to identify their contribution to knowledge. As for budgetary assessment, it seems, as seen above, that the purpose of the expenditure should shift from pure social engineering towards creating knowledge. This is a welcome development. The expenditure on random testing could then return to its natural place in public spending, that is to say as part of the “research and higher education” mission of government.