Office memorandum
Download 0.66 Mb.
|
- Navigate this page:
- 4.1 Experimental Sample Design
4*Analytical ApproachOur proposed methodology employs a controlled randomized experiment, whereby eligible primary schools in the most disadvantaged communities of Mexico are assigned to treatment and control groups. This procedure guarantees balance between treatment and control groups, in which the average characteristics of each group are similar, and subsequent differences in outcomes between treatments and controls may be attributed as causal effects of the intervention. This intervention has initiated an enhancement to the AGEs project to asses the impact of providing additional resources to the participating schools. The proposed AGEs project will be implemented in 125 treatment schools during the 2007–08 school year and will double the amount parent associations receive from an average of $600 to $1200 per school each year. Half of the money will be financed by the Ministry of Education through its usual support to these schools – all eligible schools will already be beneficiaries of the compensatory program. Funds will be transferred directly to selected schools using a trust fund specifically established for this purpose. The other half will be provided by the private sector as a public-private partnership. The private sector partners include Cinepolis (www.cinepolis.com.mx), Deutsche Bank Mexico (www.deutsche-bank.de), Fundación Televisa (www.esmas.com/fundaciontelevisa), Lazos (www.lazos.org.mx) and Western Union (www.westernunion.com). Supervision of the overall experiment will be supported by the NGO Investing in Education Foundation (www.investingineducation.org). The Mexican Ministry of Education will be in charge of implementing the project and will provide training to parent associations on how to manage the funds, how to organize meetings, and how to use student assessment information – functions it already delivers to beneficiary schools. 4.1Experimental Sample DesignThe randomization was conducted in several steps. During the period 9-20 July 2007, the research team visited Mexico City and collaborated with CONAFE in the selection and randomization of the primary schools that were to compose the evaluation sample. Schools in the treatment group will receive the extra grants for three consecutive school years. Schools in the control group will remain as counterfactual schools and will, therefore, not receive the extra benefits over the evaluation period. Both groups of schools are already incorporated into the AGEs and all will receive the base amount. First, we selected states with large indigenous rural populations, and that were well represented in the AGEs program. Budget restrictions allow the awarding of only a certain number of extra grants for the experiment each year, limiting our sample to 250 AGEs schools, of which we can only allocate extra resources to half. A statistical power calculation indicates that a sample of this size is sufficient to detect moderate student learning impacts at 95% confidence with reasonable levels of statistical power. Since 2000, almost 50,000 rural primary schools have benefited from AGEs. During school year 2006-07, there were 34,252 AGEs in 31 states. Our selected states, which have large indigenous populations as measured by their proportion in rural areas, are: Chiapas, Guerrero, Puebla and Yucatan. These four states account for 14 percent of the Mexican population overall, 22 percent of the rural population, and 37 percent of the national indigenous population. We excluded Oaxaca, even thought it has the largest indigenous population in a single state, because of problems between the teachers’ union and the government, which led to the closure of schools during most of the 2006-07 school year, and AGEs funds were not assigned. Similar problems exist to this day, and there is no guarantee that the experiment can be carried out in the state (see, for example, the daily Excelsior newspaper, 16 July 2007, “Regresa violencia a Oaxaca”). Furthermore, Oaxaca schools did not participate in ENLACE in 2006 and 2007, meaning that we do not have a baseline for the state. Our four selected states account for 17 percent of all primary public schools in Mexico, and almost 20 percent of all AGEs schools. In 2006-07, AGEs schools are distributed by state as follows: Chiapas (2,675), Guerrero (2,399), Puebla (1,265) and Yucatán (323). We randomly selected 250 schools as participants of the experiment. This was carried out using the 2007-08 database of AGEs schools provided by CONAFE, as well the national school census by SEP, which contains numerous characteristics at the primary school level for the beginning and end of each school year (see Table 1).
From the universe of AGEs schools in the four states we excluded boarding schools, schools that did not participate in ENLACE 2006, and schools that joined the mostly urban school-based management program (PEC). This left us with 5,930 potential schools to be selected for the experiment, of which 2,367 were indigenous. From these we randomly selected 250 schools using STATA software and the instruction “sample # count.” Table 2 presents the distribution of the randomized schools.
The randomization produced a distribution of indigenous and general schools that is close to the actual distribution of indigenous and general schools in the four states, as shown in Table 3. In other words, the randomly selected schools are representative of the distribution of indigenous and general schools in the four states. The 250 schools will be in the AGEs program for at least the three year duration of the experiment.
From these 250 schools, we randomly assigned 125 to treatment and 125 to control. The 125 under treatment will receive extra resources for AGEs, while the 125 under control will receive normal funds from the program. The randomization of treatment and control was done using STATA and its command “random=uniform(),” which randomly assigns values from 0 to 1 to each of the 250 schools. Then we used the command “gen var=group(),” to split the sample into two, where the 50 percent below are assigned to treatment and the 50 percent above to control. We performed an exogeneity test to assess the validity and accuracy of the randomization and check how well balanced treatment and control communities are in terms of their characteristics (see Attachment Table 4.1). In Attachment Table 4.1, we also present t-tests for the existence of other relevant programs, many of which are already generalized to most schools, such as parental training for the AGEs program, which is provided to all schools, or the national teacher incentive program (Carrera Magisterial). We also included a variable to account for the intensity of programs—that is, how many of them are received by the school concurrently. In all cases, the t-test results show that treatment and control schools are not different. This amounts to comparing the baseline values of the average outcome variables between schools in the treatment and control group. We checked that schools in treatment and control are as similar as possible by performing t-tests for the means of a series of characteristics calculated from the national school census. Randomization was run 50 times until it produced the more likely distribution according to the significance of t-tests for means. Table 4 presents the distribution of treatment and control schools according to whether they are indigenous or general schools.
It is important to note that before the final selection of schools was carried out we noted during planning meetings with CONAFE state representatives that some schools have switched to the PEC program, thus leaving the AGEs program. Since PEC provides about $5,000 annually to each participating school, compared to a maximum of $700 in AGEs, we face possible selective attrition during the years of the experiment. We analyzed this problem by trying to predict which schools are more likely to switch from AGEs to PEC by modeling the switch using 2005-06 and 2006-07 data. However, the model fit was poor and relied on very few significant variables. We thus concluded that the predicted probabilities of switching were not reliable and probably misleading. The rules for selecting PEC schools are broad, thus leaving considerable discretionary power at the hands of state officials. According to official regulations, PEC gives priority to schools in disadvantaged urban areas that voluntarily present acceptable school improvement plans. If funds remain, then these can go to indigenous peoples, schools targeting students with disabilities, multigrade schools, schools for migrants, and CONAFE’s community (non-formal) schools. Other factors associated with the allocation of PEC resources are state policies and available resources. All these factors make it extremely difficult to monitor the switching probability of schools from AGEs to PEC. In the end, all we could do was to drop those already in PEC from the potential pool of experimental schools. However, even if some schools from our sample do switch from AGEs to PEC, we will continue to follow them. Download 0.66 Mb. Share with your friends:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||