In the past few years, there has been uncertainty around whether the learning outcomes of South Africa’s schools are improving from their historically low levels.
There are three testing programmes South Africa can draw from to gauge trends, and all are international: the Trends in International Mathematics and Science Study (TIMSS), focusing on Grade 9; the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SEACMEQ), focusing on Grade 6; and the Progress in International Reading Literacy Study (PIRLS), focusing on Grade 4.
In 2012, TIMSS results pointed to substantial improvements in lower secondary maths and science since 2002. A few years later, SEACMEQ revealed improvements in the upper primary level.
Then in late 2017 came the news from PIRLS that reading in lower primary had remained static between 2011 and 2016. Was the schooling system not starting to pick up after all? The evidence was inconsistent and unclear.
In late 2019, I was asked by South Africa’s Department of Basic Education, the national authority for schools, to examine the raw data from the literacy study, which are publicly available, to verify the flat no-change trend.
My findings were surprising. The raw data appeared not to have been properly analysed in arriving at the conclusion that there was no progress. In fact, the progress was remarkably large.
It’s important to note, however, that even after improvements, South Africa still underperforms relative to most other middle income countries. What’s encouraging is that there’s a move in the right direction.
Inconsistencies
Testing done by programmes such as these involves selecting a nationally representative sample of around 300 schools in one year, and another such sample in a later year. Students in these schools write tests which repeat identical and highly confidential questions across different years.
This is what makes results comparable in ways that would never be possible in an examination system. It would be impossible to keep past examination papers secret.
If samples of schools are not nationally representative, that could result in national averages which are not comparable over time.
Was this perhaps driving a no-change trend in the Grade 4 literacy results?
Sampling problems had been found to produce inaccurate trends in a few cases outside South Africa.
I found there was nothing wrong with the sampling in the literacy study. But what was completely unexpected was to find that the 2011 to 2016 trend in classical scores didn’t correspond to the flat trend appearing in the study’s official reports. Classical scores are what teachers would be familiar with: 15 correct out of 30, meaning a 50% score.
These testing programmes use a complex statistical approach which results in another kind of score, called an item response theory score. As seen in the graph below, officially South Africa’s item response theory score in the study moved from 323 to 320.
The classical scores, in contrast, pointed to a large increase over the 2011 to 2016 period, from around 32% correct to 42% correct. Of 43 countries with a 2011 to 2016 trend, South Africa’s trend was the third-steepest increase, after those of Morocco and Oman. If correctly calculated, the item response theory scores should have pointed to a similar gain.
The inconsistency came about when the 2011 national score was converted from one scale to another by Boston College, the institution responsible for processing the data. This conversion affected only South Africa, as only South Africa took part in an easier test in 2011 and also participated in the study in 2016.
In early 2020, all references to the no-change finding were removed from the international report in response to the findings described here. Unfortunately, given that the erroneous trend was published in 2017, it remains replicated in many places.
The correction has important policy implications for South Africa. It removes the uncertainty. All three programmes now point in the same direction, which is upward. This offers hope in a context where South Africa’s underperformance in the international programmes is widely known and lamented.
What changed?
What lies behind the improvement? It’s a mix of education and non-education factors. Urbanisation has improved the access of young people to resources which facilitate learning, from electricity in the home to public libraries. In the schooling system, there’s evidence that access to textbooks has improved. Curriculum reforms have made it clearer what teachers should do.
Not acknowledging that there are improvements raises the risk of policy change, where perhaps policy stability is necessary. On the other hand, South Africa’s historical levels of performance have been so low that it would have been relatively easy to shift scores in the right direction. For further improvement to be assured, it can’t just be business as usual.
Research indicates that better teaching of reading in the early grades, a prerequisite for virtually everything else in education, is possible and necessary.
Understanding the limits
In a recent report on the attainability of the Sustainable Development Goals in the area of educational quality, I looked at what historical trends across the world suggest are the fastest possible rates of improvement in the international testing systems.
South Africa is in fact making progress not too far from the “speed limit”. An inconvenient truth about schooling systems is that when they progress, they are more like tortoises than hares. This has implications for testing programmes which monitor systemic progress. They need to be sufficiently rigorous and fine-tuned to pick up even relatively small gains.
During the past 20 years, the quality of South African schooling progressed from a level well below that of Botswana, to almost the level of Botswana. But Botswana too is an underperformer relative to income per capita.
Both countries need to improve the quality of teaching and learning much further. A proper basic schooling for all citizens is a human right. And the evidence is clear that quality schooling also bodes well for socio-economic development in the long run.
ــــــــــ
*Education economist, Stellenbosch University
