This international test showed M’sian students got smarter. But why so many ppl bising?

Last month, the Ministry of Education said that Malaysian students are finally doing better in maths, science, and reading, yay! But then DAP politicians Lim Kit Siang, Ong Kian Ming, and Tony Pua said that the Ministry might be bluffing. Or worse, that they might have rigged the results.

Of course, politicians arguing about the Malaysian education system is nothing new. But this time, they were arguing about actual data from an international test called PISA. And the best thing about actual data is that you can actually check it!

But before that, we don’t think many of you have heard about PISA, so here’s a quick introduction…

What is this PISA thing? And why does it make people upset?

PISA stands for the Programme for International Student Assessment. It’s run by the OECD, which is a bunch of (mostly) rich countries that say they want to make the world a better place.

Every three years, countries can “volunteer” some of their 15-year-old students to take PISA tests in maths, science, and reading. Countries can get quite kiasu about their PISA rankings. (Which maybe isn’t surprising given that Singapore is usually at the top haha.)

In Malaysia, people got upset a few years ago when we ranked 52nd out of 65 countries in PISA 2012. We also scored waaay below the international averages in maths and science in another international test called TIMSS 2011. (Actually, between TIMSS 1999 and 2011, our maths and science scores dropped more than any other country.)

All this criticism and malu-ation is one reason why gomen decided that our main target for education quality in the Malaysia Education Blueprint 2013–2025 is doing better in PISA and TIMSS.

So what was the drama about PISA 2015 last month?

When the OECD announced the 2015 PISA results, our Education Ministry said that Malaysians students had done a lot better in reading, maths, and science compared to the other times we took the test, proving that the Ministry had fulfilled the promises in the Education Blueprint mentioned above.

But if you try to check our results on the official PISA website, you’d find that our results weren’t published:

Which leads us to the whole hoo-hah in the first place – Why did gomen say our students had done better in PISA 2015, when Malaysia was left out of the official PISA 2015 results list? As mentioned in the introduction, this is what the Opposition was questioning:

… I combed through PISA 2015 Report … and I found to my disbelief and horror that Malaysia is not in the list of the 72 participating countries/economies… – Lim Kit Siang, from his blog

But it’s not just an Opposition thing… the pro-education Parent Action Group for Education (PAGE) has also asked the Ministry to explain what’s going on.

So is the Education Ministry correct when they say that Malaysian kids did better in PISA 2015 than before? Or is something fishy going on? (Spoiler: Probably both.)

Lets take a look at what we know based on the numbers.

Was the gomen cherry-picking “good” schools to take the test?

Exhibit A: Lots of the schools initially chosen for PISA didn’t take the test!

Two days after the PISA 2015 results came out, the Ministry put out a statement saying that 8,861 students had taken the PISA test, and that the data was valid.

But … jeng jeng jeng … the official PISA results report says this:

In Malaysia, the PISA assessment was conducted in accordance with the operational standards and guidelines of the OECD. However, the weighted response rate among the initially sampled Malaysian schools (51%) falls well short of the standard PISA response rate of 85%. Therefore, the results may not be comparable to those of other countries or to results for Malaysia from previous years.– PISA 2015 Results (Volume 1), p. 304

Because the OECD can’t test all the schools in the country, they have to pick wakil schools that will best represent a country’s students, according to their technical guidelines. At the same time they also give an 85% leeway, meaning that as long as 85% of the schools they selected took the test, all’s good.

But what happened in Malaysia is that only 51% of the selected schools took the test. In terms of numbers, the 49% that didn’t take the test equalled to 83 actual schools. Instead, these 83 schools were replaced by 77 other schools (not initially chosen by the OECD) to take the test instead.

So because such a large number of students/schools who took the test weren’t part of the original sample, no one can say for sure if the 8,843 kids who sat for the PISA test really represents ALL Malaysian 15-year-old students. For reference, in previous PISA tests, our response rates in terms of keeping to the initial samples were 99 – 100%.

But what do we know about the kids who took the test? Well…

Exhibit B: A LOT of asrama students took PISA 2015.

So, 31.4% of kids who participated in PISA 2015 were from asramas — you know, those smart flers who disappeared from your school batch after they got shiny UPSR or PMR results. But according to official gomen numbers, only 2.5% of students in Form 4 (where most 15-year-old kids are) were attending asramas that year.

Exhibit C: Malaysian kids got a lot richer between 2012 and 2015??!

Besides the tests, PISA also makes students take surveys about their school and personal background. And according to the surveys, Malaysian kids suddenly had a much better average socioeconomic status in 2015. You can take a look at the Malaysian numbers here, and for other countries here, but good luck lol.

In Malaysia, our standard of living went from being similar to Lebanon and Tunisia in 2010 and 2012, to being better than Spain in 2015!?!

If you’re wondering what having rich families have to do with this, it’s because students with higher socioeconomic status tend to do better in PISA (and in other exams). And it’s not super likely that our socioeconomic status improved so much over just three years. So maaayybee the kids who took PISA were better off than the average Malaysian kid, which means that our PISA scores would look better than they should!

Exhibit D: Malaysian kids studied reallllyy hard for PISA 2015. 

You’re actually not supposed to study for PISA (woot), because PISA wants an accurate picture of how well kids can apply reading, maths, and science skills to “real-world” problems.

But it looks like Malaysian kids had lots of kelas intervensi to prep for PISA 2015 (see here, here, here, here, and here).  Also, some PISA 2015 students went to PISA training camps in hotels.

But the funny thing is, that having said all that…

Malaysia ACTUALLY did better in PISA 2015 (probably).

Even with the exhibits we just presented, this doesn’t (necessarily) mean our PISA scores are wrong!

PISA calculates each country’s scores using a canggih maths thing called weighted averages. Which basically means that they balance out the characteristics of the students who took PISA to match the actual population.

And it turns out that the PISA 2015 calculations really did weight the asrama kids more “lightly” and the SMK kids more “heavily”.So even though more asrama kids took PISA, the calculations balanced them out in the end. (See here for my not-very-canggih maths showing this.)

And if we use canggih maths to also balance out the kids from from better socieconomic backgrounds, Malaysian kids still did better in all three PISA 2015 subjects than in 2010 or 2012.

So even though students from better socioeconomic backgrounds tend to do better, students still scored higher in 2015 than kids from similar backgrounds in previous years.

BUT we still don’t know (a) how much our scores would have improved without the extra intervensi classes, and (b) whether there are other factors (e.g. each school’s average SPM performance) that the canggih maths didn’t account for.

So … what really happened to PISA 2015?

The Education Ministry said on 16 December that they’re preparing a report to explain why Malaysia was disqualified from the main PISA results list. Until then, here are some CILISOS Conspiracy Theories that we have:

• Theory 1: PISA 2015 was fully done on computer for the first time, but not all Malaysian schools may have proper IT facilities. They might have been replaced with asramas and more privileged schools (who have better access to computers)
• Theory 2: The gomen might have been curious about the progress of asrama students, so they purposely put more asrama kids there to get a more high-res picture of their performance. If all the PISA procedures were followed properly, this wouldn’t mess up the overall Malaysian picture either.
• Theory 3: The gomen really wanted to meet KPI, so they asked the selected schools who did badly in PISA mock tests to withdraw and replaced them with schools that did better. Yes, there are mock tests  😐

For fans of conspiracy theories, do remember that there’s no concrete evidence for Theory 3. DAP MP Ong Kian Ming suspects that Theory 3 is true, since the gomen was under so much pressure to improve results. It’s a tempting possibility, because it would explain lots of confusing stuff (like the many asrama kids, the higher socioeconomic score, and the 51% thing). But the OECD themselves has stated twice (here and here) that Malaysia followed all PISA operating protocols.

Until gomen releases the report, it’s a bit sad that we don’t know if our kids scored a clean victory in PISA 2015 or not. At the same time though, it’s pretty cool that the Education ministry is responding to calls from opposition MPs and civil society in a positive way – They’re getting ready to give the public more information in an upcoming report on the PISA 2015 issue,

So here’s hopng that our kids will keep learning more in school, and we’ll all keep learning to be a more open and democratic society!