NCES Commissioner Jack Buckley’s Briefing Slide on PISA results
Conventional wisdom is that top U.S. students fare well compared to their peers across the globe. According to this line of reasoning, the US doesn’t make it on the list of the top 25 countries in math (or top 15 in reading) because America has higher poverty and racial diversity than other countries do, which drags down the national average.
Wrong.
The latest 2012 PISA test results, released Dec. 3, 2013, show that the U.S. lags among 65 countries (or sub country entities) even after adjusting for poverty. Top U.S. students are falling behind even average students in Asia. I emphasize Asia because Asian countries (or sub entities) now dominate the top 10 in all subjects: math, reading and science.
In descending order from the top spot in math, they are (1) Shanghai, (2) Singapore, (3) Hong Kong, (4) Taipei, (5) Korea, (6) Macao, (7) Japan, (8) Lichtenstein, (9) Switzerland and (10) the Netherlands. Most of these countries are also posting top-of-the charts reading scores. (Here’s the global list. See Table 1.A on page 19. I also chain the list — in two parts — at the bottom of the post for those who are having trouble clicking on the pdf file. Click on it to see a larger full-screen version.)
Let’s break down the data for the 2012 PISA (Program for International Student Assessment) conducted by the Organization for Economic Cooperation and Development (OECD) taken by 15 year olds around the world.
* The United States has a below average share of top performers in mathematics. Only 2% of students in the United States reached the highest level (Level 6) of performance in mathematics, compared with an OECD average of 3% and 31% of students in Shanghai, the top performing entity in this year’s PISA test.
* Students at the 90th percentile in the United States — the very top — are below the average student in Shanghai. Top U.S. students scored 600 in math. The average score in Shanghai was 613. (Click on chart at the top right of the page to see this in more detail).
* Massachusetts, the top performing state in the nation, did not come close to the top 10 in math. Their 15 year olds scored a 514 in mathematics, placing the state even with Germany at number 16. (To put this in context, Germany is alarmed by how low its PISA scores are.) Massachusetts did prove better in reading. Only three education systems scored higher.
* Poverty rates alone do not explain low U.S. test scores. In a telephone briefing, Andres Schleicher explained that the OECD attempted to adjust test scores for income and put all the students of the world on a level playing field. It turns out that the US has slightly lower poverty and diversity than other OECD countries on average. The average U.S. test score dropped after making this adjustment.
* There is also a problem at the bottom end in the United States. The scores of low-income Americans are exceedingly low. The U.S. has a higher percentage of kids that can’t even hit the lowest levels on the math tests than other OECD countries do on average. So, it is true that the scores of poor U.S. students are dragging the average down. Still, absent poor students, U.S. scores would still be low.
Interesting tidbits:
* Finnish slide. Seven years ago, U.S. educators and policy makers were all traveling to Finland, trying to understand the secrets behind its high achievement. Finland declined between 2006-2009 and again between 2009-2012, scoring 548 in math in 2006 and 519 in 2012. Finland is firmly out of the top 10 in math and science, although its reading scores are still high. The OECD’s Andres Schleicher says that demographic changes and immigration have not been high enough to explain the test score slide and it’s a bit of a mystery.
* Poland is showing substantial increases in test scores on all three tests, rising well above the United States.
* Vietnam’s debut on the list is very impressive. This high poverty nation falls between Austria and Germany at #17.
* Stagnation. U.S. scores on PISA exams haven’t improved over the past decade. See here. That’s a bit of a contrast from the NAEP exam where American students have been showing modest improvement. I believe the NAEP exam plays to U.S. strengths of simple equation solving. It has fewer word problems where students have to apply their knowledge to a new circumstance and write their own equations and models.
* Shanghai was also the top performer in 2009. Other provinces in China are expected to start reporting PISA results beginning with the next 2015 test and have similarly high scores.
* Asia rising. Notice the strong gains among the top performing countries. Shanghai, Singapore, and the next four education systems are all posting strong annual gains on their PISA tests. It may be that top US students aren’t getting weaker, but stagnating, while the rest of the world, especially Asia, is getting stronger.
* $$$$: The OECD data show almost no link between spending on education and PISA test results. Wealthier nations tend to score better. But the amount of money that a nation spends on education doesn’t seem to matter much. The United States is one of the biggest spenders in education, spending $115,000 per student on average between the ages of 6 and 15. The Slovak Republic spends less than half that amount at achieves similar test scores. Only four countries spend more than the United States: Austria, Norway, Luxembourg and Switzerland.
* Test quality. I took sample questions from the 2012 PISA math test and was impressed with the sample questions. Many are not multiple choice. So you can’t always use a Princeton Review technique of eliminating answer choices. You have to calculate answers yourself. I was also surprised by how many word problems there were in which you had to come up with models and equations yourself and not just solve for x in a given equation.
* Cheating. In previous posts and among colleagues, questions are coming up about cheating, especially in China. I haven’t seen evidence of widespread cheating on PISA tests that would affect a nation’s score. I know that an outside Australian contractor is involved in administering the PISA tests in China. But please comment if you have any information on PISA cheating.
Take away:
Yes, the United States has an achievement gap. Poor students are doing poorly. But our top students are nothing to brag about.
Related Stories:
Shanghai likely to repeat strong results on international PISA test in December (Nov. 18, 2013)
Data Debate: Smartest U.S. states don’t hold a candle to global competitors (Oct. 30, 2013)
Unclear where U.S. students stand in math and science (Oct. 25, 2013)
