Data Quality Campaign’s Aimee Rogstad Guidera discusses anti-data backlash and more

Aimee Rogstad Guidera, Founder and Executive Director, Data Quality Campaign

Aimee Rogstad Guidera founded the Data Quality Campaign in 2005 as a temporary advocacy group to get every state to set up its own longitudinal data system by 2009. Today, every state has a data system that tracks students from kindergarten onward.

(edited for length and clarity)

Q: Why didn’t you go out of business after you accomplished your original mission?

A:  Just because we built data collection systems, that doesn’t change anything.  We now focus on making sure states are doing the right thing to make sure the data systems are being used. The real power of data is to help educators personalize learning, and to make sure parents have the data they need about their children in the right context.

We need to build trust in the system so that people understand that this data is there to help. For a lot of educators, data is seen as a negative thing that hurts them or that embarrasses them.

Q: Perhaps teachers aren’t using the data because the data we’re collecting right now isn’t that useful?

State data systems can be helpful. But that data isn’t going to personalize learning in the classroom. Using the local and class-level data to tailor instruction for each student, that’s the next frontier.

We also need to put an emphasis on data literacy for educators.Very few education schools are teaching it. That’s a role for the states — to help train teachers in data literacy.

Q: What’s your take on the anti-data backlash?

The latest backlash on data has come more from the Right. People are concerned about government intrusion, the loss of privacy and corporate access to data.

But another backlash issue is the idea of children being boiled down to numbers. Data is somehow taking the fun out of education, or taking the art out of teaching

Data by itself is scary. But lawmakers want to know, what do we get for this $5 million reading program? And if you want to close the achievement gap, you need data to measure it.

We need to change the conversation at the grassroots level. We’re working with the American Association of School Administrators at the National PTA to create more grassroots support for education data. It’s critical get teachers to talk about how they’re using data.

Rather than talking abstract data systems, we’re focusing more on people and storytelling. The data conversation comes alive through human impact. Every number involves a kid.

Q: How do you advocate for using data?

We shine a big klieg light on what’s working so that others can learn from their experiences. For example, we highlight Kentucky because no one was using a pile of excel spreadsheets. Then the state created a one-page feedback report for every high school so that the principal could see how his school’s students were performing in college. How many of them need remediation in math and English? That’s driven changes at high schools.

We also write fact sheets on specific topics, such as on early warning systems for detecting dropouts.

One of the biggest things we do is conduct an annual survey on where states are with building the infrastructure for and using education data. The ninth one comes out in November.

As a 501(c)3, we’re not allowed to lobby. But when folks ask us for our opinion or to comment, we respond. We don’t push specific pieces of legislation.

Q: Why are you so passionate about education data?

I’ve always been interested in education. I wrote my senior thesis (at Princeton) on the role of the business community in education. Later, I worked at the National Governors Association and the National Alliance of Business (in Washington DC).  Business people would keep asking, “We keep putting money into schools, why aren’t they getting better?” And when you went into a school, you wonder that anything got accomplished. There was no access to information. They were doing things based on hunches or past experience. Business people look at information — at data — before they make decisions. We didn’t do that in education.

MIT’s Resnick on uses and risks of data usage in education

I wrote this piece earlier this summer, looking at the use of technology in education. But our conversation wandered into the domain of data and I wanted to excerpt those parts of the interview for Education By The Numbers.

MIT technology trailblazer is a critic of computerized learning

Q: What do you think of using data to influence instruction? Using big data sets to change how schools teach kids?

A:  To be honest, being at a place like MIT, people here are focused a lot at looking at data and treat data in a very privileged way. I’m often on the side of saying, “Wait a minute. We shouldn’t be designing everything just on the data.” Yes, we should take advantage of the data. But there are other ways of trying to get information as well. For example, if we want to understand how and what children learn, sitting down and talking to one of the students can also be very useful.

Q: Then why are you collecting data on Scratch usage?

A: Now that Scratch is online, we can have access to lots more data about what kids are doing. And that can be useful. If we see that certain of the programming blocks are not used, it might make us wonder, should they even be there? Or is it confusing for some reason? Should we change them to make it less confusing? It could influence our design.

If we want to see, how is it that students start using a certain concepts? When do they start using variables? Are there certain experiences that people have that are more likely to lead into using variables? There are things like that that might change some of our pedagogy.

Looking at the data might change some of our design and some of our pedagogy. But I want to be careful not to make all of our decisions based on that.

Q: What do you think of so-called adaptive learning, where computers tailor instruction for each student?

A:  Clearly there are some advantages at having certain things personalized for you. As long as it’s some options, choices and suggestions, then it’s okay. But I wouldn’t want to be limited only to what a machine suggests for me. If it’s central to my experience, if I’m categorized in a certain way and pushed down a certain path, it could make a much worse experience for me.

The machine could have students avoid things they might have been interested in. If the machine is trying to make a guess, based on how I answered one question, what would be appropriate to show me next, even if you and I answer a question the same way, it could be for different reasons. Even if we make the same mistake on the same question, it might be for different reasons. When a machine tries to make suggestions for you, a lot of time it’s wrong. It can be more frustrating than it’s worth. I personally tend to be somewhat skeptical when the machines try to be too intelligent.

One other caution would be, it’s great to have things that are specialized for me, but it’s also great to be part of a greater community.

I sometimes worry [that] it’s very easy for computers to give feedback these days. It’s seen as this great thing. Students are filling out answers to problem sets and exams. Right away it shows them if they’re right or wrong and they can get feedback right away, which can influence what they do next. Getting feedback is great. I’m all for feedback.

My concern, it’s only easy to give feedback on certain types of knowledge and certain types of activity. I think there’s a real risk, that we as a society, are going to end up giving too much privilege to the types of knowledge and the types of activity that are most easily evaluated and assessed computationally.

Q: Are you worried about more multiple-choice worksheets in our schools?

A: If that’s the result, then it’s a really bad result.

(edited for length and clarity)

Federal watchdog slams charter school data

The Government Accountability Office (GAO) issued a report in July, complaining that charter school data is so incomplete that it could not determine whether charter schools are avoiding non-English speaking students. “Specifically, for over one-third of charter schools, the field for reporting the counts of ELLs (English Language Learners) enrolled in ELL programs was left blank,” the report summary said.

The bad charter school data not only undermines the ability of researchers to figure out if charter schools are better than traditional public schools, but it also infects aggregate district-wide and state-wide education data. The GAO recommended that the Department of Education should stop reporting English-language-learner data on the district and state level if the charter school data could not be fixed.

The GAO further recommended that all charter school data be further scrutinized to see if blank data fields are ruining data sets on academic achievement and graduation.

The report, entitled Education Needs to Further Examine Data Collection on English Language Learners in Charter Schools (GAO-13-655R) was released to the public on August 15, 2013.

A call for more data, and less anonymous data, to contain college costs

In Higher Education, Data Transparency, and the Limits of Data Anonymization, Reihan Salam in the online version of the National Review writes, “I am increasingly convinced that unless governments do a better job of measuring student learning and labor market outcomes, any reform efforts will be of limited use.” In the piece Salam cites an idea from Andrew P. Kelly and Daniel K. Lautzenheiser of the American Enterprise Institute that universities and colleges shouldn’t be forced to use the same standardized test, but these “institutions should have the opportunity to choose from a menu of assessments, with the results made public. Administering an exam twice during a student’s tenure can allow institutions to measure the value added by the institution.”

With parents in revolt over plans to warehouse K-12 data at InBloom, good luck convincing Americans that we need less data privacy when it comes to university education.

The education search engines are coming

Searching the internet for recipes, academic papers or ex boyfriends is easy. But if you’re a teacher looking for a lesson plan, a textbook excerpt, or a fun brain teaser to share with your class, good luck.

For example, I just googled “multiplying exponents”. A bunch of results pop up. If you’re a parent, it would take a few minutes to digest the first five of them and then explain how to multiply 10² x 10³ to your kid. But, if you’re a teacher, you’d have to dig for quite some time before you found an age-appropriate textbook explanation, worksheet exercises, or a lesson plan you could present in a classroom.

Michael Jay, President of Educational Systemics

A lot of people are working to try to change that and give teachers the power to comb through instructional material in seconds. (To understand the kind of searches that educators are dreaming of, check out recipe searches on Google. You can search by ingredient, cooking time and calories).

The main obstacle in searching for educational material is that it first needs to be tagged with keywords by humans. Metadata is the information that describes things so that anyone can search and find them. In art, it might be the artist’s name, whether it’s a painting or a sculpture, the art movement he comes out of, etc. In education, it could be whether it’s a video or a worksheet, the subject matter and when it was created.

I talked with Michael Jay, president of Educational Systemics. He’s a former science teacher who’s been working on creating so-called metadata for instructional content. He was also a presenter at the National Center for Education Statistics (NCES) STATS-DC 2013 July data conference, where there were several sessions devoted to metadata.

The first question I had was, why do we need to spend thousands, maybe millions, to hire and pay humans to tag instructional content on the Internet? The geniuses at Google have figured out how to write algorithms to search everything else. Why is education different?, I wondered.

Jay explained the problem with education is that the indexing tools cannot figure out instructional intent. Say, for example, Jay creates an exercise where he has kids take off all their shoes. Then the kids group the shoes in various categories as a way of learning how scientists classify things.

“Unless I write out that intent of ‘categorization’ specifically, that higher level goal that you’re trying to achieve, Google can’t infer that. It has to be written explicitly,” said Jay. “We’re often not explicit about instructional intent in the activities we engage kids in.” That’s where you need humans to describe the instructional materials through metadata.

Naturally, there are a variety of competing efforts to come up with a universal metadata standard. There’s a multi-state tagging initiative. And some organizations, such as Khan Academy, are tagging their videos and online worksheets their own way.

But one of the leading metadata initiatives is something called, the Learning Resource Metadata Initiative or LRMI. It’s funded by the Gates Foundation, which is also among the funders of The Hechinger Report, where I work and which produces this blog.

LRMI has been at this for more than two years and it’s taken this group of experts, which Jay is a part of, a long time to even come to an agreement on what the categories of metadata should be. Some people wanted instructional material tagged by grade level, from preschool to college. Others wanted a child’s age specified. Some people wanted to include ratings to help teachers and parents judge the quality of content, like the star ratings on Yelp. But that “paradata” has been pushed to the back burner for another time. In the end, the LRMI working group settled on this list:

1) Target audience (students, teachers, parents)

2) Aligns with which educational standards (Common Core, Next Generation Science Standards,  other international standards)

3) Purpose (assignment, group work, field trip, reading)

4) Time required (30 minutes, 1 hour)

5) Age range of end user (5-7, 7-9)

6) Intellectual property rights (open resource, specific publisher)

Even with the categories in place, there are plenty of controversies on how to tag properly. For example, an educational video game might have multiple purposes. Educators even argue about simple things. What is a book in our digital-reader age?

Mediating these controversies is difficult. Some suggest crowd sourcing can resolve disputes. But Jay argues there should be a governance structure. “If you ask 10 educators about something, you get 15 answers. Crowd sourcing doesn’t work with education,” he said.

A big breakthrough for the metadata folks happened two months ago, when Schema.org adopted the metadata categories established by the Learning Resource Metadata Initiative (LRMI). That’s an important step because all the major search engines, from Google to Yahoo to Bing, use schema.org.

So, far only one organization has created an intelligent search engine using the new education metadata. That’s ISLE, the Illinois Shared Learning Environment, and it was rolled out only three weeks ago in July 2013. It has 170,000 educational resources tagged in its search engine. InBloom is working on a search engine using the same LRMI metadata.

The big goal is for a major search engine, such as Google, to use the LRMI metadata and launch a search engine for educational content. When that will be is anyone’s guess. Back in the fall of 2011 Google was excitedly talking about rolling out an education search engine, but it seems to have pulled back since then.

Jay says it’s a classic chicken and egg problem. “They (a major search engine) want to do it as soon as there’s a critical mass (of tagged educational content), but how do you get a critical mass until they do it.”

States and districts pull back from InBloom Data warehouse

InformationWeek Education has a good piece, with a punny wilting headline, on the growing backlash against InBloom’s plan to warehouse data for states and school districts. InformationWeek’s David F. Carr writes, “Louisiana withdrew from the project in April and other states who initially expressed an interest have backed off. Most recently, Politico reported that Guilford County, N.C., which was taking the lead on the project in that state, is putting on the brakes. By Politico’s tally, ‘that leaves New York, two Illinois districts and one Colorado district as firm participants for now; Massachusetts is on the fence.'” At issue is the concern that private student data, from telephone numbers and addresses to test scores and learning disabilities, could accidentally or improperly be released to software developers and for-profit education vendors.

Previous InBloom coverage here.

Move over U.S. News, a new ranking for universities and scientific institutions

Mapping Scientific Excellence, a new website out of Germany, has come up with a novel way to rank the world’s best universities and scientific institutions. It ranks an institution’s excellence by the rate at which it produces scientific papers that are most frequently cited. An MIT Technology review of the site, which Lutz Bornmann at the Administrative Headquarters of the Max Planck Society launched on August 7, 2013, explains that “the site counts the number of papers produced by an institution in a given discipline and then counts the number of these that are among the top 10 per cent of most highly cited. If more than ten per cent of the institution’s papers are in this category it gets a positive rating, if less than 10 per cent, it gets a negative rating.”

The captivating part is the data visualization. The website lists the top institutions by discipline and displays them on a maps where you can compare regions in different disciplines.

The website, io9, notes fascinating surprises. “In physics and astronomy, for example, two of the top three institutions in physics and astronomy are Spanish: the Institute of Photonic Sciences in Barcelona and ICREA (Institucio Catalana de Recerca i Estudis Avancats) also in Barcelona. Ranked 8th, above Harvard and MIT, is Partners Healthcare System, a non-profit healthcare organisation based in Boston that funds research, mostly in the life sciences.”

As they say, publish or perish!

Community college graduation rate perhaps double what previously thought, new data show.

My colleague Jon Marcus wrote an interesting piece today explaining why community college graduation rates are actually double what the data usually show. That’s because many students transfer and eventually earn a degree somewhere else. Instead of a dismal 18 percent graduation rate, new data from the National Student Clearinghouse suggests that the community college graduation rate is greater than 33%. (I calculated that by multiplying 60% by 25%, which is the percent of students that transfer to 4-year institutions and eventually graduate, and added that to the original 18% graduation rate).

We don’t call someone a high school drop out if they switched high schools and eventually graduate. Why do we do that for community college transfers?

Q&A with Knewton’s David Kuntz: ‘Better and faster’ learning than a traditional class?

When education investors talk about so-called adaptive learning, in which a computer tailors instructional software personally for each student, the name Knewton invariably surfaces. The ed-tech start up began five years ago as an online test prep service. But it transformed the personalization technology it uses for test prep classes into a “recommendations” engine that any software publisher or educational institution can use. Today the New York City company boasts it can teach you almost any subject better and faster than a traditional class can. At the end of 2012, 500,000 students were using its platform. By the end of this year, the company estimates it will be more than 5 million. By next year, 15 million students. Most users will be unaware that Knewton’s big data machine is the hidden engine inside the online courses provided by Pearson or Houghton Mifflin Harcourt or directly by a school, such as Arizona State University and University of Alabama.

David Kuntz

The Hechinger Report talked with David Kuntz, Knewton’s vice president of research, to understand how the company’s adaptive learning system works. Kuntz hails from the testing industry. He previously worked at Education Testing Service (ETS), which makes the GRE and administers the SAT for The College Board. Before that Kuntz worked for the company that makes the LSAT law school exam.

(Edited for length and clarity)

Question: On your company’s home page, there’s a McDonald’s hamburger counter that says you’ve served more than 276 million recommendations to students. What exactly are they? Are they like a book recommendation on Amazon?

Answer: It’s not like book recommendations on Amazon. Amazon’s goal is for you to buy the book. The goals that are driving our recommendations are the big things you need to learn. This recommendation is just one piece along the way for you to get there.

The question our machine is trying to answer is, of all of the content that’s available to me in the system, what’s the best thing to teach you next that maximizes the probability of you understanding the big things that you need to know? What’s best next?

It’s not just what you should learn next, but how you should learn it. Depending on your learning style, it might be best to introduce linear equations through a visual, geometric approach, where you plot the lines and show the intersection. For others, they might respond better to an algebraic introduction.

Q: How are the recommendations “served”?

A: That depends on how our partner [the developer of the educational application, such as Pearson or University of Arizona] designs its online course.

Sometimes, the recommendations drive the whole course experience. The student comes in, signs on, and the system will say to them, “Let’s work on this.” And they work on this. There’s formative feedback all the way through. And then the machine picks the next lesson based on how the student did in that lesson. “Now, let’s work on this other thing.”

In other cases, it may be a study aid sidebar, “Okay, you’ve just completed your assignment, and didn’t do as well as you might have liked. Here’s something you should do now that will help improve that.”

It can be tailored remediation, or the full scope and sequence of the course or a blend of those.

Q: Bring us inside your black box. How have you programmed your adaptive learning platform to come up with these recommendations?

A: We have a series of mathematical models that each try to understand a different dimension of how a student learns. We model engagement, boredom, frustration, proficiency, the extent to which a student knows or doesn’t know a particular topic. Imagine three dozen models.

Take proficiency. We use an IRT, or Item Response Theory Model, which is commonly used in testing. It estimates the probability that a student is able to do something based on an answer to a particular question.

The data gets filtered through all those models in order to come up with a recommendation in real time.

Q: Where does the data come from?

A: We know nothing about a student until he logs on. But then we can have the full click stream. That’s every mouse movement, every key press. If they click on a link to review part of a chapter and then they scroll down and scroll back up a couple times, those are things we can know. If they highlight stuff on the page, if they ask for a hint, if they select option choice C and then change their mind 4 seconds later and select option choice D, those are all pieces of information we can know. That’s part of that click stream.

Q: I’m told this is “big data.” How much data are we talking about?

A: It’s a ton. The storage component of that data is the largest portion of our Amazon Web Service’s bill (laughs). It’s fair to say that we have more than a million data points for each student who’s taking a Knewton platform course for a semester. That’s just the raw click stream data. After the raw data goes through our models, there’s exponentially more data that we’ve produced.

Q: There’s a lot of concern by parents and policymakers about how companies are exploiting and safeguarding private student data. How do you keep it private?

A: We don’t know anything personal about the student at all. We don’t know their name. We don’t know their gender. We don’t know where they live. No demographic information whatsoever. All we know is that this is a student in this particular course.

Q: Can all subjects be taught through an adaptive learning platform? Or is it best for math?

A: We love math. Math is great is because it has a rich, deep structure to it. The concepts build upon one another. Physics and chemistry are similar to math that way.

Biology has a totally different structure. It’s more about different clusters of things, connected by crosswalks.

Often, it’s less about the subject than the goals of the course.

Take freshman philosophy. If it’s a survey course of great ideas and the evolution of those great ideas, it may start with the Greek philosophers all the way up to Rene Descartes (“I think therefore I am”), up through European Western twentieth century civilization. You talk about logical positivism, and then post positivistic philosophy…

Q: (as Kuntz is rattling this off, I can’t help but multitask and Google his bio. Yep, he was a philosophy student at Brown).

A: In that case, there really is an evolution to those ideas that can be described in a knowledge graph. And our models can recommend content on this knowledge graph for students to learn.

But the other kind of freshman philosophy course is less about the subject of philosophy itself. It cares about exposing to students to some of the great ideas in the survey course. But the goal is to use these great ideas as a spur to promote creative critical analysis and discussion. In this case, most of the interaction and most of the evaluation takes place in class discussions and in written papers.

For a freshman philosophy course that is focused around teaching students how to think on their feet, come up with counter examples rapidly, and interact with other students in an engaging and intelligent way, our approach may not work as well.

Q: How does your machine decide whether to focus on a student’s weaknesses or to go deeper into an area that a student is really interested in?

A: Student engagement and interest – those are factors we try to take into account. We try to balance areas where student is having problems with areas that a student is really interested in.

If our partner [such as Pearson] has enabled direct expression of a student’s interest, we can take that data and incorporate that into the process of making a recommendation.

Q: Does the data know better than an experienced teacher’s wisdom? Does the Knewton machine ever recommend something that runs completely counter to what a veteran teacher would do?

A: One day we’ll have some really good answers to that question. What we have seen, in some cases, is that the engine has made recommendations that teachers have found surprising. But pleasantly so. Something they hadn’t anticipated that the student would need. But when presented with it, the teacher recognized that it was something good for the student to be doing.

Q: This is hard to understand.

A: It requires putting aside a lot of the things we take for granted because we grew up and were educated in the current system. We think about things in terms of syllabi and chapters. It’s hard to step back from that. Are there better and different ways that we can organize and present content?

Data systems to identify dropouts proliferate but usefulness unclear.

Sarah D. Sparks attended the same NCES STATS-DC 2013 session about dropout indicators for first graders, which I wrote about recently here and a related piece here. Sparks’s piece for Education Week, published  July 29, 2013, includes the national context that 28 states are now using early-warning data systems to help identify potential high-school dropouts, citing a count by the Data Quality Campaign.

An interesting discussion follows in the comments section. Lots of money and time are being spent on these new fancy data systems, but a question remains as to whether data mining for dropouts is useful. Are we really identifying lots of troubled students that teachers are unaware of? And what do we do about preventing someone from dropping out once we’ve identified their susceptibility? Perhaps we should spend more time on data-driven studies for drop out interventions.