If you’ve used Khan Academy, I’m sure you’re familiar with this graphic. It is shown at the end of each task to inform you of how you have progressed. You may not be familiar with the (somewhat complex) mechanics behind how students progress through each level of mastery, and that’s what I hope to clarify.

Setting the Scene

First, let’s outline some vocabulary. Khan Academy has many different exercises. Each of these exercises contains a set of problem types, and each problem type has a set of specific questions. For example, the Count to 100 exercise contains two problem types. One asks you to fill in the missing number in a giant grid.

The other asks you to do the same thing, but with colored, skip counted numbers.

Each of these problem types has several unique items to give students a variety when answering questions within the exercise. All items within an exercise are assumed to have equivalent discernibility. This is certainly false, but a compromise we have made for now. It allows us to pick an item at random whenever we think you’re ready to work on that exercise.

Tutorial Mode: Plan your own practice problems

There are three ways students work on exercises on Khan Academy. The original way is known as “tutorial mode”. Students navigate directly to the exercise they wish to practice, and the page gives them randomly selected questions. They may revisit this exercise at any time and continue to do problems as long as they like.

Learning Dashboard: Work on tasks we recommend

The new and improved ways are both accessed via the learning dashboard. The learning dashboard contains Practice Tasks and Mastery Challenges. Practice tasks give you items from a single exercise until you fulfill the completion criteria (typically, 5 correct in a row). Mastery Challenges give you a set of items from many different exercises (typically 6 different exercises) and only one chance to get each item correct. Mastery Challenges contain several different types of cards: Mastery, Review, Challenge, and Analytics. The rest of this article will focus on these two access patterns.

Practice Tasks and Mastery Cards

Practice tasks show up on your learning dashboard based on our recommendation engine’s ordering of the exercises within the mission. Once you finish a practice task, you will move into the practiced state for that exercise. We will also recompute which exercises should show up as practice tasks on your learning dashboard.

After some time (typically, 16 hours), mastery cards will become available in your Mastery Challenge for each exercise in the practiced state. If you get the mastery card correct, this will move you into level two. If the recommendation engine predicts that you are doing very well in this exercise, it may promote you all the way to level three: Mastered. There is always a delay of some time between each promotion, which is one way we use spaced repetition to enforce long-term retention.

The majority of the work you do on the Learning Dashboard will be within Practice Tasks and the resulting Mastery cards. Challenge, Review, and Analytics cards are relatively rare.

Challenge Cards

As you complete practice tasks, we may determine that you are likely to already understand the concepts present in other exercises. If you have never done a problem in an exercise, but our knowledge model predicts that you will perform very well in it, we will give you a Challenge card. If you answer this challenge card correctly, we will promote you all the way from Unpracticed to Mastered. You won’t even have to do a practice task for this exercise!

Review Cards

After you reach the Mastered level, we issue some review cards after an even longer delay. Review cards follow a spaced repetition pattern similar to mastery cards. The first review card may show up 4, 8, or 32 days after you reach Mastered. After you answer that card correctly, the next card will show up after twice as many days (8, 16, or 64). Each correct review card will again double the number of days until the next review card shows up. Once the delay between review cards reaches 256 days, it will no longer double. Any time you get a review card wrong, you will be demoted to mastery level 2. You will start the review cycle again after getting a mastery card correct.

One exception to this is if the exercise is “covered” by another exercise. If the covering exercise is Mastered, it is assumed to take care of all review cards. The covering relationships are currently hand-curated and a rather small subset of exercises.

Analytics Cards

To help us understand how all of our students are learning, we sometimes issue an analytics card. Analytics cards are currently randomly sampled from all of our math exercises, meaning that if you are working on a 3rd grade mission, you may see a Calculus exercises as an analytics card. The random sampling is how we verify that students maintain a general, long-term learning gain. Sometimes these cards cause confusion for the students who get one way outside of their ability, but the data we get from these responses has allowed us to make some very dramatic improvements to mastery progression without sacrificing long-term learning efficiency. We are looking for ways to improve this experience without degrading the data we collect.

Recap and Infographic

The learning dashboard contains two tasks: practice tasks and mastery challenges. Practice tasks contain random problems from a single exercise and require that you fulfill the completion criteria. Mastery challenges contain a mix of mastery, review, challenge, and analytics cards – each from a different exercise. The correct or incorrect responses to each of these cards will move your mastery level up or down accordingly. Once you answer all of the questions in the mastery challenge, a new one is computed. If there are no cards available for the Mastery Challenge, none is shown.

I hope that brings some clarity to how we move students through our mastery levels. Note that we are constantly experimenting with improvements to these mechanics, so this post may no longer be 100% correct by the time you read it. If you have questions, don’t hesitate to ask!