Khan Academy Mastery Mechanics

If you’ve used Khan Academy, I’m sure you’re familiar with this graphic. It is shown at the end of each task to inform you of how you have progressed. You may not be familiar with the (somewhat complex) mechanics behind how students progress through each level of mastery, and that’s what I hope to clarify.

Mastery progression icons

Setting the Scene

First, let’s outline some vocabulary. Khan Academy has many different exercises. Each of these exercises contains a set of problem types, and each problem type has a set of specific questions. For example, the Count to 100 exercise contains two problem types. One asks you to fill in the missing number in a giant grid.

Counting 1

The other asks you to do the same thing, but with colored, skip counted numbers.

Counting 2

Each of these problem types has several unique items to give students a variety when answering questions within the exercise. All items within an exercise are assumed to have equivalent discernibility. This is certainly false, but a compromise we have made for now. It allows us to pick an item at random whenever we think you’re ready to work on that exercise.

Tutorial Mode: Plan your own practice problems

There are three ways students work on exercises on Khan Academy. The original way is known as “tutorial mode”. Students navigate directly to the exercise they wish to practice, and the page gives them randomly selected questions. They may revisit this exercise at any time and continue to do problems as long as they like.

Tutorial mode

Learning Dashboard: Work on tasks we recommend

The new and improved ways are both accessed via the learning dashboard. The learning dashboard contains Practice Tasks and Mastery Challenges. Practice tasks give you items from a single exercise until you fulfill the completion criteria (typically, 5 correct in a row). Mastery Challenges give you a set of items from many different exercises (typically 6 different exercises) and only one chance to get each item correct. Mastery Challenges contain several different types of cards: Mastery, Review, Challenge, and Analytics. The rest of this article will focus on these two access patterns.

Practice tasks and mastery challenges

Practice Tasks and Mastery Cards

Practice tasks show up on your learning dashboard based on our recommendation engine’s ordering of the exercises within the mission. Once you finish a practice task, you will move into the practiced state for that exercise. We will also recompute which exercises should show up as practice tasks on your learning dashboard.

After some time (typically, 16 hours), mastery cards will become available in your Mastery Challenge for each exercise in the practiced state. If you get the mastery card correct, this will move you into level two. If the recommendation engine predicts that you are doing very well in this exercise, it may promote you all the way to level three: Mastered. There is always a delay of some time between each promotion, which is one way we use spaced repetition to enforce long-term retention.

The majority of the work you do on the Learning Dashboard will be within Practice Tasks and the resulting Mastery cards. Challenge, Review, and Analytics cards are relatively rare.

Challenge Cards

As you complete practice tasks, we may determine that you are likely to already understand the concepts present in other exercises. If you have never done a problem in an exercise, but our knowledge model predicts that you will perform very well in it, we will give you a Challenge card. If you answer this challenge card correctly, we will promote you all the way from Unpracticed to Mastered. You won’t even have to do a practice task for this exercise!

Review Cards

After you reach the Mastered level, we issue some review cards after an even longer delay. Review cards follow a spaced repetition pattern similar to mastery cards. The first review card may show up 4, 8, or 32 days after you reach Mastered. After you answer that card correctly, the next card will show up after twice as many days (8, 16, or 64). Each correct review card will again double the number of days until the next review card shows up. Once the delay between review cards reaches 256 days, it will no longer double. Any time you get a review card wrong, you will be demoted to mastery level 2. You will start the review cycle again after getting a mastery card correct.

One exception to this is if the exercise is “covered” by another exercise. If the covering exercise is Mastered, it is assumed to take care of all review cards. The covering relationships are currently hand-curated and a rather small subset of exercises.

Analytics Cards

To help us understand how all of our students are learning, we sometimes issue an analytics card. Analytics cards are currently randomly sampled from all of our math exercises, meaning that if you are working on a 3rd grade mission, you may see a Calculus exercises as an analytics card. The random sampling is how we verify that students maintain a general, long-term learning gain. Sometimes these cards cause confusion for the students who get one way outside of their ability, but the data we get from these responses has allowed us to make some very dramatic improvements to mastery progression without sacrificing long-term learning efficiency. We are looking for ways to improve this experience without degrading the data we collect.

Recap and Infographic

The learning dashboard contains two tasks: practice tasks and mastery challenges. Practice tasks contain random problems from a single exercise and require that you fulfill the completion criteria. Mastery challenges contain a mix of mastery, review, challenge, and analytics cards – each from a different exercise. The correct or incorrect responses to each of these cards will move your mastery level up or down accordingly. Once you answer all of the questions in the mastery challenge, a new one is computed. If there are no cards available for the Mastery Challenge, none is shown.

Mastery progression infographic

I hope that brings some clarity to how we move students through our mastery levels. Note that we are constantly experimenting with improvements to these mechanics, so this post may no longer be 100% correct by the time you read it. If you have questions, don’t hesitate to ask!

4 thoughts on “Khan Academy Mastery Mechanics

  1. Matt, thank you for the post. I always enjoy reading them. You said, “All items within an exercise are assumed to have equivalent discernibility. This is certainly false, but a compromise we have made for now.” I have a quick question: do you track reading level of items in any way? There are various measures of reading level, a common one being lexiles: http://en.wikipedia.org/wiki/Lexile . It seems to me that for some questions, the main problem is the reading level, not the math itself.

  2. I know this post is pretty ancient, but I have only come across it for the first time now and after reading it I finally have a name for the one thing that I really have a gripe with as far as the mastery challenges go — those damn challenge cards! I know they are a welcome feature to many, but I want an option to exclude them. I want to always take the long way and practice new skills until I am satisfied that I really understand the topic before tackling it as part of a mastery challenge. I can often get a lucky guess in on a challenge card and suddenly I don’t have to practice a skill I barely understand and worse yet that skill gets lost in the mix now that it is showing as anything from level 1 to mastered! Which makes it harder for me to find that skill again to make sure I practice it properly.

    Because of the challenge cards I often avoid doing mastery challenges at all until I have practiced everything in a given subject, which totally messes things up with the spaced repetitions, as the earlier topics in a subject might go days, weeks or even months without review because of this, unless I make the effort to review all the practice exercises up to where I currently am in a subject.

    The way I like to go through content is very structured and methodical — in order, one subject at a time, following the playlists and practices in the order they appear in the playlist and only jumping around when the play lists aren’t in an ideal order and I am stuck with practice exercises that require skills that haven’t been covered properly yet, which seems to happen more and more starting around Algebra I and up. I want the mastery challenges to be part of my daily routine of reviewing what I have been learning and making sure I am keeping those skills up to snuff as I progress. Though thanks to the challenge cards I rarely make ideal use of the mastery challenges, which means I end up undermining most of the work you are doing here.

    Please, please, consider making the challenge cards optional, they are doing far more harm than good for folks like me. Despite all that though, I do want to thank you for all the work you have done to help make KA one of the top resources for online math education!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>