Questimator: Generating Knowledge Assessments for Arbitrary Topics

Guo, Qi; Kulkarni, Chinmay; Kittur, Aniket; Bigham, Jeffrey P.; Brunskill, Emma

Questimator: Generating Knowledge Assessments for Arbitrary Topics

Qi Guo, Chinmay Kulkarni, Aniket Kittur, Jeffrey P. Bigham, Emma Brunskill

Download BibTeX Cite

Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. Pages 3726-3731.

Abstract

Formative assessments allow learners to quickly identify knowledge gaps. In traditional educational settings, expert instructors can create assessments, but in informal learning environment, it is difficult for novice learners to self assess because they don't know what they don't know. This paper introduces Questimator, an automated system that generates multiple-choice assessment questions for any topic contained within Wikipedia. Given a topic, Questimator traverses the Wikipedia graph to find and rank related topics, and uses article text to form questions, answers and distractor options. In a study with 833 participants from Mechanical Turk, we found that participants' scores on Questimator-generated quizzes correlated well with their scores on existing online quizzes on topics ranging from philosophy to economics. Also Questimator generates questions with comparable discriminatory power as existing online quizzes. Our results suggest Questimator may be useful for assessing learning in topics for which there is not an existing quiz.

Qi Guo, Chinmay Kulkarni, Aniket Kittur, Jeffrey P. Bigham, and Emma Brunskill. 2016. Questimator: Generating Knowledge Assessments for Arbitrary Topics. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-16. IJCAI, 3726–3731. https://ijcai.org/proceedings/2016/524

@inproceedings{ijcai2016p524,
  title     = {Questimator: Generating Knowledge Assessments for Arbitrary Topics},
  author    = {Guo, Qi and Kulkarni, Chinmay and Kittur, Aniket and Bigham, Jeffrey P. and Brunskill, Emma},
  booktitle = {Proceedings of the Twenty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-16}},
  publisher = {AAAI Press / International Joint Conferences on Artificial Intelligence},
  editor    = {Subbarao Kambhampati},
  pages     = {3726--3731},
  year      = {2016},
  month     = {7},
  url       = {https://ijcai.org/proceedings/2016/524},
}

TY  - CPAPER
TI  - Questimator: Generating Knowledge Assessments for Arbitrary Topics
AU  - Guo, Qi
AU  - Kulkarni, Chinmay
AU  - Kittur, Aniket
AU  - Bigham, Jeffrey P.
AU  - Brunskill, Emma
T2  - Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-16
PB  - AAAI Press / International Joint Conferences on Artificial Intelligence
PY  - 2016
SP  - 3726
EP  - 3731
UR  - https://ijcai.org/proceedings/2016/524
ER  -