The User-Language Paraphrase Challenge
Download 202.53 Kb.
|
Concluding Remarks The User-Language Paraphrase Challenge provides researchers with a large corpus of hand coded evaluations across 10 dimensions of paraphrase. Correlation and accuracy results from sophisticated and baseline variables are also provided. Researchers are encouraged to analyze the data so as to provide optimal prediction, evaluation, or categorization of the data. Researchers should consider accuracy, speed of production, and scalability in detailing their approach. The User-Language Paraphrase Corpus can be downloaded at http://csep.psyc.memphis.edu/mcnamara/link.htm. Acknowledgments This research was supported in part by the Institute for Education Sciences (IES R305G020018-02) and in part by Counter-intelligence Field Activity (CIFA H9c104-07-C-0014). The views expressed in this paper do not necessarily reflect the views of the IES or CIFA. The authors acknowledge the contributions made to this project by Vasile Rus, John Myers, Rebekah Guess, Scott Crossley, and Angela Freeman. References Aleven, V., & Koedinger, K. R. (2002). An effective meta-cognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26, 147-179. Cavazza, M., Perotto, W., & Cashman, N. (1999). The “virtual interactive presenter”: A conversational interface for interactive television. In M. Diaz, P. Owezarsji, P. Senac (Eds.)., Proceedings of the 6th International Workshop on Interactive Distributed Multimedia Systems and Telecommunications Services, IDSM’99 (pp. 235-243). Toulouse, France: Springer. Dennis, S. (2007). Introducing word order within the LSA framework. In T. Landauer, D.S. McNamara, S. Dennis, W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp 449-466). Mahwah, NJ: Erlbaum. Dolan, B., Quirk, C., & Brockett, C. (2004). Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. Proceedings of the 20th International Conference on Computational Linguistics (pp. 350-356). Geneva, Switzerland: Coling 2004. Douglas Thompson, W., & Walter, S.D. (1988). Variance and dissent: A reappraisal of the Kappa Coefficient. Journal of Clinical Edidemiol, 10, 949-958. Dolfing, H., Reitter, D., Almeida, L., Beires, N., Cody, M., Gomes, R., Robinson, K., Zielinkski, R. (2005). The FASiL Speech and Multimodal Corpora. Inter/Eurospeech 2005. Gertner, A.S. & VanLehn, K.(2000) Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, K. VanLehn (Eds.), Proceedings of the 5th International Conference on Intelligent Tutoring Systems, ITS 2000 (pp. 133-142). Montreal, Canada: ITS 2000. Graesser, A.C., McNamara, D.S., Louwerse, M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, and Computers, 36, 193-202. Graesser, A. C., Olney, A. M., Haynes, B. C., & Chipman, P. (2005). AutoTutor: A cognitive system that simulates a tutor that facilitates learning through mixed-initiative dialogue. In C. Forsythe, M. L. Bernard, & T. E. Goldsmith (Eds.), Cognitive systems: Human cognitive models in systems design. Mahwah, NJ: Erlbaum. Graesser, A.C., Person, N.K., & Magliano, J.P. (1995). Collaborative dialog patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychology, 9, 359-387. Landauer, T., McNamara, D.S., Dennis, S., & Kintsch, W. (Eds.). (2007). Handbook of Latent Semantic Analysis. Mahwah, NJ: Erlbaum. Lockelt, M., Pfleger, N., & Reithinger, N. (2007). Multi-party conversation for mixed reality. The International Journal of Virtual Reading, 6, 31-42. McCarthy, P.M., Renner, A.M., Duncan, M.G., Duran, N.D., Lightman, E.J., & McNamara, D.S. (in press). Identifying topic sentencehood. Behavior Research Methods. McCarthy, P.M., Rus, V., Crossley, S.A., Bigham, S.C., Graesser, A.C., & McNamara, D.S. (2007). Assessing entailer with a corpus of natural language. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the twentieth International Florida Artificial Intelligence Research Society Conference (pp. 247-252). Menlo Park, California: The AAAI Press. McCarthy, P.M., Rus, V., Crossley, S.A., Graesser, A.C., & McNamara, D.S. (2008). Assessing forward-, reverse-, and average-entailment indices on natural language input from the intelligent tutoring system, iSTART. In D. Wilson and G. Sutcliffe (Eds.), Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference (pp. 165-170). Menlo Park, CA: The AAAI Press. McNamara, D.S., Boonthum, C., Levinstein, I.B., & Millis, K. (2007). Evaluating self-explanations in iSTART: Comparing word-based and LSA algorithms. In T. Landauer, D.S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 227-241). Mahwah, NJ: Erlbaum. McNamara, D.S., Levinstein, I.B. & Boonthum, C. (2004). iSTART: Interactive strategy trainer for active reading and thinking. Behavior Research Methods, Instruments, and Computers, 36, 222-233. McNamara, D.S., Ozuru, Y., Graesser, A.C., & Louwerse, M. (2006). Validating Coh-Metrix. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 573-578). Austin, TX: Cognitive Science Society. Miller, G.A. (1968). Response time in man-computer conversational transactions. Proceedings of the AFIPS Fall Joint Computer Conference (pp. 81-97). San Francisco, CA: AFIPS. Millis, K., Magliano, J., Wiemer-Hastings, K., Todaro, S., & McNamara, D.S. (2007). Assessing and improving comprehension with Latent Semantic Analysis. In T. Landauer, D.S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 207-225). Mahwah, NJ: Erlbaum. Nickerson, R.S. (1969). Man computer interaction: A challenge for human factors research. Ergonomics, 12, 510-517. Penumatsa, P., Ventura, M., Graesser, A.C., Franceschetti, D.R., Louwerse, M., Hu, X., Cai, Z., & the Tutoring Research Group (2004). The right threshold value: What is the right threshold of cosine measure when using latent semantic analysis for evaluating student answers? International Journal of Artificial Intelligence Tools, 12, 257-279. Raina, R., Haghighi, A., Cox, C., Finkel, J., Michels, J., Toutanova, K., MacCartney, B., de Marneffe, M-C., Manning, C.D., & Ng, A.Y. (2005). Robust textual inference using diverse knowledge sources. Proceedings of the 1st PASCAL Challenges Workshop (pp.). Stanford, CA: Stanford University. Rehder, B., Schreiner, M.E., Wolfe, M.B., Laham, D. Landauer, T.K., & Kintsch, W. (1998). Using Latent Semantic Analysis to assess knowledge: Some technical considerations. Discourse Processes, 25, 337-354. Rus, V., Lintean, M., McCarthy, P.M., McNamara, D.S., & Graesser, A.C. (2008). Paraphrase identification with lexico-syntactic graph subsumption. In D. Wilson & G. Sutcliffe (Eds.), Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference (pp. 201-206). Menlo Park, CA: The AAAI Press. Rus, V., McCarthy, P.M., Lintean, M.C., Graesser, A.C., & McNamara, D.S. (2007). Assessing student self-explanations in an Intelligent Tutoring System. In D.S. McNamara & G. Trafton (Eds.), Proceedings of the 29th annual conference of the Cognitive Science Society (pp. 623-628). Austin, TX: Cognitive Science Society. Rus, V., McCarthy, P.M., McNamara, D.S., & Graesser, A.C. (in press [a]). Natural language understanding and assessment. In J.R. Rabuñal, J. Dorado, A. Pazos (Eds.). Encyclopedia of Artificial Intelligence. Hershey, PA: Idea Group, Inc. Rus, V., McCarthy, P.M., McNamara, D.S., & Graesser, A.C. (in press [b]). A Study of textual entailment. International Journal on Artificial Intelligence Tools. Sackman, T.R. (1972). Advanced research in online planning. In H. Sackman and R.L. Citrenbaum (Eds.), Online planning: Towards creative problem solving (pp. 3-67). Englewood Cliffs, N.J.: Prentice-Hall. Tversky, A. (1977). Features of similarity. Psychological Review, 84, 327-352. VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A. M., & Rose, C. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3-62. Wiemer-Hastings, P.M. (1999). How latent is latent semantic analysis? In T. Dean (Ed.), Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (pp. 932–941). San Francisco, CA: Morgan Kaufmann Publishers, Inc. Zmud, R.W. (1979). Individual differences and MIS success: A review of the empirical literature. Management Science, 25, 966-975. Download 202.53 Kb. Share with your friends:
|