The Technology for Effective and Efficient Learning (TEEL) Lab at Carnegie Mellon University focuses on research in learning methods, technology for learning systems, curriculum development, and workforce training.

About TEELLab

 

The Technology for Effective and Efficient Learning (TEEL) Lab at Carnegie Mellon University focuses on research in learning methods, technology for learning systems, curriculum development, and workforce training. The ultimate goal is to forge a path for lifelong learning and career ever-readiness through innovative bridging of learning between formal education, informal education, and the workplace.

The TEEL Lab develops learning platforms that support learning through discussion and collaboration, focusing on hands-on technology areas such as cloud computing, machine learning, and data science. The target interoperable online learning ecosystem under development by the team is designed to enable effective and efficient learning through iterative and interactive learning coupled with contextualized and timely feedback, and by leveraging social interactions between students as a substantial learning resource.

The lab conducts research studies on student learning and evaluates innovative approaches for incorporating social and interactive learning as a driver for developing cognitive and meta-cognitive skills and motivation. Its work is based on research spanning more than a decade in Computer Science Education, Software Engineering, Learning Sciences, and Language Technologies.

In addition to scientific findings, technical advances, and principles for instructional design, the team’s past research has also produced open source resources to share with the broader community. Specific features of the in-progress LMS-pluggable online course infrastructure is that it enables project-based learning through auto-graded projects that provide contextualized feedback. We also incorporate social learning in the form of project reflection and feedback cycles after each project, timed and role-based online learning exercises and rubric-driven peer code-review.

Members of the TEEL Lab collaborate with other groups at CMU, other universities, community colleges as well as industry. Find out how you can get involved from our Call for Participation.

Funded in part by a grant from Microsoft and grants NSF grants ACI-1443068, IIS 1822831 and IIS 1546393.

 

Foundational Research

 
  1. Chris Bogart, Marshall An, Eric Keylor, Pawanjeet Singh, Jaromir Savelka, and Majd Sakr. 2024. What Factors Influence Persistence in Project-Based Programming Courses at Community Colleges? In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024), March 20–23, 2024, Portland, OR, USA. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3626252.3630965
  2. Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, and Majd Sakr. 2024. A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education. In Proceedings of 26th Australasian Computing Education Conference (ACE ’24). ACM, New York, NY, USA, 10 page.
  3. Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023). From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-choice Questions for Programming Classes in Higher Education. arXiv preprint arXiv:2311.09518.
  4. McLaren, B.M., Herckis, L., Teffera, L., Branstetter, L., Rose, C.P., Sakr, M., Kisow, M., Reis, R., Rinsem, M., Alenius, M., & Miller, L. (2024). Thrilled by Community College Information Technology Education: Curriculum Mapping, a Learning Science Framework, and AI Learning Technologies. To be presented at AERA 2024, the 2024 Annual Meeting of American Educational Research Association (AERA). Philadelphia, PA April 11-14, 2024
  5. Savelka, Jaromir, Arav, Agarwal, Marshall, An, Chris, Bogart, Majd, Sakr. (2023) Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1. Association for Computing Machinery
  6. Sridhar, Pragnya, et al. (2023) Thrilled by Harnessing llms in curricular design: Using gpt-4 to support authoring of learning objectives. arXiv preprint arXiv:2306.17459
  7. Savelka, J., Agarwal, A., Bogart, C., Sakr, M. (2023) Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code International Conference on Computer Supported Education (CSEDU).
  8. Savelka, J., Agarwal, A., Bogart, C., Song, Y., Sakr, M. (2023) Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses? Conference on Innovation and Technology in Computer Science Education (ITiCSE).
  9. Bhat, S., Nguyen, H., Moore, S., Stamper, J., Sakr, M., Nyberg, E. (2022). Towards Automated Generation and Evaluation of Questions in Educational Domains International Conference on Educational Data Mining (EDM)
  10. S. Sankaranarayanan, S. R. Kandimalla, C. Bogart, R. C. Murray, M. Hilton, M. Sakr, C. P. Rosé (2022). Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice With Example-Based Reflection for Student Learning and Transfer Task Performance IEEE Transactions on Learning Technologies, vol 15, no. 5, pp. 594-604, 2022.
  11. Sreecharan Sankaranarayanan, Lanmingqi Ma, Siddharth Reddy Kandimalla, Ihor Markevych, Huy Nguyen, R. Charles Murray, Christopher Bogart, Michael Hilton, Majd Sakr and Carolyn Rose (2022). Collaborative Reflection “in the flow” of Programming: Designing Effective Collaborative Learning Activities in Advanced Computer Science Contexts. International Society of the Learning Sciences Annual Meeting (ISLS2022)
  12. Jiameng Du and Yifan Song, Mingxiao An, Marshall An, Christopher Bogart and Majd Sakr, (2022). Cheating detection in online assessments via timeline analysis. Special Interest Group on Computer Science Education Conference (SIGCSE), Providence, RI. - 🏆 Best Paper Award
  13. Asano, Y., Sankaranarayanan, S., Sakr, M., Bogart, C. (2021). A Thematic Summarization Dashboard for Navigating Student Reflections at Scale. International Conference on Computers in Education (ICCE).
  14. Sreecharan Sankaranarayanan, Siddharth Reddy Kandimalla, Christopher Bogart, R Charles Murray, Michael Hilton, Majd Sakr, Carolyn Rosé, (2021). Combining Collaborative Reflection based on Worked-Out Examples with Problem-Solving Practice: Designing Collaborative Programming Projects for Learning at Scale. Proc. Conf. Learning@Scale, pp. 255-258.
  15. Nguyen, H. A., Lim, M., Moore, S., Nyberg, E., Sakr, M., Stamper, J. (2021). Exploring Metrics for the Analysis of Code Submissions in an Introductory Data Science Course. Learning Analytics and Knowledge Conference.
  16. An, M., Zhang, H., Šavelka, J., Zhu, S., Bogart, C., Sakr, M. (2021). Are Working Habits Different Between Well-Performing and at-Risk Students in Online Project-Based Courses? Conference on Innovation and Technology in Computer Science Education (ITiCSE).
  17. Sreecharan Sankaranarayanan, Siddharth Reddy Kandimalla, Christopher Bogart, R Charles Murray, Haokang An, Michael Hilton, Majd Sakr, and Carolyn Rosé. (2021). Comparing Example-Based Collaborative Reflection to Problem-Solving Practice for Learning during Team-Based Software Engineering Projects. International Society for the Learning Sciences.
  18. Adams, J., Hainey, B., White, L., Foster, D., Hall, N., Hills, M., … Taglienti, C. (2020). Cloud Computing Curriculum: Developing Exemplar Modules for General Course Inclusion. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 510–511.
  19. Sankaranarayanan, S., Kandimalla, S. R., Cao, M., Maronna, I., An, H., Bogart, C., … Rosé, C. P. (2020). Designing for learning during collaborative projects online: tools and takeaways. Information and Learning Sciences, 121(7/8), 569–577.
  20. Sankaranarayanan, S., Kandimalla, S. R., Hasan, S., An, H., Bogart, C., Murray, R. C., … Rosé, C. (2020). Agent-in-the-loop: conversational agent support in service of reflection for learning during collaborative programming. International Conference on Artificial Intelligence in Education, 12164, 273–278.
  21. Sankaranarayanan, S., Kandimalla, S. R., Hasan, S., An, H., Bogart, C., Murray, R. C., … Rose, C. (2020). Creating opportunities for transactive exchange for learning in Performance-Oriented team projects. International Conference of the Learning Sciences (ICLS), 3, 1719–1720.
  22. Song, X., Goldstein, S. C., & Sakr, M. (2020). Using Peer Code Review as an Educational Tool. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 173–179.
  23. Rosé, C. P., Murray, R. C., Wang, Y., & Bao, H. (2020). Agent-Based Dyanmic Collaboration Support in a Smart Office Space. Special Interest Group on Discourse and Dialogue (SIGDIAL), 257–260.
  24. Foster, D., White, L., Erdil, D. C., Adams, J., Argüelles, A., Hainey, B., … Stott, L. (2019). Toward a cloud computing learning community. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 143–155.
  25. Goldstein, S. C., Zhang, H., Sakr, M., An, H., & Dashti, C. (2019). Understanding how work habits influence student performance. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 154–160.
  26. Sankaranarayanan, S., Dashti, C., Bogart, C., Wang, X., Marshall An, Clarence Ngoh, Michael Hilton, Sakr, M., Rosé, C. (2019). Transactivity-Based Team-Formation as a Choice: Evaluation in a Large Online Project Course. 7th International Collective Intelligence Conference.
  27. Sankaranarayanan, S., Wang, X., Dashti, C., An, M., Ngoh, C., Hilton, M., … Rosé, C. (2019). An Intelligent-Agent Facilitated Scaffold for Fostering Reflection in a Team-Based Project Course.. International Conference on Artificial Intelligence in Education, 11626, 252–256.
  28. Sankaranarayanan, S., Wang, X., Dashti, C., An, M., Ngoh, C., Hilton, M., … Rosé, C. P. (2019). Online mob programming: bridging the 21st century workplace and the classroom. Proceedings of Computer-Supported Collaborative Learning, 2, 855–856.
  29. Fiacco, J., & Rosé, C. (2018).  Towards domain general detection of transactive knowledge building behavior. Proceedings of Conference on Learning at Scale (L@S), Article No. 8, 1-11
  30. Foster, D., White, L., Adams, J., Cenk Erdil, D., Hyman, H., Kurkovsky, S., … Stott, L. (2018). Cloud computing: Developing contemporary computer science curriculum for a cloud-first future.Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 130–147., pp. 130-147. ACM, 2018.
  31. Wen, M., Maki, K., Dow, S. P., Herbsleb, J., Rosé, C. P. (2018). Supporting Virtual Team Formation through Community-Wide Deliberation, in Proceedings of the 21st ACM Conference on Computer-Supported Cooperative Work and Social Computing
  32. Sankaranarayanan, S., Dashti, C., Bogart, C., Wang, X., Sakr, M., & Rosé, C. P. (2018). When optimal team formation is a choice - Self-selection versus intelligent team formation strategies in a large online project-based course., in International Conference on Artificial Intelligence in Education (AIED), 10947, 518–531.
  33. Rosé, C. P. (2017).  Expediting the cycle of data to intervention, Learning: Research and Practice 3(1), special issue on Learning Analytics, pp 59-62
  34. Wang, X., Wen, M., Rosé, C. P. (2017).  Contrasting Explicit and Implicit Support for Transactive Exchange in Team Oriented Project Based Learning, Proceedings of Computer-Supported Collaborative Learning
  35. Rosé, C. P. & Ferschke, O. (2016).  Technology Support for Discussion Based Learning: From Computer Supported Collaborative Learning to the Future of Massive Open Online Courses, International Journal of AI in Education, 25th Anniversary Edition, volume 26(2), pp 660-678
  36. Howley, I. & Rosé, C. P. (2016).  Towards Careful Practices for Automated Linguistic
  37. Analysis of Group Learning, Journal of Learning Analytics 3(3), pp 239-262.
  38. Tomar, G., Sankaranaranayan, S., Wang, X., Rosé, C. P. (2016).  Coordinating Collaborative Chat in Massive Open Online Courses, in Proceedings of the International Conference of the Learning Sciences
  39. Nguyen, A. T., Hilton, M., Codoban, M., Nguyen, H. A., Mast, L., Rademacher, E., ... & Dig, D. (2016, November). API code recommendation using statistical learning from fine-grained changes. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 511-522). ACM.
  40. Rehman, S., Boles, J., Hammoud, M. and Sakr, M. (2015). A Cloud Computing Course: From Systems To Services, Proceedings of the 46th ACM Special Interest Group on Computer Science Education Conference (SIGCSE 2015), Kansas City, USA, March 2015.
  41. Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121.
  42. 15. Kumar, R. & Rosé, C. P. (2014). Triggering Effective Social Support for Online Groups. ACM Transactions on Interactive Intelligent Systems 3 (4).
  43. Dyke, G., Adamson, A., Howley, I., & Rosé, C. P. (2013). Enhancing Scientific Reasoning and Discussion with Conversational Agents, IEEE Transactions on Learning Technologies 6(3), special issue on Science Teaching, pp 240-247.
  44. Kumar, R. & Rosé, C. P. (2011).  Architecture for building Conversational Agents that support Collaborative Learning, IEEE Transactions on Learning Technologies, 4(1), pp 21-34

Posters

 
  1. Jaromír Šavelka, Marshall An, Chris Bogart, Majd Sakr. (2022). Measuring the Effectiveness of Rapid Occupational Training Methods in Emerging Technology Areas. Carnegie Mellon University Teaching & Learning Summit, The Eberly Center for Teaching Excellence and Educational Innovation, Pittsburgh, USA
  2. Chris Bogart, Jeremy Hoffman, Pawanjeet Singh, Jaromír Šavelka, Yufan Zhang, Marshall An, Divya Prem, Majd Sakr. (2022). Causes and effects of low self-efficacy in community college IT classrooms. Carnegie Mellon University Teaching & Learning Summit, The Eberly Center for Teaching Excellence and Educational Innovation, Pittsburgh, USA
  3. Song, E., Goldstein, S., Sakr, M., An, M., & Kandimalla, S. (2019). Online Code Review: an assessment of student engagement in peer code review. Carnegie Mellon University Teaching & Learning Summit, The Eberly Center for Teaching Excellence and Educational Innovation, Pittsburgh, USA
  4. Sankaranarayanan, S., Dashti, C., An, M., Wang, X., Rosé, C., Sakr, M., & M. Hilton. (2018). Online Mob Programming: A Collaborative Project-Based Learning Intervention. Carnegie Mellon University Teaching & Learning Summit, The Eberly Center for Teaching Excellence and Educational Innovation, Pittsburgh, USA
  5. Sakr, M., Zhang, H., Dashti, C. & An, M. (2018). Relationship between consistency of student behavior and their performance in an online course. Carnegie Mellon University Teaching & Learning Summit, The Eberly Center for Teaching Excellence and Educational Innovation, Pittsburgh, USA

Presentations and Workshops

 
  1. Marshall An, Adam Zhang, Divya Prem, Jaromir Savelka, Chris Bogart, Majd Sakr (2022). Professional development workshop for college instructors National Science Foundation Program: Improving Undergraduate STEM Education (IUSE), July 17-22, 2022, Pittsburgh, Pennsylvania, USA
  2. Majd Sakr, Marshall An, Norman L Bier (2022). Improving teaching effectiveness with project-based learning (PBL) and data-informed teaching (DIT) 2022 IUSE Summit: Propelling Change: Moving from Strategy Toward Effective & Equitable Undergraduate STEM Education, June 1-3, 2022, Washington, D.C., USA
  3. Majd Sakr, Marshall An (2022). Improving teaching effectiveness with project-based learning (PBL) and data-informed teaching (DIT) May 30, 2022, NISOD International Conference on Teaching and Leadership Excellence, Austin, USA
  4. Majd Sakr, Marshall An, David R. Galbreath, Lee Stott (2021) Sharpen Your Cloud Skills with Carnegie Mellon University and Microsoft Learn Feb 15, 2021, Microsoft Learn Live, Livestream, Global

Faculty Team

Teaching Professor, Computer Science Department

View Website

Majd Sakr

Assistant Teaching Professor, Institute for Software Research

View Website

Michael Hilton

Professor, Language Technologies Institute and Human-Computer Interaction Institute

View Website

Carolyn Rosé

Associate Professor, Computer Science Department

View Website

Seth Goldstein

Systems Scientist, Institute for Software Research

View Website

Chris Bogart

Research Associate, Computer Science Department

View Website

Jaromir Savelka

Research & Curricula AI Technicians, Computer Science Department

View Website

Can Kultur

Curricula ML Applications, Computer Science Department

View Website

Eric Keylor

Project Members

Lab Director

View Website

Heather Burte

Data Scientist

View Website

Adam Zhang

Engineering Lead

View Website

Divya Prem

Full Stack Engineer

Halim-Antoine Boukaram

Full Stack Engineer

View Website

Baljit Singh

Project Scientist

View Website

Arav Agarwal

Frontend Developer

View Website

Hanzhi Yin

UX Engineer

View Website

Aparna Maya Warrier

Full Stack Developer

View Website

Zhihan Zhang

Alumni

Project Scientist/p>

View Website

Marshall An

PhD student, Language Technologies Institute

View Website

Sreecharan Sankaranarayanan

Project Scientist

View Website

Siddharth Kandimalla

Project Administrator

View Website

Cait Frazier

Learning Engineer

View Website

Michelle Fernandez


Resources

Sail()

Sail() is an online learning platform that provides college and university instructors with job-focused technology courses created at Carnegie Mellon University that are project-based, collaborative, and use real-time feedback. Sail() offers content and scaffolded activities within the Open Learning Initiative (OLI). Our platform offers auto-graded projects with opportunities to iterate by providing timely feedback, social learning reflection and feedback cycles for each project, timed and role-based online group learning exercises, and rubric-driven peer code-review activities. The Sail() Platform enables evidence-based learning tools for instructors and researchers. Instructors can use the detailed dashboard to visually reflect on various aspects of student learning. Researchers can instrument and run various learning experiments to explore learning methods.

DiscourseDB

DiscourseDB is an NSF funded data infrastructure project designed to bridge data sources from multiple platforms for hosting those learning experiences. Our vision is to provide a common data model designed to accommodate data from diverse sources including but not limited to Chat, Threaded Discussions, Blogs, Twitter, Wikis, and Text messaging. We will make available analytics components related to constructs including role taking, help exchange, collaborative knowledge construction, showing openness, taking an authoritative stance, attitudes, confusion, alliance and opposition. In enabling application of such metrics across datasets from multiple platforms, research questions related to the mediating and moderating effect of these process and state measures on information transfer, learning, and attrition can be conducted, building on the earlier research of our team.

Bazaar

Bazaar is a publicly available architecture for orchestrating conversational agent-based support for group learning. It is a powerful tool for facilitating research in collaborative learning. It hosts a library of reusable behavioral components that each trigger a simple form of support. More complex supportive interventions are constructed by orchestrating multiple simple behaviors. Its flexibility and simplicity mean it can be used to very rapidly develop platforms for investigating a wide range of important questions within the design space of dynamic support for collaborative learning.

LightSide

The open-source LightSide platform, including the machine-learning and feature-extraction core as well as the researcher's workbench UI, has been and continues to be funded in part through Carnegie Mellon University, in particular by grants from the National Science Foundation and the Office of Naval Research. See the full acknowledgments and grant details below! We make the LightSide research platform freely available for research and education. In exchange, we ask that you provide us with basic information about who you are and how you're making use of LightSide's capabilities.

Social Recommendation

Massive Open Online Courses have experienced a recent boom in interest. Problems students struggle with in the discussion forums, such as difficulty in finding interesting discussion opportunities or attracting helpers to address posted problems, provide new opportunities for recommender systems. We developed a social recommendation technology to support help seekers in MOOC discussion forums implemented using a context-aware Matrix Factorization model to predict students' preferences for answering a given question. This recommendation framework allows for this two-way recommendation. .

Collaboration

 

The TEEL Lab at Carnegie Mellon University is excited to issue a call for collaboration on the development and deployment of our SAIL Platform and Curriculum. We plan to collaborate with faculty at community colleges on offering a variety of hands-on project-based courses as part of community college course offerings. We focus on curricula, developed by domain experts, that are targeted to the needs of a variety of learners to prepare them for jobs in the domains of cloud computing, machine learning, and data science, with the goal of helping meet the expected job growth in these fields. We prepare learners for these domains by developing foundational skills and providing opportunities to exercise industry’s best practices. This collaboration is meant to influence both the curricular content and its delivery while leveraging the expertise of faculty at community colleges in order to best meet the needs of our target learners in terms of career preparedness and the ability to cope with life-challenges that impact the learners’ ability to engage in a sustained upskilling effort. Mentorship and professional development in the domain will be made available to collaborating faculty.

Our SAIL platform offers content and scaffolded activities within the Open Learning Initiative (OLI) that integrate with Learning Management Systems that are already in use at community colleges. The design builds on an empirical foundation and software infrastructure emerging from our experience of offering online and hands-on project-based courses since 2013 to Carnegie Mellon's global campuses. Our platform offers learn-by-doing projects with scaffolding and opportunities to iterate by using auto-graded projects that provide timely and contextualized feedback. We incorporate social learning reflection and feedback cycles for each project, timed and role-based online group learning exercises and rubric-driven peer code-review opportunities. All of these are industry-standard job-relevant activities further reinforce the connection between the course content and software engineering best practices.

The TEEL Lab hopes to partner with community colleges to collaboratively adapt our existing curricular content to create new offerings or augment existing classes within the community college context. The SAIL Platform and collaboration is an effort to evaluate the efficacy of our innovative learning methods. We also are interested in research endeavors to investigate how to incorporate upskilling as part of a learner's current job.

Call to action:

  • Inform us of your interest in this collaboration
  • Identify existing programs that our initial cloud computing course offering fits within
  • Supply a list of faculty who would be interested in participating in this collaboration
  • Work through the IRB process to enable research towards continual improvement of the platform from student data

 

Positions

TEEL Lab Internships The TEEL Lab has several positions available for interns (in-person and remote). We have three types of roles we are looking to fill:
• Infrastructure development: writing and refactoring cloud apps, containerization, API development, data processing and analysis, front-end development
• Course development: writing content and auto-gradable assignments for courses related to cloud programming, Cloud DevOps, and data science/engineering
• Research: data mining and/or user centered design aimed at improving learning in online courses
Interested individuals should submit their CV here.
Full Stack Engineer, Computer Science Dept • Job Number: 2018961
The Full-Stack Engineer role includes development, maintenance and productization of large-scale Django and SpringBoot applications as well as the design and implementation of new, containerized microservices. The ideal candidate is passionate about owning, building and operating robust and cutting-edge cloud-native microservices. The candidate is detail-oriented and keen on continuously learning, being responsive, and exhibiting effective communication with team members. Learn more and apply here.
Learning Engineer / Content Developer, Computer Science Dept • Job Number: 2018983
The TEEL Lab is seeking a Learning Engineer/Content Developer for introductory CS courses at the community college level. You will work closely with CMU faculty to develop course content and activities to be deployed on our online learning platforms. This position involves being responsible for designing, authoring, testing, and reviewing course content in the relevant areas. You'll also learn how to develop conceptual content aligned with a set of hands-on projects. This is an excellent opportunity for someone to enhance their expertise in teaching introductory Computer Science topics. Learn more and apply here.

Get in touch

Email

teel@andrew.cmu.edu