Case Study: Adoption of New LMS Assessment Tool in a Technical Training Organization
Introduction
In this simulated case study, a training organization at a tech company will be adopting a new Learning Management System (LMS) in the coming months. The training organization in this scenario creates and delivers assessments using a native LMS assessment tool. In this case study, I strategically analyze the learning needs for various target stakeholders to ensure successful adoption and utilization of the new LMS assessment tool.
Note: All text included in the case study was written by me. No AI was used except for code blocks on this page.
High-Level Needs Analysis
Problem Statement
A training organization at a tech company has been using a LMS 1.0 to administer assessments to learners in its certification program for about five years. The training org uses LMS 1.0 to produce, manage, and deliver high-stakes final exams in addition to informal, formative quizzes. “LMS 2.0” will be adopted in approximately nine months; LMS 1.0 will be retired after migration is complete. A solution is needed to ensure critical stakeholders are adequately prepared for assessment creation, production, and delivery using the new LMS to avoid gaps in content development and assessment administration.
Impacted Stakeholders
| Stakeholder | Approx. Number | Role | Primary Concerns/Issues |
|---|---|---|---|
| Instructional designers | 12 | Develop assessments – use AI to generate item content based on training materials | Available item types; how to generate item content that is aligned with the new LMS |
| Subject Matter Experts (SMEs) | 15–20 | Review generated item content and provide feedback | Accuracy of assessment content |
| Production Team | 5 | Manage the production of assessments within the LMS assessment tool; make revisions to items and exams/quizzes | Assessment production processes; management of item and exam assets; item and test properties; asset generation, modification, and archiving |
| LMS Admin Specialist | 1 | Posts assessment content to correct locations in the LMS | Process for creating and managing assessment files; posting assessments to appropriate courses and locations in the LMS |
| Instructors | 50 | Deliver courses and assessments to learners | How to share assessments with learners |
| Test Takers | 1000+ | Take quizzes and tests using the LMS | Having a smooth, issue-free test-taking experience |
LMS Learning Solution Strategy
Overall Strategy – Focus on Change Management
The strategy for ensuring effective adoption of the new LMS by the impacted stakeholders should take into account ADKAR, which is a central tenet of Prosci Change Management.
In order to assist diverse stakeholders through a system change, change management is required to address each of the different stages of ADKAR. The various stakeholders will likely progress through ADKAR at different speeds and with different considerations.
ADKAR Plan
As soon as possible, share information about LMS 2.0 during team meetings where stakeholders will be in attendance. Explain what the change is and how it will impact them. (Awareness)
Administer a short Microsoft Forms survey to gauge awareness of the new LMS. (Awareness)
At later team meetings, present some examples of how LMS 2.0 will enhance the assessment creation, management, and delivery experience. Provide concrete examples of improvements to show how the new LMS will benefit the stakeholders (e.g., more efficient creation, better delivery experience). Have the executive sponsor record a short video or speak during an all-hands meeting discussing the benefit the new LMS will bring to the training org. (Desire)
Administer a short Microsoft Forms survey to measure attitudes towards the new LMS; confirm commitment to adopting the new LMS. Allow respondents to share questions and feedback about the upcoming change. (Desire)
Conduct small-group live sessions (recorded) to share processes and how-to’s to help build knowledge of various stakeholders. Develop process documents and job aids to support stakeholder knowledge and need for information. (Knowledge)
Administer a short Microsoft Forms survey to query stakeholders on their confidence in their knowledge of LMS processes and workflows. Allow respondents to share questions and feedback. (Knowledge)
Once different stakeholders begin interacting with the new LMS and completing role-specific activities related to assessments, schedule periodic “office hours” (e.g., bi-weekly) to allow stakeholders to ask questions and discuss challenges. (Ability)
Create a mini-site that highlights some accomplishments from different stakeholders in order to celebrate their wins during the adoption process. Include quotes and testimonials from managers and executive sponsor praising individual stakeholders and teams. (Reinforcement)
Deliverables
| Deliverable | Format | Purpose | Impacted Stakeholder(s) |
|---|---|---|---|
| LMS 2.0 ID Assessment Playbook | Word .docx/PDF | Job aid explaining how to create assessment content that is aligned with LMS 2.0 capabilities, assessment development best practices, and organization specifications. Covers item types and functionalities, affordances and limitations, and considerations regarding item and exam management. | Instructional designers, LMS Admin Specialist |
| Train-the-Trainer LMS 2.0 Web-Based Course | Asynchronous self-paced online course; produced using Storyline and delivered through the LMS | Overview of the LMS 2.0 assessment tool, benefits, new functionalities; how to access and share exams and quizzes with students; how to run analytics reports for scores and performance data; how to request report access | Instructors delivering assessments to students |
| LMS 2.0 Production Job Aid | Word .docx/PDF | Document assessment production instructions, including uploading/creating items, creating tests, modifying test properties, archiving items, managing item and exam assets | Production team |
| SME LMS 2.0 Assessment Overview | PowerPoint slides | A quick guide to the LMS 2.0 assessment capabilities including item types and any AI-related enhancements; geared towards educating Subject Matter Experts (SMEs) about the content capabilities of LMS 2.0 assessments | Subject Matter Experts (SMEs) |
| LMS 2.0 Exam Demo | LMS 2.0 assessment (ungraded); included as an optional activity before every exam | Demonstrate how to navigate exams and answer different item types in an ungraded, low-risk environment | Test takers |
Evaluation
How will we know if the stakeholders learned the skills needed to successfully adopt the LMS 2.0 assessment tool for their job roles?
Qualitative Surveys – Administer short surveys targeted to stakeholder groups asking the degree to which respondents are knowledgeable and confident with using different assessment functionalities and developing items. Include open-ended questions so respondents can provide details about challenges and insights.
Communication Data – Track queries via email and chat from stakeholders regarding difficulties and challenges using the LMS 2.0 assessment tool. Maintain a spreadsheet to quantify how many queries are being sent from which stakeholders. Track responses and whether further training and documentation are needed.
Productivity Data – Track work tickets related to assessment creation and production in ticket software (e.g., Jira). Create a report showing the number of assessment tickets that have been processed over the year. Identify any trends showing bottlenecks or indication of challenges. Investigate any issues if applicable.
AI Enhancements – AI tools such as Claude and Copilot can be used to quickly collect data and extract insights. Connections between AI tools and email, chat, and ticket tracking software can expedite the flow of information and allow for easier analysis of data.
Case Study Conclusion
In this case study, I presented a scenario in which a technical training organization is adopting a new LMS. The focus of the case study was the LMS assessment tool and its impact on stakeholders including instructional designers, Subject Matter Experts (SMEs), instructors, production contacts, LMS admin, and test takers. By using ADKAR, a Prosci Change Management tool, the team is able to manage the adoption process and identify areas where formal learning content is appropriate. In other cases, communications during group meetings and documentation served as instructional opportunities. I also highlighted some suggested activities to evaluate the success of the adoption and the stakeholders’ skill and confidence.