July 2026
In 2024 and 2025, MDRC partnered with SkillUp Coalition, a nonprofit organization that helps workers without bachelor’s degrees: The goal was to test whether a chatbot that used AI could lead workers to better jobs or training. The study team randomly assigned more than 5,000 job seekers to have access to the chatbot or to SkillUp's existing website. The team then tracked whether chatbot users were more or less likely to receive recommendations and explore training and job opportunities.[1] It found they were less likely to explore and act on opportunities than users of the standard website.
Since this evaluation, AI tools have advanced beyond chatbots and SkillUp has enhanced the AI tools available on its site. Yet as workforce-development organizations and community colleges increasingly use AI tools for career advising and coaching, evaluations of early-stage AI tools like this one can still help them understand what works and what does not.
- Be thoughtful about when and where AI tools are used. While using a chatbot to provide information can feel intuitive, it may instead add friction, especially when a simpler solution—like a well-designed website that makes browsing and finding information easy—already exists. A critical first step is to consider when AI can help solve a problem.
- Design tools that provide usable information and advice quickly. In this test, chatbot users had to send several messages containing their work history, location, and skills before they received a job or training recommendation, and some chatbot responses were delayed. Both experiences probably led users to stop using the tool. Designing tools to provide information more quickly can help keep users engaged.
- Help job seekers use AI tools effectively. Users did not always know what questions to ask, or had to type long responses to continue interacting with the chatbot. SkillUp staff members said they may have underestimated how hard this style of interaction would be for some users. Having human advisers and coaches use AI tools with job seekers can help them learn how to use and trust in those tools.
- Use AI to help with early exploration, rather than more targeted searches for specific jobs or training. Chatbot users tended to disengage when they had to make difficult or high-stakes decisions, such as determining whether a career fit or deciding whether to pursue a training opportunity. Rather than treating AI as a replacement for human advising, provide AI tools that job seekers can use as a first step and ensure they can ask human beings questions afterward.
AI tools have the potential to help with career navigation. But for this potential to be realized, workforce and higher education organizations need to learn from the successes and failures of tools that have been pilot tested. Ongoing assessment of new tools that truly meet job seekers’ needs is also vital. For now, lessons from these early tests can help organizations tailor AI tools to job seekers’ needs. A brief with lessons from SkillUp’s journey to integrate other AI tools into its website will be released in September 2026.
[1] The lessons in this document are drawn from Bennett Otten and Kelsey Schaberg, Using AI to Help Job Seekers: Lessons from SkillUp AI (MDRC, 2025). See that brief for more information on the evaluation and findings.
