Conversational Skill User Test

High Level Project Facts:

Research Goal:

  • Evaluate conversational skill designs MVP and understand how current builder role assumptions and responsibilities evolve within the wxO context with Gen AI (e.g. conversational skills) 

My Role: User researcher

Research Methods: User interview with clickable prototype

Background:

watsonx Orchestrate helps enterprises deploy AI assistants, agents and automation across their business. Over the year leading up to this project, the product has integrated with other IBM products such as watsonx Assistant, and other IBM Business Automation solutions. The merging of capabilities has been a challenge to the product team with concerns over the overlap (or lack of) overlap of personas, the information architecture on the product, and user flows between merged capabilities coming up as a challenge.

watsonx Orchestrate features “skills” as units of automation to complete tasks (for example: “create a job requisition” being an example of one skill). watsonx Assistant empowers enterprises to build AI powered chatbots. The product team ideated on what a successful merger between these capabilities would look like and they landed on a “conversational skill” or a capability that enables the power of skills to be interfaced with through chatbot end use experience.

But, as the team ideates this building experience, the question remains of whether the builder experience is usable.

Goals:

Evaluate conversational skill designs MVP and understand how current builder role assumptions and responsibilities evolve within the wxO context with Gen AI (e.g. conversational skills) 5 

Research questions:

Are wxO builders able to create a conversational skill experience without assistance?

What are high and low lights of the design?

What are the roles involved in building an AI assistant within wxO? Where is there handoff? What are the boundaries of these roles? 

Methods and Process:

User interview with clickable prototype

60 minute sessions consisting of:

  1. Introductory questions on their role and experience with wxO and wxA

  2. Participant click through of conversational skill MVP 1 prototype. They were asked to think aloud and asked questions throughout the experience

  3. Concluding questions on their experience and topics like builder roles, adding credentials, and value

    Scenario: You need to make a chatbot that helps managers transfer employees. The API specifications have been made for you - your goal is to help users trigger and interact with it within a chatbot conversation 

Participants:

The study included 8 participants who were current skill or AI Assistant builders and were interested in integrating assistants with their applications for digital copilot and transactional experiences. 

Recruiting was done in collaboration with product managers and tech sellers for wxO.

Analysis and Results

Key high level findings included:

  • Builders missed key tasks along the creation process that would result in the conversational skill because of content and lack of guidance to other areas in the product 

  • The concept of a “conversational skill” was confusing to participants, making them think it was a different type of skill 

  • Initial findings show use cases where there may be 2 builders involved in the end to end flow for creating a conversational skill dependent on ways a team may organize and their team’s technical composition