Test the Prototypes

Prototypes of current solutions have been developed to better understand the technical levers that can be used to create a Design Thinking experience. These prototypes will be tested with users who have been pre-determined to be interested in the topic.

To learn about my approach, initial learnings and next steps, continue reading below.

My Approach

Education

To fully comprehend the impact of Generative AI on businesses, I recognized the need to upskill my knowledge. As a visual learner, I prefer learning from industry leaders like Reid Hoffman and YouTube sources like AI Explained, which provide perspectives and opinions on the topic.

I sought insights from interviews with CEOs such as Lex Fridman's interview with OpenAI CEO Sam Altman and The Verge's interview with Microsoft CEO Satya Nadella, where they discuss their organizations' vision and how AI will play a role in their decisions.

Finally, I dove into the technology itself through videos from IBM Technology, Amazon, and various YouTubers to understand how it works and how it can influence product design decisions.

This comprehensive approach has allowed me to develop a deep understanding of Generative AI and its potential impact on businesses.

Inspirations

Upon deepening my understanding of Generative AI through research and education, I sought out real-life examples of its application to gain insights and inspiration.

In particular, I was interested in examining products that went beyond the traditional AI and chatbot experience that people are commonly familiar with and wanted to identify solutions that acted more as coaches than tools.

Through this exploration, I discovered Khan Academy's Khanmigo and Inflection AI's personal AI Pi, both of which help users understand new topics and break down challenges in a personalized and engaging manner.

By closely analyzing these products, I was able to develop a hypothesis for creating similar experiences and gain a deeper understanding of the underlying technology behind Generative AI.

This mental model not only allowed me to appreciate the transformative potential of Generative AI but also provided me with valuable insights that I could apply to my work as a venture architect and product designer.

Inspirations Khanmigo (left) and Pi (right)

Hypothesis

Recognizing my technical limitations, I understood that exploring how to train a model was beyond my skill set. Instead, I decided to focus on understanding how to leverage an existing foundational model, which is likely what most businesses will use. This involves identifying the product decisions that create specific experiences based on desired business outcomes, user experience, and technical costs.

To achieve this, I delved into prompt engineering, tuning, and system settings, such as temperature. Additionally, I explored what constitutes as "quality data" in the context of generative AI to understand how each of these factors interact with each other and how they impact the experience. Through this process, I honed my understanding of the technology and developed a list of "How Might We" statements to help me begin prototyping ideas to test.

How Might We’s

How might we design and engineer prompts to improves outputs while not causing additional overhead that can lead to systems timing out?
–
How might we tweak system settings to provide a “coach like” experience on a specific topic?
–
How might we provide a foundational model user specific context to improve it’s outputs?
–
How might we utilize prompts approaches like few shot, “step by step”, and “tree of thought” to improve outputs?

Experimenting with Voiceflow

To begin prototyping some ideas, I searched for existing solutions that would enable me to explore some of my initial hypotheses and how-might-we's. Attempting to avoid building anything from scratch, I found two solutions that, while not perfect, met the requirements needed to prototype a couple of different solutions to test.

Initial Learnings

Existing solutions like Voiceflow and MyAskAI have been useful tools for experimenting with prompt design and engineering and understanding product and architectural design decisions needed to create a Design Thinking assistant experience. However, they have limitations, particularly in multi-question flows where token and prompt settings can cause systems to time out and freeze. This emphasizes the importance of understanding the constraints of both the foundational model and the solution, and how to navigate these constraints while still delivering something of value to users.

Interestingly, I discovered that I could extract knowledge from products like Inflection AI's Pi on its tone and behaviors to help create a more personalized and engaging experience. I then took this conversation with Pi and brought it into ChatGPT to help inform a system prompt that could create a more “coach-like” experience. While this approach is not perfect due to the limitations of the solution, it has provided valuable insights for further exploration.

Outcomes as of July 2023

As more organizations experiment with Generative AI technologies, it's becoming increasingly easier to understand what makes an experience feel personalized and “magical” and a solution that users will love. Some initial takeaways and insights from my initial experiments that will inform next steps:

In addition to empathizing with users, you must also understand how to train, tune, and prompt a foundational model to ensure you’re creating a valuable assistant or complementary feature for a given solution.
–
Identifying the type of data to feed into the model and understanding how that data is being translated is crucial for knowing how to focus a foundational model and deliver a personalized experience.
–
Building an effective user interface that gathers additional context to feed back to the model and displays outputs back to the user in a clear and concise manner is important.
–
Leveraging complementary technologies such as search or relative content as part of the output can make the experience even more engaging and effective.

Through my exploration of existing solutions and my initial prototypes, I have gained valuable insights that I will use to test and iterate my ideas. Experimenting with prompt design and engineering within the context of the current solutions will also be a crucial step in creating a Generative AI experience that is both personalized and engaging.

Next Steps

Testing existing prototypes with real users to understand what is missing, what kind of questions they’re hoping to ask, and how “coach-like” the experience feels.

Solution 1 (Voiceflow)– further experimentation with prompts and conversational workflows to create a more reliable experience.

Solution 2 (MyAskAI)– further training of the model based on user questions asked that it cannot currently answer to provide a more a more expansive dataset.

Exploring and understanding the relationship between engineering and product design in the context of Generative AI.