ChatGPT Comedy Hour
Machine learning Play/Games Product design

Human Centered Evaluation of ChatGPT as a Casual Creator Tool

ChatGPT Comedy Hour is a research project that utilizes AI to study casual and technology assisted creativity. Using research methods such as surveys and interviews, I validated the characteristics that are said to be necessary in software for fun and relaxed creativity, along with other interesting thought patterns related to ChatGPT.

Student

Cat Dinh

Advisor

Juliette Powell

Abstract

In her 2015 dissertation, Kate Compton defined the genre of Casual Creators, a type of software that empowers anyone (even if they are not self proclaimed creatives) to tap into their creative side in a casual, low stakes environment. Compton sets up a framework that this type of software is most effective when it uses the right combination of “automation, generativity, surprise, and social/communal features”. In my thesis, I decided to validate and dig deeper into this framework in the context of ChatGPT. I devised a Jackbox style game in which I gave players an open ended brief, such as “Write a haiku”. It was up to them to put their own spin on it to prompt ChatGPT for an answer, such as “Write a silly haiku about how I saw my Grandpa kissing the tooth fairy on Labor Day”.  Players would then submit their favorite ChatGPT response for all other players to see, and then all players would vote for their favorite. The player with the most votes at the end wins. I ran this game three separate times with at least 5 players a session and changed factors such as failure tolerance and question formatting each time to test their effects on players’ feelings of overall creativity. Player sentiment was measured using Likert scale surveys and one-on-one interviews. In the end, I validated Compton’s framework. Players preferred to have power over control, their level of enjoyment positively correlated with their level of surprise, and the social element of the game seemed to highly affect players’ behaviors. Some discoveries I was not expecting were that players reported that having a prize did not affect their performance, and that players only felt ownership over their output if it was good (if it was bad, it was ChatGPT’s fault).

ChatGPT Comedy Hour

Research

Citation:  Casual Creators: Defining a Genre of Autotelic Creativity Support Systems by Katherine Compton

Technical Details

For this version of the game, I used votesout.org to host the answer submitting and voting. I used chat.open.ai aka ChatGPT for the generative aspect. Regarding research technology, my surveys were hosted on Qualtrics and one-on-one interviews were conducted over Zoom. There were 17 survey participants and 6 interviewees.

The Research Question: Do these factors lead people to feel their most creative and have the most fun? If so, to what extent? Trading control for power Emergence and surprise Explorability Ownership Allows for social sharingThe Research group 1: control Participants will be able to try out as many new things as they want There will be a prompt but no restrictions otherwise to what they can type into ChatGPT group 2: one chance (restricting safety & explorability) Participants will only be able to type in one prompt per question There will be no restrictions on what they can type into ChatGPT group 3: madlibs (restricting control & possibility space) Participants will be able to try out as many new things as they want There will be a Madlibs type of format that players have to use with their ChatGPT promptsTakeaways i was wrong about... People not wanting to give up control People performing differently if there was a prize on the line i was right about... Surprise is fun! Things are more fun with other people Surprises The game works better with people who know each other People only feel ownership if the output was good People loved the gameSurvey Results. One can see here at while Group 1 reported being most affected by whether or not they chose to compete, Group 3 reported the highest influence of surprise, social influences, and overall creativity.