Music Accessibility for the d/Deaf and Hard of Hearing

 

Length: 20 weeks
Team: Joyce Lin, Jessie
Mentor: Dr. Sarah Coppola
Role: Researcher, Writer
Tools: Google Docs, Google Forms, Miro, Zoom

Context

In a self-directed research study, I closely worked with a deaf interaction design student named Joyce Lin, advised by Dr. Sarah Coppola from the Human Centered Design and Engineering (HCDE) department at the University of Washington (UW), to develop an IRB-approved research protocol. The goal was to publish our research in a journal like ASSETS or CHI, and develop a working prototype to support Joyce Lin’s Design Capstone. We have been working on this project since April 2021. This research is funded by the Mary Gates Research Scholarship (Joyce Lin) and the HCDE DEI Mini Grant (Jessie Zhang).

We chose to conduct this research because we self-identify as d/Deaf and hard of hearing (DHH), we have different experiences with music, and we were curious about how we could move the d/Deaf and hard of hearing (hereafter referred to as DHH) people’s experiences with music forward. Our research consists of two stages: (1) generative research and (2) design prototyping activities. Ultimately, we hope to shift the assumptions about the d/Deaf and hard of hearing people’s experiences with music and suggest a design solution that may later be adopted by companies serving musicians, artists, and fans like Spotify, Pandora, YouTube, etc.

As with every research project, we started with a couple of hypotheses: 

 
  1. ACCESSIBILITY. Accessibility of music experiences will closely correlate with enjoyability for many DHH individuals. The more accessible the music experience is, the more enjoyable the DHH individual will find the experience.
  2. HEARING LEVEL. The level of hearing loss will play an important role in how many DHH individuals access music. The more hearing that a person has, the more likely they are to access and enjoy music.
 

We spent several months developing our IRB research protocol with critical feedback from our mentor, Dr. Sarah Coppola. As a disabled faculty member, Sarah specialized in accessibility and inclusive design. We created a quarterly milestone plan to track project progress and ensure group accountability. We leveraged our existing relationship with Disability Resources for Students at UW to coordinate accommodations like CART (real-time captioning) and ASL interpreters for our studies. When it comes to doing research, relationships are key to success!

 

Research Methods

Our formative research methods consisted of: 

  1. Literature review
  2. Surveys
  3. User interviews

These methods are discussed below.

  • The literature review enabled us to gain a deeper understanding of the relationship between music experiences and DHH experiences and to identify potential gaps in the literature that our research could address.

  • Surveys needed to be accessible in both English and American Sign Language (ASL) for many DHH people to answer. Joyce and I split the tasks evenly. For example, I’d create a script in ASL gloss (a way of writing ASL signs) to aid Joyce in signing the questions, while Joyce would build the survey in Google Forms. To prevent bad actors from poisoning our survey, particularly if they did not identify as DHH, we intentionally reached out to specific groups of people, and we did not publish our survey on online forums such as Reddit.

  • Our interviews were more complex. We interviewed 6 DHH users.

    We asked our users to share background information. We created three Design Activities for the DHH participants to engage in so that we could more readily analyze the results:

    1. Word Exercise

    2. Image Exercise

    3. Cognitive Mapping Exercise

    The goal was to uncover their sentiments about accessibility and enjoyability based on various music experiences.

 
Design Activity 1: Word Exercise

ID: Word Exercise. Review the following technology categories and list of adjectives provided. Please sort (or add your own) adjectives that describe your feeling towards each technological music experience. You can use as many or as few adjectives as you need to describe your experience.

ID: As part of the word exercise, we listed technologies and adjectives and asked participants to assign an adjective to each of the technology word. They could assign the same adjective multiple times or add their own adjective. The goal was to encourage participants to think about how they felt about each technology.

Design Activity 2: Image Exercise

ID: Image Exercise. From a list of images, choose 5-10 images and rate the images that relate to music activities along the axis of least to most access and least to most enjoyment.

Design Activity 2: Image Exercise - Images

ID: Various images that the participant can choose from and drag onto the blank graph of accessibility vs. enjoyability.

ID: A blank graph with the X axis from Least Access to Most Access and the Y axis from Least Enjoyable to Most Enjoyable.

Cognitive Mapping Exercise

ID: Cognitive Mapping Exercise.

 
 

Co-Design

We invited all interview participants to a follow-up co-design session, held via Zoom and Miro. The co-design session was broken into 3 distinct parts:

  1. Warm-up
  2. Design Activities
  3. Concept Testing

    1. Drawing Discussion: In what ways can a song be visualized? Using pen and paper, or this Figjam’s drawing tools, sketch out and discuss your ideas.

    2. [Closer Captioning]: How do you feel about Christine Sun Kim’s exploration of captions? In what ways can captioning and lyrics for songs be more descriptive? What are the drawbacks of this style of captioning?

  • We gave them the following Design Activities:

    1. Imagine you are experiencing music alone to your song of choice.

    2. Imagine you are experiencing music in the car to your song of choice.

    3. Imagine you are experiencing music at a social event.

    4. Imagine you are experiencing a live music performance with your favorite musical artist(s).

    All of these activities ended with the prompt: “How would you design lyrics, captions, and/or other music visualizations to enhance your listening experience?”

  • Participants were also invited to validate concepts.

    1. Music Video

    2. Streaming (Mobile)

    3. AR

    4. Concert Display

Research Analysis + Insights

As our research is mainly qualitative, we conducted a literature review synthesis and a thematic analysis of survey and interview responses. We used Miro to note common themes and outliers that emerged.

Literature review synthesis

Based on the existing HCI literature, we generated the following insights:

  1. DHH people rely on a combination of vibrotactile and visual experiences to consume music.

  2. DHH people rely on vibrotactile experiences to follow rhythm and beats.

  3. Many DHH people have difficulty interpreting abstract visual representations of music (e.g., spectrograms) and need additional context.

In summary, abstractions would be least helpful, and having something that’s both visual and vibrotactile would be most helpful to DHH people experiencing music.

Thematic analysis

The image above shows the first pass in affinity diagramming that we made from the raw quotes. Raw quotes were color-coded with participant numbers, and we grouped similar quotes. From there, themes emerged as is the case with the grounded theory approach. We ran the pass two more times to ensure that the themes were consistent.

Our thematic analysis revealed several findings:

  1. DHH People rely on accurate, available lyrics as a way to contextualize the songs. Participants expressed shortcomings of current captioning/lyric standards.
  2. DHH participants frequently mentioned a combination of Deaf friendly space guidelines, visuals/lighting, vibrotactile and captioning access for ideal music experiences.
  3. Engagement of music depended on circumstantial and environmental factors such as volume, background noise(s), social settings, prior knowledge, participant’s hearing, and more.
  4. Participants expressed ambivalence or had frustrations with technological tools like hearing aids, CIs, bluetooth, telecoil, and headphones due to ergonomic and pragmatic factors.

So, what does this mean?

DHH people really love music! They experience music very differently from many hearing people, which is valid. However, they have a hard time enjoying music to the fullest because they have to figure out tools and accessibility, which creates friction.

Aaand what did I learn?

It was a joy to work with a d/Deaf colleague who is an interaction designer! 100% would work with her again, with the caveat that I needed to give myself more time to follow through on my tasks. As a busy grad student also working on another capstone project, I needed time and space for a thoughtful analysis. (Big ups to G for helping us figure out the best approach to analyzing qualitative data!)

I’m really proud of our work, but if I were to do this again, I’d make sure that I could make the commitment work.