Trinity Suma, Columbia University 

Oyewole Oyekoya, CUNY Hunter College

Week 1

I first met my REU cohort the Monday after I arrived in NYC, bonding over bumper-less bowling at Frames Bowling Lounge.  Our initial meeting was refreshing; I was excited to work with all of them and make new friends.  On Tuesday, my real work picked up after a quick tour of Hunter and a meet-and-greet with all the REU mentors.  I began discussing directions for my project with Dr. Wole and outlining a project proposal.  Wednesday was the first session of VR, AR, and Mixed Reality, a class taught by Dr. Wole that I and the rest of my cohort are auditing.  For the rest of the week, I finalized my project proposal, defining my project’s uniqueness and conducting a preliminary literature review.  We wrapped up the week learning how to use Paraview and presenting our proposals.

Week 2

My work picked up this week as I began to familiarize myself with Reallusion to design the avatars for my study.  My project is ideally going to follow a bystander intervention scenario set in a pub/bar environment.  Below is my idealized script, but I will likely cut out some dialogue for simplicity. 

My scenario has five characters:

• Person A: the one bullying B
• Person B: the one being bullied by A
• Person C: another bystander
• Person D: bar owner 
• User

Below are also preliminary avatar designs for persons C and A, respectively.  I am not designing an avatar for the user since it is ideally in the first person.  I am also considering not designing one for person D for simplicity.  Only person B will be made from a headshot, and it will resemble someone the user knows.  This week, I also began working on my paper, beginning with the introduction and literature review.  Next, I want to continue creating my avatars and animate/record the audio.

Week 4

Welcome to week 4!  I can’t believe I am already halfway through this experience.  This week I finished animating my avatars on iClone with audio recordings of both my voice and my brother’s voice.  There has been more discussion about the direction of my project, but in the meantime, I worked on creating pitch variations for my audio.  Each clip has been pitched both up and down by 10%.  I chose 10% since it seemed like a good baseline to start; the clips did not sound incredibly unrealistic, but the difference was still noticeable.  Below is a sample line from the aggressor.  The first clip is the unedited recording, the second clip is the pitched-up recording, and the third clip is the pitched-down recording. 

We have decided not to abandon the bystander scenario I wrote.  Instead, it will be used as the medium to convey the altered audio.  The scenario will be presented in a survey.  The study participant will watch the scenario play out by reading the narration and watching video clips of the animated avatars.  In some cases, the participant will be presented with multiple variations of the same clips (this procedure is subject to change) in which they will have to rank the clips based on their level of aggression or assertiveness, depending on the character.  This study will allow future developers to gauge how to record and modify their audio to best convey their desired tones. 

Week 5

My progress was slower this week as we finalized the focus of my project.  After much discussion, we are going to study how various combinations of over-exaggerated, under-exaggerated, and average facial expressions and tones affect survey participants’ perceptions of aggressiveness and assertiveness (depending on the character being evaluated).  A diagram of each combination is shown below.  Nevertheless, this week I worked with Sonia and Dr. Wole to record the lines of the aggressor and bystander in my scenario with their lookalike avatars.  We have decided not to use the avatars I designed from the neutral base to maintain the lookalike avatar concept nor the audio my brother recorded.

In addition to work, I had a lot of spare time to myself, which was very healing.  I visited the MET and Guggenheim for free and met up with a friend from home.  On Thursday, the REU cohort attended a lunch cruise where we had great views of the Freedom Tower, Brooklyn Bridge, and the Statue of Liberty. 

Week 6

I had less work to do this week, but I expect it to pick up very soon.  I focused on editing all the videos of the lookalike avatars I had filmed with Sonia and Dr. Wole.  Sonia played the bystander while Dr. Wole played the aggressor; each of them filmed a variation where they underexaggerated and overexaggerated their words and facial expressions in addition to a neutral version.  From there, I exchanged the audio on each video to create 9 different variations of their words.  See the diagram above.  Here is one of the videos.  Once my videos were approved and we decided on a survey design, I created my survey in Qualtrics and am preparing to send it out early next week or sooner.  

Luckily, I was able to take advantage of the holiday weekend and joined my family in Atlantic City, NJ.  Later in the week I also went to see TWICE at Metlife Stadium. 

Week 7

This week, I finalized my survey design and sent it out to my REU cohort, the mentors, and other potential participants.  As of Friday afternoon, I have 22 responses, but not all of them are usable since they are incomplete.  I am beginning the data cleaning and analysis stages.  Given my data type and how they are categorized, I am still figuring out what tests I will use.  Dr. Wole and I have discussed non-parametric Friedman tests and two-way repeated measures ANOVA tests.  Hopefully, it will be finalized this weekend.  I have also been researching new papers that are applicable to the emotional recognition aspect of my study to include in my introduction and literature review.  

This week, my cohort also visited the ASRC again to tour the Illumination Space and the facility itself.  We also tested Sabrina’s AR app which was very fun!  I had enough time that day to visit Columbia and use one of the libraries to get some work done, which was very nice.  This weekend, I am going to Atlantic City again for my grandma’s birthday as well as taking a class at the Peloton studio near Hudson Yards. 

Week 8

Happy last week!  Thank you for following me throughout the last 8 weeks and reading my blog posts!  Over the weekend, I finally updated the introduction and literature review sections of my paper as I mentioned last week.  This week was one of my busiest as I balanced packing up my room to move out with finishing some preliminary data analysis to include in my final presentation.  Since we have yet to analyze the statistical significance data we ran, I looked at the mean and median responses for each question type.  Our results are following our original hypotheses; you can find the data in the slideshow below.  On Friday, I ground out my results and discussion sections for my paper and finished packing to go home Saturday.  I have had an amazing time this summer and will miss all of my cohort members!!

Presentation: VR REU – Final Presentation 2

Final Paper:
Trinity Suma, Birate Sonia, Kwame Agyemang Baffour, and Oyewole Oyekoya. 2023. The Effects of Avatar Voice and Facial Expression Intensity on Emotional Recognition and User Perception. In SIGGRAPH Asia 2023 Technical Communications (SA Technical Communications ’23), December 12–15, 2023, Sydney, NSW, Australia. ACM, New York, NY, USA 4 Pages. https://doi.org/10.1145/3610543.3626158 – pdf