Filip Trzcinka – Hunter College

Mentor: Daniel Chan

Week 1:

Before meeting with my mentor Daniel to narrow down what kind of project I would be working on, I decided to get ahead on my work and start on the Literature Review portion of my research paper. I wasn’t sure if I would be focusing on a physical, mental, or learning disability for the project so I began to research and read about said topics to see which direction I would prefer to take. Upon furthering my knowledge, I found myself more focused on papers that described learning and cognitive disabilities. I came up with two main research proposal ideas that I brought up to Dan when we met, and asked for his advice and for any feedback he may have. Upon conversing and pitching my ideas, he allowed me to choose which of the ideas I’d prefer to work on. After some though I decided my research project would focus on the creation of a driving simulation for student drivers who have Attention-Deficit Hyperactivity Disorder (ADHD). We planned to meet again next week after I have thoroughly researched the problems people with ADHD face when engaging in a learning activity, and what possible methods or features I should include in my simulation to aid in their learning experience. I also plan to begin mapping out and creating the simulated setting on Unity so I may also get ahead on the creation portion of the project. When we pitched our proposals to the group on Friday, Dr. Wole mentioned that he could try to connect me with someone who has had experience with making a driving simulator so that I could potentially build upon their work rather than start from scratch. Nevertheless, I will still take a look if Unity has some assets already implemented that I could use if the challenge of making a driving sim from scratch should arise.

Week 2:

This week I focused on the Literature Review section of my paper. I found a paper that described research conducted using a VR driving simulation to see if such a tool can help improve driving skills of people with Autism Spectrum Disorder. I decided to use this paper as ground work for how I’d like to develop my idea. Though their simulation was very basic, there was a test group where the simulation that used audio feedback to help remind the drivers of important rules of the road: like staying under the speed limit, staying in their lanes, etc. I considered what other features could be implemented to help those using the simulation to learn. I knew I would focus on ADHD so I read through papers where a VR driving simulation was tested with people with ADHD as a test pool, but could not find any conducted research that used an enhanced simulation instead of a basic one. Some used tools like eye trackers and such, but no real software implementations to benefit the learning experience of the user. I then looked through papers of teaching techniques used to help people with ADHD learn, with an emphasis on keeping their attention. After discussing with Dr. Wole, I went back and found papers that tested for ways to keep attention of everyone driving, besides just people with ADHD. With what I’ve read so far I created a list of features I’m hoping to implement into the driving simulation I create. The week was finished with writing my Related Works section and posting it on Overleaf with proper citations.

Week 3

With the Related Works draft completed, it was time to start the development of my driving simulation. Using the Unity game engine, I was able to import a low polygon city asset to use as the environment. I made some edits to it to make intersection lines more clear, box colliders for buildings so the user wouldn’t just phase through them, as well as adding traffic lights for every intersection. Since the traffic light assets were only for show and had no real functionality, I had to add point lights for the Red, Green, Yellow lights and I tried to write some scripts to allow the lights to change based on a timer, but unfortunately I made little progress with making that work. Will have to continue next week. I got a car asset which already includes a steering wheel object (so I would not have create my own) and imported it into the scene. I wanted a steering wheel object as I eventually hope to have the user actually grab the wheel to steer. For the car, I removed animations that were connected to the asset, added a camera to simulate the first person view then got to work on my scripts to allow the car to accelerate, reverse, and steer using the WASD keys (temporary inputs so I can test everything out at the moment) as well as having a hand brake as the space bar key. I had to take time to adjust the float values of the car’s rigid body, as well as the values for the acceleration as my car did drive pretty slow at the start. It still drives slow, but that could benefit the use for driving in a city environment. After running through the added features I plan to include with Dan, I began my work on the Methodology section of my paper as well as taking another hack at the traffic light scripts.

                                     

Aisha Frampton-Clerk, CUNY Queensborough Community College

Week 1:

I began testing some faces in the reallusion software. First, trying my own and then my boyfriend’s to see how it handled different lighting in the original headshot images and different features like facial hair. My first task next week will be to work on styling downloading hair packages and learning to manipulate them. I think these tests have given me a better idea of the scope of the software and gave me some interesting results to analyse and help shape the future of my study.

This first encounter with reallusion has helped me to understand the quality of headshot that make the best self avatars. When I start working with photographic headshots next week I will be sure to consider lighting and angle of the image.

Week 2:

This week I was focusing on finding celebrity source images to style in the Reallusion software. I first had to look for websites that provide royalty/copyright free images. I found that others had recommended flickr so I choose celebrities that had a range of images from their database. The image selection process took longer than I thought. As I tested images in the reallusion software I found that the quality of the images had to be very high as well as the angle of the face. Images with a celebrity smiling or hair over their face where difficult for the software to decifer. After finding the right pictures I put them into the reallusion software to begin styling. Using the smart hair content pack to create hairstyles that match the celebrity asethethic so they are as easy to identify as possible. I am trying to work out how I can make more custom hairstyles and clothing packs so the characters are as recognizable as possible.

Next I will be looking at how I can animate these characters, specifcally facial expressions and speech.

Week 3:

I have been importing my characters into IClone 7. I recorded a short voice memo and uploaded this to IClone 7. While there where automated lip movement and alignment to the words I had to tweak it so it fit better with the words. This included making adjustments to the facial expressions like moving eyebrows to match cadence and tone changes in the voice recording. As seen in teh face key tab selected polygons can be moved and matched to different sections of the speech to direct face movement.

Next I am going to look for recordings of the celebrity talking. I am going to look with ones that have video not just audio so I can look closely at their facial movements to model them. I also want to begin working with larger expressive movements over the whole body.

Week 4 :

This week I have been continuing to make characters and work on making them as realistic as possible. I have had some issues working with images of black celebrities. Often the software cannot pick up highlights on the face when it is selecting the color for the rest of the body. To work around this I have been selecting skin tones by hand to try and get a more accurate representation. Finding black hair textures has also been difficult as they dont come with the program. I have found in some cases layering different hair pieces in the smart hair content pack has given a thicker affect. I have also had to change some of the celebrities I chose as they did not have enough images for me to work with. I will have to test several pictures before i find one that gives an avatar that looks like the celebrity but now that i have the right images it has made styling much easier.

Here is a before and after of will smith with better original headshot and styling

Week 5:

I have been watching tutorials on how to create facial expressions/emotions onto reallusion characters in mixamo. Previously working with facial expressions exclusively in iclone 7 I am excited to see how the software differs. I want to work with the the camera plug in function as well.

I am also working on creating non celebrity headshots to make headshots that will not be familiar to subjects. with this styling is much easier as I have more control over the original images.

I have also been looking for more papers that are similar to my topic for me to use as a basis for my paper. Reading these papers in further depth has given me a lot of ideas of the features that contribute to realism and how these features can be investigated. so while it has been beneficial for understanding how to construct a research paper I have gained a better idea of what makes virtual reality real.

Week 6 :

Week 7:

I have been putting together videos of two avatars with audio ready for the questionnaire. I made 4 variations of the avatar. First a stationary image of the character then a video with audio and lip movement the next is a video including facial expressions and finally a video with full body movement. All videos have teh same audio accompanyment so as not to distract from the avatar.

I have also finished my survey by asking some questions about the video to participants. I will collect the results over the next weekend.

Week 8:

This week I was analysing the responses to my study and adding these to my paper. I completed the survey with 25 responses. I found that eye tracking had a huge affect on realism. As the second lip movement avatar was consistently ranked the least realistic and most unsettling. I was able to make some interesting conclusion about the importance of movement when creating virtual characters. I added figures that illustrate this to my paper and presentation.

Final report was submitted and accepted as a 4-pages short paper at VRST 2022:
Aisha Frampton-Clerk and Oyewole Oyekoya. 2022. Investigating the Perceived Realism of the Other User’s Look-Alike Avatars. In 28th ACM Symposium on Virtual Reality Software and Technology (VRST ’22), November29-December 1, 2022, Tsukuba, Japan. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3562939.3565636 – pdf