Deshanae Morris, Farmingdale State College

Week 1:

The first week of the REU program began on the 2nd of June where I got to meet the group I would be working with for the next 8 weeks over a fun game of bowling. Everyone was extremely welcoming and although I utterly failed the game that day I still enjoyed myself. The following day, the group got to know their mentors and toured Hunter College. For the remainder of the week, I dedicated my time to brainstorming unique ideas and diligently working on drafting and finalizing my research proposal. I also conducted a literature review to ensure that my proposal was unique. While there is still a significant amount of work ahead, I am excited about the journey and look forward to achieving much more in the upcoming weeks.

Week 2:

During the second week of the REU program, I continued to develop my proposal, literature review, and methodology. This past week has been dedicated to understanding the technical aspects of my study to begin implementing my methodology. Dr. Wole has introduced the REU group to various applications and concepts related to 3D modeling and virtual reality systems, enhancing our familiarity with these tools. I have been practicing more with Reallusion as I will need it for my research paper, but I still have much more work to do to produce my final product.

Week 3:

This past week, I have refined my methodology and concentrated on the technical aspects of my research. I have focused on creating the first of two look-alike avatars required for the study. I successfully developed a realistic avatar that closely resembles myself. I will continue using character creator in Reallusion to see if I can make my avatars more similar to the models than they already are. In the upcoming week, I plan to complete the second avatar, a male look-alike, but I am pleased with the progress thus far. Additionally, I experimented with the facial puppet and live face features in iClone to achieve the desired emotions and scenarios. Still, it was fascinating to see my look-alike avatar come to life. Additionally, Dr. Wole went over immersion, presence, and reality with the group so we were further familiarized with reality systems and immersion. In the following week, I aim to begin recording the videos needed for my survey, complete the audio recordings, and develop the survey so I can start collecting data for my research.

Week 4:

This week, we concentrated on finalizing our methodology. I continued to refine my skills with Reallusion, Character Creator, and iClone to develop two avatars, one of myself and one of Dr. Wole, that I felt confident with. Utilizing the facial puppet feature in iClone, I began varying the eye movements in these look-alike avatars. However, I will continue to explore the applications and features to enhance their eye movements and facial expressions further.

The group also dedicated time to preparing for the midterm presentation scheduled for the end of the week. We each created slideshows and demos to showcase our progress, ensuring that everyone was informed about the individual projects and the overall direction of our work. I produced several demo videos to present to the group, aiming to see if they could accurately identify the different eye movements in my avatars. This exercise provided valuable insights into how participants in my survey might respond to my research. Regarding my research objectives, I plan to conduct a survey evaluating realism, emotion/intention, perception, and comfortability using a self-introduction scenario, specifically seeing how these eye movements play a role in an observational social context.

Week 5:

This week, our entire group concentrated on developing our surveys. I did start with Google Forms, but it wasn’t able to achieve what I hoped to capture with my data so I moved over to Qualtrics. I added most of my questions and videos to the survey through Qualtrics, but Dr. Wole and I agreed that adding cropped videos of my avatars would be a compelling element so I had to add those in too. As I continue editing my survey I gain more and more confidence in my research and I am excited about what I am planning to contribute to the research community. Throughout the week, we continued expanding our knowledge of virtual reality and 3D modeling with Dr. Wole. It’s always exciting to learn something new in the tech world.

This weekend, we’re planning a lunch cruise as we approach the end of the REU. It’s a bittersweet moment, but I’m looking forward to having fun and getting to know my group even better. By the end of the weekend, my survey should be finalized, and I should have some collected data to start adding to Overleaf. I can’t wait to see how the participants respond to my work!

Week 6: 

I took the time to finalize my survey, and although I did have lots of revisions I ended up with a final product that I was really satisfied with. I used Qualtrics which took a little time to get used to because of my lack of knowledge of the software, but it ended up being rewarding when I got it to work. I ended up enjoying the process of making the survey and below I displayed the survey flow as a short insight into my survey and a component needed to create a survey using Qualtrics. I stumbled into a roadblock when developing my survey because my collected data was not displaying properly in the results section. It was hard trying to find a solution as this data is valuable to my research, but after importing my data to a copied version of my survey I managed to fix my problem. Additionally, Dr. Wole allowed the REU group to review two demo research papers on SIGGRAPH. It was refreshing to finally be on the other side, reviewing other papers instead of receiving feedback on my own. The end of the REU is approaching quickly therefore, I have to start adding and updating my research paper on Overleaf so I can present a final product.

Week 7: 

This week, we finished data collection and started exploring how the responses can be analyzed for the topics we investigated. The results I collected were pretty interesting because the comfort and realism rankings were different between all my variables (Female and Male avatar), (Normal, Random, Static), and (Full video and Cropped videos). Conducting some basic analysis I was able to see how drastic each variable affected the realism and comfort ranking of my avatars, but I still have more statistical analysis to be done as I plan to conduct ordinal regression on my variables to evaluate the significance and the cause and effect relationship between them. Over the weekend I should have finished my results and analysis to move on to the discussions and conclusions. I have lots of data and variables so analyzing the data might prove to be extensive, but Dr. Wole has helped me break down the information to figure out what I need to find from it.

Week 8: 

The last week was very busy as I including my participants spent a lot of time finishing up the final components of our paper. Additionally, this week we participated in a teacher mocap session where the group and I viewed multiple presentations from professors. One really interesting presentation was one about people telling their stories or showcasing their culture through games and the game that was displayed I found to be a funny way of showcasing a real-world struggle. To be more specific the game was about a woman of color swatting away the hands of those who want to touch her hair without permission. I was glad to support these teachers with their research and I found it to be very rewarding. Furthermore, the group prepared to give our final presentation this Thursday to those at IOWA State, the group mentors, and the other REU students. The IOWA state students presented to our REU group the following day and I found their studies to be amazing work because they take on very complex research ideas. The final day was bittersweet as I have grown quite close to the REU group, but I would not change this experience for the world. I sent my final draft paper over to SIGGRAPH for review bringing the program to completion.

Project: The Community Game Development Toolkit – Creating easy-to-use tools in Unity to help students and artists tell their story and show their artwork in a game like format.

Mentor:  Daniel Lichtman

About Me: I’m CUNY BA student majoring in Game Design and Entrepreneurship at Hunter College.

Week 1: Project Proposal and Toolkit Exploration

This week, I had the time to explore and delve into the toolkit developed by Daniel Lichtman. This toolkit was specifically designed to aid in various projects To gain a deeper understanding of its functionalities, I decided to open and explore an example project using the toolkit.

Utilizing the tools provided by Daniel Lichtman’s toolkit, I created my own custom scene with a custom HDRI and some art assets.

Using these tools I learned how they were supposed to be used and the current process for adding them to a Unity project and making a new presentation with them.

Week 3:

During this third week, I have been engaged in conducting research on VR technology. One significant milestone I achieved was writing the methodology for my upcoming VR research paper.

Additionally, I started work on a VR compatible Unity project where I created an prototype layout for an art project.

Central to this project is the implementation of a VR locomotion system, which aims to enhance user immersion and interaction within the virtual environment. Furthermore, as part of this project, I learned how to directly connect an Oculus Quest 2 headset to the Unity project.

Lastly, I started learning how to use VMD to visualize molecules and atoms.

Week 4:

This week I worked on the methodology section of my paper implementing some of the feedback from my mentor into the paper. In addition, I created a demo level and populated it with art to showcase what a potential art project would look like. It was a great way to bring my project to life within the virtual world. By carefully selecting and placing the art assets, I aimed to create an immersive experience that would captivate the viewer’s imagination. I first found some free assets on the Unity asset store to create a simple courtyard and environment with. I then found some copyright free art to place in the level as resemble what an artist would potentially do in their work.
My level results looks like the following.

To further progress the toolkit, I integrated the Oculus Quest 2 headset. I successfully connected the headset and got the Unity project to work seamlessly inside it. Now, the viewer could step into the virtual environment and feel as though they were physically present within the level.

Week 5:

During the fifth week, my main focus was on developing and adding to the website that contains all the necessary instructions for the VR project toolkit. This involved detailing the setup process and providing step-by-step guidance for implementing the toolkit with screenshots and text. Additionally, I compiled a list of settings for Professor Litchman to incorporate into the toolkit, aiming to get rid of some of the setup steps.

In order to ensure the toolkit’s effectiveness, I sought assistance from artists whom I am acquainted with. I requested their participation in testing the toolkit for the study, and fortunately, three of them agreed to help out.

Aside from the project work, we had an enjoyable experience as a team during this week. We embarked on a delightful river lunch cruise, which granted us the opportunity to admire the captivating Manhattan shoreline and the iconic Statue of Liberty. The lunch was delightful, and it provided a pleasant setting for us to get better acquainted with one another and learn about everyone’s well-being.

Week 6:
During this week, we had the July 4th holiday, which caused a slight delay in sending out the instructions to our case study testers. However, on July 5th, I promptly distributed the instructions and provided a comprehensive explanation to the testers regarding the purpose of the study and the specific objectives of the toolkit being tested.

In addition to providing instructions, I created a survey for the testers. This survey was designed to gather both quantitative and qualitative data, allowing us to gain a better understanding of the testers’ experiences. To ensure best responses, I included open-ended questions, providing the testers with the opportunity to provide detailed feedback.

Lastly, as we decided to focus on a single study instead of two, I dedicated some time to rewriting the research paper that I had previously prepared. This adjustment allowed me to revise the paper for the new study approach, ensuring the coherence and accuracy of the paper.

Week 7:
During the seventh week, my main focus was to provide assistance and guidance to the users/testers of the toolkit as they carried out their testing. I dedicated my time to addressing any questions they had and ensuring a smooth testing process for them.

In addition to supporting the testers, I devoted some time to organizing the structure of my paper. Specifically, I outlined the sections for the User Study, Results and Analysis, Discussions, and Conclusion. This preparation allowed me to establish a clear way for me to present my findings.

As I awaited the availability of the necessary data, I also began working on writing the User Study section of the paper. Since I did not have the data required for Results and Analysis at this point, I focused on writing the details of the User Study itself especially on how it was conducted.

To ensure the effective presentation of the data, I consulted with Dr. Oyewole. Together, we discussed utilizing a tabular format to convey the findings, considering the small number of participants involved in the study. This approach would help provide a concise and organized presentation of the data.

Towards the end of the week, I received the data I needed from the three testers. With this information, I am now ready to proceed with completing the remaining sections of the paper in this final week.

Nairoby Pena, Cornell University

Week 1: 

Dan and I met a couple of days before the REU began and we decided that we should start from a broad standpoint and begin to narrow in as time progressed. My first “assignment” was to research physical disabilities, learning disabilities, and mental health (emotional) issues, and the impediments that students face in higher education when they have these disabilities. During this week’s meeting, we became increasingly aware that with just 8 weeks it is unfortunately not possible to have a productive project if we take the angle of aiding mental health issues with VR. We decided that the next “assignment” will be to look for two core classes that each student must take at Hunter and Cornell (four in total) and look for the issues or impediments that a student from each of the disability groups may face in these classes (with a heavy focus on physical and learning disabilities). I will be starting on this assignment to wrap up the week as well as beginning to explore Unity, and starting to think about developing the documentation since I found one or two papers that could be a part of that. By next week we should be working on a concrete methodology for creating or enhancing VR as a tool for the impediments that students with disabilities face.

Week 2: 

This week I presented the Dan about core classes both at Cornell and Hunter which all students have to take and I talked about the impediments that students with disabilities might face in taking those classes. For example, Cornell has a swimming requirement in order to graduate and we talked about the obvious point being that every person is not physically able to jump in the pool and swim. In addition, at Hunter, there is an English course that every student is required to take which involves heavy writing and reading which students with dysgraphia or dyslexia respectively would probably struggle with. There is also an astronomy class that students can choose to take as a part of their core curriculum and unfortunately, a disability might impede a person from taking the course because of the lab component in which 9 out of 13 experiments are self-guided on a computer. Dan brought to my attention that every lab has very high tables that are made for people to be standing at and if we think about someone who might have a wheelchair this could be difficult for them to navigate as they would have to be looking up at their peers making it harder to collaborate. Dan and I also spoke about the research paper where he guided me in understanding what I could mention in my abstract. We noticed that we still do not have a clear methodology but we did talk about the fact that VR heavily exists right now in virtual museum visits and how this is a huge help for people with many different types of disabilities so this might be where the project is headed at this point.

I began to look into Unity and attempted to learn about all of its components that had to do with VR. I built my own VR space and then realized that I didn’t have a headset to try what I created; I also looked into building a microgame. 

Week 3: 

This week I focused on a deep dive into dyslexia, what it is and how it affects students. I looked at what impediments students might face when they have this disability and have to take English 120 at Hunter. I also researched how VR could or could not help students with dyslexia. Most research concludes that more scientific research is needed to prove that VR can help, however, theoretically, it can and there are VR programs that exist for students with dyslexia. After meeting with Dan, we decided that we should not focus on dyslexia because it would be difficult to come up with a project to help in minimal time. Being that there has to be a visualization aspect for the outcome of the project, I received clarity from Dr.Wole and decided that I will be creating a VR “game” or program where autistic students can engage in social modeling. This can include modules on greeting people, engaging in conversation, and how to conduct oneself in various social environments. Since it is week 3 already I will have to work rigorously to develop this. So far I have gone into unity and used the assets in order to build the virtual environment. I will be working to develop a user interface for at least three different social situations and hope to test at least one of them by the end of next week. 

We had a paper writing zoom session this week where I was able to get a draft of my abstract and begin my introduction. I will have to go in and edit this as my focus has shifted from disabilities in general to developmental disabilities (more specifically autism).

Week 4: 

Last week’s idea about social modeling has been modified so that it will be able to get done in the amount of time that is left. I will now be using a museum asset in unity and putting a human-like avatar in the virtual space. The avatar will guide the person that wears the VR headset through the museum and what the avatar does will play on a loop. (Thank you to Dr. Wole for the idea). Applying prior research to this idea, I hope to implement the general outline for social skills teaching packages this means that the program will allow for repetition of the target task (visiting a museum), verbal explanation of the social skill, the practice of skill in realistic settings (in virtual reality), role-play of target behaviors (practicing the behaviors in the virtual reality program without the guided audio).

This week I worked heavily on my paper. I was able to get through most of the introduction, related works, and some of the methodology. I will continue to add to it as the project moves along. Finally, I developed a slides presentation for the midterm check-in.

Week 5: 

This week I was able to get the museum asset in unity. I further developed the idea of social modeling through a recollection of my experience working at the Intrepid Museum with the Access program. One of the events that the program provided was early morning openings for students with disabilities, with a large portion of the population having autism. I decided that I should try to use that as the model in the museum asset. Below is a screen recording of what I have so far. I was able to edit the presentation from what the original asset had to what I wanted it to say. I hope to add audio of my voice so that everything isn’t reading based and I also will be trying to make it possible for the user to look through the museum itself; after this is done I will deploy it to the headset and make edits as needed.

I worked on the citations in my paper and revised it as my ideas developed. I am starting to think about what my results section will look like. We all visited the CUNY ASRC and got to take a tour while learning about their initiatives.

Week 6:

This week I had technical difficulties with adding audio to my visualization portion. Kwame and I spent hours trying to get the code right so that certain audios would play at each slide, however, we could not get it to work. I also looked for existing scripts that may help but they were also unable to work. I created a backup plan to have the audio play on a loop in its entirety, though this would cause people to listen to different audio portions at different points in the experience.

I sent my mentor Dan my paper for feedback and began to work on refining it and next week I hope to begin to finalize it. In addition, Dan gave me the idea to share some of the other existing VR programs that exist to explain in my final presentation so that people understand the extent to which they can be helpful for students with disabilities.

Next week I will be testing my visual aspect from Unity and making any fixes as well as focusing on the paper.

Week 7: 

I got to participate in Talia’s user study this week; it was cool to see how a fellow participant’s visualization aspect came out. Since my project does not have a user study, I focused on what I would write in my discussion for my paper. I met with Dan and he was able to give me more feedback on the paper so that I could continue to work on it. As for the technical difficulties, it seems that I will have to play the looped audio in its entirety due to the diminishing amount of time that is available. Dan and I spoke about the final presentation and what I should include which led me to begin to work on my final slides.

We got to visit the Bronx Zoo on Wednesday where we zip-lined, that was fun! We ended the week with a meeting speaking about how the final week would look and how to prepare.

Week 8: 

I spent the last week submitting my abstract, finishing my paper, and practicing my final presentation. I was able to participate in Kwame’s user study. For the final touch to my visualization portion, I put an animated avatar into the museum to make it seem as though there was a tour guide.

Overall it has been a great learning experience participating in this REU. I am grateful for all that I have learned and all of the obstacles that I was able to surpass in developing this project.

Final Paper