Nairoby Pena, Cornell University

Week 1: 

Dan and I met a couple of days before the REU began and we decided that we should start from a broad standpoint and begin to narrow in as time progressed. My first “assignment” was to research physical disabilities, learning disabilities, and mental health (emotional) issues, and the impediments that students face in higher education when they have these disabilities. During this week’s meeting, we became increasingly aware that with just 8 weeks it is unfortunately not possible to have a productive project if we take the angle of aiding mental health issues with VR. We decided that the next “assignment” will be to look for two core classes that each student must take at Hunter and Cornell (four in total) and look for the issues or impediments that a student from each of the disability groups may face in these classes (with a heavy focus on physical and learning disabilities). I will be starting on this assignment to wrap up the week as well as beginning to explore Unity, and starting to think about developing the documentation since I found one or two papers that could be a part of that. By next week we should be working on a concrete methodology for creating or enhancing VR as a tool for the impediments that students with disabilities face.

Week 2: 

This week I presented the Dan about core classes both at Cornell and Hunter which all students have to take and I talked about the impediments that students with disabilities might face in taking those classes. For example, Cornell has a swimming requirement in order to graduate and we talked about the obvious point being that every person is not physically able to jump in the pool and swim. In addition, at Hunter, there is an English course that every student is required to take which involves heavy writing and reading which students with dysgraphia or dyslexia respectively would probably struggle with. There is also an astronomy class that students can choose to take as a part of their core curriculum and unfortunately, a disability might impede a person from taking the course because of the lab component in which 9 out of 13 experiments are self-guided on a computer. Dan brought to my attention that every lab has very high tables that are made for people to be standing at and if we think about someone who might have a wheelchair this could be difficult for them to navigate as they would have to be looking up at their peers making it harder to collaborate. Dan and I also spoke about the research paper where he guided me in understanding what I could mention in my abstract. We noticed that we still do not have a clear methodology but we did talk about the fact that VR heavily exists right now in virtual museum visits and how this is a huge help for people with many different types of disabilities so this might be where the project is headed at this point.

I began to look into Unity and attempted to learn about all of its components that had to do with VR. I built my own VR space and then realized that I didn’t have a headset to try what I created; I also looked into building a microgame. 

Week 3: 

This week I focused on a deep dive into dyslexia, what it is and how it affects students. I looked at what impediments students might face when they have this disability and have to take English 120 at Hunter. I also researched how VR could or could not help students with dyslexia. Most research concludes that more scientific research is needed to prove that VR can help, however, theoretically, it can and there are VR programs that exist for students with dyslexia. After meeting with Dan, we decided that we should not focus on dyslexia because it would be difficult to come up with a project to help in minimal time. Being that there has to be a visualization aspect for the outcome of the project, I received clarity from Dr.Wole and decided that I will be creating a VR “game” or program where autistic students can engage in social modeling. This can include modules on greeting people, engaging in conversation, and how to conduct oneself in various social environments. Since it is week 3 already I will have to work rigorously to develop this. So far I have gone into unity and used the assets in order to build the virtual environment. I will be working to develop a user interface for at least three different social situations and hope to test at least one of them by the end of next week. 

We had a paper writing zoom session this week where I was able to get a draft of my abstract and begin my introduction. I will have to go in and edit this as my focus has shifted from disabilities in general to developmental disabilities (more specifically autism).

Week 4: 

Last week’s idea about social modeling has been modified so that it will be able to get done in the amount of time that is left. I will now be using a museum asset in unity and putting a human-like avatar in the virtual space. The avatar will guide the person that wears the VR headset through the museum and what the avatar does will play on a loop. (Thank you to Dr. Wole for the idea). Applying prior research to this idea, I hope to implement the general outline for social skills teaching packages this means that the program will allow for repetition of the target task (visiting a museum), verbal explanation of the social skill, the practice of skill in realistic settings (in virtual reality), role-play of target behaviors (practicing the behaviors in the virtual reality program without the guided audio).

This week I worked heavily on my paper. I was able to get through most of the introduction, related works, and some of the methodology. I will continue to add to it as the project moves along. Finally, I developed a slides presentation for the midterm check-in.

Week 5: 

This week I was able to get the museum asset in unity. I further developed the idea of social modeling through a recollection of my experience working at the Intrepid Museum with the Access program. One of the events that the program provided was early morning openings for students with disabilities, with a large portion of the population having autism. I decided that I should try to use that as the model in the museum asset. Below is a screen recording of what I have so far. I was able to edit the presentation from what the original asset had to what I wanted it to say. I hope to add audio of my voice so that everything isn’t reading based and I also will be trying to make it possible for the user to look through the museum itself; after this is done I will deploy it to the headset and make edits as needed.

I worked on the citations in my paper and revised it as my ideas developed. I am starting to think about what my results section will look like. We all visited the CUNY ASRC and got to take a tour while learning about their initiatives.

Week 6:

This week I had technical difficulties with adding audio to my visualization portion. Kwame and I spent hours trying to get the code right so that certain audios would play at each slide, however, we could not get it to work. I also looked for existing scripts that may help but they were also unable to work. I created a backup plan to have the audio play on a loop in its entirety, though this would cause people to listen to different audio portions at different points in the experience.

I sent my mentor Dan my paper for feedback and began to work on refining it and next week I hope to begin to finalize it. In addition, Dan gave me the idea to share some of the other existing VR programs that exist to explain in my final presentation so that people understand the extent to which they can be helpful for students with disabilities.

Next week I will be testing my visual aspect from Unity and making any fixes as well as focusing on the paper.

Week 7: 

I got to participate in Talia’s user study this week; it was cool to see how a fellow participant’s visualization aspect came out. Since my project does not have a user study, I focused on what I would write in my discussion for my paper. I met with Dan and he was able to give me more feedback on the paper so that I could continue to work on it. As for the technical difficulties, it seems that I will have to play the looped audio in its entirety due to the diminishing amount of time that is available. Dan and I spoke about the final presentation and what I should include which led me to begin to work on my final slides.

We got to visit the Bronx Zoo on Wednesday where we zip-lined, that was fun! We ended the week with a meeting speaking about how the final week would look and how to prepare.

Week 8: 

I spent the last week submitting my abstract, finishing my paper, and practicing my final presentation. I was able to participate in Kwame’s user study. For the final touch to my visualization portion, I put an animated avatar into the museum to make it seem as though there was a tour guide.

Overall it has been a great learning experience participating in this REU. I am grateful for all that I have learned and all of the obstacles that I was able to surpass in developing this project.

Final Paper