Home » Articles posted by VR-REU Student

Author Archives: VR-REU Student

Enhancing Virtual Exploration for Blind and Low Vision Users: In-Place Virtual Exploration with Mixed Reality Cane

Hong Zhao, CUNY Borough of Manhattan Community College

Mentors: Hao Tang and Oyewole Oyekoya


Week 1:

The day before the VR-REU program officially kicked off, Dr. Wole organized a bowling icebreaker event. It was really fun! Everyone was very enthusiastic, and I was happy to meet the other cohort. On Monday, we first completed an REU pre-program survey, then took a quick tour of Hunter College and met all the REU mentors on Zoom. My work officially started as I began discussing my project direction with Dr. Tang and getting a preliminary understanding of some of the current project code. Wednesday was the first class on VR, AR, and Mixed Reality, which primarily covered the ideal principles, how it works, and the history of VR. For the rest of the week, I reviewed some related literature and then had another discussion with Dr. Tang to finalize some research directions and complete my project proposal. Finally on Friday, we got an introduction to ParaView and presented our project proposals.


Week 2:

This week, Dr. Wole introduced us to the writing tool Overleaf. He also demonstrated how to use Tableau for optimizing data visualization. 

I completed the CITI training and successfully developed the initial version of the MR Cane control system. Here is how it works:

  • When the user long-presses the screen for two seconds, the virtual character begins to move.
  • The direction of the character’s movement is determined by the current orientation of the headset.
  • The mobile phone acts as a white cane. Users can swing their phones left or right to control the movement of the virtual cane.

Initially, I used the AR Session’s camera to capture the phone’s rotation data. However, this method proved to be imprecise when the phone was parallel to the ground, leading to suboptimal performance. To address this, I switched to using the phone’s gyroscope to obtain the rotation angles. This approach has significantly improved the test results.

Here are some key points about using the gyroscope:

  • When the device is in its default orientation, the X-axis points horizontally to the right, the Y-axis points vertically upward, and the Z-axis points away from the front of the screen. Thus, the back of the screen corresponds to a negative Z value.
  • As the device moves or rotates, these coordinate axes remain fixed relative to the phone.

This new method using the gyroscope has shown promising results in our tests, enhancing the accuracy and responsiveness of the MR Cane control system.


Arab Data Bodies Project

Lamya Serhir, CUNY Baruch College

Project: Arab Data Bodies Project

Mentors: Laila Shereen Sakr and Oyewole Oyekoya

Week 1:

The first week primarily consisted of meeting the other students, some of the mentors, and Professor Wole in addition to the project proposal. I read up on the research another student did last year for Arab Data Bodies to see how I could build on his work. Last year, he used the archive housing all the data, also known as R-Shief, to analyze the frequency of tweets, language used and general sentiment. A UML diagram of attributes like user, language used, url, tweet id, hashtag facilitated such analysis by organizing the data points. Ultimately, he used the sentiment output from such tweets to animate facial features of the avatar. 

I would like to focus making avatars of very prominent protestors that were in Tahrir Square, the center of political demonstrations in Egypt. Professor Wole recommended creating the scene such that elements of it could be used in any site of major protests, such as Alexandria and Suez. To do so, we can create crowds of people chanting and holding up signs during the protest.

The next steps are for me to get comfortable using Unity: in addition to beginner tutorials, there is a tutorial on crowd simulation that would be useful in my project. Another consideration is whether data from R-Shief archive will be beneficial, and if so, what kind of data that would be. I was thinking of basing the avatars on the most shared or viewed images or videos taken from the protests at Tahrir Square, but there are plenty of visuals available on the internet that I could use as well. 

Week 2:

This week, I focused on researching previous work done regarding VR documentaries. I found evidence about what components of VR increase the user’s sense of connectedness and how immersive documentaries create more positive attitudes towards human rights as opposed to written mediums. There is also research about the importance of social media in catalyzing the Arab Spring that I plan on using for background.

This week, I’d like to meet with my mentor to narrow down what aspects of the protests I should focus on. I plan on completing a crowd simulation that I can use to replicate a protest and finding assets within Unity that would be applicable to my project. Additionally, I’ll continue to search for relevant literature as the project progresses.

Week 3:

Professor Sakr’s team pivoted from creating a VR documentary to a video game. I learned more about the concept and inspiration artwork behind the video game, and will model my simulation after the Middle sovereign. In the world of the Arab Data Bodies Video game, there are five sovereigns each represented by a color. The Middle sovereign is represented by gold, and the theme behind it is royalty and status. I have the necessary components to make avatars move in a crowd-like fashion, so the next step is creating the environment in addition to the avatars.

Week 4:

I began creating the environment for the crowd simulation to take place (as depicted in the photos below). After consulting with the team and Professor Wole, the consensus that it would be best to focus on avatars for the remainder of the project was reached. The next step is to create avatars using generators like Metahuman and perhaps existing avatars in open source websites. There are three types of avatars I plan on creating: one with a human head, another with a robotic head and a third with a half human and half robotic head.

Week 5: 
This week, I familiarized myself with Blender so I could create the avatars for my user study. I experimented with different techniques, such as editing and sculpting to reach a desired output. I pulled aspects of the avatars such as the head and body in addition to accessories from SketchFab, which has 3D animations that can be downloaded and imported to Blender.
Week 6:
After discussing with Professor Wole, it was concluded that my project would be a developmental project since a user study would not be applicable. Therefore, I’ll be focused on writing for the following weeks and including some studies about storytelling in video games. Over the weekend, I will also import the avatars to the scene in Unity I created and animate them.
Week 7:
I spent the week revisiting my related works since my paper will be related to storytelling using VR in documentaries and video games. Most studies focus on one or the other; additionally, I’ll have to include the role of social media in the Arab Spring. I tried to import the blend files to unity, but ran into some issues transferring the textures. Moving forward, I’m focused on wrapping up the results and finalizing my paper.
Week 8:
Although this week was hectic, it was great to hear about other people’s projects and results in addition to taking their user surveys. I learned a lot over the course of this REU about my interests, strengths and weaknesses. At times, I felt like I wasn’t headed in the right direction or got frustrated when I wasn’t making as much progress as I would have liked. But I made sure to communicate my concerns with Professor Wole throughout and pivot my concept when it was clearly not working out. Although I didn’t expect to be working on a video game going into this project and found difficulty creating a thesis from my work since it was developmental and not as technical as the other projects, I think I handled the circumstances as best I could and am happy I stepped outside my comfort zone. I enjoyed the process of research and would have liked collecting and analyzing data. I hope to take on more opportunities that will allow me to do both. I also enjoyed meeting and getting to know the other students and the professor, in addition to members of the Arab Data Bodies team, all of whom were very kind, resourceful and intelligent.

Virtual Reality and Structural Racism Project

Ari Riggins, Princeton University

Project: Virtual Reality and Structural Racism Project

Mentors:  Courtney Cogburn and Oyewole Oyekoya

Week 1:

This week after meeting with Dr. Wole to discuss the specifics of the project and brainstorming ideas and research questions to be explored, I began writing my project proposal. This proposal discusses the goals and methodology for the project. 

This project aims to create an effective virtual reality based visualization that brings light to the disparities of structural racism within housing. This visualization will be based on data from different cities within the United States. We will use property value data as well as racial demographics of the areas as the input; this data will be represented as a three dimensional street or residential area with houses of changing dimensions; the dimension of the house will be proportional to its value over time with color displaying the racial component.

In addition to the project proposal, this week I also downloaded the program Unity and began getting used to it and thinking about how it could work for the project.

Week 2:

My goals for this week were mainly to learn how to use Unity to build the project and to do some background research and summarize it on the topic. I downloaded the Unity ARKit and began following some tutorials to learn how to use it. So far, I managed to make an ios AR application which uses the phone camera to display the world with an added digital cube.

cube in window

After discussion with Dr. Wole, the project idea evolved a bit to involve displaying the residential area as an augmented reality visualization where it can be viewed through a device as resting on top of a flat surface such as a table or the ground. The next step that I am currently working on in Unity is surface detection so that the visualization can align with these surfaces.

In terms of research, I found several relevant sources investigating structural racism within housing. I came across the University of Minnesota’s Mapping Prejudice project which hosts an interactive map of covenants in Minnesota where there were restrictions on the race of property owners and tenants. This project provides a view of one method of visualization for data on racial discrimination within housing.

Week 3:

This week was spent focusing mostly on the data. I met with Dr. Cogburn and Dr. Wole and we discussed a more specific view of the visualization. Dr. Cogburn brought up the reference of a report done by Brookings Edu which investigates the devaluation of Black homes and neighborhoods; this report will serve as the jumping-off point for the data of this project as well as a reference for discussion of the topic.

The data used in the report comes from the American Community Survey performed by the US Census Bureau and from Zillow. It will be necessary to find similar data from the census for this project. We decided that currently, the project should focus on one geographic area as a case study of the overall inequality. The city I am planning to focus on is Rochester New York; it was represented in the Brookings report and was shown to have a large disparity in the valuation of Black and White homes.

Week 4:

This week in unity I continued working with the ARKit to detect surfaces and display the visualization on them. We discussed the data after running into a roadblock where we did not have access to all of the information we wanted. The Brookings report had not provided the names of specific towns and areas that we found to be comparable so we cannot find data on them individually. However, we are able to use the reported data by changing our visualization a bit. Instead of being on a timeline, the houses will be on a sliding scale by the factor of race.

I also gave my midterm presentation this week which helped me solidify my background research for the project, as well as explain it in a clear manner.

Week 5:

This week I was mostly working in unity. I found a free house asset that works for the project and I used the ARKit to place this on any detected plane. I also worked on getting a United States map to serve as the basis of the visualization on the plane. We decided to use multiple locations from the Brookings report as case studies, so now I am still working to write the script which changes the house size in accordance with this data. Now that I have the pieces working, I need to arrange the scene and scale everything, as well as create some instructions for use.

I have also been working on my paper and am currently thinking about the methodology section.

Week 6:

This week in terms of writing the paper I made a short draft of my abstract and began working on the methods section. I worked in unity to get the house asset into AR and to write a script to add the growing animation in the video below. I added an input to the house which dictates the disparity which should be displayed through the amount of growth of the house. I also looking into changing the color of the house and having it fade from one color to another. When meeting with my mentors, they suggested that I try some different approaches to the overall visualization such as adding avatars to depict the neighborhood demographics of the house and changing the color of the house to green or some other monetary representation to depict the change in value.

Week 7:

This week, I have been working to get my demo finished. I fixed my shrinking issue with the house and I added the color change to the roof, though I still have to sync these two processes. In meeting with my mentors we decided that I should focus on completing this one scene instead of working on two due to the limited time left. We also discussed the background of the scene and things I could add to make it feel more like a neighborhood. We also discussed labeling and how I could make clear the data which the visualization is actually conveying. At this point, my work is going to be finishing this demo and focusing on wrapping up all we discussed and what I’ve worked on into a presentation.

Week 8:

In the final week, I was majorly focused on preparing for my presentation and finishing up every aspect of the project. I also worked to finish the paper along with my presentation. In terms of my visualization, I had the case study visualization of one house changing in size and color, but at my mentor meeting we discussed the significance of the color and other possibilities. I ended up making two other versions of the visualizations using different colormaps representing the racial make-up of the communities.

Final Report

Hunter College
City University of New York
695 Park Ave
New York, NY 10065

Telephone: +1 (212) 396-6837
Email: oo700 at hunter dot cuny dot edu

Follow me:
wolexvr on Twitter  wole-oyekoya-5b753610 on Linkedin  woleucl's YouTube channel  oyekoya's Github profile  Googlescholar