Project: The Community Game Development Toolkit – Creating easy-to-use tools in Unity to help students and artists tell their story and show their artwork in a game like format.

Mentor:  Daniel Lichtman

About Me: I’m CUNY BA student majoring in Game Design and Entrepreneurship at Hunter College.

Week 1: Project Proposal and Toolkit Exploration

This week, I had the time to explore and delve into the toolkit developed by Daniel Lichtman. This toolkit was specifically designed to aid in various projects To gain a deeper understanding of its functionalities, I decided to open and explore an example project using the toolkit.

 

Utilizing the tools provided by Daniel Lichtman’s toolkit, I created my own custom scene with a custom HDRI and some art assets.

Using these tools I learned how they were supposed to be used and the current process for adding them to a Unity project and making a new presentation with them.

Week 3:

During this third week, I have been engaged in conducting research on VR technology. One significant milestone I achieved was writing the methodology for my upcoming VR research paper.

Additionally, I started work on a VR compatible Unity project where I created an prototype layout for an art project.

Central to this project is the implementation of a VR locomotion system, which aims to enhance user immersion and interaction within the virtual environment. Furthermore, as part of this project, I learned how to directly connect an Oculus Quest 2 headset to the Unity project.

Lastly, I started learning how to use VMD to visualize molecules and atoms.

Week 4:

This week I worked on the methodology section of my paper implementing some of the feedback from my mentor into the paper. In addition, I created a demo level and populated it with art to showcase what a potential art project would look like. It was a great way to bring my project to life within the virtual world. By carefully selecting and placing the art assets, I aimed to create an immersive experience that would captivate the viewer’s imagination. I first found some free assets on the Unity asset store to create a simple courtyard and environment with. I then found some copyright free art to place in the level as resemble what an artist would potentially do in their work.
My level results looks like the following.

To further progress the toolkit, I integrated the Oculus Quest 2 headset. I successfully connected the headset and got the Unity project to work seamlessly inside it. Now, the viewer could step into the virtual environment and feel as though they were physically present within the level.

Week 5:

During the fifth week, my main focus was on developing and adding to the website that contains all the necessary instructions for the VR project toolkit. This involved detailing the setup process and providing step-by-step guidance for implementing the toolkit with screenshots and text. Additionally, I compiled a list of settings for Professor Litchman to incorporate into the toolkit, aiming to get rid of some of the setup steps.

In order to ensure the toolkit’s effectiveness, I sought assistance from artists whom I am acquainted with. I requested their participation in testing the toolkit for the study, and fortunately, three of them agreed to help out.

Aside from the project work, we had an enjoyable experience as a team during this week. We embarked on a delightful river lunch cruise, which granted us the opportunity to admire the captivating Manhattan shoreline and the iconic Statue of Liberty. The lunch was delightful, and it provided a pleasant setting for us to get better acquainted with one another and learn about everyone’s well-being.

Week 6:
During this week, we had the July 4th holiday, which caused a slight delay in sending out the instructions to our case study testers. However, on July 5th, I promptly distributed the instructions and provided a comprehensive explanation to the testers regarding the purpose of the study and the specific objectives of the toolkit being tested.

In addition to providing instructions, I created a survey for the testers. This survey was designed to gather both quantitative and qualitative data, allowing us to gain a better understanding of the testers’ experiences. To ensure best responses, I included open-ended questions, providing the testers with the opportunity to provide detailed feedback.

Lastly, as we decided to focus on a single study instead of two, I dedicated some time to rewriting the research paper that I had previously prepared. This adjustment allowed me to revise the paper for the new study approach, ensuring the coherence and accuracy of the paper.

Week 7:
During the seventh week, my main focus was to provide assistance and guidance to the users/testers of the toolkit as they carried out their testing. I dedicated my time to addressing any questions they had and ensuring a smooth testing process for them.

In addition to supporting the testers, I devoted some time to organizing the structure of my paper. Specifically, I outlined the sections for the User Study, Results and Analysis, Discussions, and Conclusion. This preparation allowed me to establish a clear way for me to present my findings.

As I awaited the availability of the necessary data, I also began working on writing the User Study section of the paper. Since I did not have the data required for Results and Analysis at this point, I focused on writing the details of the User Study itself especially on how it was conducted.

To ensure the effective presentation of the data, I consulted with Dr. Oyewole. Together, we discussed utilizing a tabular format to convey the findings, considering the small number of participants involved in the study. This approach would help provide a concise and organized presentation of the data.

Towards the end of the week, I received the data I needed from the three testers. With this information, I am now ready to proceed with completing the remaining sections of the paper in this final week.

 

Nairoby Pena, Cornell University

Week 1: 

Dan and I met a couple of days before the REU began and we decided that we should start from a broad standpoint and begin to narrow in as time progressed. My first “assignment” was to research physical disabilities, learning disabilities, and mental health (emotional) issues, and the impediments that students face in higher education when they have these disabilities. During this week’s meeting, we became increasingly aware that with just 8 weeks it is unfortunately not possible to have a productive project if we take the angle of aiding mental health issues with VR. We decided that the next “assignment” will be to look for two core classes that each student must take at Hunter and Cornell (four in total) and look for the issues or impediments that a student from each of the disability groups may face in these classes (with a heavy focus on physical and learning disabilities). I will be starting on this assignment to wrap up the week as well as beginning to explore Unity, and starting to think about developing the documentation since I found one or two papers that could be a part of that. By next week we should be working on a concrete methodology for creating or enhancing VR as a tool for the impediments that students with disabilities face.

Week 2: 

This week I presented the Dan about core classes both at Cornell and Hunter which all students have to take and I talked about the impediments that students with disabilities might face in taking those classes. For example, Cornell has a swimming requirement in order to graduate and we talked about the obvious point being that every person is not physically able to jump in the pool and swim. In addition, at Hunter, there is an English course that every student is required to take which involves heavy writing and reading which students with dysgraphia or dyslexia respectively would probably struggle with. There is also an astronomy class that students can choose to take as a part of their core curriculum and unfortunately, a disability might impede a person from taking the course because of the lab component in which 9 out of 13 experiments are self-guided on a computer. Dan brought to my attention that every lab has very high tables that are made for people to be standing at and if we think about someone who might have a wheelchair this could be difficult for them to navigate as they would have to be looking up at their peers making it harder to collaborate. Dan and I also spoke about the research paper where he guided me in understanding what I could mention in my abstract. We noticed that we still do not have a clear methodology but we did talk about the fact that VR heavily exists right now in virtual museum visits and how this is a huge help for people with many different types of disabilities so this might be where the project is headed at this point.

I began to look into Unity and attempted to learn about all of its components that had to do with VR. I built my own VR space and then realized that I didn’t have a headset to try what I created; I also looked into building a microgame. 

Week 3: 

This week I focused on a deep dive into dyslexia, what it is and how it affects students. I looked at what impediments students might face when they have this disability and have to take English 120 at Hunter. I also researched how VR could or could not help students with dyslexia. Most research concludes that more scientific research is needed to prove that VR can help, however, theoretically, it can and there are VR programs that exist for students with dyslexia. After meeting with Dan, we decided that we should not focus on dyslexia because it would be difficult to come up with a project to help in minimal time. Being that there has to be a visualization aspect for the outcome of the project, I received clarity from Dr.Wole and decided that I will be creating a VR “game” or program where autistic students can engage in social modeling. This can include modules on greeting people, engaging in conversation, and how to conduct oneself in various social environments. Since it is week 3 already I will have to work rigorously to develop this. So far I have gone into unity and used the assets in order to build the virtual environment. I will be working to develop a user interface for at least three different social situations and hope to test at least one of them by the end of next week. 

We had a paper writing zoom session this week where I was able to get a draft of my abstract and begin my introduction. I will have to go in and edit this as my focus has shifted from disabilities in general to developmental disabilities (more specifically autism).

Week 4: 

Last week’s idea about social modeling has been modified so that it will be able to get done in the amount of time that is left. I will now be using a museum asset in unity and putting a human-like avatar in the virtual space. The avatar will guide the person that wears the VR headset through the museum and what the avatar does will play on a loop. (Thank you to Dr. Wole for the idea). Applying prior research to this idea, I hope to implement the general outline for social skills teaching packages this means that the program will allow for repetition of the target task (visiting a museum), verbal explanation of the social skill, the practice of skill in realistic settings (in virtual reality), role-play of target behaviors (practicing the behaviors in the virtual reality program without the guided audio).

This week I worked heavily on my paper. I was able to get through most of the introduction, related works, and some of the methodology. I will continue to add to it as the project moves along. Finally, I developed a slides presentation for the midterm check-in.

Week 5: 

This week I was able to get the museum asset in unity. I further developed the idea of social modeling through a recollection of my experience working at the Intrepid Museum with the Access program. One of the events that the program provided was early morning openings for students with disabilities, with a large portion of the population having autism. I decided that I should try to use that as the model in the museum asset. Below is a screen recording of what I have so far. I was able to edit the presentation from what the original asset had to what I wanted it to say. I hope to add audio of my voice so that everything isn’t reading based and I also will be trying to make it possible for the user to look through the museum itself; after this is done I will deploy it to the headset and make edits as needed.

I worked on the citations in my paper and revised it as my ideas developed. I am starting to think about what my results section will look like. We all visited the CUNY ASRC and got to take a tour while learning about their initiatives.

Week 6:

This week I had technical difficulties with adding audio to my visualization portion. Kwame and I spent hours trying to get the code right so that certain audios would play at each slide, however, we could not get it to work. I also looked for existing scripts that may help but they were also unable to work. I created a backup plan to have the audio play on a loop in its entirety, though this would cause people to listen to different audio portions at different points in the experience.

I sent my mentor Dan my paper for feedback and began to work on refining it and next week I hope to begin to finalize it. In addition, Dan gave me the idea to share some of the other existing VR programs that exist to explain in my final presentation so that people understand the extent to which they can be helpful for students with disabilities.

Next week I will be testing my visual aspect from Unity and making any fixes as well as focusing on the paper.

Week 7: 

I got to participate in Talia’s user study this week; it was cool to see how a fellow participant’s visualization aspect came out. Since my project does not have a user study, I focused on what I would write in my discussion for my paper. I met with Dan and he was able to give me more feedback on the paper so that I could continue to work on it. As for the technical difficulties, it seems that I will have to play the looped audio in its entirety due to the diminishing amount of time that is available. Dan and I spoke about the final presentation and what I should include which led me to begin to work on my final slides.

We got to visit the Bronx Zoo on Wednesday where we zip-lined, that was fun! We ended the week with a meeting speaking about how the final week would look and how to prepare.

Week 8: 

I spent the last week submitting my abstract, finishing my paper, and practicing my final presentation. I was able to participate in Kwame’s user study. For the final touch to my visualization portion, I put an animated avatar into the museum to make it seem as though there was a tour guide.

Overall it has been a great learning experience participating in this REU. I am grateful for all that I have learned and all of the obstacles that I was able to surpass in developing this project.

Final Paper