The Next Challenge

Challenge 01

If the two performers sit next to each other in the real and see each other next to each other in the VR world (only at the beginning) too. How would I unlock the HMD presence in VR to not be dependent on the user’s head?

  • Because I want to screenshot the phone’s screen and place these screenshots around each performer. And I want each screenshot to have colliders/physics engine. These screenshots will act like walls that are generated by touch gestures. Therefore, I need to find a way to unlock the VR HMD presence in VR. In VR, each performers will end up being far away from each other. In real life, they will still sit next to each other when they took out the headset.


Challenge 02

How to add the second and third and so-on plugins into your working project?

I can add the first plugins to the project file by creating a C++ project. But for the second and third, I couldn't do it and I ended up recreating the whole project file from scratch and re-added the plugins at the same time.


Challenge 03

UE Blueprint

Instantiate object (wall/rectangle with colliders/physics engine) from touch gesture and place the object around the performer with materials that is captured from the performer's phone screen.

Multiplayer Demo

The past week, I've been trying to figure out how to do multiplayer in VR and UE (Unreal Engine). There are 2 ways I can do this.

  1. Plan A: Follow UE Tutorials step by step and make Multiplayers Blueprints
  2. Plan B: Used the existed Template.

Plan A : Follow UE Tutorials step by step and make Multiplayers Blueprints

I decided to go with Plan A before I'm going to use the template because I wanted to learn the software, UE. I've been watching and following Official UE tutorials online [ Link ]. Although they didn't cover the VR part, I think they did a very good job explaining the basics of how it works. Even though I understand maybe 20% of it all. But, I learned a lot from this tutorial.

After following the UE tutorials Link (Blueprint Multiplayer), I had this up and running. I managed to spawn 2 players from 2 PCs. One is a host and another is a client. They both used the same file.

And following more from the UE tutorials, I think I did something went wrong along the way because I'm stuck on the "Loading Page".... and I'm not able to spawn inside the scene anymore or I am spawned in the scene but the Loading screen won't disappear.

 I've HIT a ROAD BLOCK....
So, let's move to Plan B --->


Plan B : Used the existed Template : Github Link

It took me the the whole day from 9am - 9pm to make this template work. There are so many things I tried and figured it out like:

  • I cannot used the NYU wifi network. I have to get into  itpsandbox. (for both PCs)
  • But, I could not get into ITP wifi sandbox for both PCs....but luckily I found ethernet LAN Lines that are both itpsandbox. After I tried so many times with normal NYU LAN. The LAN has to be itpsandbox.
  • Both host and client PC cannot log in to Steam account while using the template. Otherwise, the client won't be able to join and spawn into the scene while the host can.

Succesful Multiplayer Test

Yayyyyyyy!!!!! In this video, you can see that they are looking in the mirror in VR and standing next to each other but in the real world they are actually behind each other. You can see more clearly when they're pretending to fight. This happened because I did not do the room set up properly and I point the Vive controllers to the monitors at different location.

Grab Objects

Macarena Dance

Network Connection MUST be itpsandbox

 

Unsuccessful Multiplayer Connection Test

The Left PC is the Client. The Right PC is the Host.  This unsuccessful connection happened when I'm signed in to Steam account and not in the same itpsandbox LAN connection.

Websocket - Unreal Engine

This week I've been busy with trying to send live data of touch gestures into UE(Unreal Engine). Trying to recognize mobile gesture input (tap & swipe (Left, Right, Top, Bottom)) while using a native social media app.

Challenge:

It is very extremely difficult to do this. I'm still a noob in coding skill and javascript and knowledge of mobile operating systems and devices (iOS & Android). Not to mention security issues.

Solution:

After talking to many of my friends about my thesis nonstop for the whole week about this problem I have, bits and pieces of solutions seemed to slowly fit together. So for my performance piece, instead of the 2 users are using their phone through a native app, they would have to use their phone through a web browser like chrome. I believe I would need to implement these elements together:

Demo 01 - Socket.io + hammer.js + Touch Gestures

Github - Code test touch gestures with server

Touch Gestures break down:

  • Pan Left = 1
  • Pan Right = 2
  • Pan Up = 3
  • Pan Down = 4 
  • Tap = 5

With Wippy's and Yuli's help, I'm able to get 5 touch gesture recognized from mobile through web server.

Touch Gesture hammer.js Socket

Touch gestures test + hammer.js + socket.io

Finding the IP address in windows (took me longer than I expected...)

Demo 02 - Socket.io + hammer.js + Touch Gestures + Unreal Engine

Building on top of Demo 01, I was able to send one gesture in at the moment. I'm getting familiar with Unreal Engine's Blueprint to recognize all of them.

Touch gesture number 2 = pan right

Demo 02 - UPDATE - Socket.io + hammer.js + Touch Gestures + Unreal Engine [Toggle Light]

One more challenge

I have to find a way to make chrome extensions or firefox add-ons on mobile. From my research, chrome extensions is NOT available for mobile....

Questions

  1. How to add the second and third and so-on plugins? I can do add the first plugins into the project file by creating a C++ project. But for the second and third, I couldn't do it and I ended up recreating the whole project file from scratch and re-added the plugins at the same time.

Hand Gestures

For iPhone

1. Touchpose - Github 

2. Fingertips - Github

For Android

  • Settings > Developer Mode > Tap Build Number 7 times > Input > Show Taps & Pointer Location

Thesis Description

Descriptions

Technology is changing our lives and impacting human behavior in ways we have never seen before. Our relationship with one another has changed as it is no longer bounded in one contiguous space but by the reach of our technology network. We experience this interconnectedness through numerous applications on our smartphones. We constantly stare, tap, and swipe without realizing the consequences and the implications of these actions. However, I believe the more we use our phone to stay connected, the less we are connected in the real world. 

My project aims to focus on these mundane behavior and often normative behavior (tap, swipe, scroll) and attempt to translate these physical interactions into virtual parameters via Virtual Reality Technology. A parody of repeated physical interaction we do every day without thinking. 

  • Log in - Tap App - Tap Swipe Tap - Lock the phone. REPEAT
  • Wake up - Shower - Eat - Work - Sleep. REPEAT

I wish to translate this cause and effect in VR that one tap means more than just a simple visual and sound feedback/response you see on screen. You don’t see it but our action/input with our smart devices, this electronically networks establish a new type of relationship to the widely scattered people and places we care about. Networks connect and propagate the effects of our action. Today we can do unto others at a great distance, and they can unto us. I’m attempting to visualize these hidden connections we all have and that smartphones make us closer to people they are far away from, but farther from the people they’re close to.

The project is a multi-performance and game-like storytelling experience of multiple users (2 users) where they are the creator of the virtual world. The experience requires each user’s physical input/interaction into virtual parameters via VR controllers (I want to use smartphones instead of VR controllers). At the beginning of the experience, they will physically sit next to each other and once they put on the VR headset, they will see a white world (a blank canvas) with their phones’ screens. The more they use (swipe, tap, scroll) their smartphone in VR, the more content (wireframes) will be generated and eventually they can no longer see each other in VR because they would feel a gradual shift away from each other during the game.  

For now, I will focus on 2 people as my user subjects. They are friends and use their phone often. I will use a Moment App to collect and log data for one week. The Moment App tracks what apps my users used the most and how much time they spend connected.  The VR environment content will depend on the what App my users used the most. I’m thinking of using existed UX/UI wireframes that procedurally created by user’s input(tap/swipe/scroll). After I’ve collected what App they used the most, the test user subjects will have to play my VR game too.

 

Personal Statement

As a trained Architect, I have always had a keen awareness of our surroundings and have always designed for the built world. Data and information gathered from physical systems and sensors help architects to design smart homes and eventually smart cities. I feel like architect’s role in the new digital age will not only shape, arrange, and connect spaces in the physical world to satisfy human needs. But we need to be able to speculate and be aware of the connected space in the virtual world too because networks have become part of our daily life.

In a rhizomatic way, humans have a symbiotic relationship with technology and similar to how every human is created uniquely. The relationship with our phones is also uniquely different. We are co-creating and uniquely tailoring our personal space in the digital world that is customized to our needs and interests.

"If we understand what is happenings, and if we can conceive and explore alternative futures, we can find opportunities to intervene, sometimes to resist, to organize, to legate, to plan, and to design”, William J Mitchell

Design Elements

1. Environment

I was inspired by the concept of a Never Ending Running game like "Minion Rush". I love this idea of the never-ending game because it's a metaphor of how we go through life day-in/day-out in a loop like an endless cycle of life.  Wake up - Shower - Eat - Commute - Work - Eat - Commute - Eat - Shower - Sleep - REPEAT. 

2. Interaction

The content that will be constructed will be based on Touch Gesture Interactions of the users with their mobile phone (VR Controllers).

  • SWIPE
  • TAP
  • TYPING

3. Environment Contents

   3.1. iPhone UX/UI Interface

  • The UX/UI interface is based the data I collect from my user subjects.

___ To be continued ___

 

   3.2. OpenStreet Map

  • I've been looking into a quick and easy way to generate city or things that resembles city and I came across this OpenStreet Map Plug-in. I'll test this out soon. Check out the Github page.
  • I'm thinking of importing only the street component to be used in my project as road tracks so that I can define and limits my user/player routes. Otherwise, they will move all over the place in a vast space of VR world. 

___ To be continued ___

 

 

Data Log

  • I will be using an App called Moment to log and collect their relationship with their phones and with each other. Moment App tracks how much my user subjects use their phone each day for a week.
  • I will have 2 user test subject for my performance.
  • The goal for this is to find out which app these 2 users have in common.
    • Then (I think) I will use the top 5 apps as my main content in creating the VR world.
    • For Example, if the top 5 apps are Instagram, Messages, Gmail, WhatsApp, Facebook; These UXUI wireframes will be procedurally generated in my world.

Color Pallete

I came across this blog when I was researching on picking color palette for my experience. I find it quite useful.

I want my experience to start off with a white world like a blank canvas and keep a monochrome or limited tone. I'd like to keep it minimal and highlight main events for dramatic effect. 

Possible Highlighted Color

  • Way Finding - Like in "Super Hot!"
  • Triggered events
  • Lighting Cues

Tools I used to pick color scheme are:

Only God Forgives

  • Red = emotions at a heightened level, saturated to convey that through-line of tension from all ranges of the characters' feelings.
  • Blue = a somber tranquility almost - characters are left to linger in their worlds to think in the blue that overwhelms them.
  • Yellow = a culmination of the previous two - where their collisions are what make up the frame.

Sin City

  • I really like high contrast with limited color used.
  • My world would be similar in color usage but will be heavy on white than black