This week I've been busy with trying to send live data of touch gestures into UE(Unreal Engine). Trying to recognize mobile gesture input (tap & swipe (Left, Right, Top, Bottom)) while using a native social media app.
Challenge:
It is very extremely difficult to do this. I'm still a noob in coding skill and javascript and knowledge of mobile operating systems and devices (iOS & Android). Not to mention security issues.
Solution:
After talking to many of my friends about my thesis nonstop for the whole week about this problem I have, bits and pieces of solutions seemed to slowly fit together. So for my performance piece, instead of the 2 users are using their phone through a native app, they would have to use their phone through a web browser like chrome. I believe I would need to implement these elements together:
- socket.io
- chrome extension or firefox add-ons
- hammer.js - Add touch gestures to webpage
- socket.io plugins for Unreal Engine
Demo 01 - Socket.io + hammer.js + Touch Gestures
Github - Code test touch gestures with server
Touch Gestures break down:
- Pan Left = 1
- Pan Right = 2
- Pan Up = 3
- Pan Down = 4
- Tap = 5
With Wippy's and Yuli's help, I'm able to get 5 touch gesture recognized from mobile through web server.
Touch gestures test + hammer.js + socket.io
Finding the IP address in windows (took me longer than I expected...)
Demo 02 - Socket.io + hammer.js + Touch Gestures + Unreal Engine
Building on top of Demo 01, I was able to send one gesture in at the moment. I'm getting familiar with Unreal Engine's Blueprint to recognize all of them.
Touch gesture number 2 = pan right
Demo 02 - UPDATE - Socket.io + hammer.js + Touch Gestures + Unreal Engine [Toggle Light]
One more challenge
I have to find a way to make chrome extensions or firefox add-ons on mobile. From my research, chrome extensions is NOT available for mobile....
Questions
- How to add the second and third and so-on plugins? I can do add the first plugins into the project file by creating a C++ project. But for the second and third, I couldn't do it and I ended up recreating the whole project file from scratch and re-added the plugins at the same time.