The "Vibey Transcribey" - Dom's dissertation

Hello, friends! For quite some time now I have been hiding away while working on my dissertation in Human Factors Psychology. Inspired by wisdom from Universal Design principles and the social model of disability, what I’ve created is a wearable device that converts components of speech to vibrating patterns presented on the arm, a novel encoding scheme to do that conversion, training software in the form of mini-games played on a PC, and a set of experiments to test the whole system.

I’ve been eager to share it here, but I can’t share too many details yet, as the extra knowledge could influence the results if any of my participants happen to read into what I post. I can share videos of the training games once all of my participants have been briefed, and even more details once data collection is complete if anyone is interested. I’ll keep this thread updated with details as I can share them. For now, I can share this:

The device, which I’m calling the “Vibey Transcribey” is based on the ESP32 and uses 16 LRAs. Vowels and consonants are rendered to different locations on the forearm, pitch is rendered to the bicep, and loudness moderates the intensity of the vibrations. In its current form the device only works with pre-processed speech. The speech-processing software is mostly written in Python and the training games were created with Unreal Engine 4. The device interacts directly with the training software, so you could call it a gaming peripheral. The PCBs were designed with LibrePCB, and fabricated/populated by PCBWay. Enclosures, jigs, and other parts were designed FreeCAD and Blender and 3D printed on my Prusa i3 MK3S.

When the project is completed, everything will be shared on the project’s Open Science Framework page. This includes the speech processing software, the experiment’s evaluation software, CAD designs, electronics schematics, statistical analysis code, etc. I intend to submit for publication, although I haven’t settled on any journals yet.

Now that I have approval from the university and my dissertation committee, I can finish building devices and collecting data, which is why I’m posting today. I have already recruited a few participants, but need more volunteers - twelve total. All of the participants will do a pre-test and, six weeks later, a post-test. Half of the participants will take one of my devices home with them along with the training software, while the other participants will receive no training and serve as a control group. Participants who are assigned to the trained group will be expected to play the training games at least 2-3 times per week for the six week period between the pre- and post-tests, and each training session should last at least 30 minutes.

To participate, you must:

  • Be at least 18 years old
  • Be a native English speakers
  • Not be fluent in any other spoken language
  • Not have any diagnosed language disorders
  • Be right-hand dominant
  • Be at least a moderate video-game player
  • Have access to a modern gaming PC with a controller

If anyone here is interested in being a participant, I have a brief screener survey that you will need to complete. I was fortunate to receive a small amount of funding, but not enough to cover all of the hardware costs, so I’m unable to offer any compensation for participation other than my eternal gratitude and the opportunity to be a part of (what I consider to be) pretty cool science. Fair warning though, the pre-test and post-test are tedious and each take 2 hours. If you are assigned to the control group, you’ll only experience the pre- and post-tests and none of the training games. Please do not apply if you are not willing and able to be in either group.

14 Likes

That’s pretty cool…I hope it turns out to be extremely useful for those who can benefit from it. I especially like the name – a classic “Domism.” Keep us posted…I look forward to seeing a lot of mention down the road on sites like Hackaday, Engadget, etc.

I’ll be able to say “I knew him before he became famous!”

P.S. “Dr. Dom” has a nice ring to it…

3 Likes

Super cool. I would participate, but I don’t qualify as a moderate gamer… if you need a poor gamer in a different part of the study, flag me down. :slight_smile:

3 Likes

I’ve participated in an earlier set of trials for an unrelated project Dom was involved with. I highly recommend that if you’re interested that you volunteer.

Although I say unrelated, the other project also addressed a way to allow differently abled people to more fully participate in common activities.

3 Likes

I should’ve specified more clearly that by “moderate gamer” I mean in time spent playing video games, not skill level

2 Likes

Does it work with cellphones?

Cuz there is no tactile feedback on my emulator

It can work over USB, wifi, or Bluetooth. For my study though it’s USB only to ensure reliable power and data.

If the hardware you’re emulating has no tactile feedback, why would an emulator? lol

Funny you ask though - for the first version of the Vibey Transcribey, I implemented an Android TTS system. Well, Android sees my software as a TTS engine, but instead of synthesizing speech, it generates the vibrations on the device. This allowed every app that could be used with a TTS to work with the Vibey Transcribey, which was pretty cool. The current version of the hardware and encoding scheme is a bit more complicated than that first version, so that doesn’t work anymore, but it could. For now though, my scope is focused on what’s necessary to complete the dissertation.

2 Likes

my scope is focused on what’s necessary to complete the dissertation.

Good plan. Not sure I could ever focus that long on one project. Mad props.

1 Like

I don’t know how public you want the Vibey Transcribey to be at this point, but there is this at Hackaday: Show Us Your Odd Inputs And Peculiar Peripherals! | Hackaday

2 Likes

I saw that, but the prize is pretty small for the effort required. The paper I’m writing is 80+ pages so far and I haven’t even collected any data yet. Organizing and formatting everything for HaD feels daunting. If the contest was a few months later, I probably would put in the energy just for the cool-factor of it, but I don’t even have the mind-space for it right now lol

4 Likes

@dom When I first read this, I really wanted to participate. But I don’t fit the criteria. :pleading_face::pensive: Seems really heckin’ awesome, though! You’re doing great!!!

1 Like

Thanks, I really appreciate the kind words!

Finding participants has been a little tricky. If the devices were cheaper/easier to make, I could have more participants and could then relax the participation requirements a little because I could handle confounding variables better. I still have a few more to make, so hopefully I’ll be able to find more participants soon! :crossed_fingers:t4:

3 Likes

Hello again!

My doctoral defense is officially scheduled for next Tuesday, Nov 29th at 8am Central. It will be online over Zoom, and you’re all invited.

If you’re interested in attending let me know via reply or DM and I’ll send you the online meeting info.

5 Likes