Sound-to-Touch Sensory Substituion in Java
Tactile Waves is an open source digital signal processing (DSP) library for sound-to-touch sensory substitution research and development. It has been designed to enable mobile devices to be used as the 'back end' of a sound-to-touch sensory substitution system. Using Tactile Waves, audio can be acquired and preprocessed, and important features can be extracted and sent to sensory substitution hardware worn on the body via Bluetooth.
Tactile Waves can be useful even if you are'nt interested in sensory substitution. As required by sensory substitution applications, Tactile Waves provides a plethora of speech processing functions and audio stream management. These can be used to build audio analysis and feature extraction applications for Android and Java such as a Pitch Detector or Formant Frequency Estimator. Tactile Waves could also be used as a preprocessor and feature extractor for an Automatic Speech Recognition (ASR) system. The possibilities are endless, and it is my hope that users will be encouraged to push the boundaries of the library (and contribute their extensions!).
Tactile Waves is available as a remote Android Library, as well as a standard JAR file.
If you are developing an Android Application, the easiest way to use Tactile waves is to add a dependency to the remote library.
Add the following dependency to your apps build.gradle file:
dependencies {
implementation 'com.funkatronics.code:tactilewaves:1.0.0'
}
Tactile Waves provides a system for handling audio recording, buffering, windowing and processing in a few easy to use objects within the tactilewaves.dsp package.
The WaveManager object represents the core of Tactile Waves' audio engine. It reads audio samples from a WaveInputStream, and generates a WaveFrame for each frame of audio acquired. Each WaveFrame is passed through a chain of WaveProcessor objects that perform some useful processing on the frame, and optionally store extracted audio features in the frame's feature set. Processed frames are then sent to any WaveListener objects that have been added to the WaveManager. These listeners can be used to trigger UI updates, or send extracted audio features to sensory substitution hardware.
WaveInputStream is an abstract class that contains an underlying InputStream to acquire audio bytes from a stream and convert them audio samples. A WaveManager object is used to read a WaveInputStream and generate windowed frames of audio.
The WaveFrame object represents a single frame of audio. A WaveManager will create WaveFrame objects according the the buffer and overlap length provided. Each WaveFrame is then passed through the WaveManager's processing chain
Tactile Waves uses the WaveProcessor interface to build objects that can be inserted into a WaveManager's processing chain. Any object that implements this interface can be added to the chain to perform some type of processing or analysis on each frame of audio.
After a WaveFrame has been passed through the entire processing chain, it is sent to any subscribed WaveFrameListener objects. This can be used to trigger events that are dependent on processed audio frames such as UI updates or Bluetooth transmission.
A WaveManager is initialized with a WaveInputStream to read audio from, along with a buffer and overlap length.
To obtain a stream from an Android device's microphone use:
WaveInputStream inputStream = new AndroidWaveInputStream(bufferSize);
A WaveManager can then be instantiated:
WaveManager waveManager = new WaveManager(inputStream, bufferSize, overlap);
A newly instantiated WaveManager will have an empty processing chain. To add processing to the chain, use the addEffectToChain method to add any object that implements the WaveProcessor interface. Tactile Waves provides several ready to use processors, and users can easily build their own.
For example, to estimate the pitch of incoming audio, a PitchProcessor can be used. Add a new PitchProcessor the the Wave Manager's processing chain:
waveManager.addEffectToChain(new PitchProcessor(bufferSize));
Add any listeners to the WaveManager using the addListener() method:
waveManager.addListener(listener);
The WaveManager is now ready to be started. Calling start() will start processing audio in a new thread.
waveManager.start();
Calling stop() will stop this thread.
waveManager.stop();
Tactile Waves has been tested on a Moto G5 Plus running Android 7.0 on a Snapdragon 625 as well as a Windows laptop with an Intel i7-8550U processor. Real-time operation has been confirmed on these devices. More Android device tests will be performed in the near future, but any modern Android device should be more than capable of processing audio in real-time with Tactile Waves.
The built-in StopWatch class can be used in a WaveFrameListener object to measure the elapsed time between completed audio frames, to ensure real-time operation on your device/system.
Version 1.0.1 - 2018-03-22
Minor changes and fixes, see the changelog for more info
Version 1.0.0 - 2018-03-09
First public release
This software is licensed under the GNU General Public License - see the license.txt file for details
This project was funded by a Creative Works Award from CSU Ventures.
Joren Six for his TarsosDSP Library that provided an excellent learning resource in the early days of my thesis
JJ Moritz at Sapien, LLC for bringing me into CSU's sensory substitution research and providing assistance and resources throughout the project
Dr. Leslie Stone-Roy and Dr. John Williams at Colorado State University for their advice and guidance
Drop me a note:
Denver, CO
funkatronicsmail@gmail.com