# HLA Files (Android)

# Concept

HLA (Hapticlabs Android) files are used to express haptic signals in a format that corresponds to Androids VibrationEffect.createWaveform.

In addition, HLA files define which audio signals are played corresponding to the haptic signals, if any.

Furthermore, these files contain metadata regarding the haptic signal, such as its title and duration.

# Structure

HLA files are JSON files that contain the following fields:

ProjectName [string]: The title of the haptic project.
TrackName [string]: The title of the haptic track.
Duration [number]: The duration of the haptic signal in milliseconds.
Timings [array<number>]: An array of integers that define the timing of the haptic signal as required by VibrationEffect.createWaveform.
Amplitudes [array<number>]: An array of integers, range 0-255, that define the amplitude of the haptic signal as required by VibrationEffect.createWaveform.
Repeat [number]: The number of times the haptic signal is repeated or -1 to indicate no repetition, as required by VibrationEffect.createWaveform.
RequiredAudioFiles [array<string>]: An array of strings that define the audio files that are referenced by this HLA file.
Audios [array<AudioFile>]: An array of objects that define the audio files that are referenced by this HLA file:
- AudioFile [{Filename: string, Time: number}]: An object that defines an audio file and the timestamp when it is played, relative to the start of the signal.
  - Filename [string]: The relative path (from the .hla location) to the audio file.
  - Time [number]: The timestamp in milliseconds when the audio file is played.

# Playback

# React Native

The react-native-hapticlabs npm package provides a simple way to play back HLA files on Android devices:

import { playHLA } from 'react-native-hapticlabs';

// Play an OGG file with haptic feedback
playHLA('path/to/file.hla');

# Kotlin

To play back haptic signals encoded in HLA files using Kotlin, you can use the following code snippet (see the implementation for the react-native package):

val data: String

try {
  val file = File("path/to/file.hla")
  val fis = FileInputStream(file)
  val dataBytes = ByteArray(file.length().toInt())
  fis.read(dataBytes)
  fis.close()
  data = String(dataBytes, StandardCharsets.UTF_8)
} catch (e: IOException) {
    e.printStackTrace()
    promise.reject("Error reading file", e)
    return
}

// Parse the file to a JSON
val gson = Gson()
val jsonObject = gson.fromJson(data, JsonObject::class.java)

// Extracting Amplitudes array
val amplitudesArray = jsonObject.getAsJsonArray("Amplitudes")
val amplitudes = IntArray(amplitudesArray.size())
for (i in 0 until amplitudesArray.size()) {
    amplitudes[i] = amplitudesArray[i].asInt
}

// Extracting Repeat value
val repeat = jsonObject.get("Repeat").asInt

// Extracting Timings array
val timingsArray = jsonObject.getAsJsonArray("Timings")
val timings = LongArray(timingsArray.size())
for (i in 0 until timingsArray.size()) {
    timings[i] = timingsArray[i].asLong
}


val durationMs = jsonObject.get("Duration").asLong

val audiosArray = jsonObject.getAsJsonArray("Audios")

// Prepare the vibration
val vibrationEffect = VibrationEffect.createWaveform(timings, amplitudes, repeat)
val vibratorManager = reactContext.getSystemService(Context.VIBRATOR_MANAGER_SERVICE) as VibratorManager
val vibrator = vibratorManager.getDefaultVibrator()

// [Optional] Prepare the audio

// Execute the vibration
vibrator.vibrate(vibrationEffect)

Playing audio files in sync with the haptic output of Vibrator.vibrate is not straightforward on Android. We found that AudioTrack provided the best results for synchronizing audio and haptic signals because the audio signal can be preloaded and subsequently played back with relatively low (however still nonzero) latency. This can be implemented as follows (see the implementation for the react-native package):

/* A class to handle the audio playback */
class AudioTrackPlayer(private val filePath: String, private val reactContext: ReactApplicationContext) {
    private var audioTrack: AudioTrack? = null
    private var extractor: MediaExtractor? = null
    private var codec: MediaCodec? = null
    private var inputBuffers: Array<ByteBuffer>? = null
    private var outputBuffers: Array<ByteBuffer>? = null
    private var info: MediaCodec.BufferInfo? = null
    private var isEOS = false

    /**
     * Preload the audio data from the file. This sets up the MediaExtractor and
     * MediaCodec and prepares the AudioTrack for playback.
     */
    fun preload() {
        extractor = MediaExtractor()
        try {
            extractor?.setDataSource(filePath)
            var format: MediaFormat? = null

            // Find the first audio track in the file
            for (i in 0 until extractor!!.trackCount) {
                format = extractor!!.getTrackFormat(i)
                val mime = format.getString(MediaFormat.KEY_MIME)
                if (mime?.startsWith("audio/") == true) {
                    extractor!!.selectTrack(i)
                    codec = MediaCodec.createDecoderByType(mime)
                    codec?.configure(format, null, null, 0)
                    break
                }
            }

            if (codec == null) {
                return // No suitable codec found
            }

            codec?.start()

            info = MediaCodec.BufferInfo()

            // Set up AudioTrack
            val sampleRate = format!!.getInteger(MediaFormat.KEY_SAMPLE_RATE)
            val channelCount = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT)
            val channelConfig = if (channelCount == 1) AudioFormat.CHANNEL_OUT_MONO else AudioFormat.CHANNEL_OUT_STEREO
            val audioFormat = AudioFormat.ENCODING_PCM_16BIT

            // Load the entire audio file into the AudioTrack
            val byteArrayOutputStream = ByteArrayOutputStream()

            while (!isEOS) {
                if (!isEOS) {
                    val inIndex = codec!!.dequeueInputBuffer(10000)
                    if (inIndex >= 0) {
                        val buffer = codec!!.getInputBuffer(inIndex)
                        if (buffer != null) {
                          val sampleSize = extractor!!.readSampleData(buffer, 0)
                          if (sampleSize < 0) {
                              codec!!.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM)
                              isEOS = true
                          } else {
                              codec!!.queueInputBuffer(inIndex, 0, sampleSize, extractor!!.sampleTime, 0)
                              extractor!!.advance()
                          }
                        }
                    }
                }

                val outIndex = codec!!.dequeueOutputBuffer(info!!, 10000)
                when (outIndex) {
                    MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
                    }
                    MediaCodec.INFO_TRY_AGAIN_LATER -> {}
                    else -> {
                        val outBuffer = codec!!.getOutputBuffer(outIndex)
                        val chunk = ByteArray(info!!.size)
                        if (outBuffer != null){
                          outBuffer.get(chunk)
                          outBuffer.clear()
                          // Copy the chunk into the full buffer
                          try {
                              byteArrayOutputStream.write(chunk)
                          } catch (e: IOException) {
                              e.printStackTrace()
                          }
                          codec!!.releaseOutputBuffer(outIndex, false)
                        }
                    }
                }

                if (info!!.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) {
                    break
                }
            }
            codec!!.stop()
            codec!!.release()
            extractor!!.release()

            val fullBuffer = byteArrayOutputStream.toByteArray()

            audioTrack = AudioTrack.Builder()
                .setAudioAttributes(
                    AudioAttributes.Builder()
                        .setUsage(AudioAttributes.USAGE_ASSISTANCE_SONIFICATION)
                        .setContentType(AudioAttributes.CONTENT_TYPE_SONIFICATION)
                        .build()
                )
                .setAudioFormat(
                    AudioFormat.Builder()
                        .setSampleRate(sampleRate)
                        .setChannelMask(channelConfig)
                        .setEncoding(audioFormat)
                        .build()
                )
                .setBufferSizeInBytes(fullBuffer.size)
                .setTransferMode(AudioTrack.MODE_STATIC)
                .build()

            audioTrack?.write(fullBuffer, 0, fullBuffer.size)

        } catch (e: IOException) {
            e.printStackTrace()
        }
    }

    /**
     * Internal method to handle audio playback. This method should be run in a
     * separate thread.
     */
    fun playAudio() {
        audioTrack?.play()
    }
}

// Plays the .hla file at the specified path. Make sure the file is not compressed and its associated audio files are in the same directory.
fun playHLA(path: String, handler: Handler)  {
  val data: String

  // Load the HLA file
  try {
    val file = File(path)
    val fis = FileInputStream(file)
    val dataBytes = ByteArray(file.length().toInt())
    fis.read(dataBytes)
    fis.close()
    data = String(dataBytes, StandardCharsets.UTF_8)
  } catch (e: IOException) {
      e.printStackTrace()
      promise.reject("Error reading file", e)
      return
  }

  // Parse the file to a JSON
  val gson = Gson()
  val jsonObject = gson.fromJson(data, JsonObject::class.java)

  // Extracting Amplitudes array
  val amplitudesArray = jsonObject.getAsJsonArray("Amplitudes")
  val amplitudes = IntArray(amplitudesArray.size())
  for (i in 0 until amplitudesArray.size()) {
      amplitudes[i] = amplitudesArray[i].asInt
  }

  // Extracting Repeat value
  val repeat = jsonObject.get("Repeat").asInt

  // Extracting Timings array
  val timingsArray = jsonObject.getAsJsonArray("Timings")
  val timings = LongArray(timingsArray.size())
  for (i in 0 until timingsArray.size()) {
      timings[i] = timingsArray[i].asLong
  }

  val durationMs = jsonObject.get("Duration").asLong

  val audiosArray = jsonObject.getAsJsonArray("Audios")

  // Prepare the vibration
  val vibrationEffect = VibrationEffect.createWaveform(timings, amplitudes, repeat)
  val vibratorManager = reactContext.getSystemService(Context.VIBRATOR_MANAGER_SERVICE) as VibratorManager
  val vibrator = vibratorManager.getDefaultVibrator()

  // Prepare containers for the audio data
  val audioTrackPlayers = Array(audiosArray.size()) { AudioTrackPlayer("", reactContext) }
  val audioDelays = IntArray(audiosArray.size())

  // Get the directory of the hla file
  val audioDirectoryPath = path.substringBeforeLast('/')

  // Prepare the audio files
  for (i in 0 until audiosArray.size()) {
      val audioObject = audiosArray[i].asJsonObject

      // Get the "Time" value
      val time = audioObject.get("Time").asInt

      // Get the "Filename" value
      val fileName = audioDirectoryPath + "/" + audioObject.get("Filename").asString

      // Create an AudioTrackPlayer for each audio file
      val audioTrackPlayer = AudioTrackPlayer(fileName, reactContext)

      // Preload the audio file
      audioTrackPlayer.preload()

      // Store the AudioTrackPlayer and its corresponding delay
      audioTrackPlayers[i] = audioTrackPlayer
      audioDelays[i] = time
  }

  val startTime = SystemClock.uptimeMillis()

  // Schedule the audio playback
  for (i in 0 until audiosArray.size()) {
      handler?.postAtTime({
          audioTrackPlayers[i].playAudio()
      }, startTime + audioDelays[i])
  }

  // Execute the vibration
  vibrator.vibrate(vibrationEffect)
}

Note: The haptic and audio signals are handled separately when using Vibrator.vibrate. Perfect synchronization can not be guaranteed due to the unpredictable latency of the audio playback. We strongly recommend using OGG files for synchronised haptics and audio playback if your target devices have high haptic capabilities as that approach handles haptic and audio signals as one signal and thus provides the best synchronization.