Houndify React Native

This module allows you to make requests to the Houndify API from a React-Native application. To use it, you must set up this library on your React-Native app as well as houndify on your server. We'll explain both parts below.

NOTE: This is in beta.

If you are building on iOS: You must be using react-native <= 59.8.X

This library:

• Supports streaming! This means your users will see their speech as they talk, without waiting for a whole audio file to send.
• Supports TTS (Text to Speech) This means your app will be able to speak to your user out loud.
• Works hand in hand with Houndify's main sdk, houndify.
• Requires houndify >= 3.0.0 on your server. If you don't know what we mean by this, keep reading the next paragraph :).

Server Setup

We recommend you set up your proxy server first. This isn't too bad, especially if you are familiar with express. This way, you can test your React-Native app as soon as you get it running.

Proxy server may sound scary, but to be honest, we think you can set this up in under 30 seconds. Ready?

index.js

const express = require("express");
const app = express();
 
const expressWs = require("express-ws")(app);
const {createReactNativeProxy} = require("houndify").HoundifyExpress;
 
app.use("/houndifyReactNativeProxy", createReactNativeProxy(express, 'CLIENT_ID', 'CLIENT_KEY'));
 
app.listen(3000);

That's your proxy server. Put that on a machine, add in your Client ID and Client Key from houndify.com, run it with node index.js and you're done.

By default, express will run this on http://localhost:3000. Wherever it runs, make a note of the host and port. For example, localhost:3000. You'll need this when setting up the client.

If you are adding this to an existing express server, the important parts are:

1. Importing express-ws and initializing it with app.
2. Importing createReactNativeProxy from houndify.
   1. If you're using other parts of houndify in your app, we trust you know how too include this module as well.
3. Use app.use to forward all requests to /houndify-proxy to this new proxy.

Explanation

This library uses 'websockets', these allow efficient two way communication between a server and a client. express-ws allows express to use these websockets.

All the heavy lifting here is done in createReactNativeProxy. You just need to pass in a clientId and clientKey, and tell express to send it requests.

We take express as an argument because the express-ws library modifies the express object. Sometimes, re-requiring express in another file doesn't have these modifications, so for safety we just use the same object.

TIP: Test an app using a server on your computer.

You may not want to deal with uploading your server to a remote host--when you're working quickly and testing, running on localhost is by far the simplest way to develop. Using a tool such as ngrok let's you run a local server which can be accessed through a normal URL.

This way you can test houndify-react-native in an app while running a server on your computer.

React Native Installation

To use this module in your react-native app, go to your app's root directory and use the following commands.

1. npm install houndify-react-native --save or yarn add houndify-react-native (Based on your system you may need to run as sudo)
2. react-native link react-native-audio-record
3. react-native link react-native-tts

Houndify React Native depends on some native (ios and android specifc) modules for audio, using react-native link makes sure these are configured properly.

Remember to add Internet and Microphone permissions to your application. You may need to request permissions in your app, or manually approve them from settings.

Known Installation Issues

Sometimes the react-native build system has issues with native modules. Your app may fail to build, with errors such as:

1. XCode build error 65
2. node-gyp errors
3. Could not mkdir...Usually involving fsevents.

if these or other errors occur, the following commands usually help.

1. npm i -g node-gyp node-gyp is needed when working with some native modules.
2. sudo node-gyp rebuild -g --unsafe-perm Helps node-gyp.
3. sudo chmod -R 777 node_modules This makes sure any build tools have write access to the node_modules directory.
4. npm install

Depending on your setup, you may need to run these as sudo. If something goes wrong, use sudo rm -rf node_modules to remove node_modules, and try issuing the above commands again.

Usage

There are a couple ways to use this package. You can either use the HoundifyReactNative class directly, or use the HoundifyComponent wrapper we've built. The wrapper removes some boilerplate, so we recommend you use it unless you have a reason not to.

HoundifyComponent (Recommended)

This is the simplest way to setup HoundifyComponent. We'll explain each part below, but there are a couple of important parts:

1. Props (e.g. host)
2. Render Props (e.g. transcription, writtenResponse). Pay close attention to the syntax here.

<HoundifyComponent
    host="ws://localhost:3000"
>
    // These are "render props". They are described below.
    {({transcription, writtenResponse}) => (
        /**Your Components Here*/
    )}
</HoundifyComponent>

Props

Only the host prop is shown, but more are available.

Available Props

Property	Type	Required	Description
host	string	Yes	The websocket host of the proxy server being used. This is described more in the SDK, but basically, if you are running a server at http://example.com, this will be ws://example.com. Notice there is no trailing slash.
tts	boolean	No, by default it is true.	If true, HoundifyComponent will read responses out loud to your user. It uses the spokenResponse property, which is also passed through render props.
requestInfo	Object	No	See the houndify SDK for more info. This lets you provide metadata such as location and device information.
conversationState	Object	No	This allows you to keep track of context over the course of a conversation. If not supplied, it will be automatically managed. This is reccomended.
Hooks: These usually aren't needed, but notify you when certain events take place.
onStart	callback	No	Called when recording begins.
onStop	callback	No	Called when recording ends.
onTranscription	callback (transcription:string) => void	No	Called when a new partial transcription is available. This is also passed in the transcription render prop.
onResponse	callback (response:object, writtenResponse:string, spokenResponse:string, conversationState:object) => void	No	Called when a new response is available. This is also passed in render props.
onVolume	callback (volume:float) => void	No	Called when a new volume is available. This is also passed in render props.
onError	callback (error:object) => void	No	Called if an error occurs. This is also passed in render props.
getRecorder	callback (recorder:HoundifyReactNative) => void	No	Provides access to the HoundifyReactNative class being used internally

Why do Hooks exist?

Ideally, any part of your app that directly depends on the houndify recording (e.g. showing transcription, showing response) is inside HoundifyComponent. But, sometimes you may need to access a piece of information from outside of your app's render function.

For example, say you want to show an animated readout of the volume. Without going into the details of React Native Animation, this may require some setup in other parts of your component. For this case, you could use the onVolume callback to store volume in global state.

Or perhaps you want your back button to automatically end recording. You could use getRecorder to store the recorder and call recorder.abort() automatically when the user exits. Hooks provide flexibility.

Render Props

What are Render Props

HoundifyComponent uses the HoundifyReactNative API to internally keep track of state regarding a user's request. It needs to pass this state into child components, so that your app can access it.

This is how child components are usually written:

<Parent>
    <Child/>
</Parent>

This is how they are written with render props:

<Parent>
    {(arg1, arg2...) =>
        <Child/>
    }
</Parent>

Here, Parent's direct child is not <Child/>, but instead a function that returns <Child/>. Parent, in our case HoundifyComponent, calls this function and uses it to pass in important state information.

Render Props in HoundifyComponent

transcription and writtenResponse are examples of properties that HoundifyComponent keeps track of and passes into your app. When a user makes a request, these properties are updated periodically and provided to any components your write. (Review the syntax above and note the use of object destructuring)

Available Properties

Property	Type	Description
transcription	string	The current transcription of the user's speech. This will update periodically while a recording is being made.
response	JSON Object (HoundServer object)	The full response from Houndify. See the Houndify Web SDK and for more details.
spokenResponse	string	What your app should 'say' to the user, if this functionality exists. This is contained in the response object, but has been extracted for convenience.
writtenResponse	string	What your app should 'write' to the user, if this functionality exists. This is contained in the response object, but has been extracted for convenience.
isRecording	boolean	Whether the app is currently recording.
start	callback	Calling this function will start recording.
abort	callback	Calling this function will stop recording.

How do I show Cards with information based on the response?

If you've used any virtual assistant, you've seen cards. For example, context specific UI for weather, stocks, information, and so on. The response render prop contains all the information you need to show these custom UI elements. See the Houndify Web SDK or HoundServer object specification for more information on how to understand this response.

Also, feel free to start console.log-ing and explore for yourself! Hint: Look for the AllResults, InformationNuggets, and NuggetKind properties in the response. Generally, InformationNugget -> UI Element.

A simple component implementing HoundifyComponent

The following is a simple component like what you may need to build for your app.

It uses two text fields to show the results of a query, and a button to toggle recording component.

import React, {Component} from 'react';
 
class MyReactNativeComponent extends Component{
    ...
    render(){
        return(
          <HoundifyComponent host='ws://example.com'>
              {
                  ({transcription, writtenResponse, isRecording, start, abort}) => (
                     <>
                     <Text>{transcription}</Text>
                     <Text>{writtenResponse}</Text>
                     <Button onPress={isRecording ? abort : start} title='Click Me!'/>
                     </>   
                  )
               }
          </HoundifyComponent>
        )
    }
}

HoundifyReactNative

Of course, you can also use this class directly instead of the React component. Most of the features are very similar to above, it just involves a bit more boilerplate.

 
class MyReactNativeComponent extends Component{
    constructor(props){
        ...
        this.recorder = new ReactNativeComponent({
            ...props
        })
    }
 
    ... somewhere else in the component
    this.recorder.start() // Starts recording.
    this.recorder.abort() // Prematurely ends recording.
}

Here, props is exactly like the props for HoundifyComponent. (Except for tts). You can see that even this way is not too complicated, but you must use the hooks provided through props to keep track of response, transcription, etc yourself. Using this class directly does not provide the automatic state management of HoundifyComponent. For most applications, you are best of using HoundifyComponent and using the provider hooks to give other parts of your app access where needed.