Real-time automatic speech recognition and natural language understanding tools in one flexible API
Website | Docs | Discussions | Blog | Podcast
Speech recognition polyfill
Polyfill for the SpeechRecognition standard on web, using Speechly as the underlying API. The primary use of this library is to enable speech recognition on browsers that would not normally support it natively.
Speechly offers a free tier for its speech recognition API with a generous usage limit.
Useful links
- Quickstart
- Browser support
- Handling errors
- Examples
- Integrating with react-speech-recognition
- Limitations
- Type docs
- Contributing
- About Speechly
Quickstart
Installation
npm install --save @speechly/speech-recognition-polyfill
Basic usage
First, you need a Speechly Application ID:
- Log in to Speechly Dashboard
- Open Create a new application
- Give your application a name and press Create application
- Deploy the application
- Copy the App ID, you'll need it the next step.
Once you have your App ID, you can use it to create a recognition object that can start transcribing anything the user speaks into the microphone:
import { createSpeechlySpeechRecognition } from '@speechly/speech-recognition-polyfill';
const appId = '<your_speechly_app_id>';
const SpeechlySpeechRecognition = createSpeechlySpeechRecognition(appId);
const speechRecognition = new SpeechlySpeechRecognition();
Before you start using speechRecognition
to start transcribing, you should provide a callback to process any transcripts that get generated. A common use case is to match the transcript against a list of commands and perform an action when you detect a match. Alternatively, you may want to display the transcript in the UI. Here's how to set the callback:
speechRecognition.onresult = ({ results }) => {
const transcript = results[0][0].transcript;
// Process the transcript
};
You may also want to configure the recognition object:
// Keep transcribing, even if the user stops speaking
speechRecognition.continuous = true;
// Get transcripts while the user is speaking, not just when they've finished
speechRecognition.interimResults = true;
With your recognition object configured, you're ready to start transcribing by using the start()
method. To comply with rules set by browsers, this must be triggered by a user action such as a button click. For example, in a React component this could look like:
const startTranscribing = () => {
speechRecognition.start();
};
// When rendering component
<button onClick={startTranscribing}>Push to talk</button>
After calling start()
, the microphone will be turned on and the recognition object will start passing transcripts to the callback you assigned to onresult
. If you want to stop transcribing, you can call the following:
speechRecognition.stop();
Browser support
This polyfill will work on browsers that support the MediaDevices and AudioContext APIs, which covers roughly 95% of web users in 2022. The exceptions are Internet Explorer and most browsers from before 2016. On these browsers, an error will be thrown when creating a SpeechlySpeechRecognition
object.
The SpeechlySpeechRecognition
class offers the hasBrowserSupport
flag to check whether the browser supports the required APIs. We recommend you do the following when creating your speech recognition object:
if (SpeechlySpeechRecognition.hasBrowserSupport) {
const speechRecognition = new SpeechlySpeechRecognition();
// Use speech recognition
} else {
// Show some fallback UI
}
Handling errors
A common error case is when the user chooses not to give permission for the web app to access the microphone. This, and any other error emitted by this polyfill, can be handled via the onerror
callback. In such cases, it's advised that you render some fallback UI as these errors will usually mean that voice-driven features will not work and should be disabled:
import { MicrophoneNotAllowedError } from '@speechly/speech-recognition-polyfill';
...
speechRecognition.onerror = (event) => {
if (event === MicrophoneNotAllowedError) {
// Microphone permission denied - show some fallback UI
} else {
// Unable to start transcribing - show some fallback UI
}
};
Examples
The following examples use React to demonstrate how this polyfill can be used in real web components.
Matching commands
A common use case is to enable the user to control a web app using their voice. The following example has a "hold to talk" button that enables transcription while held down. It provides a list of commands that, when matched by anything the user says, will be displayed. In practice, these matched commands could be used to perform actions.
import React, { useState, useEffect } from 'react';
import { createSpeechlySpeechRecognition } from '@speechly/speech-recognition-polyfill';
const appId = '<your_speechly_app_id>';
const SpeechlySpeechRecognition = createSpeechlySpeechRecognition(appId);
const speechRecognition = new SpeechlySpeechRecognition();
speechRecognition.continuous = true;
speechRecognition.interimResults = true;
const COMMANDS = ['PLAY', 'PAUSE', 'REWIND'];
export default () => {
const [matchedCommand, setMatchedCommand] = useState('');
const handleResult = ({ results }) => {
const { transcript } = results[0][0];
COMMANDS.forEach(command => {
if (transcript.indexOf(command) !== -1) {
setMatchedCommand(command);
}
});
};
useEffect(() => {
speechRecognition.onresult = handleResult;
});
return (
<div>
<button
onTouchStart={speechRecognition.start}
onMouseDown={speechRecognition.start}
onTouchEnd={speechRecognition.stop}
onMouseUp={speechRecognition.stop}
>Hold to talk</button>
<span>{matchedCommand}</span>
</div>
);
};
Displaying a transcript
You may want to simply display everything the user says as text, for composing a message for example. This example uses the same button as before. The transcripts are combined and collected in a local state, which is displayed as one piece of text.
import React, { useState, useEffect, useCallback } from 'react';
import { createSpeechlySpeechRecognition } from '@speechly/speech-recognition-polyfill';
const appId = '<your_speechly_app_id>';
const SpeechlySpeechRecognition = createSpeechlySpeechRecognition(appId);
const speechRecognition = new SpeechlySpeechRecognition();
speechRecognition.continuous = true;
export default () => {
const [transcript, setTranscript] = useState('');
const handleResult = useCallback(({ results }) => {
const newTranscript = [transcript, results[0][0].transcript].join(' ');
setTranscript(newTranscript);
}, [transcript]);
useEffect(() => {
speechRecognition.onresult = handleResult;
});
return (
<div>
<button
onTouchStart={speechRecognition.start}
onMouseDown={speechRecognition.start}
onTouchEnd={speechRecognition.stop}
onMouseUp={speechRecognition.stop}
>Hold to talk</button>
<span>{transcript}</span>
</div>
);
};
Integrating with react-speech-recognition
This polyfill is compatible with react-speech-recognition
, a React hook that manages the transcript for you and allows you to provide more powerful commands. For React web apps, we recommend you combine these libraries. See its README for full guidance on how to use react-speech-recognition
. It can be installed with:
npm install --save react-speech-recognition
Below is an example with more complex commands, which print a message in response to each command matched. For example, saying "Bob is my name" will result in the message "Hi Bob!".
import React, { useState } from 'react';
import SpeechRecognition, { useSpeechRecognition } from 'react-speech-recognition';
import { createSpeechlySpeechRecognition } from '@speechly/speech-recognition-polyfill';
const appId = '<your_speechly_app_id>';
const SpeechlySpeechRecognition = createSpeechlySpeechRecognition(appId);
SpeechRecognition.applyPolyfill(SpeechlySpeechRecognition);
export default () => {
const [message, setMessage] = useState('');
const commands = [
{
command: '* is my name',
callback: (name) => setMessage(`Hi ${name}!`),
matchInterim: true
},
{
command: 'My top sports are * and *',
callback: (sport1, sport2) => setMessage(`#1: ${sport1}, #2: ${sport2}`)
},
{
command: 'Goodbye',
callback: () => setMessage('So long!'),
matchInterim: true
},
{
command: 'Pass the salt (please)',
callback: () => setMessage('My pleasure')
}
];
const {
transcript,
listening,
browserSupportsSpeechRecognition,
isMicrophoneAvailable
} = useSpeechRecognition({ commands });
const listenContinuously = () => SpeechRecognition.startListening({ continuous: true });
if (!browserSupportsSpeechRecognition) {
return <span>No browser support</span>
}
if (!isMicrophoneAvailable) {
return <span>Please allow access to the microphone</span>
}
return (
<div>
<p>Microphone: {listening ? 'on' : 'off'}</p>
<button
onTouchStart={listenContinuously}
onMouseDown={listenContinuously}
onTouchEnd={SpeechRecognition.stopListening}
onMouseUp={SpeechRecognition.stopListening}
>Hold to talk</button>
<p>{transcript}</p>
<p>{message}</p>
</div>
);
};
Limitations
While this polyfill is intended to enable most use cases for voice-driven web apps, it does not implement the full W3C specification for SpeechRecognition
, only a subset:
-
start()
method -
stop()
method -
abort()
method -
continuous
property -
interimResults
property -
onresult
property -
onend
property - a callback that is fired whenstop()
orabort()
is called -
onerror
property - a callback that is fired when an error occurs when attempting to start speech recognition
Some notable limitations:
- The
lang
property is currently unsupported, defaulting to English transcription -
onresult
will only receive the most recent speech recognition result (the utterance that the user is in the process of saying or has just finished saying) and does not store a history of all transcripts. This can easily be resolved by either managing your own transcript state (see the Displaying a transcript example above) or usingreact-speech-recognition
to do that for you - Transcripts are generated in uppercase letters without punctuation. If needed, you can transform them using toLowerCase()
-
onerror
currently only supports thenot-allowed
(user denied permission to use the microphone) error and theaudio-capture
error, which is emitted for any other case where speech recognition fails. The full list in the spec can be found here
Contributing
For a guide on how to develop speech-recognition-polyfill
and contribute changes, see CONTRIBUTING.md.
About Speechly
Speechly is a developer tool for building real-time multimodal voice user interfaces. It enables developers and designers to enhance their current touch user interface with voice functionalities for better user experience. Speechly key features:
Speechly key features
- Fully streaming API
- Multi modal from the ground up
- Easy to configure for any use case
- Fast to integrate to any touch screen application
- Supports natural corrections such as "Show me red – i mean blue t-shirts"
- Real time visual feedback encourages users to go on with their voice