Sketch-thru-Plan (STP) is a Natural Language Planning Engine that analyzes combined speech and sketches and produces interpretations of user intentions in terms of symbols placed on a map, and higher-level constructs that correlate multiple symbols into intended actions (or tasks).
STP is a multimodal system - it produces interpretations based on multiple kinds of user input, most commonly speech and sketch. The interpretations fuses these modalities, for example, identifying a location from the sketch, and the intended semantics from the speech, for instance when a user sketches a line and says "phase line blue". The interpretation of multiple modalities also makes the recognizer more robust through mutual disambiguation - if there is high confidence for example that the user sketched an area, rather than a line, then those speech hypothesis that are known to refer to areas are considered more likely, even if they did not rank too well as an independent interpretation of the speech.
STP is a generic engine, which can be configured to provide interpretations in multiple domains. The primary area of application is military planning, supporting fast creation of Courses of Action (COAs). STP enhances user cognition by letting them focus on the creative aspect of planning, keeping focus on the task, not the tool.
Plans designed in STP are executable with little additional user intervention other than the symbol laydown using speech and sketch. The resulting plans are ready to be sent to simulators for adjudication, and to Command and Control (C2) systems.
- Sketch-thru-Plan (STP) Engine (v5.9.3+) running on an accessible server
- Most samples will require a PC or Mac with a working microphone, mouse or stylus
You can get the SDK from npm:
npm install --save sketch-thru-plan-sdk
Or you can embed directly as a script using jsdelivr. As always, it is recommended that a specific version be used rather than @latest to prevent breaking changes from affecting existing code
<!-- Include _after_ the external services such as the Microsoft Cognitive Services Speech -->
<script src="https://cdn.jsdelivr.net/npm/sketch-thru-plan-sdk@latest/dist/sketch-thru-plan-sdk-bundle-min.js"></script>The SDK is built as a UMD library, and is therefore compatible with plain vanilla (IIFE), AMD and CommonJS. Also included is an ESM bundle (sketch-thru-plan-sdk-bundle.esm.js).
When used in vanilla javascript, an StpSDK exported global can be used to access the SDK types:
const stpsdk = new StpSDK.StpRecognizer(stpconn);In typescript, import sketch-thru-plan-sdk after installing via npm:
import * as StpSDK from "sketch-thru-plan-sdk";
const stpsdk = new StpSDK.StpRecognizer(stpconn);Or import individual types:
import { StpWebSocketsConnector, StpRecognizer, StpSymbol } from "sketch-thru-plan-sdk";
const stpsdk = new StpRecognizer(stpconn);Quickstart and samples are available in https://github.com/hyssostech/sketch-thru-plan-sdk-js