Chat about a video clip (or without the video clip) using the powerful OpenAI ChatGPT (hosted in OpenAI or Microsoft Azure) or Google Gemini (hosted in Google Could).
chat-about-video is an open-source NPM package designed to accelerate the development of conversation applications about video content. Harnessing the capabilities of ChatGPT from Microsoft Azure or OpenAI, as well as Gemini from Google, this package opens up a range of usage scenarios with minimal effort.
Key features:
- ChatGPT models hosted in both Azure and OpenAI are supported.
- Gemini models hosted in Google Cloud are supported.
- Frame images are extracted from the input video, and uploaded for ChatGPT/Gemini to consume.
- It can automatically retry on receiving throttling (HTTP status code 429) and error (HTTP status code 5xx) responses from the API.
- Options supported by the underlying API are exposed for customisation.
- It can also be used in scenario that no video needs to be involved, that means it can be used for "normal" text chats.
To use chat-about-video in your Node.js application,
add it as a dependency along with other necessary packages based on your usage scenario.
Below are examples for typical setups:
# ChatGPT on OpenAI or Azure with Azure Blob Storage
npm i chat-about-video @azure/openai @ffmpeg-installer/ffmpeg @azure/storage-blob
# Gemini in Google Cloud
npm i chat-about-video @google/generative-ai @ffmpeg-installer/ffmpeg
# ChatGPT on OpenAI or Azure with AWS S3
npm i chat-about-video @azure/openai @ffmpeg-installer/ffmpeg @handy-common-utils/aws-utils @aws-sdk/s3-request-presigner @aws-sdk/client-s3ChatGPT
To use ChatGPT hosted on OpenAI or Azure:
npm i @azure/openaiGemini
To use Gemini hosted on Google Cloud:
npm i @google/generative-aiffmpeg
If you need ffmpeg for extracting video frame images, ensure it is installed. You can use a system package manager or an NPM package:
sudo apt install ffmpeg
# or
npm i @ffmpeg-installer/ffmpegAzure Blob Storage
To use Azure Blob Storage for frame images (not needed for Gemini):
npm i @azure/storage-blobAWS S3
To use AWS S3 for frame images (not needed for Gemini):
npm i @handy-common-utils/aws-utils @aws-sdk/s3-request-presigner @aws-sdk/client-s3There are two approaches for feeding video content to ChatGPT. chat-about-video supports both of them.
Frame image extraction:
- Integrate ChatGPT from Microsoft Azure or OpenAI effortlessly.
- Utilize ffmpeg integration provided by this package for frame image extraction or opt for a DIY approach.
- Store frame images with ease, supporting Azure Blob Storage and AWS S3.
- GPT-4o and GPT-4 Vision Preview hosted in Azure allows analysis of up to 10 frame images.
- GPT-4o and GPT-4 Vision Preview hosted in OpenAI allows analysis of more than 10 frame images.
Video indexing with Microsoft Azure:
- Exclusively supported by GPT-4 Vision Preview from Microsoft Azure.
- Ingest videos seamlessly into Microsoft Azure's Video Retrieval Index.
- Automatic extraction of up to 20 frame images using Video Retrieval Indexer.
- Default integration of speech transcription for enhanced comprehension.
- Flexible storage options with support for Azure Blob Storage and AWS S3.
chat-about-video supports sending Video frames directly to Google's API without a cloud storage.
- Utilize ffmpeg integration provided by this package for frame image extraction or opt for a DIY approach.
- Number of frame images is only limited by Gemini API in Google Cloud.
ChatAboutVideo and Conversation are generic classes.
Use them without concrete generic type parameters when you want the flexibility to easily switch between ChatGPT and Gemini.
Otherwise, you may want to use concrete type. Below are some examples:
// cast to a concrete type
const castToChatGpt = chat as ChatAboutVideoWithChatGpt;
// you can also just leave the ChatAboutVideo instance generic, but narrow down the conversation type
const conversationWithGemini = (await chat.startConversation(...)) as ConversationWithGemini;
const conversationWithChatGpt = await (chat as ChatAboutVideoWithChatGpt).startConversation(...);To access the underlying API wrapper, use the getApi() function on the ChatAboutVideo instance.
To get the raw API client, use the getClient() function on the awaited object returned from getApi().
Intermediate files, such as extracted frame images, can be saved locally or in the cloud.
To remove these files when they are no longer needed, remember to call the end() function
on the Conversation instance when the conversion finishes.
If you would like to customise how frame images are extracted and stored, consider these:
- In the options object passed to the constructor of
ChatAboutVideo, there's a propertyextractVideoFrames. This property allows you to customise how frame images are extracted.-
format,interval,limit,width,height- These allows you to specify your expectation on the extraction. -
deleteFilesWhenConversationEnds- This flag allows you to specify whether you want extracted frame images to be deleted from the local file system when the conversation ends, or not. -
framesDirectoryResolver- You can supply a function for determining where extracted frame image files should be stored locally. -
extractor- You can supply a function for doing the extraction.
-
- In the options object passed to the constructor of
ChatAboutVideo, there's a propertystorage. For ChatGPT, storing frame images in the cloud is recommended. You can use this property to customise how frame images are stored in the cloud.-
azureStorageConnectionString- If you would like to use Azure Blob Storage, you need to put the connection string in this property. If this property does not have a value,ChatAboutVideowould assume that you'd like to use AWS S3, and default AWS identity/credential will be picked up from the OS. -
storageContainerName,storagePathPrefix- They allows you to specify where those images should be stored. -
downloadUrlExpirationSeconds- For images stored in the cloud, presigned download URLs with expiration are generated for ChatGPT to access. This property allows you to control the expiration time. -
deleteFilesWhenConversationEnds- This flag allows you to specify whether you want extracted frame images to be deleted from the cloud when the conversation ends, or not. -
uploader- You can supply a function for uploading images into the cloud.
-
In the options object passed to the constructor of ChatAboutVideo, there's a property clientSettings,
and there's another property completionSettings. Settings of the underlying model can be configured
through those two properties.
You can also override settings using the last parameter of startConversation(...) function on ChatAboutVideo,
or the last parameter of say(...) function on Conversation.
// This is a demo utilising GPT-4o or Vision preview hosted in OpenAI.
// OpenAI API allows more than 10 (maximum allowed by Azure's OpenAI API) images to be supplied.
// Video frame images are uploaded to Azure Blob Storage and then made available to GPT from there.
//
// This script can be executed with a command line like this from the project root directory:
// export OPENAI_API_KEY=...
// export AZURE_STORAGE_CONNECTION_STRING=...
// export OPENAI_MODEL_NAME=...
// export AZURE_STORAGE_CONTAINER_NAME=...
// ENABLE_DEBUG=true DEMO_VIDEO=~/Downloads/test1.mp4 npx ts-node test/demo1.ts
//
import { consoleWithColour } from '@handy-common-utils/misc-utils';
import chalk from 'chalk';
import readline from 'node:readline';
import { ChatAboutVideo, ConversationWithChatGpt } from 'chat-about-video';
async function demo() {
const chat = new ChatAboutVideo(
{
credential: {
key: process.env.OPENAI_API_KEY!,
},
storage: {
azureStorageConnectionString: process.env.AZURE_STORAGE_CONNECTION_STRING!,
storageContainerName: process.env.AZURE_STORAGE_CONTAINER_NAME || 'vision-experiment-input',
storagePathPrefix: 'video-frames/',
},
completionOptions: {
deploymentName: process.env.OPENAI_MODEL_NAME || 'gpt-4o', // 'gpt-4-vision-preview', // or gpt-4o
},
extractVideoFrames: {
limit: 100,
interval: 2,
},
},
consoleWithColour({ debug: process.env.ENABLE_DEBUG === 'true' }, chalk),
);
const conversation = (await chat.startConversation(process.env.DEMO_VIDEO!)) as ConversationWithChatGpt;
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const prompt = (question: string) => new Promise<string>((resolve) => rl.question(question, resolve));
while (true) {
const question = await prompt(chalk.red('\nUser: '));
if (!question) {
continue;
}
if (['exit', 'quit', 'q', 'end'].includes(question)) {
await conversation.end();
break;
}
const answer = await conversation.say(question, { maxTokens: 2000 });
console.log(chalk.blue('\nAI:' + answer));
}
console.log('Demo finished');
rl.close();
}
demo().catch((error) => console.log(chalk.red(JSON.stringify(error, null, 2))));// This is a demo utilising GPT-4 Vision preview hosted in Azure.
// Azure Video Retrieval Indexer is used for extracting information from the input video.
// Information in Azure Video Retrieval Indexer is supplied to GPT.
//
// This script can be executed with a command line like this from the project root directory:
// export AZURE_OPENAI_API_ENDPOINT=..
// export AZURE_OPENAI_API_KEY=...
// export AZURE_OPENAI_DEPLOYMENT_NAME=...
// export AZURE_STORAGE_CONNECTION_STRING=...
// export AZURE_STORAGE_CONTAINER_NAME=...
// export AZURE_CV_API_KEY=...
// ENABLE_DEBUG=true DEMO_VIDEO=~/Downloads/test1.mp4 npx ts-node test/demo2.ts
//
import { consoleWithColour } from '@handy-common-utils/misc-utils';
import chalk from 'chalk';
import readline from 'node:readline';
import { ChatAboutVideo, ConversationWithChatGpt } from 'chat-about-video';
async function demo() {
const chat = new ChatAboutVideo(
{
endpoint: process.env.AZURE_OPENAI_API_ENDPOINT!,
credential: {
key: process.env.AZURE_OPENAI_API_KEY!,
},
storage: {
azureStorageConnectionString: process.env.AZURE_STORAGE_CONNECTION_STRING!,
storageContainerName: process.env.AZURE_STORAGE_CONTAINER_NAME || 'vision-experiment-input',
storagePathPrefix: 'video-frames/',
},
completionOptions: {
deploymentName: process.env.AZURE_OPENAI_DEPLOYMENT_NAME || 'gpt4vision',
},
videoRetrievalIndex: {
endpoint: process.env.AZURE_CV_API_ENDPOINT!,
apiKey: process.env.AZURE_CV_API_KEY!,
createIndexIfNotExists: true,
deleteIndexWhenConversationEnds: true,
},
},
consoleWithColour({ debug: process.env.ENABLE_DEBUG === 'true' }, chalk),
);
const conversation = (await chat.startConversation(process.env.DEMO_VIDEO!)) as ConversationWithChatGpt;
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const prompt = (question: string) => new Promise<string>((resolve) => rl.question(question, resolve));
while (true) {
const question = await prompt(chalk.red('\nUser: '));
if (!question) {
continue;
}
if (['exit', 'quit', 'q', 'end'].includes(question)) {
await conversation.end();
break;
}
const answer = await conversation.say(question, { maxTokens: 2000 });
console.log(chalk.blue('\nAI:' + answer));
}
console.log('Demo finished');
rl.close();
}
demo().catch((error) => console.log(chalk.red(JSON.stringify(error, null, 2)), (error as Error).stack));// This is a demo utilising GPT-4o or Vision preview hosted in Azure.
// Up to 10 (maximum allowed by Azure's OpenAI API) frames are extracted from the input video.
// Video frame images are uploaded to Azure Blob Storage and then made available to GPT from there.
//
// This script can be executed with a command line like this from the project root directory:
// export AZURE_OPENAI_API_ENDPOINT=..
// export AZURE_OPENAI_API_KEY=...
// export AZURE_OPENAI_DEPLOYMENT_NAME=...
// export AZURE_STORAGE_CONNECTION_STRING=...
// export AZURE_STORAGE_CONTAINER_NAME=...
// ENABLE_DEBUG=true DEMO_VIDEO=~/Downloads/test1.mp4 npx ts-node test/demo3.ts
import { consoleWithColour } from '@handy-common-utils/misc-utils';
import chalk from 'chalk';
import readline from 'node:readline';
import { ChatAboutVideo, ConversationWithChatGpt } from 'chat-about-video';
async function demo() {
const chat = new ChatAboutVideo(
{
endpoint: process.env.AZURE_OPENAI_API_ENDPOINT!,
credential: {
key: process.env.AZURE_OPENAI_API_KEY!,
},
storage: {
azureStorageConnectionString: process.env.AZURE_STORAGE_CONNECTION_STRING!,
storageContainerName: process.env.AZURE_STORAGE_CONTAINER_NAME || 'vision-experiment-input',
storagePathPrefix: 'video-frames/',
},
completionOptions: {
deploymentName: process.env.AZURE_OPENAI_DEPLOYMENT_NAME || 'gpt4vision',
},
},
consoleWithColour({ debug: process.env.ENABLE_DEBUG === 'true' }, chalk),
);
const conversation = (await chat.startConversation(process.env.DEMO_VIDEO!)) as ConversationWithChatGpt;
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const prompt = (question: string) => new Promise<string>((resolve) => rl.question(question, resolve));
while (true) {
const question = await prompt(chalk.red('\nUser: '));
if (!question) {
continue;
}
if (['exit', 'quit', 'q', 'end'].includes(question)) {
await conversation.end();
break;
}
const answer = await conversation.say(question, { maxTokens: 2000 });
console.log(chalk.blue('\nAI:' + answer));
}
console.log('Demo finished');
rl.close();
}
demo().catch((error) => console.log(chalk.red(JSON.stringify(error, null, 2))));// This is a demo utilising Google Gemini through Google Generative Language API.
// Google Gemini allows more than 10 (maximum allowed by Azure's OpenAI API) frame images to be supplied.
// Video frame images are sent through Google Generative Language API directly.
//
// This script can be executed with a command line like this from the project root directory:
// export GEMINI_API_KEY=...
// ENABLE_DEBUG=true DEMO_VIDEO=~/Downloads/test1.mp4 npx ts-node test/demo4.ts
import { consoleWithColour } from '@handy-common-utils/misc-utils';
import chalk from 'chalk';
import readline from 'node:readline';
import { HarmBlockThreshold, HarmCategory } from '@google/generative-ai';
import { ChatAboutVideo, ConversationWithGemini } from 'chat-about-video';
async function demo() {
const chat = new ChatAboutVideo(
{
credential: {
key: process.env.GEMINI_API_KEY!,
},
clientSettings: {
modelParams: {
model: 'gemini-1.5-flash',
},
},
extractVideoFrames: {
limit: 100,
interval: 0.5,
},
},
consoleWithColour({ debug: process.env.ENABLE_DEBUG === 'true' }, chalk),
);
const conversation = (await chat.startConversation(process.env.DEMO_VIDEO!)) as ConversationWithGemini;
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const prompt = (question: string) => new Promise<string>((resolve) => rl.question(question, resolve));
while (true) {
const question = await prompt(chalk.red('\nUser: '));
if (!question) {
continue;
}
if (['exit', 'quit', 'q', 'end'].includes(question)) {
await conversation.end();
break;
}
const answer = await conversation.say(question, {
safetySettings: [{ category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT, threshold: HarmBlockThreshold.BLOCK_NONE }],
});
console.log(chalk.blue('\nAI:' + answer));
}
console.log('Demo finished');
rl.close();
}
demo().catch((error) => console.log(chalk.red(JSON.stringify(error, null, 2))));- aws
- azure
- azure/client-hack
- azure/video-retrieval-api-client
- chat
- chat-gpt
- gemini
- index
- storage
- storage/types
- types
- utils
- video
- video/ffmpeg
- video/types
azure/video-retrieval-api-client.VideoRetrievalApiClient
• new VideoRetrievalApiClient(endpointBaseUrl, apiKey, apiVersion?)
| Name | Type | Default value |
|---|---|---|
endpointBaseUrl |
string |
undefined |
apiKey |
string |
undefined |
apiVersion |
string |
'2023-05-01-preview' |
▸ createIndex(indexName, indexOptions?): Promise<void>
| Name | Type |
|---|---|
indexName |
string |
indexOptions |
CreateIndexOptions |
Promise<void>
▸ createIndexIfNotExist(indexName, indexOptions?): Promise<void>
| Name | Type |
|---|---|
indexName |
string |
indexOptions? |
CreateIndexOptions |
Promise<void>
▸ createIngestion(indexName, ingestionName, ingestion): Promise<void>
| Name | Type |
|---|---|
indexName |
string |
ingestionName |
string |
ingestion |
IngestionRequest |
Promise<void>
▸ deleteDocument(indexName, documentUrl): Promise<void>
| Name | Type |
|---|---|
indexName |
string |
documentUrl |
string |
Promise<void>
▸ deleteIndex(indexName): Promise<void>
| Name | Type |
|---|---|
indexName |
string |
Promise<void>
▸ getIndex(indexName): Promise<undefined | IndexSummary>
| Name | Type |
|---|---|
indexName |
string |
Promise<undefined | IndexSummary>
▸ getIngestion(indexName, ingestionName): Promise<IngestionSummary>
| Name | Type |
|---|---|
indexName |
string |
ingestionName |
string |
Promise<IngestionSummary>
▸ ingest(indexName, ingestionName, ingestion, backoff?): Promise<void>
| Name | Type |
|---|---|
indexName |
string |
ingestionName |
string |
ingestion |
IngestionRequest |
backoff |
number[] |
Promise<void>
▸ listDocuments(indexName): Promise<DocumentSummary[]>
| Name | Type |
|---|---|
indexName |
string |
Promise<DocumentSummary[]>
▸ listIndexes(): Promise<IndexSummary[]>
Promise<IndexSummary[]>
chat.ChatAboutVideo
| Name | Type |
|---|---|
CLIENT |
any |
OPTIONS |
extends AdditionalCompletionOptions = any
|
PROMPT |
any |
RESPONSE |
any |
• new ChatAboutVideo<CLIENT, OPTIONS, PROMPT, RESPONSE>(options, log?)
| Name | Type |
|---|---|
CLIENT |
any |
OPTIONS |
extends AdditionalCompletionOptions = any
|
PROMPT |
any |
RESPONSE |
any |
| Name | Type |
|---|---|
options |
SupportedChatApiOptions |
log |
undefined | LineLogger<(message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void> |
| Property | Description |
|---|---|
Protected apiPromise: Promise<ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE>> |
|
Protected log: undefined | LineLogger<(message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void> |
|
Protected options: SupportedChatApiOptions
|
▸ getApi(): Promise<ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE>>
Get the underlying API instance.
Promise<ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE>>
The underlying API instance.
▸ startConversation(options?): Promise<Conversation<CLIENT, OPTIONS, PROMPT, RESPONSE>>
Start a conversation without a video
| Name | Type | Description |
|---|---|---|
options? |
OPTIONS |
Overriding options for this conversation |
Promise<Conversation<CLIENT, OPTIONS, PROMPT, RESPONSE>>
The conversation.
▸ startConversation(videoFile, options?): Promise<Conversation<CLIENT, OPTIONS, PROMPT, RESPONSE>>
Start a conversation about a video.
| Name | Type | Description |
|---|---|---|
videoFile |
string |
Path to a video file in local file system. |
options? |
OPTIONS |
Overriding options for this conversation |
Promise<Conversation<CLIENT, OPTIONS, PROMPT, RESPONSE>>
The conversation.
chat.Conversation
| Name | Type |
|---|---|
CLIENT |
any |
OPTIONS |
extends AdditionalCompletionOptions = any
|
PROMPT |
any |
RESPONSE |
any |
• new Conversation<CLIENT, OPTIONS, PROMPT, RESPONSE>(conversationId, api, prompt, options, cleanup?, log?)
| Name | Type |
|---|---|
CLIENT |
any |
OPTIONS |
extends AdditionalCompletionOptions = any
|
PROMPT |
any |
RESPONSE |
any |
| Name | Type |
|---|---|
conversationId |
string |
api |
ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE> |
prompt |
undefined | PROMPT
|
options |
OPTIONS |
cleanup? |
() => Promise<any> |
log |
undefined | LineLogger<(message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void> |
| Property | Description |
|---|---|
Protected api: ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE> |
|
Protected Optional cleanup: () => Promise<any> |
|
Protected conversationId: string
|
|
Protected log: undefined | LineLogger<(message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void, (message?: any, ...optionalParams: any[]) => void> |
|
Protected options: OPTIONS
|
|
Protected prompt: undefined | PROMPT
|
▸ end(): Promise<void>
Promise<void>
▸ getApi(): ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE>
Get the underlying API instance.
ChatApi<CLIENT, OPTIONS, PROMPT, RESPONSE>
The underlying API instance.
▸ getPrompt(): undefined | PROMPT
Get the prompt for the current conversation. The prompt is the accumulated messages in the conversation so far.
undefined | PROMPT
The prompt which is the accumulated messages in the conversation so far.
▸ say(message, options?): Promise<undefined | string>
Say something in the conversation, and get the response from AI
| Name | Type | Description |
|---|---|---|
message |
string |
The message to say in the conversation. |
options? |
Partial<OPTIONS> |
Options for fine control. |
Promise<undefined | string>
The response/completion
chat-gpt.ChatGptApi
• new ChatGptApi(options)
| Name | Type |
|---|---|
options |
ChatGptOptions |
| Property | Description |
|---|---|
Protected client: OpenAIClient
|
|
Protected Optional extractVideoFrames: Pick<ExtractVideoFramesOptions, "height"> & Required<Omit<ExtractVideoFramesOptions, "height">> |
|
Protected options: ChatGptOptions
|
|
Protected storage: Required<Pick<StorageOptions, "uploader">> & StorageOptions
|
|
Protected tmpDir: string
|
|
Protected Optional videoRetrievalIndex: Required<Pick<VideoRetrievalIndexOptions, "createIndexIfNotExists" | "deleteDocumentWhenConversationEnds" | "deleteIndexWhenConversationEnds">> & VideoRetrievalIndexOptions
|
▸ appendToPrompt(newPromptOrResponse, prompt?): Promise<ChatRequestMessageUnion[]>
Append a new prompt or response to the form a full prompt. This function is useful to build a prompt that contains conversation history.
| Name | Type | Description |
|---|---|---|
newPromptOrResponse |
ChatCompletions | ChatRequestMessageUnion[] |
A new prompt to be appended, or previous response to be appended. |
prompt? |
ChatRequestMessageUnion[] |
The conversation history which is a prompt containing previous prompts and responses. If it is not provided, the conversation history returned will contain only what is in newPromptOrResponse. |
Promise<ChatRequestMessageUnion[]>
The full prompt which is effectively the conversation history.
▸ buildTextPrompt(text, _conversationId?): Promise<{ prompt: ChatRequestMessageUnion[] }>
Build prompt for sending text content to AI
| Name | Type | Description |
|---|---|---|
text |
string |
The text content to be sent. |
_conversationId? |
string |
Unique identifier of the conversation. |
Promise<{ prompt: ChatRequestMessageUnion[] }>
An object containing the prompt.
▸ buildVideoPrompt(videoFile, conversationId?): Promise<BuildPromptOutput<ChatRequestMessageUnion[], { deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions>>
Build prompt for sending video content to AI. Sometimes, to include video in the conversation, additional options and/or clean up is needed. In such case, options to be passed to generateContent function and/or a clean up call back function will be returned in the output of this function.
| Name | Type | Description |
|---|---|---|
videoFile |
string |
Path to the video file. |
conversationId? |
string |
Unique identifier of the conversation. |
Promise<BuildPromptOutput<ChatRequestMessageUnion[], { deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions>>
An object containing the prompt, optional options, and an optional cleanup function.
▸ Protected buildVideoPromptWithFrames(videoFile, conversationId?): Promise<BuildPromptOutput<ChatRequestMessageUnion[], { deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions>>
| Name | Type |
|---|---|
videoFile |
string |
conversationId |
string |
Promise<BuildPromptOutput<ChatRequestMessageUnion[], { deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions>>
▸ Protected buildVideoPromptWithVideoRetrievalIndex(videoFile, conversationId?): Promise<BuildPromptOutput<ChatRequestMessageUnion[], { deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions>>
| Name | Type |
|---|---|
videoFile |
string |
conversationId |
string |
Promise<BuildPromptOutput<ChatRequestMessageUnion[], { deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions>>
▸ generateContent(prompt, options): Promise<ChatCompletions>
Generate content based on the given prompt and options.
| Name | Type | Description |
|---|---|---|
prompt |
ChatRequestMessageUnion[] |
The full prompt to generate content. |
options |
{ deploymentName: string } & AdditionalCompletionOptions & GetChatCompletionsOptions
|
Optional options to control the content generation. |
Promise<ChatCompletions>
The generated content.
▸ getClient(): Promise<OpenAIClient>
Get the raw client. This function could be useful for advanced use cases.
Promise<OpenAIClient>
The raw client.
▸ getResponseText(result): Promise<undefined | string>
Get the text from the response object
| Name | Type | Description |
|---|---|---|
result |
ChatCompletions |
the response object |
Promise<undefined | string>
▸ isServerError(error): boolean
Check if the error is a server error.
| Name | Type | Description |
|---|---|---|
error |
any |
any error object |
boolean
true if the error is a server error, false otherwise.
▸ isThrottlingError(error): boolean
Check if the error is a throttling error.
| Name | Type | Description |
|---|---|---|
error |
any |
any error object |
boolean
true if the error is a throttling error, false otherwise.
gemini.GeminiApi
• new GeminiApi(options)
| Name | Type |
|---|---|
options |
GeminiOptions |
| Property | Description |
|---|---|
Protected client: GenerativeModel
|
|
Protected extractVideoFrames: Pick<ExtractVideoFramesOptions, "height"> & Required<Omit<ExtractVideoFramesOptions, "height">> |
|
Protected options: GeminiOptions
|
|
Protected tmpDir: string
|
▸ appendToPrompt(newPromptOrResponse, prompt?): Promise<Content[]>
Append a new prompt or response to the form a full prompt. This function is useful to build a prompt that contains conversation history.
| Name | Type | Description |
|---|---|---|
newPromptOrResponse |
Content[] | GenerateContentResult
|
A new prompt to be appended, or previous response to be appended. |
prompt? |
Content[] |
The conversation history which is a prompt containing previous prompts and responses. If it is not provided, the conversation history returned will contain only what is in newPromptOrResponse. |
Promise<Content[]>
The full prompt which is effectively the conversation history.
▸ buildTextPrompt(text, _conversationId?): Promise<{ prompt: Content[] }>
Build prompt for sending text content to AI
| Name | Type | Description |
|---|---|---|
text |
string |
The text content to be sent. |
_conversationId? |
string |
Unique identifier of the conversation. |
Promise<{ prompt: Content[] }>
An object containing the prompt.
▸ buildVideoPrompt(videoFile, conversationId?): Promise<BuildPromptOutput<Content[], GeminiCompletionOptions>>
Build prompt for sending video content to AI. Sometimes, to include video in the conversation, additional options and/or clean up is needed. In such case, options to be passed to generateContent function and/or a clean up call back function will be returned in the output of this function.
| Name | Type | Description |
|---|---|---|
videoFile |
string |
Path to the video file. |
conversationId |
string |
Unique identifier of the conversation. |
Promise<BuildPromptOutput<Content[], GeminiCompletionOptions>>
An object containing the prompt, optional options, and an optional cleanup function.
▸ generateContent(prompt, options): Promise<GenerateContentResult>
Generate content based on the given prompt and options.
| Name | Type | Description |
|---|---|---|
prompt |
Content[] |
The full prompt to generate content. |
options |
GeminiCompletionOptions |
Optional options to control the content generation. |
Promise<GenerateContentResult>
The generated content.
▸ getClient(): Promise<GenerativeModel>
Get the raw client. This function could be useful for advanced use cases.
Promise<GenerativeModel>
The raw client.
▸ getResponseText(result): Promise<undefined | string>
Get the text from the response object
| Name | Type | Description |
|---|---|---|
result |
GenerateContentResult |
the response object |
Promise<undefined | string>
▸ isServerError(error): boolean
Check if the error is a server error.
| Name | Type | Description |
|---|---|---|
error |
any |
any error object |
boolean
true if the error is a server error, false otherwise.
▸ isThrottlingError(error): boolean
Check if the error is a throttling error.
| Name | Type | Description |
|---|---|---|
error |
any |
any error object |
boolean
true if the error is a throttling error, false otherwise.
azure/video-retrieval-api-client.CreateIndexOptions
| Property | Description |
|---|---|
Optional features: IndexFeature[] |
|
Optional metadataSchema: IndexMetadataSchema
|
|
Optional userData: object
|
azure/video-retrieval-api-client.DocumentSummary
| Property | Description |
|---|---|
createdDateTime: string
|
|
documentId: string
|
|
Optional documentUrl: string
|
|
lastModifiedDateTime: string
|
|
Optional metadata: object
|
|
Optional userData: object
|
azure/video-retrieval-api-client.IndexFeature
| Property | Description |
|---|---|
Optional domain: "surveillance" | "generic"
|
|
Optional modelVersion: string
|
|
name: "vision" | "speech"
|
azure/video-retrieval-api-client.IndexMetadataSchema
| Property | Description |
|---|---|
fields: IndexMetadataSchemaField[] |
|
Optional language: string
|
azure/video-retrieval-api-client.IndexMetadataSchemaField
| Property | Description |
|---|---|
filterable: boolean
|
|
name: string
|
|
searchable: boolean
|
|
type: "string" | "datetime"
|
azure/video-retrieval-api-client.IndexSummary
| Property | Description |
|---|---|
createdDateTime: string
|
|
eTag: string
|
|
Optional features: IndexFeature[] |
|
lastModifiedDateTime: string
|
|
name: string
|
|
Optional userData: object
|
azure/video-retrieval-api-client.IngestionRequest
| Property | Description |
|---|---|
Optional filterDefectedFrames: boolean
|
|
Optional generateInsightIntervals: boolean
|
|
Optional includeSpeechTranscript: boolean
|
|
Optional moderation: boolean
|
|
videos: VideoIngestion[] |
azure/video-retrieval-api-client.IngestionStatusDetail
| Property | Description |
|---|---|
documentId: string
|
|
documentUrl: string
|
|
lastUpdatedTime: string
|
|
succeeded: boolean
|
azure/video-retrieval-api-client.IngestionSummary
| Property | Description |
|---|---|
Optional batchName: string
|
|
createdDateTime: string
|
|
Optional fileStatusDetails: IngestionStatusDetail[] |
|
lastModifiedDateTime: string
|
|
name: string
|
|
state: "NotStarted" | "Running" | "Completed" | "Failed" | "PartiallySucceeded"
|
azure/video-retrieval-api-client.VideoIngestion
| Property | Description |
|---|---|
Optional documentId: string
|
|
documentUrl: string
|
|
Optional metadata: object
|
|
mode: "update" | "remove" | "add"
|
|
Optional userData: object
|
types.AdditionalCompletionOptions
| Property | Description |
|---|---|
Optional backoffOnServerError: number[] |
Array of retry backoff periods (unit: milliseconds) for situations that the server returns 5xx response |
Optional backoffOnThrottling: number[] |
Array of retry backoff periods (unit: milliseconds) for situations that the server returns 429 response |
Optional startPromptText: string
|
The user prompt that will be sent before the video content. If not provided, nothing will be sent before the video content. |
Optional systemPromptText: string
|
System prompt text. If not provided, a default prompt will be used. |
types.BuildPromptOutput
| Name |
|---|
PROMPT |
OPTIONS |
| Property | Description |
|---|---|
Optional cleanup: () => Promise<any> |
|
Optional options: Partial<OPTIONS> |
|
prompt: PROMPT
|
types.ChatApi
| Name | Type |
|---|---|
CLIENT |
CLIENT |
OPTIONS |
extends AdditionalCompletionOptions
|
PROMPT |
PROMPT |
RESPONSE |
RESPONSE |
▸ appendToPrompt(newPromptOrResponse, prompt?): Promise<PROMPT>
Append a new prompt or response to the form a full prompt. This function is useful to build a prompt that contains conversation history.
| Name | Type | Description |
|---|---|---|
newPromptOrResponse |
PROMPT | RESPONSE
|
A new prompt to be appended, or previous response to be appended. |
prompt? |
PROMPT |
The conversation history which is a prompt containing previous prompts and responses. If it is not provided, the conversation history returned will contain only what is in newPromptOrResponse. |
Promise<PROMPT>
The full prompt which is effectively the conversation history.
▸ buildTextPrompt(text, conversationId?): Promise<{ prompt: PROMPT }>
Build prompt for sending text content to AI
| Name | Type | Description |
|---|---|---|
text |
string |
The text content to be sent. |
conversationId? |
string |
Unique identifier of the conversation. |
Promise<{ prompt: PROMPT }>
An object containing the prompt.
▸ buildVideoPrompt(videoFile, conversationId?): Promise<BuildPromptOutput<PROMPT, OPTIONS>>
Build prompt for sending video content to AI. Sometimes, to include video in the conversation, additional options and/or clean up is needed. In such case, options to be passed to generateContent function and/or a clean up call back function will be returned in the output of this function.
| Name | Type | Description |
|---|---|---|
videoFile |
string |
Path to the video file. |
conversationId? |
string |
Unique identifier of the conversation. |
Promise<BuildPromptOutput<PROMPT, OPTIONS>>
An object containing the prompt, optional options, and an optional cleanup function.
▸ generateContent(prompt, options?): Promise<RESPONSE>
Generate content based on the given prompt and options.
| Name | Type | Description |
|---|---|---|
prompt |
PROMPT |
The full prompt to generate content. |
options? |
OPTIONS |
Optional options to control the content generation. |
Promise<RESPONSE>
The generated content.
▸ getClient(): Promise<CLIENT>
Get the raw client. This function could be useful for advanced use cases.
Promise<CLIENT>
The raw client.
▸ getResponseText(response): Promise<undefined | string>
Get the text from the response object
| Name | Type | Description |
|---|---|---|
response |
RESPONSE |
the response object |
Promise<undefined | string>
▸ isServerError(error): boolean
Check if the error is a server error.
| Name | Type | Description |
|---|---|---|
error |
any |
any error object |
boolean
true if the error is a server error, false otherwise.
▸ isThrottlingError(error): boolean
Check if the error is a throttling error.
| Name | Type | Description |
|---|---|---|
error |
any |
any error object |
boolean
true if the error is a throttling error, false otherwise.
types.ChatApiOptions
| Name |
|---|
CS |
CO |
| Property | Description |
| ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---- | ---- | ---- | ------- | ------- | ---- | ----- | -------- | --- |
| Optional clientSettings: CS | |
| Optional completionOptions: AdditionalCompletionOptions & CO | |
| credential: Object | Type declaration
| Name | Type |
| :------ | :------ |
| key | string | |
| Optional endpoint: string | |
| Optional tmpDir: string | Temporary directory for storing temporary files.
If not specified, then the temporary directory of the OS will be used. |
types.ExtractVideoFramesOptions
| Property | Description |
|---|---|
Optional deleteFilesWhenConversationEnds: boolean
|
Whether files should be deleted when the conversation ends. |
Optional extractor: VideoFramesExtractor
|
Function for extracting frames from the video. If not specified, a default function using ffmpeg will be used. |
Optional format: string
|
Image format of the extracted frames. Default value is 'jpg'. |
Optional framesDirectoryResolver: (inputFile: string, tmpDir: string, conversationId: string) => string
|
Function for determining the directory location for storing extracted frames. If not specified, a default function will be used. The function takes three arguments: |
Optional height: number
|
Video frame height, default is undefined which means the scaling will be determined by the videoFrameWidth option. If both videoFrameWidth and videoFrameHeight are not specified, then the frames will not be resized/scaled. |
Optional interval: number
|
Intervals between frames to be extracted. The unit is second. Default value is 5. |
Optional limit: number
|
Maximum number of frames to be extracted. Default value is 10 which is the current per-request limitation of ChatGPT Vision. |
Optional width: number
|
Video frame width, default is 200. If both videoFrameWidth and videoFrameHeight are not specified, then the frames will not be resized/scaled. |
types.StorageOptions
| Property | Description |
|---|---|
Optional azureStorageConnectionString: string
|
|
Optional deleteFilesWhenConversationEnds: boolean
|
Whether files should be deleted when the conversation ends. |
Optional downloadUrlExpirationSeconds: number
|
Expiration time for the download URL of the frame images in seconds. Default is 3600 seconds. |
Optional storageContainerName: string
|
Storage container for storing frame images of the video. |
Optional storagePathPrefix: string
|
Path prefix to be prepended for storing frame images of the video. Default is empty. |
Optional uploader: FileBatchUploader
|
Function for uploading files |
types.VideoRetrievalIndexOptions
| Property | Description |
|---|---|
apiKey: string
|
|
Optional createIndexIfNotExists: boolean
|
|
Optional deleteDocumentWhenConversationEnds: boolean
|
|
Optional deleteIndexWhenConversationEnds: boolean
|
|
endpoint: string
|
|
Optional indexName: string
|
## Modules |
▸ createAwsS3FileBatchUploader(s3Client, expirationSeconds, parallelism?): FileBatchUploader
| Name | Type | Default value |
|---|---|---|
s3Client |
S3Client |
undefined |
expirationSeconds |
number |
undefined |
parallelism |
number |
3 |
Re-exports CreateIndexOptions
Re-exports DocumentSummary
Re-exports IndexFeature
Re-exports IndexMetadataSchema
Re-exports IndexMetadataSchemaField
Re-exports IndexSummary
Re-exports IngestionRequest
Re-exports IngestionStatusDetail
Re-exports IngestionSummary
Re-exports PaginatedWithNextLink
Re-exports VideoIngestion
Re-exports VideoRetrievalApiClient
▸ createAzureBlobStorageFileBatchUploader(blobServiceClient, expirationSeconds, parallelism?): FileBatchUploader
| Name | Type | Default value |
|---|---|---|
blobServiceClient |
BlobServiceClient |
undefined |
expirationSeconds |
number |
undefined |
parallelism |
number |
3 |
▸ fixClient(openAIClient): void
| Name | Type |
|---|---|
openAIClient |
any |
void
- CreateIndexOptions
- DocumentSummary
- IndexFeature
- IndexMetadataSchema
- IndexMetadataSchemaField
- IndexSummary
- IngestionRequest
- IngestionStatusDetail
- IngestionSummary
- VideoIngestion
Ƭ PaginatedWithNextLink<T>: Object
| Name |
|---|
T |
| Name | Type |
|---|---|
nextLink? |
string |
value |
T[] |
Ƭ ChatAboutVideoWith<T>: ChatAboutVideo<ClientOfChatApi<T>, OptionsOfChatApi<T>, PromptOfChatApi<T>, ResponseOfChatApi<T>>
| Name |
|---|
T |
Ƭ ChatAboutVideoWithChatGpt: ChatAboutVideoWith<ChatGptApi>
Ƭ ChatAboutVideoWithGemini: ChatAboutVideoWith<GeminiApi>
Ƭ ConversationWith<T>: Conversation<ClientOfChatApi<T>, OptionsOfChatApi<T>, PromptOfChatApi<T>, ResponseOfChatApi<T>>
| Name |
|---|
T |
Ƭ ConversationWithChatGpt: ConversationWith<ChatGptApi>
Ƭ ConversationWithGemini: ConversationWith<GeminiApi>
Ƭ SupportedChatApiOptions: ChatGptOptions | GeminiOptions
Ƭ ChatGptClient: OpenAIClient
Ƭ ChatGptCompletionOptions: { deploymentName: string } & AdditionalCompletionOptions & Parameters<OpenAIClient["getChatCompletions"]>[2]
Ƭ ChatGptOptions: { extractVideoFrames?: ExtractVideoFramesOptions ; storage: StorageOptions ; videoRetrievalIndex?: VideoRetrievalIndexOptions } & ChatApiOptions<OpenAIClientOptions, ChatGptCompletionOptions>
Ƭ ChatGptPrompt: Parameters<OpenAIClient["getChatCompletions"]>[1]
Ƭ ChatGptResponse: ChatCompletions
Ƭ GeminiClient: GenerativeModel
Ƭ GeminiClientOptions: Object
| Name | Type |
|---|---|
modelParams |
ModelParams |
requestOptions? |
RequestOptions |
Ƭ GeminiCompletionOptions: AdditionalCompletionOptions & Omit<GenerateContentRequest, "contents">
Ƭ GeminiOptions: { clientSettings: GeminiClientOptions ; extractVideoFrames: ExtractVideoFramesOptions } & ChatApiOptions<GeminiClientOptions, GeminiCompletionOptions>
Ƭ GeminiPrompt: GenerateContentRequest["contents"]
Ƭ GeminiResponse: GenerateContentResult
Re-exports AdditionalCompletionOptions
Re-exports BuildPromptOutput
Re-exports ChatAboutVideo
Re-exports ChatAboutVideoWith
Re-exports ChatAboutVideoWithChatGpt
Re-exports ChatAboutVideoWithGemini
Re-exports ChatApi
Re-exports ChatApiOptions
Re-exports ClientOfChatApi
Re-exports Conversation
Re-exports ConversationWith
Re-exports ConversationWithChatGpt
Re-exports ConversationWithGemini
Re-exports ExtractVideoFramesOptions
Re-exports FileBatchUploader
Re-exports OptionsOfChatApi
Re-exports PromptOfChatApi
Re-exports ResponseOfChatApi
Re-exports StorageOptions
Re-exports SupportedChatApiOptions
Re-exports VideoFramesExtractor
Re-exports VideoRetrievalIndexOptions
Re-exports extractVideoFramesWithFfmpeg
Re-exports lazyCreatedFileBatchUploader
Re-exports lazyCreatedVideoFramesExtractor
Re-exports FileBatchUploader
▸ lazyCreatedFileBatchUploader(creator): FileBatchUploader
| Name | Type |
|---|---|
creator |
Promise<FileBatchUploader> |
Ƭ FileBatchUploader: (dir: string, relativePaths: string[], containerName: string, blobPathPrefix: string) => Promise<{ cleanup: () => Promise<any> ; downloadUrls: string[] }>
▸ (dir, relativePaths, containerName, blobPathPrefix): Promise<{ cleanup: () => Promise<any> ; downloadUrls: string[] }>
Function that uploads files to the cloud storage.
####### Parameters
| Name | Type | Description |
|---|---|---|
dir |
string |
The directory path where the files are located. |
relativePaths |
string[] |
An array of relative paths of the files to be uploaded. |
containerName |
string |
The name of the container where the files will be uploaded. |
blobPathPrefix |
string |
The prefix for the blob paths (file paths) in the container. |
####### Returns
Promise<{ cleanup: () => Promise<any> ; downloadUrls: string[] }>
A Promise that resolves with an object containing an array of download URLs for the uploaded files and a cleanup function to remove the uploaded files from the container.
- AdditionalCompletionOptions
- BuildPromptOutput
- ChatApi
- ChatApiOptions
- ExtractVideoFramesOptions
- StorageOptions
- VideoRetrievalIndexOptions
Ƭ ClientOfChatApi<T>: T extends ChatApi<infer CLIENT, any, any, any> ? CLIENT : never
| Name |
|---|
T |
Ƭ OptionsOfChatApi<T>: T extends ChatApi<any, infer OPTIONS, any, any> ? OPTIONS : never
| Name |
|---|
T |
Ƭ PromptOfChatApi<T>: T extends ChatApi<any, any, infer PROMPT, any> ? PROMPT : never
| Name |
|---|
T |
Ƭ ResponseOfChatApi<T>: T extends ChatApi<any, any, any, infer RESPONSE> ? RESPONSE : never
| Name |
|---|
T |
▸ effectiveExtractVideoFramesOptions(options?): Pick<ExtractVideoFramesOptions, "height"> & Required<Omit<ExtractVideoFramesOptions, "height">>
Calculate the effective values for ExtractVideoFramesOptions by combining the default values and the values provided
| Name | Type | Description |
|---|---|---|
options? |
ExtractVideoFramesOptions |
the options containing the values provided |
Pick<ExtractVideoFramesOptions, "height"> & Required<Omit<ExtractVideoFramesOptions, "height">>
The effective values for ExtractVideoFramesOptions
▸ effectiveStorageOptions(options): Required<Pick<StorageOptions, "uploader">> & StorageOptions
Calculate the effective values for StorageOptions by combining the default values and the values provided
| Name | Type | Description |
|---|---|---|
options |
StorageOptions |
the options containing the values provided |
Required<Pick<StorageOptions, "uploader">> & StorageOptions
The effective values for StorageOptions
▸ effectiveVideoRetrievalIndexOptions(options): Required<Pick<VideoRetrievalIndexOptions, "createIndexIfNotExists" | "deleteDocumentWhenConversationEnds" | "deleteIndexWhenConversationEnds">> & VideoRetrievalIndexOptions
Calculate the effective values for VideoRetrievalIndexOptions by combining the default values and the values provided
| Name | Type | Description |
|---|---|---|
options |
VideoRetrievalIndexOptions |
the options containing the values provided |
Required<Pick<VideoRetrievalIndexOptions, "createIndexIfNotExists" | "deleteDocumentWhenConversationEnds" | "deleteIndexWhenConversationEnds">> & VideoRetrievalIndexOptions
The effective values for VideoRetrievalIndexOptions
Re-exports VideoFramesExtractor
Re-exports extractVideoFramesWithFfmpeg
▸ lazyCreatedVideoFramesExtractor(creator): VideoFramesExtractor
| Name | Type |
|---|---|
creator |
Promise<VideoFramesExtractor> |
▸ extractVideoFramesWithFfmpeg(inputFile, outputDir, intervalSec, format?, width?, height?, startSec?, endSec?, limit?): Promise<{ cleanup: () => Promise<any> ; relativePaths: string[] }>
Function that extracts frame images from a video file.
| Name | Type | Description |
|---|---|---|
inputFile |
string |
Path to the input video file. |
outputDir |
string |
Path to the output directory where frame images will be saved. |
intervalSec |
number |
Interval in seconds between each frame extraction. |
format? |
string |
Format of the output frame images (e.g., 'jpg', 'png'). |
width? |
number |
Width of the output frame images in pixels. |
height? |
number |
Height of the output frame images in pixels. |
startSec? |
number |
Start time of the video segment to extract in seconds, inclusive. |
endSec? |
number |
End time of the video segment to extract in seconds, exclusive. |
limit? |
number |
Maximum number of frames to extract. |
Promise<{ cleanup: () => Promise<any> ; relativePaths: string[] }>
An object containing an array of relative paths to the extracted frame images and a cleanup function for deleting those files.
Ƭ VideoFramesExtractor: (inputFile: string, outputDir: string, intervalSec: number, format?: string, width?: number, height?: number, startSec?: number, endSec?: number, limit?: number) => Promise<{ cleanup: () => Promise<any> ; relativePaths: string[] }>
▸ (inputFile, outputDir, intervalSec, format?, width?, height?, startSec?, endSec?, limit?): Promise<{ cleanup: () => Promise<any> ; relativePaths: string[] }>
Function that extracts frame images from a video file.
####### Parameters
| Name | Type | Description |
|---|---|---|
inputFile |
string |
Path to the input video file. |
outputDir |
string |
Path to the output directory where frame images will be saved. |
intervalSec |
number |
Interval in seconds between each frame extraction. |
format? |
string |
Format of the output frame images (e.g., 'jpg', 'png'). |
width? |
number |
Width of the output frame images in pixels. |
height? |
number |
Height of the output frame images in pixels. |
startSec? |
number |
Start time of the video segment to extract in seconds, inclusive. |
endSec? |
number |
End time of the video segment to extract in seconds, exclusive. |
limit? |
number |
Maximum number of frames to extract. |
####### Returns
Promise<{ cleanup: () => Promise<any> ; relativePaths: string[] }>
An object containing an array of relative paths to the extracted frame images and a cleanup function for deleting those files.