llm-management-updated

LLM Management Service (Updated)

An updated version of the LLM Management Service that uses Enapso MongoDB ORM for machine management.

Features

• ORM-based MongoDB integration using @innotrade/enapso-mongo-orm
• Simple interface for machine management
• Automatic machine synchronization every 2 hours
• VastAI API integration for real machine data

Prerequisites

• Node.js (v14+)
• MongoDB (running locally or accessible URL)
• VastAI API Key (optional, for fetching real machine data)

Installation

1. Clone the repository
2. Navigate to the project directory
3. Install dependencies:

npm install

4. Create a .env file in the root directory with the following configuration:

PORT=3200
MONGO_URL=mongodb://localhost:27017/llm-management
LOG_LEVEL=debug
VASTAI_API_KEY=your_vastai_api_key_here

Adjust the values according to your environment. The VASTAI_API_KEY is optional - if not provided, the service will fall back to using mock machine data.

Usage

Starting the Server

npm start

For development with auto-reload:

npm run dev

Testing the ORM Endpoint

To test just the ORM functionality:

npm run orm

This will start the server with the ORM endpoint exposed on port 3200 and print usage examples.

To test the ORM client that interacts with the endpoint:

npm run orm-client

This will send a test request to the ORM endpoint and display the results.

ORM Endpoint

The primary way to interact with the service is through the ORM endpoint at http://localhost:3200/orm. This accepts POST requests to interact with the Machine model using the ENAPSO Mongo ORM.

Standard CRUD Operations

The Machine model supports standard CRUD operations:

1. Read (Find Machines):

{
  "class": "Machine",
  "method": "read",
  "filter": [
    { "property": "provider", "value": "vastai" },
    { "property": "gpu_count", "operator": "$gte", "value": 1 }
  ],
  "sort": [ { "property": "price_per_hour", "direction": "ASC" } ],
  "limit": 5
}

2. Get by ID:

{
  "class": "Machine",
  "method": "getById",
  "_id": "mongoose_object_id_string"
}

3. Create Machine (Manual):

{
  "class": "Machine",
  "method": "create",
  "args": {
    "data": {
      "id": "manual-machine-01",
      "name": "My Manual Machine",
      "provider": "manual",
      "price_per_hour": 0.5,
      "gpu_name": "Custom GPU",
      "gpu_count": 2
      // ... other fields
    }
  }
}

4. Update Machine:

{
  "class": "Machine",
  "method": "update",
  "args": {
    "_id": "mongoose_object_id_string",
    "data": {
      "name": "Updated Machine Name",
      "instanceStatus": "idle"
    }
  }
}

5. Delete Machine:

{
  "class": "Machine",
  "method": "delete",
  "args": {
    "_id": "mongoose_object_id_string"
  }
}

VastAI Machine Operations

The Machine model provides custom methods for managing Vast.ai instances:

1. Start Machine:

{
  "class": "Machine",
  "method": "startMachine",
  "args": {
    "machineId": "12345",
    "options": {
      "model": "llama3",
      "diskSpace": 215,
      "timeout": 300000,
      "checkInterval": 10000
    }
  }
}

You can use either machineId or ask_contract_id to identify the machine. Both now use the original Vast.ai ID without the "vastai-" prefix that was used in earlier versions.

The response will return immediately with a success status and instance ID. The service then continues monitoring the instance startup in the background and updates the machine's record in the database once running.

When the machine reaches 'running' status, the system will automatically create an accessUrl field that provides direct access to the WebUI (typically on port 9090). This URL is constructed using the machine's public IP address and exposed port, and can be used to access the Ollama WebUI interface directly from a browser.

2. Stop Machine:

{
  "class": "Machine",
  "method": "stopMachine",
  "args": {
    "machineId": "12345" 
  }
}

You can use any of these parameters to identify the machine:

• machineId: The machine ID in the database (same as the original Vast.ai ID)
• ask_contract_id: The original Vast.ai offer ID (same as machineId)
• instanceId: The ID returned when starting a machine

3. Get Machine Status:

{
  "class": "Machine",
  "method": "getMachineStatus",
  "args": {
    "machineId": "12345" 
  }
}

This will return the current status of the machine (idle, starting, running, stopping, stopped, error). If the machine has an instanceId, it will query Vast.ai for the latest status and update the database record if needed.

Machine Synchronization

The service automatically syncs machine data from Vast.ai to the MongoDB database on startup and every 2 hours. If you provide a valid VastAI API key in the .env file, the service will fetch real-time machine offers from Vast.ai.

The synchronization process:

1. Fetches available machine offers from Vast.ai
2. Maps them to the Machine schema format
3. Stores them in MongoDB through the ORM layer

To manually trigger synchronization, you can restart the service.

Development

Project Structure

• lib/
  • OrmService.js - Core service that defines the Machine model and its methods
  • VastAIService.js - Handles Vast.ai API interactions (machine start/stop/status)
  • MachineSyncService.js - Fetches and syncs machine offers from Vast.ai
  • LLMService.js - Main service that orchestrates all other services
  • config.js - Application configuration
• index.js - Application entry point
• .env - Environment variables (ignored by git)
• .env.example - Template for environment variables
• delete-machines.js - Utility script to delete all machines from the database
• package.json - Project metadata and dependencies

Schema Definition

The Machine schema includes fields for machine specifications, location, pricing, and instance state:

• Basic Info: id, name, provider
• Hardware: gpu_name, gpu_count, total_flops, cuda_cores, cpu_cores, cpu_ram, disk_space
• Pricing: price_per_hour
• Location: country, city, latitude, longitude
• Instance Status: instanceStatus, instanceId, instanceHost, instancePorts, accessUrl, instanceStartTime, instanceStopTime, instanceError
• Metadata: lastSyncTime, timestamps

Adding Custom Methods

You can extend the Machine schema with more methods in OrmService.js by adding them to the methods object in the schema definition. For example:

// Add this to the methods object in machineSchemaDefinition.methods
async findByGpuName(body) {
    const { args } = body;
    const gpuName = args?.gpuName;
    
    if (!gpuName) {
        return { 
            success: false, 
            status: 400, 
            message: 'gpuName is required in args' 
        };
    }
    
    try {
        // 'this.constructor' refers to the Model when called via ORM
        const results = await this.constructor.find({ gpu_name: gpuName });
        return {
            success: true,
            status: 200,
            message: 'Ok',
            data: results,
            count: results.length
        };
    } catch (error) {
        return { 
            success: false, 
            status: 500, 
            message: error.message 
        };
    }
}

You can then call this method via the ORM endpoint:

{
  "class": "Machine",
  "method": "findByGpuName",
  "args": {
    "gpuName": "RTX 4090"
  }
}