alican-test-cli

Needle Control CLI Tool

needlectl is local client for MacOS, Windows, Linux to sync files to Needle.

Features

• Real-time file monitoring and synchronization
• Automatic file state management
• Support for multiple file types
• Recursive and non-recursive folder monitoring
• Efficient handling of file changes, additions, and deletions

Quick Start

First, install PM2 globally (required for daemon management):

npm install -g pm2

Then install needlectl command line tool globally using your favourite package manager, we recommend using bun:

bun install -g @needle-ai/needlectl

For npm users:

npm install -g @needle-ai/needlectl

Follow these steps to set up:

1. Verify installation:

needlectl --version

2. Set your API key:

needlectl api-key set <your-needle-api-key>

3. Create a collection:

needlectl collections create --name "My Collection" --model basilikum-minima

Or use an existing collection ID from Needle.

4. Create a connector (replace with your collection ID):

# For recursive monitoring of the folder
needlectl connectors create --collection-id=<collection-id> -r /path/to/folder

# For non-recursive monitoring of the folder
needlectl connectors create --collection-id=<collection-id> -f /path/to/folder

5. Start the daemon:

needlectl start

Voilà! The daemon will start monitoring folders provided and sync files to Needle. Go to Needle and start asking questions right away.

Development Setup

To develop and test the CLI tool locally:

1. Install PM2 globally (required for daemon management):

npm install -g pm2

2. Clone the repository and install dependencies:

git clone <repository-url>
cd needle-cli-tool
bun install

3. Set development environment:

export NODE_ENV=development

4. Build the project:

bun run build

5. Link the package globally:

bun link

Now you can use needlectl command locally, and any changes you make will be reflected after rebuilding (bun run build).

Development Workflow

1. Make your changes in the src directory
2. Rebuild the project: bun run build
3. Test your changes using the needlectl command
4. For debugging, check the logs:

   tail -f logs/needle-daemon-out.log
   tail -f logs/needle-daemon-error.log

Command Reference

Collection Management

# Create a collection
needlectl collections create --name "Test Collection" --model basilikum-minima

# List collections
needlectl collections list

# Get collection details
needlectl collections get <collection-id>

Connector Management

# Create a connector
needlectl connectors create --collection-id=<id> -r /path/to/folder

# List connectors (with tree view)
needlectl connectors ls
# Shows a formatted table with:
# - Connector details (ID, Collection, Device, MAC)
# - Folder tree structure for recursive monitoring
# Example output:
# CONNECTORS
# ────────────────────────────────────────
# ID: local_123456789
# Collection: clt_987654321
# Device: my-device (darwin)
# MAC: xx:xx:xx:xx:xx:xx
#
# Monitored Folders:
# my_folder (recursive)
# ├── file1.txt
# ├── subfolder/
# │   ├── file2.txt
# │   └── file3.txt
# └── file4.txt
# ────────────────────────────────────────

# Get connector details
needlectl connectors get <connector-id>

Daemon Control

# Start the daemon
needlectl start
# Output: Successfully started daemon with PID <number>
# Or: Needle daemon with PID <number> is already running

# Check daemon status
needlectl status
# Output: Needle daemon with PID <number> is running
# Or: No running needle daemon found

# Stop the daemon
needlectl stop
# Output: Stopped daemon with PID <number>
# Or: No running needle daemon found

# View daemon logs
needlectl logs
# Shows detailed daemon activity logs

# Restart the daemon
needlectl stop && needlectl start

Initial Sync Behavior

When setting up a new monitoring environment:

1. You can create files first, then set up the connector and start the daemon:

   # Create test files
   mkdir -p my_folder && echo "content" > my_folder/file.txt
   
   # Create connector
   needlectl connectors create --collection-id=<id> -r my_folder
   
   # Start daemon
   needlectl start

   The daemon will perform an initial sync of all existing files.

2. Or you can set up the connector and daemon first, then add files:

   # Set up monitoring
   needlectl connectors create --collection-id=<id> -r my_folder
   needlectl start
   
   # Add files later
   echo "content" > my_folder/file.txt

   The daemon will detect and sync new files as they are added.

In both cases, the daemon will:

• Perform initial sync of existing files when started
• Monitor for new files and changes
• Maintain proper file state tracking

File Synchronization

The daemon automatically:

• Monitors specified folders for changes
• Syncs new files to Needle platform
• Handles file updates (delete + create)
• Processes file deletions
• Maintains file state tracking

Supported File Types

• PDF (application/pdf)
• Text (text/plain)
• Word (application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document)
• Excel (application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
• PowerPoint (application/vnd.ms-powerpoint, application/vnd.openxmlformats-officedocument.presentationml.presentation)
• CSV (text/csv)
• HTML (text/html)
• Calendar (text/calendar)

This list grows every day, for more up-to-date information please check Needle.

Architecture

File State Management

The tool maintains file states in .needle/config.json:

• Tracks file hashes to detect changes
• Stores Needle file IDs
• Manages sync timestamps
• Handles file deletions

Monitoring Modes

1. Recursive (-r flag)

   • Monitors entire folder tree
   • Includes all subfolders
   • No depth limit

2. Non-recursive (-f flag)

   • Only monitors specified folder
   • Ignores subfolders

Error Handling & Logging

Error Handling

• Detailed error logging
• Automatic retry on failures
• Graceful shutdown handling
• State persistence

Logs

Logs are stored in the logs directory:

needlectl logs

or read from the file directly:

# View live logs
tail -f logs/needle-daemon-out.log

# View error logs
tail -f logs/needle-daemon-error.log

Common Issues

1. "API key not found"

   • Solution: Set API key using needlectl api-key set <key>

2. "Collection not found"

   • Solution: Verify collection ID using needlectl collections list

3. "Upload failed"

   • Check API key validity
   • Verify file size limits
   • Check network connection

Testing

1. Create test directory:

mkdir -p test_folder

2. Add test files:

echo "Test content" > test_folder/test.txt
echo "Test HTML" > test_folder/test.html

3. Test operations:

# Modify file
echo "Updated content" >> test_folder/test.txt

# Delete file
rm test_folder/test.html

Troubleshooting

Common Scenarios

1. Files not syncing:

# Check daemon status
needlectl status

# Restart daemon
needlectl stop && needlectl start

# Check logs
needlectl logs

2. Collection issues:

# Verify collection exists
needlectl collections list

# Check collection details
needlectl collections get <collection-id>

3. Connector issues:

# List connectors
needlectl connectors ls

# Recreate connector if needed
needlectl connectors create --collection-id=<id> -r /path/to/folder