A sophisticated, enterprise-grade HTTP client for Node.js and edge environments featuring intelligent cookie management, advanced web crawling capabilities, and comprehensive automation tools.
- Overview
- Installation
- Quick Start
- Core Features
- Web Crawler
- Configuration
- TypeScript Support
- Environment Detection
- Error Handling
- HTTP Methods Reference
- Performance & Security
- License
Uniqhtt represents a paradigm shift in HTTP client architecture, engineered specifically for modern applications requiring sophisticated request management, intelligent session handling, and seamless web automation. Built with a dual-runtime foundation, it delivers consistent performance across Node.js environments and edge computing platforms while maintaining enterprise-grade reliability and security standards.
- 🌐 Universal Runtime Compatibility - Seamlessly operates across Node.js, Deno, Bun, and edge environments including Cloudflare Workers
- 🍪 Intelligent Cookie Ecosystem - Advanced session management powered by
tough-cookie
with automatic persistence, domain handling, and multi-format serialization - 🕷️ Enterprise Web Crawler - Production-ready crawling engine featuring DOM manipulation, intelligent caching, and event-driven architecture
- 🔄 Adaptive Request Orchestration - Sophisticated queue management with configurable concurrency, rate limiting, and priority handling
- 🛡️ Multi-Protocol Proxy Integration - Comprehensive proxy support including SOCKS4/5, HTTP/HTTPS with authentication and failover mechanisms
- 📥 High-Performance File Operations - Memory-efficient streaming downloads with real-time progress tracking and metadata extraction
- 🔄 Resilient Retry Framework - Intelligent retry logic with exponential backoff, circuit breaking, and conditional retry strategies
- 🗜️ Transparent Content Processing - Automatic decompression handling for gzip, brotli, and deflate with streaming optimization
- 🔐 Security-First Design - Modern TLS contexts, certificate validation, secure cookie handling, and header sanitization
- 📊 TypeScript Native - Complete type safety with comprehensive interfaces and generic support for enhanced developer experience
npm install uniqhtt
yarn add uniqhtt
pnpm add uniqhtt
Experience the elegance of Uniqhtt's intuitive API designed for both simplicity and power:
import uniqhtt from 'uniqhtt';
// Effortless GET request with automatic response parsing
const response = await uniqhtt.get<User[]>('https://api.example.com/users');
console.log(response.data); // Fully typed response data
// Streamlined JSON POST with intelligent content-type handling
const user = await uniqhtt.postJson<User>('https://api.example.com/users', {
name: 'John Doe',
email: 'john@example.com',
preferences: { theme: 'dark', notifications: true }
});
// Form-encoded POST with automatic serialization
const loginResult = await uniqhtt.postForm<AuthResponse>('https://api.example.com/login', {
username: 'john',
password: 'secret',
rememberMe: true
});
console.log(`Welcome ${loginResult.data.user.name}!`);
Uniqhtt's file download capabilities provide enterprise-grade performance with comprehensive progress tracking and metadata extraction:
import { DownloadResponse } from 'uniqhtt';
// Direct-to-disk streaming download with automatic directory creation
const downloadResult: DownloadResponse<null> = await uniqhtt.download(
'https://example.com/large-dataset.zip',
'./downloads/dataset.zip'
);
const downloadResult2: DownloadResponse<null> = await uniqhtt.post('https://example.com/large-dataset.zip', {},
{saveTo: './downloads/dataset.zip'}
);
console.log(`✅ Download completed successfully:`);
console.log(` File: ${downloadResult.fileName}`);
console.log(` Size: ${downloadResult.size}`);
console.log(` Speed: ${downloadResult.downloadSpeed}`);
console.log(` Duration: ${downloadResult.totalTime}`);
// Alternative syntax for file downloads with full response access
const fileResponse = await uniqhtt.get<null>('https://example.com/report.pdf', {
saveTo: './reports/monthly-report.pdf'
});
Uniqhtt's cookie management system represents a sophisticated approach to session handling, providing seamless persistence and intelligent domain management across complex web interactions:
import { Uniqhtt, Cookie, SerializedCookie } from 'uniqhtt';
const client = new Uniqhtt();
// Programmatic cookie creation with comprehensive attribute support
const sessionCookie = new Cookie({
key: 'session_token',
value: 'eyJhbGciOiJIUzI1NiIs...',
domain: '.example.com',
path: '/',
secure: true,
httpOnly: true,
sameSite: 'strict',
expires: new Date(Date.now() + 24 * 60 * 60 * 1000) // 24 hours
});
client.setCookies([sessionCookie]);
// Bulk cookie import from various formats
const netscapeCookies = `
# Netscape HTTP Cookie File
.example.com TRUE / TRUE ${Date.now() + 86400000} auth_token abc123
.example.com TRUE /api FALSE 0 api_key xyz789
`;
client.setCookies(netscapeCookies);
// Automatic cookie persistence across requests
const dashboardData = await client.get<DashboardData>('https://example.com/dashboard');
const apiResponse = await client.post<ApiResult>('https://example.com/api/data', payload);
// Advanced cookie introspection and management
const currentCookies = client.getCookies();
console.log('Cookie String:', currentCookies.string); // HTTP Cookie header format
console.log('Netscape Format:', currentCookies.netscape); // For browser import/export
console.log('Serialized:', currentCookies.serialized); // For JSON storage
Uniqhtt's proxy infrastructure supports enterprise networking environments with comprehensive protocol support and intelligent failover mechanisms:
import { Uniqhtt, IProxy } from 'uniqhtt';
// Advanced SOCKS5 proxy configuration with authentication
const socksProxy: IProxy = {
protocol: 'socks5',
host: '127.0.0.1',
port: 9050,
username: 'proxy_user',
password: 'secure_password',
keepAlive: true,
timeout: 30000,
maxSockets: 10
};
const client = new Uniqhtt({ proxy: socksProxy });
// Corporate HTTP proxy with custom configuration
const corporateProxy: IProxy = {
protocol: 'https',
host: 'proxy.corporate.com',
port: 8080,
username: 'employee_id',
password: 'access_token',
rejectUnauthorized: false, // For internal certificates
keepAliveMsecs: 60000
};
// All requests automatically route through configured proxy
const externalData = await client.get<ExternalApiResponse>('https://api.external.com/data');
console.log('Request routed through proxy:', externalData.status);
The integrated queue management system provides sophisticated traffic control, ensuring respectful interaction with target servers while maximizing throughput efficiency:
import { Uniqhtt, queueOptions } from 'uniqhtt';
// Enterprise-grade queue configuration for high-throughput scenarios
const queueConfig: queueOptions = {
concurrency: 5, // Maximum simultaneous requests
interval: 1000, // Time window in milliseconds
intervalCap: 2, // Requests allowed per interval
timeout: 30000, // Individual request timeout
throwOnTimeout: true, // Fail fast on queue timeout
autoStart: true // Begin processing immediately
};
const client = new Uniqhtt({
queueOptions: {
enable: true,
options: queueConfig
}
});
// Bulk operations automatically respect queue constraints
const endpoints: string[] = [
'https://api.example.com/users',
'https://api.example.com/products',
'https://api.example.com/orders',
// ... hundreds more URLs
];
const results = await Promise.all(
endpoints.map((url, index) =>
client.get<ApiResponse>(`${url}?page=${index}`)
)
); // Automatically throttled to queue specifications
console.log(`Processed ${results.length} requests with optimal rate limiting`);
Uniqhtt's retry framework implements intelligent failure recovery with adaptive backoff strategies and conditional retry logic:
import { UniqhttResponse, UniqhttError } from 'uniqhtt';
// Sophisticated retry configuration with exponential backoff
const resilientResponse = await uniqhtt.get<CriticalData>('https://unreliable-api.com/critical-data', {
retry: {
maxRetries: 5,
retryDelay: 1000, // Initial delay: 1 second
incrementDelay: true // Exponential backoff: 1s, 2s, 4s, 8s, 16s
},
timeout: 10000
});
console.log('Data retrieved successfully:', resilientResponse.data);
Advanced multipart handling with comprehensive file upload support and intelligent content-type detection:
import { FormData, UniqhttResponse } from 'uniqhtt';
import { readFileSync } from 'fs';
// Comprehensive multipart form construction
const formData = new FormData();
formData.append('document', readFileSync('./contract.pdf'), {
filename: 'contract-v2.pdf',
contentType: 'application/pdf'
});
formData.append('metadata', JSON.stringify({
title: 'Service Agreement',
version: '2.1',
confidential: true
}), { contentType: 'application/json' });
formData.append('signature', 'John Doe');
formData.append('timestamp', new Date().toISOString());
// Streamlined multipart upload with progress tracking
const uploadResult = await uniqhtt.postMultipart<UploadResponse>(
'https://api.example.com/documents/upload',
formData,
{
timeout: 120000, // Extended timeout for large files
headers: {
'X-Upload-Session': 'session_12345',
'X-Client-Version': '1.0.0'
}
}
);
console.log(`Document uploaded: ${uploadResult.data.documentId}`);
Uniqhtt's integrated crawler represents a production-ready web automation platform, featuring intelligent DOM manipulation, advanced caching strategies, and event-driven data extraction architecture.
The crawler provides an elegant, declarative approach to web data extraction with automatic queue management and intelligent content processing:
import uniqhtt from 'uniqhtt';
// Configure enterprise crawler with comprehensive options
const crawler = uniqhtt.crawler({
crawlerOptions: {
queueOptions: { concurrency: 3 },
baseUrl: 'https://ecommerce.example.com',
enableCache: true,
cacheTTL: 1000 * 60 * 60, // 1 hour cache
timeout: 15000,
maxRetryAttempts: 3,
retryDelay: 2000,
headers: {
'User-Agent': 'Uniqhtt-Crawler/1.0 (+https://company.com/bot)',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
}
}
});
// Event-driven data extraction with intelligent selectors
crawler
.onSelection<HTMLAnchorElement>('a[href*="/products/"]', (productLink) => {
const productUrl = productLink.getAttribute('href');
if (productUrl) {
crawler.visit(productUrl); // Queues for processing
}
})
.onSelection<HTMLElement>('.product-title', (titleElement) => {
console.log('Product Found:', titleElement.textContent?.trim());
})
.onSelection<HTMLElement>('.price', (priceElement) => {
const price = priceElement.textContent?.match(/\$(\d+\.?\d*)/)?.[1];
if (price) {
console.log('Price:', parseFloat(price));
}
})
.onJsonData<ProductData>((apiData) => {
// Handle JSON API responses within pages
console.log('API Data:', apiData.products.length, 'products loaded');
});
// Initiate crawling process (synchronous queue addition)
crawler.visit('/catalog');
crawler.visit('/featured-products');
crawler.visit('/new-arrivals');
// Wait for all queued operations to complete (asynchronous)
await crawler.waitForAll(); // Wait for all queued requests
console.log('✅ Crawling completed successfully');
Advanced crawler configuration supporting differentiated handling of internal and external requests:
// Primary crawler for main domain with proxy routing
const advancedCrawler = uniqhtt.crawler({
crawlerOptions: {
queueOptions: { concurrency: 4 },
baseUrl: 'https://main-site.com',
socksProxies: [
{
protocol: 'socks5',
host: '127.0.0.1',
port: 9050
}
],
retryOnProxyError: true,
enableCache: true,
debug: true
},
// Secondary client for external domain handling
crawler2Options: {
timeout: 8000,
useProxy: false, // Bypass proxy for external requests
concurrency: 2, // Lower concurrency for external sites
respectQueueOption: true, // Honor queue settings
maxRetryAttempts: 1 // Reduced retries for external resources
}
});
// Intelligent request routing based on domain
advancedCrawler
.onSelection<HTMLAnchorElement>('a[href]', (link) => {
const href = link.getAttribute('href');
if (!href) return;
const url = new URL(href, 'https://main-site.com');
if (url.hostname === 'main-site.com') {
// Internal links use primary client (with proxy)
crawler.visit(url.pathname);
} else {
// External links use secondary client (direct connection)
crawler.visit2(url.href);
}
});
crawler.visit('/');
await crawler.waitForAll(); // Wait for all queued requests
Comprehensive event system for sophisticated data extraction and processing workflows:
interface ProductData {
id: string;
name: string;
price: number;
availability: boolean;
}
const dataCrawler = uniqhtt.crawler({
crawlerOptions: {
queueOptions: { concurrency: 5 },
baseUrl: 'https://api-driven-site.com',
enableCache: true
}
});
// Multiple event handlers for comprehensive data capture
dataCrawler
.onDocument((document) => {
// Full document processing
const title = document.querySelector('title')?.textContent;
console.log('Page Title:', title);
})
.onResponse<string>((response) => {
// Access to full HTTP response
console.log(`Response: ${response.status} - ${response.finalUrl}`);
console.log(`Content-Type: ${response.contentType}`);
})
.onJsonData<ProductData[]>((jsonData) => {
// Automatic JSON parsing and type safety
jsonData.forEach(product => {
console.log(`Product: ${product.name} - $${product.price}`);
});
})
.onRawData((buffer) => {
// Raw response data access for custom processing
console.log(`Received ${buffer.length} bytes`);
})
.onAnchor((anchor) => {
// Specialized anchor element handling
const href = anchor.href;
const text = anchor.textContent;
console.log(`Link: ${text} -> ${href}`);
})
.onAttribute('data-product-id', (productId) => {
// Extract specific attributes across all elements
console.log('Product ID found:', productId);
});
dataCrawler.visit('/products');
await dataCrawler.waitForAll(); // Wait for all queued requests
Establish consistent behavior across all requests with comprehensive global settings:
import { Uniqhtt } from 'uniqhtt';
const client = new Uniqhtt({
baseURL: 'https://api.enterprise.com/v2',
timeout: 30000,
headers: {
'User-Agent': 'Enterprise-App/2.1.0',
'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIs...',
'X-API-Version': '2.1',
'X-Client-ID': 'app_12345'
},
// Browser behavior simulation
mimicBrowser: true,
// Global retry policy
retry: {
maxRetries: 3,
retryDelay: 1500,
incrementDelay: true
},
// Advanced networking options
rejectUnauthorized: true,
useSecureContext: true,
// Cookie management
enableCookieJar: true,
// Queue configuration for all requests
queueOptions: {
enable: true,
options: {
concurrency: 10,
interval: 500,
intervalCap: 5
}
}
});
Fine-tune individual requests with granular control over behavior and processing:
import { UniqhttResponse } from 'uniqhtt';
const customResponse = await uniqhtt.get<ApiData>('https://api.example.com/sensitive-data', {
headers: {
'X-Request-ID': crypto.randomUUID(),
'X-Priority': 'high'
},
timeout: 5000,
maxRedirects: 3,
// Custom response validation
validateStatus: (status: number): boolean => {
return status >= 200 && status < 300;
},
// Redirect handling with custom logic
onRedirect: ({ url, status, sameDomain, method }) => {
console.log(`Redirecting to: ${url.href} (${status})`);
if (!sameDomain) {
console.warn('Cross-domain redirect detected');
return {
redirect: false,
message: 'Cross-domain redirects not allowed'
};
}
return {
redirect: true,
method: status === 303 ? 'GET' : method,
withoutBody: status === 303
};
},
// Custom error recovery
retry: {
maxRetries: 2,
retryDelay: 1000,
incrementDelay: false
}
});
Uniqhtt's TypeScript integration provides comprehensive type safety with intelligent inference and generic support:
import uniqhtt, {
UniqhttResponse,
UniqhttError,
HttpConfig,
DownloadResponse
} from 'uniqhtt';
// Define API response interfaces
interface User {
id: number;
name: string;
email: string;
profile: {
avatar_url: string;
preferences: Record<string, any>;
};
}
interface ApiError {
error: string;
code: number;
details?: string[];
}
// Fully typed request with comprehensive error handling
async function fetchUserData(userId: number): Promise<User> {
try {
const response: UniqhttResponse<User> = await uniqhtt.get<User>(
`https://api.example.com/users/${userId}`,
{
headers: {
'Accept': 'application/json',
'X-Request-ID': crypto.randomUUID()
},
timeout: 10000
} as HttpConfig
);
// Full type safety on response properties
console.log(`User: ${response.data.name}`);
console.log(`Status: ${response.status}`);
console.log(`Content-Type: ${response.contentType}`);
console.log(`Final URL: ${response.finalUrl}`);
return response.data;
} catch (error) {
if (error instanceof UniqhttError) {
// Typed error handling
const errorData = error.response.data as ApiError;
console.error(`API Error: ${errorData.error} (${errorData.code})`);
console.error(`Status: ${error.response.status}`);
console.error(`URL: ${error.response.finalUrl}`);
}
throw error;
}
}
// Generic utility for typed API calls
async function apiCall<T, E = ApiError>(
endpoint: string,
options?: HttpConfig
): Promise<T> {
const response = await uniqhtt.get<T>(endpoint, options);
return response.data;
}
// Usage with full type inference
const users = await apiCall<User[]>('/users');
const settings = await apiCall<AppSettings>('/settings');
Uniqhtt automatically adapts to different runtime environments while maintaining consistent API behavior:
// Node.js environment - Full feature set including file system operations
import uniqhtt from 'uniqhtt/nodejs';
import { UniqhttNode } from 'uniqhtt';
const nodeClient = new UniqhttNode({
useCurl: true, // Enable cURL backend for advanced features
httpAgent: customHttpAgent,
httpsAgent: customHttpsAgent
});
// Edge environment - Optimized for serverless and edge computing
import uniqhtt from 'uniqhtt/edge';
import { UniqhttEdge } from 'uniqhtt';
const edgeClient = new UniqhttEdge({
timeout: 5000, // Shorter timeouts for edge environments
maxRedirects: 3
});
// Automatic detection (recommended) - Chooses optimal implementation
import uniqhtt, { Uniqhtt } from 'uniqhtt';
// Runtime detection happens automatically
const response = await uniqhtt.get('https://api.example.com/data');
Comprehensive error management with detailed context and intelligent recovery strategies:
import { UniqhttError, UniqhttResponse } from 'uniqhtt';
async function robustApiCall<T>(url: string): Promise<T> {
try {
const response = await uniqhtt.get<T>(url, {
retry: {
maxRetries: 3,
retryDelay: 1000,
incrementDelay: true
},
timeout: 15000
});
return response.data;
} catch (error) {
if (error instanceof UniqhttError) {
// Comprehensive error context
const errorInfo = {
message: error.message,
code: error.code,
status: error.response.status,
statusText: error.response.statusText,
url: error.response.finalUrl,
headers: error.response.headers,
data: error.response.data
};
console.error('Request failed:', errorInfo);
// Conditional error handling based on status
switch (error.response.status) {
case 401:
console.error('Authentication required');
// Trigger re-authentication flow
break;
case 403:
console.error('Access forbidden');
// Handle authorization failure
break;
case 429:
console.error('Rate limit exceeded');
// Implement backoff strategy
break;
case 500:
case 502:
case 503:
case 504:
console.error('Server error - retry may succeed');
// Server errors might be temporary
break;
default:
console.error('Unexpected error');
}
} else {
// Handle non-Uniqhtt errors
console.error('Unknown error:', error);
}
throw error;
}
}
Complete API reference for all supported HTTP methods with consistent TypeScript signatures:
// Core HTTP methods with full type support
const getData = await uniqhtt.get<ResponseType>(url, config);
const postResult = await uniqhtt.post<ResponseType>(url, data, config);
const putResult = await uniqhtt.put<ResponseType>(url, data, config);
const patchResult = await uniqhtt.patch<ResponseType>(url, data, config);
const deleteResult = await uniqhtt.delete<ResponseType>(url, config);
const headResult = await uniqhtt.head<ResponseType>(url, config);
const optionsResult = await uniqhtt.options<ResponseType>(url, config);
// Specialized content-type methods
const jsonResponse = await uniqhtt.postJson<ApiResponse>(url, jsonData, config);
const formResponse = await uniqhtt.postForm<FormResult>(url, formData, config);
const multipartResponse = await uniqhtt.postMultipart<UploadResult>(url, formData, config);
// File operations
const downloadResult: DownloadResponse<null> = await uniqhtt.download(url, localPath, config);
// Method variants for different HTTP verbs
const putJsonResult = await uniqhtt.putJson<UpdateResult>(url, jsonData, config);
const patchFormResult = await uniqhtt.patchForm<PatchResult>(url, formData, config);
// Buffer response handling
const bufferResponse: UniqhttResponse<Buffer> = await uniqhtt.get(url, {
returnBuffer: true
});
Uniqhtt implements numerous performance enhancements for optimal throughput and resource utilization:
- Intelligent Connection Pooling: Automatic HTTP/HTTPS agent management with configurable pool sizes and keep-alive settings
- Streaming Architecture: Memory-efficient processing for large files and datasets with minimal memory footprint
- Adaptive Compression: Automatic negotiation and decompression of gzip, brotli, and deflate with streaming optimization
- Smart Caching: Built-in response caching for crawler operations with TTL management and intelligent invalidation
- Queue Optimization: Sophisticated request queuing with priority handling and adaptive concurrency control
Enterprise-grade security implementations ensure safe operation in production environments:
- Modern TLS Configuration: Custom TLS contexts with contemporary cipher suites and protocol version enforcement
- Certificate Validation: Comprehensive certificate chain validation with custom CA support
- Secure Cookie Handling: Proper implementation of secure, httpOnly, and SameSite cookie attributes
- Header Sanitization: Automatic cleanup and validation of security-sensitive headers
- Proxy Security: Secure credential handling for authenticated proxy connections with encryption support
// Security-focused configuration example
const secureClient = new Uniqhtt({
rejectUnauthorized: true,
useSecureContext: true,
headers: {
'Strict-Transport-Security': 'max-age=31536000; includeSubDomains',
'X-Content-Type-Options': 'nosniff',
'X-Frame-Options': 'DENY'
},
enableCookieJar: true,
// Additional security configurations automatically applied
});
MIT © Uniqhtt Contributors
We welcome contributions! Please see our Contributing Guide for details on our development process, coding standards, and submission guidelines.
Uniqhtt - Where sophisticated HTTP client architecture meets elegant developer experience. From simple API integrations to complex web automation workflows, Uniqhtt scales with your requirements while maintaining enterprise-grade reliability and performance.