> For the complete documentation index, see [llms.txt](https://infronai.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infronai.gitbook.io/docs/support/provider-integration.md). # Provider Integration To be eligible to provide inference on Infron you must have the following: ### 1. List Models Endpoint You must implement an endpoint that returns all models that should be served by Infron. At this endpoint, please return a list of all available models on your platform. Below is an example of the response format: ```json { "data": [ { // Required "id": "anthropic/claude-sonnet-4", "hugging_face_id": "", // required if the model is on Hugging Face "name": "Anthropic: Claude Sonnet 4", "created": 1690502400, "input_modalities": ["text", "image", "file"], "output_modalities": ["text"], "quantization": "fp8", "context_length": 1000000, "max_output_length": 128000, "pricing": { "prompt": "0.000008", // pricing per 1 token "completion": "0.000024", // pricing per 1 token "cache_read": "0.0000008", "image": "0", // pricing per 1 image "request": "0", // pricing per 1 request "input_cache_read": "0", // pricing per 1 token "input_cache_write_5m": "0", // pricing per 1 token "input_cache_write_1h": "0" // pricing per 1 token }, "supported_sampling_parameters": ["temperature", "stop"], "supported_features": [ "tools", "json_mode", "structured_outputs", "web_search", "reasoning" ], // Optional "description": "Anthropic's flagship model...", "deprecation_date": "2025-06-01T15:00:00Z", // ISO 8601 date or UTC hour "is_ready": true, // false to keep the model staged-but-hidden on Infron "is_free": false, // true to mark as a free endpoint "capacity_tpm": 1000000, // input tokens per minute capacity for this model (optional) "infron": { "slug": "anthropic/claude-sonnet-4" }, "datacenters": [ { "country_code": "US" // `Iso3166Alpha2Code` } ] } ] } ``` The `id` field should be the exact model identifier that Infron will use when calling your API. The `pricing` fields are in string format to avoid floating point precision issues, and must be in USD. Valid input modalities are: `text`, `image`, `file`, `audio`, `video`. Valid output modalities are: `text`, `image`, `embeddings`, `audio`, `video`, `rerank`, `speech`, `transcription`. Valid quantization values are: `int4`, `int8`, `fp4`, `fp6`, `fp8`, `fp16`, `bf16`, `fp32`. Valid sampling parameters are: `temperature`, `top_p`, `top_k`, `min_p`, `top_a`, `frequency_penalty`, `presence_penalty`, `repetition_penalty`, `stop`, `seed`, `max_tokens`, `logit_bias`. Valid features are: `tools`, `json_mode`, `structured_outputs`, `logprobs`, `web_search`, `reasoning`. **Tiered Pricing** For models with different pricing based on context length (e.g., long context pricing), you can provide `pricing` as an array of tiers instead of a single object: ```json { "pricing": [ { "prompt": "0.000002", // base tier pricing per 1 token "completion": "0.000012", // base tier pricing per 1 token "image": "0.01", // pricing per 1 image (base tier only) "request": "0", // pricing per 1 request (base tier only) "input_cache_read": "0.000001" // base tier pricing per 1 token }, { "prompt": "0.000004", // long context tier pricing per 1 token "completion": "0.000018", // long context tier pricing per 1 token "input_cache_read": "0.000002", // long context tier pricing per 1 token "min_context": 200000 // minimum input tokens for this tier to apply } ] } ``` When using tiered pricing, the first tier (index 0) is the base pricing that applies when input tokens are below the `min_context` threshold. The second tier applies when input tokens meet or exceed the `min_context` value. **Deprecation Date** If a model is scheduled for deprecation, include the `deprecation_date` field in ISO 8601 format. Infron accepts either a date-only value or a specific UTC hour: ```json { "id": "anthropic/claude-2.1", "deprecation_date": "2025-06-01" } ``` * Use `YYYY-MM-DD` for date-only deprecations. Date-only values default to 13:00 UTC on that date. * Use `YYYY-MM-DDTHH:00:00Z` to request a specific UTC hour, for example `2025-06-01T15:00:00Z`. When Infron's provider monitor detects a deprecation date or time, it will automatically update the endpoint to display deprecation warnings to users. Models past their deprecation time may be automatically hidden from the marketplace. **Controlling Launch with `is_ready`** By default, when Infron's provider monitor sees a new model in your `/v1/models` response, it auto-stages the endpoint, runs baseline tests, and unhides it (makes it live) once the tests pass and pricing is configured. If you need to upload a model ahead of an announcement — or temporarily take a model offline — set the optional boolean `is_ready` field: ```json { "id": "your-org/upcoming-model", "is_ready": false } ``` Behavior: * `is_ready: false` skips baseline tests for newly-staged endpoints, keeping them hidden, and auto-hides any matching endpoint that is currently live. Use this to upload a model in advance of launch, or to take a live model offline coordinated with us. * `is_ready: true` and an omitted/absent field both preserve the default auto-stage and auto-unhide behavior. **Free Model Variants with `is_free`** If you want to offer a free version of a model, set `is_free: true`: ```json { "id": "your-org/your-model", "is_free": true } ``` Behavior: * `is_free: true` marks the endpoint as a free endpoint (`:free` suffix). * Any upstream `pricing` sent alongside `is_free: true` is ignored — free endpoints always have zero cost. * `is_free: false` or an omitted field preserves the default behavior (standard paid variant). **Capacity with `capacity_tpm`** Report your per-model throughput capacity so Infron can make better routing and capacity-planning decisions. The value is in **input tokens per minute**: ```json { "id": "your-org/your-model", "capacity_tpm": 5000000 } ``` * The value is an integer representing the input tokens per minute your infrastructure can process for this model (i.e. prompt/input throughput, not output generation). * Omitting the field or setting it to `null` leaves the capacity unknown (the default). * Infron's provider monitor auto-applies capacity changes when they appear in your `/v1/models` response. ### 2. Auto Top Up or Invoicing For Infron to use the provider we must be able to pay for inference automatically. This can be done via auto top up or invoicing. ### 3. Uptime Monitoring & Traffic Routing Infron automatically monitors provider reliability and adjusts traffic routing based on uptime metrics. Your endpoint's uptime is calculated as: **successful requests ÷ total requests** (excluding user errors). **Errors that affect your uptime:** * Authentication issues (401) * Model not found (404) * All server errors (500+) * Mid-stream errors * Successful requests with error finish reasons * Rate limiting (429) **Errors that DON'T affect uptime:** * Bad requests (400) - user input errors * Oversized payloads (413) - user input errors This system ensures traffic automatically flows to the most reliable providers while giving temporary issues time to resolve. ### 4. Performance Metrics Infron publicly tracks TTFT (time to first token) and throughput (tokens/second) for all providers on each model page. Throughput is calculated as: **output tokens ÷ generation time**, where generation time includes fetch latency (time from request to first server response), TTFT, and streaming time. This means any queueing on your end will show up in your throughput metrics. To keep your metrics competitive: * Return early 429s if under load, rather than queueing requests * Stream tokens as soon as they're available * If processing takes time (e.g. reasoning models), send SSE comments as keep-alives so we know you're still working on the request. Otherwise we may cancel with a fetch timeout and fallback to another provider ### **5. How deprioritization thresholds work** For each model, we compare every provider's signal values against the group of providers serving that model. We use a **median + MAD** (median absolute deviation) approach rather than simple averages, which keeps thresholds stable even when one provider is a significant outlier. Each signal has a different sensitivity: * **Throughput** -- providers falling more than **2 standard deviations** below the median are deprioritized. The wider margin accounts for natural throughput variance caused by time-of-day load patterns. * **Tool-calling success rate** -- providers falling more than **2 standard deviations** below the median are deprioritized. Success rates cluster near 100%, so this wider margin avoids penalizing normal noise while catching genuinely broken endpoints. A minimum of **4 providers** serving the same model is required before statistical thresholds are computed. Below that count, no deprioritization is applied for that signal. Endpoints are placed into one of three tiers: 1. **Good** -- sufficient data and no signals below threshold. These receive top routing priority. 2. **Insufficient data** -- not enough recent traffic to evaluate. These sort behind known-good providers but ahead of deprioritized ones. An endpoint needs at least 100 general requests (30-minute window) and 200 tool-call requests (2-hour window) before it can be evaluated. 3. **Deprioritized** -- one or more signals fell below threshold. These are routed to last. Consistent rate limiting (429s) can reduce the volume of successful requests available for evaluation, making it harder for us to collect enough benchmark data to place your endpoint in the top tier. Returning early 429s is still preferred over queueing, but minimizing rate limits where possible helps ensure your endpoint has sufficient data for a fair evaluation. ### **6. How to improve your ranking** To maximize the traffic routed to your endpoints: * **Maintain high tool-call reliability** -- ensure your endpoint returns well-formed tool call responses consistently. * **Optimize throughput** -- minimize queueing and stream tokens as soon as they are available (see Performance Metrics above). * **Return early 429s under load** -- rather than queueing and degrading throughput, return rate limit errors so we can retry with another provider and your metrics stay healthy. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://infronai.gitbook.io/docs/support/provider-integration.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.