Ctrlk
ModelsLoginLLMs.txt
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Provider Integration

To be eligible to provide inference on Infron you must have the following:

1. List Models Endpoint

You must implement an endpoint that returns all models that should be served by Infron. At this endpoint, please return a list of all available models on your platform. Below is an example of the response format:

{
  "data": [
    {
      // Required
      "id": "anthropic/claude-sonnet-4",
      "hugging_face_id": "", // required if the model is on Hugging Face
      "name": "Anthropic: Claude Sonnet 4",
      "created": 1690502400,
      "input_modalities": ["text", "image", "file"],
      "output_modalities": ["text"],
      "quantization": "fp8",
      "context_length": 1000000,
      "max_output_length": 128000,
      "pricing": {
        "prompt": "0.000008", // pricing per 1 token
        "completion": "0.000024", // pricing per 1 token
        "cache_read": "0.0000008",
        "image": "0", // pricing per 1 image
        "request": "0", // pricing per 1 request
        "input_cache_read": "0", // pricing per 1 token
        "input_cache_write_5m": "0", // pricing per 1 token
        "input_cache_write_1h": "0" // pricing per 1 token
      },
      "supported_sampling_parameters": ["temperature", "stop"],
      "supported_features": [
        "tools",
        "json_mode",
        "structured_outputs",
        "web_search",
        "reasoning"
      ],
      // Optional
      "description": "Anthropic's flagship model...",
      "deprecation_date": "2025-06-01T15:00:00Z", // ISO 8601 date or UTC hour
      "is_ready": true, // false to keep the model staged-but-hidden on Infron
      "is_free": false, // true to mark as a free endpoint
      "capacity_tpm": 1000000, // input tokens per minute capacity for this model (optional)
      "infron": {
        "slug": "anthropic/claude-sonnet-4"
      },
      "datacenters": [
        {
          "country_code": "US" // `Iso3166Alpha2Code`
        }
      ]
    }
  ]
}

The id field should be the exact model identifier that Infron will use when calling your API.

The pricing fields are in string format to avoid floating point precision issues, and must be in USD.

Valid input modalities are: text, image, file, audio, video.

Valid output modalities are: text, image, embeddings, audio, video, rerank, speech, transcription.

Valid quantization values are: int4, int8, fp4, fp6, fp8, fp16, bf16, fp32.

Valid sampling parameters are: temperature, top_p, top_k, min_p, top_a, frequency_penalty, presence_penalty, repetition_penalty, stop, seed, max_tokens, logit_bias.

Valid features are: tools, json_mode, structured_outputs, logprobs, web_search, reasoning.

Tiered Pricing

For models with different pricing based on context length (e.g., long context pricing), you can provide pricing as an array of tiers instead of a single object:

When using tiered pricing, the first tier (index 0) is the base pricing that applies when input tokens are below the min_context threshold. The second tier applies when input tokens meet or exceed the min_context value.

Deprecation Date

If a model is scheduled for deprecation, include the deprecation_date field in ISO 8601 format. Infron accepts either a date-only value or a specific UTC hour:

  • Use YYYY-MM-DD for date-only deprecations. Date-only values default to 13:00 UTC on that date.

  • Use YYYY-MM-DDTHH:00:00Z to request a specific UTC hour, for example 2025-06-01T15:00:00Z.

When Infron's provider monitor detects a deprecation date or time, it will automatically update the endpoint to display deprecation warnings to users. Models past their deprecation time may be automatically hidden from the marketplace.

Controlling Launch with is_ready

By default, when Infron's provider monitor sees a new model in your /v1/models response, it auto-stages the endpoint, runs baseline tests, and unhides it (makes it live) once the tests pass and pricing is configured. If you need to upload a model ahead of an announcement — or temporarily take a model offline — set the optional boolean is_ready field:

Behavior:

  • is_ready: false skips baseline tests for newly-staged endpoints, keeping them hidden, and auto-hides any matching endpoint that is currently live. Use this to upload a model in advance of launch, or to take a live model offline coordinated with us.

  • is_ready: true and an omitted/absent field both preserve the default auto-stage and auto-unhide behavior.

Free Model Variants with is_free

If you want to offer a free version of a model, set is_free: true:

Behavior:

  • is_free: true marks the endpoint as a free endpoint (:free suffix).

  • Any upstream pricing sent alongside is_free: true is ignored — free endpoints always have zero cost.

  • is_free: false or an omitted field preserves the default behavior (standard paid variant).

Capacity with capacity_tpm

Report your per-model throughput capacity so Infron can make better routing and capacity-planning decisions. The value is in input tokens per minute:

  • The value is an integer representing the input tokens per minute your infrastructure can process for this model (i.e. prompt/input throughput, not output generation).

  • Omitting the field or setting it to null leaves the capacity unknown (the default).

  • Infron's provider monitor auto-applies capacity changes when they appear in your /v1/models response.

2. Auto Top Up or Invoicing

For Infron to use the provider we must be able to pay for inference automatically. This can be done via auto top up or invoicing.

3. Uptime Monitoring & Traffic Routing

Infron automatically monitors provider reliability and adjusts traffic routing based on uptime metrics. Your endpoint's uptime is calculated as: successful requests ÷ total requests (excluding user errors).

Errors that affect your uptime:

  • Authentication issues (401)

  • Model not found (404)

  • All server errors (500+)

  • Mid-stream errors

  • Successful requests with error finish reasons

  • Rate limiting (429)

Errors that DON'T affect uptime:

  • Bad requests (400) - user input errors

  • Oversized payloads (413) - user input errors

This system ensures traffic automatically flows to the most reliable providers while giving temporary issues time to resolve.

4. Performance Metrics

Infron publicly tracks TTFT (time to first token) and throughput (tokens/second) for all providers on each model page.

Throughput is calculated as: output tokens ÷ generation time, where generation time includes fetch latency (time from request to first server response), TTFT, and streaming time. This means any queueing on your end will show up in your throughput metrics.

To keep your metrics competitive:

  • Return early 429s if under load, rather than queueing requests

  • Stream tokens as soon as they're available

  • If processing takes time (e.g. reasoning models), send SSE comments as keep-alives so we know you're still working on the request. Otherwise we may cancel with a fetch timeout and fallback to another provider

5. How deprioritization thresholds work

For each model, we compare every provider's signal values against the group of providers serving that model. We use a median + MAD (median absolute deviation) approach rather than simple averages, which keeps thresholds stable even when one provider is a significant outlier.

Each signal has a different sensitivity:

  • Throughput -- providers falling more than 2 standard deviations below the median are deprioritized. The wider margin accounts for natural throughput variance caused by time-of-day load patterns.

  • Tool-calling success rate -- providers falling more than 2 standard deviations below the median are deprioritized. Success rates cluster near 100%, so this wider margin avoids penalizing normal noise while catching genuinely broken endpoints.

A minimum of 4 providers serving the same model is required before statistical thresholds are computed. Below that count, no deprioritization is applied for that signal.

Endpoints are placed into one of three tiers:

  1. Good -- sufficient data and no signals below threshold. These receive top routing priority.

  2. Insufficient data -- not enough recent traffic to evaluate. These sort behind known-good providers but ahead of deprioritized ones. An endpoint needs at least 100 general requests (30-minute window) and 200 tool-call requests (2-hour window) before it can be evaluated.

  3. Deprioritized -- one or more signals fell below threshold. These are routed to last.

Consistent rate limiting (429s) can reduce the volume of successful requests available for evaluation, making it harder for us to collect enough benchmark data to place your endpoint in the top tier. Returning early 429s is still preferred over queueing, but minimizing rate limits where possible helps ensure your endpoint has sufficient data for a fair evaluation.

6. How to improve your ranking

To maximize the traffic routed to your endpoints:

  • Maintain high tool-call reliability -- ensure your endpoint returns well-formed tool call responses consistently.

  • Optimize throughput -- minimize queueing and stream tokens as soon as they are available (see Performance Metrics above).

  • Return early 429s under load -- rather than queueing and degrading throughput, return rate limit errors so we can retry with another provider and your metrics stay healthy.

PreviousJoin Community

Last updated