Overview
Groq runs open-weight LLMs (Llama, Mixtral, Whisper) on LPU hardware that delivers very low latency. The Automize connector covers Chat Completions, transcription, translation, and model metadata via Groq's OpenAI-compatible API.
- Regions: Single global API at api.groq.com. The OpenAI-compatible surface lives at /openai/v1/* – same request shape as OpenAI Chat.
- Plans: Free tier has generous per-model RPM/TPM but caps daily tokens. Paid tier (on-demand) removes the daily cap and raises RPM. Whisper STT (transcribe + translate) bills per audio second.
Setting up the connection
Auth type: Bearer Token.
- In Groq Console → API Keys, click Create API Key (starts with gsk_).
- Paste as the API token. Connector sends Authorization: Bearer <key>.
Find your credentials at https://console.groq.com/keys.
Rate limits
Per-model RPM (requests/min), TPM (tokens/min), RPD (requests/day), TPD (tokens/day) – all four are tracked separately and any can fire 429. The X-RateLimit-Remaining-* headers expose live state; connector backs off on 429.
Data model
Groq's surface is intentionally narrow – it's an inference layer,
not a model owner. You pick a model + send a request:
Chat OpenAI-compatible chat completions
Audio Transcribe Whisper STT, source language preserved
Audio Translate Whisper-to-English (target=English only)
Models metadata + capability flags
Models accessible (subject to availability changes):
llama-3.3-70b-versatile general-purpose, tool-use capable
llama-3.1-8b-instant tiny + fast, weaker reasoning
llama-3.2-* (vision) multimodal variants
mixtral-8x7b-32768 long context, sparse-MoE
whisper-large-v3 STT (transcribe)
whisper-large-v3-turbo STT, faster, lossy on rare languages
Tool/function calling support is per-model. llama-3.3-70b-versatile
is the most reliable; smaller models occasionally hallucinate tool
names or skip required arguments.
Output streaming exists in the API; this connector resolves the
full response per call rather than streaming token-by-token.Available operations
5 operations available. Click any row to jump to its detail.
| Operation | Category | What it does |
|---|---|---|
| Chat Completion | General | Generate a chat completion |
| Get Model | General | Get model details |
| List Models | General | List available models |
| Transcribe Audio | General | Transcribe audio file via Whisper |
| Translate Audio | General | Translate audio to English via Whisper |
Operations
Chat Completion
Generate a chat completion
| Parameter | Required | Type | Description |
|---|---|---|---|
Model model | No | text | |
Messages messages | No | text | |
Temperature temperature | No | text | |
Max Tokens max_tokens | No | text | |
Top P top_p | No | text | |
Stop stop | No | text | |
Tools tools | No | text |
Tips
- OpenAI-compatible – same messages/role/content shape as OpenAI Chat. Tools/function calling supported on llama-3.1-* and llama-3.3-70b-versatile.
- max_tokens caps output. temperature=0 for deterministic; default 1.0 can vary across identical inputs.
- Stream support exists in the API but the connector resolves the full response.
Get Model
Get model details
| Parameter | Required | Type | Description |
|---|---|---|---|
Model Id model_id | Yes | text |
Tips
- GET /openai/v1/models/{model_id} – single model metadata.
- Useful as a credential health-check (cheap, deterministic) and to confirm a model_id is still available before kicking off a chain of Chat calls.
List Models
List available models
No input parameters.
Tips
- GET /openai/v1/models – returns every model accessible to the API key, with object='model', owned_by='Groq', and active flag.
- context_window varies by model – pin to the listed max in production; over-context returns a generic 400.
Transcribe Audio
Transcribe audio file via Whisper
| Parameter | Required | Type | Description |
|---|---|---|---|
File file | No | text | |
Model model | No | text | |
Language language | No | text | |
Prompt prompt | No | text | |
Response Format response_format | No | text | |
Temperature temperature | No | text |
Tips
- POST /openai/v1/audio/transcriptions – model=whisper-large-v3 or whisper-large-v3-turbo. Audio file (up to 25 MB).
- response_format=verbose_json returns word-level timestamps; default is text.
Translate Audio
Translate audio to English via Whisper
| Parameter | Required | Type | Description |
|---|---|---|---|
File file | No | text | |
Model model | No | text | |
Prompt prompt | No | text | |
Response Format response_format | No | text | |
Temperature temperature | No | text |
Tips
- Same as transcribe but outputs English regardless of source language. Use the English-target flow only – for other targets, transcribe then translate downstream.
FAQ
- Why is Groq so much faster than OpenAI for the same model?
- Different hardware. Groq's LPU is designed for sequential token generation – they trade batching efficiency for time-per-token. Expect 5-10x faster output for streaming workloads.
- Are responses identical to OpenAI's hosted Llama?
- Same weights, different inference stack. Outputs at temperature=0 are typically identical, but token-level scheduling can differ on edge cases (tool-use formatting, very long contexts).
- TPM vs RPM – which usually fires first?
- Depends on your prompt size. Heavy RAG prompts (8k+ input tokens) hit TPM first; chatty agentic loops hit RPM. Read X-RateLimit-Remaining-Tokens vs Remaining-Requests after the first call to calibrate.
- Tool calls work?
- Yes – tools=[{type:'function', function:{name,description,parameters}}] then read response.choices[0].message.tool_calls. llama-3.3-70b-versatile is the most reliable; smaller models hallucinate tool names.