# Querying text models
Fireworks.ai offers an OpenAI-compatible REST API for querying text models. There are several ways to interact with it:
* The [Fireworks Python client library](https://docs.fireworks.ai/tools-sdks/python-client/installation)
* The [web console](https://fireworks.ai/)
* [LangChain](https://python.langchain.com/docs/integrations/providers/fireworks)
* Directly invoking the [REST API](https://docs.fireworks.ai/api-reference/post-completions) using your favorite tools or language
* The [OpenAI Python client](https://github.com/openai/openai-python)
### [](https://docs.fireworks.ai/guides/querying-text-models#using-the-web-console)Using the web console
All Fireworks models can be accessed through the web console at [fireworks.ai](https://fireworks.ai/). Clicking on a model will take you to the playground where you can enter a prompt along with additional request parameters.
Non-chat models will use the [completions API](https://docs.fireworks.ai/api-reference/post-completions) which passes your input directly into the model.
Models with a conversation config are considered chat models (also known as instruct models). By default, chat models will use the [chat completions API](https://docs.fireworks.ai/api-reference/post-chatcompletions) which will automatically format your input with the conversation style of the model. Advanced users can revert back to the completions API by unchecking the “Use chat template” option.
### [](https://docs.fireworks.ai/guides/querying-text-models#using-the-api)Using the API
#### [](https://docs.fireworks.ai/guides/querying-text-models#chat-completions-api)Chat completions API
Models with a conversation config have the [chat completions API](https://docs.fireworks.ai/api-reference/post-completions) enabled. These models are typically tuned with specific conversation styles for which they perform best. For example, Llama chat models use the following [template](https://gpus.llm-utils.org/llama-2-prompt-template/):
> \\[INST] <\>
>
> <\>
>
> {user\_message\_1} \[/INST]
Some templates can support multiple chat messages as well. In general, we recommend users use the chat completions API whenever possible to avoid common prompt formatting errors. Even small errors like misplaced whitespace may result in poor model performance.
Here are some examples of calling the chat completions API:
Python (Fireworks)Python (OpenAI 1.x)Python (OpenAI 0.x)cURL
```python
from fireworks.client import Fireworks
client = Fireworks(api_key="")
response = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3-8b-instruct",
messages=[{
"role": "user",
"content": "Say this is a test",
}],
)
print(response.choices[0].message.content)
```
[****](https://docs.fireworks.ai/guides/querying-text-models#overriding-the-system-prompt)**Overriding the system prompt**
A conversation style may include a default system prompt. For example, Llama 2 models use the default Llama prompt:
> You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
For styles that support a system prompt, you may override this prompt by setting the first message with the role `system`. For example:
JSON
```json
[
{
"role": "system",
"content": "You are a pirate."
},
{
"role": "user",
"content": "Hello, what is your name?"
}
]
```
To completely omit the system prompt, you can set `content` to the empty string.
The process of generating a conversation-formatted prompt will depend on the conversation style used. To verify the exact prompt used, turn on [`echo`](https://docs.fireworks.ai/guides/querying-text-models#echo).
#### [](https://docs.fireworks.ai/guides/querying-text-models#completions-api)Completions API
Text models generate text based on the provided input prompt. All text models support this basic [completions API](https://docs.fireworks.ai/api-reference/post-completions). Using this API, the model will successively generate new tokens until either the maximum number of output tokens has been reached or if the model’s special end-of-sequence (EOS) token has been generated.
Most models will automatically prepend the beginning-of-sequence (BOS) token (e.g. ``) to your prompt input. You can always double-check by passing [raw\_output](https://docs.fireworks.ai/guides/querying-text-models#raw-output) and inspecting the resulting `prompt_token_ids`.
Here are some examples of calling the completions API:
Python (Fireworks)Python (OpenAI 1.x)Python (OpenAI 0.x)cURL
```python
from fireworks.client import Fireworks
client = Fireworks(api_key="")
response = client.completion.create(
model="accounts/fireworks/models/llama-v3-8b-instruct",
prompt="Say this is a test",
)
print(response.choices[0].text)
```
### [](https://docs.fireworks.ai/guides/querying-text-models#getting-usage-info)Getting usage info
The returned object will contain a `usage` field containing
* The number of prompt tokens ingested
* The number of completion tokens (i.e. the number of tokens generated)
### [](https://docs.fireworks.ai/guides/querying-text-models#advanced-options)Advanced options
See the API reference for the [completions](https://docs.fireworks.ai/api-reference/post-completions) and [chat completions](https://docs.fireworks.ai/api-reference/post-completions) APIs for a detailed description of these options.
#### [](https://docs.fireworks.ai/guides/querying-text-models#streaming)Streaming
By default, results are returned to the client once the generation is finished. Another option is to stream the results back, which is useful for chat use cases where the client can incrementally see results as each token is generated.
Here is an example with the completions API:
Python (Fireworks)Python (OpenAI 1.x)Python (OpenAI 0.x)cURL
```python
from fireworks.client import Fireworks
client = Fireworks(api_key="")
response_generator = client.completion.create(
model="accounts/fireworks/models/llama-v3p1-405b-instruct",
prompt="Say this is a test",
stream=True,
)
for chunk in response_generator:
print(chunk.choices[0].text)
```
and one with the chat completions API:
Python (Fireworks)Python (OpenAI 1.x)Python (OpenAI 0.x)
```python
from fireworks.client import Fireworks
client = Fireworks(api_key="")
response_generator = client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-8b-instruct",
messages=[{
"role": "user",
"content": "Say this is a test",
}],
stream=True,
)
for chunk in response_generator:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
#### [](https://docs.fireworks.ai/guides/querying-text-models#async-mode)Async mode
The Python client library also supports asynchronous mode for both completion and chat completion.
Python (Fireworks)Python (OpenAI 1.x)
```python
import asyncio
from fireworks.client import AsyncFireworks
client = AsyncFireworks(api_key="")
async def main():
stream = client.completion.acreate(
model="accounts/fireworks/models/llama-v3p1-8b-instruct",
prompt="Say this is a test",
stream=True,
)
async for chunk in stream:
print(chunk.choices[0].text, end="")
asyncio.run(main())
```
#### [](https://docs.fireworks.ai/guides/querying-text-models#predicted-outputs)Predicted Outputs
See [Using Predicted Outputs](https://docs.fireworks.ai/guides/predicted-outputs)
#### [](https://docs.fireworks.ai/guides/querying-text-models#sampling-options)Sampling options
The API auto-regressively generates text based on choosing the next token using the probability distribution over the space of tokens. For detailed information on how to implement these options, please refer to the [Chat Completions](https://docs.fireworks.ai/api-reference/post-chatcompletions) or [Completions](https://docs.fireworks.ai/api-reference/post-completions) API documentation.
[****](https://docs.fireworks.ai/guides/querying-text-models#multiple-choices)**Multiple choices**
By default, the API will return a single generation choice per request. You can create multiple generations by setting the `n` parameter to the number of desired choices. The returned `choices` array will contain the result of each generation.
[****](https://docs.fireworks.ai/guides/querying-text-models#max-tokens)**Max tokens**
`max_tokens` or `max_completion_tokens` defines the maximum number of tokens the model can generate, with a default of 2000. If the combined token count (prompt + output) exceeds the model’s limit, it automatically reduces the number of generated tokens to fit within the allowed context.
[****](https://docs.fireworks.ai/guides/querying-text-models#temperature)**Temperature**
Temperature allows you to configure how much randomness you want in the generated text. A higher temperature leads to more “creative” results. On the other hand, setting a temperature of 0 will allow you to generate deterministic results which is useful for testing and debugging.
[****](https://docs.fireworks.ai/guides/querying-text-models#top-p)**Top-p**
Top-p (also called nucleus sampling) is an alternative to sampling with temperature, where the model considers the results of the tokens with top\_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.
[****](https://docs.fireworks.ai/guides/querying-text-models#top-k)**Top-k**
Top-k is another sampling method where the k most probable tokens are filtered and the probability mass is redistributed among tokens.
[****](https://docs.fireworks.ai/guides/querying-text-models#min-p)**Min-p**
[`min_p`](https://arxiv.org/abs/2407.01082) specifies a probability threshold to control which tokens can be selected during generation. Tokens with probabilities lower than this threshold are excluded, making the model more focused on higher-probability tokens. The default value varies, and setting a lower value ensures more variety, while a higher value produces more predictable, focused outputs.
[****](https://docs.fireworks.ai/guides/querying-text-models#repetition-penalty)**Repetition penalty**
LLMs are sometimes prone to repeat a single character or a sentence. Using a frequency and presence penalty can reduce the likelihood of sampling repetitive sequences of tokens. They work by directly modifying the model’s logits (un-normalized log-probabilities) with an additive contribution.
> logits\[j] -= c\[j] \* frequency\_penalty + (c\[j] > 0 ? 1 : 0) \* presence\_penalty
where
* `logits[j]` is the logits of the j-th token
* `c[j]` is how often that token was sampled before the current position
The [`repetition_penalty`](https://arxiv.org/pdf/1909.05858.pdf) modifies the logit (raw model output) for repeated tokens. If a token has already appeared in the prompt or output, the penalty is applied to its probability of being selected again.
**Key differences to keep in mind:**
* `frequency_penalty`: Works on how often a word has been used, increasing the penalty for more frequent words. OAI compatible.
* `presence_penalty`: Penalizes words once they appear, regardless of frequency. OAI compatible.
* `repetition_penalty`: Adjusts the likelihood of repeated tokens based on previous appearances, providing an exponential scaling effect to control repetition more precisely, including from the prompt.
[****](https://docs.fireworks.ai/guides/querying-text-models#mirostat-learning-rate-and-target)**Mirostat (learning rate and target)**
The [Mirostat algorithm](https://arxiv.org/abs/2007.14966) is a sampling method that helps keep the output’s unpredictability, or perplexity, at a set target. It adjusts token probabilities as the text is generated to balance between more diverse or more predictable results. This is useful when you need steady control over how random or focused the text output should be.
There are two parameters that can be adjusted:
* `mirostat_target`: Sets the desired level of unpredictability (perplexity) for the Mirostat algorithm. A higher target results in more diverse output, while a lower target keeps the text more predictable.
* `mirostat_lr`: Controls how quickly the Mirostat algorithm adjusts token probabilities to reach the target perplexity. A lower learning rate makes the adjustments slower and more gradual, while a higher rate speeds up the corrections.
[****](https://docs.fireworks.ai/guides/querying-text-models#logit-bias)**Logit bias**
Parameter that modifies the likelihood of specified tokens appearing. Pass in a Dict\[int, float] that maps a token\_id to a logits bias value between -200.0 and 200.0. For example
python
```
client.completions.create(
model="...",
prompt="...",
logit_bias={0: 10.0, 2: -50.0}
)
```
### [](https://docs.fireworks.ai/guides/querying-text-models#debugging-options)Debugging options
#### [](https://docs.fireworks.ai/guides/querying-text-models#ignore-eos)Ignore EOS
This option allows you to control whether the model stops when it generated the End of Sequence (EOS) token. This is helpful primarily for performance benchmarking to reliably generate exactly `max_tokens`. Note the quality of the output may degrade as we override model’s decision to generate EOS token.
#### [](https://docs.fireworks.ai/guides/querying-text-models#logprobs)Logprobs
The `logprobs` parameter determines how many token probabilities are returned. If set to N, it will return log (base e) probabilities for N+1 tokens: the chosen token plus the N most likely alternative tokens.
The log probabilities will be returned in a LogProbs object for each choice.
* `tokens` contains each token of the chosen result.
* `token_ids` contains the integer IDs of each token of the chosen result.
* `token_logprobs` contains the logprobs of each chosen token.
* `top_logprobs` will be a list whose length is the number of tokens of the output. Each element is a dictionary of size `logprobs`, from the most likely tokens at the given position to their respective log probabilities.
When used in conjunction with echo, this option can be set to see how the model tokenized your input.
#### [](https://docs.fireworks.ai/guides/querying-text-models#top-logprobs)Top logprobs
Setting the `top_logprobs` parameter to an integer value in conjunction with `logprobs=True` will also return the above information but in an OpenAI client-compatible format.
#### [](https://docs.fireworks.ai/guides/querying-text-models#echo)Echo
Setting the `echo` parameter to true will cause the API to return the prompt along with the generated text. This can be used in conjunction with the chat completions API to verify the prompt template used. It can also be used in conjunction with logprobs to see how the model tokenized your input.
#### [](https://docs.fireworks.ai/guides/querying-text-models#raw-output)Raw output
This is an unstable, experimental API. It may change at any time and should not be relied upon for production use cases.
Setting the `raw_output` parameter to true will cause the API to return a `raw_output` object in the response containing addititional debugging information with regards to how the raw prompt and completion response as seen/produced by the model.
* `prompt_fragments` - Pieces of the prompt (like individual messages) before truncation and concatenation.
* `prompt_token_ids` - Fully tokenized prompt as seen by the model.
* `completion` - Raw completion produced by the model before any tool calls are parsed.
* `completion_logprobs` - Log probabilities for the completion. Only populated if `logprobs` is specified in the request.
### [](https://docs.fireworks.ai/guides/querying-text-models#appendix)Appendix
#### [](https://docs.fireworks.ai/guides/querying-text-models#tokenization)Tokenization
Language models read and write text in chunks called tokens. In English, a **token** can be as short as one character or as long as one word (e.g., a or apple), and in some languages, tokens can be even shorter than one character or even longer than one word.
Different model families use different **tokenizers**. The same text might be translated to different numbers of tokens depending on the model. It means that generation cost may vary per model even if the model size is the same. For the Llama model family, you can use [this tool](https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/) to estimate token counts. The actual number of tokens used in prompt and generation is returned in the `usage` field of the API response.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://fireworks.gitbook.io/fireworks/inference/querying-text-models.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.