Skip to main content
Version: PromptQL

Providers & Models

Introduction​

PromptQL supports configuring different LLM providers and models to tailor the experience to your application's needs.

This gives you the flexibility to choose the most performance-efficient and cost-effective solution for your use case, and the freedom to switch between providers and models as needed depending the task.

For example, you can use one model for conversational tasks and another for advanced AI primitives:

Example of globals/metadata/promptql-config.hml
kind: PromptQlConfig
version: v2
definition:
  llm:
    provider: openai
    model: o3-mini
    apiKey:
      valueFromEnv: OPENAI_API_KEY
  aiPrimitivesLlm:
    provider: openai
    model: gpt-4o
    apiKey:
      valueFromEnv: OPENAI_API_KEY
  overrideAiPrimitivesLlm:
    summarize:
      provider: anthropic
      model: claude-3-5-sonnet-latest
      apiKey:
        valueFromEnv: ANTHROPIC_API_KEY
Mapping environment variables

If you do specify environment variables in your promptql-config.hml, don't forget to add them to the globals subgraph's subgraph.yaml under the envMapping section.

llm​

The llm configuration is used to define the LLM provider and model for conversational tasks in your application. In the example above, we're using openai as the provider with the o3-mini model.

aiPrimitivesLlm​

The aiPrimitivesLlm configuration is used to define the LLM provider and model for AI primitives in your application. This is used for tasks such as program generation and execution. In the example above, we're using openai as the provider with the gpt-4o model.

If aiPrimitiveLlm is not specified, the llm configuration is used for AI primitives as well.

overrideAiPrimitivesLlm​

If you need to use a different LLM for a specific AI primitive, you can use the overrideAiPrimitivesLlm configuration. This is useful if you want to use a different model for a specific primitive. Example primitives are summarize, classify, and extract. In the example above, we're using anthropic as the provider with the claude-3-5-sonnet-latest model for the summarize primitive.

Any specific AI primitive will first use the LLM specified in overrideAiPrimitivesLlm before falling back to the aiPrimitivesLlm configuration if nothing is specified, and then the llm configuration if nothing is specified there either.

Available providers & models​

Anthropic​

To use an Anthropic model, set the provider to anthropic. The following have been tested with PromptQL:

  • claude-3-5-sonnet-latest
  • claude-3-7-sonnet-latest

AWS Bedrock​

To use a Bedrock-wrapped model, set the provider to bedorck. The following have been tested with PromptQL:

  • Claude 3.5 Sonnet
  • Claude 3.7 Sonnet

NB: For Bedrock models, you'll need to provide a model_id that resembles this string:

arn:aws:bedrock:<AWS region>:<AWS account ID>:inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0

Google Gemini​

To use a Google Gemini model, set the provider to gemini. The following have been tested with PromptQL:

  • gemini-1.5-flash
  • gemini-2.0-flash

Google Vertex AI​

To use Google Vertex AI, set the provider to vertex. You must provide a publisher-prefixed modelId:

  • google/<gemini-model> (e.g., google/gemini-2.0-flash-001, google/gemini-2.5-flash)
  • anthropic/<claude-model> (e.g., anthropic/claude-3-7-sonnet)

Notes:

Example configuration:

Example Vertex (Google)
kind: PromptQlConfig
version: v2
definition:
  llm:
    provider: vertex
    modelId: google/gemini-2.5-flash
    googleProject: my-gcp-project # required
    googleLocation: us-central1 # required region
    googleServiceAccountKeyJson:
      valueFromEnv: GOOGLE_SERVICE_ACCOUNT_KEY_JSON
Example Vertex (Anthropic)
kind: PromptQlConfig
version: v2
definition:
  llm:
    provider: vertex
    modelId: anthropic/claude-3-7-sonnet
    googleProject: my-gcp-project
    googleLocation: us-east5
    googleServiceAccountKeyJson:
      valueFromEnv: GOOGLE_SERVICE_ACCOUNT_KEY_JSON
Service account credentials
  • The googleServiceAccountKeyJson must be a JSON string for the Google service account key, not a file path.
  • Store the entire JSON in an environment variable and reference it via valueFromEnv.
  • Remember to map the env var in your globals/subgraph.yaml envMapping.

Required permissions (IAM roles)​

To call Vertex AI from PromptQL, the Google service account must have sufficient permissions.

Recommended:

  • Production (least privilege): create a custom role with only aiplatform.endpoints.predict permissions
  • Development/trials: grant the broader roles/aiplatform.user for simplicity

Minimum permissions (only if you invoke Endpoints):

  • Endpoints (custom/managed online prediction):
    • aiplatform.endpoints.predict

Create a custom aiplatform.endpoints.predict-only role and bind it to your service account:

# Create a custom role (add/remove permissions as needed)
gcloud iam roles create vertexPredictor \
  --project=YOUR_PROJECT \
  --title="Vertex Predictor" \
  --permissions="aiplatform.endpoints.predict" \
  --stage=GA

# Grant the custom role to your service account
gcloud projects add-iam-policy-binding YOUR_PROJECT \
  --member="serviceAccount:SA_NAME@YOUR_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_PROJECT/roles/vertexPredictor"

Alternatively, grant the broader built-in role (not least-privilege):

gcloud projects add-iam-policy-binding YOUR_PROJECT \
  --member="serviceAccount:SA_NAME@YOUR_PROJECT.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

Note:

  • Grant in the project from which you invoke Vertex AI (the project associated with your credentials).
  • Ensure your googleLocation is a supported region; see links above for region availability.

Hasura​

With Hasura—used as the default provider—there is no specific model necessary in your configuration.

HIPAA Compliance

If you plan to interact with PHI using PromptQL, you should not use the Hasura provider as it is not HIPAA compliant. You will have to bring your own LLM with another supported provider.

Microsoft Azure​

To use an Azure foundational model, set the provider to azure.

OpenAI​

To use an OpenAI model, set the provider to openai. The following have been tested with PromptQL:

  • o1
  • o3-mini
  • gpt-4o
Base Model

When using the hasura provider, the default model is claude-3-5-sonnet-latest. This is the recommended model for PromptQL program generation.

Troubleshooting If you see “model not supported in streamGenerateContent”, ensure:
  • modelId uses the publisher prefix (google/... or anthropic/...)
  • googleLocation is valid for the chosen model (see location references above) :::

Considerations​

  • The model key is not supported when using the hasura provider.
  • The value for a model key is always in the dialect of the provider's API.
  • If ai_primitives_llm is not defined, it defaults to the provider specified in the llm configuration.
  • system_instructions are optional but recommended to customize the behavior of your LLM.
Set your API key as an environment variable

If you're utilizing a provider aside from Hasura, you'll need to add your environment variable to the globals subgraph so the container running PromptQL has access to it.

As an example, in globals/subgraph.yaml:
kind: Subgraph
version: v2
definition:
  name: globals
  generator:
    rootPath: .
  includePaths:
    - metadata
  envMapping:
    ANTHROPIC_API_KEY:
      fromEnv: ANTHROPIC_API_KEY

Be sure to also include the key-value pair for your API key (<PROVIDER>_API_KEY=your-key) in your project's .env files.