Skip to main content
The vLLM Provider supports querying language models deployed with vLLM for prompt-based interactions.

​
Authentication

This provider requires authentication.
  • api_url: vLLM API endpoint URL (required: True, sensitive: False)
  • api_key: Optional API key if your vLLM deployment requires authentication (required: False, sensitive: True)

​
In workflows

This provider can be used in workflows. As “step” to query data, example: 
steps:
    - name: Query vllm
      provider: vllm
      config: "{{ provider.my_provider_name }}"
      with:
        prompt: {value}  
        temperature: {value}  
        model: {value}  
        max_tokens: {value}  
        structured_output_format: {value}
Check the following workflow example:

​
Connecting with the Provider

To connect to a vLLM deployment:
  1. Deploy your vLLM instance or obtain the API endpoint of an existing deployment
  2. Configure the API URL in your provider configuration
  3. If your deployment requires authentication, configure the API key