ChatVertexAI

This will help you getting started with ChatVertexAI chat models. For detailed documentation of all ChatVertexAI features and configurations head to the API reference.

Overview

Integration details

LangChain.js supports Google Vertex AI chat models as an integration. It supports two different methods of authentication based on whether you’re running in a Node environment or a web environment.

Class	Package	Local	Serializable	PY support	Package downloads	Package latest
ChatVertexAI	@langchain/google-vertexai	❌	✅	✅

Model features

Tool calling	Structured output	JSON mode	Image input	Audio input	Video input	Token-level streaming	Token usage	Logprobs
✅	✅	❌	✅	✅	✅	✅	✅	❌

Setup

To access ChatVertexAI models you’ll need to setup Google VertexAI in your Google Cloud Platform (GCP) account, save the credentials file, and install the @langchain/google-vertexai integration package.

Credentials

Head to GCP and generate a credentials file. Once you’ve done this set the GOOGLE_APPLICATION_CREDENTIALS environment variable:

export GOOGLE_APPLICATION_CREDENTIALS="path/to/your/credentials.json"

If running in a web environment, you should set the GOOGLE_VERTEX_AI_WEB_CREDENTIALS environment variable as a JSON stringified object, and install the @langchain/google-vertexai-web package:

GOOGLE_VERTEX_AI_WEB_CREDENTIALS={"type":"service_account","project_id":"YOUR_PROJECT-12345",...}

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

Installation

The LangChain ChatVertexAI integration lives in the @langchain/google-vertexai package:

tip

See this section for general instructions on installing integration packages.

npm

npm i @langchain/google-vertexai

yarn

yarn add @langchain/google-vertexai

pnpm

pnpm add @langchain/google-vertexai

Or if using in a web environment:

npm

npm i @langchain/google-vertexai-web

yarn

yarn add @langchain/google-vertexai-web

pnpm

pnpm add @langchain/google-vertexai-web

Instantiation

Now we can instantiate our model object and generate chat completions:

import { ChatVertexAI } from "@langchain/google-vertexai";
// Uncomment the following line if you're running in a web environment:
// import { ChatVertexAI } from "@langchain/google-vertexai-web"

const llm = new ChatVertexAI({
  model: "gemini-1.5-pro",
  temperature: 0,
  maxRetries: 2,
  authOptions: {
    // ... auth options
  },
  // other params...
});

Invocation

const aiMsg = await llm.invoke([
  [
    "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
  ],
  ["human", "I love programming."],
]);
aiMsg;

AIMessageChunk {
  "content": "J'adore programmer. \n",
  "additional_kwargs": {},
  "response_metadata": {},
  "tool_calls": [],
  "tool_call_chunks": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 20,
    "output_tokens": 7,
    "total_tokens": 27
  }
}

console.log(aiMsg.content);

J'adore programmer.

Chaining

We can chain our model with a prompt template like so:

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You are a helpful assistant that translates {input_language} to {output_language}.",
  ],
  ["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
  input_language: "English",
  output_language: "German",
  input: "I love programming.",
});

AIMessageChunk {
  "content": "Ich liebe das Programmieren. \n",
  "additional_kwargs": {},
  "response_metadata": {},
  "tool_calls": [],
  "tool_call_chunks": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 15,
    "output_tokens": 9,
    "total_tokens": 24
  }
}

Multimodal

The Gemini API can process multimodal inputs. The example below demonstrates how to do this:

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatVertexAI } from "@langchain/google-vertexai";
import fs from "node:fs";

const llmForMultiModal = new ChatVertexAI({
  model: "gemini-pro-vision",
  temperature: 0.7,
});

const image = fs
  .readFileSync("../../../../../examples/hotdog.jpg")
  .toString("base64");
const promptForMultiModal = ChatPromptTemplate.fromMessages([
  [
    "human",
    [
      {
        type: "text",
        text: "Describe the following image.",
      },
      {
        type: "image_url",
        image_url: "data:image/png;base64,{image_base64}",
      },
    ],
  ],
]);

const multiModalRes = await promptForMultiModal.pipe(llmForMultiModal).invoke({
  image_base64: image,
});

console.log(multiModalRes.content);

 The image shows a hot dog in a bun. The hot dog is grilled and has a red color. The bun is white and soft.

Tool calling

ChatVertexAI also supports calling the model with a tool:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { zodToGeminiParameters } from "@langchain/google-vertexai/utils";
import { z } from "zod";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const calculatorSchema = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const geminiCalculatorTool = {
  functionDeclarations: [
    {
      name: "calculator",
      description: "A simple calculator tool",
      parameters: zodToGeminiParameters(calculatorSchema),
    },
  ],
};

const llmWithTool = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
}).bindTools([geminiCalculatorTool]);

const toolRes = await llmWithTool.invoke("What is 1628253239 times 81623836?");
console.dir(toolRes.tool_calls, { depth: null });

[
  {
    name: 'calculator',
    args: { number2: 81623836, operation: 'multiply', number1: 1628253239 },
    id: 'a219d75748f445ab8c7ca8b516898e18',
    type: 'tool_call'
  }
]

withStructuredOutput

Alternatively, you can also use the withStructuredOutput method:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { z } from "zod";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const calculatorSchemaForWSO = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const llmWithStructuredOutput = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
}).withStructuredOutput(calculatorSchemaForWSO, {
  name: "calculator",
});

const wsoRes = await llmWithStructuredOutput.invoke(
  "What is 1628253239 times 81623836?"
);
console.log(wsoRes);

{ operation: 'multiply', number1: 1628253239, number2: 81623836 }

VertexAI tools agent

The Gemini family of models not only support tool calling, but can also be used in the Tool Calling agent. Here’s an example:

import { z } from "zod";

import { tool } from "@langchain/core/tools";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatVertexAI } from "@langchain/google-vertexai";
// Uncomment this if you're running inside a web/edge environment.
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const llmAgent = new ChatVertexAI({
  temperature: 0,
  model: "gemini-1.5-pro",
});

// Prompt template must have "input" and "agent_scratchpad input variables"
const agentPrompt = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant"],
  ["placeholder", "{chat_history}"],
  ["human", "{input}"],
  ["placeholder", "{agent_scratchpad}"],
]);

// Mocked tool
const currentWeatherTool = tool(async () => "28 °C", {
  name: "get_current_weather",
  description: "Get the current weather in a given location",
  schema: z.object({
    location: z.string().describe("The city and state, e.g. San Francisco, CA"),
  }),
});

const agent = await createToolCallingAgent({
  llm: llmAgent,
  tools: [currentWeatherTool],
  prompt: agentPrompt,
});

const agentExecutor = new AgentExecutor({
  agent,
  tools: [currentWeatherTool],
});

const input = "What's the weather like in Paris?";
const agentRes = await agentExecutor.invoke({ input });

console.log(agentRes.output);

The weather in Paris, France is 28 degrees Celsius.

API reference

For detailed documentation of all ChatVertexAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_google_vertexai.ChatVertexAI.html