TL;DR:
Main Topic: Integrating Generative AI and Large Language Models (LLMs) into Java applications using the Spring AI framework without switching to Python.
Core Architecture & Features:
ChatClient API: Recommended, fluent API for model calls, streaming (Flux<String>), and structured JSON output (mapping to Java Records/POJOs).
Provider Abstraction: Unified interface for OpenAI, Anthropic Claude, Google Gemini, Amazon Bedrock, and local models via Ollama.
Tool Calling (Function Calling): Declarative method execution by AI via @Tool and @ToolParam annotations (automatically controlled by the ToolCallingAdvisor).
Retrieval Augmented Generation (RAG): Native VectorStore integration (e.g., Qdrant, PgVector, Redis) via the Builder pattern and QuestionAnswerAdvisor.
State & Context: Management of the conversation history via MessageChatMemoryAdvisor, requiring a mandatory CONVERSATION_ID.
Observability & Security: Integrated Micrometer/OpenTelemetry metrics (including token consumption); securing MCP endpoints via OAuth2 or API keys (mcp-server-security).
Primary Purpose: Enables Java and Spring Boot applications to directly utilize LLM features, RAG pipelines, and agent workflows within the existing enterprise infrastructure.
Inhaltsverzeichnis
This article was automatically translated. Original article.
AI has made its way into virtually every product team, and with it comes a question that has long preoccupied Java developers: Do you actually have to switch to Python for serious AI features? For some time now, the short answer has been: No. Spring AI provides the longer answer.
With the GA release of Spring AI 2.0 in June 2026, the project has reached a stable foundation—built on Spring Boot 4 and Spring Framework 7. This article explains what Spring AI is all about, what a minimal setup looks like, and what to keep in mind in day-to-day use. It’s intended for developers who want to get a solid first impression before integrating the framework into an existing project.
| Spring AI | 2.0.0 (GA, June 2026), available via Maven Central |
| Platform | Spring Boot 4.0 |
| Java | 17 as a minimum; 21+ recommended |
| Libraries | Jackson 3 instead of Jackson 2; consistently annotated with “null” (JSpecify) |
What is Spring AI—and what problem does it solve?
Spring AI is a framework from the Spring community that integrates AI capabilities into Spring applications without requiring developers to leave their familiar technology stack. It provides abstractions for working with models from various providers—such as those from OpenAI, Anthropic (Claude), or Google (Gemini).
Conceptually, the project borrows heavily from Python libraries like LangChain and LlamaIndex. The difference: Spring AI thinks in Spring terms from the very beginning. Dependency injection, auto-configuration, the usual design patterns—all of that remains intact. For a seasoned Spring team, this significantly lowers the barrier to entry because no second mental model is required.
Spring AI or Python After All?
No one disputes that Python is the top choice in the AI and ML landscape. So why Spring AI? The reason rarely lies in pure AI performance, but rather in integration.
Imagine an application that has grown over the years—Spring Petclinic is the classic example, built on Spring Boot, Thymeleaf, and JPA. Such systems were never designed for AI. The alternative to Spring AI would be to set up a second infrastructure in Python alongside it: a separate service, separate authentication, additional network hops, and another CI/CD pipeline. All of that comes at a cost before the first feature is even up and running.
Spring AI takes the opposite approach. Just as Spring Data provides an abstraction layer over various databases—you s
Key Concepts
There are a few terms you should know before getting started.
Models – the actual AI algorithms. Spring AI supports the major providers: OpenAI, Anthropic Claude, Google Gemini, Amazon Bedrock, Ollama for local models, and others. In version 2.0, the framework for OpenAI, Anthropic, and Google uses the official vendor SDKs directly, making new model features available more quickly.
Tokens & Embeddings – Models don’t process words, but rather tokens, which are text fragments. Embeddings translate text into numerical vectors, allowing semantic proximity to be expressed mathematically. This is the foundation for vector search and RAG.
Prompts – the instructions given to the model. For dynamic prompts, Spring AI relies on templates (via StringTemplate), into which variables are inserted at runtime.
ChatClient – the central, fluent API for communicating with the model. In version 2.0, the ChatClient is explicitly the recommended entry point; the lower-level ChatModel is only needed for special cases.
Advisors – a kind of AOP for LLM calls. They intercept requests and responses: The SimpleLoggerAdvisor logs requests, while the MessageChatMemoryAdvisor automatically appends the conversation history.
Structured Output – By default, LLMs respond in free-form text. Spring AI can configure the model to return machine-readable JSON instead, which is mapped directly to a Java record or POJO.
Tool Calling – The model may call defined
The Minimal Setup
Thanks to auto-configuration, getting started is quick.
1. Dependencies
Since version 2.0, the stable artifacts have been available in Maven Central—you no longer need an additional snapshot or milestone repository. The best way to manage versions is via the BOM (here, Gradle Kotlin DSL):
|
1 2 3 4 |
dependencies { implementation(platform("org.springframework.ai:spring-ai-bom:2.0.0")) implementation("org.springframework.ai:spring-ai-starter-model-openai") } |
For Anthropic or Google, you simply swap out the starter (spring-ai-starter-model-anthropic, spring-ai-starter-model-google-genai)—the rest of the code remains the same. That’s exactly the point.
2. Configuration
The API key is retrieved from an environment variable.
|
1 2 3 4 5 6 7 8 |
spring: ai: openai: api-key: ${OPENAI_API_KEY} chat: options: model: gpt-5-mini temperature: 0.0 # 0.0 = deterministisch, höhere Werte = kreativer |
3. A First Controller
You can use ChatClient.Builder to inject a ready-to-use client:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
@RestController @RequestMapping("/api/ai") public class TranslationController { private final ChatClient chatClient; public TranslationController(ChatClient.Builder builder) { this.chatClient = builder.build(); } @GetMapping("/translate") public Map<String, String> translate(@RequestParam String text) { String response = chatClient.prompt() .user("Übersetze den folgenden Text ins Spanische: " + text) .call() .content(); return Map.of("translation", response); } } |
The call generates the request to the provider in the background and returns plain text. That’s all it takes for the first result.
Responses Directly as Objects
Often, you want structure rather than plain text. The ChatClient can map the response directly to a record using .entity(…):
|
1 2 3 4 5 6 |
record Translation(String original, String translated, String targetLanguage) {} Translation result = chatClient.prompt() .user("Übersetze 'Guten Morgen' ins Spanische.") .call() .entity(Translation.class); |
Spring AI instructs the model to return the appropriate JSON and deserializes it for us.
Streaming
With longer responses, you don’t want to wait until the entire text is available. .stream() returns the response as a Flux, piece by piece:
|
1 2 3 4 5 6 7 |
@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> stream(@RequestParam String message) { return chatClient.prompt() .user(message) .stream() .content(); } |
Strengths and Weaknesses—An Honest Assessment
No tech stack is all sunshine and roses. Spring AI clearly excels in three areas.
The first is model interchangeability. Switching from OpenAI to a model run locally via Ollama is often just a matter of updating dependencies and a few properties; the application code remains unaffected. This makes A/B testing between providers and mixing commercial and local models effortless.
The second is integration with Spring Boot. Vector databases like Qdrant, PgVector, or Redis connect to the application via the usual auto-configuration—saving a lot of setup work.
The third is security. The Model Context Protocol (MCP) allows AI endpoints to be secured, either via OAuth2 (as specified in the standard) or, where no OAuth2 infrastructure exists, via an API key.
However, there are limitations. The project has long moved at a rapid pace. With 2.0 GA, the API surface has stabilized and is consistently zero-annotated, but anyone migrating from the 1.x world should definitely read the upgrade notes—quite a few things have been renamed or removed between versions. Fine-tuning remains challenging: A good overall configuration consisting of the model, vector store, and prompt strategy isn’t achieved by default, but rather through measurement and readjustment. And local open-source models are often still inferior to commercial models in terms of dialogue handling; the framework cannot abstract this away.
In Practice: Proven Patterns
Anyone who dives deeper will encounter the same challenges as everyone else. Here are a few recommendations.
RAG Instead of Fine-Tuning
If the model needs to access internal company knowledge, Retrieval Augmented Generation (RAG) is usually the more cost-effective approach than fine-tuning. Unstructured content—such as tickets and wiki articles—is stored as embeddings in a vector store. At runtime, the pipeline searches for semantically relevant fragments and provides them to the model as context.
The vector store is configured as a bean. Since version 1.0, Spring AI has consistently used builders instead of multi-argument constructors:
|
1 2 3 4 5 6 7 |
@Bean QdrantVectorStore vectorStore(QdrantClient client, EmbeddingModel embeddingModel) { return QdrantVectorStore.builder(client, embeddingModel) .collectionName("incidents") .initializeSchema(true) .build(); } |
You don’t have to build the search manually. QuestionAnswerAdvisor handles retrieval and prompt enrichment:
|
1 2 3 |
ChatClient chatClient = builder .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore).build()) .build(); |
If you need more control, you can access the Store directly—using the Builder here as well:
|
1 2 3 4 5 6 |
SearchRequest request = SearchRequest.builder() .query(text) .topK(5) .build(); List<Document> hits = vectorStore.similaritySearch(request); |
Take System Prompts Seriously
Models are also willing to answer questions that have nothing to do with the actual task. A system prompt sets clear guidelines:
|
1 2 3 4 5 6 7 8 9 |
@Bean ChatClient chatClient(ChatClient.Builder builder) { return builder .defaultSystem(""" Du bist ein Assistent für die Verwaltung einer Tierklinik. Beantworte ausschließlich Fragen, die die Tierklinik betreffen. """) .build(); } |
Move Prompts to Separate Files
These kinds of instructions can quickly become lengthy. Strings concatenated within Java code are difficult to maintain—and product owners or prompt engineers can’t even access them. A better approach is to place the prompt in a file. Markdown works well because it allows for a structure that the models can reliably process. Spring AI loads .st templates by default; a simple Markdown file works just as well.
File located at src/main/resources/prompts/chat-system.md:
|
1 2 3 4 5 6 7 |
# Rolle Du bist ein Assistent für eine Tierklinik. # Regeln - Beantworte nur Fragen zur Tierklinik. - Gib niemals sensible Kundendaten preis. - Wenn du etwas nicht weißt, sage das offen. |
Load via @Value as Classpath-Ressource:
|
1 2 3 4 5 6 7 8 9 10 |
@Configuration public class ChatClientConfig { @Value("classpath:prompts/chat-system.md") private Resource systemPrompt; @Bean ChatClient chatClient(ChatClient.Builder builder) { return builder.defaultSystem(systemPrompt).build(); } } |
Advantage: The Java class remains clean, and the prompt can be customized independently of the code.
Tools Instead of Blind Answers
With Tool Calling, you give the model access to your own systems—it decides for itself if and when to call a method. The streamlined approach uses the @Tool annotation on standard service methods. The description is key: it’s what tells the model when the tool is appropriate.
A common pitfall: The annotation is located in org.springframework.ai.tool.annotation.Tool—not in the chat.model package.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import org.springframework.ai.tool.annotation.Tool; import org.springframework.ai.tool.annotation.ToolParam; import org.springframework.stereotype.Service; @Service public class WeatherService { @Tool(description = "Liefert aktuelle Temperatur und Wetter für eine Stadt.") public String getCurrentWeather( @ToolParam(description = "Name der Stadt") String city) { return new ExternalWeatherApiClient().fetchWeatherForCity(city); } } |
To register the tool, pass the Bean—not the method name as a string. Per request using .tools(…), or for all requests in the builder using .defaultTools(…):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
@RestController public class WeatherChatController { private final ChatClient chatClient; public WeatherChatController(ChatClient.Builder builder, WeatherService weatherService) { this.chatClient = builder .defaultTools(weatherService) .build(); } @GetMapping("/chat") public String chat(@RequestParam String message) { return chatClient.prompt() .user(message) .call() .content(); } } |
Spring AI automatically generates the JSON schema expected by the model from the method signature, calls the method as needed, and returns the result. In version 2.0, the ChatModels no longer execute this loop themselves—instead, the ChatClient automatically registers a ToolCallingAdvisor for this purpose. Anyone working directly with the ChatModel must control the tool execution themselves.
The older approach using java.util.function.Function beans with @Description still exists; however, for new projects, @Tool is the more direct approach.
Managing the Conversation History
LLMs are stateless—without a history, the model cannot understand follow-up questions. The MessageChatMemoryAdvisor automatically appends the latest messages. Spring Boot already configures a ChatMemory bean (a MessageWindowChatMemory with an in-memory repository) for this purpose; you just need to integrate it:
|
1 2 3 4 5 6 7 8 |
@Bean ChatClient chatClient(ChatClient.Builder builder, ChatMemory chatMemory) { return builder .defaultAdvisors( MessageChatMemoryAdvisor.builder(chatMemory).build(), new SimpleLoggerAdvisor()) .build(); } |
If you want to set the window size yourself, you must explicitly create the Memory:
|
1 2 3 |
ChatMemory chatMemory = MessageWindowChatMemory.builder() .maxMessages(20) .build(); |
One detail that is required in current versions: Every call made through the Memory Advisor requires a conversation ID to ensure that conversations remain separate:
|
1 2 3 4 5 |
chatClient.prompt() .user(message) .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId)) .call() .content(); |
The previously common syntax using new InMemoryChatMemory() is no longer used—the constructor and class have been replaced by the Builder API.
Securing Endpoints
In enterprise environments, securing AI integration is not just a “nice-to-have.” When integrating systems via MCP, MCP servers must be protected according to the specification—primarily via OAuth2. Where this is not feasible, the community module mcp-security offers an API key option. It does not originate from Spring Security Core, but from org.springaicommunity:mcp-server-security (for Spring AI 2.x, the 0.1.x version line).
|
1 2 3 4 5 6 7 8 9 |
@Bean SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception { // McpApiKeyConfigurer stammt aus dem Community-Modul mcp-server-security return http .authorizeHttpRequests(auth -> auth.anyRequest().authenticated()) .with(McpApiKeyConfigurer.mcpServerApiKey(), apiKey -> apiKey.apiKeyRepository(apiKeyRepository())) .build(); } |
The apiKeyRepository is required—the configuration will not start without a key source. The server is then called with an X-API-key: id.secret header; the secret portion is bcrypt-hashed on the server side.
What Else 2.0 Brings to the Table
A few points that are interesting beyond day-to-day operations.
MCP has been fully integrated into the core in 2.0. An application can simultaneously act as an MCP client—consuming external tools such as file system or database access—and as an MCP server, offering its own business logic as tools. Streamable HTTP is the new standard for transport; SSE is considered deprecated, while stdio remains for local processes.
Observability is built-in. Spring AI generates micrometer spans and OpenTelemetry-compatible metrics for model and tool calls, including token consumption—which helps with cost control.
The entire API is null-annotated via JSpecify. For Kotlin users, this translates to true nullable and non-nullable types that the compiler checks.
In production, it’s also worth planning for retry and rate-limiting behavior, as well as sensible error handling, from the very beginning: LLM APIs don’t always respond, and they don’t always respond quickly. And when setting up a new project, start.spring.io takes care of selecting the model and vector store starters for you.
Conclusion
Spring AI bridges the gap between the established Java world and the rapid development in generative AI without requiring you to leave the ecosystem. Through abstractions like the ChatClient, straightforward tool calling, and the integration of vector stores, you can create value instead of rebuilding infrastructure.
With version 2.0, the entire framework rests on a stable, consistent foundation. For those familiar with the Spring Boot ecosystem who want to integrate LLMs, this provides a straightforward path. The pragmatic approach: start small with the ChatClient, experiment with structured output, and later expand the architecture toward RAG.
