Software Architecture in the AI Era: What Actually Changes

Software architecture in the AI era extends far beyond calling a model through an SDK. Teams now add AI gateways, new communication protocols, evaluation suites, prompt-injection defenses and fingerprint-based caching around every model call. This article walks through each of those pieces, why they exist, and what a developer needs to learn that a prompt file alone cannot cover.

What Changes in Software Architecture in the AI Era

Software architecture in the AI era changes less at the model call and more in everything around it: routing, protocols, security, evaluation, caching and observability. An SDK is just another way to make a request. The engineering work sits in how you plug model capabilities into applications without breaking correctness, cost control or safety. This article covers the seven components that matter most, based on a recent Full Cycle discussion of how AI reshapes system design.

The recurring theme is that developers stop programming against model names and start programming against capabilities and intents. Just as few application developers care whether their deploy lands on Kubernetes or a VPS, the argument goes, few will need to know whether a summarization task resolves to one provider model or another. The architecture absorbs that decision.

AI Agents Are More Than a Prompt in a Markdown File

An AI agent is a system that uses a model to pursue a goal, often by calling tools, rather than answering a single prompt. Search the term and you will find agents marketed for marketing, sales, law and development, which makes the concept feel broader than it is. In practice, many simple agents reduce to a Markdown File holding a prompt, but production agents need tool calls, state handling and guardrails around that file.

Coding assistants illustrate the spectrum. Claude Code, Anthropic agentic terminal tool, GitHub Copilot, GitHub's AI pair programmer, Codex from OpenAI, and Gemini from Google all matter, but knowing their keystrokes is the visible tip of the iceberg. The architectural work underneath them is what a serious engineering interview probes. This is also where practices such as Speck Driven Development, Loop Engineer and Harness Engineer fit: they structure how a developer works with these tools, not what runs in production.

New Protocols: MCP and Agent Communication

AI workloads introduced communication protocols alongside HTTP, TCP and gRPC. The most cited is the Model Context Protocol (MCP), an open standard for connecting models to external tools and data sources. The protocol specification moved in 2025 to a streamable, stateless-oriented HTTP transport, replacing the earlier HTTP+SSE approach, which materially changes how servers are deployed and scaled.

A second protocol, the Agent2Agent (A2A) protocol, addresses direct agent-to-agent communication, originally introduced by Google in April 2025 and later placed under the Linux Foundation. Alongside these protocols, a body of AI-specific design patterns is forming so teams stop reinventing solutions the industry already converged on, much as object-oriented design patterns standardized common structures decades ago.

Protocol	What it connects	Notable detail
MCP (Model Context Protocol)	Models to external tools and data	Moved to a streamable, stateless-oriented HTTP transport in 2025
A2A (Agent2Agent)	Agents to other agents	Introduced by Google in April 2025, now under the Linux Foundation
HTTP / gRPC / TCP	Services to services	The pre-AI baseline; AI protocols layer on top of these habits

AI Gateway: Route by Intent, Not by Model Name

An AI gateway is a control layer that every model call passes through, the same role an API gateway plays for service traffic. The developer requests a capability such as 'text summarizer'; the gateway resolves that intent to a configured provider model, with fallback targets, retry counts, timeout policies and per-provider cost, latency and usage tracking already set by the platform team.

This separation matters because the company, not the developer, holds the provider contracts. Picking a model ad hoc wastes money the organization already negotiated. As models become commodities, the developer's job shifts to stating the intent, and the gateway keeps provider details fully transparent. The transcript's worked example: a request for 'text summarizer' maps to GPT 5.4 mini, falls back to 4.1 mini on failure, with configured retry counts and timeouts. The developer never names a model.

In the gateway pattern described in the Full Cycle course material, the component concentrates these responsibilities:

1. Model routing from custom capability names to provider models
2. Timeout and retry policies, with fallback to a cheaper or secondary model
3. Authentication, authorization and rate limiting on every model call
4. Budgets, governance and audit logs showing who used what
5. Caching and response optimization, which the source argues is one of the highest-leverage cost controls

Both open-source and commercial AI gateway solutions exist; evaluating at least one is worth the time before an architecture review forces the issue. Every caller that needs a model in production lands here: web apps, mobile clients, internal services and API integrations.

Security: Prompt Injection and the New Top Ten Lists

AI introduces attack classes that conventional application security did not cover. A chat interface lets any user write arbitrary text at your system, attempting prompt injection or jailbreaks to extract data another employee should not see or to trigger unauthorized actions. The classic comparison is SQL injection: parameter manipulation with a defined boundary. With a model, the input is free-form language and the instructions themselves are attack surface.

The OWASP GenAI security project published its LLM Top 10 list, refreshed in 2025, covering risks such as prompt injection and sensitive data disclosure, and added a separate Top 10 for agentic AI risks. Two principles from the source deserve emphasis: the system prompt is not a secret, and a model call is not a security boundary. Controls must run before and after the call, and audits must bracket both the model call and any tool call. Prompts can reduce risk; they cannot guarantee it.

Evaluation and Model Trade-offs

Evaluation is the automated testing layer that verifies an agent still behaves as intended after every prompt or model change. Change 'be polite' in a prompt and one behavior improves while another quietly breaks. No agent should reach production without this harness, because every prompt edit can invalidate previously working behavior. The source describes two complementary styles:

1. Probabilistic evaluation: define criteria such as rudeness, task completion, repetition, output over 300 characters, or correct handoff to a human when a customer starts swearing, then measure outputs against them statistically.
2. Deterministic evaluation: assert exact call sequences (tool A, then B, then C) or required tool usage regardless of order.

Model choice is a quantified trade-off among latency, quality and cost. The source's worked example, laid out side by side:

Model option	Accuracy	Cost per million tokens	Latency
Baseline	80%	R$ 10	faster
Upgrade	85%	R$ 15	up to 3 seconds slower

The right answer depends on scope. If poor quality is strangling the business, pay more. If errors are tolerable and fallbacks exist, keep the cheaper model. The same arithmetic applies to latency: paying the same for a three-second-longer answer only works when users can wait.

Caching with Fingerprints and AI Observability

Caching looks familiar until AI enters the composition. A normal cache hits on a key and invalidates on a TTL or content change. With AI, the worst outcome is serving a stale answer confidently wrong. The mitigation described is a fingerprint: hash the prompt, the policy rules, the model identifier, the context document and the result together. For example, a request to 'summarize this new law' combines an objective-tone instruction, an access policy, a model version and a stored 2024 law document into one hash. On a repeat request, only an identical fingerprint earns a cached response; any change to prompt, model or context invalidates the whole set, even when the user's question is unchanged. Classifying or re-ranking the prompt does not help: if the model or context changed, the cache is gone.

Observability extends along the same lines. The classic three pillars of logs, metrics and traces, with events sometimes cited as a fourth, all apply, but logs need stricter filtering because model traffic carries sensitive data. Traces gain the most value: they show the step-by-step path through an application, from the incoming request through each model call, which is exactly where AI failures hide.

Two adjacent pressures round out the picture. Messaging pipelines built on Kafka-class tools, RabbitMQ, SQS and SNS assumed millisecond responses; model calls can take 10 seconds, forcing rethinking around queues and streaming. And cloud platforms have layered AI services on top of compute basics: Vertex AI on Google Cloud, plus AI service catalogs on AWS and Azure, alongside vector databases for semantic search. Relearning the provider landscape is part of the job now.

Design Docs Still Matter for AI Context

Documentation has always been hard to keep current, and AI does not fix that. The source stresses design docs as a discipline: RFCs, ADRs, product documents, postmortems, incident responses and feature specs each serve a different part of a system. Feeding the wrong document type to a model as context can mislead it into wrong decisions. Knowing which document belongs where is now an input-quality problem, not just a writing problem.

FAQ

Is learning to code with AI tools really urgent?

The source argues it is: teams that code with AI deliver substantially faster, and a developer who cannot will fall behind on output. Speed alone is not enough, though; delivering well is what the surrounding architecture makes possible.

What is an AI gateway in one sentence?

It is a control layer through which all model calls pass, resolving developer-stated intents to provider models while enforcing routing, fallback, retry, budget, auth and caching policies centrally.

Why can't a system prompt secure an AI application?

Prompts are user-visible instruction text, not a trust boundary, and models can be manipulated through prompt injection. Security controls and audits must run before and after every model and tool call.

When should AI cache entries be invalidated?

Whenever any component of the fingerprint changes: the prompt, the policy, the model, the context document or the rules. A repeated question with a changed context must miss the cache and be regenerated.

What is the difference between probabilistic and deterministic evaluation?

Probabilistic evaluation scores agent behavior against statistical criteria such as tone or task completion; deterministic evaluation asserts exact tool-call sequences or required tool usage. Most production agents need both.

What replaced HTTP and SSE in MCP?

In 2025 the MCP specification moved to a streamable, stateless-oriented HTTP transport. The change affects how MCP servers are deployed and scaled, removing the need to hold long-lived connections per client.

Do developers still need to know which model runs behind a capability?

Usually no. The company negotiates provider contracts and the AI gateway routes intents to models, so naming a specific model in application code often wastes money the organization already secured. Platform and infrastructure teams are the exception.

How does AI change messaging and streaming pipelines?

Queues such as Kafka, RabbitMQ, SQS and SNS were designed around millisecond throughput. Model calls can take several seconds, so consumers, timeouts and backpressure all need redesign around the slower, variable response time.

Which cloud AI services should a developer learn first?

Start with the managed model platform on your current provider, such as Vertex AI on Google Cloud, then the equivalent catalogs on AWS and Azure, plus a vector database for semantic search. Expect to relearn parts of each provider's catalog.

From Watched Video to Written Reference

The argument of this piece is that the durable skills sit around the model call, not inside it, and the same is true of the knowledge in a long technical video like this one. The reasoning, the trade-off examples and the caching walkthrough deserve to outlive the play button as a document your team can search and cite.

If you have explanations, interviews or lessons sitting in your own YouTube videos, Skala Blog turns them into structured articles: paste the URL, get a transcription, and generate a publishable draft that keeps the evidence chain intact.

Source video

Keep Learning

This walkthrough draws on the Full Cycle discussion, and more of this material appears in Gustavo Dev Doido's ecosystem, including Crazystack Typescript and the Bootcamp do Dev Doido. If you want structured depth on the same topics, see https://crazystack.com.br.