What we learned from building cloud agents

By “cloud agent,” we mean an agent whose runtime is hosted remotely and can operate independently of any process on your local machine.

These agents take many forms¹: Meta Muse, Claude Tag, OpenAI Codex Cloud, Sierra, ElevenLabs Agents, Higgsfield Supercomputer.

Organizations should be able to build and deploy agents like these for their own use cases, with the capabilities and security guarantees they need.

We found that flexible, composable infrastructure components provide a better foundation for cloud agents than a monolithic solution or SaaS product.

Over the past year, we have built and battle-tested these infrastructure components and open-sourced one of them: the Unreal Agent harness. Today, we want to discuss the rest of the cloud agent stack.

At Unreal Labs, we help companies build, deploy and evaluate cloud agents for their use cases. Get in touch: contact@unreallabs.ai

#Sandbox

Tinkering with agents as a hobby is fun. You purchase a Mac mini, connect your OpenAI subscription to an agent like Hermes, click through a basic setup, and voila, you can interact with the little guy (or gal!) helping you with whatever.

Building agents for real businesses is less fun. Thinking through all the edge cases of what smart AIs can do, cost controls, and evals makes your head spin. Don’t worry, today we’re going through all this step by step.

First off, do you expect more than one user interacting with agents in your system? Great, then you need to isolate the agents from one another. The best way to do that is to use sandboxes².

There are many sandbox solutions available today. We won’t cover them in depth in this post, but here are a few high-level thoughts worth entertaining:

• Are you putting an agent harness inside the sandbox or outside of it?
• If the harness lives outside the sandbox, does your sandbox provider allow you to pause the sandbox and reduce costs? Sandboxes based on Firecracker microVMs usually allow you to do so. Depending on the workload, you can reduce the cost of running agents. For example, pausing the sandbox during inference.

#Storage and Persistence

A sandbox gives an agent a dedicated space to work, but that space may be temporary. The organization’s context, skills, and persistent memory need to remain available whenever the agent starts.

We’ve seen several approaches to file storage for agents. The main ones are persistent volumes and S3 snapshots. One approach that seems overlooked, but that we think suits many internal deployments, is mounting subdirectories over NFS. We treat this storage as a large cache for reads and a staging area for writes.

This makes sandbox preparation fast, while mounts and permissions provide baseline isolation between agents’ workspaces.

For example, conversational agents need access to previous messages. We expose a view of message history as a read-only directory, so agents can consult it without modifying the underlying records. The database remains the source of truth for conversation history.

#Memory

What should an agent remember across runs? For some agents, maintainers choose and manage the persistent documents available to the agent. Others maintain their own memory and need a way to update it from past work.

For persistent agents that manage their own memory, a separate background agent can review logs and existing memories, then update the organization’s shared memory. This process is often called dreaming. We prefer meditation: the name better captures reflecting on past experience and organizing what the agent has learned.

#Agent Egress Proxy

The agent needs to access external services such as Google Drive and internal services such as a CMS. These requests often require credentials (auth), which poses a problem: how do we let the agent use an account without giving it the account’s secrets?

An egress proxy handles this. All traffic leaving the sandbox passes through it, whether the destination is an external API or another service in your system. The proxy can:

• Intercept requests, check their destination, and inject the required credentials.
• Block requests to destinations the agent isn’t allowed to access.
• Inspect and modify request bodies.

It’s best to keep credential management and storage separate from the proxy service, partly for security reasons. The proxy retrieves the credentials for each request from the Credential Broker, which checks whether the requester is permitted to make that type of call. Credentials are added after the request leaves the sandbox, so the agent never handles them.

#Credential Broker and Secret Storage

The Credential Broker manages account connections. It handles connecting accounts, checks who can use each connection, and refreshes auth tokens in the background, where applicable.

Another reason to keep it separate from the egress proxy is that account connections are shared across the stack. The proxy needs credentials to forward agent requests, while other services need them for work such as communicating through Slack. A dedicated broker lets these components share connections and access rules without each implementing its own credential management.

For storing API keys and tokens, we’ve found OpenBao (a Linux Foundation fork of Vault) reliable and easy to deploy and manage. But storing secrets is only part of the job. The Credential Broker also tracks who connected each account, the connection’s status, and where its credentials are stored. We keep those records in Postgres.

#Messaging Gateway

Messaging is one of the places where these shared account connections are used. A connected Slack account lets the system communicate, but we still need to decide which messages require the agent’s attention and where its replies should go.

The Messaging Gateway handles this through a common interface for Slack, iMessage, Telegram, WhatsApp, internal chat, Instant Bloomberg, or any other messaging service. Its responsibilities include:

• Incoming messages. Routes messages that require the agent’s attention, such as a Slack mention, to the Agent Orchestrator.
• Outgoing messages. Sends the agent’s responses to the intended channel or thread, checking that the destination exists.
• Conversation storage. Durably saves and retrieves conversations belonging to each organization or user.
• Messaging permissions. Enforces rules about which channels an agent can send messages to.

#Agent Orchestrator

An incoming message may find an idle agent, an agent already working, or no running sandbox at all. The Agent Orchestrator handles these cases: it prepares the environment, manages execution, and delivers queued messages when the agent is ready.

Its main responsibilities are to:

• Create a sandbox with access to the required files and services
• Prepare an agent, start it and manage its lifecycle
• Save execution traces and stream conversation output

It also manages scheduled runs, tracks execution status, and cleans up idle sandboxes.

Incoming messages go through a queue, allowing new requests to arrive while an agent is working. This matters when someone sends follow-up instructions or several people talk to an agent in a shared channel. The Messaging Gateway decides which messages enter the queue, and the orchestrator delivers them when the agent is ready.

#Custom chat interfaces

A custom chat interface is optional. Some agents interact with users entirely through messaging platforms such as Slack. Others need their own interface in a web or mobile app. For these agents, the orchestrator can store conversation output and stream live updates, letting users reopen conversations, revisit results, and follow messages, tool calls, and reasoning as they appear.

Take web chat as an example. In these deployments, PostgreSQL provides durable storage for conversation history, while Redis Streams serves active conversations and their live updates. This lets the web interface load history and follow ongoing work without bottlenecks on the slow storage.

For building the web interface, assistant-ui has been a good fit. Its external-store runtime connects to the application’s own backend, and its message format has been flexible enough to support different client requirements.

#Harness and SDK

The infrastructure around the agent depends on the harness to execute tools and report what is happening.

A harness runs the agent loop: it calls the model, executes tools, and maintains conversation state.

First-party agent SDKs are available from both providers: Anthropic offers the Claude Agent SDK, while OpenAI offers the OpenAI Agents SDK and the Codex SDK.

In our experience, these SDKs are still evolving. Maintaining integrations requires ongoing work as SDK versions and models change and new features are enabled. We’ve encountered several challenges:

• Model compatibility: The Claude Agent SDK is built for Claude and Anthropic’s API. Running Kimi K3 through Fireworks’ Anthropic-compatible endpoint exposed tool-call ID collisions. Direct tests showed that reused IDs caused Claude Code to drop results from subsequent model requests, despite successful tool execution. API compatibility alone does not guarantee harness compatibility.
• Keeping models and runtimes aligned: Newer models can work poorly with older harness and SDK versions, making upgrades necessary. These upgrades can also require integration changes: background tasks, for example, needed new completion logic to prevent early exits and hangs.
• Tool restrictions: Disabling a tool should also remove instructions to use it. In the Claude Agent SDK, blocking AskUserQuestion removed its definition, but planning tools still told the model to call it. Following those instructions can lead to a rejected tool call, wasting tokens and adding cost because of the harness’s own conflicting instructions.

These SDKs are open source, and we’ve contributed multiple fixes to the Claude Agent SDK.

These experiences, together with our own ideas about how a harness should work, led us to develop Unreal Agent. Read more about Unreal Agent on our blog.

#Building your cloud agent

We’ve covered the infrastructure we use to build cloud agents and some of the problems we’ve encountered running them in production. Different agents need different parts of this stack. We believe having flexible and sometimes reusable components makes it easier to build for those differences.

If you’re building a cloud agent, we’d love to work together! At Unreal Labs, we help companies build, deploy, and evaluate agents for their use cases. Get in touch: contact@unreallabs.ai.

#References