LLM Agents (Orchestrator)¶
pulsatrix has a native C++ library for LLM agents. It is a client for the Anthropic API (Claude) and for OpenAI-compatible local servers (llama-server, vLLM, Ollama), with streaming and retries, a tool loop, and a tool registry that checks the model's inputs. It needs no Python and no SDK.
Planned
An MCP (Model Context Protocol) server, which will let Claude and other agents call pulsatrix's explainers as tools, serving the same tool registry. See the roadmap.
Building it¶
The orchestrator is a separate library, off by default, so the core library never takes on networking dependencies:
cmake -S . -B build -DPULSATRIX_ENABLE_ORCHESTRATOR=ON
cmake --build build
HTTP comes from cpp-httplib (MIT, one header, fetched at a pinned version). Two settings control TLS:
| Option | Default | What it does |
|---|---|---|
PULSATRIX_ORCHESTRATOR_OPENSSL |
AUTO |
HTTPS through OpenSSL 3: AUTO uses it when CMake finds it, ON requires it, OFF builds plain HTTP only. Certificates are checked against the system's trust store, including Windows' and macOS'. |
PULSATRIX_ORCHESTRATOR_CURL |
OFF |
Also builds a libcurl backend. libcurl brings its own TLS, Schannel on Windows, so it's the way to get HTTPS on Windows without OpenSSL. |
Plain HTTP is enough for a local server such as llama-server or Ollama. The Claude API needs HTTPS.
Without OpenSSL, AUTO builds anyway
If CMake doesn't find OpenSSL, AUTO only prints a status line, and every https://
request (every Claude call) then fails at run time with an HttpError of kind Tls. Pass
-DPULSATRIX_ORCHESTRATOR_OPENSSL=ON to make a missing OpenSSL a configure error instead.
HttpsSupported(HttpBackend::Httplib) tells a program at run time.
To use it from your own CMake project, link pulsatrix_orchestrator in the same build, or
pulsatrix::orchestrator from an installed pulsatrix:
find_package(pulsatrix 1.1 REQUIRED)
if(pulsatrix_HAS_ORCHESTRATOR)
target_link_libraries(my_app PRIVATE pulsatrix::orchestrator)
endif()
The package finds OpenSSL and libcurl for you. Linking the static libraries by hand needs
-lssl -lcrypto too (and -lcurl with the libcurl backend).
Its headers are under pulsatrix/orch/ and its names in pulsatrix::orch, since
pulsatrix::Agent already names the reinforcement-learning policy.
Talking to a model¶
A Provider sends a conversation to a model and streams the answer back. There are two:
MakeAnthropicProvider()for the Claude API;MakeOpenAiCompatibleProvider()for a server with OpenAI's chat-completions API, such as llama-server, vLLM or Ollama running on your own machine.
Both take the same ChatRequest and return the same ChatResponse, so code written for one
works with the other:
#include "pulsatrix/orch/provider.hpp"
using namespace pulsatrix::orch;
std::unique_ptr<Provider> claude = MakeAnthropicProvider(AnthropicOptions::FromEnv()); // ANTHROPIC_API_KEY
ChatRequest request;
request.model = "claude-opus-5-5";
request.system = "Answer in one sentence.";
request.messages.push_back(Message::User("What does layer-wise relevance propagation conserve?"));
ChatResponse answer = claude->chat(request, {[](std::string_view text) { std::cout << text; }});
std::cout << "\n" << answer.usage.output_tokens << " tokens\n";
The callback prints the answer as it arrives. answer.message holds the whole thing.
max_tokens, the longest answer allowed, is 4096 unless you set it.
Local servers¶
For a local model, only the provider and the model name change:
OpenAiCompatibleOptions local;
local.base_url = "http://127.0.0.1:11434/v1"; // Ollama; llama-server's default is http://127.0.0.1:8080/v1
std::unique_ptr<Provider> model = MakeOpenAiCompatibleProvider(local);
request.model = "qwen2.5:7b"; // a model the server has, for example one pulled with `ollama pull`
- A server that wants a token (vLLM started with
--api-key, or a hosted endpoint) gets it fromlocal.api_key = Secret("..."); it's sent as a bearer token. - Tool calls need a model that was trained for them, and a server set up to parse them; check your server's documentation.
- A tool's
strictflag is passed on to the server, which may ignore it. The tool registry below checks inputs either way.
Tools and the tool loop¶
Define each tool once in a ToolRegistry: a name, a description, a
JSON Schema for its input and the function that runs it. The registry
gives the provider the tool definitions and gives RunToolLoop() an executor. The loop asks the
model, runs each tool the model calls, sends the results back, and repeats until the model
answers:
#include "pulsatrix/orch/tool_registry.hpp"
ToolRegistry tools;
tools.add({"add", "Adds two numbers a and b and returns their sum.",
pulsatrix::ParseJson(R"({"type": "object",
"properties": {"a": {"type": "number"}, "b": {"type": "number"}},
"required": ["a", "b"], "additionalProperties": false})"),
[](const pulsatrix::JsonValue& input) -> ToolOutcome {
const double sum = input.find("a")->as_double() + input.find("b")->as_double();
return {pulsatrix::WriteJsonCompact(sum)};
},
/*strict=*/true}); // strict tool use: Claude only writes inputs that match the schema
request.tools = tools.specs();
ToolLoopResult result = RunToolLoop(*claude, request, tools.executor());
std::cout << result.last.message.text() << "\n";
The registry checks every input against its tool's schema before the function sees it, so the
function above can rely on a and b being there and being numbers. (A schema's "integer"
admits 1.0 and numbers too large for int64_t, as JSON Schema says it should, so a function
that calls as_int64() can still throw.) Every mistake goes back to the model as an error
result it can act on, never as an exception in your program:
- an input that doesn't match the schema lists each problem and where it is, as a
JSON Pointer (
/ais membera,/items/0the first element):The input for add doesn't match its schema: - /a must be a number, not a string; - arguments that weren't valid JSON, which small local models sometimes write;
- a tool name that doesn't exist:
There is no tool named "sub". The tools are: add.; - a function that throws:
The tool failed:and the exception's message. A function can also return{"message", /*is_error=*/true}itself.
The messages never quote the model's values back to it.
executor() refers to the registry, so keep the registry alive while the loop runs.
RunToolLoop() takes any function from ToolCall to ToolOutcome, so a registry is optional.
The loop's fourth argument, ToolLoopOptions, has:
max_turns(20): the most requests the loop makes. If the model still wants tools then,result.hit_turn_limitis true and the conversation ends on tool results, soresult.last.message.text()may be empty.stream: callbacks for the streamed text and thinking, as inchat().on_message: called with each message the loop adds, the model's answers and the tool results, for logging or showing progress.
result.messages is the whole conversation, ready to continue with another request, and
result.usage adds up the tokens of every request.
Which schemas work¶
JSON Schema has several versions ("drafts"). Claude, OpenAI-compatible servers and MCP clients don't all assume the same one, so a schema may use only the keywords that mean the same in draft-7 and 2020-12, the two in use:
| For | Keywords |
|---|---|
| any value | type (a name or an array of names), enum, const |
| objects | properties, required, additionalProperties, minProperties, maxProperties |
| arrays | items (one schema for every element), minItems, maxItems, uniqueItems |
| strings | minLength, maxLength (counted in characters, not bytes) |
| numbers | minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf |
| combining | allOf, anyOf, oneOf, not |
| notes, not checked | title, description, default, examples, format, $comment, $schema |
add() refuses any other keyword rather than not enforce it. That includes $ref and
$defs/definitions (spelled differently in the two drafts; write the schema out), items as
an array (2020-12 calls it prefixItems), and pattern (check the string in the tool).
ValidateJson() and CheckSchema() in json_schema.hpp work on their own as well.
Strict tool use. With strict = true, Claude's sampling is constrained so that it can only
write inputs that match the schema. Strict mode supports fewer schemas, and add() refuses a
strict tool whose schema it can't take, with the reason. A strict schema needs
"additionalProperties": false on every object, and can't have:
minimum,maximum,exclusiveMinimum,exclusiveMaximumormultipleOf;minLengthormaxLength;maxItems,uniqueItems, orminItemsabove 1;minPropertiesormaxProperties;oneOfornot;enumvalues that are arrays or objects, ortrue/falsein place of a schema.
StrictModeProblem() says the same for any schema. The API also limits a whole request to 20
strict tools, 24 optional parameters and 16 parameters with a choice of types (anyOf or a
type array) across its strict tools. Those aren't checked here; the API answers 400.
A complete program¶
#include <csignal>
#include <iostream>
#include <memory>
#include <string>
#include "pulsatrix/orch/provider.hpp"
#include "pulsatrix/orch/tool_registry.hpp"
using namespace pulsatrix::orch;
int main(int argc, char** argv) {
#ifndef _WIN32
std::signal(SIGPIPE, SIG_IGN); // see "HTTP and streaming" below
#endif
const bool local = argc > 1 && std::string(argv[1]) == "--local";
try {
std::unique_ptr<Provider> model;
ChatRequest request;
if (local) {
OpenAiCompatibleOptions options;
options.base_url = "http://127.0.0.1:11434/v1"; // Ollama
model = MakeOpenAiCompatibleProvider(options);
request.model = "qwen2.5:7b";
} else {
model = MakeAnthropicProvider(AnthropicOptions::FromEnv());
request.model = "claude-opus-5-5";
}
ToolRegistry tools;
tools.add({"add", "Adds two numbers a and b and returns their sum.",
pulsatrix::ParseJson(R"({"type": "object",
"properties": {"a": {"type": "number"}, "b": {"type": "number"}},
"required": ["a", "b"], "additionalProperties": false})"),
[](const pulsatrix::JsonValue& input) -> ToolOutcome {
return {pulsatrix::WriteJsonCompact(input.find("a")->as_double() +
input.find("b")->as_double())};
}});
request.tools = tools.specs();
request.messages.push_back(Message::User("What is 1234.5 + 6789.25? Use the add tool."));
ToolLoopOptions options;
options.stream.on_text = [](std::string_view text) { std::cout << text << std::flush; };
const ToolLoopResult result = RunToolLoop(*model, request, tools.executor(), options);
std::cout << "\n(" << result.turns << " requests, " << result.usage.input_tokens << " tokens in, "
<< result.usage.output_tokens << " out)\n";
return result.hit_turn_limit ? 2 : 0;
} catch (const ProviderError& e) {
std::cerr << "the provider failed: " << e.what() << "\n";
} catch (const HttpError& e) {
std::cerr << "the request couldn't be sent: " << e.what() << "\n";
} catch (const std::exception& e) { // a missing key, or a bad request
std::cerr << e.what() << "\n";
}
return 1;
}
What the providers handle for you¶
-
Retries. These are retried, by default twice:
- overloads (529) and server errors (500, 502, 503, 504);
- request timeouts (408);
- rate limits (429);
- connections that fail, time out or break;
- an overload or server error reported partway through Claude's stream.
A retry waits a random time up to 1 s, then up to 2 s, doubling to at most 30 s, so many clients don't retry in step. When the server sends
retry-after, the retry waits that long instead, but aretry-afterover 60 s fails at once. A 429 withoutretry-afterfrom the Claude API is the monthly spend cap, which retrying can't fix, so it fails at once too. Nothing is retried once part of the answer has reached your callback, since a retry would repeat it.AnthropicOptions::retryandOpenAiCompatibleOptions::retry(aRetryPolicy) change all of this;max_retries = 0turns it off. - Errors.chat()andRunToolLoop()throw: -ProviderErrorfor an error response, or when the retries run out. It carries the HTTPstatus(), the API's errortype()(for exampleoverloaded_error) and therequest_id()to quote to support. A connection that kept failing has status 0 and typetransport. -HttpErrorfor a request that can't be sent at all: no TLS in this build (Tls), a bad URL or header (InvalidRequest), or a response over the size limit (TooLarge). -std::invalid_argumentfor a request without a model or messages, or withmax_tokensbelow 1.Making the provider can fail too:
AnthropicOptions::FromEnv()throwsstd::runtime_errorwhenANTHROPIC_API_KEYisn't set, andMakeAnthropicProvider()throwsstd::invalid_argumentfor an empty key. - Prompt caching (Claude). The API can reuse the start of a prompt it saw recently, which makes it cheaper and faster. The providers mark the tool definitions, the system prompt and the conversation so far for caching, so each turn of a tool loop pays full price only for what's new.answer.usage.cache_read_input_tokensshows it working. Prompts under 512 tokens aren't cached. - Thinking. Current Claude models reason before answering, and send that reasoning as "thinking" blocks. In a tool loop they have to be sent back unchanged, and they are;on_thinkingshows the thinking as it streams. - Tool choice. Current Claude models (Opus 5.5, Sonnet 5.5, Fable 5.1) reject forcing a particular tool with HTTP 400, so the providers never do. Setallow_tool_calls = falseto tell the model not to call any. - Local models. A small model sometimes writes tool arguments that aren't valid JSON. The OpenAI-compatible provider keeps them as a string incall.inputinstead of failing, and the registry reports the mistake back to the model.
Threads¶
A provider handles one request at a time: give each thread its own. A ToolRegistry may run
tools from several threads at once, as long as your functions allow it, but finish adding tools
before any loop starts.
API keys¶
orch::Secret holds a credential. It has no operator<< and doesn't convert to a string, so a
key can't reach a log line or an error message by accident. The one way to read it is reveal():
#include "pulsatrix/orch/secret.hpp"
std::optional<pulsatrix::orch::Secret> key = pulsatrix::orch::Secret::FromEnv("ANTHROPIC_API_KEY");
if (!key) {
std::cerr << "set ANTHROPIC_API_KEY\n";
return 1;
}
std::cout << "using key " << key->redacted() << "\n"; // "using key <secret, 108 characters>"
// headers.emplace_back("x-api-key", key->reveal()); // the only place the value goes
The memory holding the key is overwritten when the Secret is destroyed or assigned over.
HTTP and streaming¶
The providers are built on this layer. Use it directly for an API they don't cover.
MakeHttpTransport() returns a transport: libcurl when the build has it, cpp-httplib otherwise.
MakeHttpTransport(HttpBackend::Httplib) or (HttpBackend::Curl) picks one. send() waits for
the whole response; stream() passes a successful response's body to a callback as it arrives,
which is how LLM APIs stream their answers. SseParser turns those chunks into server-sent
events (SSE), the text/event-stream format the APIs stream in:
#include "pulsatrix/orch/http.hpp"
#include "pulsatrix/orch/sse.hpp"
using namespace pulsatrix::orch;
HttpRequest request;
request.url = "https://api.anthropic.com/v1/messages";
request.add_secret_header("x-api-key", *key); // redacted by request.describe()
request.add_header("anthropic-version", "2023-06-01");
request.add_header("content-type", "application/json");
request.body = R"({"model": "claude-opus-5-5", "max_tokens": 256, "stream": true,
"messages": [{"role": "user", "content": "Hello"}]})";
SseParser parser([](const SseEvent& event) {
if (event.type == "content_block_delta") {
std::cout << event.data << "\n"; // a JSON object; the providers above parse these
} // other types, such as ping, are skipped
});
std::unique_ptr<HttpTransport> http = MakeHttpTransport();
HttpResponse response = http->stream(request, [&](std::string_view chunk) {
parser.feed(chunk);
return true; // false cancels the request
});
parser.finish();
if (!response.ok()) {
std::cerr << "HTTP " << response.status << ": " << response.body << "\n"; // the error, collected
}
What the transport does for you:
- No redirects. A redirect would send your key to a host you didn't choose, so a 3xx is returned as a response instead.
- Limits.
connect_timeout(10 s) andread_timeout, the longest the server may go without sending a byte (10 minutes, since a model can think for a while).max_response_bytes(64 MiB) stops a server that never stops sending. - Certificate checks can't be turned off. To trust a company proxy or your own server,
pass its CA certificate:
MakeHttpTransport({"/path/to/ca.pem"}). It replaces the system's trust store. A provider takes the transport as its second argument:MakeAnthropicProvider(options, MakeHttpTransport({"/path/to/ca.pem"})). - Errors. A response with a 4xx or 5xx status is a response. A request that got no
response throws
HttpError, whosekind()says why:Connect,Tls,Timeout,TooLarge,Protocol,InvalidRequestorOther. A header value with a CR or LF is refused before anything is sent, so a value can't inject headers of its own. - Threads. A transport handles one request at a time.
The parser follows the SSE specification (part of the WHATWG HTML standard): any line ending,
chunks split anywhere, comments, id and retry. It passes every event type on; skip the ones
you don't handle, since APIs add new ones.
On Linux and macOS, a server that closes the connection while OpenSSL is writing can raise
SIGPIPE, which ends the process by default. A program that streams should ignore it:
std::signal(SIGPIPE, SIG_IGN);.
JSON¶
The orchestrator uses pulsatrix's own JSON (pulsatrix/json.hpp), not a separate library. Its
parser is strict and refuses nesting deeper than 256 levels, which is what reading a remote
server's responses calls for. WriteJsonCompact() writes a value on one line with no whitespace,
for request bodies and for newline-delimited protocols such as MCP over stdio:
pulsatrix::JsonValue message{pulsatrix::JsonValue::Object{}};
message.add("role", "user");
message.add("content", "Which input mattered most?\nAnswer briefly.");
std::string body = pulsatrix::WriteJsonCompact(message);
// {"role":"user","content":"Which input mattered most?\nAnswer briefly."}
If your own code uses nlohmann::json, include pulsatrix/viz/mime_bundle.hpp and
nlohmann::json j = value; converts a JsonValue. pulsatrix itself doesn't depend on nlohmann.