Context

I work on an automated triage and repair platform for cloud AI compute: standard compute hosts and GPU systems such as NVIDIA GB200. When a host fails during first-time provisioning, or while being recycled after a customer releases it, the platform opens a case, with exactly one active case per host.

A workflow engine moves each case through a state machine. It runs a triage sequence specific to the host's architecture, a rules engine picks the single highest-priority fault (fixing one often clears the others), and a hardware or software repair recipe sends the host back to service.

The platform has three parts. The service API is the front door for cases, logs, and sequences. The worker runs the state machine. The rules (triage sequences, repair recipes, policies, ticket templates) live in an object storage bucket and are served to the worker through the service. Each availability domain runs independently with its own rules bucket, and releases go out in order: infrastructure, then rules, then service and worker together.

The problem

Rules shipped separately from code, and a rules deployment simply overwrote the files the service was reading. That produced two failure modes.

Non-atomic deployments. Old files could be deleted before new ones finished uploading. A failed upload took about 10 minutes to reapply, and any case that read rules in that window could find them missing.

Version drift. New rules often depend on new worker code: a new action, policy, or parser. Because rules deploy first, the service could hand the worker instructions it didn't understand yet. The gap was usually 15 to 20 minutes. The worst we saw was six hours.

Either way, cases broke between triage or repair stages and were sidelined for manual investigation. Getting one back on track took at least six hours, often with expensive GPU capacity sitting idle.

The approach I rejected

My first idea was major/minor versioning across both repositories: a major bump means "needs a new worker," a minor bump means "config-only."

The design

Two ideas replace the old model: capabilities replace version numbers, and blue/green folders with a runtime-owned pointer replace in-place overwrites.

Capabilities instead of versions

Every policy, enum, parser, or worker action is a capability. Each rules bundle ships a manifest of its required capabilities, and each service build ships a file of its supported capabilities. Compatibility becomes a set check.

Gate 1: build time

The rules repository is pinned as a submodule in the deployment repo. When an engineer moves that pointer, a pre-commit hook generates one manifest containing both what the rules require and what the targeted service version supports. CI fails the build on any mismatch, catching the most common mistake: merging config that depends on new code without pointing it at the new service version.

Trade-off: regenerating the manifest inside CI would be stricter, but it was much more complex. I chose a pre-commit hook with a hard gate. A missing or stale manifest blocks the release, and the README gives a one-command fix.

Gate 2: runtime, with blue/green

The build gate proves compatibility with a service version, not necessarily the one running in a given availability domain at a given moment. So the final decision happens at runtime.

How a release flows

1. Rules deploy to whichever folder is inactive. The live folder is never touched.
2. Only after every file lands does the pipeline set the request to pending activation, a dependency enforced in Terraform. A deployment that dies halfway never announces itself.
3. The service checks for a pending activation every 10 minutes, on startup, and on the first case after a deploy. It compares the pending folder's required capabilities with its own supported set.
4. If compatible, it flips the pointer, then resets the request to no-op. If not, it keeps serving the current rules safely.
5. When the matching service version rolls out, its first check succeeds and it flips.

Config-only releases, such as reusing an existing GB200 capability for GB300, activate within minutes. Releases that depend on new code wait automatically for that code. Nobody has to decide which case applies.

Key decisions

• The pipeline never touches the pointer after bootstrap. Rules deployments run plan → human approval → apply, and apply can fail partway. If the pipeline owned the one file the service can't live without, a failure could delete it. Making the service the single writer removes that risk.
• Commit markers on both sides. The rules request and app-info files are written last, so nothing ever acts on a half-finished deployment.
• Flip first, then clear the request. If the service crashes between the two writes, the retry repeats an idempotent flip instead of losing the activation.
• Bootstrap migration. On the first run the pipeline detects the legacy layout, seeds both folders with the current rules, and points to blue, giving release and test engineers a predictable starting state.
• Legacy fallback. The original root folder stays in place, and the service falls back to it if anything in the blue/green path looks wrong.

Edge cases

• Concurrent writers. Many service instances may check at once. Writes to the pointer and request use ETag-based conditional writes and checksums, so exactly one instance wins and the rest re-read.
• Rules that never become compatible. If a bad rule set stays pending because no released service supports it, a newer rules deployment may overwrite the inactive folder once the pending request is more than six hours old. The live folder is never touched.
• Service-only deployments. A mitigation release with unchanged rules is safe by construction: no pending request means no flip.

Simplifying my own design

An earlier version had the pipeline choose between "staged" and "ready" states, so the service had to publish its runtime capabilities back to storage for the pipeline to read. Once the runtime gate became the single source of truth, that step stopped carrying weight: the service that has the capabilities makes the decision itself. I'm removing it, which leaves fewer states, fewer writers, and less to go wrong.

Rollout and impact

I split activation from consumption. The control plane (bootstrap, flips, pointer and request handling) is live in all ~100 availability domains. Reading rules from blue/green is behind a feature flag in a single low-traffic region with two availability domains; everywhere else, and as an automatic fallback, the service still reads the legacy folder. This proved out the state machine fleet-wide while limiting risk to production repairs. Fleet-wide rollout of rule consumption follows the same flag.

• Exposure removed. Batched rollouts leave a 15–20 minute rules-to-service gap in each availability domain. That's up to ~100 cases per domain, or up to ~10,000 cases per release across the fleet, that could break and be sidelined.
• Recovery time avoided. Each sidelined case takes 6+ hours of manual work to recover.
• No new process. No version numbers to manage. The build gate and runtime check enforce compatibility automatically.

What I'd do next

• Move manifest generation fully into CI once the build tooling allows it, closing the skipped-hook gap.
• Alert when a request stays pending past a threshold, so incompatible rules surface well before the six-hour override.
• Show the active rule set and any capability gaps per availability domain on an operational dashboard.

Context

In real-time account-to-account payments, money moves between banks in seconds, often through a central clearing house. Our service sat beside that flow. For each payment, a client (the sending bank, the receiving bank, or the clearing house) sent us the transaction: amount, currency, sender and receiver accounts. We returned a risk score from 1 to 99, produced by a downstream ML model from the transaction and historical data. The client decided whether to approve or decline; Visa never made that call.

Our first production client was a national clearing house carrying most of its country's real-time payment traffic. Every millisecond we added was a millisecond added to someone's payment.

Architecture

The service ran active-active across two data centers behind a geo-proximity load balancer, so either site could take all traffic instantly if the other went down. Each VM ran two gateway and two orchestrator processes, with a local Envoy proxy spreading requests across them.

Two small Redis-backed checks sat on every request:

• Entitlement: is this branch code valid for this bank?
• Allowlist: is this a known high-value account? Those accounts are still scored, but flagged so their unusually large transactions don't skew the model.

Problem 1: the orchestrator kept dying

Under production-like load, the gateway started returning bursts of 502s. The cause was the orchestrator process dying outright.

The DAG engine ran independent steps in parallel goroutines, passing attribute maps from step to step. Maps that steps were meant to write to already used sync.Map. But some maps were meant to be read-only, and over time a few code paths had started writing into them. When two parallel steps hit one of those maps at once, Go's runtime raised a fatal concurrent map writes error. It can't be recovered, so it killed the whole process and took half a node's capacity with it.

Problem 2: Redis lookups blowing the latency budget

The end-to-end target was 20 ms at p99.95, with two Redis lookups on every request. At the worst point, individual workflow nodes needed timeouts as high as 300 ms just to keep requests from failing.

The data wasn't the problem: each check read from only 10,000–20,000 records. We profiled garbage collection and the Redis client and tried the usual levers (bigger and smaller pools, dial and read timeouts). None of it moved the needle.

Working with the Redis platform team, we confirmed that server-side lookup time was tiny; the time was going into the network path. That led me to the root cause: the client's maximum connection age was effectively zero. Pool tuning couldn't help, because the pool never actually reused anything.

Per-node timeouts came down from 300 ms to 20 ms, a budget that now only exists to absorb rare spikes.

Results

• 99.999% availability for real-time fraud scoring.
• p99.95 latency at or under 20 ms end to end, at up to 1,000 TPS of live payment traffic.
• Per-node timeouts tightened from 300 ms to 20 ms, with typical node latency of 2–3 ms.
• Monitoring and alerting expanded with Prometheus metrics and Grafana dashboards.

The next chapter: confidential compute

The next market was the UK, the largest account-to-account payments market. Data-residency rules required financial data to stay in-country, and Visa had a single data center in Europe, which ruled out the active-active model there. The answer was AWS.

Visa's cybersecurity standards required more than a private VPC: sensitive payment data could never sit unencrypted outside a hardware-isolated environment. Encryption at rest and TLS in transit weren't enough; the data also had to be protected while in use. So we re-architected the service to run inside AWS Nitro Enclaves: isolated slices of an EC2 instance's CPU and memory with no network, no persistent storage, and no root access, unreachable even by the parent instance.

We were the first team at Visa to take a service to the cloud, and the first to work out what confidential compute meant in practice. There was no internal playbook, and the learning curve was steep.

Build it ourselves or wait for a wrapper

One option was a third-party product: we would hand over Docker images of the gateway and orchestrator, and the vendor's wrapper would handle running them inside enclaves. Given our timeline, we chose to build it ourselves instead: our own listeners, proxies, and startup wiring. It cost us the learning curve up front, but it meant we understood every hop a request took.

Getting traffic in

An enclave can't open a TCP port, so the gateway's HTTPS listener was replaced with a vsock listener, wrapped in TLS so that client TLS terminates inside the enclave and plaintext never exists on the parent. On the parent instance, a socat service run under systemd (restart on failure) listens on the TCP port and forwards each connection to the enclave's context ID and port.

Getting traffic out

Every dependency is reached the same way in reverse. The services connect to localhost ports inside the enclave; a vsock proxy pair, one inside and one on the parent, carries each connection out. That meant one proxy per Redis endpoint and one per Kafka broker, each a systemd unit. Application code barely changed; the plumbing moved into infrastructure.

The hard part: wiring it all together

No single proxy was difficult. Getting all of them to work together was. Every external connection needed a matching pair: a local port inside the enclave, a vsock port and context ID, and a proxy on the parent pointing at the real endpoint. All of it had to be running, in the right order, before the service could serve traffic, and at first much of it was wired by hand.

• One proxy per Kafka broker, not one per cluster. We started with a single proxy for Kafka. A Kafka client uses its first connection only to fetch the cluster layout, then connects to each broker directly at its advertised address. One proxy got that first call through, and everything after it failed.
• Mismatches all looked the same. A wrong local port or context ID on either side didn't produce a useful error. Traffic quietly went nowhere. Debugging meant tracing a request hop by hop across the boundary: enclave socket, vsock, parent proxy, real endpoint.
• Startup order mattered. Proxies on both sides had to be up before the services inside the enclave tried to connect, so we brought them up as systemd services ahead of the enclave workload.

By the end we had put the whole setup in one place: the port and context ID map, the proxy units, and the order to start them, so the next team wouldn't have to rediscover it.

Build and run

Each service was packaged as a Docker image, converted with nitro-cli build-enclave into a signed enclave image file (EIF), and launched with a fixed vCPU count, memory allocation, and context ID that the parent's proxies address it by.

Trade-offs we planned around

• Latency. Every network call now crosses an extra proxy hop, on a service with a 20 ms budget. The connection-reuse lesson from the Redis fix mattered even more here: a warm connection pays the vsock cost once, not per request.
• Capacity. Enclave CPU and memory are carved out of the parent instance, so instance sizing has to cover both. Not every instance type supports enclaves.
• Topology. Gateways and orchestrators moved onto separate instances, each behind a network load balancer, with ElastiCache for Redis and Kinesis Firehose carrying audit data into the analytics platform.

I worked on this design and implementation but moved on before it reached production, so its production results aren't mine to claim.

What I took from it

• Measure where the time goes before tuning. Pool and timeout tweaks did nothing until server-side timings showed the lookups were fast and the network path wasn't.
• A contract in a comment isn't a contract. "This map is read-only" has to be enforced in code, especially once work runs in parallel.
• Being first means writing the playbook. With no internal precedent, the most useful thing we produced besides the service was a clear map of how it all connects.
• Security requirements reshape architecture. Confidential compute moved TLS termination and encryption inside the enclave and turned every network call into a proxied socket.

Context

The platform is OCI's tier-0 ticketing system, think Jira for the cloud: internal incident tickets, customer support tickets, and large-scale events. It runs active-passive across regions and serves an entire realm from one place, so it has to stay up when almost everything else is down. That's why it runs its own gateway instead of the shared API gateway product: if that product fails, tickets about the failure still need to be filed.

Requests pass through the gateway (rate limiting, allowlisting, routing) to the core API, which owns ticket logic and validation. Attachments go to object storage, with their metadata in an attachments table.

The ask was to raise the 10 MB attachment limit, initially to 2 GB. We started with the customer support domain. Internal teams can be asked to zip or split files over chat; external customers can't.

How uploads worked before

An upload was a single request: the whole file in one body, through the gateway to the core API. The core API inspected it with Apache Tika, checked the size and the extension against an allow list, wrote it to object storage with one PutObject call, and stored the object's location in the attachments table.

That's fine at 10 MB. At hundreds of megabytes, a single request body through two Java services means large memory footprints, long-held connections, timeouts, and retries that start over from byte zero.

Options I considered

The design

One endpoint, three stages, mirroring object storage's create, upload-part, and commit calls.

• Declare up front. The client starts by stating the file's name, size, extension, and checksum. We validate the claim, store it in the attachments table, create the multipart upload, and return our attachment ID with the storage upload ID.
• Small requests only. Each part is an ordinary request, no bigger than the old upload limit, so our services never hold more than one chunk at a time. We check that the attachment and upload IDs belong together, forward the part, and hand the ETag back to the client.
• Safe retries. If a part times out after storage already accepted it, the client simply resends it. Only the part numbers and ETags sent at commit go into the final object, so duplicate parts never corrupt the file.
• Verify before trust. After commit, the core API reads the merged object's metadata and compares size, checksum, and extension with the original claim. Any mismatch fails the commit with a 4xx.
• No orphans. A lifecycle policy deletes abandoned or failed multipart uploads after 24 hours.
• Hard limit and soft limit. The system was built and tested to a 1 GB hard limit. Customer support runs with a flag-controlled soft limit of 200 MB, which covers 99.9% of requests; going higher is a configuration change, not a code change.

Keeping it simple for callers

Three calls is more work for clients, so I hid it. Upstream of the gateway, a proxy service routes each ticketing call to the platform or to Jira, depending on the project and realm. I built a client library, shipped as a jar that proxy depends on, so callers still make one upload call; the library streams the file, splits it into chunks, and runs the three stages. The ticketing UI got the same logic: files up to 10 MB use the original single-request API, larger ones use the chunked flow.

The 502 mystery

To speed uploads up, the client library sent chunks in parallel with a thread pool. There were several API instances to spread them across, so parallelism looked free.

Then uploads started failing with 502 Bad Gateway, and our gateway logs showed nothing: the failing requests never reached it. That pointed upstream. The load balancer's logs showed the requests arriving, alongside spikes in bandwidth usage, and a difference in request size between chunks that got through and chunks that didn't.

Raising the load balancer's bandwidth helped, but didn't fix it. So I tested the three variables together: number of parallel chunks, chunk size, and bandwidth.

The extra bandwidth made more parallelism possible; the smaller chunks made it work. Upload issues were concentrated in the first month after launch and settled once this landed.

Results

• Upload capacity raised from 10 MB to 1 GB, tested end to end.
• 200 MB live in production for the customer support domain, across every region and realm the platform serves.
• No unverified files. Every byte passes through our services, and nothing is visible until it matches its declaration.
• Stable after the first month, now in maintenance mode, with expansion to the other domains under way.

What I'd do next

• Check file type from the content itself (its leading bytes) rather than the declared extension, so a renamed file can't slip through.
• Have the server tell clients the chunk size and parallelism to use, so limits can change without shipping new clients.