Systems and AI workflow research
What makes tooling reusable across systems?
Suppose we have a blurred image on the GPU and want to put an overlay on top of it. The overlay comes from a different component, perhaps a script or a CPU image library. When the overlay changes, we'd like to use the new pixels without repeating the blur or downloading the background. That sounds like a reasonable request, but it asks the integration to preserve something neither component can guarantee on its own.
The application has to connect them without taking over their memory management or depending on their private image types. It also needs a useful answer if an operation isn't available, the interfaces disagree, or a provider refuses the work. Those decisions become harder to manage if every consumer has to make them by recognizing which implementation it was given.
The image operation is one instance of a broader tooling problem. We often build a useful capability inside one application, then rebuild much of its integration when we want to use it somewhere else. I'm exploring whether TTX can make the agreement around that capability reusable too. Could the same provider serve a game engine, a build tool and an agent, with each negotiating the interface it needs?
I've been using the Godot lab to work through that question. TTX supplies the contracts and data transport that let independently implemented GDScript, native C++ and CUDA image operations participate in one application. Each provider can specialize behind an agreement its callers understand. The lab gives us a concrete place to examine what that agreement has to preserve and what it costs.
Where a common interface starts to leak
One way to connect these images is to require every provider to supply an ordinary CPU pixel buffer. That makes the compositor easy to write. Unfortunately, an image already on the GPU now has to cross back to the CPU before the caller can use it. If the next operation runs on the GPU too, we may immediately upload those same bytes again. The abstraction has hidden the storage choice by making us pay to undo it.
We could instead teach the consumer to recognize a CUDA image and use a special path. That can work well when we own both sides and expect them to evolve together. Across independently maintained providers, though, the consumer starts collecting knowledge about device contexts, private handles and lifetimes. Adding another implementation means finding every place that made a decision from that identity.
My performance engineering background tends to pull me toward data oriented design. Considering an algorithm and its storage together opens up a lot of design space, and I want a provider to take advantage of that. In this lab the script provider performs spatial convolution, the native CPU provider uses PocketFFT, and the CUDA provider computes on device memory. Requiring their internals to look alike would remove much of the reason to have separate implementations.
The shared agreement therefore describes the operation and the interfaces needed to use it. For example, the convolution contract specifies how the kernel is centered and how pixels outside the image are treated. A caller needs those facts to know what result it is asking for. It doesn't need to know which FFT library produces that result, or whether the provider uses an FFT at all.
Agreeing on what we can call
In TTX, a contract has a UUID that gives the request an identity independent of the provider. The consumer can ask whether an image supports convolution without constructing the callable interface or observing its pixels. That answer is useful for discovery, but there is still an agreement to establish before making a call.
Imagine two independently built modules that both recognize the convolution UUID. One expects a kernel record by value and the other expects a pointer to that record. The operation name agrees, but calling one through the other's function type would be unsafe. This is the sort of assumption that matching headers can hide until a component is built or loaded separately.
Binding checks the interface representation as well as the UUID. Both sides describe their API record, including the calling convention and the argument and result forms of its functions. TTX compiles those descriptions into canonical bytes that can be prepared ahead of binding. The binding implementation compares the prepared representations before supplying the API. A disagreement is returned to the caller before it obtains an interface it could invoke incorrectly.
The resulting API can contain an opaque receiver and function pointers supplied by the provider. The caller invokes those functions with the receiver; it doesn't cast the receiver to a private image class. The image graph retains the bound interface for repeated calls on that receiver and keeps the supplying image and module alive. Copying function pointers alone would not keep their code or their object alive.
Stability here means that a consumer can continue relying on the agreed behavior while a provider changes its implementation. The interface can evolve too, but incompatible changes need an explicit migration or another supported contract. Detecting a disagreement prevents an unsafe call; it doesn't make an incompatible replacement work automatically.
That gives the consumer a usable call boundary. The provider still owes it the behavior named by the contract. Matching representations can't prove that a convolution computes the right pixels, so we need to test that separately.
Following the overlay into the GPU
Once the background supplies a composition interface, the consumer can pass it the overlay as another image object. The CUDA compositor knows its own background buffer. To read the foreign overlay, it asks through the overlay's pixel contract, without identifying its concrete implementation.
That observation establishes a data Flow: the requester describes the pixel form it needs, and both sides agree on a supported way to access it. A provider might lend stable memory or populate storage supplied by the requester. These are separate promises from supplying an image operation. Acquiring a callable doesn't require extracting the pixels it will operate on.
The current CUDA compositor observes the overlay into host storage and uploads it. Its background remains in device memory, and the result is another device image. This path pays for moving the foreign input while preserving the provider's control over the data it already owns.
Upload the changed overlay.
Use the resident buffer.
Replacing the overlay also needs to preserve the work that produced the background. The lab's expression graph tracks changes in its inputs, so it can reevaluate composition without reevaluating an unchanged blur. This belongs to the image graph. The data transport layer doesn't know what a blur is and can't decide which computation should be cached.
The native image test checks both responsibilities. Discovering and acquiring an image operation must leave the transfer counters unchanged. Updating only the CPU overlay must upload it once, preserve the existing blur, and leave the CUDA download and plan-build counts unchanged while producing the composite.
We then request the result's pixels separately and compare them with an independent alpha-composition calculation. Small convolutions are checked against a direct spatial calculation in double precision. Two FFT implementations agreeing with each other would be weaker evidence, since they could share the same mistake. The tests need to distinguish correct output from unnecessary work that happens to produce correct output.
There is still room to improve this path. The compositor currently reads a foreign overlay through host storage even when a more direct exchange might be possible. Avoiding that transfer would require an agreement about the usable memory and its lifetime. Recognizing that both objects happen to come from CUDA wouldn't settle whether they share a device context or whether either one can lend its buffer.
What happens when the provider says no?
If we ask the GDScript provider to produce the blurred background, a valid convolution interface doesn't mean it will accept every workload. Its spatial implementation runs synchronously, so a large image and kernel can monopolize the runtime. The provider owns a work budget and checks each request against it. This check also runs when convolution is called directly, so hiding an expensive option in the demo controls isn't the only thing preventing it from executing.
This separates three questions that are easy to collapse into one: whether the provider recognizes the contract, whether we can establish a callable interface, and whether it accepts this request. A positive answer to the first two doesn't entitle the consumer to bypass the third.
The same distinction matters during transport negotiation. Flow can try another mutually supported protocol when a candidate is unavailable. After a Shared agreement has been selected, however, a failure to acquire its lifetime is returned to the caller. The transport doesn't silently switch to another access mechanism to work around the failed acquisition.
An application could choose a different provider or offer a smaller workload, but that would be an explicit policy decision. Preserving the refusal means it can make that decision with the provider's constraints still intact. The fulfillment checks exercise refusal propagation as well as operation identity: two operations taking the same argument shape must not become interchangeable just because one is easier to invoke.
Taking the same agreements into a browser
Moving the lab into a browser gave me a more demanding version of the reuse question. The application still wanted image operations, but it would be running in WebAssembly with a different set of services available. A useful outcome would be to keep using the providers that could run there and receive an ordinary unavailable result for those that couldn't. The native CUDA service wasn't going to appear just because its interface had a portable description.
Some changes belonged in TTX. Our initial data format assumed eight-byte pointers; this target needed four-byte pointers and its own calling convention. Those facts had to participate in representation agreement. We could then reject incompatible interfaces before calling them, using the same mechanism as the native providers. Other changes needed a closer look at where the implementation was putting them.
Perimortem's native File implementation couldn't provide its filesystem guarantees on this target. The agent's first response was to exclude it from the build. That removed the immediate compilation obstacle, but left future callers with a missing implementation where they should have received a failure result. Keeping the API and reporting failure through its existing optional or boolean results let the caller decide what to try next. Godot could still provide resources through its own facilities. It took review to get that distinction back into the implementation.
We also found a less obvious leak in image acquisition. The project already had an import service mapping configured names to modules. One consumer used that service, while the image loader assembled a filename from the provider's name. The default names worked, so the existing tests passed. Giving the CPU module an unfamiliar alias made the difference visible: the sampling interface could open it, the image inspector accepted the configuration, and image creation failed.
The correction was to use the existing importer and retain the module it supplied. The new test also removes the configuration and releases the source Resource before reading a derived image. Acquisition and lifetime both have to survive the substitution. Matching pixels under the default configuration had told us very little about either assumption.
Getting a build through was only part of the experiment. I wanted to see whether the new environment could be supported without leaving each consumer with more private conventions to understand. The contracts gave us specific promises to review and test, but we still had to catch the implementations that bypassed them.
What did the cleanup actually save?
The pointer extension also acquired an ABI wrapper whose job turned out to be selecting pointer width. Schema and the representation compiler already needed that fact, so we removed the separate owner and passed the width directly. The affected production files lost 80 lines. Calling convention stayed with each callable, where it describes how that function can be invoked.
I'd like that to have been a neat performance win too. The measurements didn't support it. In a focused native benchmark, a Fragment copy of 1,024 scalar values went from about 1.79 microseconds before the cleanup to 1.99 afterward, an 11.5% regression. Binding and wide lookup were nearly unchanged. These were medians from nine runs of each version, with the variants interleaved and outputs checked. They measure the native implementation, not browser latency.
Specializing the whole compiler for each pointer width was another tempting approach. It increased compiler object code from roughly 46 KB to 87 KB without a useful speed gain, so we kept one constexpr implementation. Static descriptions can still be prepared during C++ compilation by passing a constant width through that same code. Having less source to maintain was worthwhile, but it didn't settle the execution cost. The remaining regressions still need investigation.
The engine payload was a different kind of cost. The stock Godot Web engine was 42.04 MiB, mostly executable code, while our extension was a separate 0.97 MiB. A lab-specific build without XR, 3D, physics and unused optional modules reduced the engine to 24.43 MiB. That was a choice about what this host needed to provide, with fallback text rendering for the lab's Latin UI in place of advanced text shaping. These are engine configuration savings, independent of TTX's representation format.
There is a useful distinction between that configuration and excluding File. Godot has an explicit optional class surface, and a host can advertise a smaller one. File already had an API through which a caller could observe failure. We needed to preserve those different promises, rather than make every unsupported operation somebody else's platform check.
The Web port evidence records these comparisons and their limits. The reduced engine passes browser checks for text, controls, textures and scene reconstruction. A GDScript image provider also creates and inverts an image through the TTX bridge. Its result remains readable after the caller releases the original image and provider references. The complete lab still needs browser provider packaging and configuration. Those are separate results, and a working image probe doesn't make the rest of the port complete.
What this changes for people and agents
Before the Web port, I gave a fresh agent a much narrower convolution migration. It had no conversation history, but it did have explicit compatibility requirements and the existing tests. It found the native and script bridge paths and reused the representation check without adding a compatibility API. The patch was a useful example of a change following the intended boundaries.
The broader port had architectural instructions and a long conversation behind it, and still produced the workarounds above. I had to question the build change, the extra ABI owner, and the assumptions about performance. That review is part of the work it took to reach the result. Leaving it out would make the successful parts look considerably easier to obtain than they were.
These weren't comparable tasks, so I can't use them to claim that one context strategy or architecture makes agents more productive. They do expose a familiar maintenance problem. A local fix can leave another loader, state holder or platform convention for future contributors to understand. Splitting that code into more classes doesn't necessarily improve matters, and compressing it into one large function can make the same decisions harder to find.
What helped here was asking who should make each decision, then testing a consumer that didn't happen to share our defaults. The unfamiliar import alias found a gap that the ordinary image tests missed. The Wasm File check created real files through the host runtime first, then verified that the unsupported File operations failed without altering them. Each test made a particular promise observable.
TTX began as a way to make pluggable systems manageable for people. The agent work is helping me examine how much of that model is actually expressed in the code, and how much I still carry in my head. A useful contract needs enough meaning in its interface, comments and tests for the next contributor to change the system without reconstructing our conversation. That remains work we have to do, even when generating the implementation is quick.
Reusing the capability in another tool
The next integrations need to exercise those same questions with different consumers. An asset pipeline could request image operations without starting an editor. An agent could use a bridge through the Model Context Protocol, which provides tool discovery and invocation with schema-described arguments. I'd like to find out how much of that bridge can be shared while native callers continue using the same providers directly.
A callable's binary description won't tell that bridge whether a string names a file or whether an operation overwrites it. Those meanings need to come from the provider, and the host needs a policy for what it exposes. The Source and toolchain work gives us another setting for this: an editor, a build process and an agent could ask the same provider for diagnostics while presenting and acting on the answers differently. Neither direction has been established by the image lab.
For now, the browser work gives us plenty to finish. I want to follow it through deployment and another independently supplied provider, accounting for the adaptation and review along the way. If that next integration requires us to teach another consumer our private naming or storage conventions, we have more work to do on the agreement that was supposed to make the tool reusable.