Source and dialects

An image provider makes a convenient example because the result is easy to see. The same separation becomes more demanding when the thing being supplied is a language. An editor wants names and provenance, a terminal wants execution facts, and a build system wants to discover the products it can request. None of them should have to become part of the parser to get an answer.

The Source work is separating those responsibilities in Tetrodotoxin. Source owns the observation and its provenance. A selected tokenizer interprets bytes, and a Dialect executes a range of the resulting command stream. The models it exposes can then be used without inheriting its source syntax.

Getting from bytes to a cursor

A Source observation exposes its content through a binding surface. The host can select a tokenization policy appropriate to those bytes, rather than require Source itself to know every language it might encounter. The tokenizer owns its output publication, so a C provider doesn't need to adopt a C++ Arena to take part.

The reference tokens are small copied values containing a code and a provider-local locator. Spelling and source coordinates stay behind the provider. That lets a consumer carry tokens around without retaining a separate view in each one or assuming how the provider stores its source.

A Cursor adds a logical position and access to that token service. Its operation table asks about absolute indices; each Cursor keeps its own progress. Copying one therefore gives us a fork that can look ahead or be handed to a nested operation. The C++ facade can cache the current token because the provider promises stable observations at an index.

Text views borrow from the supplying lifetime. A consumer that needs the bytes to outlive the Cursor must copy them or retain the appropriate owner. Similarly, forking the position doesn't roll back diagnostics or other effects produced by an invoked dialect. Those are different responsibilities from deciding where the next token will be read.

Executing a dialect

A Dialect receives a Cursor at the point where it should start and executes a range of the command stream. Parsing a declaration is one useful application, but the interface also permits work whose purpose is a side effect or a request to another system. Describing every dialect as an AST builder would quietly put the parser back in charge of the whole model.

The retained result of one invocation is a Monograph. It is exposed as an Abstract policy through a Publication, and it defines what another participant can observe from that invocation. Its owner may retain private storage, child publications or both.

This matters for nesting. An outer invocation can consume its own part of the stream, hand a Cursor to another dialect, and retain the returned Monograph. It can also forward that publication unchanged when it already provides the result required. Neither case requires a Monograph to be a file root or requires the two dialects to share an allocator.

Keeping the model after the source has done its work

The Library dialect supplies syntax for constructing Type and Execution facts. Authored Source policies add the location, name and other information that comes from the text. A Shader or App implementation can then ask about the underlying model without pretending that its objects were created by Library's parser.

That is the point of separating the model from its authoring language. App needs to establish that a callable satisfies its entry requirements. It shouldn't need to care whether the callable came from a Library source, another language or a loaded provider. The terminal consumes the agreed facts and emits a result under its own lifetime.

The focused Source and Library tests cover the portable Cursor, a bounded function grammar, nested invocation ownership and emitted code that remains usable after discovery and provider release. Extending the rest of Library and the Build/Package/Workspace path is the next part of making this useful as a whole toolchain.

The Source contracts and model implementations show where that separation is being made.