Source and dialects

In Tetrodotoxin, Source owns the observation and its provenance. A selected tokenizer interprets bytes, and a Dialect executes a range of the resulting command stream.

Getting from bytes to a cursor

A Source observation exposes its content through a binding surface. The host can select a tokenization policy appropriate to those bytes, rather than require Source itself to know every language it might encounter. The tokenizer owns its output and its lifetime, so a C provider doesn't need to adopt a C++ Arena to take part.

The reference tokens are small copied values containing a code and a provider-local locator. Spelling and source coordinates stay behind the provider. That lets a consumer carry tokens around without retaining a separate view in each one or assuming how the provider stores its source.

A Cursor adds a logical position and access to that token service. Its operation table asks about absolute indices; each Cursor keeps its own progress. Copying one therefore gives us a fork that can look ahead or be handed to a nested operation. The C++ facade can cache the current token because the provider promises stable observations at an index.

Text views borrow from the supplying lifetime. A consumer that needs the bytes to outlive the Cursor must copy them or retain the appropriate owner. Similarly, forking the position doesn't roll back diagnostics or other effects produced by an invoked dialect.

Executing a dialect

A Dialect receives a Cursor at the point where it should start and executes a range of the command stream. Parsing a declaration is one useful application, but the interface also permits work whose purpose is a side effect or a request to another system.

The retained result of one invocation is a Monograph. It is exposed as an Abstract interface, and it defines what another participant can observe from that invocation. Its return contract states how the caller retains access. The provider may keep private storage and retained child results.

An outer invocation can consume its own part of the stream, hand a Cursor to another dialect, and retain the returned Monograph. It can also forward that result unchanged when it already provides the answer required. Neither case requires a Monograph to be a file root or requires the two dialects to share an allocator.

The Source contracts and model implementations define these interfaces and their retention rules. Toolchain composition follows the resulting models through App and a terminal.