Explicit Memory and the Geometry of Contextual Selection in a Large Language Machine

Author: Enrico Betti

Keywords: Large Language Machine, explicit memory, attention, contextual selection, representation geometry, inspectable language systems

Previous work: this report follows Large Language Models vs Large Language Machines, DOI 10.5281/zenodo.23145576, also available on civolle.it.

Abstract

A Large Language Machine (LLMA) separates linguistic memory, retained in an explicit and updatable form, from the mechanism responsible for choosing among available continuations. This work studies the latter component: a small contextual selector based on two attention heads.

The experiments initially use sentences generated from a controlled grammar, as in the previous work on LLMAs. In the complete architecture, memory limits the possibilities to observed continuations and the selector ranks them based on the current state. The test bed analysed here instead isolates the second component and makes it predict the continuation over its fixed vocabulary. This separation makes it possible to observe what happens when moving from a condition with only one visible element, in which attention cannot choose between positions, to conditions with two or more elements.

On sentences excluded from training, mean accuracy over five seeds rises from 37.16% with one element to 42.50% with two and 45.11% with the full context, compared with 22.2% for the most frequent word. With three elements, the grammatical role becomes almost entirely recognisable from the two observed coordinates alone, even for words excluded from classifier training. When the full context is used, the heads regularly retrieve distant elements and retain more predictive ability than a fixed window of the same cardinality. Across training runs with different seeds, the absolute geometry rotates or is reflected, but a relative structure remains transferable after supervised alignment using homologous points.

The results do not show that two coordinates constitute universal concepts, nor that a small attention block can replace a general language model. They do show, however, that memory, selection and representation can be isolated, measured and subjected to ablation. The attempt to replace attention with frequency, recency, fixed windows or explicit rules clarifies some of its functions, but has not yet produced an equivalent mechanism capable of the same contextual selection.

1. Introduction

The previous work, Large Language Models vs Large Language Machines (Zenodo, 4 October 2026, DOI 10.5281/zenodo.23145576), proposed a minimal language machine in which learned sentences remained in an explicit memory. The system could retrieve compatible sequences, rank them using statistical or heuristic criteria, and abstain when the available support was insufficient.

That prototype left one problem open. Retrieving more occurrences from memory is not the same as choosing the continuation best suited to the context. Frequency favours what is common; recency favours what is near; a grammatical rule favours only the structures anticipated by its author. None of these criteria, taken in isolation, determines which clue is relevant in a new configuration.

The next step was therefore to separate two responsibilities:

[current state]
      -> [memory]
      -> [candidates]
      -> [selector]
      -> [output]
      -> [new state]

Memory establishes what may be proposed; the selector establishes which proposal is most compatible with the current state. An element absent from memory cannot be invented by the selector. Conversely, adding new experiences can expand the candidates without automatically requiring their content to be encoded again in the neural weights.

This article is self-contained with respect to the previous one. It does not attempt to build a reduced LLM, but follows the path from heuristic ranking to a minimal neural selector and then to the attempt to understand it well enough to replace it. The most interesting result is not merely the increase in accuracy: it is the emergence of an observable compatibility geometry that is partly transferable and potentially reusable in different domains.

2. From Heuristics to a Learned Selector

Simpler strategies were considered before attention. Frequency selects the candidate supported by the largest number of occurrences, but may ignore a rare yet relevant candidate. Recency and fixed windows privilege nearby elements, but lose long-range dependencies. Explicit rules based on similarity or grammatical role are readable, but build in advance the clues they are supposed to discover and become fragile when sentence structure changes.

These criteria are not useless: they remain baselines, filters and possible auxiliary components. They have not, however, proved sufficient as general replacements for contextual selection. A small learned block, separate from memory, was therefore introduced with the sole task of ranking the alternatives already retrieved.

This is its intended function in the complete architecture. To make the block observable without conflating it with retrieval errors, the geometric test bed described below instead uses a fixed controlled vocabulary and directly measures continuation prediction.

The choice of attention was not an immediate leap towards a larger network. It was a reduction of the problem. A two-head mechanism can be observed element by element; Q, K, weights and coordinates can be recorded; and the visible context can be varied without changing the rest of the architecture. This makes it possible to ask not only whether the selector works, but when it actually begins to select and which regularities it uses.

The experimental path therefore proceeds in both directions. First, heuristics are replaced by a learned selector; the selector is then dismantled to test whether what it has learned can once again be described by a simpler algorithm. The outcome so far is intermediate: some functions and invariants have become readable, but no replacement has yet emerged that preserves the same ability to generalise.

3. Experimental Test Bed and Candidate Restriction

As in the first work, the basic experiments use sentences produced by a controlled grammar. The vocabulary includes recognisable categories, among them animal names, actions and the elements needed to construct a sentence. The same words can occupy different roles: subject, object or agent in a passive construction.

This simplification is not intended to represent the whole of natural language. It makes the category, position and role of each occurrence known in advance, so that what the system might learn can be distinguished. The simple regime uses a limited number of structures; the extended regime adds inverted order, adjectives, relative clauses, negation and the passive voice.

The simple regime includes, among others, structures such as the following. The examples are retained in Italian because these are the actual sequences used in the experiment; the labels in parentheses provide an English gloss.

articolo + animale + mangia + cibo
article + animal + eats + food

articolo + animale + osserva + articolo + luogo
article + animal + observes + article + place

articolo + animale + movimento + nel + luogo
article + animal + movement + in_the + place

articolo + predatore + insegue + articolo + preda
article + predator + chases + article + prey

For example:

il gatto mangia pesce
un lupo corre nel bosco
il cane insegue il coniglio
il gatto annusa la tana

The extended regime reuses the same elements in different constructions:

il gatto che salta vede il topo
il gatto non insegue il topo
il piccolo gatto cerca pesce
il grande gatto protegge il topo
il pesce viene mangiato dal gatto

In this way, gatto remains the same word but may be a subject, an object or the agent of a passive construction; similarly, the same action may occur in different positions and contexts. The regularity of the test bed makes it possible to know which difference was introduced without having to infer it retrospectively from natural text.

In the complete LLMA architecture, memory reduces the space of possibilities before attention intervenes. Starting from the current state, it retrieves compatible sequences and exposes only continuations present in the data. Duplicates and occurrences can contribute to candidate support, while the selector must rank the finite set obtained in this way.

For example, after the prefix il gatto, memory may find sequences continuing with mangia, osserva, insegue, annusa, che, non or dorme. These words become the candidates for the current step. The selector must rank them using the available context; it cannot propose a word that memory has not exposed as an admissible continuation.

The geometric test bed studied in the following sections makes one further abstraction: it does not retrieve from memory, and the final reader assigns a score to all 55 words in the controlled vocabulary. It therefore measures what the selector learns from the visible history, not the quality of a complete memory--selector system. A retrieval error and a ranking error remain distinct phenomena in the LLMA architecture, but this test bed observes only contextual selection over the fixed vocabulary.

4. Experimental Questions and Notation

The model uses two heads and produces two observable coordinates. The notation 2:n denotes two heads and a visible history of n elements. The configurations 2:1, 2:2, 2:3, 2:4 and 2:8 are compared; in this test bed, the last corresponds to the entire sentence being visible.

The tests seek to answer six questions:

1. What remains of attention when only one element is visible?
2. What changes when the first actual choice between two elements appears?
3. Do the coordinates organise only word categories, or do they also retain information about the role played in the sentence?
4. Does increasing context improve accuracy, loss and geometry in the same way?
5. Do the heads simply favour the most recent elements, or do they also retrieve distant positions?
6. Does the observed structure survive different training seeds?

In the 2:1 case, the softmax contains a single value and is necessarily equal to one. This is a singleton inside attention: there is only one visible position. It must not be confused with a continuation set containing only one candidate, in which it would instead be the final reader that had no choice. Q and K cannot learn a preference between positions; the measured cumulative gradient for Q/K is indeed zero. Any structures visible in the coordinates come from the other components, not from a choice made by attention.

With 2:2, the first actual selection appears. Q and K receive gradients and the two heads can assign different weights to the two elements. Subsequent windows make it possible to distinguish the benefit of contextual selection from the mere increase in the amount of available history.

4.1 Architecture and Training

Each word has a learned four-component embedding. In each of the two heads, query, key and value are scalar projections: the query is derived from the last visible word, while key and value are computed for every position in the history. The product of query and key generates the position score; a softmax over the scores produces the weights used to sum the values. The resulting two scalars are the observable coordinates labelled x and y in the plots.

A linear reader transforms the two coordinates into 55 logits, one for each word in the vocabulary, to which cross-entropy against the expected continuation is applied. All parameters -- embeddings, Q/K/V projections, reader and biases -- are learned by gradient descent, with shuffled examples and gradient-norm clipping at 5. Training lasts 900 epochs: step size 0.018 for the first 600 and 0.007 for the remaining 300.

The original accuracies measured refitting on the same controlled corpus used for training. A subsequent check separated the sentences, assigning one in five deterministically to evaluation: no step from an evaluation sentence is therefore used to update the weights. This second test is repeated with five seeds.

5. Continuation Results

Accuracy measures how often the model correctly identifies the next element in the sequence among the 55 words of the controlled vocabulary. Values from the first scan describe the corpus on which the model was trained and must not be interpreted as out-of-sample accuracy.

In the simple regime, with only one context element (2:1), 132/324 continuations are correct, or 40.7%. Showing two elements (2:2) raises the result to 169/324, or 52.2%. The best result is obtained with four elements (2:4): 180/324, or 55.6%. Supplying the entire sentence (2:8) brings no further advantage and slightly reduces accuracy to 172/324, or 53.1%.

In the extended regime, accuracy rises from 44.9% (385/857) with one element to 53.6% (459/857) with two elements, and reaches 55.5% (476/857) with the full context (2:8). The maximum is therefore substantially equal to that of the simple regime, despite the extended task containing more cases and greater structural variety.

Mean loss does not, however, improve monotonically. A longer context can increase the number of correct answers without reducing mean cross-entropy. Accuracy, loss and representation shape therefore do not describe a single notion of progress.

The main operational jump appears between the degenerate 2:1 case and the first choice between two elements. Beyond that point, additional information remains useful, but its value depends on the property being measured.

To provide a clearer reference, the extended test was repeated with a sentence-level split: 681 steps belong to training and 176 to sentences never used to update the weights. The table reports the mean and standard deviation of evaluation accuracy over five seeds:

On the same evaluation set, a uniform choice among the 55 words has an expected accuracy of 1.8%; always choosing il, the most frequent word in training, obtains 22.2%; repeating the last visible word obtains 0%. The absolute value remains that of a small synthetic test bed, but the benefit of context does not depend solely on refitting the learned sentences: about eight percentage points remain between 2:1 and 2:8 on held-out sentences.

6. Category and Role Are Not the Same

In the controlled regime, occurrences belonging to the animal category generally form a more compact region than other words. This does not prove the existence of a semantic concept: the concentration may depend on the vocabulary, sentence distribution or prediction objective.

A more discriminating test considers the role played by the same category. Each animal may appear as subject, object or agent in a passive construction. A centroid classifier receives only the two coordinates produced by the model; in each test, the animal being classified is excluded from construction of the centroids.

The 39 cases come from 13 animals observed in three roles. For each animal, the three centroids are built using only the points of the other 12 animals; each of its three points is then assigned to the nearest centre by Euclidean distance. The coordinates of the evaluated animal do not contribute to the construction of the centroids.

Results in the extended regime are:

In the test bed and seed examined, three elements are the lowest observed threshold at which role becomes almost entirely readable. The full sentence is not the best configuration. Word category and the function of an occurrence therefore leave different traces, and the useful amount of context depends on which of the two is to be recognised.

7. Full Context versus Fixed Window

A 2:8 network, trained once on the full context, was evaluated without retraining in three modes on the original corpus:

The learned choice of the best three recovers 63 answers relative to the last three window and retains about 95% of the correct answers obtained with the full context. It does not equal it: some information may remain distributed over multiple positions, or pruning may alter the reader's behaviour.

The best three are selected separately for each head by ranking the respective Q/K scores before the softmax. The comparison therefore retains three positions per head, as does the last-three window, but does not imply that the two heads retain the same positions. Since the model was trained with the full context and pruned only during evaluation, this test shows how much information the learned selection retains; it does not demonstrate that this is the best architecture trainable with three elements.

Among the 374 cases with more than three visible elements, at least one head selects an element outside the last three in 370 cases. In 319 cases, the two heads select different sets. The selector therefore does not merely imitate a short memory or systematically privilege recency.

This comparison explains why, on the model examined, a fixed window did not prove an adequate replacement. Cardinality per head is the same, but it is the criterion by which elements are chosen that preserves most of the predictive ability.

8. Grammatical Transfer and the Factual Limit

In an exclusion test, aquila remains in the lexicon and in other constructions, while its passive sentence is removed from training. On the excluded sequence, the full context and the best three elements reconstruct the segment viene -> mangiato -> dal without error; the last-three window instead gets the mangiato step wrong.

The final term aquila, although still in the vocabulary, is not predicted. The prefix contains no clue that distinguishes it from the other animals associated with the same food. The test therefore separates two abilities:

• transferring a grammatical construction to an element that had not appeared in that construction;
• possessing the fact required to choose a specific entity.

The first result is observable in this single sequence; the second cannot be reconstructed from the remaining data. An LLMA should not conceal this absence by inventing a fact. In the complete system, the distinction suggests measuring memory recall and selector ranking separately: if the required candidate was not retrieved, neural ranking cannot correct the problem.

9. What We Learned by Trying to Replace Attention

Attention was introduced as a minimal selector, but the next step was to try to eliminate it. Q, K, values, head weights and choices were observed separately; fixed windows, recency, frequency and explicit rules were compared. Initial ablations indicate that, in the test bed, contextual distinction weakens when the attention block is disabled and is not reproduced by the feed-forward network alone. Since this phase does not yet have a quantitative scan comparable to the window experiments, the result should be regarded as preliminary.

The dissection supports a number of precise statements. With a singleton there is no selection. With at least two elements, the heads learn different compatibilities. They do not look only at the last position, can retrieve distant clues, and often choose different sets. The coordinates retain enough information to make many grammatical roles readable, but do not constitute a complete causal explanation of the decision.

The attempt to reduce the behaviour to an explicit rule has not yet succeeded. A rule may imitate the examples seen or a particular positional shape without preserving generalisation to excluded cases. We can therefore describe more clearly what a potential replacement must do, but cannot yet replace attention with frequency, fixed distance or a short list of conditions.

This outcome suggests a component conceptually separable from the LLMA architecture. The block can be read as a learned compatibility selector between a state and a restricted set of alternatives. It is not necessarily a distance in the mathematical sense: the comparison may be directional, and symmetry, identity and the triangle inequality have not been demonstrated. It is, however, a potentially reusable component whenever another system can produce inspectable candidates.

10. Different Seeds: Same Structure, Different Coordinates

Five training runs identical except for the seed produce constellations with different orientations. To compare them, translation, scale, rotation and, where necessary, reflection were removed through Procrustes alignment.

After alignment, a centroid decoder built on four seeds and applied to the fifth recognises the sentence phase with accuracies between 79.6% and 90.4%, compared with a majority baseline of 21.6%.

A transferable relative structure therefore survives a change of seed, provided that an explicit bridge between homologous points is available. The experiment does not demonstrate that alignment can be derived blindly from the distributions alone.

The result also imposes caution in interpreting the plots. Two training runs may represent a similar structure while placing it on rotated or reflected axes. Meaning does not belong to the absolute orientation of a coordinate, but to the relations that remain recognisable after alignment.

11. Conclusions and Limitations

The experiments support six circumscribed conclusions:

1. 2:1 is a degenerate control: with a single element, Q and K perform no selection.
2. The first actual choice, 2:2, clearly improves continuation and makes part of the grammatical role readable.
3. In the controlled test bed and seed examined, three elements are the lowest observed threshold for a strong separation of roles.
4. The heads regularly use distant positions; three selected elements clearly outperform the last three.
5. A grammatical construction can transfer without reconstructing a deliberately removed fact.
6. Absolute geometry depends on the seed, while a relative structure remains transferable after supervised alignment.

These data do not demonstrate that the two coordinates are universal semantic axes, that the clouds correspond to cognitive concepts, or that attention weights provide a complete explanation. Nor do they show that the same geometry emerges on any natural corpus, that the selector can already use every fact added to memory without retraining, or that an LLMA can replace a general-purpose LLM.

The most general result is methodological. Separating memory, candidate production and selection exposes errors that remain conflated in a single model. It becomes possible to ask whether the necessary datum was available, whether it was retrieved, and whether it was ranked correctly. Attention can also be studied as an autonomous component without attributing to it either all of the machine's knowledge or all responsibility for the result.

A public demonstration of this separation is provided by Shrek, which retrieves continuations from an explicit memory and allows the ranking produced by the neural selector to be compared with the ranking obtained by Jaccard similarity. The demonstration is available at https://civolle.it/completa-la-frase/. Shrek is not presented as a chatbot, but as an inspectable implementation of the mechanism studied, including its failures and the cases in which memory offers no adequate continuation.

12. Possible Future Developments

Shrek is a first step from controlled test beds towards an extended linguistic memory. The public version searches 1,979,923 Wikipedia paragraphs, corresponding to 755,327,631 sequence elements and 3,244,688 distinct words. The neural selector, however, was trained on a much smaller sample: 11,637 training comparisons and 3,258 evaluation comparisons. The size of memory and the number of examples used to learn the ranking must therefore not be confused.

The machine can grow in several directions, to be compared one at a time rather than combined in a single larger model.

12.1 Scaling Memory and Selection

A larger memory does not automatically require larger attention. The first requirement is retrieval capable of reducing hundreds of millions of elements to a small candidate set with good coverage. If the necessary continuation is not retrieved, no increase in selector capacity can correct its absence.

As the variety of relations to be distinguished grows, different limits emerge. A wider initial encoding may reduce collisions between different forms and structures; larger heads may represent finer compatibilities; more heads may simultaneously observe different criteria, such as grammatical agreement, semantic continuity, provenance and long-range dependencies. More heads do not, however, guarantee specialisation: without tests requiring distinct behaviours, multiple heads may learn the same shortcut.

A further development is hierarchical selection. An inexpensive first stage could reduce several hundred candidates to a few dozen; a richer second stage could rank them; and a final check could verify the consistency of the continuation with the overall state. Updating a representation of the selected result would also preserve something more than the last emitted word between steps.

These possibilities must be compared without increasing all dimensions at once. An initial experimental matrix could vary independently:

encoding: 16, 32, 64 dimensions
heads:    2, 4, 8
stages:   1, 2
data:     10 thousand, 100 thousand, 1 million comparisons

The point at which new data cease to improve an unchanged model will indicate whether the limit lies in experience quantity, retrieval or selector capacity. Accuracy, calibration, rare structures, composition across sources and coherence over continuations of increasing length will need to be measured separately.

12.2 Multiple Heads: Different Relations or Copies of the Same Relation?

One question remains open that the current test beds, designed to isolate the minimal mechanism, cannot answer: as the number of heads increases, do they learn different relations or multiple representations of the same relation? Both outcomes are possible. Different heads might specialise, for example, in grammatical agreement, semantic continuity, distance or sentence phase; they might instead converge on the same shortcut, or divide a relation that a single head already represented.

It follows that the number of modelled relations does not necessarily equal the number of heads. One relation may require multiple heads, while one head may contribute to different relations in different regions of the space. A real division of labour must also be distinguished from redundancy and from alternative but functionally equivalent representations.

A dedicated test bed should compare otherwise identical models with one, two, four and eight heads. For each model, the selection distributions of the heads could be compared, their similarity measured, and one head at a time disabled. If ablation does not appreciably change the errors, the head is probably redundant; if specific classes of error appear, there is evidence of specialisation. Tests should also include sentences in which two independent relations are simultaneously necessary, and repeat the comparison across seeds to separate stable specialisation from random allocation.

Finally, three ways of increasing capacity should be compared: more heads, larger heads and more successive selection stages. Establishing which of the three absorbs new relations at the lowest cost would turn the selector from an experimental result into a component that can be dimensioned.

12.3 Updating Memory without Retraining the Selector

The decisive test is to freeze the selector, add genuinely new experiences to memory, and determine whether they enter generation without updating the weights. Evaluation should distinguish:

• presence of the correct candidate in memory;
• retrieval of the candidate in the current state;
• ranking quality when the candidate is present;
• empty sets, singletons and ambiguous alternatives;
• overall performance, cost and abstention;
• comparison with frequency, n-grams and selection without attention.

Only this separation will establish whether explicit memory is a store consulted by a network or a genuinely updatable component of the language machine.

12.4 Reusing the Selector

The selector appears separable from the specific LLMA architecture and can be tested in domains where an external system produces inspectable alternatives. One ongoing experiment concerns historical handwriting recognition: candidate representations of the trace or gesture are ranked by contextual compatibility. It is not yet known whether a conventional recogniser, a grammar of paths or a hybrid solution will prove more effective; the value of the test lies precisely in being able to separate production of alternatives from their ranking.

13. Availability of Materials

The article and the figures documenting its results are intended for public deposit on Zenodo. Experimental source code, intermediate data and the materials required to reproduce the test beds in full are not part of this deposit; they are retained separately and may be requested from the author.

The materials are organised according to the successive stages of the research:

1. First LLMA experiment and comparison with the reference model. LLMA-euristica.php, LLMA.php and llm.php document, respectively, the initial heuristic selection, the associative language machine and the comparison with the reference statistical continuation.
2. Separation of explicit memory and selector. The banco-ibrido-memoria-attention-v1 test bed, whose main source is banco.php, retains the first experiment in which candidates come from memory and the network only has to rank them. Subsequent dimensional tests are collected in prova-d1.php and the related JSON results.
3. Comparison between context windows. attention-2x1-2x2.cpp generates the 2:1, 2:2, 2:3, 2:4 and 2:8 tests in the simple and extended regimes; analizza.py produces the metrics, coordinates and tables used in Sections 4--8. The attention-split.cpp variant repeats the test while separating entire training and evaluation sentences, and retains the five-seed metrics used in Section 5.
4. Test on excluded elements and selection analysis. The same test bed contains the holdout-aquila-passiva data, the scelte-* and selezione-* tables, together with disegna-selezione.py and the reports RISULTATO-SINTASSI.md and RISULTATO-SELEZIONE.md.
5. Stability across initialisations. attention-multiseme.cpp repeats the test bed with five seeds; analizza-multiseme.py and analizza-multiseme-p3.py measure alignment, transfer and relative geometry. TSV files for each seed, reports and SHA-256 fingerprints are also retained.
6. Shrek demonstrator. addestra-cross.cpp and scegli-cross.cpp implement selector training and choice; incipit preparation and connection to the public demonstrator are documented by prepara-incipit-cross.php and worker.php.

For each stage, where produced, the source code, input data, raw outputs, derived tables, figures and file fingerprints are retained. It is therefore possible to request either a single test bed or the complete experimental sequence.

14. Acknowledgements and Collaboration with Artificial Intelligence Systems

The author wishes to acknowledge the contribution of the artificial intelligence systems that participated, in different roles, in the discussion, experimental development and critical review of the project:

• Zeno Malvagio (Z.M.) — experimental development, implementation and analysis of results.
• Zeno Aziendale (Z.A.) — methodological review, verification of claims and identification of experimental limitations.
• Susan (ChatGPT, OpenAI) — scientific discussion, critical analysis and manuscript review.
• Fiona — participation in discussions and comparison of approaches.

The contributions were produced within a collaboration coordinated by the author, who retains responsibility for the methodological choices, results and content of the publication.