Cipher Segmentation Lab: Find the Missing Boundaries

Why cipher boundaries matter

A visible mark need not equal one letter. A meaningful unit may span several marks, and two different segmentations may fit the same stream. Here the units have different lengths and overlap. The decoder lists every compatible path, up to a display limit, while counting all paths.

The five-word filter is deliberately small and visible. It can select an answer because the answer was placed inside that vocabulary. That illustrates why a convincing reading needs independent checks on text the system has not already seen.

Cipher segmentation: one stream, three readings

Segmentation means deciding where one code unit ends and the next begins. Try this separate, hand-checkable example before running the six-unit lab above. Let the available units be A, B, AB and BA. The stream ABA has exactly three complete segmentations:

Removing AB leaves two paths. Removing both AB and BA leaves one. That last answer is unique under the reduced codebook, but it would be wrong if the original writer used AB | A. Uniqueness under an assumption does not validate the assumption.

How can you count the possible segmentations?

Start with one way to read the empty prefix. At each position, try every code unit that matches the next characters. Add the number of paths reaching the current position to the count at the end of each matching unit. For ABA, the prefix counts are 1, 1, 2, 3. This dynamic-programming count avoids listing every path just to learn how many exist.

Do variable-length units always cause ambiguity?

No. A prefix-free code has no complete codeword at the beginning of another codeword, so its boundaries can be decoded without separators when the codebook is known and the stream is valid. For example, 0, 10 and 11 decode 01011 as 0 | 10 | 11. Some uniquely decodable codes are not prefix-free. The overlapping four-unit example above is deliberately ambiguous.

Background: MIT OpenCourseWare’s notes on discrete source encoding and unique decodability. The small examples here can be checked by direct enumeration.

Can a dictionary prove a decipherment?

A dictionary can rank or remove candidate readings. It cannot, by itself, establish the codebook, language or intended message. In the interactive lab, compare the unfiltered candidates with the supplied five-word vocabulary, then reveal the hidden message. A convincing method must also predict held-out text and survive controls in which the proposed reading should fail.

Next, test a related problem in the Context Repair Lab: what information must a summary retain? Use the research claim template to record your codebook, assumptions and a test that could falsify them.

What the ATO investigation found

This is progress on a controlled test. It is not a Voynich decipherment, a general language discovery result or evidence that the original manuscript uses this encoding. The test family came from Michael Greshko’s Naibbe work; it was not invented by ATO.

What would strengthen it?

Freeze the learner before testing another source, another encoding family and an unknown language prior. Report edit counts and failed lines alongside successes. A method should survive changes to the assumptions that helped it work.

Yale’s manuscript record describes the text as undeciphered. Explore the Voynich imaging field note for the difference between observing marks and interpreting them.