Tessel libraries

Tessel libraries / text

text 0.3.0

Text tools: Unicode characters and normalization, words and sentences, hyphenation, spelling, fuzzy matching, stemming, search and readability

tessel add text

MIT licence · 3 versions · used by 0 projects · published 2026-10-31

Text tools for Tessel programs: characters as a reader sees them, normalization, words and sentences, hyphenation, spelling, fuzzy matching, stemming, full-text search and readability scores. It follows Unicode 17's rules, and the results are checked against Unicode's own test files and against Python libraries.

import text

fn main() {
    let note = "Don't skip it: the beam's depth is 3.5 m. Check e.g. the 1,000 kN case first! 👨‍👩‍👧 approves."
    print(text.words(note))
    print(text.wordCount(note))
    print(text.sentences(note))
    print("👨‍👩‍👧".count)
    print(text.graphemeCount("👨‍👩‍👧"))

    let h = text.englishHyphenator()
    print(h.hyphenate("reinforcement"))
    print(h.parts("hyphenation"))

    print(text.levenshtein("kitten", "sitting"))
    print(text.closest("lenght", ["length", "height", "width"]))
    print(text.fuzzyFilter("opfl", ["Open File", "Open Folder", "Copy File", "Toggle Fold"]))
}

prints

["Don't", "skip", "it", "the", "beam's", "depth", "is", "3.5", "m", "Check", "e.g", "the", "1,000", "kN", "case", "first", "approves"]
17
["Don't skip it: the beam's depth is 3.5 m. ", "Check e.g. the 1,000 kN case first! ", "👨‍👩‍👧 approves."]
5
1
re-in-force-ment
["hy", "phen", "ation"]
3
["length", "height"]
["Open File", "Open Folder", "Copy File"]

Add it to a program with tessel add text, then import text.

Positions

Tessel's strings count characters as Unicode code points: count, substring and character(at:) all work that way. Every position this library gives or takes is a code point index, so it can go straight into substring. Code points aren't always what a reader calls a character, though. The family emoji above is 5 code points but one grapheme cluster, so carets and Backspace should move by nextGrapheme and previousGrapheme.

Characters

Function
codePoints(_ s: String) -> [Int] The code points of s.
codePoint(_ ch: String) -> Int The code point of the first character, -1 for "".
fromCodePoints(_ cps: [Int]) -> String The text of some code points (U+FFFD for numbers that aren't code points).
category(_ c: Int) -> String The General_Category: "Lu", "Ll", "Nd", "Po", "Sc", "So"…
isLetter, isUppercase, isLowercase, isMark, isDigit, isNumber, isPunctuation, isSymbol, isSpace, isAlphanumeric (_ c: Int) -> Bool, by the Unicode properties.
caseFold(_ s: String) -> String The form for comparing without case: "Straße" and "STRASSE" both fold to "strasse".
sameIgnoringCase(_ a: String, _ b: String) -> Bool Whether the texts are equal apart from case, however their accents are written.
combiningClass(_ c: Int) -> Int The Canonical_Combining_Class: 0 for most characters, 230 for most accents above.
unicodeVersion "17.0.0".

Normalization

The same text can be stored as different code points. For example, "é" can be one code point (U+00E9), or "e" followed by a combining accent (U+0301):

let typed = "Cafe" + text.fromCodePoints([769])
print(typed == "Café")              // false
print(text.nfc(typed) == "Café")    // true
print(text.nfkc("file №2"))          // file No2

nfc (composed: the usual form for storing text), nfd (decomposed), nfkc and nfkd follow UAX #15. The K forms also turn ligatures, full-width letters, superscripts and the like into plain characters, which is useful for searching and comparing. All 20,034 cases of Unicode's NormalizationTest.txt pass, and every other code point is left as it is.

Grapheme clusters, words and sentences

These follow the default rules of UAX #29. A list of breaks starts at 0 and ends at the text's count.

Function
graphemes(_ s) -> [String], graphemeCount(_ s) -> Int, graphemeBreaks(_ s) -> [Int] Characters as a reader counts them: an accented letter, a flag, an emoji with a skin tone, CR LF.
nextGrapheme(_ s, _ at) -> Int, previousGrapheme(_ s, _ at) -> Int Where the caret goes with the arrow keys, and how far Backspace deletes.
words(_ s) -> [String], wordCount(_ s) -> Int The words: pieces with a letter or number in them. "don't", "e.g", "3.5", "1,000" and "foo_bar" each count as one word. Chinese and Japanese ideographs count one each.
wordPieces(_ s) -> [Piece], wordBreaks(_ s) -> [Int] All the pieces, words and the rest, with from, to and isWord.
wordAt(_ s, _ at) -> Piece The piece at a position: what a double click selects.
nextWordEnd, nextWordStart, previousWordStart (_ s, _ at) -> Int: Option/Ctrl+arrow movement.
sentences(_ s) -> [String], sentenceBreaks(_ s) -> [Int] Sentences, each with the spaces after it. No break after the full stop in "3.5" or in "e.g. this". The rules can't tell that "Mr. Smith" doesn't end a sentence.

Speed: a 342,000-character text (about 100 pages) is word-counted in 10 ms and split into sentences in 6 ms.

Hyphenation

let h = text.englishHyphenator()           // make once and keep: about 4 ms
h.hyphenate("hyphenation")                 // "hy-phen-ation"
h.points("structural")                     // [5]
h.hyphenateText("Reinforced concrete")     // soft hyphens (U+00AD) wherever a word may break

This is Liang's method, the one TeX uses. englishHyphenator has TeX's American English patterns (hyph-en-us.tex, 4,938 patterns and Knuth's exceptions). It keeps at least 2 letters before the first break and 3 after the last; englishHyphenator(left:right:) changes that. For other languages, give hyphenatorFromTex a pattern file from hyph-utf8, or give hyphenator the patterns and exceptions themselves. Words containing anything other than letters ("don't", "3rd") aren't hyphenated.

Spelling

var sp = text.speller(text.systemWordList(), user: "Tessel\n")
sp.isCorrect("reinforcement")       // true
sp.isCorrect("recieve")             // false
for m in sp.misspellings("Teh beam was stoped by Tessel's sensor.") {
    print("{m.word} at {m.from}-{m.to}: {sp.suggest(m.word, limit: 3)}")
}
// Teh at 0-3: ["The", "Ten", "Tea"]
// stoped at 13-19: ["stopped", "stiped", "stooped"]
let line = sp.addWord("Syncster")   // "Syncster\n": append it to the user's dictionary file
text.autocorrect(text.autocorrectDefaults(), "Teh")   // "The"

A Speller is built from a word list with one word per line. systemWordList() reads /usr/share/dict/words (236,000 words, built in about 17 ms) and gives "" on Windows. A word also counts as right when it is a listed word with a regular ending (-s, -ed, -ing, -er, -est, -ly, -ness, un-), following the usual spelling rules: "stopping" is right but "stoping" isn't, and "shelves" is right but "shelfs" isn't. Capitals, contractions, possessives, hyphenated words, acronyms, numbers and web addresses are handled too. suggest ranks words one or two edits away by how likely the slip is: swapped and doubled letters, letters left out, neighbouring keys and wrong vowels. It also fixes endings ("basicly" → "basically") and splits words run together ("alot" → "a lot"). addWord, ignoreAll, ignoreOnce and resetIgnored work like the same commands in Word. autocorrect fixes about 140 common typing slips and keeps their capitals.

This is the spell checker from the Quill word processor. With the system's list, Quill's tests flag nothing in 1,548 words of ordinary prose, and offer the intended word first for 42 of 42 common misspellings and 62 of 62 harder ones. It checks words at about 4 million a second.

Fuzzy matching

Function
levenshtein(_ a, _ b) -> Int Insertions, deletions and substitutions.
damerauLevenshtein(_ a, _ b) -> Int The same, also counting a swap of two neighbouring characters as one edit (the "optimal string alignment" version).
similarity(_ a, _ b) -> Float 0 to 1: 1 − (insertions + deletions) / (both lengths). This is rapidfuzz's ratio / 100.
jaro, jaroWinkler(_ a, _ b, prefixWeight: 0.1) For short texts such as names.
closest(_ word, _ choices, limit: 3, cutoff: 0.6) -> [String] "Did you mean…": the most similar choices, ignoring case.
fuzzyMatch(_ pattern, _ item) -> FuzzyMatch? Whether the pattern's letters appear in the item in order. Gives a score and the matched positions, for showing them in bold.
fuzzyFilter(_ pattern, _ items) -> [String] The matching items, best first, as in a command palette.

fuzzyMatch scores a matched letter higher when it starts a word (after a space or punctuation, or a capital after a small letter) or follows the previous match directly. Each letter skipped costs a point, so "opfl" ranks "Open File" before "Open Folder".

Stemming

text.stem("running")        // "run"
text.stems("The engineers were checking the bending of reinforced beams")
// ["the", "engin", "were", "check", "the", "bend", "of", "reinforc", "beam"]

This is Snowball's English (Porter2) stemmer. It gives the same stem as Snowball for all 42,649 words of Snowball's own test list. A stem isn't always a word ("engin", "happi"), so compare stems with stems.

var help = text.SearchIndex()
let pages = [
    "Simply supported beams: bending moment and shear force diagrams.",
    "Reinforced concrete columns under axial load and bending.",
    "Steel beam connections: bolted and welded.",
    "The moment of inertia of a section.",
]
for p in pages {
    let ignored = help.add(p)                     // gives the document's number: 0, 1, 2…
}
for hit in help.search("beam bending") {
    print("{hit.document} {hit.score.formatted(decimals: 3)}  {pages[hit.document]}")
}
print(help.search("\"bending moment\"").count)    // a phrase: 1
print(help.search("reinf", prefix: true)[0].document)   // while typing: 1
0 0.531  Simply supported beams: bending moment and shear force diagrams.
2 0.327  Steel beam connections: bolted and welded.
1 0.304  Reinforced concrete columns under axial load and bending.
1
1

Results are ranked by BM25, as in Lucene and Elasticsearch (k1 = 1.2, b = 0.75). The scores match Python's bm25s. Documents and queries are cut into words, case folded, stripped of Snowball's 174 English stopwords and stemmed, so "beam" finds "beams". SearchIndex(stemming: false, stopwords: []) matches whole words only. A "phrase in quotes" must appear with its words in order. prefix: true lets the last word be cut short, for searching as you type. addTerms and searchTerms take terms that you split up yourself.

Readability

let st = text.textStats("The cat sat on the mat. It was happy. Readability formulas estimate the education needed to understand a passage.")
print("{st.sentences} sentences, {st.words} words, {st.syllables} syllables")   // 3 sentences, 19 words, 35 syllables
print(st.fleschReadingEase().formatted(decimals: 1))                             // 44.6
print(st.fleschKincaidGrade().formatted(decimals: 1))                            // 8.6

TextStats holds the counts and has fleschReadingEase, fleschKincaidGrade, gunningFog, smogIndex, colemanLiauIndex, automatedReadabilityIndex and readingMinutes. syllables(word) counts by English spelling rules. On the 35,627 words that the CMU Pronouncing Dictionary shares with /usr/share/dict/words, the count matches CMU's for 93.6% of them, and for 97.5% of the words of ordinary prose. That is close enough for formulas that average over a whole text. It does get some words wrong ("created" counts as 2).

Tests

tessel run tests checks:

  • the category of all 1,114,112 code points against UnicodeData.txt, and White_Space against PropList.txt;
  • all 20,034 cases of Unicode's NormalizationTest.txt, and that every other code point stays as it is;
  • case folding against CaseFolding.txt and Python's str.casefold;
  • all 3,222 cases of Unicode's GraphemeBreakTest.txt, WordBreakTest.txt and SentenceBreakTest.txt;
  • hyphenation of 14,000 words against pyphen with the same patterns;
  • 1,509 pairs of distances against rapidfuzz;
  • stems of Snowball's whole English test list (42,649 words);
  • BM25 scores of 400 queries over 300 documents against bm25s;
  • syllable counts of 35,627 words, against the rules' first version in Python and against the CMU dictionary;
  • the spelling tests from Quill, including the system word list;
  • the rest by hand, plus speed.

tests/cases.json is made by tools/oracle/oracle.py. Running it needs the downloaded files (python3 tools/fetch.py) and a venv with pyphen, rapidfuzz, snowballstemmer, bm25s and cmudict. tools/gen/gen_tables.py and tools/gen/hyph.py write unicode_data.tsl and hyphen_en.tsl from the same files.

Licence

MIT. The English hyphenation patterns in hyphen_en.tsl are © Gerard D.C. Kuiken, and may be copied and changed as long as their notice (kept at the top of that file) stays with them. The stopword list is Snowball's (BSD licence).

Used by

No library in this index or public GitHub repository uses text yet. Use it with tessel add text and it shows up here.

Versions

VersionTagCommitPublishedUses
0.3.0v0.3.041ce8425162026-10-31–
0.2.0v0.2.0874b0e948b2026-10-31–
0.1.0v0.1.01803b58f7e2026-10-31–

Each version's entry in the index records its commit and the checksum of its code, and tessel add checks the download against them.