Python — strings and text processing

Almost all the data a language model meets is text. Being able to process text in Python is therefore fundamental.

A string is a sequence of characters. You can:

• index and slice it: s[0], s[-1], s[2:5]
• normalise it: s.lower(), s.strip()
• split and join: s.split(), " ".join(items)
• replace: s.replace("a", "b")
• ask questions: "ai" in s, s.startswith("Hi")

Strings are immutable: the methods give you a new string, the original is unchanged.

line = "  Anna, 14, Gothenburg  "
parts = [p.strip() for p in line.strip().split(",")]
print(parts)                     # ['Anna', '14', 'Gothenburg']
name, age, city = parts
print(f"{name} is {int(age) + 1} next year")   # Anna is 15 next year

text = "AI is fun. AI is hard."
print(text.lower().count("ai"))          # 2
words = text.lower().replace(".", "").split()
print(len(words), words[:3])             # 6 ['ai', 'is', 'fun']

Tokenisation in a language model starts exactly like this: normalise, split, count.