Lexer
Registry record
- CopyrightID
-
0x15f6a03a232a1bca0f2cb86fcdd29af406b60abfd239e40d4b5ff70fc98e3978
- EvidenceClass
-
chain-registered
- EvidenceClassStatement
-
This record has a CopyrightID on CopyrightChains and no confirmed Bitcoin attestation that verifies against this record, so no OTS fields are served. Verification establishes record integrity only. It says nothing about who made the work, who holds rights in it, or when it was made.
Lexical tokenization is conversion of a text into (semantically or syntactically) meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping symbols, data types and language keywords. Lexical tokenization is related to the type of tokenization used in large language models (LLMs) but with two differences. First, lexical tokenization is usually based on a lexical grammar, whereas LLM tokenizers are usually probability-based. Second, LLM tokenizers perform a second step that converts the tokens into numerical values.
Similar Artists