Myrddin Dundragon
7c564d18a2
This makes it so that the TokenStream and all it's associated Token types use a generic when dealing with the variant of the Token. Span was also given the ability to merge with another span. This will make it easier to track the span as users group TokenTypes together to make their domain specific types. All tests and examples were updated for this change. The version was incremented to 0.2.0.
Rune
Rune is a high-performance, customizable lexical analysis library written in Rust.
It transforms source files into tokens using a fast, cache-friendly design.
“Turn raw text into structured meaning — like spellcraft for source code.”
Features
- Basic tokenization: Whitespace, text, numbers, symbols, and newlines.
- Flat
TokenStreamdesign: Optimized for speed and cache locality. - Custom transforms: Supply your own function to turn base tokens into domain-specific ones (like Markdown, HTML, or custom Domain Specific Languages).
- Iterators and mutation: Traverse or modify tokens efficiently.
Getting Started
Add to your project
This library is hosted on the CyberMages registry. To add it to a project the CyberMages registry will need to be added to Cargo as per the Cargo Book.
First add the registry to your cargo config.toml file.
[registries.cybermages]
index = "sparse+https://workshop.cybermages.tech/api/packages/CyberMages/cargo/"
Then add this to your Cargo.toml file.
[dependencies]
rune = { version = "0.2.0", registry = "cybermages" }