Myrddin Dundragon
6984455c12
The code now returns a result that has the TokenStream. For now it is returning any Error, but later there should be a lexical analysis error type.
Rune
Rune is a high-performance, customizable lexical analysis library written in Rust.
It transforms source files into tokens using a fast, cache-friendly design.
“Turn raw text into structured meaning — like spellcraft for source code.”
Features
- Basic tokenization: Whitespace, text, numbers, symbols, and newlines.
- Flat
TokenStreamdesign: Optimized for speed and cache locality. - Custom transforms: Supply your own function to turn base tokens into domain-specific ones (like Markdown, HTML, or custom Domain Specific Languages).
- Iterators and mutation: Traverse or modify tokens efficiently.
Getting Started
Add to your project
This library is hosted on the CyberMages registry. To add it to a project the CyberMages registry will need to be added to Cargo as per the Cargo Book.
First add the registry to your cargo config.toml file.
[registries.cybermages]
index = "sparse+https://workshop.cybermages.tech/api/packages/CyberMages/cargo/"
Then add this to your Cargo.toml file.
[dependencies]
rune = { version = "0.3.0", registry = "cybermages" }
Copyright & License
Copyright 2025 CyberMages LLC
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this library except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS