6 Commits

Author SHA1 Message Date
7c564d18a2 [#3] TokenStream now hold generic variants.
This makes it so that the TokenStream and all it's associated Token
types use a generic when dealing with the variant of the Token.

Span was also given the ability to merge with another span. This
will make it easier to track the span as users group TokenTypes
together to make their domain specific types.

All tests and examples were updated for this change.

The version was incremented to 0.2.0.
2025-04-16 01:56:37 -04:00
f924811c47 [#1] For some reason gitignore file didn't commit. 2025-04-16 00:29:18 -04:00
86001bac38 [#1] Set swap files to be ignored files.
Also removed the accidentaly committed swap files.
2025-04-15 21:21:33 -04:00
acf869efbb [#2] A data-oriented Lexer.
I took the Token module from the Arcanum project and brought it over
to here. It was a nice data oriented way of handling the Tokens.

I then created a Lexer that can scan a file or text and allow the
user to transform the scanned tokens before the final token array
is returned. This should allow for more complex and specific tokens
to be created for whatever domain is being targeted.

I also added basic library examples and testing.

Finally, I made sure the documentation generated nicely.

This is now marked as version: 0.1.0
2025-04-15 21:17:28 -04:00
0350a151a9 [#1] Initial library setup.
Setting up our Rust format file and README.
2025-04-15 16:52:31 -04:00
54db6247ae Initial commit 2025-04-15 11:32:53 -04:00