I am making a lexer, don't tell me to not do because I already did most of it.
Currently it makes an array of tokens and that's it.
I would like to know, what functions the lexer needs to provide and a brief explanation of what each function needs to do.
I'll accept the most complete list.
An example function would be:
**next**: Consume the current token and return it
Also, should the lexer have the `expect` function or should the interpreter implement it?
By the way, the lexer constructor accepts a string as argument and make the lexical analyses and store all the tokens in the "tokens" variable.
以上就是What functions a lexer needs to provide?的详细内容，更多请关注web前端其它相关文章！