I am making a lexer, don't tell me to not do because I already did most of it. Currently it makes an array of tokens and that's it. I would like to know, what functions the lexer needs to provide and a brief explanation of what each function needs to do. I'll accept the most complete list. An example function would be: **next**: Consume the current token and return it Also, should the lexer have the `expect` function or should the interpreter implement it? By the way, the lexer constructor accepts a string as argument and make the lexical analyses and store all the tokens in the "tokens" variable. The language is javascript, so I can't overload operators.
en.wikipedia.org/wiki/Lexical_analysis