Token state
Change state from having lists of keywords, identifiers, etc., to a list of token objects, where each token object has properties such as "isKeyword", "isIdentifier", etc.
Change state from having lists of keywords, identifiers, etc., to a list of token objects, where each token object has properties such as "isKeyword", "isIdentifier", etc.