Real databases turn a SQL string into a stream of tokens before they ever try to understand its structure -- Postgres's lexer and DuckDB's tokenizer both do this same job, just with a lot more grammar to cover.

Implement tokenize(sql): given a SQL string, return a list of Token(kind, value) objects. Token kinds:

• KEYWORD -- one of SELECT, FROM, WHERE, AND, OR (case-insensitive, normalize to uppercase)
• IDENT -- an identifier (table/column name), case preserved
• NUMBER -- a numeric literal, kept as a string
• STRING -- a single-quoted string literal, with the quotes stripped
• SYMBOL -- punctuation/operators: , ( ) ; = < > * <= >= !=

-- starts a line comment (skip to the next newline, or end of input) and produces no token at all -- not a SYMBOL, not anything.

The missing link: a string literal isn't "scan to the next quote"

That's the version that looks obviously correct and is subtly wrong. SQL's actual rule (Postgres, MySQL, SQLite all agree on this) is: a literal single quote INSIDE a string is written as two single quotes in a row. 'it''s' is ONE string token whose value is it's -- not the string it immediately followed by a stray, mis-parsed s'. A tokenizer that stops at the first ' it sees will silently produce garbage on any string containing an apostrophe, which is exactly the kind of user data (names, addresses, free text) a real query is full of.

Examples

tokenize("SELECT id FROM users")
-> [Token("KEYWORD","SELECT"), Token("IDENT","id"), Token("KEYWORD","FROM"), Token("IDENT","users")]

tokenize("WHERE age >= 18")
-> [Token("KEYWORD","WHERE"), Token("IDENT","age"), Token("SYMBOL",">="), Token("NUMBER","18")]

tokenize("WHERE name = 'it''s'")
-> [Token("KEYWORD","WHERE"), Token("IDENT","name"), Token("SYMBOL","="), Token("STRING","it's")]

Constraints

• Multi-character operators (<=, >=, !=) must not be split into two SYMBOL tokens.
• Keywords are case-insensitive; identifiers are not.
• Whitespace between tokens is not itself a token, and neither is a -- comment.
• A doubled '' inside a string is one literal quote character in the token's value, not the end of the string.

Real-world grounding

• Postgres's lexer is a flex grammar: src/backend/parser/scan.l -- search it for xqdouble (the doubled-quote-escape rule) and <xc> (its comment-handling start condition).
• DuckDB's tokenizer lives under src/parser/peg/tokenizer/

Both do the same conceptual job as this exercise, just with a lot more SQL grammar to recognize.