tokenizer package

The zero-copy SQL lexer that turns raw bytes into position-tagged tokens.

Import path: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer (pkg.go.dev)

Part of the GoSQLX API reference.

Functions

GetTokenizer() *Tokenizer

Retrieve tokenizer from pool.

tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz) // ALWAYS defer

PutTokenizer(t *Tokenizer)

Return tokenizer to pool.

Type: Tokenizer

Tokenize(input []byte) ([]models.TokenWithSpan, error)

Tokenize SQL with zero-copy operation and position tracking.

tokens, err := tkz.Tokenize([]byte("SELECT * FROM users"))

TokenizeContext(ctx context.Context, input []byte) ([]models.TokenWithSpan, error)

Tokenize with context support.

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
tokens, err := tkz.TokenizeContext(ctx, []byte("SELECT * FROM users"))

Supported Token Types

Features:

• O(1) integer-based token type checking via the ModelType field
• Unicode support (UTF-8)
• Dialect-specific tokens (PostgreSQL, MySQL, SQL Server, Oracle, SQLite)
• Zero-copy operations
• Position tracking (line, column, offset)
• Scientific notation support (1.23e4, 1.23E+4)
• Triple-quoted strings (Python-style)
• Escape sequences (\n, \t, \r, \, ', ")