Replace ParseDate with unified Ragel -G2 scanner + recursive descent parser

Delete the 1650-line multi-pass constraint-satisfaction engine
(timeparser.cpp) and replace it with date_scan.rl: a Ragel -G2
scanner that tokenizes date strings into a flat token array, plus a
small recursive descent parser that dispatches on the leading token.

The scanner handles month names, day-of-week names, timezone
abbreviations, military timezone letters, meridian (AM/PM), ordinal
suffixes, and numbers — all as DFA alternations with no hash tables
or linear searches.

The parser dispatches on the first non-whitespace token:
  4+ digit number → ISO 8601 family (extended, basic, ordinal, week)
  Month name      → US order (Mmm DD [YYYY] HH:MM:SS / legacy)
  1-2 digit + Month → European order (DD Mmm YYYY HH:MM:SS)
  Day-of-week     → strip prefix, re-dispatch

This is a single code path — always exercised, always tested.  The
old architecture had two independent parsers (do_convtime + ParseDate)
where one could be broken without anyone noticing.

Sub-second precision: unified handling across all formats, accumulating
up to 7 fractional digits as hectonanoseconds (100ns), matching the
internal FACTOR_100NS_PER_SECOND resolution.

All 998 smoke tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Stephen Dennis 2026-04-07 00:10:55 -06:00
parent a96448cbbd
commit 4884ba3dba
7 changed files with 4076 additions and 1657 deletions

282
docs/design-date-parser.md Normal file
View file

@ -0,0 +1,282 @@
# Design: Unified Date Parser
## Status
Proposed — replaces both `do_convtime()` and `ParseDate()` with a
single Ragel -G2 scanner + recursive descent parser.
## Background
`convtime()` currently has two independent code paths:
1. **`do_convtime()`** — Hand-written parser for the legacy format
`[Ddd] Mmm DD HH:MM:SS[.frac] YYYY`. Works correctly.
2. **`ParseDate()`** — 1650-line multi-pass constraint-satisfaction
engine. Was completely broken from 2.13 through 2.14 due to a
`mux_min(int,int)` truncating `size_t`. Nobody noticed for months.
Having two parsers means two sub-second algorithms, two sets of
validation rules, two things to maintain, and the ability for one to
be broken without anyone knowing. The fix is to make everything
one thing.
## Design Principle
One parser, always used, always tested. If it breaks, every
`convsecs()`/`convtime()` round-trip fails and 900+ smoke tests
scream. No silent fallback to a different code path.
## Architecture
```
convtime(input, [zone], [precision])
→ date_parse(input) → FIELDEDTIME + zone info
→ apply timezone logic
→ return epoch string
```
`date_parse()` is the unified entry point. It replaces both
`lta.SetString()` (which called `do_convtime()`) and `ParseDate()`.
### Ragel -G2 Scanner
The scanner tokenizes the input into a flat array of typed tokens:
```c
enum DateToken {
TOK_EOF,
TOK_NUM, // sequence of digits, value + digit count stored
TOK_MONTH, // Jan..Dec / January..December → 1..12
TOK_DOW, // Sun..Sat / Sunday..Saturday → 0..6
TOK_TZ_NAME, // UTC, GMT, EST, PST, CST, MST, EDT, PDT, CDT, MDT,
// HST, AKST, AKDT, BST, IST, CET, CEST, EET, EEST,
// AEST, AST, ADT, UT → offset in minutes
TOK_TZ_MIL, // A..Z (except J) → offset in minutes
TOK_MERIDIAN, // AM=0, PM=12
TOK_SUFFIX, // st, nd, rd, th
TOK_T, // letter T (ISO separator)
TOK_W, // letter W (ISO week prefix)
TOK_Z, // letter Z (UTC indicator)
TOK_DASH, // -
TOK_PLUS, // +
TOK_COLON, // :
TOK_DOT, // .
TOK_COMMA, // ,
TOK_SPACE, // whitespace (collapsed)
};
```
Ragel handles the name matching via DFA alternation. Month names
(24 alternations), day-of-week (14), timezone abbreviations (~25),
and meridian (2) compile into a single goto-driven state machine.
This is a jump table — faster than the old `ParseThreeLetters()`
hash comparison or the `PD_TextTable` linear scan.
Numbers carry their digit count, which the parser uses for
disambiguation (1-2 digits = day/hour/minute/second, 3 digits =
day-of-year, 4 digits = year, 5+ digits = extended year).
### Recursive Descent Parser
After scanning, the parser dispatches on the leading token(s):
```
parse_datetime(tokens):
skip optional TOK_DOW [TOK_COMMA] [TOK_SPACE]
if peek TOK_NUM(4+ digits) or TOK_PLUS or TOK_DASH(followed by digits(4+))
→ parse_iso(tokens)
if peek TOK_MONTH
→ parse_month_leading(tokens)
if peek TOK_NUM(1-2 digits) and peek+1 is TOK_MONTH
→ parse_day_leading(tokens)
→ fail
```
Each handler is a straight left-to-right walk consuming expected
tokens. No backtracking.
#### parse_iso — ISO 8601 family
```
year = consume NUM(4+ digits)
if peek TOK_W → ISO week date
consume W, NUM(2) week
optional DASH, NUM(1) day-of-week
elif peek DASH + NUM(3) → ISO ordinal
consume DASH, NUM(3) day-of-year
elif peek DASH + NUM(1-2) → ISO extended
consume DASH, NUM(1-2) month, DASH, NUM(1-2) day
elif peek NUM(4) (MMDD compact) → ISO basic
split NUM into month(2) + day(2)
elif peek NUM(3) (DDD compact) → ISO basic ordinal
day-of-year = NUM
parse_time_part(tokens) (see below)
```
#### parse_month_leading — Mmm DD[,] YYYY time [TZ]
```
month = consume TOK_MONTH
day = consume NUM(1-2)
optional TOK_SUFFIX, TOK_COMMA
optional TOK_SPACE
year = consume NUM (any digit count for year)
parse_time_part(tokens)
```
This covers:
- `Jan 01 00:00:00 2000` (after time, year — legacy order variant)
- `January 1st, 2026 10:43:00 UTC`
- `Apr 6 2026 10:43:00 EST`
Note: The legacy `do_convtime()` format puts time before year:
`Mmm DD HH:MM:SS YYYY`. The unified parser needs to handle both
orders. After consuming month and day, if the next token looks like
a time (NUM:NUM), parse time then year. If it looks like a year
(4-digit NUM), parse year then time.
```
if peek NUM + TOK_COLON → time first, then year
parse_time_part(tokens)
year = consume NUM
else
year = consume NUM
parse_time_part(tokens)
```
#### parse_day_leading — DD Mmm YYYY time [TZ]
```
day = consume NUM(1-2)
optional TOK_SUFFIX
month = consume TOK_MONTH
optional TOK_COMMA
year = consume NUM
parse_time_part(tokens)
```
#### parse_time_part — HH:MM:SS[.frac] [TZ]
```
optional TOK_T or TOK_SPACE (separator)
hour = consume NUM(1-2)
consume TOK_COLON
minute = consume NUM(2)
optional:
consume TOK_COLON
second = consume NUM(2)
optional:
consume TOK_DOT
frac = consume NUM (1-9 digits, right-pad to 7 with zeros)
optional TOK_MERIDIAN (adjust hour: 12→0, add PM offset)
parse_timezone(tokens)
```
#### parse_timezone
```
if peek TOK_Z → UTC (offset = 0)
if peek TOK_TZ_NAME → named offset
if peek TOK_TZ_MIL → military offset
if peek TOK_PLUS or TOK_DASH → numeric offset
sign = consume
offset = consume NUM(4) as HHMM, or NUM(2) HH + COLON + NUM(2) MM
else → no timezone specified
```
## Sub-second Handling
Unified across all formats. After the seconds, if a `.` follows:
1. Consume up to 9 digits.
2. Right-pad with zeros to exactly 7 digits.
3. Store as hectonanoseconds (100ns units).
4. Split into milliseconds (digits 1-3), microseconds (4-6),
nanoseconds (7, ×100) for FIELDEDTIME.
This replaces both `ParseDecimalSeconds()` in `do_convtime()` and
the broken inline parser in the old `PD_GetFields()`.
## Supported Formats
### ISO 8601
```
2026-04-06T10:43:00Z
2026-04-06T10:43:00+0000
2026-04-06T10:43:00-07:00
2026-04-06T10:43:00.1234567Z
2026-04-06 10:43:00 UTC
20260406T104300Z
2026-096T10:43:00Z
2026W15-1T10:43:00Z
2026-04-06
-1605-120T12:34:56Z
```
### Legacy / Name-based
```
Jan 01 00:00:00 2000 convsecs() output
Wed Jun 24 10:22:54 1992 with day-of-week
Jun 24 10:22:54.123456 1992 with sub-seconds
April 6, 2026 10:43:00 UTC full month, comma
Apr 6 2026 10:43:00 EST year before time
January 1st 2000 00:00:00 Z ordinal suffix
6 Apr 2026 10:43:00 GMT European order
Mon, 6 Apr 2026 10:43:00 +0000 RFC 2822-like
Apr 6 2026 10:43:00pm Z AM/PM
```
### Explicitly Not Supported
```
04/06/2026 ambiguous (MM/DD vs DD/MM)
6.4.2026 ambiguous (DD.MM vs MM.DD)
Apr 6 26 ambiguous (year 26 vs day 26?)
```
## File Layout
```
mux/lib/date_scan.rl Ragel source (scanner)
mux/lib/date_scan.cpp Generated (read-only, chmod a-w)
mux/lib/date_parse.cpp Recursive descent parser
mux/lib/timeparser.cpp Deleted
mux/lib/timeutil.cpp do_convtime() removed
mux/include/timeutil.h ParseDate() signature unchanged or
replaced by date_parse()
```
Alternatively, the parser can live in the same `.rl` file after
the `%%write` block, keeping scanner and parser together.
## Testing
- `testcases/convtime_fn.mux` — legacy format tests (must still pass)
- `testcases/parsedate_fn.mux` — update with ISO + name-based tests
- All 998+ smoke tests must continue to pass
- The round-trip `convtime(convsecs(N, utc), utc) == N` must hold
for all valid N
## Migration
1. Write `date_scan.rl` (scanner) and parser.
2. Wire as the sole implementation of date parsing.
3. Remove `do_convtime()` and old `ParseDate()`.
4. Verify all smoke tests pass.
5. Update `help convtime` examples.
## Size Estimate
- Ragel scanner: ~150 lines of `.rl` (names are just alternations)
- Recursive descent parser: ~200 lines of C++
- Total: ~350 lines replacing ~1850 lines
(`do_convtime` ~130 + `ParseDate` ~1650 + helpers)

View file

@ -13,6 +13,7 @@ Do not hand-edit generated files in this repository. Edit the source input and r
| `mux/modules/engine/art_scan.cpp` | `mux/modules/engine/art_scan.rl` | `ragel -G2` |
| `mux/modules/engine/ast_scan.cpp` | `mux/modules/engine/ast_scan.rl` | `ragel -G2` |
| `mux/lib/color_ops.c` | `mux/lib/color_ops.rl` | `ragel -G2 -C` |
| `mux/lib/date_scan.cpp` | `mux/lib/date_scan.rl` | `ragel -G2` |
| `mux/muxescape/muxescape.cpp` | `mux/muxescape/muxescape.rl` | `ragel -G2` |
| `mux/include/utf8tables.h` | Unicode inputs under `utf/` | `utf/` pipeline / `make` |
| `mux/lib/utf8tables.cpp` | Unicode inputs under `utf/` | `utf/` pipeline / `make` |

View file

@ -12,7 +12,7 @@ AM_CFLAGS = -O2 -Wall -Wextra -Wno-implicit-fallthrough -std=c11
LIBMUX_SRC = libmux.cpp \
sha1.cpp svdrand.cpp svdhash.cpp \
timeutil.cpp timeabsolute.cpp timedelta.cpp timeparser.cpp timezone.cpp \
timeutil.cpp timeabsolute.cpp timedelta.cpp date_scan.cpp timezone.cpp \
utf8_collate.cpp utf8_grapheme.cpp utf8_normalize.cpp utf8tables.cpp \
strtod.cpp alarm.cpp mathutil.cpp stringutil.cpp alloc.cpp dbutil.cpp

View file

@ -269,7 +269,7 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -I$(top_srcdir)/ganl/include -I$(top_srcdi
AM_CFLAGS = -O2 -Wall -Wextra -Wno-implicit-fallthrough -std=c11
LIBMUX_SRC = libmux.cpp \
sha1.cpp svdrand.cpp svdhash.cpp \
timeutil.cpp timeabsolute.cpp timedelta.cpp timeparser.cpp timezone.cpp \
timeutil.cpp timeabsolute.cpp timedelta.cpp date_scan.cpp timezone.cpp \
utf8_collate.cpp utf8_grapheme.cpp utf8_normalize.cpp utf8tables.cpp \
strtod.cpp alarm.cpp mathutil.cpp stringutil.cpp alloc.cpp dbutil.cpp
@ -488,13 +488,17 @@ all-local: libmux.so
-include $(LIBMUX_CXX_OBJS:.lo=.d)
-include $(LIBMUX_C_OBJS:.lo=.d)
# Ragel .rl -> .c
# Ragel .rl -> .c — output is read-only to prevent accidental edits.
# Ragel .rl -> .c/.cpp — output is read-only to prevent accidental edits.
color_ops.c: color_ops.rl $(top_srcdir)/include/color_ops.h
chmod u+w $@ 2>/dev/null || true
ragel -G2 -C -o $@ $<
chmod a-w $@
date_scan.cpp: date_scan.rl $(top_srcdir)/include/timeutil.h
chmod u+w $@ 2>/dev/null || true
ragel -G2 -o $@ $<
chmod a-w $@
libmux.so: $(LIBMUX_OBJS)
$(CXX) $(CXXFLAGS) $(DYNAMICLIB_CXXFLAGS) -Wl,-soname,libmux.so -o $@ $(LIBMUX_OBJS) $(LIBS) $(DL_LIB) $(SQL_LIBS)

2508
mux/lib/date_scan.cpp Normal file

File diff suppressed because it is too large Load diff

1277
mux/lib/date_scan.rl Normal file

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff