mirror of
https://github.com/brazilofmux/tinymux
synced 2026-08-13 00:23:11 -04:00
Replace ParseDate with unified Ragel -G2 scanner + recursive descent parser
Delete the 1650-line multi-pass constraint-satisfaction engine (timeparser.cpp) and replace it with date_scan.rl: a Ragel -G2 scanner that tokenizes date strings into a flat token array, plus a small recursive descent parser that dispatches on the leading token. The scanner handles month names, day-of-week names, timezone abbreviations, military timezone letters, meridian (AM/PM), ordinal suffixes, and numbers — all as DFA alternations with no hash tables or linear searches. The parser dispatches on the first non-whitespace token: 4+ digit number → ISO 8601 family (extended, basic, ordinal, week) Month name → US order (Mmm DD [YYYY] HH:MM:SS / legacy) 1-2 digit + Month → European order (DD Mmm YYYY HH:MM:SS) Day-of-week → strip prefix, re-dispatch This is a single code path — always exercised, always tested. The old architecture had two independent parsers (do_convtime + ParseDate) where one could be broken without anyone noticing. Sub-second precision: unified handling across all formats, accumulating up to 7 fractional digits as hectonanoseconds (100ns), matching the internal FACTOR_100NS_PER_SECOND resolution. All 998 smoke tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
a96448cbbd
commit
4884ba3dba
7 changed files with 4076 additions and 1657 deletions
282
docs/design-date-parser.md
Normal file
282
docs/design-date-parser.md
Normal file
|
|
@ -0,0 +1,282 @@
|
|||
# Design: Unified Date Parser
|
||||
|
||||
## Status
|
||||
|
||||
Proposed — replaces both `do_convtime()` and `ParseDate()` with a
|
||||
single Ragel -G2 scanner + recursive descent parser.
|
||||
|
||||
## Background
|
||||
|
||||
`convtime()` currently has two independent code paths:
|
||||
|
||||
1. **`do_convtime()`** — Hand-written parser for the legacy format
|
||||
`[Ddd] Mmm DD HH:MM:SS[.frac] YYYY`. Works correctly.
|
||||
|
||||
2. **`ParseDate()`** — 1650-line multi-pass constraint-satisfaction
|
||||
engine. Was completely broken from 2.13 through 2.14 due to a
|
||||
`mux_min(int,int)` truncating `size_t`. Nobody noticed for months.
|
||||
|
||||
Having two parsers means two sub-second algorithms, two sets of
|
||||
validation rules, two things to maintain, and the ability for one to
|
||||
be broken without anyone knowing. The fix is to make everything
|
||||
one thing.
|
||||
|
||||
## Design Principle
|
||||
|
||||
One parser, always used, always tested. If it breaks, every
|
||||
`convsecs()`/`convtime()` round-trip fails and 900+ smoke tests
|
||||
scream. No silent fallback to a different code path.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
convtime(input, [zone], [precision])
|
||||
→ date_parse(input) → FIELDEDTIME + zone info
|
||||
→ apply timezone logic
|
||||
→ return epoch string
|
||||
```
|
||||
|
||||
`date_parse()` is the unified entry point. It replaces both
|
||||
`lta.SetString()` (which called `do_convtime()`) and `ParseDate()`.
|
||||
|
||||
### Ragel -G2 Scanner
|
||||
|
||||
The scanner tokenizes the input into a flat array of typed tokens:
|
||||
|
||||
```c
|
||||
enum DateToken {
|
||||
TOK_EOF,
|
||||
TOK_NUM, // sequence of digits, value + digit count stored
|
||||
TOK_MONTH, // Jan..Dec / January..December → 1..12
|
||||
TOK_DOW, // Sun..Sat / Sunday..Saturday → 0..6
|
||||
TOK_TZ_NAME, // UTC, GMT, EST, PST, CST, MST, EDT, PDT, CDT, MDT,
|
||||
// HST, AKST, AKDT, BST, IST, CET, CEST, EET, EEST,
|
||||
// AEST, AST, ADT, UT → offset in minutes
|
||||
TOK_TZ_MIL, // A..Z (except J) → offset in minutes
|
||||
TOK_MERIDIAN, // AM=0, PM=12
|
||||
TOK_SUFFIX, // st, nd, rd, th
|
||||
TOK_T, // letter T (ISO separator)
|
||||
TOK_W, // letter W (ISO week prefix)
|
||||
TOK_Z, // letter Z (UTC indicator)
|
||||
TOK_DASH, // -
|
||||
TOK_PLUS, // +
|
||||
TOK_COLON, // :
|
||||
TOK_DOT, // .
|
||||
TOK_COMMA, // ,
|
||||
TOK_SPACE, // whitespace (collapsed)
|
||||
};
|
||||
```
|
||||
|
||||
Ragel handles the name matching via DFA alternation. Month names
|
||||
(24 alternations), day-of-week (14), timezone abbreviations (~25),
|
||||
and meridian (2) compile into a single goto-driven state machine.
|
||||
This is a jump table — faster than the old `ParseThreeLetters()`
|
||||
hash comparison or the `PD_TextTable` linear scan.
|
||||
|
||||
Numbers carry their digit count, which the parser uses for
|
||||
disambiguation (1-2 digits = day/hour/minute/second, 3 digits =
|
||||
day-of-year, 4 digits = year, 5+ digits = extended year).
|
||||
|
||||
### Recursive Descent Parser
|
||||
|
||||
After scanning, the parser dispatches on the leading token(s):
|
||||
|
||||
```
|
||||
parse_datetime(tokens):
|
||||
skip optional TOK_DOW [TOK_COMMA] [TOK_SPACE]
|
||||
|
||||
if peek TOK_NUM(4+ digits) or TOK_PLUS or TOK_DASH(followed by digits(4+))
|
||||
→ parse_iso(tokens)
|
||||
|
||||
if peek TOK_MONTH
|
||||
→ parse_month_leading(tokens)
|
||||
|
||||
if peek TOK_NUM(1-2 digits) and peek+1 is TOK_MONTH
|
||||
→ parse_day_leading(tokens)
|
||||
|
||||
→ fail
|
||||
```
|
||||
|
||||
Each handler is a straight left-to-right walk consuming expected
|
||||
tokens. No backtracking.
|
||||
|
||||
#### parse_iso — ISO 8601 family
|
||||
|
||||
```
|
||||
year = consume NUM(4+ digits)
|
||||
|
||||
if peek TOK_W → ISO week date
|
||||
consume W, NUM(2) week
|
||||
optional DASH, NUM(1) day-of-week
|
||||
elif peek DASH + NUM(3) → ISO ordinal
|
||||
consume DASH, NUM(3) day-of-year
|
||||
elif peek DASH + NUM(1-2) → ISO extended
|
||||
consume DASH, NUM(1-2) month, DASH, NUM(1-2) day
|
||||
elif peek NUM(4) (MMDD compact) → ISO basic
|
||||
split NUM into month(2) + day(2)
|
||||
elif peek NUM(3) (DDD compact) → ISO basic ordinal
|
||||
day-of-year = NUM
|
||||
|
||||
parse_time_part(tokens) (see below)
|
||||
```
|
||||
|
||||
#### parse_month_leading — Mmm DD[,] YYYY time [TZ]
|
||||
|
||||
```
|
||||
month = consume TOK_MONTH
|
||||
day = consume NUM(1-2)
|
||||
optional TOK_SUFFIX, TOK_COMMA
|
||||
optional TOK_SPACE
|
||||
year = consume NUM (any digit count for year)
|
||||
parse_time_part(tokens)
|
||||
```
|
||||
|
||||
This covers:
|
||||
- `Jan 01 00:00:00 2000` (after time, year — legacy order variant)
|
||||
- `January 1st, 2026 10:43:00 UTC`
|
||||
- `Apr 6 2026 10:43:00 EST`
|
||||
|
||||
Note: The legacy `do_convtime()` format puts time before year:
|
||||
`Mmm DD HH:MM:SS YYYY`. The unified parser needs to handle both
|
||||
orders. After consuming month and day, if the next token looks like
|
||||
a time (NUM:NUM), parse time then year. If it looks like a year
|
||||
(4-digit NUM), parse year then time.
|
||||
|
||||
```
|
||||
if peek NUM + TOK_COLON → time first, then year
|
||||
parse_time_part(tokens)
|
||||
year = consume NUM
|
||||
else
|
||||
year = consume NUM
|
||||
parse_time_part(tokens)
|
||||
```
|
||||
|
||||
#### parse_day_leading — DD Mmm YYYY time [TZ]
|
||||
|
||||
```
|
||||
day = consume NUM(1-2)
|
||||
optional TOK_SUFFIX
|
||||
month = consume TOK_MONTH
|
||||
optional TOK_COMMA
|
||||
year = consume NUM
|
||||
parse_time_part(tokens)
|
||||
```
|
||||
|
||||
#### parse_time_part — HH:MM:SS[.frac] [TZ]
|
||||
|
||||
```
|
||||
optional TOK_T or TOK_SPACE (separator)
|
||||
hour = consume NUM(1-2)
|
||||
consume TOK_COLON
|
||||
minute = consume NUM(2)
|
||||
optional:
|
||||
consume TOK_COLON
|
||||
second = consume NUM(2)
|
||||
optional:
|
||||
consume TOK_DOT
|
||||
frac = consume NUM (1-9 digits, right-pad to 7 with zeros)
|
||||
optional TOK_MERIDIAN (adjust hour: 12→0, add PM offset)
|
||||
parse_timezone(tokens)
|
||||
```
|
||||
|
||||
#### parse_timezone
|
||||
|
||||
```
|
||||
if peek TOK_Z → UTC (offset = 0)
|
||||
if peek TOK_TZ_NAME → named offset
|
||||
if peek TOK_TZ_MIL → military offset
|
||||
if peek TOK_PLUS or TOK_DASH → numeric offset
|
||||
sign = consume
|
||||
offset = consume NUM(4) as HHMM, or NUM(2) HH + COLON + NUM(2) MM
|
||||
else → no timezone specified
|
||||
```
|
||||
|
||||
## Sub-second Handling
|
||||
|
||||
Unified across all formats. After the seconds, if a `.` follows:
|
||||
|
||||
1. Consume up to 9 digits.
|
||||
2. Right-pad with zeros to exactly 7 digits.
|
||||
3. Store as hectonanoseconds (100ns units).
|
||||
4. Split into milliseconds (digits 1-3), microseconds (4-6),
|
||||
nanoseconds (7, ×100) for FIELDEDTIME.
|
||||
|
||||
This replaces both `ParseDecimalSeconds()` in `do_convtime()` and
|
||||
the broken inline parser in the old `PD_GetFields()`.
|
||||
|
||||
## Supported Formats
|
||||
|
||||
### ISO 8601
|
||||
|
||||
```
|
||||
2026-04-06T10:43:00Z
|
||||
2026-04-06T10:43:00+0000
|
||||
2026-04-06T10:43:00-07:00
|
||||
2026-04-06T10:43:00.1234567Z
|
||||
2026-04-06 10:43:00 UTC
|
||||
20260406T104300Z
|
||||
2026-096T10:43:00Z
|
||||
2026W15-1T10:43:00Z
|
||||
2026-04-06
|
||||
-1605-120T12:34:56Z
|
||||
```
|
||||
|
||||
### Legacy / Name-based
|
||||
|
||||
```
|
||||
Jan 01 00:00:00 2000 convsecs() output
|
||||
Wed Jun 24 10:22:54 1992 with day-of-week
|
||||
Jun 24 10:22:54.123456 1992 with sub-seconds
|
||||
April 6, 2026 10:43:00 UTC full month, comma
|
||||
Apr 6 2026 10:43:00 EST year before time
|
||||
January 1st 2000 00:00:00 Z ordinal suffix
|
||||
6 Apr 2026 10:43:00 GMT European order
|
||||
Mon, 6 Apr 2026 10:43:00 +0000 RFC 2822-like
|
||||
Apr 6 2026 10:43:00pm Z AM/PM
|
||||
```
|
||||
|
||||
### Explicitly Not Supported
|
||||
|
||||
```
|
||||
04/06/2026 ambiguous (MM/DD vs DD/MM)
|
||||
6.4.2026 ambiguous (DD.MM vs MM.DD)
|
||||
Apr 6 26 ambiguous (year 26 vs day 26?)
|
||||
```
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
mux/lib/date_scan.rl Ragel source (scanner)
|
||||
mux/lib/date_scan.cpp Generated (read-only, chmod a-w)
|
||||
mux/lib/date_parse.cpp Recursive descent parser
|
||||
mux/lib/timeparser.cpp Deleted
|
||||
mux/lib/timeutil.cpp do_convtime() removed
|
||||
mux/include/timeutil.h ParseDate() signature unchanged or
|
||||
replaced by date_parse()
|
||||
```
|
||||
|
||||
Alternatively, the parser can live in the same `.rl` file after
|
||||
the `%%write` block, keeping scanner and parser together.
|
||||
|
||||
## Testing
|
||||
|
||||
- `testcases/convtime_fn.mux` — legacy format tests (must still pass)
|
||||
- `testcases/parsedate_fn.mux` — update with ISO + name-based tests
|
||||
- All 998+ smoke tests must continue to pass
|
||||
- The round-trip `convtime(convsecs(N, utc), utc) == N` must hold
|
||||
for all valid N
|
||||
|
||||
## Migration
|
||||
|
||||
1. Write `date_scan.rl` (scanner) and parser.
|
||||
2. Wire as the sole implementation of date parsing.
|
||||
3. Remove `do_convtime()` and old `ParseDate()`.
|
||||
4. Verify all smoke tests pass.
|
||||
5. Update `help convtime` examples.
|
||||
|
||||
## Size Estimate
|
||||
|
||||
- Ragel scanner: ~150 lines of `.rl` (names are just alternations)
|
||||
- Recursive descent parser: ~200 lines of C++
|
||||
- Total: ~350 lines replacing ~1850 lines
|
||||
(`do_convtime` ~130 + `ParseDate` ~1650 + helpers)
|
||||
|
|
@ -13,6 +13,7 @@ Do not hand-edit generated files in this repository. Edit the source input and r
|
|||
| `mux/modules/engine/art_scan.cpp` | `mux/modules/engine/art_scan.rl` | `ragel -G2` |
|
||||
| `mux/modules/engine/ast_scan.cpp` | `mux/modules/engine/ast_scan.rl` | `ragel -G2` |
|
||||
| `mux/lib/color_ops.c` | `mux/lib/color_ops.rl` | `ragel -G2 -C` |
|
||||
| `mux/lib/date_scan.cpp` | `mux/lib/date_scan.rl` | `ragel -G2` |
|
||||
| `mux/muxescape/muxescape.cpp` | `mux/muxescape/muxescape.rl` | `ragel -G2` |
|
||||
| `mux/include/utf8tables.h` | Unicode inputs under `utf/` | `utf/` pipeline / `make` |
|
||||
| `mux/lib/utf8tables.cpp` | Unicode inputs under `utf/` | `utf/` pipeline / `make` |
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ AM_CFLAGS = -O2 -Wall -Wextra -Wno-implicit-fallthrough -std=c11
|
|||
|
||||
LIBMUX_SRC = libmux.cpp \
|
||||
sha1.cpp svdrand.cpp svdhash.cpp \
|
||||
timeutil.cpp timeabsolute.cpp timedelta.cpp timeparser.cpp timezone.cpp \
|
||||
timeutil.cpp timeabsolute.cpp timedelta.cpp date_scan.cpp timezone.cpp \
|
||||
utf8_collate.cpp utf8_grapheme.cpp utf8_normalize.cpp utf8tables.cpp \
|
||||
strtod.cpp alarm.cpp mathutil.cpp stringutil.cpp alloc.cpp dbutil.cpp
|
||||
|
||||
|
|
|
|||
|
|
@ -269,7 +269,7 @@ AM_CPPFLAGS = -I$(top_srcdir)/include -I$(top_srcdir)/ganl/include -I$(top_srcdi
|
|||
AM_CFLAGS = -O2 -Wall -Wextra -Wno-implicit-fallthrough -std=c11
|
||||
LIBMUX_SRC = libmux.cpp \
|
||||
sha1.cpp svdrand.cpp svdhash.cpp \
|
||||
timeutil.cpp timeabsolute.cpp timedelta.cpp timeparser.cpp timezone.cpp \
|
||||
timeutil.cpp timeabsolute.cpp timedelta.cpp date_scan.cpp timezone.cpp \
|
||||
utf8_collate.cpp utf8_grapheme.cpp utf8_normalize.cpp utf8tables.cpp \
|
||||
strtod.cpp alarm.cpp mathutil.cpp stringutil.cpp alloc.cpp dbutil.cpp
|
||||
|
||||
|
|
@ -488,13 +488,17 @@ all-local: libmux.so
|
|||
-include $(LIBMUX_CXX_OBJS:.lo=.d)
|
||||
-include $(LIBMUX_C_OBJS:.lo=.d)
|
||||
|
||||
# Ragel .rl -> .c
|
||||
# Ragel .rl -> .c — output is read-only to prevent accidental edits.
|
||||
# Ragel .rl -> .c/.cpp — output is read-only to prevent accidental edits.
|
||||
color_ops.c: color_ops.rl $(top_srcdir)/include/color_ops.h
|
||||
chmod u+w $@ 2>/dev/null || true
|
||||
ragel -G2 -C -o $@ $<
|
||||
chmod a-w $@
|
||||
|
||||
date_scan.cpp: date_scan.rl $(top_srcdir)/include/timeutil.h
|
||||
chmod u+w $@ 2>/dev/null || true
|
||||
ragel -G2 -o $@ $<
|
||||
chmod a-w $@
|
||||
|
||||
libmux.so: $(LIBMUX_OBJS)
|
||||
$(CXX) $(CXXFLAGS) $(DYNAMICLIB_CXXFLAGS) -Wl,-soname,libmux.so -o $@ $(LIBMUX_OBJS) $(LIBS) $(DL_LIB) $(SQL_LIBS)
|
||||
|
||||
|
|
|
|||
2508
mux/lib/date_scan.cpp
Normal file
2508
mux/lib/date_scan.cpp
Normal file
File diff suppressed because it is too large
Load diff
1277
mux/lib/date_scan.rl
Normal file
1277
mux/lib/date_scan.rl
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue