Fix flaky timestamp test that only passed in Asia/Shanghai TZ. Now parses and compares the instant instead of literal string.
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
Aligns three Go DLA / PDF post-processing behaviors with the Python `deepdoc` reference so the Go PDF pipeline matches Python's DLA region / annotation semantics.
- Add Go `EmailParser` support for Outlook `.msg` (OLE2/CFB) files via
the `gomsg` library, in addition to the existing `.eml` (RFC 5322)
support. The `.msg` hard-error is gone; emails with `.msg` attachments
are now ingested end-to-end.
- Re-chunk email attachments into retrievable text (user-oriented). Each
attachment is re-parsed by its file extension through the shared parser
registry and folded back into the same document, so attachment content
becomes searchable. This mirrors Python's legacy `rag/app/email.py`.
Binary attachments (images/audio/video/folders) are skipped by design.
- Restore a corrupted `sample.msg` test fixture and add guards so binary
fixtures are never mangled again (`.gitattributes` marks `*.msg` binary;
`check_files.py` skips NUL-byte files). Also made `check_files.py`
ruff-clean.
Adds a characterization / regression test
(`pdf_postprocess_measure_test.go`) that pins the **figure-aware reading
order** produced by the PDF adapter layer
(`internal/parser/parser/pdf_postprocess.go`).
Stop flattening `<table>` into a single text blob. A `<table>` now emits:
1. an inlined `doc_type_kwd:"text"` item keeping the `<table>…</table>`
markup (row/column structure survives for embedding/retrieval/LLM rendering),
2. a structured `doc_type_kwd:"table"` / `ck_type:"table"` item appended
after the walk, consumed by the downstream chunker.
Centralize the shared golden-doc + alignment helpers in `align_test.go` so the format-specific PRs (text&code, markdown golden, HTML) reuse one implementation instead of each carrying their own copy of the scaffolding.
Aligns the Go EML parser (`internal/parser/parser/email_parser.go`) with the Python flow parser (`rag/flow/parser/parser.py:_email`) on two structural points so the Go path is a faithful drop-in for the Python path on `.eml` inputs.
Restore the deduplication that was dropped when #17926 was merged.
`CompileDelimiterPatternList` now keeps a `seen` set and collapses
equivalent active entries (both backtick-inner and bare) into a single
alternation. This PR also removes the dead code that the re-review
surfaced.
`compileChildrenPattern` re-implemented the delimiter-list compile
inline with two divergences from the shared
`CompileDelimiterListPattern`:
- It never stripped backticks, so a backtick-wrapped
`children_delimiter` like `` `###` `` matched the **literal wrapped
token** rather than the inner `###`.
- It sorted by **byte length** instead of rune count (`sortSlice`), so
multi-byte delimiters could be ordered incorrectly and a longer
delimiter could fail to win over a shorter prefix.
Net effect: inline prose stays on one line (`Hello World`), real `<br>` boundaries survive (including before tags and repeated breaks), and source formatting whitespace no longer over-splits.
## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
- The `eng` flag passed to `removeTOCWord`/`removeContentsTable` was
hardcoded `false`, so English documents used the CJK 3-char TOC prefix
instead of Python's 2-word English prefix. This caused English tables of
contents to be under-deleted (left in indexed text) or over-deleted
(body paragraphs sharing the 3-char prefix dropped).
- `eng` is now derived from the parsed item content via `isEnglishItems`
(a port of Python `is_english`/`_is_english`), mirroring Python's
content heuristic.
## Changes
- `internal/parser/parser/text_toc.go`: add
`isEnglishItems`/`isEnglishTexts` (ASCII-ratio >80% ⇒ English,
fullmatch-anchored regex).
- `internal/parser/parser/docx_parser.go`: pass
`isEnglishItems(sections)` / `isEnglishItems(lineItems)` to
`removeTOCWord` (json + markdown paths).
- `internal/parser/parser/html_parser.go`: pass `isEnglishItems(items)`
to `removeContentsTable`.
- `docs/migration_python_go_diff.md`: close the 1.9/1.10 `eng` residual
and resolve the contradictory Parser 2.11 "Partially fixed" note.
## Notes
- `remove_toc` is a Parser-stage, DSL-configured feature (`remove_toc:
true/false` in the parser family setup). This change only fixes the
internal English/CJK prefix decision; it does not alter the pipeline or
the DSL contract. `remove_header_footer` (precise match) is unaffected.
- `eng` is auto-derived from content rather than exposed as a new config
knob, to stay faithful to Python behavior.
## Test plan
- `CGO_ENABLED=0 go test ./internal/parser/parser/ -run
'TestRemoveContentsTable|TestIsEnglishTexts|TestRemoveTOCWordEnglishDetection'`
- Added `TestIsEnglishTexts` (ASCII-ratio cases) and
`TestRemoveTOCWordEnglishDetection` (English TOC no longer over-deletes
"Chapter 2 Method").
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
xugangqiang
jay77721
Yash Raj Pandey
Harsh Kashyap
euvre
天海蒼灆
buua436
Taranum Wasu
Lynn
Jin Hai
Sandkiloconnect[bot]
Wang Qi