Commit Graph
157 Commits
Author SHA1 Message Date
CCLUCKY 84e09c0c00 fix(parser): keep PaddleOCR figure regions as media items (#21452) 2026-10-29 20:21:45 +08:00
天海蒼灆 869e4d5cf1 feat(parser): parse MinerU 4 output and verify the endpoint (#21625) 2026-10-29 18:45:54 +08:00
jay77721 5b397cecd2 feat(parser): hand off EPUB images as media refs (#21677) 2026-10-29 17:20:44 +08:00
xugangqiang 7402f3df1d fix(parser): drop EE-specific references from OSS parse-method comment (#21674) 2026-10-29 15:30:34 +08:00
jay77721 4b69a7f41c fix(parser): preserve Markdown image references in chunk text (#21608) 2026-10-29 11:05:48 +08:00
Taranum Wasu 863304fe78 fix(parser): emit page positions for SoMark JSON blocks (#21324) 2026-10-29 11:03:50 +08:00
Lukas 2d4f2796b2 fix(parser): recognize a Table of Contents heading when removing a TOC (#21331) 2026-10-28 21:29:49 +08:00
jay77721 d697b0410a feat(parser): add media references for on-demand image materialization (#21561) 2026-10-28 19:47:53 +08:00
CCLUCKY de59ebf378 fix(parser): honour the pipeline context in SoMark and Docling PDF calls (#21536) 2026-10-28 15:52:16 +08:00
CCLUCKY 0c8af782cb fix(parser): read PDF bookmarks into remote parse results (#21499) 2026-10-28 15:28:30 +08:00
CCLUCKY aea8be4777 fix(parser): apply the OpenDataLoader request timeout (#21527) 2026-10-28 12:44:02 +08:00
CCLUCKY 2eae6eb63b fix(parser): carry OpenDataLoader PDF positions through to parse items (#21450) 2026-10-28 10:44:39 +08:00
Taranum Wasu acd6d887a5 fix(parser): emit Docling page positions and real page_count (#21330) 2026-10-28 09:55:28 +08:00
jay77721 6e2214aaa8 fix(parser): preserve table structure across document formats (#21504) 2026-10-27 16:54:40 +08:00
jay77721 3e313daf6f fix(ingestion): support legacy .xls BIFF8 as tag source (#21483)
## Summary
Tag files with `.xls` extension were accepted by the extension check in
`parseTagSourceByFilename` but always produced an empty tag index:
`parseXLSXTagSource` used only `excelize.OpenReader`, which rejects OLE
BIFF workbooks (`unsupported workbook file format`). This makes legacy
Excel 97-2003 tag files unusable while the docs claim `XLSX, XLS, CSV,
TXT` support.

This PR routes OLE-magic `.xls` bytes to a new BIFF8 reader
(`go-excel-reader`, already used by the document `XLSParser`), applying
the identical per-row rule as `.xlsx`.

## Related Issue
None (follow-up of #10602 / #10660 which fixed document-side `.xls`; tag
side was left behind).

## Changes
- `internal/ingestion/component/tag_source.go`
- Detect OLE compound-document magic and route to new
`parseBIFF8TagSource` (fallback to `excelize` for OOXML-renamed `.xls`).
- `parseBIFF8TagSource`: per-sheet, no-header, first two non-empty cells
form `(content, tags)`; single-cell rows accumulate; extra cells
ignored.
- Extract shared `accumulateTagCells` helper (removes xlsx/BIFF8 loop
duplication); simplify `tagCellString` (drop redundant `string` case).
  - Fix `unsupported extension` error text to include `.xls`.
- `internal/ingestion/component/tag_biff8_test.go` (new):
`TestParseBIFF8TagSource` with an xlwt-generated BIFF8 fixture (3
examples incl. single-cell accumulation).

## Test Plan
- `bash build.sh --test ./internal/ingestion/component/ -run
'TestParseBIFF8TagSource|TestParseTagSourceByFilename$|TestParseXLSXTagSource'
-v` — all PASS.
- `bash build.sh --test ./internal/ingestion/component/` — full package
PASS (~16s).
- Real-file check: document `XLSParser` parses 4 genuine `.xls` fixtures
(`produced=true`); same bytes via tag path previously returned
`count=0`, now return 3 tag examples.
2026-10-26 17:34:59 +08:00
jay77721 b5e8a0a2c3 refactor(parser): carry spreadsheet merge geometry through chunkers (#21463) 2026-10-26 14:23:47 +08:00
xugangqiang 0410c4a266 refactor(parser): drop ParseResult.OutputFormat/Markdown, gate on Produced() (#21414) 2026-10-23 19:11:10 +08:00
jay77721 fd675ebde2 fix(deepdoc): preserve complete PDF figures (#21390) 2026-10-23 17:06:46 +08:00
xugangqiang fced7746a3 fix(parser): make IsPDFParseMethod recognize registered external analyzer methods (#21383) 2026-10-23 14:26:06 +08:00
jay77721 17f173c288 refactor(parser): remove dead output_format readers (#21336) 2026-10-22 17:40:41 +08:00
xugangqiangandCodeBuddy Code fa0bae42b0 refactor(parser): remove dead ParseResult.Text field and text output path (#21334)
## Summary

The DSL no longer supports `output_format` selection.
`normalizeParserJSON` (in `internal/ingestion/component/parser.go`)
hardcodes `setup["output_format"] = "json"` for every parser setup, so
the `ParseResult.Text` field — populated only on the now-unreachable
text output path — is dead.

This change removes `ParseResult.Text` and every read/write site:

- `internal/parser/parser/parse_result.go`: drop the `Text` field and
update payload-family helpers (`isPayloadZero`, `payloadFamily`, etc.).
- `internal/parser/parser/email_parser.go` / `doc_parser.go`: drop the
text-output branches and `output_format:"text"` handling.
- `internal/ingestion/component/parser.go`: the normalization already
forces `json`; remove the text branch in `normalizeParserJSON` /
`buildParserOutputs`.
- `internal/service/file/file_content.go`: `parseResultText` no longer
reads `res.Text`; it falls back to `res.Markdown` for legacy
markdown-only payloads.
- `internal/service/document/document_artifact.go`: DOC preview now
reads preview text from the JSON items.
- `internal/parser/parser/pdf_result.go`: drop the dead `case
result.Text != ""` fallback in `NormalizeResult` (no PDF producer sets
`parser.ParseResult.Text`; confirmed by grep).

## Test plan

- Updated / removed obsolete text-path unit tests; added guard tests
(`text_field_removed_test.go`, `text_field_removed_doc_test.go`)
asserting the field is gone and JSON is the only output family.
- `bash build.sh --test ./internal/parser/parser/...
./internal/ingestion/component/... ./internal/service/file/...
./internal/service/document/...` — all green.
- Full `bash build.sh --test` regression — all green (no FAIL).
- `bash build.sh --go` builds both `ragflow-cli` and `ragflow_server`.

## Notes

`deepdoctype.ParseResult` (a different struct in
`internal/deepdoc/parser/type`) still has its own `Text` / `Sections`
fields; those are unrelated and intentionally untouched.

---------

Co-authored-by: CodeBuddy Code <noreply@codebuddy.ai>
2026-10-22 16:23:48 +08:00
Lukas 4969b7b41a fix(parser): place a table's footer rows after its body (#21283)
### Summary

`tableutil.ParseTableHTML` builds table rows in source order. HTML 4
required `<tfoot>` to come before `<tbody>`, and browsers draw it below
the body wherever it stands. Tables written in that order are still
common: generated reports, documentation tools, older CMS exports. For
those tables the footer row landed between the header and the data:

```html
<table>
  <thead><tr><th>Item</th><th>Qty</th></tr></thead>
  <tfoot><tr><th>Total</th><th>3</th></tr></tfoot>
  <tbody><tr><td>pen</td><td>1</td></tr><tr><td>ink</td><td>2</td></tr></tbody>
</table>
```

```text
main:         Item | Qty / Total | 3 / pen | 1 / ink | 2    HeaderRows = 2
this branch:  Item | Qty / pen | 1 / ink | 2 / Total | 3    HeaderRows = 1
```

A footer of `<th>` cells was also counted as a second header row,
because the header count is the leading run of `<th>` rows.

### What changed

`ParseTableHTML` collects the rows of a `<tfoot>` apart and appends them
after the body. Their `<th>` flags go with them, so they no longer
extend the header. Tables without a `<tfoot>` are unchanged. Nested
tables are still folded into their cell by `cellText`, as before.

The HTML parser's table items, the chunker and the deepdoc converter all
build their grids through `ParseTableHTML` / `BuildTableData`, so they
all get the rendered order.

### Testing

`TestParseTableHTMLFooterRowsComeLast` uses the table above and expects
the rows in rendered order with one header row. It fails on `main` with
`HeaderRows = 2`.

```
CGO_ENABLED=0 go test ./internal/parser/...                    # ok (chunk, parser, tableutil)
TIKTOKEN_CACHE_DIR=… go test ./internal/ingestion/component/chunker/ ./internal/service/file/
# chunker: only TestPaddleOCRHighlightsThroughPipeline fails, identically on main (local environment)
go vet ./internal/parser/tableutil/ && gofmt -l internal/parser/   # clean
```
2026-10-22 15:15:07 +08:00
Lukas 8e31338dd2 fix(parser): leave out the content of <template> elements (#21284) 2026-10-22 14:38:16 +08:00
jay77721 7aac10a22b refactor(ingestion): remove all ck_type tombstones (#21317) 2026-10-22 14:34:28 +08:00
jay77721 d4dd1eeb28 fix(parser): normalize inline image base64 before chunk upload (#21286) 2026-10-21 19:22:51 +08:00
jay77721 93b314f306 fix(chunker): preserve list-form PDF positions in title chunkers (#21258) 2026-10-21 18:41:31 +08:00
jay77721 32f0114c95 refactor(parser): unify PDF JSON post-processing (#21228) 2026-10-21 17:46:25 +08:00
jay77721 11ca59d511 refactor(ingestion): retire ck_type in favor of layout and doc_type (#21259) 2026-10-21 17:01:20 +08:00
Lukas 7ee5174fc3 fix(parser): break EPUB text at the end of every block element (#21104) 2026-10-21 15:36:55 +08:00
xugangqiang 3a4cef0b77 fix(parser): merge .doc header-row table fragments split by office_oxide (#21222) 2026-10-20 16:08:26 +08:00
jay77721 f402c001be fix(parser): preserve PaddleOCR PDF highlight positions (#21205) 2026-10-20 15:46:31 +08:00
Xinliang Zheng ab85152d0f fix(parser): use selected tenant SoMark and OpenDataLoader instances (#21112) 2026-10-20 13:39:49 +08:00
xugangqiang 4c036bd750 fix(parser): caption DOCX tables with their heading hierarchy (#21202) 2026-10-20 13:37:29 +08:00
xugangqiang 404950b415 fix(parser): emit DOC tables as structured items instead of plain text (#21201) 2026-10-20 13:37:00 +08:00
xugangqiang 5f1d452d63 feat(parser): add per-method external DeepDoc analyzer dispatch (#21182) 2026-10-20 13:36:04 +08:00
Sourav Rajvi 1b6c087b39 fix(chunker): group DOCX paragraphs into custom-delimiter records (#20713) 2026-10-20 13:27:30 +08:00
Lukas 781397b3ba fix(parser): start a new line before every HTML block element (#21107) 2026-10-20 11:20:31 +08:00
Lukas 0cdbe73d4d fix(parser): read text and inline elements under <body> as one item (#21108) 2026-10-19 19:17:51 +08:00
Lukas 6488930b70 fix(parser): decode numeric and all named character references in EPUB text (#21103) 2026-10-19 16:02:35 +08:00
Lukas 4c2b098c93 fix(parser): keep JSON numbers and <, >, & as written in JSON items (#21101) 2026-10-19 15:24:03 +08:00
Lukas fbae305f3e fix(parser): write an email's text output in a fixed field order (#21100) 2026-10-19 15:16:59 +08:00
xugangqiang 5cb4abb40a Remove RowsToHTML; render PDF tables via canonical RenderTableHTML (#21083) 2026-10-16 22:08:09 +08:00
xugangqiang b073eaabc0 refactor(parser): introduce typed Position API for DocItem positions (#21049) 2026-10-16 22:05:17 +08:00
jay77721 a2f22bebb5 fix(parser): grow truncated rows to host inherited merge values (#21044) 2026-10-16 12:29:39 +08:00
xugangqiang eabad0c74c refactor(parser): type ParseResult.JSON as []DocItem boundary adapter (#20998) 2026-10-15 20:15:40 +08:00
jay77721 89a7734408 fix(parser): inherit merged data cells, strip invisible chars, trim dataless columns (#20996) 2026-10-15 19:17:53 +08:00
xugangqiang 66f15c779d Remove dead ParseResult.HTML field and redundant EmailParser json-path Text (#20992) 2026-10-15 19:15:50 +08:00
Taranum Wasu 4dfcab718d fix(parser): forward caller ctx through PDF TCADP path (#17673) 2026-10-15 19:09:30 +08:00
xugangqiang 9fe7fec2c6 Render table chunks as HTML on Text at the chunker (#20918) 2026-10-15 09:58:38 +08:00
xugangqiang 5d35f968b0 refactor(parser): bake merged-cell geometry into the structured TableData grid (#20832) 2026-10-13 13:50:21 +08:00