mtfyun/ragflow
1
0 Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-11-10 13:24:05 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,914 Commits 3 Branches 191 Tags
main
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
jay77721 70b09ce5af fix(chunk): replace chunk image on edit instead of appending (#22175) 2026-11-09 19:18:32 +08:00
jay77721 2b4d628d51 refactor(ingestion): drop inert parser family setup keys (#22163) 2026-11-09 17:38:59 +08:00
jay77721 d4ff41a57d refactor(ingestion): remove retired parser config references (#22153) 2026-11-09 17:37:53 +08:00
jay77721 64bc9f3f40 refactor(ingestion): remove retired parser options (#22147) 2026-11-09 14:33:43 +08:00
jay77721 eeeda36046 refactor(ingestion): remove media flattening option (#22144) 2026-11-09 13:47:50 +08:00
jay77721 8ba216b1c1 fix(deepdoc): preserve missed PDF table rows and merged headers (#22073) 2026-11-06 18:03:00 +08:00
jay77721 3a6b6dca78 refactor(ingestion): centralize component names and identity checks (#22081) 2026-11-06 17:03:32 +08:00
jay77721 4d20ef45f6 docs: record name column widening in database change log (#22030) 2026-11-05 18:04:41 +08:00
jay77721 afd9a7f1da fix(parser): preserve Office table cell spans (#22012) 2026-11-05 15:41:17 +08:00
jay77721 6ece20514f fix(core): enforce consistent name and alias length limits (#22020) 2026-11-05 15:32:06 +08:00
jay77721 aed68a2697 fix(ingestion): preserve PDF sentences across page boundaries (#21982) 2026-11-05 14:49:43 +08:00
jay77721 b2cf72d714 fix(deepdoc): keep DLA-labeled body prose out of table captions (#21979) 2026-11-04 19:53:45 +08:00
jay77721 5b54cb0dad feat(ingestion): record pipeline version on operation logs (#21916) 2026-11-04 18:31:42 +08:00
jay77721 bb5437e693 perf(pdf): read page directives once per document (#21950) 2026-11-04 15:34:02 +08:00
jay77721 9ed193674c feat(ingestion): recognize image tables through the DeepDoc page pipeline (#21911) 2026-11-04 13:14:38 +08:00
jay77721 8d44dca39b fix(ingestion): share extractor concurrency across tag extraction (#21891) 2026-11-03 18:48:40 +08:00
jay77721 5dc7683aa4 fix(parser): detect Office content before selecting Go parsers (#21909) 2026-11-03 17:38:56 +08:00
jay77721 8e0ed2bc15 fix(chunker): align PDF child previews with source positions (#21853) 2026-11-02 19:17:20 +08:00
jay77721 0c7fb3d7d7 perf(chunker): coalesce PDF renders and share crop workers (#21850) 2026-11-02 17:46:55 +08:00
jay77721 c6c9df03a2 test(mediacache): cover real Kvrocks lifecycle and index boundary (#21851) 2026-11-02 16:10:16 +08:00
jay77721 08460530d5 feat(parser): hand off Markdown and HTML remote images as media refs (#21764) 2026-11-02 11:59:01 +08:00
jay77721 1191d500a6 fix(chunker): prevent PDF child chunks from overwriting each other (#21773) 2026-11-02 10:44:29 +08:00
jay77721 dce85b3043 fix(parser): keep recovered PDF TOC leader lines out of the body guard (#21768) 2026-10-30 20:05:23 +08:00
jay77721 41753ee5a1 fix(chunker): drop DOCX custom-delimiter record prefix (#21760) 2026-10-30 16:50:12 +08:00
jay77721 aaa8e05354 feat(parser): hand off Office images as media refs (#21700) 2026-10-30 16:48:51 +08:00
jay77721 9ba49c523e feat(media): harden run media cache lifecycle and limits (#21694) 2026-10-30 12:41:23 +08:00
jay77721 5b397cecd2 feat(parser): hand off EPUB images as media refs (#21677) 2026-10-29 17:20:44 +08:00
jay77721 fc766ce3d7 fix(parser): remove duplicate figures for extracted PDF tables (#21673) 2026-10-29 16:37:23 +08:00
jay77721 4b69a7f41c fix(parser): preserve Markdown image references in chunk text (#21608) 2026-10-29 11:05:48 +08:00
jay77721 d697b0410a feat(parser): add media references for on-demand image materialization (#21561) 2026-10-28 19:47:53 +08:00
jay77721 6e2214aaa8 fix(parser): preserve table structure across document formats (#21504) 2026-10-27 16:54:40 +08:00
jay77721 292db08b91 fix(preview): highlight all spreadsheet chunk ranges (#21511) 2026-10-27 14:22:02 +08:00
jay77721 1a6a382a2f fix(deepdoc): preserve table rows before cell assignment (#21489) 2026-10-27 11:55:17 +08:00
jay77721 3e313daf6f fix(ingestion): support legacy .xls BIFF8 as tag source (#21483) 
## Summary
Tag files with `.xls` extension were accepted by the extension check in
`parseTagSourceByFilename` but always produced an empty tag index:
`parseXLSXTagSource` used only `excelize.OpenReader`, which rejects OLE
BIFF workbooks (`unsupported workbook file format`). This makes legacy
Excel 97-2003 tag files unusable while the docs claim `XLSX, XLS, CSV,
TXT` support.

This PR routes OLE-magic `.xls` bytes to a new BIFF8 reader
(`go-excel-reader`, already used by the document `XLSParser`), applying
the identical per-row rule as `.xlsx`.

## Related Issue
None (follow-up of #10602 / #10660 which fixed document-side `.xls`; tag
side was left behind).

## Changes
- `internal/ingestion/component/tag_source.go`
- Detect OLE compound-document magic and route to new
`parseBIFF8TagSource` (fallback to `excelize` for OOXML-renamed `.xls`).
- `parseBIFF8TagSource`: per-sheet, no-header, first two non-empty cells
form `(content, tags)`; single-cell rows accumulate; extra cells
ignored.
- Extract shared `accumulateTagCells` helper (removes xlsx/BIFF8 loop
duplication); simplify `tagCellString` (drop redundant `string` case).
  - Fix `unsupported extension` error text to include `.xls`.
- `internal/ingestion/component/tag_biff8_test.go` (new):
`TestParseBIFF8TagSource` with an xlwt-generated BIFF8 fixture (3
examples incl. single-cell accumulation).

## Test Plan
- `bash build.sh --test ./internal/ingestion/component/ -run
'TestParseBIFF8TagSource|TestParseTagSourceByFilename$|TestParseXLSXTagSource'
-v` — all PASS.
- `bash build.sh --test ./internal/ingestion/component/` — full package
PASS (~16s).
- Real-file check: document `XLSParser` parses 4 genuine `.xls` fixtures
(`produced=true`); same bytes via tag path previously returned
`count=0`, now return 3 tag examples.
 2026-10-26 17:34:59 +08:00
jay77721 bea5b62faa fix(ingestion): contain process-fatal errors and bound parser work (#21480) 2026-10-26 17:32:47 +08:00
jay77721 b5e8a0a2c3 refactor(parser): carry spreadsheet merge geometry through chunkers (#21463) 2026-10-26 14:23:47 +08:00
jay77721 2fc2b8888d fix(document): persist parser models with the selected pipeline (#21418) 2026-10-26 13:34:44 +08:00
jay77721 914bcb0307 fix(deepdoc): resolve conflicting caption labels by confidence (#21415) 2026-10-26 11:18:27 +08:00
jay77721 1b8ab33c2c fix(deepdoc): keep native PDF text when OCR detection misses it (#21399) 2026-10-26 11:17:28 +08:00
jay77721 651a1c65fc fix(deepdoc): crop clean media previews without context bands (#21412) 2026-10-23 19:09:58 +08:00
jay77721 fd675ebde2 fix(deepdoc): preserve complete PDF figures (#21390) 2026-10-23 17:06:46 +08:00
jay77721 17f173c288 refactor(parser): remove dead output_format readers (#21336) 2026-10-22 17:40:41 +08:00
jay77721 c34e873824 fix(sdk): align Python SDK with RAGFlow 1.0 dataset API (#21319) 2026-10-22 15:45:08 +08:00
jay77721 9cb5539c01 docs(api): fix dataset parse_type examples and selection rules (#21333) 2026-10-22 15:14:58 +08:00
jay77721 7aac10a22b refactor(ingestion): remove all ck_type tombstones (#21317) 2026-10-22 14:34:28 +08:00
jay77721 d4dd1eeb28 fix(parser): normalize inline image base64 before chunk upload (#21286) 2026-10-21 19:22:51 +08:00
jay77721 93b314f306 fix(chunker): preserve list-form PDF positions in title chunkers (#21258) 2026-10-21 18:41:31 +08:00
jay77721 32f0114c95 refactor(parser): unify PDF JSON post-processing (#21228) 2026-10-21 17:46:25 +08:00
jay77721 11ca59d511 refactor(ingestion): retire ck_type in favor of layout and doc_type (#21259) 2026-10-21 17:01:20 +08:00
jay77721 b84cadea97 fix(ingestion): block video parse while keeping visual upload routing (#21245) 2026-10-21 16:16:39 +08:00
jay77721 18d3da2756 refactor(pipeline): remove dead output_format from ingestion templates (#21266) 2026-10-21 13:57:02 +08:00
jay77721 ac6876d9e4 fix(ingestion): migrate composite PaddleOCR layout selectors (#21236) 2026-10-20 17:30:09 +08:00
jay77721 f402c001be fix(parser): preserve PaddleOCR PDF highlight positions (#21205) 2026-10-20 15:46:31 +08:00
jay77721 f42999e01a fix(web): correct PaddleOCR provider links to AI Studio (#21196) 2026-10-20 11:10:50 +08:00
jay77721 cfd41e569f feat(pipeline): support image and email in general template (#21072) 2026-10-19 14:37:44 +08:00
jay77721 1d2d049702 fix(ingestion): document and validate extractor timeout env (#21119) 2026-10-19 13:07:23 +08:00
jay77721 0ad97e70af fix(chunker): keep structured tables and images in the title family (#21040) 2026-10-16 18:37:03 +08:00
jay77721 a2f22bebb5 fix(parser): grow truncated rows to host inherited merge values (#21044) 2026-10-16 12:29:39 +08:00
jay77721 78faa36bd6 fix(ingestion): scope extractor timeout to single LLM call (#21003) 2026-10-16 09:44:39 +08:00
jay77721 89a7734408 fix(parser): inherit merged data cells, strip invisible chars, trim dataless columns (#20996) 2026-10-15 19:17:53 +08:00
jay77721 5753d7b27f fix(parser): keep short-line body pages out of PDF TOC removal (#20988) 2026-10-15 19:06:26 +08:00
jay77721 b859e0460e fix(dao): migrate Python-era flat ingestion config onto pipeline components (#20986) 2026-10-15 19:03:57 +08:00
jay77721 9f94f13cd4 fix(chunker): keep ancestor headings on every cap-split hierarchy piece (#20953) 2026-10-15 14:02:08 +08:00
jay77721 a2a629e972 fix(chunker): stop charging the replicated table header against the split budget (#20845) 2026-10-15 11:43:27 +08:00
jay77721 09ef155b0c fix(web): filter empty metadata values in dataset meta_fields count (#20933) 2026-10-14 16:34:54 +08:00
jay77721 a93f416558 fix(ingestion): report knowledge compiler progress fractions (#20902) 2026-10-13 21:09:28 +08:00
jay77721 8efbe73da0 fix(chunker): preserve whole lines in PDF text previews (#20890) 2026-10-13 16:54:19 +08:00
jay77721 c80d15b37c fix(ingestion): fetch a PDF source before image admission in VLM enhancement (#20846) 2026-10-13 16:06:34 +08:00
jay77721 db125c7188 fix(syncer): fail closed on non-positive connector refresh freq (#20871) 2026-10-13 15:30:24 +08:00
jay77721 deb1235be4 feat(parser): require a text source for images when vision enhancement is off (#20839) 2026-10-12 20:18:01 +08:00
jay77721 0ae367476b fix(chunker): slice General split positions by visible-text ratio (#20827) 2026-10-12 17:49:13 +08:00
jay77721 ccb78ef37d fix(ingestion): preserve spreadsheet identity in rebuilt chunks (#20825) 2026-10-12 17:37:39 +08:00
jay77721 030d08ac5d fix(deepdoc): keep an attached figure caption on its own line (#20819) 2026-10-12 17:31:45 +08:00
jay77721 5fc413b957 fix(ingestion): preserve typed metadata during merges (#20808) 2026-10-12 17:14:26 +08:00
jay77721 845e94d701 fix(chunker): preserve PDF and Markdown sentence delimiters (#20812) 2026-10-12 16:51:05 +08:00
jay77721 35de240c39 feat(ingestion): add progress fraction reporting to VisionEnhancement and cropImageChunks (#20800) 2026-10-12 13:47:54 +08:00
jay77721 877138c98e fix(web): restrict spreadsheet parser methods to supported options (#20749) 2026-10-12 10:29:06 +08:00
jay77721 bb18560be8 fix(chunker): preserve punctuation in general TXT and DOCX chunks (#20751) 2026-10-12 10:15:25 +08:00
jay77721 95df031cb0 refactor(parser): split image OCR into an independent switch (#20683) 2026-10-12 10:12:44 +08:00
jay77721 829aeb2f29 fix(deepdoc): stop PDF caption merge from dropping text and figures (#20650) 2026-10-10 13:34:48 +08:00
jay77721 e8f7f97237 fix(ingestion): persist and honor global vision settings (#20474) 2026-10-09 13:15:48 +08:00
jay77721 4071451ea4 refactor(deps): use Go dependency downloader as sole entrypoint (#20360) 2026-09-30 18:02:03 +08:00
jay77721 285b2c2903 fix(parser): restore picture OCR independently of VLM enhancement (#20424) 2026-09-30 14:32:48 +08:00
jay77721 6304825676 fix(parser): preserve DOCX XML character references (#20359) 2026-09-30 12:03:58 +08:00
jay77721 83b8e2ca24 fix(document): clone parser_config to break dataset aliasing; drop dead parser-config helpers (#20381) 2026-09-29 19:05:42 +08:00
jay77721 31fd92c2fa feat(parser): add opt-in VLM enhancement for images (#20166) 2026-09-29 15:54:28 +08:00
jay77721 9b67a4112a refactor(parser): unify spreadsheet tables on a segmented HTML wire (#20358) 2026-09-29 15:00:46 +08:00
jay77721 78154e1e28 fix(ingestion): read Python logs in Go backend (#20214) 2026-09-28 13:06:54 +08:00
jay77721 2be666b44a fix(router): accept beta tokens on document image routes for shared chats (#20185) 2026-09-24 19:20:52 +08:00
jay77721 b227b950db fix(dataset): read document-declared tag sources in tag aggregation (#20119) 2026-09-24 11:50:13 +08:00
jay77721 efd15d1851 fix(chunk): support image updates on the chunk PATCH endpoint (#20107) 2026-09-23 19:58:30 +08:00
jay77721 328fadf1f8 fix(parser): replace slog.Debug with common.Debug+zap in toc_header_footer (#20071) 2026-09-23 10:16:00 +08:00
jay77721 83635b5ad6 refactor(parser): drop unreachable headerColEnd fallbacks from XLSX table items (#20053) 2026-09-23 09:45:59 +08:00
jay77721 dae8016532 fix(parser): catch tight page numbers and site-promo headers/footers (#20019) 2026-09-23 09:45:34 +08:00
jay77721 ee5c314c14 fix(ingestion): derive component param whitelist from component schemas (#20060) 2026-09-23 09:43:10 +08:00
jay77721 a4c0fc8e76 fix(document): route image and presentation uploads to their dedicated parsers (#20061) 2026-09-23 09:41:49 +08:00
jay77721 338703565a refactor(go): unify Go logging on the zap logger (#20026) 2026-09-22 19:40:08 +08:00
jay77721 82b45422e0 fix(ingestion, web): converge cancels fast and stop logging them as errors (#19992) 2026-09-22 14:02:34 +08:00
jay77721 83c47d0251 fix(parser): route UI "Plain Text" PDF method to the plain-text parser (#19979) 2026-09-21 17:32:36 +08:00
jay77721 12e6d888db feat(ingestion): report sub-component parsing progress instead of a frozen 20% (#19965) 2026-09-21 17:26:07 +08:00