mtfyun/ragflow
1
0 Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-11-12 06:10:52 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,605 Commits 3 Branches 194 Tags
ef84683084cbf94bc616db8efec00182827446de
Commit Graph
98 Commits
Author SHA1 Message Date
jay77721 5b397cecd2 feat(parser): hand off EPUB images as media refs (#21677) 2026-10-29 17:20:44 +08:00
jay77721 d697b0410a feat(parser): add media references for on-demand image materialization (#21561) 2026-10-28 19:47:53 +08:00
buua436 ad5c331c0b fix: remove system variable API routes (#21410) 2026-10-23 16:59:28 +08:00
Jin Hai 1d7744f944 CLI: model group (#21392) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-10-23 15:48:38 +08:00
Jin Hai b73f030931 CLI: add show model stats (#21376) 
```
SHOW MODEL STATS;
SHOW MODEL STATS DAYS 7; 
SHOW MODEL STATS FROM '2024-01-01' TO '2024-01-31';

SHOW MODEL STATS DAYS 7 'openai';                        -- provider
SHOW MODEL STATS DAYS 7 'openai' 'prod';                 -- provider + instance
SHOW MODEL STATS DAYS 7 'openai' 'prod' 'gpt-4o';        -- provider + instance + model

SHOW MODEL STATS DAYS 7 BY DATE;
SHOW MODEL STATS DAYS 7 BY MODEL;
SHOW MODEL STATS DAYS 7 BY INSTANCE;
SHOW MODEL STATS DAYS 7 BY PROVIDER;

SHOW MODEL STATS DAYS 30 'openai' BY DATE;
SHOW MODEL STATS DAYS 30 'openai' 'prod' BY MODEL;
SHOW MODEL STATS DAYS 30 'openai' 'prod' 'gpt-4o' BY DATE;
SHOW MODEL STATS FROM '2024-01-01' TO '2024-01-31' 'azure' BY INSTANCE;
SHOW MODEL STATS FROM '2024-01-01' TO '2024-01-31' BY PROVIDER;
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-10-23 13:29:40 +08:00
lemonzoey 102d719fd5 fix(cli): send the dataset_id filter and guard the memory lookup type assertion (#21193) 2026-10-21 18:12:37 +08:00
Jin Hai 3f3f588abe CLI: add show quota (#21243) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-10-20 18:54:51 +08:00
Wang Qi 863fee004c Refactor: Delete not used APIs (#20894) 2026-10-13 17:48:55 +08:00
xugangqiang b75e195f22 feat(parser): unify BuiltIn and Pipeline parser selection into one dropdown (#20603) 2026-10-09 13:25:09 +08:00
Jin Hai 950b0fa4a3 Go cli: refactor (#20249) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-09-29 14:20:09 +08:00
Jin Hai 4ba5479210 Go cli: refactor api command auth (#20247) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-09-29 14:20:02 +08:00
Jin Hai 204d82ac89 Go cli: refactor api command (#20239) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-09-29 14:19:57 +08:00
Jin Hai b63d4124e7 Go: refactor cli, remove getDatasetID (#20236) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-09-29 14:19:40 +08:00
Jin Hai 16de4bfcb6 Go CLI: set/show cpu/memory/concurrency (#20002) 
```
RAGFlow(api/default)> show cores;
ragflow-cli error: Get cores not implemented
RAGFlow(api/default)> set cores 1;
ragflow-cli error: Set cores not implemented
RAGFlow(api/default)> show memory;
ragflow-cli error: Get memory not implemented
RAGFlow(api/default)> set memory 1;
ragflow-cli error: Set memory not implemented
RAGFlow(api/default)> show concurrency;
ragflow-cli error: Get concurrency not implemented
RAGFlow(api/default)> show concurrency 1;
ragflow-cli error: Get concurrency not implemented
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-09-21 21:50:13 +08:00
qinling0210 25a7c14e4f fix(tag): rebuild tagging on a Go tag source index (#19985) 
### Summary

rebuild tagging on a Go tag source index

```
The following endpoints are removed in GO because there is no tag datatset inGO.
GET	/api/v1/datasets/:dataset_id/tags 
PUT	/api/v1/datasets/:dataset_id/tags 
DELETE	/api/v1/datasets/:dataset_id/tags 
```
 2026-09-21 19:23:29 +08:00
Jin Hai 84b7608668 Go CLI: fix benchmark (#19946) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-09-21 14:45:22 +08:00
Wang Qi fe501ebb04 Refactor: move chat message/reference to dedicated tables (#19577) 2026-09-15 15:52:12 +08:00
qinling0210 aa78e8d224 Remove internal cli 'insert chunks from file' and 'insert metadata from file' (#19591) 2026-09-14 14:23:04 +08:00
Wang Qi b7e577ce48 Refactor: combine 3 retrieval/search API as one (#19303) 2026-09-08 11:41:46 +08:00
mkaaad 0dbfaa8b67 feat(cli): add sync logs listing commands for the user CLI (#18195) 
### Summary
```
RAGFlow(api/default)> LIST SYNC LOGS WITH PAGE_SIZE 2 PAGE 2;
Total: 764
+----------------------------------+-------------------------+-------------+-----------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| connector_id                     | docs_removed_from_index | error_count | error_msg | id                               | kb_id                            | kb_name | new_docs_indexed | prune_freq | refresh_freq | status | task_type | time_started        | total_docs_indexed | update_date         |
+----------------------------------+-------------------------+-------------+-----------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |           | fb55b5f1fb754475834cd9eac8b52054 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:45 | 16                 | 2026-08-13 09:49:46 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |           | 3cad779d7d2e4efb8d8d2dbf05d45653 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:42 | 16                 | 2026-08-13 09:49:43 |
+----------------------------------+-------------------------+-------------+-----------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+

```
```
RAGFlow(api/default)> LIST SYNC LOGS;
Total: 764
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| connector_id                     | docs_removed_from_index | error_count | error_msg                                                                                  | id                               | kb_id                            | kb_name | new_docs_indexed | prune_freq | refresh_freq | status | task_type | time_started        | total_docs_indexed | update_date         |
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 1           | sync task failed: failed to fetch URL: Get "https://xxxxxxxxxxxx/rss.xml": EOF             | 87592e5623fc4284bf69b9290f322718 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 4      | sync      | 2026-08-13 09:49:48 | 16                 | 2026-08-13 09:51:01 |
| 570138201cf44e1180711f1ebdf8492f | 0                       | 1           | sync task failed: failed to fetch URL: Get "https://www.ruanyifeng.com/blog/atom.xml": EOF | 456adbbf21a147af99adfce4405abd99 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 1            | 4      | sync      | 2026-08-13 09:50:00 | 3                  | 2026-08-13 09:50:18 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | fb55b5f1fb754475834cd9eac8b52054 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:45 | 16                 | 2026-08-13 09:49:46 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 3cad779d7d2e4efb8d8d2dbf05d45653 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:42 | 16                 | 2026-08-13 09:49:43 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | dd09320aa5b34d6cbcfa7b03ede68c4a | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:36 | 16                 | 2026-08-13 09:49:38 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | afbcee73240f420ca1997e6506f387fb | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:33 | 16                 | 2026-08-13 09:49:34 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 7d018146a0f14a25b7263d5980fdc30f | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:30 | 16                 | 2026-08-13 09:49:32 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 520e0d6f421e40628ab541fc4fee1a69 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:27 | 16                 | 2026-08-13 09:49:28 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 3d3ce49825de4ef99642e7994ae954e3 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:21 | 16                 | 2026-08-13 09:49:22 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 1c511f9b4a7c425fb30a956589ca56fa | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:18 | 16                 | 2026-08-13 09:49:19 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | b9ab73c3f48c4f47a84fd55b542b32b4 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:15 | 16                 | 2026-08-13 09:49:16 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | db0cad90a7744d758b24d2e3d3828963 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:12 | 16                 | 2026-08-13 09:49:13 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | a501eadb2cce452187f3c86431352c70 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:08 | 16                 | 2026-08-13 09:49:10 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | b4265a3b7baf4466a525b91a9fe21b71 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:03 | 16                 | 2026-08-13 09:49:05 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | a6ad590931c046dda5fc3222b4010c06 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:00 | 16                 | 2026-08-13 09:49:02 |
......
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+

```
list all logs
```
RAGFlow(api/default)> LIST DATASET 'test' SYNC LOGS;
Total: 764
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| connector_id                     | docs_removed_from_index | error_count | error_msg                                                                                  | id                               | kb_id                            | kb_name | new_docs_indexed | prune_freq | refresh_freq | status | task_type | time_started        | total_docs_indexed | update_date         |
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 1           | sync task failed: failed to fetch URL: Get "https://xxxxxxxxxxxx/rss.xml": EOF             | 87592e5623fc4284bf69b9290f322718 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 4      | sync      | 2026-08-13 09:49:48 | 16                 | 2026-08-13 09:51:01 |
| 570138201cf44e1180711f1ebdf8492f | 0                       | 1           | sync task failed: failed to fetch URL: Get "https://www.ruanyifeng.com/blog/atom.xml": EOF | 456adbbf21a147af99adfce4405abd99 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 1            | 4      | sync      | 2026-08-13 09:50:00 | 3                  | 2026-08-13 09:50:18 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | fb55b5f1fb754475834cd9eac8b52054 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:45 | 16                 | 2026-08-13 09:49:46 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 3cad779d7d2e4efb8d8d2dbf05d45653 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:42 | 16                 | 2026-08-13 09:49:43 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | dd09320aa5b34d6cbcfa7b03ede68c4a | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:36 | 16                 | 2026-08-13 09:49:38 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | afbcee73240f420ca1997e6506f387fb | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:33 | 16                 | 2026-08-13 09:49:34 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 7d018146a0f14a25b7263d5980fdc30f | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:30 | 16                 | 2026-08-13 09:49:32 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 520e0d6f421e40628ab541fc4fee1a69 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:27 | 16                 | 2026-08-13 09:49:28 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 3d3ce49825de4ef99642e7994ae954e3 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:21 | 16                 | 2026-08-13 09:49:22 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | 1c511f9b4a7c425fb30a956589ca56fa | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:18 | 16                 | 2026-08-13 09:49:19 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | b9ab73c3f48c4f47a84fd55b542b32b4 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:15 | 16                 | 2026-08-13 09:49:16 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | db0cad90a7744d758b24d2e3d3828963 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:12 | 16                 | 2026-08-13 09:49:13 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | a501eadb2cce452187f3c86431352c70 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:08 | 16                 | 2026-08-13 09:49:10 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | b4265a3b7baf4466a525b91a9fe21b71 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:03 | 16                 | 2026-08-13 09:49:05 |
| 02b5b14183884445bd55ad6b9d322204 | 0                       | 0           |                                                                                            | a6ad590931c046dda5fc3222b4010c06 | 0e6e28e65b7044e58070e0e4252319a6 | test    | 0                | 5          | 0            | 3      | sync      | 2026-08-13 09:49:00 | 16                 | 2026-08-13 09:49:02 |
......
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
```
 2026-08-13 18:39:29 +08:00
Jin Hai f621b4c7b4 Go: refactor config (#17678) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-08-01 22:06:56 +08:00
Jin Hai d19a036cda Go: add context to lots of interface (#17253) 
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-22 22:30:57 +08:00
Zhichang Yu 12787996d1 feat(agent): Go ingestion pipeline progress mirroring and DeepDOC parser hardening (#16795) 
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.

feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.

refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.

refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.

feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).

refactor: translate Chinese prose comments to English in non-harness Go
files.

chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
 2026-07-10 10:36:10 +08:00
Zhichang Yu 014c3f634f Align Go ingestion boundaries with Python (#16647) 
Moves doc_id blob resolution into Parser, tightens chunker/tokenizer to
Python output_format semantics, updates extractor list handling, and
fixes real-template integration tests.
 2026-07-05 20:43:52 +08:00
qinling0210 7862f69f39 Implement chat completions in go (#16491) 
### Summary

POST   /api/v1/chat/completions
 2026-07-01 15:52:52 +08:00
Jin Hai bd56a1473f Go CLI: merge function (#16458) 
### Summary

1. remove unused code.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-30 15:47:26 +08:00
Jin Hai 7c1edca15e Go CLI: fix api commands (#16457) 
### Summary

As title.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-29 19:09:32 +08:00
Jin Hai 3202ec6abf Go CLI: refactor commands (#16447) 
### Summary

1. Move debug commands to dev file.
2. Refactor some commands syntax

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-29 17:03:26 +08:00
Jin Hai d4ef3d21d1 Go CLI: Add create and drop commands (#16430) 
### What problem does this PR solve?

1. Add CREATE and DROP DATASET / MEMORY / AGENT / SEARCH / CHAT.
2. Add option to build.sh to strip RAGFlow binary.

### Type of change

- [x] Refactoring

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-29 11:13:14 +08:00
Zhichang Yu 0c3952147c fix(codeql): close remaining 44 CodeQL alerts post-merge (#16408) 
## Summary

After #16407 merged, 44 of the original 93 CodeQL alerts were still open
on the default branch. This PR closes the remaining ones by:

1. **Moving 32 existing `// codeql[...]` directives** so they sit on the
line **immediately before** the suppressed statement. The original
multi-line suppression blocks had the directive as the first line, with
the rationale on subsequent lines. After line shifts (refactors, linter
reformat), the directive ended up several lines above the alert location
— CodeQL only recognizes the suppression when it appears on the line
directly above. (32 alerts across 27 files.)

2. **Adding 9 new `// codeql[...]` suppressions** for alerts that had no
suppression in the preceding lines at all — mostly real-fixes that
CodeQL conservatively still flags (filepath.Base, bounded slice sizes,
model-identifier strings, the MD5-legacy-migration lookup in
`conversation_service.py`).

## Files changed

- `api/db/services/conversation_service.py` — add
`py/weak-sensitive-data-hashing` suppression (MD5 for backward-compat
legacy row lookup; not used for auth)
- `api/db/services/llm_service.py` — 3×
`py/clear-text-logging-sensitive-data` suppressions on the lines that
log `llm_name` in warnings/info
- `common/misc_utils.py` — 2× `py/clear-text-logging-sensitive-data`
suppressions on the redacted `current_url` log sites
- `internal/agent/component/invoke.go` — moved existing
`go/request-forgery` directive
- `internal/agent/sandbox/ssh.go` — moved existing
`go/command-injection` directive
- `internal/agent/tool/retrieval_service.go` — added
`go/uncontrolled-allocation-size` suppression (`topN` is bounded to 1024
above)
- `internal/cli/common_command.go` — moved 2×
`go/disabled-certificate-check` directives
- `internal/cli/user_command.go` — added `go/clear-text-logging`
suppression (filepath.Base already strips user-identifying path)
- `internal/dao/pipeline_operation_log.go` — moved 2× `go/sql-injection`
directives
- `internal/dao/user_canvas.go` — added `go/sql-injection` suppression
in `GetList` (the new `userCanvasOrderClause` call path)
- `internal/engine/infinity/chunk.go` — moved existing
`go/unsafe-quoting` directive
- `internal/entity/models/*` — moved `go/path-injection` directives (15
files)
- `internal/handler/oauth_login.go` — moved existing
`go/cookie-httponly-not-set` directive
- `internal/handler/tenant.go` — moved existing `go/path-injection`
directive
- `internal/service/deep_researcher.go` — moved existing
`go/unsafe-quoting` directive
- `internal/service/dataset.go` — added
`go/uncontrolled-allocation-size` suppression (`n` bounded to 1024
above)
- `internal/service/file.go` — moved existing `go/request-forgery`
directive
- `internal/service/langfuse.go` — moved 2× `go/request-forgery`
directives
- `internal/utility/mcp_client.go` — moved 3× `go/request-forgery`
directives
- `internal/utility/smtp.go` — moved existing `go/email-injection`
directive
- `rag/prompts/generator.py` — added
`py/clear-text-logging-sensitive-data` suppression
- `web/.../use-provider-fields.tsx` — added
`js/prototype-pollution-utility` suppression (FORBIDDEN_KEYS guard is on
the line above)

## Why the previous PR left alerts open

`// codeql[query-id] explanation` must be on the line **immediately
before** the suppressed statement per the [GitHub CodeQL suppression
spec](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning-with-codeql/suppressing-code-scanning-alerts).
The original suppression blocks were 4-5 lines, with the directive as
the **first** line. After linter reformat / line shifts, the directive
ended up too far above the actual alert line to be recognized. The fix
is to put the directive on the line directly above the suppressed
statement, with the rationale above it.

## Test plan

- All 9 modified Python files `ast.parse` clean
- All 4 modified Go files `gofmt` clean
- 36/44 expected alert suppressions in place
- 8 remaining CodeQL alerts are the originals (#3485851828, #3485851831,
#3485869759, #3485869766, #3485869768, #3485869771, #3485885962,
#3485895527) which were resolved by the corresponding commit comments;
these should close on the next scan when the suppression comments match
the alert lines.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-06-29 09:45:16 +08:00
Zhichang Yu 195bfffb5e fix(security): address 93 CodeQL code-scanning alerts across 61 files (#16407) 
## Summary

Resolves all 93 open alerts at
https://github.com/infiniflow/ragflow/security/code-scanning by rule:

| Rule | Count | Treatment |
|------|-------|-----------|
| py/clear-text-logging-sensitive-data | 23 | Real fix — log scrubbing |
| go/path-injection | 15 | Real fix where possible, suppression with
rationale |
| go/request-forgery | 8 | Suppression with rationale
(operator-controlled URLs) |
| go/clear-text-logging | 10 | Real fix — log scrubbing |
| go/unsafe-quoting | 5 | Real fix — escape or refactor |
| go/sql-injection | 3 | Real fix — orderby whitelist + CodeQL comment |
| go/uncontrolled-allocation-size | 2 | Real fix — cap to 1024 |
| go/incorrect-integer-conversion | 3 | Real fix — ParseInt + range
check |
| go/insecure-hostkeycallback | 1 | Real fix — known_hosts file |
| go/disabled-certificate-check | 2 | Suppression with rationale |
| go/command-injection | 1 | Suppression (sanitized via shq()) |
| go/email-injection | 1 | Suppression with rationale |
| go/cookie-httponly-not-set | 1 | Suppression (SPA bootstrap) |
| js/stack-trace-exposure | 1 | Real fix — generic client message |
| js/prototype-pollution-utility | 1 | Real fix — reject
__proto__/constructor/prototype |
| py/weak-sensitive-data-hashing | 1 | Real fix — MD5 → SHA-256 |
| py/incomplete-url-substring-sanitization | 3 | Real fix —
urlparse(hostname) |
| py/paramiko-missing-host-key-validation | 1 | Real fix —
load_system_host_keys + RejectPolicy |
| cpp/integer-multiplication-cast-to-long | 2 | Real fix — cast to
size_t |

## Real fixes (with measurable security improvement)

**SSH host key verification (Go + Python)**  
Replace `InsecureIgnoreHostKey()` / `paramiko.AutoAddPolicy()` with
proper host key verification against a known_hosts file (configurable
via `SSH_KNOWN_HOSTS` env / `known_hosts` config field; fail-closed when
unset). Loads `~/.ssh/known_hosts` first via `load_system_host_keys()`
so existing setups keep working.

**SQL injection in `user_canvas`**  
Add `userCanvasOrderableColumns` whitelist + `userCanvasOrderClause`
helper. Both `GetList()` and `ListByTenantIDs()` now route the
user-supplied `orderby` query param through the helper, defaulting to
`create_time` on miss.

**SQL injection in `pipeline_operation_log`**  
Existing whitelist documented via CodeQL comment.

**Real SQL injection in `infinity/chunk.go:931`**  
Escape `'` → `''` on user-controlled `questionText` before splicing into
`filter_fulltext(...)` SQL filter.

**Real SQL injection in `elasticsearch/sql.go:75`**  
Defense-in-depth escape on tokenizer output before splicing into
`MATCH(...)`.

**Python code injection in `result_protocol.go`**  
Replace raw JSON literal embedding into Python/JS expressions with
base64 + `json.loads` / `JSON.parse(Buffer.from(...,
'base64').toString('utf8'))`. Eliminates both the unsafe-quoting sink
and the brittleness of mixing JSON true/false/null with Python syntax.

**URL substring check bypass in `embedding_model.py`**  
Replace `if "dashscope-intl.aliyuncs.com" in u` with
`urlparse(u).hostname == "dashscope-intl.aliyuncs.com"` so a base_url
like `https://attacker.example/?u=dashscope-intl.aliyuncs.com` cannot
bypass the routing.

**Prototype pollution in `setNestedValue` (TS)**  
Reject `__proto__`/`constructor`/`prototype` keys before any assignment.

**Integer overflow**  
- scrypt params via `ParseInt` + non-positive check
(`internal/common/password.go`)
- `topN` and `n` caps to 1024 (retrieval_service.go, dataset.go)
- `nalloc*statesize` cast to `size_t` (cpp/re2/onepass.cc)

**Cookie httponly**  
Set explicitly with rationale: this is the OAuth bootstrap cookie
intentionally read by the SPA.

**Stack trace exposure**  
Replace `error.message` in HTTP 500 response with generic `"internal
error"`; full error still logged server-side via `console.error`.

**Weak hashing**  
MD5 → SHA-256 for deterministic `conv_id` derivation
(`conversation_service.py`).

**Log scrubbing**  
Remove or redact user-controlled / sensitive content from clear-text
logs across 8 ingestion parsers, `llm_service.py` ×11,
`tenant_llm_service.py` ×7, `misc_utils.py` ×4, `redis_conn.py` ×10,
`conftest.py` ×4, `init_data.py`, `dataset_api_service.py`,
`generator.py`, `mysql_migration.py`, `cli.go`, `user_command.go`,
`pdf_parser.go`. Most patterns converted to parameterized logging
(`logging.info("...: %d", n)`) or static messages.

## CodeQL suppressions (each with rationale)

For alerts where the data flow is genuinely safe but CodeQL can't see
the context — operator-controlled URLs, sanitized inputs, etc. — I added
`// codeql[go/<rule>] <rationale>` annotations rather than dismissing
them, so future readers can audit the rationale inline:

- `internal/agent/component/invoke.go:135` — Invoke is a generic canvas
HTTP client
- `internal/service/langfuse.go` ×2 — host is per-tenant operator config
- `internal/service/file.go:1184` — already SSRF-guarded by
`assertURLSafe`
- `internal/utility/mcp_client.go` ×3 — already `AssertURLSafe` +
IP-pinned
- `internal/entity/models/bedrock.go` — sigv4-signed request, URL can't
be tampered
- `internal/service/deep_researcher.go:269` — `callback` is SSE display
string, not SQL
- `internal/engine/infinity/chunk.go:346` — UUIDs can't contain `'` (RFC
4122)
- `internal/cli/common_command.go` ×2 — CLI trusts operator-configured
URL
- `internal/utility/smtp.go:194` — msg is server-built, not user form
input
- `internal/entity/models/*` ×14 (path-injection) — audio file paths are
caller-supplied

## Test plan

- ✅ All 13 modified Go packages build cleanly
- ✅ 663 tests pass across `internal/agent/sandbox`, `internal/common`,
`internal/agent/component`, `internal/engine/infinity`, `internal/dao`
- ✅ All 11 modified Python files parse via `ast.parse`
- ✅ TypeScript `tsc --noEmit` clean on the modified
`use-provider-fields.tsx`
- ✅ `node --check` clean on the modified JS file

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-06-29 09:45:16 +08:00
Jin Hai f763044889 Go CLI: Fix show admin server and api server (#16382) 
### What problem does this PR solve?

RAGFlow(api/default)> show admin server;

RAGFlow(api/default)> show api server 'default';

RAGFlow(admin)> show admin server;

RAGFlow(admin)> show api server 'default';

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-26 19:16:14 +08:00
Jin Hai 8bc27d8df1 Go CLI: fix show variable (#16370) 
### What problem does this PR solve?

```
RAGFlow(api/default)> show var 'mail.port';
+-----------+-----------+--------------+-------+
| data_type | name      | setting_type | value |
+-----------+-----------+--------------+-------+
| integer   | mail.port | config       | 30    |
+-----------+-----------+--------------+-------+
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-26 13:51:56 +08:00
Jin Hai dbefadd86a Go CLI: refactor (#16355) 2026-06-25 20:36:50 +08:00
Jin Hai edfa9be67f Go CLI: fix list provider instance tasks (#16345) 2026-06-25 15:49:31 +08:00
Jin Hai 17b066e6ae Go CLI: fix list dataset files by dataset name (#16341) 
### What problem does this PR solve?

```
RAGFlow(api/default)> list dataset 'ccc' files;
Total: 1
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-25 13:41:58 +08:00
Jin Hai 7ef4a4a06a Go CLI: list provider instance models, sync and list provider (#16311) 
### What problem does this PR solve?

```
RAGFlow(api/default)> list provider 'zhipu-ai' instance 'test' models sync;
+------------+---------------+------------+-------------+------------------+---------------------------------------------+
| dimensions | max_dimension | max_tokens | model_types | name             | thinking                                    |
+------------+---------------+------------+-------------+------------------+---------------------------------------------+
|            |               | 128000     | [chat]      | glm-4.5@z-ai     | map[clear_thinking:true default_value:true] |
|            |               | 128000     | [chat]      | glm-4.5-air@z-ai | map[clear_thinking:true default_value:true] |
|            |               | 202752     | [chat]      | glm-4.6@z-ai     | map[clear_thinking:true default_value:true] |
|            |               | 202752     | [chat]      | glm-4.7@z-ai     | map[clear_thinking:true default_value:true] |
|            |               | 202752     | [chat]      | glm-5@z-ai       | map[clear_thinking:true default_value:true] |
|            |               | 200000     | [chat]      | glm-5-turbo@z-ai | map[clear_thinking:true default_value:true] |
|            |               | 202752     | [chat]      | glm-5.1@z-ai     | map[clear_thinking:true default_value:true] |
|            |               |            | [chat]      | glm-5.2@z-ai     |                                             |
+------------+---------------+------------+-------------+------------------+---------------------------------------------+

RAGFlow(api/default)> list provider 'zhipu-ai' instance 'test' models;

RAGFlow(api/default)> list dataset 'aaa' ingestion tasks;

RAGFlow(api/default)> list dataset '0abe79f9423311f1ad8d38a74640adcc' documents;

```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-25 10:01:21 +08:00
Jin Hai 1fc02606ea Go CLI: fix key commands (#16306) 
### What problem does this PR solve?

```
RAGFlow(api/default)> set key 'ragflow-JgnarFSCUiV99oOvvMDei7ZzZg1cVlqGd1AMHrHeKE4';
SUCCESS
RAGFlow(api/default)> unset key;
SUCCESS

RAGFlow(api/default)> list provider 'zhipu-ai' instances;

RAGFlow(api/default)> list providers;

RAGFlow(api/default)> list available providers;

RAGFlow(api/default)> list provider 'zhipu-ai' instance 'test' models;
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-24 18:48:09 +08:00
Jin Hai 9624f70b22 Go CLI: refactor (#16299) 
```
RAGFlow(api/default)> list dataset 'e93ab2c04ad111f1b17438a74640adcc' documents;
Total: 1

RAGFlow(api/default)> list datasets;


RAGFlow(api/default)> list chats;
Total: 2

RAGFlow(api/default)> list agents;
Total: 1

RAGFlow(api/default)> list searches;
Total: 1

RAGFlow(api/default)> list keys;
+----------------------------------+---------------+----------------------------------+-----------------------------------------------------+---------------+
| beta                             | create_time   | tenant_id                        | token                                               | update_time   |
+----------------------------------+---------------+----------------------------------+-----------------------------------------------------+---------------+
| GKsLEdSUkl76gJz1k_4fJpSQRIlWsiki | 1782285917523 | 2ba4881420fa11f19e9c38a74640adcc | ragflow-JgnarFSCUiV99oOvvMDei7ZzZg1cVlqGd1AMHrHeKE4 | 1782285917523 |
+----------------------------------+---------------+----------------------------------+-----------------------------------------------------+---------------+
RAGFlow(api/default)> create key;
SUCCESS

RAGFlow(api/default)> drop key 'ragflow-aA4R7AuUD158yh2LDh7IDBiqwOKFDKeTwUSQSLVdPdM';
SUCCESS
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-24 16:50:40 +08:00
Jin Hai e15130534f Go: default public key (#16265) 
### What problem does this PR solve?

Provider default public key for CLI

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-23 17:43:26 +08:00
Jin Hai dec2ce4a60 Go CLI: admin model framework (#16252) 2026-06-23 16:57:05 +08:00
Jin Hai b661e9c19e Go CLI: admin list providers (#16243) 
### What problem does this PR solve?

```
RAGFlow(admin)> list providers;
+----------------------+-------------------------------------------------------------+
| command              | error                                                       |
+----------------------+-------------------------------------------------------------+
| list_model_providers | 'list model providers' is implemented in enterprise edition |
+----------------------+-------------------------------------------------------------+

RAGFlow(admin)> add provider 'zhipu-ai';
+-------------+-----------------------------------------------------------+
| field       | value                                                     |
+-------------+-----------------------------------------------------------+
| command     | add_model_provider                                        |
| error       | 'add model provider' is implemented in enterprise edition |
| provider_id | admin                                                     |
| user_id     | zhipu-ai                                                  |
+-------------+-----------------------------------------------------------+

RAGFlow(admin)> delete provider 'zhipu-ai';
+-------------+--------------------------------------------------------------+
| field       | value                                                        |
+-------------+--------------------------------------------------------------+
| command     | delete_model_provider                                        |
| error       | 'delete model provider' is implemented in enterprise edition |
| provider_id | admin                                                        |
| user_id     | zhipu-ai                                                     |
+-------------+--------------------------------------------------------------+

RAGFlow(admin)> add provider 'zhipu-ai' instance 'instance1';
+---------------+-----------------------------------------------------------+
| field         | value                                                     |
+---------------+-----------------------------------------------------------+
| command       | add_model_instance                                        |
| error         | 'add model instance' is implemented in enterprise edition |
| instance_name | instance1                                                 |
| provider_id   | zhipu-ai                                                  |
| user_id       | admin                                                     |
+---------------+-----------------------------------------------------------+

RAGFlow(admin)> delete provider 'zhipu-ai' instance 'test'
+-------------+--------------------------------------------------------------+
| field       | value                                                        |
+-------------+--------------------------------------------------------------+
| instances   | [test]                                                       |
| provider_id | zhipu-ai                                                     |
| user_id     | admin                                                        |
| command     | delete_model_provider                                        |
| error       | 'delete model instance' is implemented in enterprise edition |
+-------------+--------------------------------------------------------------+

RAGFlow(admin)> add provider 'zhipu-ai' instance 'instance1' model 'xxx';
+---------------+--------------------------------------------------+
| field         | value                                            |
+---------------+--------------------------------------------------+
| command       | add_model                                        |
| error         | 'add model' is implemented in enterprise edition |
| instance_name | instance1                                        |
| model_names   | [xxx]                                            |
| provider_id   | zhipu-ai                                         |
| user_id       | admin                                            |
+---------------+--------------------------------------------------+

RAGFlow(admin)> delete provider 'zhipu-ai' instance 'test' model 'xxx';
+---------------+------------------------------------------------------+
| field         | value                                                |
+---------------+------------------------------------------------------+
| command       | delete_model_provider                                |
| error         | 'delete models' is implemented in enterprise edition |
| instance_name | test                                                 |
| models        | [xxx]                                                |
| provider_id   | zhipu-ai                                             |
| user_id       | admin                                                |
+---------------+------------------------------------------------------+

```

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-23 10:26:31 +08:00
qinling0210 563d855780 Implement OpenAI chat completions in GO (#16177) 
### What problem does this PR solve?

Implement OpenAI chat completions in GO

POST /api/v1/openai/<chat_id>/chat/completions

OpenAI chat cli: internal/development.md

### Type of change

- [x] Refactoring
 2026-06-18 18:07:27 +08:00
Jin Hai 32d5c0039b Go: refactor model API to accept model id (#15999) 
### What problem does this PR solve?

Not not only model_name@instance_name@provider_name is acceptable, but
also model_id is acceptable.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-15 10:10:14 +08:00
Jin Hai e89afbae21 Go: file parser config (#15989) 
### What problem does this PR solve?

Add parser config

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-13 19:40:43 +08:00
Jin Hai 234f1b7cff Go: add office_oxide and parse docx file. (#15976) 
### What problem does this PR solve?

As title.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-12 20:28:15 +08:00
Jin Hai 115b730d07 Go: parse ingestion DSL (#15938) 
PR #15938

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-12 17:58:36 +08:00
Jin Hai e96bc37d06 Go: use NATS as the message queue (#15327) 
### What problem does this PR solve?

```
RAGFlow(admin)> mq publish 'msg2';
SUCCESS
RAGFlow(admin)> mq publish 'msg3';
SUCCESS
RAGFlow(admin)> mq list;
+---------+---------------+
| message | subject       |
+---------+---------------+
| msg1    | tasks.RAGFLOW |
| msg2    | tasks.RAGFLOW |
| msg3    | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull 2;
+---------+---------------+
| message | subject       |
+---------+---------------+
| msg1    | tasks.RAGFLOW |
| msg2    | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull noack;
+---------+---------------+
| message | subject       |
+---------+---------------+
| abc     | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq show
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| ack_pending_count | consumer_count | memory | message_count | pending_count | redelivered_count | waiting_count |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| 2                 | 1              | 0      | 2             | 0             | 1                 | 0             |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+

RAGFlow(admin)> list ingestors;
+--------------+-------------------------------------------+--------+
| host         | name                                      | status |
+--------------+-------------------------------------------+--------+
| 192.168.1.38 | ingestor-8f0e4bd5650a4ac58b0151969fbf6935 | alive  |
+--------------+-------------------------------------------+--------+

RAGFlow(admin)> list ingestion tasks;
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| document_id                      | id                               | status    | step | user        | user_id                          |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| ffe64fae423411f1a2d938a74640adcc | 90d3d0f6528941c1ac8eb0360effccc4 | COMPLETED | 5    | aaa@aaa.com | 2ba4881420fa11f19e9c38a74640adcc |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+

RAGFlow(admin)> remove ingestion tasks '90d3d0f6528941c1ac8eb0360effccc4';
+---------+----------------------------------+
| delete  | task_id                          |
+---------+----------------------------------+
| success | 90d3d0f6528941c1ac8eb0360effccc4 |
+---------+----------------------------------+

RAGFlow(admin)> stop ingestion tasks 'e89e20d9a25848a1b79bd9345ddbfe1d';
+----------+----------------------------------+
| status   | task_id                          |
+----------+----------------------------------+
| STOPPING | e89e20d9a25848a1b79bd9345ddbfe1d |
+----------+----------------------------------+

# Publish a message
RAGFlow(admin)> mq publish 'cdd';
SUCCESS

# List current tasks in the message queue
RAGFlow(admin)> mq list
+----------------------------------+---------------+
| message                          | subject       |
+----------------------------------+---------------+
| 7ce392a3c1624cd2be4b5276e8825059 | tasks.RAGFLOW |
+----------------------------------+---------------+

# Consume a task from the message queue
RAGFlow(admin)> mq pull
+------+-----+----------------+
| ack  | id  | type           |
+------+-----+----------------+
| true | cdd | ingestion_test |
+------+-----+----------------+

# User mode
# List ingestion tasks, followed by dataset id
RAGFlow(user)> list ingestion tasks from '0abe79f9423311f1ad8d38a74640adcc';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d |        | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

RAGFlow(user)> list ingestion tasks;
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-06-02T19:02:31+08:00 | 1780398151417 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | e89e20d9a25848a1b79bd9345ddbfe1d |        | COMPLETED | 2026-06-02T19:02:52+08:00 | 1780398172208 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

# Create an ingestion task
# First argument is document id, second argument is dataset id
RAGFlow(user)> start ingestion 'ffe64fae423411f1a2d938a74640adcc' from '0abe79f9423311f1ad8d38a74640adcc';
+----------------------------------+-------------------------------------------+
| document_id                      | result                                    |
+----------------------------------+-------------------------------------------+
| ffe64fae423411f1a2d938a74640adcc | task_id: 8d758cd14a8b4ba8ab505003fb52017d |
+----------------------------------+-------------------------------------------+

# Pause an ingestion task, first argument is ingestion id
RAGFlow(user)> stop ingestion '8d758cd14a8b4ba8ab505003fb52017d';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d |        | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

# Delete an ingestion task
RAGFlow(api/default)> remove ingestion tasks 'f366450a27d54677aec1c7090add30f0';
+---------+----------------------------------+
| remove  | task_id                          |
+---------+----------------------------------+
| success | f366450a27d54677aec1c7090add30f0 |
+---------+----------------------------------+

```

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-12 14:56:44 +08:00
Jin Hai ca00d23aac Go: add parse and chunk command (#15936) 
### What problem does this PR solve?

Two commands are used for ingestion file testing
```
RAGFlow(api/default)> chunk 'file' with 'dsl';
Chunk file: file, DSL: dsl
SUCCESS
RAGFlow(api/default)> parse file 'filename' chat 'xxx';
Success to parse local file "filename", vision: , chat: xxx, asr: , ocr: , embedding: , doc_parse: 
SUCCESS
```

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-11 13:33:26 +08:00
Yingfeng 01a2a44766 Clean CLI for filesystem (#15838) 
### Type of change

- [x] Refactoring
 2026-06-09 17:00:10 +08:00