Fix Python S3 bucket creation in non-us-east-1 regions — now sends LocationConstraint via CreateBucketConfiguration.
---------
Co-authored-by: Sbaaoui Idriss <112825897+6ba3i@users.noreply.github.com>
Fix SereneDB available_int=0 filter dropped as falsy — now handles it before skip, matching other backends. ?available=false now returns only disabled chunks.
Fix#18873 — add missing get_scores() to OBConnection for SeekDB score extraction. Vector/hybrid retrieval and Chat RAG now work.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Fix ensure_v1() mangling Gemini's v1beta/openai endpoint by appending /v1. Now recognizes any segment starting with v{digit} as already versioned.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Add chunk_order_int to OceanBase chunk schema and migration. Defaults to 0 on NULL or missing field. Tests added.
Fixes#16967
---------
Co-authored-by: zq <zhouquan1511@163.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Fix OpenSearch update missing kb_id scope. Unlike Elasticsearch, OSConnection.update dropped the knowledgebaseId argument, causing updates (like clearing pagerank) to apply to all datasets in a tenant. Now adds kb_id to the query condition.
### Summary
@JinHai-CN — the cleanup you invited in #18478. This is the same finding
CodeRabbit raised against the You.com connector there, applied to the
three that shipped before it.
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
### Summary
Fix OpenSearch retrieval returning `vector_similarity = 0.000` for every
chunk when hybrid search is enabled.
On the OpenSearch backend, retrieval uses a second KNN-only search
(`Dealer._knn_scores()`) to recover per-chunk cosine scores for
reranking. That pass intentionally sends `MatchDenseExpr(...,
{"similarity": 0.0})` to mean “no minimum similarity cutoff.”
However, `OSConnection.search()` was incorrectly mapping `similarity` to
the KNN clause `boost` field:
```python
knn_query[vector_column_name]["boost"] = similarity
```
With `similarity=0.0`, this produced `boost=0.0`, which zeroed out KNN
`_score` values. `get_scores()` then returned `0.0` for every hit, so
`vector_similarity` was always zero and hybrid ranking ignored the
vector component — with no exception raised.
This is separate from the `get_scores()` `AttributeError` crash
addressed in #14970 / #15390; here retrieval succeeds but vector scores
are silently lost.
### Summary
Adds You.com as a built-in Web Search provider for RAGFlow Chat,
alongside Tavily and Querit, using the provider-neutral dispatch #17813
put in place. No changes to existing Tavily or Querit behaviour.
You.com runs its own web index and returns several extracted passages
per result rather than a single meta description, so retrieved chunks
arrive with usable context.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
### Summary
This PR adds [Serply](https://serply.io) as a third web search provider
for chat assistants, alongside the existing Tavily and Querit options.
### Summary
GaussDB DocEngine could return no chunks for conversational queries even
when relevant content was available. `Dealer.search()` supplies
`minimum_should_match` (30%, then 10% on retry), but the GaussDB adapter
discarded it and built a single `plainto_tsquery` from every token. This
effectively required all conversational filler terms to match.
Follow-up to #17526 ("Refactor: merge dataset scope graph"), which introduced two code paths that touch Infinity columns the deployed schema does not declare. This PR makes the runtime robust against the old schema while also adding the new column to the new schema so freshly created tables are correct.
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…rphaned clusters
### Summary
**Problem**
When deleting a document from the knowledge base, the parent cluster
could never be located and was therefore never cleaned up, leaving
orphaned clusters in Infinity indefinitely.
**Root Cause**
InfinityConnection.field_keyword() treats all _kwd-suffixed columns as
keyword lists and applies .split("###") to them in get_fields().
parent_kwd was missing from the exclusion list. This caused a parent
cluster name like "Toronto Transit & Authority Control 5e8d5eb6" to
become ['Toronto Transit & Authority Control 5e8d5eb6'] (a
single-element list). When that list was fed into _nav_cluster_id(), it
produced a different hash than the actual cluster's hash (computed from
the plain string), so every deletion-time cluster lookup silently
failed.
**Fix**
Added "parent_kwd" to the exclusion list in field_keyword(), alongside
docnm_kwd, important_kwd, question_kwd, etc.
### What problem does this PR solve?
Fix Infinity compatibility issues in knowledge compilation.
This change:
- Stores compilation source ID lists as JSON arrays in Infinity.
- Parses JSON array fields when reading compiled documents.
- Uses `json_contains` for filtering JSON array fields.
- Adds the missing `name` column to the Infinity mapping.
- Updates dataset navigation KNN search to use the unified
`MatchDenseExpr` interface.
- Handles unavailable embeddings without querying an invalid `q_0_vec`
field.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
When a table dataset's `field_map` is missing or stale,
`aggregate_table_doc_metadata` falls back to probing chunk dictionaries
for each column's Elasticsearch field key. It currently performs that
probe only once, against the first dictionary chunk, and caches `(None,
"none")` if the field is absent there.
Sparse table rows commonly omit empty columns. If the first row has no
`notes` field but a later row contains `notes_raw`, the cached miss
causes every later row to be skipped and the document-level `notes`
metadata is silently lost. The result depends only on row order:
```python
chunks = [{}, {"notes_raw": "Handle with care"}]
aggregate_table_doc_metadata(chunks, task) # before: {}
aggregate_table_doc_metadata(list(reversed(chunks)), task)
# before: {"notes": ["Handle with care"]}
```
This was also identified in CodeRabbit's review of the merged
table-metadata implementation in #15780, but remained unfixed after that
PR merged:
https://github.com/infiniflow/ragflow/pull/15780#pullrequestreview-4448490676
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
When the initial lookup found no key for a column, retry the existing
`_resolve_es_chunk_field_key` against the current chunk. Cache the first
successful resolution so subsequent rows retain the existing fast path.
Field-map-backed columns and columns found in the first chunk are
unchanged.
### Testing
- Added `test_aggregate_auto_mode_probes_later_sparse_chunks` with an
empty first row and a populated second row.
- Confirmed red→green: before the fix the assertion received `{}`; after
the fix it receives `{"notes": ["Handle with care"]}`.
- Full existing `test_table_metadata_aggregation.py`: **15 passed**.
- `ruff check` and `ruff format --check`: clean.
- `compileall` for both changed files: clean.
The local test environment did not contain the repository's full service
dependency set and had a corrupt pre-existing NLTK `wordnet.zip`. The
test module does not use those services or corpora, so the run stubbed
only `common.settings` engine flags, `json_repair`, and the global
conftest's NLTK resource lookup; the production module and aggregation
tests themselves ran unchanged.
### Duplicate-work check
Checked all currently open PRs (including changed file paths) and found
none touching `rag/utils/table_es_metadata.py` or its aggregation test.
The earlier #15780 review is historical context, not active competing
work.
### Disclosure
AI-assisted (Codex): the candidate came from an AI-assisted review
queue. I independently reproduced the order-dependent data loss against
the real module, checked the historical review and all open PR file
paths, and ran the regression plus full existing test file before
submitting.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sbaaoui Idriss
Anupam Mediratta
huyua9
Wenchao An
Huang-404-Q
SYED ALI ABBAS RAHIL
MarMar Labs
Wang Qi
kic635andJin Hai
Andrew ChenandClaude Opus 4.8
wangyunlai
Paul Yao
Brian SparkerandBrian Sparker
Andras BARTHA
Serply
Sevenzuo
ump45nose
zws202606
S
qinling0210
buua436
EthanZhang
deadtricksterandClaude Opus 4.8
Kevin Hu
Yingfeng
Andrew ChenandClaude Opus 4.8
Lynn
Zhichang Yu
Öndery