Add Sitemap data source — ingests web pages from sitemap.xml with incremental sync. Supports HTML→Markdown extraction, PDF discovery, URL filtering, SSRF protection.
### Summary
Add Xquik as a new data source for X (Twitter) post ingestion. Supports Latest/Top queries, cursor pagination, incremental sync, and maps engagement/media to RAGFlow documents. Stops duplicate requests on repeated cursors. Tests and docs included.
---------
Co-authored-by: kriptoburak <kriptoburak@users.noreply.github.com>
Adds Azure DevOps as a new data source, syncing repository files and pull requests. Supports both hosted and self-hosted. Excludes wikis, work items, and binaries. Includes checkpoint/resume, change detection via commit IDs, and handles Azure's quirky 203 auth response. Tests passed against real deployment.
Gonghan-PrincessandGonghan-Princess
VincentLambert
Burak Bayırandkriptoburak
Erhan Aydın
YanZhang