salix/madre
1
0
Fork 0
Code Issues Pull requests 2 Projects Releases 1 Packages Wiki Activity Actions 2

Gather the search index at a slower pace #3

Open
salix wants to merge 2 commits from feat/slower-index-gathering into main
pull from: feat/slower-index-gathering
merge into: salix:main
Conversation 0 Commits 2 Files changed 1 +20 -6
salix commented 2026-10-19 10:51:45 +02:00
Owner
Copy link

What & why

The background search-index crawler rested only 150 ms between 100-event batches and paced each room independently, so a catch-up run fired six or seven back-pagination requests a second — for work the user never asked for. This PR paces the crawl like Element Desktop's search crawler instead: one batch per pass with rooms taking turns, and a three-second rest after every request no matter the room, so the catch-up emits at most one back-pagination request every three seconds however many rooms are behind. The batch size (100 events) and five-second idle wait already matched Element.

Verify

  • Open the app on an encrypted account whose index is still catching up: back-pagination requests arrive at most once every three seconds, and the search-index progress in preferences keeps advancing
  • cargo check -p madre-core
## What & why The background search-index crawler rested only 150 ms between 100-event batches and paced each room independently, so a catch-up run fired six or seven back-pagination requests a second — for work the user never asked for. This PR paces the crawl like Element Desktop's search crawler instead: one batch per pass with rooms taking turns, and a three-second rest after every request no matter the room, so the catch-up emits at most one back-pagination request every three seconds however many rooms are behind. The batch size (100 events) and five-second idle wait already matched Element. ## Verify - Open the app on an encrypted account whose index is still catching up: back-pagination requests arrive at most once every three seconds, and the search-index progress in preferences keeps advancing - `cargo check -p madre-core`
Gather the search index at a slower pace
All checks were successful
Build x86_64 Flatpak / flatpak (pull_request) Successful in 10m10s
Details
fa13873f52 
The backfill crawler rested 150 ms between 100-event batches, so a
catch-up run hammered the homeserver with six or seven requests a
second for work the user never asked for. Rest a full second between
batches instead: about one request a second, a typical room still
caught up in minutes, and the connection stays free for the sync.
Pace the search index crawl like Element's
Some checks are pending
Build x86_64 Flatpak / flatpak (pull_request) Waiting to run
Details
59af2265b1 
Element Desktop's search crawler rests three seconds between
100-event requests and idles for five; our rest was per room, so the
request rate still grew with the number of rooms catching up. Pace
the whole crawl instead: one batch per pass, rooms taking turns in a
round robin, and a three-second rest after every request, whatever
the room. The five-second idle wait already matched.
Some checks are pending
Build x86_64 Flatpak / flatpak (pull_request) Waiting to run
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/slower-index-gathering:feat/slower-index-gathering
git switch feat/slower-index-gathering

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff feat/slower-index-gathering
git switch feat/slower-index-gathering
git rebase main
git switch main
git merge --ff-only feat/slower-index-gathering
git switch feat/slower-index-gathering
git rebase main
git switch main
git merge --no-ff feat/slower-index-gathering
git switch main
git merge --squash feat/slower-index-gathering
git switch main
git merge --ff-only feat/slower-index-gathering
git switch main
git merge feat/slower-index-gathering
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
salix/madre!3
No description provided.