Storage, sync, and lists

Where things are stored, sync between browsers, and subscribing to lists. Part of Experiments and decisions. Newest notes go at the top of each section.

FOSS tools default list (2026-10-04)

• Added: a separate bundled, default list labels LibreSpeed and cobalt.tools with the neutral FOSS tag, and makes AI slop available for manual tagging. Kept it separate from Reference rather than broadening that list's meaning; the rules do not change ranking or hide results.
• Compatibility: default subscriptions remain a fallback only when none are stored. Existing users are not automatically resubscribed and can add FOSS tools from Settings → Lists → More lists.
• Not checked: the live sites' source and licence pages. The list reflects the requested classification and does not claim anything about each hosted service's privacy practices.

Passphrase changes that race or lose their answer (2026-09-29)

• Found: a review of the passphrase change turned up three gaps. It ran in the Settings page, so the lock that keeps syncs from overlapping (which only works within one script) didn't hold back the background script's syncs; a sync's last attempt writes without If-Match and could put back a file under the old passphrase. When the server saved the new file but the answer was lost, the old passphrase stayed saved and the next sync failed. And a server disconnected during the change was connected again when it finished.
• Fixed: Settings asks the background script to change the passphrase (change-passphrase), where it queues behind syncs. When the save gets no answer or a server error, Anubis reads the file back and keeps the new passphrase if it opens the file; when the server can't be reached, it keeps the old one with an unconfirmed status telling the user what to do. The new passphrase is saved only while the same connection is still there.
• Checked: unit tests for a lost answer, the server going away after the save, and a disconnect during the change each failed before the fix. The webdav e2e part still changes the passphrase through Settings.

WebDAV passphrase changes and recovery (2026-09-29)

• Built: Settings can change an established encryption passphrase. Anubis reads and decrypts the current encrypted file, encrypts the same data with the new passphrase, and replaces it only with a strong If-Match ETag. A missing/weak ETag, concurrent edit, wrong current key, or failed write leaves the saved passphrase unchanged. Other browsers must get the new passphrase before their next sync.
• Rejected: running a normal merge sync before rotating the key. That could write changes using the old passphrase before a later rotation failure; the rotation now changes only the encryption on the exact server copy it read.
• Test correction: the encrypted envelope is format 2, but its decrypted backup payload remains format 1. The first browser assertion checked the payload as though it were the envelope; it now checks both layers separately.
• Recovery: added a specific concurrent-change status with a retry instruction. Existing status messages already explain how to recover from denied permission, login, folder, network, unreadable-file, wrong-key, and plaintext-downgrade errors; the sync guide now explains updating other browsers after rotation.
• Ranking explanations: the result menu already identified each list and its action, so no second explanation system was needed. It now also shows the exact matched list rule and line number; the matcher already retained both for report links.
• Checked: unit tests cover successful rotation, wrong current passphrase, a concurrent ETag change, server rejection, and matcher reasons. The browser check now requires a matched rule in the result menu. The Settings and result-menu documentation screenshots are regenerated from the mocks.

End-to-end WebDAV encryption (2026-09-29)

• Built: new connections encrypt the sync file by default with AES-256-GCM; PBKDF2-SHA-256 derives the key from a user-chosen passphrase. A random salt and nonce are stored with the ciphertext. The passphrase stays in that browser's local extension storage, never browser sync or the WebDAV request.
• Why this design: Web Crypto provides AES-GCM in both target browsers, so no dependency or downloaded code is needed. GCM provides confidentiality and detects modified ciphertext in one operation; a fresh random nonce is generated for every encryption, and a random salt makes each derived key unique. PBKDF2-SHA-256 is available in the same API and slows offline guessing of the passphrase. This is not a substitute for a strong passphrase or protection for a compromised browser profile.
• Migration and safeguards: existing plaintext connections can opt in without losing their merge base; their next successful write replaces the file with ciphertext. A browser without the passphrase, or with the wrong one, cannot overwrite an encrypted file. Once encryption is established, Anubis rejects a plaintext downgrade. An encrypted file can still be deleted or replayed by its server; this is not an availability or rollback defense.
• Recovery trade-off: losing the passphrase makes the server copy unreadable; data still present in each browser can be exported separately. The passphrase is available to Anubis in local extension storage so background syncing can work, so it does not protect a compromised browser profile. Plain backup exports are not encrypted by WebDAV encryption.
• Checked: WebDAV unit coverage verifies ciphertext has no readable list content or passphrase, cross-browser decryption, plaintext migration, missing/wrong passphrase non-overwrite, and downgrade rejection. Updated the connect screen, privacy policy, sync guide, store notes, and generated screenshots.
• Layout iteration: the first generated screenshot showed the long passphrase hint expanding the form's max-content label column and pushing inputs offscreen. Grouping the passphrase fields into their own grid fixed it; the refreshed image shows the complete form.

Reporting mistakes to lists

Asked for: a way to report a wrong result, like SponsorBlock's feedback loop but without a backend. Lists already live in Git repositories, so a report can be an issue there.

• Shipped: under Why in the result menu, "Wrong? Report it to list" for each list that weighed the result. It opens a pre-filled issue: the result's address without its query or fragment (which can carry session details), what the list does with it, the rules that matched with their line numbers, and a "What should change:" prompt. The issue is in English whatever the interface language, since it's for the list's maintainers. Line numbers come from the copy Anubis has, which can be older than the repository's, so the rules' text goes in too.
• Where it goes: the list's ! issues:, else the repository its ! homepage: or its own address points into (raw.githubusercontent.com/<owner>/<repo>/…, GitLab's …/-/raw/…, Codeberg's …/raw/…). Gists and other hosts have no tracker to derive, so they get no link. None of the seven third-party lists in the directory sets ! issues: or ! homepage: (checked 2026-09-29), so without deriving it the feature would reach only Anubis's own lists.
• Suggestions stay opt-in. "Suggest it to…" still needs ! issues:: a report of a mistake is an ordinary bug report for any repository, but unsolicited additions aren't. Suggestions now skip lists that already weigh the result, since for those the report is the right link; before, such a list could be offered a suggestion.
• Considered: a "Report" link on each line under Why. Rejected because one list can give several reasons for one result; one link per list, named, also says where the report goes.
• Fixed later: a suggestion's example result dropped only the query (split('?')), so an address with just a #fragment kept it. Both issues now build the address the same way (plainAddress in utils/subscriptions.ts), without query or fragment.
• The report and suggestion lines are the first of the menu's text in messages.json. tList in utils/i18n.ts puts the links in place of $1, joined with Intl.ListFormat ("A or B").
• Unverified: what GitHub does when a repository has issues turned off (it should show the repository), or has issue forms with blank issues turned off (it may show the template chooser and drop the pre-filled text).

Storage

• Syncing between browsers (2026-09-29): optional, through a WebDAV server the user connects in Settings → Sync; browser sync stays the default and runs either way. (Firefox for Android, which doesn't sync add-on data, gets sync this way too.)

  • Why WebDAV. uBlacklist offers Google Drive, Dropbox, OneDrive, and WebDAV. Each of the first three needs the project to register an app with the service and keep it, the identity permission for the sign-in window, and for Google a review; together with its sync loop that's about 1,900 lines, not counting settings screens and tests. WebDAV needs none of that and no new permission: connecting asks for the one host through the optional https://*/* that list subscriptions already use. The cost is on the user, who needs an account with a service that offers it (Koofr and InfiniCLOUD have free plans, Nextcloud works), and usually an app password. Dropbox is the lighter follow-up if people ask for something they already have.
  • One file, the backup format. anubis-sync.json holds what Export backup saves, so Restore backup reads it too. It's read and written whole: small, and one file means one ETag to guard.
  • Merging, not newest-wins. uBlacklist keeps the newer copy of each part. Here each sync merges three ways against what both sides had at the last one (local:webdavBase), so a site hidden in Firefox and one tagged in Chrome both survive, and a ranking changed on one side combines with a tag added on the other. The personal list merges line by line, keyed by what a line is about (a site, a tag definition, a ! name: header, or the line itself), so moved lines still line up; lines only the other side has go after the line they follow there. 6,000 new sites merge in well under a second. When both sides changed the same thing, the browser syncing now wins.
  • The first sync uses what a fresh install has as its base, with the server winning: an untouched browser takes everything (without bringing back the default fandom.com the other browser removed), and one that already had its own sites keeps them. A test that took the local copy as the base still passed the first case; the second, a browser with its own sites, is what tells them apart.
  • Saving in between. The write carries If-Match with the ETag just read; a server that got another browser's save in between refuses it (412) and the sync merges again, up to three times, the last without the condition. A weak ETag (W/…) never matches under If-Match, so it isn't sent. A change made here while the server answered starts the round again too.
  • When. A few seconds after a change here (skipped when everything still matches the base, so the sync's own writes don't start another), on startup and after an update, and when a search page asks, at most every 5 minutes. That's the moment changes matter, and it needs no alarms permission.
  • Consent. Firefox counts sending anything off the browser as data transmission. The list is a set of sites, browsingActivity in its terms, declared as optional and requested with the host when the user presses Connect. Settings might also count as technicalAndInteraction; not declared, since that one shows a switch at install for a feature most people won't use. The login stays in local: storage, never in sync.
  • Verified: merge and sync logic in unit tests with two browsers and a fake server (tests/webdav.test.ts), including another browser's save in between and a file Anubis can't read (left alone). In Chromium (node e2e/run.mjs webdav), against a mock server with no CORS headers: Connect, the file created with If-None-Match: *, a site added on the server arriving on Sync now, and a settings change here reaching the server a few seconds later.
  • Consent regression (2026-09-29): the first permission request used whether consent was already granted to decide whether to ask for it, so a fresh Firefox 140 connection omitted the data prompt. Background sync also treated a missing consent grant as an unsupported browser and fell back to checking only the host; if checking data_collection threw, it made the same fallback. Either path could send the list without consent. Requests now require browsingActivity before any transfer. Anubis now requires Firefox 142 or later in its manifest; tests ensure a missing permission or failed consent check cannot transfer data.
  • Still unverified: real servers (Koofr, InfiniCLOUD, Nextcloud, where signed-in cookies are one known trap, so requests send none) and Firefox's actual consent prompt in a real browser.
  • Found on the way: a list from a host other than GitHub, arriving through sync or a backup, couldn't download in the other browser, and Update now didn't ask for the host. It now asks, from the click.

• Sync (2026-09-29): browser sync already carried everything but downloaded lists, in Firefox and Chrome alike, with nothing to set up in Anubis. Three weak points, now handled:

  • Size. Sync holds about 100 KB, so a list of about 3,300 sites was the most that synced, and importing Brave's Hacker News Goggle (6,238 sites, 174 KB as text) left the list on one computer. The list is now compressed before it's split into chunks (deflate-raw through the browser's CompressionStream, then base64, as uBlacklist does with LZString): that Goggle takes 57 KB, about 3× less, so some 10,000 sites fit. Where compression isn't available (before Chrome 103 or Firefox 113) the list is saved as plain text, as before. Settings → Sync (then under Share and back up) shows the room used.
  • Lists read short. Chrome syncs each chunk on its own, so another computer could read the new count of chunks while some still held old text, or chunks from two computers mixed, and an edit made then saved the broken list. The meta item now carries a checksum of the whole text; a list that doesn't match it isn't used, and this computer's last good copy (local:personalCopy) stands in. An edit waits 1.5 seconds for the rest to arrive before saving on top of the copy. Watching the chunks as well as the meta item re-reads the list when late chunks land. The copy also saves unpacking the chunks on every search page. Checked in Chromium (node e2e/run.mjs sync): a change from the result menu is saved compressed with its checksum, and a count of chunks arriving before the chunks keeps the old list on the page until they land.
  • Nothing said how to turn it on. Firefox only syncs add-ons' data with Add-ons ticked in its sync settings, about every 10 minutes, and Firefox for Android doesn't sync it at all. Settings and the guide's new page (Syncing between computers) now say so.
  • Still unverified: syncing between two real computers (the sandbox had one browser and no account), and CompressionStream in Firefox content scripts, where streams have had trouble with Xray wrappers in the past. If it fails there, an edit from a search page saves plain text, and the background script, which reads the list whenever it arrives from sync, keeps the copy that search pages read up to date.
  • Not done here: syncing between Firefox and Chrome, which browser sync can't do. See the next entry.

• The personal list is text in the list format (so it can be published as-is), stored in storage.sync split into chunks of at most 7 KB, since sync allows 8 KB per item and about 100 KB in total. If it outgrows sync it falls back to local storage and settings say so.

• Bug found: the first chunking split on newlines and re-joined with \n, which would have corrupted a single line longer than a chunk. Chunks are now cut by encoded size and joined with nothing; a test round-trips 3,000 lines with non-ASCII text.

• Default subscriptions are not written to storage on install. On a second device, sync data can arrive after onInstalled, and writing defaults would overwrite the user's synced subscriptions. Instead, "no stored subscriptions" means "the defaults".

• The old sync:blockedSites array is migrated into the personal list on install and on first read.

• Overlapping writes (2026-09-29): every change reads the stored value, changes it and writes it back. Only personal list edits were queued, so two quick changes on one page could read the same value and one would be lost: two engine switches, two clean-up switches, or two lists subscribed from More lists at once. Settings, tag choices, subscriptions, and the downloaded lists now each have a queue per page (writeQueue in utils/storage.ts), and changes that depend on the current value (one engine among many) are made inside it.

• Not done: one queue across pages. Content scripts can't share a lock with extension pages (navigator.locks belongs to the search page's origin there), so it would mean sending every change to the background script as data, a rewrite of every edit. Losing a change that way takes two changes in two windows within a few milliseconds of each other.

• Slow patterns (2026-09-29): a list's /regex/ rules run on every result of every search, in the page, and a pattern like /(a+)+$/ backtracks for seconds on the wrong address. JavaScript can't stop a running regex, so they're checked when the list is read: a group that repeats something and is repeated itself (nestedRepeat in utils/listformat.ts) or a pattern over 1,000 characters is skipped as a list error. The check goes by structure, so it also turns down a few safe patterns like ([a-z]+\.)*. None of the seven third-party lists in the directory has a repeated group at all (the Huge AI Blocklist's 92 regexes are all like /pinterest.+\/name/, under 100 characters), so none lost a rule.

• "Update all" could be ignored (2026-09-29): the background runs one update at a time, and "Update all" joined a routine check already running (which only updates lists that are due) instead of updating every list. It now runs straight after that check.

• Import took half a minute (2026-09-29): importing a big list rewrote the whole personal list once per site, so Brave's Hacker News Goggle (6,238 sites) froze Settings for 33 seconds. Sites are now gathered and written in one pass (setSites in utils/personal.ts): 42 ms, with byte-identical results for every list in the directory and the test imports.

Subscriptions and permissions

• Verified end to end: pressing Subscribe on "Stack Overflow copies" in the settings downloaded the real file from raw.githubusercontent.com inside the extension, with no host permission. (In the development sandbox Chromium had to be told to trust the network proxy's CA; e2e/run.mjs does that by public key when PROXY_CA_CERT is set, rather than turning certificate checks off.)

• raw.githubusercontent.com and gist.githubusercontent.com send Access-Control-Allow-Origin: *, so lists there download with no host permission at all. Other hosts ask for permission to that one host when you subscribe (optional_host_permissions on Chrome, optional_permissions on Firefox MV2, which WXT doesn't convert automatically).

• Updates run when the browser starts and when a search page loads, at most every 30 minutes, instead of using the alarms permission.

• Real lists parsed while testing: Brave's Hacker News Goggle (6,238 rules in 14 ms), Tech blogs, Rust, Copycats, and No Pinterest Goggles; laylavish's AI blocklist in uBlacklist format (1,673 rules; one line skipped for a TLD wildcard, which isn't a valid match pattern); arosh's Stack Overflow and GitHub copy lists. A README fetched by mistake parses to zero rules and is rejected.

Subscribe links (2026-09-29)

Subscribe on the lists directory, and links list authors share, lead to the guide's subscribe?url=…&name=… page, the same shape as uBlacklist's. Anubis opens Settings → Lists with a "Subscribe to …?" panel there; nothing is added until you press Subscribe, since anyone can make a link. A list from the directory shows the directory's name and description, whatever name the link gives.

• How the link reaches the extension. uBlacklist redirects its subscribe page to its options page with a declarativeNetRequest rule. That needs host access to its site, which it asks for with a separate "enable subscription links" button (so the first link someone follows does nothing), and the options page listed in web_accessible_resources, which lets any page frame it. Anubis uses a content script that matches only the subscribe page: no new permission, no web-accessible page, and it works from the first click. The cost is one more site in the install prompt (bishop-v.github.io), explained in the privacy page and the store notes.
• Where settings open. Turning the subscribe page's own tab into settings (tabs.update) would leave the subscribe page in its history, so Back would open settings again, and Chrome has no way to replace the entry (Firefox's loadReplace). Settings open in a new tab beside it instead, and the subscribe page goes back to where the link was, or closes if it was opened in a tab of its own. Back or Forward onto it does nothing (its navigation type is back_forward); the page itself explains and links to itself.
• VitePress's router. It follows same-site links without loading a page, and a content script only runs on a page load, so a plain link on the directory would never reach Anubis. Its router leaves links with a target alone (seen in its source), so the directory's carry target="_self"; the check against the built site below confirms it.
• Checked in Chromium: node e2e/run.mjs subscribe-link (mock directory, subscribe page, and list; the directory tab goes back, a new tab closes, Forward doesn't reopen settings, a second link says you already subscribe), and the built docs site served at its real address, with and without the extension. Not yet run in Firefox, nor on the live site, which isn't published until GitHub Pages is on.

Importing from other tools

• Shipped: Settings → Share and back up → Import sites takes uBlacklist rules, a HOHSER JSON export, a Goggle, or a plain domain list and merges whole-site entries into your list. HOHSER's FULL_HIDE becomes Hide, PARTIAL_HIDE becomes Lower, and its three highlight colours become highlight-1..3 tags set to highlight. Rules that need URL patterns or regular expressions can't be expressed as whole-site entries, so they're counted and left out, with a pointer to subscribe to the original list instead.
• Bug found: the confirmation vanished. Saving triggers a debounced re-render of the section, which ran after the message had already been shown once. Messages now stay for a few seconds across re-renders (entrypoints/options/flash.ts); the Lists section had the same problem.