fix(app): fold accents and punctuation in normalize_query
normalize_query only split whitespace, joined and lowercased, so beyonce and beyoncé, ac/dc and acdc, sigur ros and sigur rós were DISTINCT keys - in the cache and, more importantly, in the learned ranking. Whatever the user taught under one spelling was invisible under the other. It is a miss problem only: folding can merge two keys but can never split one, so no false positive is possible. Punctuation is DROPPED rather than turned into a space, and that choice is what makes ac/dc match acdc - a space would give "ac dc" and leave the two apart, which is the case the fold exists for. Existing whitespace is untouched, so "me and mr. johnson" still becomes "me and mr johnson". The fold is hand-written instead of NFKD plus combining-mark stripping, because that needs unicode-normalization, which is not in this workspace's lock file and would be a new dependency in a crate BOTH frontends compile. The table covers Latin-1 Supplement and common Latin Extended-A; anything outside it passes through unchanged and keeps its own key, exactly as before. Non-Latin scripts are asserted to survive intact. Changing the key orphans every bucket persisted under the old rule, so the loader re-keys on load and SUMS the scores of any two buckets that collapse into one - the same arithmetic the user would have produced by interacting with both spellings under the new rule. Re-keying is idempotent, so it costs one pass and then nothing, and a load never writes: opening the app must not rewrite the file. Measured against the owner's real ranking store BEFORE this was written, as the contract requires: 143 buckets, 9 re-keyed, 0 collisions. So on the machine that matters this is a pure re-key with nothing merged and nothing lost. The collision arithmetic still exists because a future collision is possible, and it is covered by a test built from that same shape. Contract: qbz-nix-docs/qt-frontend/2026-08-03-cortinilla-parity, commit S-N3 (owner ruling R8, option i). Gate: cargo check -p qbz-app clean, cargo test -p qbz-app 207 passed (2 new: the fold, the migration). cc
This commit is contained in:
2 files changed
+171
-8
No files matched your search
@@ -53,16 +53,79 @@ pub const ARTIST_CACHE_FILE: &str = "search_artist_cache.json";
|
||||
// Key normalization — THE cache key
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Normalize a raw query into the canonical cache key: lowercase, trimmed,
|
||||
/// internal whitespace runs collapsed to single spaces.
|
||||
/// Normalize a raw query into the canonical cache key: accents folded to
|
||||
/// their ASCII base, punctuation dropped, lowercased, trimmed, internal
|
||||
/// whitespace runs collapsed to single spaces.
|
||||
///
|
||||
/// The accent + punctuation folding was added 2026-08-03. Before it,
|
||||
/// `beyonce` / `beyoncé`, `ac/dc` / `acdc` and `sigur ros` / `sigur ros` with
|
||||
/// an accent were DISTINCT keys, so whatever the user taught the ranking under
|
||||
/// one spelling was invisible under the other. It is a MISS problem only — no
|
||||
/// false positive is possible, because folding can merge two keys but never
|
||||
/// splits one.
|
||||
///
|
||||
/// Punctuation is DROPPED, not turned into a space, and that choice is what
|
||||
/// makes `ac/dc` and `acdc` the same key — mapping it to a space would give
|
||||
/// `ac dc` and leave the two apart, which is the case the fold exists for.
|
||||
/// Existing whitespace is untouched, so `me and mr. johnson` still becomes
|
||||
/// `me and mr johnson`.
|
||||
///
|
||||
/// This is the ONLY definition of the cache key; `search_service.rs` and
|
||||
/// `search_ranking.rs` import it from here.
|
||||
pub fn normalize_query(q: &str) -> String {
|
||||
q.split_whitespace()
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ")
|
||||
.to_lowercase()
|
||||
let folded: String = q
|
||||
.chars()
|
||||
.map(fold_char)
|
||||
.filter(|c| c.is_alphanumeric() || c.is_whitespace())
|
||||
.collect();
|
||||
folded.split_whitespace().collect::<Vec<_>>().join(" ").to_lowercase()
|
||||
}
|
||||
|
||||
/// Fold one character to its unaccented ASCII base, or return it unchanged.
|
||||
///
|
||||
/// Hand-written rather than NFKD + combining-mark stripping, because that
|
||||
/// needs `unicode-normalization`, which is not in this workspace's lock file
|
||||
/// and would be a new dependency in a crate BOTH frontends compile. The table
|
||||
/// covers Latin-1 Supplement and the common Latin Extended-A letters, which is
|
||||
/// the realistic space for music metadata; anything outside it passes through
|
||||
/// unchanged and simply keeps its own key, exactly as before.
|
||||
///
|
||||
/// Verified against the owner's real ranking store (143 buckets): 9 keys
|
||||
/// change form, 0 collide.
|
||||
fn fold_char(c: char) -> char {
|
||||
match c {
|
||||
'à' | 'á' | 'â' | 'ã' | 'ä' | 'å' | 'ā' | 'ă' | 'ą' => 'a',
|
||||
'À' | 'Á' | 'Â' | 'Ã' | 'Ä' | 'Å' | 'Ā' | 'Ă' | 'Ą' => 'A',
|
||||
'ç' | 'ć' | 'č' | 'ĉ' | 'ċ' => 'c',
|
||||
'Ç' | 'Ć' | 'Č' | 'Ĉ' | 'Ċ' => 'C',
|
||||
'ď' | 'đ' => 'd',
|
||||
'Ď' | 'Đ' => 'D',
|
||||
'è' | 'é' | 'ê' | 'ë' | 'ē' | 'ĕ' | 'ė' | 'ę' | 'ě' => 'e',
|
||||
'È' | 'É' | 'Ê' | 'Ë' | 'Ē' | 'Ĕ' | 'Ė' | 'Ę' | 'Ě' => 'E',
|
||||
'ğ' | 'ĝ' | 'ġ' | 'ģ' => 'g',
|
||||
'Ğ' | 'Ĝ' | 'Ġ' | 'Ģ' => 'G',
|
||||
'ì' | 'í' | 'î' | 'ï' | 'ī' | 'ĭ' | 'į' | 'ı' => 'i',
|
||||
'Ì' | 'Í' | 'Î' | 'Ï' | 'Ī' | 'Ĭ' | 'Į' | 'İ' => 'I',
|
||||
'ł' | 'ĺ' | 'ľ' | 'ļ' => 'l',
|
||||
'Ł' | 'Ĺ' | 'Ľ' | 'Ļ' => 'L',
|
||||
'ñ' | 'ń' | 'ň' | 'ņ' => 'n',
|
||||
'Ñ' | 'Ń' | 'Ň' | 'Ņ' => 'N',
|
||||
'ò' | 'ó' | 'ô' | 'õ' | 'ö' | 'ø' | 'ō' | 'ŏ' | 'ő' => 'o',
|
||||
'Ò' | 'Ó' | 'Ô' | 'Õ' | 'Ö' | 'Ø' | 'Ō' | 'Ŏ' | 'Ő' => 'O',
|
||||
'ŕ' | 'ř' | 'ŗ' => 'r',
|
||||
'Ŕ' | 'Ř' | 'Ŗ' => 'R',
|
||||
'ś' | 'š' | 'ş' | 'ŝ' => 's',
|
||||
'Ś' | 'Š' | 'Ş' | 'Ŝ' => 'S',
|
||||
'ť' | 'ţ' => 't',
|
||||
'Ť' | 'Ţ' => 'T',
|
||||
'ù' | 'ú' | 'û' | 'ü' | 'ū' | 'ŭ' | 'ů' | 'ű' | 'ų' => 'u',
|
||||
'Ù' | 'Ú' | 'Û' | 'Ü' | 'Ū' | 'Ŭ' | 'Ů' | 'Ű' | 'Ų' => 'U',
|
||||
'ý' | 'ÿ' => 'y',
|
||||
'Ý' | 'Ÿ' => 'Y',
|
||||
'ź' | 'ż' | 'ž' => 'z',
|
||||
'Ź' | 'Ż' | 'Ž' => 'Z',
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -259,6 +322,42 @@ fn page<T>(items: Vec<T>) -> SearchResultsPage<T> {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn normalize_folds_accents_and_punctuation() {
|
||||
// The whole point: these pairs used to be DIFFERENT keys, so anything
|
||||
// learned under one spelling was invisible under the other.
|
||||
assert_eq!(normalize_query("Beyoncé"), normalize_query("beyonce"));
|
||||
assert_eq!(normalize_query("Sigur Rós"), normalize_query("sigur ros"));
|
||||
assert_eq!(normalize_query("AC/DC"), normalize_query("acdc"));
|
||||
assert_eq!(normalize_query("Lääz Rockit"), "laaz rockit");
|
||||
assert_eq!(normalize_query("Siempre te extraño"), "siempre te extrano");
|
||||
|
||||
// Punctuation is DROPPED, not spaced — that is what makes ac/dc match
|
||||
// acdc. Existing whitespace survives.
|
||||
assert_eq!(normalize_query("Cro-Mags"), "cromags");
|
||||
assert_eq!(normalize_query("Tom's Diner"), "toms diner");
|
||||
assert_eq!(normalize_query("Me and Mr. Johnson"), "me and mr johnson");
|
||||
|
||||
// The pre-existing contract still holds.
|
||||
assert_eq!(normalize_query(" Pink Floyd "), "pink floyd");
|
||||
|
||||
// Idempotent: a normalized key normalizes to itself. The one-shot
|
||||
// ranking migration depends on this.
|
||||
for q in ["Beyoncé", "AC/DC", "Cro-Mags", " Pink Floyd "] {
|
||||
let once = normalize_query(q);
|
||||
assert_eq!(normalize_query(&once), once, "not idempotent for {q:?}");
|
||||
}
|
||||
|
||||
// Folding can MERGE two keys but must never SPLIT one, or a user would
|
||||
// lose a bucket instead of gaining a match.
|
||||
assert!(!normalize_query("radiohead").is_empty());
|
||||
assert_eq!(normalize_query("radiohead"), "radiohead");
|
||||
|
||||
// Non-Latin scripts pass through unchanged rather than being mangled.
|
||||
assert_eq!(normalize_query("東京"), "東京");
|
||||
assert_eq!(normalize_query("Кино"), "кино");
|
||||
}
|
||||
|
||||
use super::*;
|
||||
|
||||
fn unique_test_dir(name: &str) -> PathBuf {
|
||||
|
||||
@@ -246,10 +246,42 @@ impl SearchRanking {
|
||||
continue;
|
||||
}
|
||||
max_order = max_order.max(bucket.order);
|
||||
self.order.insert(bucket.query.clone(), bucket.order);
|
||||
self.ranking.insert(bucket.query, map);
|
||||
// ONE-SHOT KEY MIGRATION (2026-08-03). `normalize_query` gained
|
||||
// accent + punctuation folding, so a bucket persisted under the
|
||||
// OLD rule can carry a key the new rule would never produce —
|
||||
// `lääz rockit` where lookups now ask for `laaz rockit`. Without
|
||||
// this, every such bucket becomes unreachable and the user
|
||||
// perceives it as "my search suddenly got dumber".
|
||||
//
|
||||
// Re-keying is idempotent: a key already in the new form
|
||||
// normalizes to itself, so this costs one pass and then nothing.
|
||||
// Two old keys CAN collapse into one; their scores are SUMMED,
|
||||
// which is the same arithmetic a user would have produced by
|
||||
// interacting with both spellings under the new rule.
|
||||
//
|
||||
// Measured on the owner's real store before this shipped: 143
|
||||
// buckets, 9 re-keyed, 0 collisions.
|
||||
let key = normalize_query(&bucket.query);
|
||||
match self.ranking.get_mut(&key) {
|
||||
Some(existing) => {
|
||||
for (k, v) in map {
|
||||
let slot = existing.entry(k).or_insert(0);
|
||||
*slot = (*slot + v).min(MAX_SCORE);
|
||||
}
|
||||
// Keep the MORE RECENT of the two recency stamps.
|
||||
let prev = self.order.get(&key).copied().unwrap_or(0);
|
||||
self.order.insert(key, prev.max(bucket.order));
|
||||
}
|
||||
None => {
|
||||
self.order.insert(key.clone(), bucket.order);
|
||||
self.ranking.insert(key, map);
|
||||
}
|
||||
}
|
||||
}
|
||||
self.tick = max_order;
|
||||
// The re-keyed state only reaches disk on the next `record`. That is
|
||||
// deliberate: a load must not write, or merely opening the app would
|
||||
// rewrite the file.
|
||||
}
|
||||
|
||||
/// Serialize the current in-memory state and write it to disk. Best-effort:
|
||||
@@ -517,6 +549,38 @@ mod tests {
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn old_keys_are_migrated_and_collisions_sum() {
|
||||
// A store written under the OLD normalize rule (accents and
|
||||
// punctuation preserved) must stay reachable under the new one.
|
||||
let dir = unique_test_dir("migrate");
|
||||
std::fs::create_dir_all(dir.join("search")).unwrap();
|
||||
let legacy = r#"{
|
||||
"buckets": [
|
||||
{ "query": "beyonce", "order": 1,
|
||||
"entities": [ { "kind": "artist", "id": "1", "score": 3 } ] },
|
||||
{ "query": "beyoncé", "order": 5,
|
||||
"entities": [ { "kind": "artist", "id": "1", "score": 2 },
|
||||
{ "kind": "album", "id": "9", "score": 4 } ] },
|
||||
{ "query": "lääz rockit", "order": 2,
|
||||
"entities": [ { "kind": "artist", "id": "7", "score": 1 } ] }
|
||||
]
|
||||
}"#;
|
||||
std::fs::write(dir.join("search").join("search_ranking.json"), legacy).unwrap();
|
||||
|
||||
let r = SearchRanking::new(&dir);
|
||||
// The two spellings collapsed into ONE bucket and their scores SUMMED.
|
||||
assert_eq!(r.score_for("beyonce", "artist", "1"), 5, "3 + 2");
|
||||
assert_eq!(r.score_for("beyoncé", "artist", "1"), 5, "same bucket now");
|
||||
assert_eq!(r.score_for("beyonce", "album", "9"), 4, "carried over");
|
||||
// A re-keyed bucket with no collision keeps its score and is reachable
|
||||
// under the query the user will actually type.
|
||||
assert_eq!(r.score_for("laaz rockit", "artist", "7"), 1);
|
||||
assert_eq!(r.score_for("Lääz Rockit", "artist", "7"), 1);
|
||||
drop(r);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn background_write_lands_without_a_drop() {
|
||||
// The round-trip test above drops the store first, so it exercises the
|
||||
|
||||
Reference in new issue
Block a user