perf: allow request bursts, reuse album track metadata; feat: qobuz lyrics

Two related changes: a 68-track Qobuz album was gated by the API throttle
rather than the network, and Qobuz lyrics are now supported, which is what the
faster album path had to be designed around.

--- perf ---

ratelimit: token bucket instead of fixed spacing
  The limiter spaced every request exactly interval apart with no burst, so at
  the default 60 requests/minute it allowed exactly one request per second no
  matter how many workers ran. Measured: 8 requests took 7s with 1 worker and
  7s with 6, i.e. max_connections had no effect on anything touching the API.
  Since a track needs its metadata and stream URL before downloading, workers
  could only start a track every two seconds, which is why an album looked
  serial even at 57 MiB/s per track.

  New() now takes a burst (providers pass max_connections), idle time accrues
  budget up to that burst, and burst is clamped to one minute's worth of
  requests so a large max_connections cannot bypass a small configured rate.

app: reuse the track objects already in the album payload
  ripAlbum fetched the album, kept only the track ids, then asked the provider
  for every track again, doubling the request count. Providers opt in via
  AlbumTracksAreComplete(); only Qobuz does, because it is only verified there
  and because Tidal does extra work in its track lookup.

  Verified against the live Qobuz API on a 68-track release: the embedded track
  objects already carry title, version, track_number, media_number,
  parental_warning, isrc, copyright, performer, performers, composer, duration,
  audio_info, maximum_bit_depth, maximum_sampling_rate and
  release_date_original. The only field a per-track lookup adds that this app
  reads is the nested album object, so albumTrackMetadata attaches the parent
  album with its own track list stripped, or a 68-track album would hold 68
  copies of itself. trackMetaUsable refetches anyway when title or track number
  is missing, so a thin payload cannot produce a wrongly named file.

  Measured end-to-end through the real app and limiter at the shipped defaults,
  68 tracks: 137 API calls / 2m11s before, 69 calls / 1m3s after.

--- qobuz lyrics ---

Qobuz's main API does not serve lyrics: has_lyrics is only a flag, extra=lyrics
adds nothing, and track/getLyrics does not exist. Its apps use a two-step flow
instead, confirmed live:

  GET track/lyricsUrl?track_id=&app_id=&request_ts=&request_sig=
    -> {"track_id","album_id","lyrics_url"}      (signed CloudFront URL)
  GET <lyrics_url>
    -> {"original":{"type","lang","lines":[{"line","start","end"}]},
        "translation_langs":[...],"writers","publishers":[...]}

type is plain, lsync or wsync; start/end are milliseconds; the line text is
under "line". Passing a language returns a translation instead of the original,
and only for languages that exist for the track (en 403s on an English song),
so the language parameter is deliberately omitted. A 403 on the signed URL is
how Qobuz represents "no lyrics", so it is not treated as an error.

Lyrics are exposed through a TrackLyrics method rather than inside GetMetadata,
because album rips skip the per-track metadata call. That keeps the fast path:
with lyrics on, an album costs the same two provider calls per track it always
did, and the lyrics document itself comes from a CDN that does not consume the
provider's request budget. With metadata.exclude = ["lyrics"] it drops to one
call per track and the full speedup applies.

Existing behaviour is preserved: Tidal and Deezer already attach lyrics in
their metadata, so attachLyrics leaves them alone, and a lyrics failure is
logged and ignored rather than failing a download.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
JorenandClaude Opus 5 committed 2026-10-03 21:55:48 +02:00
1 parent 26fb466afc
commit da4add57fe
13 files changed
+1029 -31

No files matched your search

+49 -14
View File
@@ -1,3 +1,5 @@
// Package ratelimit provides the API request throttle shared by the provider
// clients.
package ratelimit
import (
@@ -6,20 +8,54 @@ import (
"time"
)
// Limiter is a token bucket: it enforces a sustained rate of
// requestsPerMinute while allowing up to burst requests to proceed
// immediately when budget has accrued.
//
// It used to space every request exactly interval apart, with no burst. That
// made max_connections meaningless for anything that goes through the API:
// with the default 60 requests/minute, six download workers each needing a
// metadata call and a stream-URL call could only start one track per two
// seconds, so an album appeared to download serially even though the transfers
// themselves were fast.
type Limiter struct {
interval time.Duration
burst int
mu sync.Mutex
mu sync.Mutex
// next is the time the next token becomes available. It is allowed to
// trail now by up to (burst-1)*interval, which is what accrues burst
// budget while no requests are being made.
next time.Time
}
func New(requestsPerMinute int) *Limiter {
// New returns a limiter allowing requestsPerMinute sustained, with burst
// requests permitted back-to-back. A non-positive requestsPerMinute disables
// throttling entirely.
//
// burst is clamped to at least 1 and to at most one minute's worth of
// requests, so a very low rate cannot be bypassed by a large burst.
func New(requestsPerMinute int, burst int) *Limiter {
if requestsPerMinute <= 0 {
return &Limiter{}
}
return &Limiter{interval: time.Minute / time.Duration(requestsPerMinute)}
if burst < 1 {
burst = 1
}
if burst > requestsPerMinute {
burst = requestsPerMinute
}
return &Limiter{
interval: time.Minute / time.Duration(requestsPerMinute),
burst: burst,
}
}
// Wait blocks until the caller may issue a request, or until ctx is done.
//
// Budget is reserved at call time, so a cancelled Wait still consumes its
// slot. That keeps the sustained rate honest under cancellation, at the cost
// of briefly under-using the budget.
func (l *Limiter) Wait(ctx context.Context) error {
if l.interval <= 0 {
return nil
@@ -27,22 +63,21 @@ func (l *Limiter) Wait(ctx context.Context) error {
l.mu.Lock()
now := time.Now()
if l.next.IsZero() {
l.next = now.Add(l.interval)
l.mu.Unlock()
return nil
// Accrue budget for idle time, capped at burst tokens.
earliest := now.Add(-time.Duration(l.burst-1) * l.interval)
if l.next.Before(earliest) {
l.next = earliest
}
wake := l.next
if now.After(wake) {
l.next = now.Add(l.interval)
l.mu.Unlock()
return nil
}
l.next = l.next.Add(l.interval)
l.mu.Unlock()
timer := time.NewTimer(time.Until(wake))
delay := time.Until(wake)
if delay <= 0 {
return nil
}
timer := time.NewTimer(delay)
defer timer.Stop()
select {
+92 -4
View File
@@ -2,19 +2,25 @@ package ratelimit
import (
"context"
"sync"
"testing"
"time"
)
func TestLimiterDisabled(t *testing.T) {
l := New(-1)
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("Wait() error = %v", err)
for _, rpm := range []int{-1, 0} {
l := New(rpm, 6)
for i := 0; i < 10; i++ {
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("New(%d).Wait() error = %v", rpm, err)
}
}
}
}
func TestLimiterContextCancel(t *testing.T) {
l := New(1)
// burst 1 means strict spacing, so the second request must block.
l := New(1, 1)
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("first Wait() error = %v", err)
}
@@ -26,3 +32,85 @@ func TestLimiterContextCancel(t *testing.T) {
t.Fatalf("expected context error")
}
}
// TestLimiterAllowsBurst is the regression test for album rips appearing
// serial: with the old strict-spacing limiter, max_connections workers could
// not get their API calls through concurrently, so only one track downloaded
// at a time.
func TestLimiterAllowsBurst(t *testing.T) {
const burst = 6
l := New(60, burst) // the shipped defaults
start := time.Now()
for i := 0; i < burst; i++ {
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("Wait() %d error = %v", i, err)
}
}
elapsed := time.Since(start)
// 60 rpm is one token per second; the burst must not have waited.
if elapsed > 200*time.Millisecond {
t.Fatalf("%d burst requests took %v, want them to proceed immediately", burst, elapsed)
}
}
// TestLimiterEnforcesSustainedRate checks the burst does not become a way to
// exceed the configured rate: once the bucket is drained, requests are spaced
// by the interval again.
func TestLimiterEnforcesSustainedRate(t *testing.T) {
// 6000 rpm = 10ms per token, so this stays fast.
const burst = 3
l := New(6000, burst)
ctx := context.Background()
for i := 0; i < burst; i++ {
if err := l.Wait(ctx); err != nil {
t.Fatal(err)
}
}
// The bucket is drained; the next two must each wait about one interval.
start := time.Now()
for i := 0; i < 2; i++ {
if err := l.Wait(ctx); err != nil {
t.Fatal(err)
}
}
elapsed := time.Since(start)
if elapsed < 15*time.Millisecond {
t.Fatalf("post-burst requests took %v, want them throttled to ~10ms each", elapsed)
}
}
// TestLimiterBurstCappedByRate stops a large max_connections from letting a
// deliberately tiny rate through in one go.
func TestLimiterBurstCappedByRate(t *testing.T) {
l := New(2, 64)
if l.burst != 2 {
t.Fatalf("burst = %d, want it clamped to the rate (2)", l.burst)
}
}
func TestLimiterConcurrentCallersShareTheRate(t *testing.T) {
const burst = 8
l := New(6000, burst)
var wg sync.WaitGroup
start := time.Now()
for i := 0; i < burst; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := l.Wait(context.Background()); err != nil {
t.Errorf("Wait() error = %v", err)
}
}()
}
wg.Wait()
// All within the burst, so none should have blocked meaningfully.
if elapsed := time.Since(start); elapsed > 200*time.Millisecond {
t.Fatalf("%d concurrent burst requests took %v", burst, elapsed)
}
}