Author SHA1 Message Date
JorenandClaude Opus 5 d7cc4e639c fix: keep verse breaks in qobuz plain lyrics, handle timestamped blanks
The no-space form after the timestamp is correct and unchanged: LRC is
[mm:ss.xx]text, and the Deezer output already matches. Two real issues behind
that question, though.

Qobuz separates verses with an entry whose line is empty and which has no
start or end at all. Those were dropped entirely, so the plain lyrics tag came
out as one undifferentiated block. They are now kept as single blank lines,
collapsed so a run of separators cannot produce several blanks, and trimmed at
both ends so a tag never begins or ends with one. Checked against the live
track: 41 plain lines with 8 stanza breaks, where before it was 33 flat lines.

They are still excluded from the LRC, which is the correct behaviour rather
than an omission: a synced entry needs a timestamp, and these have none, so
emitting one would park a blank lyric at the wrong moment.

Separately, an empty line that *does* carry a timestamp is a different thing:
a display-clear marker. Qobuz does not appear to emit those, but other lyrics
sources do, and the parser now handles them properly by writing a bare
[mm:ss.xx] entry into the LRC and keeping it out of the plain text.

The LRC itself is unchanged for real data: 33 entries, every one timestamped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 22:05:20 +02:00
JorenandClaude Opus 5 da4add57fe perf: allow request bursts, reuse album track metadata; feat: qobuz lyrics
Two related changes: a 68-track Qobuz album was gated by the API throttle
rather than the network, and Qobuz lyrics are now supported, which is what the
faster album path had to be designed around.

--- perf ---

ratelimit: token bucket instead of fixed spacing
  The limiter spaced every request exactly interval apart with no burst, so at
  the default 60 requests/minute it allowed exactly one request per second no
  matter how many workers ran. Measured: 8 requests took 7s with 1 worker and
  7s with 6, i.e. max_connections had no effect on anything touching the API.
  Since a track needs its metadata and stream URL before downloading, workers
  could only start a track every two seconds, which is why an album looked
  serial even at 57 MiB/s per track.

  New() now takes a burst (providers pass max_connections), idle time accrues
  budget up to that burst, and burst is clamped to one minute's worth of
  requests so a large max_connections cannot bypass a small configured rate.

app: reuse the track objects already in the album payload
  ripAlbum fetched the album, kept only the track ids, then asked the provider
  for every track again, doubling the request count. Providers opt in via
  AlbumTracksAreComplete(); only Qobuz does, because it is only verified there
  and because Tidal does extra work in its track lookup.

  Verified against the live Qobuz API on a 68-track release: the embedded track
  objects already carry title, version, track_number, media_number,
  parental_warning, isrc, copyright, performer, performers, composer, duration,
  audio_info, maximum_bit_depth, maximum_sampling_rate and
  release_date_original. The only field a per-track lookup adds that this app
  reads is the nested album object, so albumTrackMetadata attaches the parent
  album with its own track list stripped, or a 68-track album would hold 68
  copies of itself. trackMetaUsable refetches anyway when title or track number
  is missing, so a thin payload cannot produce a wrongly named file.

  Measured end-to-end through the real app and limiter at the shipped defaults,
  68 tracks: 137 API calls / 2m11s before, 69 calls / 1m3s after.

--- qobuz lyrics ---

Qobuz's main API does not serve lyrics: has_lyrics is only a flag, extra=lyrics
adds nothing, and track/getLyrics does not exist. Its apps use a two-step flow
instead, confirmed live:

  GET track/lyricsUrl?track_id=&app_id=&request_ts=&request_sig=
    -> {"track_id","album_id","lyrics_url"}      (signed CloudFront URL)
  GET <lyrics_url>
    -> {"original":{"type","lang","lines":[{"line","start","end"}]},
        "translation_langs":[...],"writers","publishers":[...]}

type is plain, lsync or wsync; start/end are milliseconds; the line text is
under "line". Passing a language returns a translation instead of the original,
and only for languages that exist for the track (en 403s on an English song),
so the language parameter is deliberately omitted. A 403 on the signed URL is
how Qobuz represents "no lyrics", so it is not treated as an error.

Lyrics are exposed through a TrackLyrics method rather than inside GetMetadata,
because album rips skip the per-track metadata call. That keeps the fast path:
with lyrics on, an album costs the same two provider calls per track it always
did, and the lyrics document itself comes from a CDN that does not consume the
provider's request budget. With metadata.exclude = ["lyrics"] it drops to one
call per track and the full speedup applies.

Existing behaviour is preserved: Tidal and Deezer already attach lyrics in
their metadata, so attachLyrics leaves them alone, and a lyrics failure is
logged and ignored rather than failing a download.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 21:55:48 +02:00
JorenandClaude Opus 5 26fb466afc refactor: finish the jsonutil dedupe; back up config before auto-upgrade
Duplicated helpers
  cmd/rip had its own copies of six helpers that already existed in
  internal/jsonutil: asString (StringFromAny), searchInt (IntFromAny),
  nestedSearchInt and nestedSearchString (NestedAny/NestedString),
  firstNonEmpty (FirstNonEmpty) and firstPositiveInt. firstPositiveInt was
  duplicated a third and fourth time in provider/yandex and provider/beatport,
  so it moves into jsonutil alongside the rest. jsonutil is back to 100%
  coverage.

  searchBool is deliberately left alone, with a comment saying why: it accepts
  only a real bool, and switching it to jsonutil.BoolFromAny would start
  coercing strings and numbers, changing how search results get labelled.

config auto-upgrade is no longer destructive
  UpgradeOutdated re-serializes the whole config from the parsed struct, so any
  key this version does not know about is dropped, and it runs automatically
  from run() without prompting. It now copies the original to
  <path>.bak-<unixtime> (0600, since it holds the same credentials) and
  validates the result before writing. It returns the backup path so the CLI
  can tell the user where it went. The comment states plainly that this is a
  version stamp rather than a migration framework.

cmd/rip tests
  The command bodies are testable now that they are functions returning exit
  codes rather than arms of a switch that called os.Exit. Covers the usage
  errors for every command, the quality floor on `rip id`, config path output,
  the --version flag, and that applyQualityOverride reaches all six sources and
  clamps only the two with a floor. cmd/rip coverage 17% -> 26%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:26:24 +02:00
JorenandClaude Opus 5 b5d4d5688d refactor: consolidate the duplicated download and worker-pool code
internal/download
  The ~40-line progress-bar decorator stack was copy-pasted into three download
  paths and had started to drift; it is now newDownloadBar, which returns a nil
  bar when progress is off (so callers use it unconditionally) plus a finish
  func to defer. The three hand-rolled read/write loops that differed only in
  whether they touched a bar become copyWithProgress.

  FileDeezerEncrypted and FileYandexEncrypted were ~90% identical: mkdir, GET,
  status check, create output, bar, cleanup-on-failure, Content-Length check,
  fsync. That scaffolding is now streamEncrypted, and each function supplies
  only its decryption step. Also switches the io.EOF/io.ErrUnexpectedEOF
  comparisons to errors.Is.

internal/app
  Three copies of "coerce a JSON array that may be []any or []map[string]any"
  become anySlice, which also reports whether the value was an array at all so
  callers can still distinguish missing from empty.

  The worker-pool dispatch existed three times and the copies disagreed: two
  counted failures under a mutex and logged inline with the track id, while the
  third collected errors in a channel and logged them all after wg.Wait()
  without the id. runTrackWorkers unifies them on the inline form, which
  reports failures as they happen and names the track, and keeps the mutex held
  across the log so failure lines cannot interleave.

Tests
  The progress-bar paths had no coverage at all, because progress is disabled
  when not attached to a terminal; four new tests force it on via the existing
  STREAMRIP_GO_FORCE_PROGRESS escape hatch and cover the determinate bar, the
  unknown-length spinner, Deezer decryption with a bar attached, and the
  failure path still removing the partial file. internal/download coverage
  53% -> 63%.

  Three cancellation tests pin the Ctrl-C fix. Verified as genuine regression
  tests by reverting the guards: without them the rip walks all 40 tracks of a
  cancelled album, logs a failure for each, and Rip() returns nil.

Adds store.SQLite.IsFailed, the read complement to MarkFailed, used by the
bookkeeping test to confirm failure records still land under a cancelled
context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:23:24 +02:00
JorenandClaude Opus 5 78d4f0316d refactor: split the command dispatch out of main.go
Completes the restructuring started when main() became run() int. main.go goes
from 1217 lines to 130: global option handling, config loading, signal setup,
and a dispatch switch that delegates to one function per command.

The command bodies move to cmd_rip.go (url, file, id), cmd_search.go (search,
lastfm), cmd_config.go (config, database) and cmd_smoke.go (the developer smoke
commands), each taking (ctx, cfg, gopts, args) and returning an exit code.

They also stop reading the process global. Previously run() did

    os.Args = append([]string{os.Args[0], gopts.command}, gopts.commandArgs...)

and the arms then indexed back into os.Args[2:], os.Args[5:] and so on, so the
already-parsed options were round-tripped through os.Args. The arms now take
the parsed gopts.commandArgs directly, which is a uniform index shift of two
(os.Args[5:] -> args[3:], len(os.Args) < 5 -> len(args) < 3).

No behaviour change. Verified by running each command's argument validation and
help path against the built binary, plus the existing suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:17:24 +02:00
JorenandClaude Opus 5 ff5fb1cdb5 chore: add README, Makefile, CI, lint config and --version
The repo had no README, Makefile, CI, linter config or version string.

- README.md: requirements (ffmpeg/ffprobe, plus yt-dlp for SoundCloud), build,
  the per-source credentials table, usage, the quality scale and the download
  database's migration behaviour. Every claim was checked against the code
  rather than written from memory, which is how the quality-range bug below was
  found. It also records the two deferred refactors (no shared HTTP layer, and
  metadata as map[string]any) so they are documented rather than lore, and
  notes that there is no LICENSE file.
- Makefile: build, install, test, race, cover, fmt, fmt-check, vet, lint,
  check, clean, help. `make build` stamps version/commit/date via -ldflags.
- .github/workflows/ci.yml: gofmt, vet, build, test and test -race, plus a
  lint job. The linter is intentionally unpinned, with a comment: a
  golangci-lint built against an older Go cannot analyse a go1.25 module, which
  is the same reason a locally installed staticcheck built with go1.24 cannot
  run here.
- .golangci.yml: standard set plus bodyclose, errorlint, nilerr and
  govet/copylocks. copylocks matters now that config.Config carries a mutex;
  bodyclose and errorlint target defects this codebase actually has.
- `rip --version` / `-V`, documented in the help output, falling back to the
  VCS revision the toolchain embeds for `go install` builds.

Fixes a bug introduced in the previous commit: Validate capped deezer quality
at 2 and soundcloud at 1, but config.toml.example documents deezer as 2/3/4 and
the global -q flag accepts 0-4, so a documented config would have failed to
load. deezer's buildFormatPriority treats anything >= 2 as FLAC and soundcloud
ignores the value entirely, so the real ceiling is 4 everywhere; only the Qobuz
and Beatport floor of 1 is a genuine constraint. Pinned with a test that walks
the whole documented range.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:14:59 +02:00
JorenandClaude Opus 5 384f32cd67 refactor: move vendor app credentials and signing keys into config
The provider apps' own identifiers and protocol keys were scattered as
constants across six packages. They are not user secrets and stay baked in so
the tool still works out of the box, but they are now mirrored into the config
so that when a provider rotates one, a user can fix their install with a config
edit instead of waiting for a release.

Moved, each with the current value as the default in internal/config:
  qobuz     mobile_app_id, mobile_app_secret
  tidal     app_client_id, app_client_secret
  deezer    api_key, gateway_key, blowfish_secret
  yandex    sign_key, legacy_mp3_sign_salt
  beatport  app_client_id

applyVendorDefaults backfills any empty value, so configs written before these
keys existed keep working and the clients never have to nil-check them. Each
client reads through a small accessor; the package-level signing helpers
(mobileSignRequest, yandexDownloadSign, legacyDirectURL, deriveGatewayAuthToken
and friends, deriveDeezerBlowfishKey) now take the key as a parameter instead
of closing over a constant. The blowfish secret reaches internal/download
through a field on Downloader set by app.New.

The blowfish key derivation XORs 16 bytes of the secret, so a mistyped override
would have panicked; a wrong-length value is now rejected with a warning and
the built-in value is used. Tests pin the length and that the qobuz mobile
secret is valid hex, since the HKDF step requires it.

Also drops soundcloud.client_id and soundcloud.app_version: soundcloud goes
through yt-dlp, which supplies its own, so nothing read them.

config.toml.example documents each new key as "leave empty unless the vendor
rotates theirs".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:11:41 +02:00
JorenandClaude Opus 5 b3b6a2eb1d refactor: delete dead code, dedupe helpers, add tests for untested packages
Dead code
  Removed, each verified to have no non-test caller: promptSearchSelection
  (77 lines, superseded by promptSearchSelectionMenu), AddPlaylistByTrackIDs,
  download.NewWithVerifySSL, qobuz.Client.GetTrackMetadata,
  tidal.Client.trackSupportsAtmos (test-only, and the comment above
  GetDownloadable already documented its removal from the live path) and its
  now-orphaned test, yandex.canonicalTrackRequestID (a no-op returning its
  trimmed input; the call site uses strings.TrimSpace directly) and the unused
  defaultEstimatedKbps constant.

Config keys that did nothing
  qobuz.download_booklets and misc.check_for_updates were declared, defaulted
  and documented in config.toml.example but read by no code. The whole
  [youtube] block went too: there is no YouTube provider. go-toml ignores
  unknown keys, verified, so existing user configs still load.

Duplicated helpers
  tidal.stringify and yandex.firstNonEmpty were byte-identical copies of
  jsonutil.StringFromAny and jsonutil.FirstNonEmpty. displayArtistNames
  existed three times, byte-identical, in internal/app, provider/yandex and
  provider/beatport; it now lives in jsonutil. The CLI's normalizeCodec
  delegates to config.NormalizeCodec so the flag and the config file are
  checked by the same rules.

Tests for packages that had none
  internal/jsonutil (0% -> 100%) and internal/netutil (0% -> 78%), the latter
  covering redactURL, which is a security control, and the zero-timeout
  contract that internal/download and the qobuz body copy depend on.
  internal/config rose to 82% with coverage of the new Validate(), the
  0600 permissions, CloneConfigData not sharing slices, and a concurrent
  UpdateTokens test that exercises the mutex under -race.

Test suite is ~85x faster
  Provider tests built clients with the default 60 requests/minute, so the
  rate limiter slept 1s per HTTP call: deezer took 31s, tidal 11s, qobuz 8s.
  Each provider test package now builds its config through a testConfigData
  helper that disables throttling. Whole suite: ~60s -> 0.7s, and 2.8s
  under -race.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:05:55 +02:00
JorenandClaude Opus 5 201aa5daa7 fix: database migration, config validation and provider client defects
Completes the correctness pass.

Download database migration no longer destroys dedupe state
  The pre-source-column migration carried every legacy row over with
  source='' and then dropped downloads_legacy, while IsDownloaded matched
  source exactly. The result was that every previously recorded download
  looked new, the user silently re-downloaded their whole library, and the
  original rows were gone. Now the file is copied to <path>.bak-<unixtime>
  first, downloads_legacy is kept, and IsDownloaded falls back to the
  sourceless rows (over-matching across sources costs a skipped re-download,
  which is the cheaper error). Schema state is tracked in PRAGMA user_version
  so later migrations can be ordered explicitly. Databases already migrated by
  the old code are detected on open.

  Also enables WAL and a 10s busy_timeout and caps the pool at one connection,
  so a concurrent reader no longer gets SQLITE_BUSY with no retry, and builds
  the DSN through url.URL so a path containing '?' is not parsed as query
  parameters.

--quality now applies to every source
  applyGlobalConfigOverrides set only qobuz, tidal, deezer and soundcloud, so
  `rip -q 4 url <beatport-url>` was a silent no-op even though
  qualityForSource reads Yandex and Beatport. All six are set, with the
  per-source floor applied and reported rather than silently ignored.

config: validation and a fixed reset
  Load checked nothing but misc.version, so a negative max_connections or a
  bogus conversion codec was accepted and only surfaced mid-rip. Adds
  Validate() covering per-source quality ranges, max_connections,
  requests_per_minute and truncate_to, and normalizes the conversion codec
  from the file through the same path as --codec. `config reset` assigned one
  value to both Config.File and Config.Session, so their Secrets slices shared
  a backing array; it now takes separate copies.

  Tests cover the shipped defaults and config.toml.example, which is how the
  over-strict artwork max-width bound in the first draft was caught: -1 is the
  documented "do not downscale" sentinel.

soundcloud: use the shared HTTP client
  It was the only provider constructing its own http.Client, so it silently
  ignored downloads.verify_ssl and max_connections and bypassed the transport
  that redacts credentials from -vv logs. It also had no rate limiter at all.

qobuz: mobile fallback could not download a long track
  http.Client.Timeout bounds the whole request including the body read, so the
  30s API client aborted any track that took longer than 30s to stream. The
  body copy now uses a no-timeout client, as internal/download already does.

qobuz: HKDF no longer fails silently
  hexDecodeOrNil discarded its error, so a non-hex app secret keyed HKDF on
  nil, derived a wrong KEK and produced silently corrupt audio instead of an
  error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 16:58:59 +02:00
JorenandClaude Opus 5 c030d1be09 fix: resource leaks, cancellation, races and path traversal
Six correctness bugs that the existing tests could not see. The suite passed
and `go test -race` was clean throughout, because nothing exercised concurrent
token refresh, shared-folder artwork, or hostile metadata.

main(): extract run() int
  main() was a single 1196-line function that registered 20
  `defer mainApp.Close()` handlers and then called os.Exit 150 times, so none
  of them ever ran. Close() is what closes the SQLite handle, flushes the
  progress bars and removes the artwork temp dirs, so every failed rip leaked
  temp directories. main() is now a wrapper around run() int and the command
  bodies return exit codes.

Ctrl-C now stops a rip
  ctx was threaded everywhere but never checked: zero ctx.Err()/ctx.Done()
  in internal/app and internal/download. Cancellation left Rip() iterating
  the whole remaining queue, and the retry ladder retried on a dead context,
  so one Ctrl-C cost three doomed attempts per remaining track. Adds checks to
  the dispatch loops, the worker-spawn loops and the retry ladder, and routes
  the MarkFailed/MarkDownloaded bookkeeping through a detached context so the
  records survive cancellation. Convert and TagFLAC now take a ctx and use
  exec.CommandContext, so a stalled ffmpeg/ffprobe no longer outlives Ctrl-C.

Config race
  Provider token refresh mutated the shared *config.Config and called
  SaveFile() from inside download workers with no lock, racing the struct
  fields and issuing concurrent os.WriteFile against one path. Adds a mutex to
  config.Config plus UpdateTokens(), and routes all five refresh sites through
  it. Each provider client gets an authMu guarding loggedIn and its
  login/refresh path; beatport needed it most, since ensureToken is reached
  from apiRequest.

Artwork cover race
  artwork.Prepare was check-then-write on a path shared by every worker in a
  playlist rip. Two workers both saw a missing cover.jpg, both created it, and
  the loser's cleanup deleted the winner's finished file. Adds a per-folder
  lock and downloads via a unique temp file + rename. The regression test
  fails against the old code with 15 of 16 workers getting no cover.

Path traversal from remote metadata
  naming.CleanName stripped <>:"/\|?* and control chars but not "." , so a
  title of ".." survived into filepath.Join and escaped the download root.
  SoundCloud and Tidal playlist names are user-generated. Now rejects "." and
  "..", trims trailing dots, handles Windows device names, adds DEL to the
  invalid set and caps components at 255 bytes without splitting a rune.
  app.go gains safeJoin as a second line of defence.

Credentials no longer world-readable
  The config holds plaintext provider passwords, Deezer ARL cookies and OAuth
  tokens and was written 0o644. Now 0o600, with 0o700 directories, and an
  existing 0o644 file is tightened on write.

Also: deletes the 9 duplicated *-rip-smoke dev commands (~300 lines fully
subsumed by `rip id`) and the unreachable second dev-help arm; registers
image/png so PNG covers are downscaled instead of being silently saved as
cover.jpg containing PNG bytes; adds the missing .gitignore, which was keeping
a 21MB build artifact one `git add -A` away from the repo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 16:52:58 +02:00
50 changed files with 5891 additions and 2387 deletions

No files matched your search

+61
View File
@@ -0,0 +1,61 @@
name: CI
on:
push:
branches: [master]
pull_request:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
- name: gofmt
run: |
out="$(gofmt -l .)"
if [ -n "$out" ]; then
echo "These files are not gofmt-clean:"
echo "$out"
exit 1
fi
- name: go vet
run: go vet ./...
- name: Build
run: go build ./...
- name: Test
run: go test ./...
- name: Test (race)
run: go test -race ./...
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
# .golangci.yml uses the v2 config schema, so this needs golangci-lint
# v2.x and action v7+. Left unpinned deliberately: a golangci-lint built
# against an older Go toolchain cannot analyse this module at all (the
# same reason a locally installed staticcheck built with go1.24 fails on
# go.mod's go 1.25). Pin it once a known-good version is confirmed.
- uses: golangci/golangci-lint-action@v7
with:
version: latest
args: --timeout=5m
+21
View File
@@ -0,0 +1,21 @@
# Build output
/rip
/rip.exe
*.test
# Local runtime state
*.db
*.db-wal
*.db-shm
config.toml
*.bak-*
# Scratch files
albums.txt
*.log
coverage.out
# Editor / OS
.DS_Store
.idea/
.vscode/
+39
View File
@@ -0,0 +1,39 @@
version: "2"
run:
timeout: 5m
tests: true
linters:
default: standard
enable:
- bodyclose # unclosed HTTP response bodies: this app makes a lot of requests
- errorlint # == on errors instead of errors.Is; the download code had several
- copyloopvar
- misspell
- nilerr
- unconvert
- usestdlibvars
- whitespace
settings:
govet:
enable:
- copylocks # config.Config carries a mutex and must not be copied
- nilness
staticcheck:
checks:
- all
exclusions:
presets:
- std-error-handling
rules:
# Test helpers legitimately ignore errors on cleanup paths.
- path: _test\.go
linters:
- errcheck
formatters:
enable:
- gofmt
+63
View File
@@ -0,0 +1,63 @@
BINARY := rip
PKG := ./cmd/rip
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
COMMIT ?= $(shell git rev-parse HEAD 2>/dev/null)
DATE ?= $(shell date -u +%Y-%m-%dT%H:%M:%SZ)
LDFLAGS := -X main.version=$(VERSION) -X main.commit=$(COMMIT) -X main.date=$(DATE)
.DEFAULT_GOAL := build
.PHONY: build
build: ## Build the rip binary
go build -ldflags "$(LDFLAGS)" -o $(BINARY) $(PKG)
.PHONY: install
install: ## Install rip into GOBIN
go install -ldflags "$(LDFLAGS)" $(PKG)
.PHONY: test
test: ## Run the test suite
go test ./...
.PHONY: race
race: ## Run the test suite under the race detector
go test -race ./...
.PHONY: cover
cover: ## Run tests and report per-package coverage
go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out | tail -1
.PHONY: fmt
fmt: ## Format all Go files
gofmt -w .
.PHONY: fmt-check
fmt-check: ## Fail if any file is not gofmt-clean
@out="$$(gofmt -l .)"; \
if [ -n "$$out" ]; then echo "not gofmt-clean:"; echo "$$out"; exit 1; fi
.PHONY: vet
vet: ## Run go vet
go vet ./...
.PHONY: lint
lint: ## Run golangci-lint if it is installed
@if command -v golangci-lint >/dev/null 2>&1; then \
golangci-lint run; \
else \
echo "golangci-lint not installed; skipping"; \
echo " go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest"; \
fi
.PHONY: check
check: fmt-check vet test ## Everything CI runs
.PHONY: clean
clean: ## Remove build and coverage artifacts
rm -f $(BINARY) coverage.out
.PHONY: help
help: ## List targets
@grep -hE '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | awk -F':.*?## ' '{printf " %-12s %s\n", $$1, $$2}'
+172
View File
@@ -0,0 +1,172 @@
# streamrip-go
A Go port of [streamrip](https://github.com/nathom/streamrip): a command-line
music downloader for Qobuz, Tidal, Deezer, SoundCloud, Yandex Music and
Beatport. Downloads tracks, albums, playlists, artist discographies and labels,
tags them, optionally embeds cover art and converts the result.
You need your own paid account for any source you use. This tool does not
provide or circumvent access to anything.
## Requirements
- Go 1.25 or newer (to build)
- `ffmpeg` and `ffprobe` — tagging and format conversion
- `yt-dlp` — required for SoundCloud only
```sh
# Arch
sudo pacman -S ffmpeg yt-dlp
# Debian/Ubuntu
sudo apt install ffmpeg && pipx install yt-dlp
# macOS
brew install ffmpeg yt-dlp
```
## Build
```sh
make build # -> ./rip, with version info stamped in
make install # into $(go env GOBIN)
./rip --version
```
Or without make: `go build ./cmd/rip`.
## Configuration
The first run writes a config file and tells you where. It lives at
`$XDG_CONFIG_HOME/streamrip/config.toml` (`~/Library/Application
Support/streamrip/config.toml` on macOS).
```sh
rip config path # print the path
rip config open # open in $EDITOR
rip config reset # restore defaults
```
`config.toml.example` documents every key. The file holds credentials in
plaintext — provider passwords, the Deezer ARL cookie and OAuth tokens — so it
is created `0600`. Keep it that way.
Credentials you need to set, per source:
| Source | Required config |
|------------|-----------------|
| Qobuz | `email_or_userid` + `password_or_token` (md5 of your password), or `use_auth_token = true` with an auth token |
| Tidal | `access_token` / `refresh_token` |
| Deezer | `arl` (recommended), or `email` + `password` |
| Yandex | `access_token` |
| Beatport | `username` + `password` |
| SoundCloud | none; handled via `yt-dlp` |
Lyrics (plain and synced `.lrc`-style) are fetched for Qobuz, Tidal and Deezer.
Each costs one extra API request per track, so `metadata.exclude = ["lyrics"]`
turns them off for a faster rip.
Each provider section also has vendor `app_client_id`-style keys. Leave them
empty: they default to values built into the binary. They exist so that if a
provider rotates one, you can patch your install with a config edit instead of
waiting for a release.
## Usage
```sh
rip url https://play.qobuz.com/album/abc123
rip url https://www.beatport.com/release/example/123 --force
rip id qobuz album abc123
rip search qobuz album "kind of blue"
rip search tidal track "teardrop" --first
rip file urls.txt # one URL per line, or a JSON list of IDs
rip lastfm --source qobuz <lastfm-playlist-url>
rip database downloads # what has already been downloaded
rip database failed
```
Global options (these must come *before* the command):
```
-f, --folder <path> Override downloads folder
-q, --quality <0-4> Override provider quality
-c, --codec <codec> Convert after download: ALAC, FLAC, OGG, MP3, AAC
-ndb, --no-db Ignore the download database
--no-progress Disable progress bars
--no-ssl-verify Disable TLS verification
--config-path <path> Use a custom config file
-v, -vv Verbose logging (-vv also logs HTTP requests)
-V, --version Show version
-h, --help Show help
```
Run `rip help <command>` for per-command help.
### Quality
`-q` is a 0–4 scale mapped onto each provider's own ladder. Qobuz and Beatport
have no tier 0, so `-q 0` is raised to 1 for those and reported on stderr.
| Source | Range | 0 | Top tier |
|------------|-------|---|----------|
| Qobuz | 1–4 | — | FLAC 24-bit up to 192 kHz |
| Tidal | 0–4 | LOW (HE-AAC) | HI_RES_LOSSLESS |
| Deezer | 0–4 | MP3 128 | FLAC (anything ≥ 2) |
| Yandex | 0–4 | LOW | lossless |
| Beatport | 1–4 | — | FLAC 16/44.1 |
| SoundCloud | 0–4 | ignored | provider-defined |
Each provider saturates at the top of its own ladder, so values above its
highest real tier are accepted and behave as that tier.
### Download database
Completed downloads are recorded in a SQLite database and skipped next time.
`--force` / `--ignore-db` overrides this for one run; `-ndb` disables it
entirely. Paths are configurable under `[database]`.
If you are upgrading from a build that predates source tracking, the database
is migrated on first open and a copy is kept alongside it as
`downloads.db.bak-<timestamp>`.
## Development
```sh
make check # gofmt + vet + tests, what CI runs
make test
make race
make cover
make lint # needs golangci-lint
```
The test suite runs in about a second and makes no network requests; provider
tests drive `httptest` servers.
Layout:
```
cmd/rip CLI entry point, one file per command group
internal/app orchestration: queue, resolve, rip
internal/provider/* one client per source, behind provider.Client
internal/download HTTP downloads, progress bars, Deezer/Yandex decryption
internal/audio/{tag,convert} ffmpeg tagging and conversion
internal/{naming,artwork,store,config,netutil,ratelimit,jsonutil,verbose}
```
### Known rough edges
Two deliberate pieces of technical debt, both documented rather than hidden:
- **No shared HTTP layer.** The request/decode/retry sequence is reimplemented
in each provider client, and retry on 429/5xx exists in only two of six. A
`provider/transport` package would remove roughly 700 lines.
- **Metadata is `map[string]any`.** `provider.Client` returns untyped maps, so
each client hand-builds an undeclared "Qobuz-shaped" schema that
`cmd/rip/search.go` then unpicks in a long switch. A typed media model is the
natural next step; `internal/provider/qobuz/model.go` is a started-and-parked
attempt at it.
## License
No license file is present, so this is "all rights reserved" by default. Add
one before distributing. Note that upstream streamrip is GPL-3.0.
+53 -28
View File
@@ -2,6 +2,7 @@ package main
import ( import (
"fmt" "fmt"
"os"
"strconv" "strconv"
"strings" "strings"
@@ -26,6 +27,7 @@ type globalOptions struct {
noSSLVerify bool noSSLVerify bool
verbose int verbose int
help bool help bool
version bool
command string command string
commandArgs []string commandArgs []string
} }
@@ -49,6 +51,9 @@ func parseGlobalArgs(args []string) (globalOptions, error) {
case isHelpArg(arg): case isHelpArg(arg):
opts.help = true opts.help = true
return opts, nil return opts, nil
case arg == "--version" || arg == "-V":
opts.version = true
return opts, nil
case arg == "-ndb" || arg == "--no-db": case arg == "-ndb" || arg == "--no-db":
opts.noDB = true opts.noDB = true
case arg == "--no-progress": case arg == "--no-progress":
@@ -124,16 +129,10 @@ func parseGlobalArgs(args []string) (globalOptions, error) {
return opts, nil return opts, nil
} }
// normalizeCodec delegates to config so the --codec flag and a codec set in
// the config file are validated by the same rules.
func normalizeCodec(raw string) (string, error) { func normalizeCodec(raw string) (string, error) {
codec := strings.ToUpper(strings.TrimSpace(raw)) return config.NormalizeCodec(raw)
switch codec {
case "ALAC", "FLAC", "MP3", "AAC", "VORBIS":
return codec, nil
case "OGG":
return "VORBIS", nil
default:
return "", fmt.Errorf("unsupported codec %q (expected ALAC, FLAC, OGG, MP3, AAC)", raw)
}
} }
func applyGlobalConfigOverrides(cfg *config.Config, opts globalOptions) { func applyGlobalConfigOverrides(cfg *config.Config, opts globalOptions) {
@@ -144,10 +143,7 @@ func applyGlobalConfigOverrides(cfg *config.Config, opts globalOptions) {
cfg.Session.Database.DownloadsEnabled = false cfg.Session.Database.DownloadsEnabled = false
} }
if opts.qualitySet { if opts.qualitySet {
cfg.Session.Qobuz.Quality = opts.quality applyQualityOverride(cfg, opts.quality)
cfg.Session.Tidal.Quality = opts.quality
cfg.Session.Deezer.Quality = opts.quality
cfg.Session.Soundcloud.Quality = opts.quality
} }
if opts.codecSet { if opts.codecSet {
cfg.Session.Conversion.Enabled = true cfg.Session.Conversion.Enabled = true
@@ -161,6 +157,50 @@ func applyGlobalConfigOverrides(cfg *config.Config, opts globalOptions) {
} }
} }
// minQualityForSource is the lowest quality id each source accepts. Qobuz and
// Beatport have no equivalent of the "lowest" tier that 0 means elsewhere.
var minQualityForSource = map[string]int{
"qobuz": 1,
"beatport": 1,
}
// applyQualityOverride applies a global -q/--quality to every source.
//
// It previously set only qobuz, tidal, deezer and soundcloud, so
// `rip -q 4 url <beatport-url>` was a silent no-op even though
// Main.qualityForSource reads Yandex and Beatport too. Sources with a higher
// floor are clamped rather than erroring, because one global flag has to work
// across a mixed queue.
func applyQualityOverride(cfg *config.Config, quality int) {
clamped := map[string]int{}
for _, source := range []string{"qobuz", "tidal", "deezer", "soundcloud", "yandex", "beatport"} {
q := quality
if min, ok := minQualityForSource[source]; ok && q < min {
q = min
clamped[source] = q
}
switch source {
case "qobuz":
cfg.Session.Qobuz.Quality = q
case "tidal":
cfg.Session.Tidal.Quality = q
case "deezer":
cfg.Session.Deezer.Quality = q
case "soundcloud":
cfg.Session.Soundcloud.Quality = q
case "yandex":
cfg.Session.Yandex.Quality = q
case "beatport":
cfg.Session.Beatport.Quality = q
}
}
for _, source := range []string{"qobuz", "beatport"} {
if q, ok := clamped[source]; ok {
fmt.Fprintf(os.Stderr, "note: %s has no quality %d; using %d for %s\n", source, quality, q, source)
}
}
}
func errorWithActionableHint(err error, opts globalOptions) string { func errorWithActionableHint(err error, opts globalOptions) string {
if err == nil { if err == nil {
return "" return ""
@@ -204,18 +244,3 @@ func parseQuality(raw string, min int, max int) (int, error) {
} }
return q, nil return q, nil
} }
func asString(v any) string {
switch t := v.(type) {
case string:
return t
case int:
return strconv.Itoa(t)
case int64:
return strconv.FormatInt(t, 10)
case float64:
return strconv.FormatFloat(t, 'f', -1, 64)
default:
return ""
}
}
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/config"
"golang.org/x/term"
)
// Commands that inspect local state: config and database.
func cmdConfig(_ context.Context, cfg *config.Config, _ globalOptions, args []string) int {
var err error
if len(args) < 1 {
printCommandHelp(os.Stdout, "config")
return 2
}
switch args[0] {
case "open":
vim := false
for _, arg := range args[1:] {
switch arg {
case "-v", "--vim":
vim = true
default:
fmt.Fprintf(os.Stderr, "option error: unknown option %q\n", arg)
return 2
}
}
fmt.Printf("opening file at %s\n", cfg.Path)
if err = openConfigInEditor(cfg.Path, vim); err != nil {
fmt.Fprintf(os.Stderr, "open config error: %v\n", err)
return 1
}
case "reset":
yes := false
for _, arg := range args[1:] {
switch arg {
case "-y", "--yes":
yes = true
default:
fmt.Fprintf(os.Stderr, "option error: unknown option %q\n", arg)
return 2
}
}
if !yes {
if !term.IsTerminal(int(os.Stdin.Fd())) {
fmt.Fprintln(os.Stderr, "reset requires --yes in non-interactive mode")
return 2
}
ok, askErr := promptYesNo(fmt.Sprintf("Are you sure you want to reset the config file at %s? [y/N]: ", cfg.Path))
if askErr != nil {
fmt.Fprintf(os.Stderr, "prompt error: %v\n", askErr)
return 1
}
if !ok {
fmt.Println("reset aborted")
return 0
}
}
def := config.DefaultConfigData()
// Separate copies: assigning the same value to both halves made
// File.Qobuz.Secrets and Session.Qobuz.Secrets share one backing
// array.
cfg.File = config.CloneConfigData(def)
cfg.Session = config.CloneConfigData(def)
if err = cfg.SaveFile(); err != nil {
fmt.Fprintf(os.Stderr, "reset config error: %v\n", err)
return 1
}
fmt.Printf("reset the config file at %s\n", cfg.Path)
case "path":
if len(args) > 1 {
fmt.Fprintf(os.Stderr, "option error: unexpected argument %q\n", args[1])
return 2
}
fmt.Printf("config path: '%s'\n", cfg.Path)
default:
fmt.Fprintf(os.Stderr, "unknown config command: %s\n", args[0])
return 2
}
return 0
}
func cmdDatabase(_ context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 2 || args[0] != "browse" {
printCommandHelp(os.Stdout, "database")
return 2
}
table := strings.ToLower(strings.TrimSpace(args[1]))
switch table {
case "downloads":
rows, listErr := listDownloadsRows(cfg.Session.Database.DownloadsPath)
if listErr != nil {
fmt.Fprintf(os.Stderr, "database browse error: %v\n", listErr)
return 1
}
fmt.Println("downloads database")
fmt.Println("row id")
for i, id := range rows {
fmt.Printf("%02d %s\n", i, id)
}
case "failed":
rows, listErr := listFailedRows(cfg.Session.Database.FailedDownloadsPath)
if listErr != nil && isNoSuchTableErr(listErr) && cfg.Session.Database.FailedDownloadsPath != cfg.Session.Database.DownloadsPath {
rows, listErr = listFailedRows(cfg.Session.Database.DownloadsPath)
}
if listErr != nil {
fmt.Fprintf(os.Stderr, "database browse error: %v\n", listErr)
return 1
}
fmt.Println("failed downloads database")
fmt.Println("row source media_type id")
for i, row := range rows {
fmt.Printf("%02d %s %s %s\n", i, row.Source, row.MediaType, row.ID)
}
default:
fmt.Fprintf(os.Stderr, "invalid database %q. choose downloads or failed\n", table)
return 2
}
return 0
}
+200
View File
@@ -0,0 +1,200 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/config"
)
// Commands that queue media and rip it: url, file and id.
func cmdURL(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 1 {
printCommandHelp(os.Stdout, "url")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
rawArgs := make([]string, 0, len(args))
ignoreDB := false
for _, arg := range args {
if arg == "--force" || arg == "--ignore-db" {
ignoreDB = true
continue
}
rawArgs = append(rawArgs, arg)
}
mainApp.IgnoreDB = ignoreDB || gopts.noDB
added := 0
for _, raw := range rawArgs {
if addURLToQueue(ctx, mainApp, raw) {
added++
}
}
if added == 0 {
fmt.Println("nothing to rip")
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("url rip complete (%d item(s))\n", added)
return 0
}
func cmdFile(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 1 {
printCommandHelp(os.Stdout, "file")
return 2
}
ignoreDB := false
for _, arg := range args[1:] {
switch arg {
case "--force", "--ignore-db":
ignoreDB = true
default:
fmt.Fprintf(os.Stderr, "option error: unknown option %q\n", arg)
return 2
}
}
content, err := os.ReadFile(args[0])
if err != nil {
fmt.Fprintf(os.Stderr, "read file error: %v\n", err)
return 1
}
idItems, urls, repeated, jsonInput, err := parseFileInput(content)
if err != nil {
fmt.Fprintf(os.Stderr, "file parse error: %v\n", err)
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = ignoreDB || gopts.noDB
added := 0
if jsonInput {
fmt.Printf("detected json file. loading %d item(s)\n", len(idItems))
for _, item := range idItems {
if err = mainApp.AddByID(ctx, item.Source, item.MediaType, item.ID); err != nil {
fmt.Printf("add failed: source=%s type=%s id=%s err=%v\n", item.Source, item.MediaType, item.ID, err)
continue
}
added++
}
} else {
if repeated > 0 {
fmt.Printf("found %d repeated url(s)\n", repeated)
}
fmt.Printf("detected list of urls. loading %d item(s)\n", len(urls))
for _, raw := range urls {
if addURLToQueue(ctx, mainApp, raw) {
added++
}
}
}
if added == 0 {
fmt.Println("nothing to rip")
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("file rip complete (%d item(s))\n", added)
return 0
}
func cmdID(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 3 {
printCommandHelp(os.Stdout, "id")
return 2
}
source := strings.ToLower(strings.TrimSpace(args[0]))
mediaType := strings.ToLower(strings.TrimSpace(args[1]))
itemID := strings.TrimSpace(args[2])
opts, err := parseSmokeOptions(args[3:], 0, 4)
if err != nil {
fmt.Fprintf(os.Stderr, "option error: %v\n", err)
return 2
}
if opts.qualitySet {
switch source {
case "qobuz":
if opts.quality < 1 {
fmt.Fprintf(os.Stderr, "quality error: qobuz quality must be 1-4\n")
return 2
}
cfg.Session.Qobuz.Quality = opts.quality
case "tidal":
cfg.Session.Tidal.Quality = opts.quality
case "yandex":
cfg.Session.Yandex.Quality = opts.quality
case "beatport":
if opts.quality < 1 {
fmt.Fprintf(os.Stderr, "quality error: beatport quality must be 1-4\n")
return 2
}
cfg.Session.Beatport.Quality = opts.quality
}
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = opts.ignoreDB || gopts.noDB
if err = mainApp.AddByID(ctx, source, mediaType, itemID); err != nil {
fmt.Fprintf(os.Stderr, "add error: %v\n", err)
return 1
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("id rip complete: source=%s type=%s id=%s\n", source, mediaType, itemID)
return 0
}
+281
View File
@@ -0,0 +1,281 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/config"
"golang.org/x/term"
)
// Commands that discover media before ripping it: search and lastfm.
func cmdSearch(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
var err error
var source, mediaType string
var sopts searchOptions
if len(args) < 3 {
if !term.IsTerminal(int(os.Stdin.Fd())) {
printCommandHelp(os.Stdout, "search")
return 2
}
source, mediaType, sopts, err = promptSearchInteractive(cfg.Session.CLI.MaxSearchResults)
if err != nil {
fmt.Fprintf(os.Stderr, "search prompt error: %v\n", err)
return 2
}
} else {
source = strings.ToLower(strings.TrimSpace(args[0]))
mediaType = strings.ToLower(strings.TrimSpace(args[1]))
sopts, err = parseSearchArgs(args[2:], cfg.Session.CLI.MaxSearchResults)
}
if err != nil {
fmt.Fprintf(os.Stderr, "search option error: %v\n", err)
return 2
}
if sopts.first && sopts.outputFile != "" {
fmt.Fprintln(os.Stderr, "cannot choose --first and --output-file together")
return 2
}
if !isAllowedSearchSource(source) {
fmt.Fprintf(os.Stderr, "unsupported search source %q\n", source)
return 2
}
if !isAllowedMediaType(mediaType) {
fmt.Fprintf(os.Stderr, "unsupported media type %q\n", mediaType)
return 2
}
if source == "soundcloud" && mediaType != "track" && mediaType != "playlist" {
fmt.Fprintln(os.Stderr, "soundcloud search currently supports media types track and playlist")
return 2
}
if source == "yandex" && mediaType != "track" && mediaType != "album" && mediaType != "playlist" && mediaType != "artist" {
fmt.Fprintln(os.Stderr, "yandex search currently supports media types track, album, playlist, and artist")
return 2
}
if source == "beatport" && mediaType != "track" && mediaType != "album" && mediaType != "label" {
fmt.Fprintln(os.Stderr, "beatport search currently supports media types track, album, and label")
return 2
}
if sopts.query == "" {
fmt.Fprintln(os.Stderr, "search query cannot be empty")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, source)
if err != nil {
fmt.Fprintf(os.Stderr, "%s login error: %v\n", source, err)
return 1
}
pages, err := provider.Search(ctx, mediaType, sopts.query, sopts.limit)
if err != nil {
fmt.Fprintf(os.Stderr, "search error: %v\n", err)
return 1
}
results := normalizeSearchResults(source, mediaType, pages)
if len(results) == 0 {
fmt.Println("no results")
return 0
}
if sopts.outputFile != "" {
fmt.Printf("results: %d\n", len(results))
for i, result := range results {
fmt.Printf("%2d. id=%s | %s\n", i+1, result.ID, result.Title)
}
if err = writeSearchResultsToFile(source, mediaType, results, sopts.outputFile); err != nil {
fmt.Fprintf(os.Stderr, "write results error: %v\n", err)
return 1
}
fmt.Printf("wrote %d results to %s\n", len(results), sopts.outputFile)
return 0
}
if sopts.first {
fmt.Printf("results: %d\n", len(results))
fmt.Printf(" 1. id=%s | %s\n", results[0].ID, results[0].Title)
results = results[:1]
}
if sopts.noDownload {
fmt.Printf("results: %d\n", len(results))
for i, result := range results {
fmt.Printf("%2d. id=%s | %s\n", i+1, result.ID, result.Title)
}
return 0
}
if !sopts.first {
fmt.Printf("results: %d\n", len(results))
}
if sopts.first {
selection := []int{0}
mainApp.IgnoreDB = sopts.ignoreDB || gopts.noDB
skippedDownloaded := 0
added := 0
for _, idx := range selection {
item := results[idx]
if !sopts.ignoreDB {
already, checkErr := mainApp.Store.IsDownloaded(ctx, source, item.ID)
if checkErr == nil && already {
skippedDownloaded++
fmt.Printf("skip (already downloaded): id=%s | %s\n", item.ID, item.Title)
continue
}
}
if err = mainApp.AddByID(ctx, source, mediaType, item.ID); err != nil {
fmt.Printf("add failed: id=%s err=%v\n", item.ID, err)
continue
}
added++
}
if added == 0 {
if skippedDownloaded > 0 {
fmt.Println("selected item was already downloaded (use --force to redownload)")
} else {
fmt.Println("nothing selected to download")
}
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("search download complete (%d item(s))\n", added)
return 0
}
if !term.IsTerminal(int(os.Stdin.Fd())) {
fmt.Println("non-interactive input; use `rip id` to download specific results")
return 0
}
selection, err := promptSearchSelectionMenu(source, mediaType, sopts.query, results)
if err != nil {
fmt.Fprintf(os.Stderr, "selection error: %v\n", err)
return 2
}
if len(selection) == 0 {
fmt.Println("download cancelled")
return 0
}
mainApp.IgnoreDB = sopts.ignoreDB || gopts.noDB
skippedDownloaded := 0
added := 0
for _, idx := range selection {
item := results[idx]
if !sopts.ignoreDB {
already, checkErr := mainApp.Store.IsDownloaded(ctx, source, item.ID)
if checkErr == nil && already {
skippedDownloaded++
fmt.Printf("skip (already downloaded): id=%s | %s\n", item.ID, item.Title)
continue
}
}
if err = mainApp.AddByID(ctx, source, mediaType, item.ID); err != nil {
fmt.Printf("add failed: id=%s err=%v\n", item.ID, err)
continue
}
added++
}
if added == 0 {
if skippedDownloaded > 0 {
fmt.Println("all selected items were already downloaded (use --force to redownload)")
} else {
fmt.Println("nothing selected to download")
}
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("search download complete (%d item(s))\n", added)
return 0
}
func cmdLastFM(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
opts, parseErr := parseLastFMArgs(args, cfg.Session.LastFM.Source, cfg.Session.LastFM.FallbackSource)
if parseErr != nil {
fmt.Fprintf(os.Stderr, "lastfm option error: %v\n", parseErr)
printCommandHelp(os.Stdout, "lastfm")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = gopts.noDB
title, tracks, err := fetchLastFMPlaylist(ctx, cfg.Session.Downloads.VerifySSL, opts.PlaylistURL)
if err != nil {
fmt.Fprintf(os.Stderr, "lastfm parse error: %v\n", err)
return 1
}
if len(tracks) == 0 {
fmt.Println("no tracks found in playlist")
return 0
}
fmt.Printf("lastfm playlist: %s (%d tracks)\n", title, len(tracks))
resolvedTracks, err := resolveLastFMTracks(ctx, mainApp, opts, tracks)
if err != nil {
fmt.Fprintf(os.Stderr, "lastfm resolve error: %v\n", err)
return 1
}
if len(resolvedTracks) == 0 {
fmt.Println("no lastfm tracks resolved")
return 0
}
playlistID := fmt.Sprintf("lastfm:%s", strings.ToLower(strings.ReplaceAll(title, " ", "_")))
refs := make([]app.PlaylistTrackRef, 0, len(resolvedTracks))
for _, item := range resolvedTracks {
refs = append(refs, app.PlaylistTrackRef{Source: item.Source, ID: item.ID})
}
if addErr := mainApp.AddMixedPlaylistByTrackRefs(ctx, playlistID, title, refs); addErr != nil {
fmt.Printf("playlist queue failed: err=%v\n", addErr)
fmt.Println("no lastfm playlists queued")
return 0
}
fmt.Printf("queued lastfm playlist: %s (%d tracks)\n", title, len(refs))
if len(refs) == 0 {
fmt.Println("no lastfm playlists queued")
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %v\n", err)
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %v\n", err)
return 1
}
fmt.Printf("lastfm rip complete (%d track(s) across 1 playlist)\n", len(resolvedTracks))
return 0
}
+310
View File
@@ -0,0 +1,310 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/provider"
)
// Developer smoke commands, listed by "rip dev-help". These exercise a single
// provider call and print the result; they are not part of the supported CLI.
func cmdSoundcloudSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip soundcloud-smoke <soundcloud_url>")
return 2
}
meta, err := fetchSoundcloudOEmbed(ctx, cfg.Session.Downloads.VerifySSL, strings.TrimSpace(args[0]))
if err != nil {
fmt.Fprintf(os.Stderr, "soundcloud smoke error: %v\n", err)
return 1
}
fmt.Printf("soundcloud oembed ok: title=%q author=%q provider=%q\n", jsonutil.StringFromAny(meta["title"]), jsonutil.StringFromAny(meta["author_name"]), jsonutil.StringFromAny(meta["provider_name"]))
return 0
}
func cmdQobuzSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip qobuz-smoke <track_id> [quality]")
return 2
}
opts, err := parseSmokeOptions(args[1:], 1, 4)
if err != nil {
fmt.Fprintf(os.Stderr, "option error: %v\n", err)
return 2
}
if opts.qualitySet {
cfg.Session.Qobuz.Quality = opts.quality
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "qobuz")
if err != nil {
fmt.Fprintf(os.Stderr, "qobuz login error: %v\n", err)
return 1
}
trackID := args[0]
meta, err := provider.GetMetadata(ctx, trackID, "track")
if err != nil {
fmt.Fprintf(os.Stderr, "metadata error: %v\n", err)
return 1
}
title, _ := meta["title"].(string)
d, err := provider.GetDownloadable(ctx, trackID, cfg.Session.Qobuz.Quality)
if err != nil {
fmt.Fprintf(os.Stderr, "downloadable error: %v\n", err)
return 1
}
fmt.Printf("qobuz ok: title=%q quality=%d ext=%s\n", title, cfg.Session.Qobuz.Quality, d.Extension)
fmt.Printf("stream_url=%s\n", d.URL)
return 0
}
func cmdQobuzConvertRipSmoke(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 2 {
fmt.Println("usage: rip qobuz-convert-rip-smoke <track_id> <codec> [quality] [--force|--ignore-db]")
return 2
}
opts, err := parseSmokeOptions(args[2:], 1, 4)
if err != nil {
fmt.Fprintf(os.Stderr, "option error: %v\n", err)
return 2
}
if opts.qualitySet {
cfg.Session.Qobuz.Quality = opts.quality
}
cfg.Session.Conversion.Enabled = true
cfg.Session.Conversion.Codec = strings.ToUpper(strings.TrimSpace(args[1]))
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = opts.ignoreDB || gopts.noDB
trackID := args[0]
if err = mainApp.AddByID(ctx, "qobuz", "track", trackID); err != nil {
fmt.Fprintf(os.Stderr, "add error: %v\n", err)
return 1
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %v\n", err)
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %v\n", err)
return 1
}
fmt.Printf("qobuz convert rip smoke complete (codec=%s)\n", cfg.Session.Conversion.Codec)
return 0
}
func cmdQobuzSearchSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip qobuz-search-smoke <query>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "qobuz")
if err != nil {
fmt.Fprintf(os.Stderr, "qobuz login error: %v\n", err)
return 1
}
query := strings.Join(args, " ")
pages, err := provider.Search(ctx, "album", query, 10)
if err != nil {
fmt.Fprintf(os.Stderr, "search error: %v\n", err)
return 1
}
for _, page := range pages {
albums, ok := page["albums"].(map[string]any)
if !ok {
continue
}
items, ok := albums["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
item, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(item["id"])
title := jsonutil.StringFromAny(item["title"])
version := jsonutil.StringFromAny(item["version"])
bitDepth := jsonutil.StringFromAny(item["maximum_bit_depth"])
sr := jsonutil.StringFromAny(item["maximum_sampling_rate"])
if version != "" {
title = title + " (" + version + ")"
}
fmt.Printf("album_id=%s | %s | %sB-%skHz\n", id, title, bitDepth, sr)
}
}
return 0
}
func cmdTidalSearchSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip tidal-search-smoke <query>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "tidal")
if err != nil {
fmt.Fprintf(os.Stderr, "tidal login error: %v\n", err)
return 1
}
query := strings.Join(args, " ")
pages, err := provider.Search(ctx, "album", query, 10)
if err != nil {
fmt.Fprintf(os.Stderr, "search error: %v\n", err)
return 1
}
for _, page := range pages {
items, ok := page["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
wrapper, ok := raw.(map[string]any)
if !ok {
continue
}
item, ok := wrapper["item"].(map[string]any)
if !ok {
item = wrapper
}
fmt.Printf("album_id=%s | %s\n", jsonutil.StringFromAny(item["id"]), jsonutil.StringFromAny(item["title"]))
}
}
return 0
}
func cmdTidalMetadataSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 2 {
fmt.Println("usage: rip tidal-metadata-smoke <track|album|playlist|artist> <id>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "tidal")
if err != nil {
fmt.Fprintf(os.Stderr, "tidal login error: %v\n", err)
return 1
}
mediaType := args[0]
itemID := args[1]
meta, err := provider.GetMetadata(ctx, itemID, mediaType)
if err != nil {
fmt.Fprintf(os.Stderr, "metadata error: %v\n", err)
return 1
}
title := jsonutil.StringFromAny(meta["title"])
if title == "" {
title = jsonutil.StringFromAny(meta["name"])
}
trackCount := 0
if tracksMap, ok := meta["tracks"].(map[string]any); ok {
if items, ok := tracksMap["items"].([]map[string]any); ok {
trackCount = len(items)
} else if anyItems, ok := tracksMap["items"].([]any); ok {
trackCount = len(anyItems)
}
}
fmt.Printf("tidal metadata ok: type=%s id=%s title=%q tracks=%d\n", mediaType, itemID, title, trackCount)
return 0
}
func cmdTidalVideoSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip tidal-video-smoke <video_id>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
providerClient, err := mainApp.GetLoggedInProvider(ctx, "tidal")
if err != nil {
fmt.Fprintf(os.Stderr, "tidal login error: %v\n", err)
return 1
}
videoProvider, ok := providerClient.(interface {
GetVideoDownloadable(context.Context, string) (*provider.Downloadable, error)
})
if !ok {
fmt.Fprintln(os.Stderr, "tidal provider does not support video downloadable")
return 1
}
videoID := strings.TrimSpace(args[0])
meta, err := providerClient.GetMetadata(ctx, videoID, "video")
if err != nil {
fmt.Fprintf(os.Stderr, "video metadata error: %v\n", err)
return 1
}
d, err := videoProvider.GetVideoDownloadable(ctx, videoID)
if err != nil {
fmt.Fprintf(os.Stderr, "video downloadable error: %v\n", err)
return 1
}
title := jsonutil.StringFromAny(meta["title"])
if title == "" {
title = jsonutil.StringFromAny(meta["name"])
}
fmt.Printf("tidal video ok: id=%s title=%q ext=%s\n", videoID, title, d.Extension)
fmt.Printf("stream_url=%s\n", d.URL)
return 0
}
+248
View File
@@ -0,0 +1,248 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"streamrip-go/internal/config"
)
// testConfig returns a config rooted in a temp dir, so command functions can
// run without touching the user's real config or download folder.
func testConfig(t *testing.T) *config.Config {
t.Helper()
tmp := t.TempDir()
d := config.DefaultConfigData()
d.Downloads.Folder = filepath.Join(tmp, "downloads")
d.Database.DownloadsEnabled = false
d.Database.FailedDownloadsEnabled = false
d.CLI.ProgressBars = false
return &config.Config{Path: filepath.Join(tmp, "config.toml"), File: d, Session: d}
}
// The command functions became testable when main() was split into run() int
// plus one function per command; previously they were arms of a 1196-line
// switch that called os.Exit directly.
func TestCommandsRejectMissingArguments(t *testing.T) {
tests := []struct {
name string
run func(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int
args []string
}{
{"url with no args", cmdURL, nil},
{"file with no args", cmdFile, nil},
{"config with no args", cmdConfig, nil},
{"database with no args", cmdDatabase, nil},
{"database without browse", cmdDatabase, []string{"downloads"}},
{"database browse without table", cmdDatabase, []string{"browse"}},
{"id with too few args", cmdID, []string{"qobuz", "album"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := tc.run(context.Background(), testConfig(t), globalOptions{}, tc.args)
if got != 2 {
t.Fatalf("exit code = %d, want 2 (usage error)", got)
}
})
}
}
func TestCmdDatabaseRejectsUnknownTable(t *testing.T) {
if got := cmdDatabase(context.Background(), testConfig(t), globalOptions{}, []string{"browse", "bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdConfigRejectsUnknownSubcommand(t *testing.T) {
if got := cmdConfig(context.Background(), testConfig(t), globalOptions{}, []string{"bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdConfigPathPrintsPath(t *testing.T) {
cfg := testConfig(t)
out := captureStdout(t, func() {
if got := cmdConfig(context.Background(), cfg, globalOptions{}, []string{"path"}); got != 0 {
t.Errorf("exit code = %d, want 0", got)
}
})
if !strings.Contains(out, cfg.Path) {
t.Fatalf("output %q does not contain the config path %q", out, cfg.Path)
}
}
func TestCmdConfigPathRejectsExtraArgument(t *testing.T) {
if got := cmdConfig(context.Background(), testConfig(t), globalOptions{}, []string{"path", "extra"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdFileReportsMissingFile(t *testing.T) {
missing := filepath.Join(t.TempDir(), "nope.txt")
if got := cmdFile(context.Background(), testConfig(t), globalOptions{}, []string{missing}); got != 1 {
t.Fatalf("exit code = %d, want 1 (read error)", got)
}
}
func TestCmdFileRejectsUnknownOption(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "urls.txt")
if err := os.WriteFile(path, []byte("https://play.qobuz.com/album/abc\n"), 0o600); err != nil {
t.Fatal(err)
}
if got := cmdFile(context.Background(), testConfig(t), globalOptions{}, []string{path, "--bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdURLRejectsInvalidURL(t *testing.T) {
// Nothing resolvable is queued, so this reports "nothing to rip" and exits 0.
out := captureStdout(t, func() {
if got := cmdURL(context.Background(), testConfig(t), globalOptions{}, []string{"not-a-url"}); got != 0 {
t.Errorf("exit code = %d, want 0", got)
}
})
if !strings.Contains(out, "nothing to rip") {
t.Fatalf("output %q should report that nothing was queued", out)
}
}
func TestCmdIDRejectsQualityBelowSourceFloor(t *testing.T) {
// Qobuz has no quality 0.
if got := cmdID(context.Background(), testConfig(t), globalOptions{}, []string{"qobuz", "album", "abc", "0"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
if got := cmdID(context.Background(), testConfig(t), globalOptions{}, []string{"beatport", "album", "abc", "0"}); got != 2 {
t.Fatalf("beatport exit code = %d, want 2", got)
}
}
func TestCmdIDRejectsUnknownOption(t *testing.T) {
if got := cmdID(context.Background(), testConfig(t), globalOptions{}, []string{"qobuz", "album", "abc", "--bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestSmokeCommandsRejectMissingArguments(t *testing.T) {
tests := []struct {
name string
run func(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int
args []string
}{
{"soundcloud", cmdSoundcloudSmoke, nil},
{"qobuz", cmdQobuzSmoke, nil},
{"qobuz convert", cmdQobuzConvertRipSmoke, []string{"id"}},
{"qobuz search", cmdQobuzSearchSmoke, nil},
{"tidal search", cmdTidalSearchSmoke, nil},
{"tidal metadata", cmdTidalMetadataSmoke, []string{"track"}},
{"tidal video", cmdTidalVideoSmoke, nil},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := tc.run(context.Background(), testConfig(t), globalOptions{}, tc.args); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
})
}
}
// captureStdout swaps os.Stdout for a pipe while fn runs. The commands print
// user-facing output with fmt.Println rather than taking a writer.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
orig := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
done := make(chan string, 1)
go func() {
var sb strings.Builder
buf := make([]byte, 4096)
for {
n, readErr := r.Read(buf)
if n > 0 {
sb.Write(buf[:n])
}
if readErr != nil {
break
}
}
done <- sb.String()
}()
fn()
_ = w.Close()
os.Stdout = orig
out := <-done
_ = r.Close()
return out
}
func TestVersionOutput(t *testing.T) {
var sb strings.Builder
printVersion(&sb)
out := sb.String()
if !strings.HasPrefix(out, "rip ") {
t.Errorf("version output should start with \"rip \": %q", out)
}
for _, want := range []string{"go:", "target:"} {
if !strings.Contains(out, want) {
t.Errorf("version output missing %q: %q", want, out)
}
}
}
func TestParseGlobalArgsVersionFlag(t *testing.T) {
for _, flag := range []string{"--version", "-V"} {
opts, err := parseGlobalArgs([]string{flag})
if err != nil {
t.Fatalf("parseGlobalArgs(%q) error = %v", flag, err)
}
if !opts.version {
t.Errorf("parseGlobalArgs(%q) did not set version", flag)
}
}
}
func TestApplyQualityOverrideCoversEverySource(t *testing.T) {
cfg := testConfig(t)
applyQualityOverride(cfg, 4)
got := map[string]int{
"qobuz": cfg.Session.Qobuz.Quality,
"tidal": cfg.Session.Tidal.Quality,
"deezer": cfg.Session.Deezer.Quality,
"soundcloud": cfg.Session.Soundcloud.Quality,
"yandex": cfg.Session.Yandex.Quality,
"beatport": cfg.Session.Beatport.Quality,
}
for source, q := range got {
if q != 4 {
t.Errorf("%s quality = %d, want 4; -q must reach every source", source, q)
}
}
}
func TestApplyQualityOverrideClampsSourceFloor(t *testing.T) {
cfg := testConfig(t)
applyQualityOverride(cfg, 0)
if cfg.Session.Qobuz.Quality != 1 {
t.Errorf("qobuz quality = %d, want it clamped to 1", cfg.Session.Qobuz.Quality)
}
if cfg.Session.Beatport.Quality != 1 {
t.Errorf("beatport quality = %d, want it clamped to 1", cfg.Session.Beatport.Quality)
}
if cfg.Session.Tidal.Quality != 0 {
t.Errorf("tidal quality = %d, want 0 preserved", cfg.Session.Tidal.Quality)
}
}
+5 -4
View File
@@ -44,6 +44,7 @@ Global options:
--no-ssl-verify Disable TLS verification --no-ssl-verify Disable TLS verification
--config-path <path> Use a custom config file --config-path <path> Use a custom config file
-v, -vv Verbose logging -v, -vv Verbose logging
-V, --version Show version
-h, --help Show help -h, --help Show help
Run 'rip help <command>' for command-specific help. Run 'rip help <command>' for command-specific help.
@@ -151,10 +152,10 @@ Inspect local download databases.
func printDeveloperHelp(w io.Writer) { func printDeveloperHelp(w io.Writer) {
fmt.Fprint(w, `Developer smoke commands: fmt.Fprint(w, `Developer smoke commands:
soundcloud-smoke soundcloud-smoke
qobuz-smoke, qobuz-rip-smoke, qobuz-convert-rip-smoke qobuz-smoke, qobuz-convert-rip-smoke, qobuz-search-smoke
qobuz-album-rip-smoke, qobuz-playlist-rip-smoke, qobuz-artist-rip-smoke, qobuz-label-rip-smoke
qobuz-search-smoke
tidal-search-smoke, tidal-metadata-smoke, tidal-video-smoke tidal-search-smoke, tidal-metadata-smoke, tidal-video-smoke
tidal-rip-smoke, tidal-album-rip-smoke, tidal-playlist-rip-smoke, tidal-artist-rip-smoke
The per-media-type rip smoke commands were removed; use the equivalent
rip id <source> <media-type> <id>
`) `)
} }
+5 -4
View File
@@ -13,6 +13,7 @@ import (
"strings" "strings"
"streamrip-go/internal/app" "streamrip-go/internal/app"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/urlparse" "streamrip-go/internal/urlparse"
) )
@@ -94,12 +95,12 @@ func parseFileInput(content []byte) ([]fileIDItem, []string, int, bool, error) {
if !ok { if !ok {
return nil, nil, 0, true, fmt.Errorf("json item %d must be an object", i+1) return nil, nil, 0, true, fmt.Errorf("json item %d must be an object", i+1)
} }
source := strings.ToLower(strings.TrimSpace(asString(entry["source"]))) source := strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(entry["source"])))
mediaType := strings.ToLower(strings.TrimSpace(asString(entry["media_type"]))) mediaType := strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(entry["media_type"])))
if mediaType == "" { if mediaType == "" {
mediaType = strings.ToLower(strings.TrimSpace(asString(entry["mediaType"]))) mediaType = strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(entry["mediaType"])))
} }
id := strings.TrimSpace(asString(entry["id"])) id := strings.TrimSpace(jsonutil.StringFromAny(entry["id"]))
if source == "" || mediaType == "" || id == "" { if source == "" || mediaType == "" || id == "" {
return nil, nil, 0, true, fmt.Errorf("json item %d missing source/media_type/id", i+1) return nil, nil, 0, true, fmt.Errorf("json item %d missing source/media_type/id", i+1)
} }
+5 -13
View File
@@ -13,6 +13,7 @@ import (
"time" "time"
"streamrip-go/internal/app" "streamrip-go/internal/app"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
) )
@@ -241,8 +242,8 @@ func extractLastFMTitleArtistPairs(page string) []lastFMTrack {
if len(dataPairs) > 0 { if len(dataPairs) > 0 {
out := make([]lastFMTrack, 0, len(dataPairs)) out := make([]lastFMTrack, 0, len(dataPairs))
for _, m := range dataPairs { for _, m := range dataPairs {
title := html.UnescapeString(strings.TrimSpace(firstNonEmpty(m[1], m[2]))) title := html.UnescapeString(strings.TrimSpace(jsonutil.FirstNonEmpty(m[1], m[2])))
artist := html.UnescapeString(strings.TrimSpace(firstNonEmpty(m[3], m[4]))) artist := html.UnescapeString(strings.TrimSpace(jsonutil.FirstNonEmpty(m[3], m[4])))
if title == "" || artist == "" { if title == "" || artist == "" {
continue continue
} }
@@ -256,8 +257,8 @@ func extractLastFMTitleArtistPairs(page string) []lastFMTrack {
titles := lastFMTitleTagsRe.FindAllStringSubmatch(page, -1) titles := lastFMTitleTagsRe.FindAllStringSubmatch(page, -1)
out := make([]lastFMTrack, 0, len(titles)/2) out := make([]lastFMTrack, 0, len(titles)/2)
for i := 0; i+1 < len(titles); i += 2 { for i := 0; i+1 < len(titles); i += 2 {
titleRaw := strings.TrimSpace(firstNonEmpty(titles[i][1], titles[i][2])) titleRaw := strings.TrimSpace(jsonutil.FirstNonEmpty(titles[i][1], titles[i][2]))
artistRaw := strings.TrimSpace(firstNonEmpty(titles[i+1][1], titles[i+1][2])) artistRaw := strings.TrimSpace(jsonutil.FirstNonEmpty(titles[i+1][1], titles[i+1][2]))
if strings.EqualFold(titleRaw, "Play on YouTube") || strings.EqualFold(artistRaw, "Play on YouTube") { if strings.EqualFold(titleRaw, "Play on YouTube") || strings.EqualFold(artistRaw, "Play on YouTube") {
continue continue
} }
@@ -271,15 +272,6 @@ func extractLastFMTitleArtistPairs(page string) []lastFMTrack {
return out return out
} }
func firstNonEmpty(items ...string) string {
for _, item := range items {
if strings.TrimSpace(item) != "" {
return strings.TrimSpace(item)
}
}
return ""
}
func extractLastFMTracksFromMirrorMarkdown(md string) (string, []lastFMTrack) { func extractLastFMTracksFromMirrorMarkdown(md string) (string, []lastFMTrack) {
lines := strings.Split(strings.ReplaceAll(md, "\r\n", "\n"), "\n") lines := strings.Split(strings.ReplaceAll(md, "\r\n", "\n"), "\n")
title := "" title := ""
+43 -1132
View File
File diff suppressed because it is too large. Load diff
+90 -213
View File
@@ -110,84 +110,6 @@ func parseSearchArgs(args []string, defaultLimit int) (searchOptions, error) {
}, nil }, nil
} }
func promptSearchSelection(results []searchResult) ([]int, error) {
reader := bufio.NewReader(os.Stdin)
for {
fmt.Print("Select results to download (e.g. 1,3-5; a=all; q=cancel): ")
line, err := reader.ReadString('\n')
if err != nil {
return nil, err
}
line = strings.TrimSpace(line)
if line == "" || strings.EqualFold(line, "q") || strings.EqualFold(line, "quit") {
return nil, nil
}
if strings.EqualFold(line, "a") || strings.EqualFold(line, "all") {
out := make([]int, 0, len(results))
for i := range results {
out = append(out, i)
}
return out, nil
}
selected := map[int]struct{}{}
chunks := strings.Split(line, ",")
ok := true
for _, raw := range chunks {
part := strings.TrimSpace(raw)
if part == "" {
continue
}
if strings.Contains(part, "-") {
bounds := strings.SplitN(part, "-", 2)
if len(bounds) != 2 {
ok = false
break
}
start, err1 := strconv.Atoi(strings.TrimSpace(bounds[0]))
end, err2 := strconv.Atoi(strings.TrimSpace(bounds[1]))
if err1 != nil || err2 != nil || start <= 0 || end <= 0 || start > end {
ok = false
break
}
for i := start; i <= end; i++ {
if i > len(results) {
ok = false
break
}
selected[i-1] = struct{}{}
}
if !ok {
break
}
continue
}
idx, err := strconv.Atoi(part)
if err != nil || idx <= 0 || idx > len(results) {
ok = false
break
}
selected[idx-1] = struct{}{}
}
if !ok || len(selected) == 0 {
fmt.Println("Invalid selection, try again.")
continue
}
out := make([]int, 0, len(selected))
for idx := range selected {
out = append(out, idx)
}
for i := 1; i < len(out); i++ {
for j := i; j > 0 && out[j] < out[j-1]; j-- {
out[j], out[j-1] = out[j-1], out[j]
}
}
return out, nil
}
}
func promptSearchSelectionMenu(source, mediaType, query string, results []searchResult) ([]int, error) { func promptSearchSelectionMenu(source, mediaType, query string, results []searchResult) ([]int, error) {
if len(results) == 0 { if len(results) == 0 {
return nil, nil return nil, nil
@@ -408,38 +330,38 @@ func normalizeSearchResults(source, mediaType string, pages []map[string]any) []
if !ok { if !ok {
continue continue
} }
id := asString(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
title := asString(itm["title"]) title := jsonutil.StringFromAny(itm["title"])
if title == "" { if title == "" {
title = asString(itm["name"]) title = jsonutil.StringFromAny(itm["name"])
} }
if version := asString(itm["version"]); version != "" { if version := jsonutil.StringFromAny(itm["version"]); version != "" {
title += " (" + version + ")" title += " (" + version + ")"
} }
artist := nestedSearchString(itm, "artist", "name") artist := jsonutil.NestedString(itm, "artist", "name")
if artist == "" { if artist == "" {
artist = nestedSearchString(itm, "performer", "name") artist = jsonutil.NestedString(itm, "performer", "name")
} }
album := nestedSearchString(itm, "album", "title") album := jsonutil.NestedString(itm, "album", "title")
trackCount := searchInt(itm["tracks_count"]) trackCount := jsonutil.IntFromAny(itm["tracks_count"])
if trackCount == 0 { if trackCount == 0 {
trackCount = searchInt(itm["track_count"]) trackCount = jsonutil.IntFromAny(itm["track_count"])
} }
explicit := searchBool(itm["parental_warning"]) explicit := searchBool(itm["parental_warning"])
date := firstNonEmpty( date := jsonutil.FirstNonEmpty(
asString(itm["release_date_original"]), jsonutil.StringFromAny(itm["release_date_original"]),
asString(itm["release_date"]), jsonutil.StringFromAny(itm["release_date"]),
asString(itm["releaseDate"]), jsonutil.StringFromAny(itm["releaseDate"]),
asString(itm["streamStartDate"]), jsonutil.StringFromAny(itm["streamStartDate"]),
nestedSearchString(itm, "album", "release_date_original"), jsonutil.NestedString(itm, "album", "release_date_original"),
nestedSearchString(itm, "album", "release_date"), jsonutil.NestedString(itm, "album", "release_date"),
) )
releases := 0 releases := 0
if mediaType == "artist" { if mediaType == "artist" {
releases = firstPositiveInt( releases = jsonutil.FirstPositiveInt(
searchInt(itm["albums_count"]), jsonutil.IntFromAny(itm["albums_count"]),
searchInt(itm["albumsCount"]), jsonutil.IntFromAny(itm["albumsCount"]),
nestedSearchInt(itm, "albums", "total"), jsonutil.IntFromAny(jsonutil.NestedAny(itm, "albums", "total")),
) )
} }
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit}) appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
@@ -457,39 +379,39 @@ func normalizeSearchResults(source, mediaType string, pages []map[string]any) []
if wrapped, ok := itm["item"].(map[string]any); ok { if wrapped, ok := itm["item"].(map[string]any); ok {
itm = wrapped itm = wrapped
} }
id := asString(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
title := asString(itm["title"]) title := jsonutil.StringFromAny(itm["title"])
if title == "" { if title == "" {
title = asString(itm["name"]) title = jsonutil.StringFromAny(itm["name"])
} }
artist := nestedSearchString(itm, "artist", "name") artist := jsonutil.NestedString(itm, "artist", "name")
if artist == "" { if artist == "" {
if artists, ok := itm["artists"].([]any); ok && len(artists) > 0 { if artists, ok := itm["artists"].([]any); ok && len(artists) > 0 {
if a0, ok := artists[0].(map[string]any); ok { if a0, ok := artists[0].(map[string]any); ok {
artist = asString(a0["name"]) artist = jsonutil.StringFromAny(a0["name"])
} }
} }
} }
album := nestedSearchString(itm, "album", "title") album := jsonutil.NestedString(itm, "album", "title")
trackCount := searchInt(itm["numberOfTracks"]) trackCount := jsonutil.IntFromAny(itm["numberOfTracks"])
if trackCount == 0 { if trackCount == 0 {
trackCount = searchInt(itm["tracks_count"]) trackCount = jsonutil.IntFromAny(itm["tracks_count"])
} }
explicit := searchBool(itm["explicit"]) explicit := searchBool(itm["explicit"])
date := firstNonEmpty( date := jsonutil.FirstNonEmpty(
asString(itm["releaseDate"]), jsonutil.StringFromAny(itm["releaseDate"]),
asString(itm["streamStartDate"]), jsonutil.StringFromAny(itm["streamStartDate"]),
asString(itm["release_date"]), jsonutil.StringFromAny(itm["release_date"]),
nestedSearchString(itm, "album", "releaseDate"), jsonutil.NestedString(itm, "album", "releaseDate"),
nestedSearchString(itm, "album", "release_date"), jsonutil.NestedString(itm, "album", "release_date"),
) )
releases := 0 releases := 0
if mediaType == "artist" { if mediaType == "artist" {
releases = firstPositiveInt( releases = jsonutil.FirstPositiveInt(
searchInt(itm["numberOfAlbums"]), jsonutil.IntFromAny(itm["numberOfAlbums"]),
searchInt(itm["albums_count"]), jsonutil.IntFromAny(itm["albums_count"]),
searchInt(itm["albumsCount"]), jsonutil.IntFromAny(itm["albumsCount"]),
nestedSearchInt(itm, "albums", "total"), jsonutil.IntFromAny(jsonutil.NestedAny(itm, "albums", "total")),
) )
} }
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit}) appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
@@ -509,25 +431,25 @@ func normalizeSearchResults(source, mediaType string, pages []map[string]any) []
if !ok { if !ok {
continue continue
} }
id := asString(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
title := asString(itm["title"]) title := jsonutil.StringFromAny(itm["title"])
if title == "" { if title == "" {
title = asString(itm["name"]) title = jsonutil.StringFromAny(itm["name"])
} }
artist := nestedSearchString(itm, "artist", "name") artist := jsonutil.NestedString(itm, "artist", "name")
album := nestedSearchString(itm, "album", "title") album := jsonutil.NestedString(itm, "album", "title")
trackCount := searchInt(itm["nb_tracks"]) trackCount := jsonutil.IntFromAny(itm["nb_tracks"])
explicit := searchBool(itm["explicit_lyrics"]) explicit := searchBool(itm["explicit_lyrics"])
date := firstNonEmpty( date := jsonutil.FirstNonEmpty(
asString(itm["release_date"]), jsonutil.StringFromAny(itm["release_date"]),
nestedSearchString(itm, "album", "release_date"), jsonutil.NestedString(itm, "album", "release_date"),
) )
releases := 0 releases := 0
if mediaType == "artist" { if mediaType == "artist" {
releases = firstPositiveInt( releases = jsonutil.FirstPositiveInt(
searchInt(itm["nb_album"]), jsonutil.IntFromAny(itm["nb_album"]),
searchInt(itm["albums_count"]), jsonutil.IntFromAny(itm["albums_count"]),
searchInt(itm["numberOfAlbums"]), jsonutil.IntFromAny(itm["numberOfAlbums"]),
) )
} }
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit}) appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
@@ -542,13 +464,13 @@ func normalizeSearchResults(source, mediaType string, pages []map[string]any) []
if !ok { if !ok {
continue continue
} }
id := asString(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
title := asString(itm["title"]) title := jsonutil.StringFromAny(itm["title"])
artist := nestedSearchString(itm, "artist", "name") artist := jsonutil.NestedString(itm, "artist", "name")
trackCount := searchInt(itm["tracks_count"]) trackCount := jsonutil.IntFromAny(itm["tracks_count"])
date := firstNonEmpty( date := jsonutil.FirstNonEmpty(
asString(itm["release_date"]), jsonutil.StringFromAny(itm["release_date"]),
asString(itm["display_date"]), jsonutil.StringFromAny(itm["display_date"]),
) )
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Date: date, TrackCount: trackCount}) appendUnique(searchResult{ID: id, Title: title, Artist: artist, Date: date, TrackCount: trackCount})
} }
@@ -562,34 +484,34 @@ func normalizeSearchResults(source, mediaType string, pages []map[string]any) []
if !ok { if !ok {
continue continue
} }
id := asString(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
title := asString(itm["title"]) title := jsonutil.StringFromAny(itm["title"])
if title == "" { if title == "" {
title = asString(itm["name"]) title = jsonutil.StringFromAny(itm["name"])
} }
artist := nestedSearchString(itm, "artist", "name") artist := jsonutil.NestedString(itm, "artist", "name")
if artist == "" { if artist == "" {
artist = nestedSearchString(itm, "performer", "name") artist = jsonutil.NestedString(itm, "performer", "name")
} }
album := nestedSearchString(itm, "album", "title") album := jsonutil.NestedString(itm, "album", "title")
trackCount := firstPositiveInt( trackCount := jsonutil.FirstPositiveInt(
searchInt(itm["trackCount"]), jsonutil.IntFromAny(itm["trackCount"]),
searchInt(itm["track_count"]), jsonutil.IntFromAny(itm["track_count"]),
searchInt(itm["tracks_count"]), jsonutil.IntFromAny(itm["tracks_count"]),
) )
explicit := searchBool(itm["explicit"]) explicit := searchBool(itm["explicit"])
date := firstNonEmpty( date := jsonutil.FirstNonEmpty(
asString(itm["release_date"]), jsonutil.StringFromAny(itm["release_date"]),
asString(itm["releaseDate"]), jsonutil.StringFromAny(itm["releaseDate"]),
nestedSearchString(itm, "album", "release_date"), jsonutil.NestedString(itm, "album", "release_date"),
nestedSearchString(itm, "album", "releaseDate"), jsonutil.NestedString(itm, "album", "releaseDate"),
) )
releases := 0 releases := 0
if mediaType == "artist" { if mediaType == "artist" {
releases = firstPositiveInt( releases = jsonutil.FirstPositiveInt(
searchInt(itm["albums_count"]), jsonutil.IntFromAny(itm["albums_count"]),
searchInt(itm["numberOfAlbums"]), jsonutil.IntFromAny(itm["numberOfAlbums"]),
nestedSearchInt(itm, "albums", "total"), jsonutil.IntFromAny(jsonutil.NestedAny(itm, "albums", "total")),
) )
} }
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit}) appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
@@ -604,18 +526,18 @@ func normalizeSearchResults(source, mediaType string, pages []map[string]any) []
if !ok { if !ok {
continue continue
} }
id := asString(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
title := asString(itm["title"]) title := jsonutil.StringFromAny(itm["title"])
if title == "" { if title == "" {
title = asString(itm["name"]) title = jsonutil.StringFromAny(itm["name"])
} }
if version := asString(itm["version"]); version != "" { if version := jsonutil.StringFromAny(itm["version"]); version != "" {
title += " (" + version + ")" title += " (" + version + ")"
} }
artist := nestedSearchString(itm, "artist", "name") artist := jsonutil.NestedString(itm, "artist", "name")
album := nestedSearchString(itm, "album", "title") album := jsonutil.NestedString(itm, "album", "title")
trackCount := firstPositiveInt(searchInt(itm["tracks_count"]), searchInt(itm["track_count"])) trackCount := jsonutil.FirstPositiveInt(jsonutil.IntFromAny(itm["tracks_count"]), jsonutil.IntFromAny(itm["track_count"]))
date := firstNonEmpty(asString(itm["release_date_original"]), asString(itm["release_date"]), nestedSearchString(itm, "album", "release_date_original")) date := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(itm["release_date_original"]), jsonutil.StringFromAny(itm["release_date"]), jsonutil.NestedString(itm, "album", "release_date_original"))
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, TrackCount: trackCount}) appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, TrackCount: trackCount})
} }
} }
@@ -686,55 +608,10 @@ func formatSearchOptionLabel(index int, mediaType string, r searchResult) string
} }
} }
func nestedSearchString(v map[string]any, keys ...string) string { // searchBool is deliberately stricter than jsonutil.BoolFromAny: it accepts
cur := any(v) // only a real bool. The explicit/parental-warning fields it reads are proper
for _, key := range keys { // booleans in every provider's search payload, and widening this to coerce
m, ok := cur.(map[string]any) // strings and numbers would change how results are labelled.
if !ok {
return ""
}
cur = m[key]
}
return asString(cur)
}
func nestedSearchInt(v map[string]any, keys ...string) int {
cur := any(v)
for _, key := range keys {
m, ok := cur.(map[string]any)
if !ok {
return 0
}
cur = m[key]
}
return searchInt(cur)
}
func firstPositiveInt(values ...int) int {
for _, v := range values {
if v > 0 {
return v
}
}
return 0
}
func searchInt(v any) int {
switch t := v.(type) {
case int:
return t
case int64:
return int(t)
case float64:
return int(t)
case string:
i, _ := strconv.Atoi(t)
return i
default:
return 0
}
}
func searchBool(v any) bool { func searchBool(v any) bool {
b, ok := v.(bool) b, ok := v.(bool)
return ok && b return ok && b
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"fmt"
"io"
"runtime"
"runtime/debug"
)
// version is overridden at build time:
//
// go build -ldflags "-X main.version=$(git describe --tags --always --dirty)"
//
// It stays "dev" for a plain `go build`. See the Makefile.
var version = "dev"
// commit and date are optional build-time stamps, also set with -X.
var (
commit = ""
date = ""
)
// buildVersion resolves the version string, preferring the ldflags value and
// falling back to the VCS revision the Go toolchain embeds for `go install`.
func buildVersion() string {
v := version
c := commit
if c == "" {
if info, ok := debug.ReadBuildInfo(); ok {
for _, s := range info.Settings {
if s.Key == "vcs.revision" {
c = s.Value
}
if s.Key == "vcs.time" && date == "" {
date = s.Value
}
}
}
}
if len(c) > 12 {
c = c[:12]
}
if c != "" {
v += " (" + c + ")"
}
return v
}
func printVersion(w io.Writer) {
fmt.Fprintf(w, "rip %s\n", buildVersion())
if date != "" {
fmt.Fprintf(w, "built: %s\n", date)
}
fmt.Fprintf(w, "go: %s\n", runtime.Version())
fmt.Fprintf(w, "target: %s/%s\n", runtime.GOOS, runtime.GOARCH)
}
+40 -19
View File
@@ -10,8 +10,13 @@ concurrency = true
# The maximum number of tracks to download at once # The maximum number of tracks to download at once
# Set to -1 for no limit # Set to -1 for no limit
max_connections = 6 max_connections = 6
# Max number of API requests per source per minute # Max number of API requests per source per minute. Set to -1 for no limit.
# Set to -1 for no limit #
# This is the usual cause of a large album feeling slow: every track needs a
# stream-URL lookup, so a 68-track album is ~69 requests and at 60/minute that
# is a minute of waiting before the audio is even reached. Requests are
# allowed to burst up to max_connections, so raising this is what lets several
# tracks download at once. Raise it if your provider tolerates it.
requests_per_minute = 60 requests_per_minute = 60
# Verify SSL certificates for API connections # Verify SSL certificates for API connections
# Set to false only if certificate verification fails (not recommended) # Set to false only if certificate verification fails (not recommended)
@@ -21,9 +26,6 @@ verify_ssl = true
# Quality ladder (global scale is 0-4, Qobuz supports 1-4): # Quality ladder (global scale is 0-4, Qobuz supports 1-4):
# 1 = MP3 320, 2 = FLAC 16/44.1, 3 = FLAC 24-bit <=96kHz, 4 = FLAC 24-bit <=192kHz # 1 = MP3 320, 2 = FLAC 16/44.1, 3 = FLAC 24-bit <=96kHz, 4 = FLAC 24-bit <=192kHz
quality = 3 quality = 3
# Download booklet PDFs when available
download_booklets = true
# Authenticate using Qobuz auth token instead of email/password hash # Authenticate using Qobuz auth token instead of email/password hash
use_auth_token = false use_auth_token = false
# If use_auth_token=true, set your user id. Otherwise set your email. # If use_auth_token=true, set your user id. Otherwise set your email.
@@ -34,6 +36,11 @@ password_or_token = ""
app_id = "" app_id = ""
# Managed automatically by streamrip-go # Managed automatically by streamrip-go
secrets = [] secrets = []
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Qobuz rotates theirs and you need to
# patch your install without waiting for a new release.
mobile_app_id = ""
mobile_app_secret = ""
[tidal] [tidal]
# Quality ladder: # Quality ladder:
@@ -53,6 +60,11 @@ access_token = ""
refresh_token = "" refresh_token = ""
# Unix timestamp when access_token expires # Unix timestamp when access_token expires
token_expiry = 0 token_expiry = 0
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Tidal rotates theirs and you need to
# patch your install without waiting for a new release.
app_client_id = ""
app_client_secret = ""
[deezer] [deezer]
# Quality ladder: # Quality ladder:
@@ -67,6 +79,13 @@ email = ""
password = "" password = ""
# Optional cached Deezer refresh token. Managed automatically when available. # Optional cached Deezer refresh token. Managed automatically when available.
refresh_token = "" refresh_token = ""
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Deezer rotates theirs and you need to
# patch your install without waiting for a new release.
api_key = ""
gateway_key = ""
# Must be exactly 16 characters if set.
blowfish_secret = ""
[yandex] [yandex]
# Quality ladder: # Quality ladder:
@@ -76,6 +95,11 @@ quality = 2
access_token = "" access_token = ""
# Cached current account uid. Managed automatically when available. # Cached current account uid. Managed automatically when available.
user_id = "" user_id = ""
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Yandex rotates theirs and you need to
# patch your install without waiting for a new release.
sign_key = ""
legacy_mp3_sign_salt = ""
[beatport] [beatport]
# Quality ladder: # Quality ladder:
@@ -93,21 +117,14 @@ access_token = ""
refresh_token = "" refresh_token = ""
# Unix timestamp when access_token expires # Unix timestamp when access_token expires
token_expiry = 0 token_expiry = 0
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Beatport rotates theirs and you need to
# patch your install without waiting for a new release.
app_client_id = ""
[soundcloud] [soundcloud]
# Quality is currently provider-defined (keep 0) # Quality is currently provider-defined (keep 0)
quality = 0 quality = 0
# Managed automatically when available
client_id = ""
app_version = ""
[youtube]
# Only 0 is currently supported
quality = 0
# Download video streams together with audio when supported
download_videos = false
# Folder used for video outputs
video_downloads_folder = "/path/to/StreamripDownloads/YouTubeVideos"
[database] [database]
# Track IDs already downloaded are stored here and skipped next time # Track IDs already downloaded are stored here and skipped next time
@@ -157,7 +174,13 @@ set_playlist_to_album = true
renumber_playlist_tracks = true renumber_playlist_tracks = true
# Separator used when a provider exposes multiple artists as separate values # Separator used when a provider exposes multiple artists as separate values
artist_separator = "; " artist_separator = "; "
# Metadata fields to exclude from tagging # Metadata fields to exclude from tagging.
#
# "lyrics" is worth knowing about: Qobuz, Tidal and Deezer each serve lyrics
# from a separate endpoint, so fetching them costs one extra API request per
# track and is subject to requests_per_minute above. Set
# exclude = ["lyrics"] if you do not want lyrics and would rather have the
# faster rip.
exclude = [] exclude = []
[filepaths] [filepaths]
@@ -189,5 +212,3 @@ max_search_results = 100
[misc] [misc]
# Metadata used for config compatibility checks # Metadata used for config compatibility checks
version = "2.2.0" version = "2.2.0"
# Notify when a new version is available
check_for_updates = true
+365 -250
View File
@@ -11,6 +11,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
"time"
"streamrip-go/internal/artwork" "streamrip-go/internal/artwork"
"streamrip-go/internal/audio/convert" "streamrip-go/internal/audio/convert"
@@ -52,6 +53,7 @@ type ripTrackOptions struct {
albumEmbedCover string albumEmbedCover string
albumArtist string albumArtist string
prefetched *provider.Downloadable prefetched *provider.Downloadable
prefetchedMeta map[string]any
index int index int
total int total int
albumDiscTotal int albumDiscTotal int
@@ -87,7 +89,29 @@ var (
) )
type trackTagger interface { type trackTagger interface {
TagFLAC(path string, meta tag.Metadata, coverPath string) error TagFLAC(ctx context.Context, path string, meta tag.Metadata, coverPath string) error
}
// trackLyricsProvider is implemented by providers that can fetch lyrics for a
// track independently of its metadata lookup.
//
// Tidal and Deezer attach lyrics inside GetMetadata, which album rips skip
// when they reuse the album payload's track objects. Qobuz exposes this
// instead, so lyrics work on both paths.
type trackLyricsProvider interface {
TrackLyrics(ctx context.Context, trackID string) (plain string, synced string, err error)
}
// albumTracksSelfDescribing is implemented by providers whose album payload
// embeds track objects complete enough to tag from, making the per-track
// metadata call redundant.
//
// This is opt-in per provider rather than assumed, because it is only true
// where it has been checked against the live API, and because some providers
// do extra work inside their track lookup: Tidal, for instance, fetches lyrics
// there, which an album payload does not carry.
type albumTracksSelfDescribing interface {
AlbumTracksAreComplete() bool
} }
type videoDownloadableProvider interface { type videoDownloadableProvider interface {
@@ -128,6 +152,9 @@ func New(cfg *config.Config) (*Main, error) {
Pending: []media.Pending{}, Pending: []media.Pending{},
Media: []media.Media{}, Media: []media.Media{},
} }
// The downloader does the Deezer decryption, so it needs the protocol
// secret from config rather than a constant of its own.
m.DL.DeezerBlowfishSecret = cfg.Session.Deezer.BlowfishSecret
verbose.SetSink(func(msg string) { m.DL.Logf("%s", msg) }) verbose.SetSink(func(msg string) { m.DL.Logf("%s", msg) })
return m, nil return m, nil
} }
@@ -161,6 +188,251 @@ func (m *Main) logf(format string, args ...any) {
fmt.Printf(format, args...) fmt.Printf(format, args...)
} }
// bookkeepingTimeout bounds the detached store writes below. They must not
// inherit a cancelled context, or a single Ctrl-C silently loses the record of
// which tracks failed.
const bookkeepingTimeout = 5 * time.Second
// markFailed records a failed download. It deliberately detaches from ctx so
// the write still lands when the rip was cancelled, which is exactly when the
// failure record matters most.
func (m *Main) markFailed(ctx context.Context, source, mediaType, id string) {
if m.Store == nil {
return
}
writeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), bookkeepingTimeout)
defer cancel()
if err := m.Store.MarkFailed(writeCtx, source, mediaType, id); err != nil {
m.logf("warning: could not record failure for %s/%s/%s: %v\n", source, mediaType, id, err)
}
}
// markDownloaded records a completed download, detached from ctx for the same
// reason as markFailed: the file is already on disk, so losing the record would
// cause a needless re-download later.
func (m *Main) markDownloaded(ctx context.Context, source, id string) error {
if m.Store == nil {
return nil
}
writeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), bookkeepingTimeout)
defer cancel()
return m.Store.MarkDownloaded(writeCtx, source, id)
}
// defaultMaxWorkers is used when max_connections is unset or nonsensical.
const defaultMaxWorkers = 6
// runTrackWorkers rips ids, either serially or across max_connections workers
// depending on config, and returns how many failed.
//
// This replaces three copy-pasted dispatch blocks that had drifted apart: two
// counted failures under a mutex and logged inline, while the third collected
// errors in a channel and logged them all after wg.Wait() without the track id.
// The inline form is kept, since it reports failures as they happen and names
// the track. pos is 1-based to match the progress output's numbering.
func (m *Main) runTrackWorkers(ctx context.Context, ids []string, rip func(pos int, id string) error) int {
failures := 0
maxWorkers := m.Config.Session.Downloads.MaxConnections
if !m.Config.Session.Downloads.Concurrency || maxWorkers == 1 {
for i, id := range ids {
if ctx.Err() != nil {
break
}
if err := rip(i+1, id); err != nil {
failures++
m.logf("track failed: id=%s reason=%v\n", id, err)
}
}
return failures
}
if maxWorkers <= 0 {
maxWorkers = defaultMaxWorkers
}
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
var mu sync.Mutex
for i, id := range ids {
if ctx.Err() != nil {
break
}
wg.Add(1)
sem <- struct{}{}
go func(pos int, tid string) {
defer wg.Done()
defer func() { <-sem }()
if err := rip(pos, tid); err != nil {
// Holding mu across the log keeps failure lines from
// interleaving with each other.
mu.Lock()
failures++
m.logf("track failed: id=%s reason=%v\n", tid, err)
mu.Unlock()
}
}(i+1, id)
}
wg.Wait()
return failures
}
// metadataExcluded reports whether a tag was opted out via
// metadata.exclude in the config.
func (m *Main) metadataExcluded(field string) bool {
for _, k := range m.Config.Session.Metadata.Exclude {
if strings.EqualFold(strings.TrimSpace(k), field) {
return true
}
}
return false
}
// attachLyrics fills in lyrics for providers that fetch them separately from
// track metadata. It is a no-op when the metadata already carries lyrics (as
// Tidal's and Deezer's does), when the provider has no lyrics support, or when
// the user excluded them.
func (m *Main) attachLyrics(ctx context.Context, p provider.Client, meta map[string]any, trackID string) {
if meta == nil || m.metadataExcluded("lyrics") {
return
}
if jsonutil.StringFromAny(meta["lyrics"]) != "" || jsonutil.StringFromAny(meta["lyrics_synced"]) != "" {
return
}
lp, ok := p.(trackLyricsProvider)
if !ok {
return
}
plain, synced, err := lp.TrackLyrics(ctx, trackID)
if err != nil {
// Lyrics are cosmetic; never fail a download over them.
m.logf("lyrics unavailable for id=%s: %v\n", trackID, err)
return
}
if plain != "" {
meta["lyrics"] = plain
}
if synced != "" {
meta["lyrics_synced"] = synced
}
}
// trackMetaUsable reports whether a track object carried over from an album
// payload is complete enough to tag and name a file from.
//
// It is a belt-and-braces check on top of the provider opting in: title and
// track number determine the output filename, so if either is missing the
// caller refetches rather than writing a wrongly named file.
func trackMetaUsable(meta map[string]any) bool {
if len(meta) == 0 {
return false
}
if strings.TrimSpace(jsonutil.StringFromAny(meta["title"])) == "" {
return false
}
return jsonutil.IntFromAny(meta["track_number"]) > 0
}
// albumTrackMetadata builds the per-track metadata for an album rip from the
// track objects already embedded in the album payload.
//
// Verified against the live Qobuz API: the only field a dedicated track
// lookup adds that this app reads is the nested album object, which is the
// parent album we already have. The album copy has its own track list removed,
// so attaching it to every track does not duplicate the whole album n times.
func albumTrackMetadata(albumMeta map[string]any, rawItems []any) map[string]map[string]any {
albumForTrack := make(map[string]any, len(albumMeta))
for k, v := range albumMeta {
if k == "tracks" {
continue
}
albumForTrack[k] = v
}
out := make(map[string]map[string]any, len(rawItems))
for _, raw := range rawItems {
item, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(item["id"])
if id == "" {
continue
}
meta := make(map[string]any, len(item)+1)
for k, v := range item {
meta[k] = v
}
if _, has := meta["album"]; !has {
meta["album"] = albumForTrack
}
out[id] = meta
}
return out
}
// anySlice normalizes a JSON array that may have been decoded as either
// []any or []map[string]any, which differs per provider because some clients
// hand-build their metadata maps instead of round-tripping through encoding/json.
//
// It reports false when the value is neither, so callers can distinguish
// "missing" from "empty".
func anySlice(v any) ([]any, bool) {
switch items := v.(type) {
case []any:
return items, true
case []map[string]any:
out := make([]any, 0, len(items))
for _, item := range items {
out = append(out, item)
}
return out, true
default:
return nil, false
}
}
// safeJoin joins base with parts and asserts the result stays inside the
// configured downloads folder.
//
// naming.CleanName already rejects "." and ".." components, so this should
// never fire; it is a second line of defence because every component here is
// built from provider-supplied metadata, and writing outside the download root
// is the one failure mode worth being paranoid about.
func (m *Main) safeJoin(base string, parts ...string) string {
joined := filepath.Join(append([]string{base}, parts...)...)
root := strings.TrimSpace(m.Config.Session.Downloads.Folder)
if root == "" {
return joined
}
if pathWithin(root, joined) {
return joined
}
m.logf("warning: refusing to write outside the download folder (%q); using %q instead\n", joined, root)
return filepath.Join(root, "Unknown")
}
// pathWithin reports whether target is root itself or sits beneath it.
func pathWithin(root, target string) bool {
absRoot, err := filepath.Abs(root)
if err != nil {
return false
}
absTarget, err := filepath.Abs(target)
if err != nil {
return false
}
rel, err := filepath.Rel(absRoot, absTarget)
if err != nil {
return false
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return false
}
return true
}
func (m *Main) GetLoggedInProvider(ctx context.Context, source string) (provider.Client, error) { func (m *Main) GetLoggedInProvider(ctx context.Context, source string) (provider.Client, error) {
p, ok := m.Providers[source] p, ok := m.Providers[source]
if !ok { if !ok {
@@ -184,7 +456,7 @@ func (m *Main) AddByID(ctx context.Context, source, mediaType, id string) error
ResolveFn: func(ctx context.Context) (media.Media, error) { ResolveFn: func(ctx context.Context) (media.Media, error) {
meta, err := p.GetMetadata(ctx, id, mediaType) meta, err := p.GetMetadata(ctx, id, mediaType)
if err != nil { if err != nil {
_ = m.Store.MarkFailed(ctx, source, mediaType, id) m.markFailed(ctx, source, mediaType, id)
return nil, err return nil, err
} }
@@ -224,47 +496,6 @@ func (m *Main) AddByID(ctx context.Context, source, mediaType, id string) error
return nil return nil
} }
func (m *Main) AddPlaylistByTrackIDs(ctx context.Context, source, playlistID, playlistName string, trackIDs []string) error {
p, err := m.GetLoggedInProvider(ctx, source)
if err != nil {
return err
}
if strings.TrimSpace(playlistName) == "" {
playlistName = playlistID
}
ids := make([]string, 0, len(trackIDs))
for _, id := range trackIDs {
id = strings.TrimSpace(id)
if id != "" {
ids = append(ids, id)
}
}
if len(ids) == 0 {
return fmt.Errorf("playlist %q has no track ids", playlistName)
}
pending := media.PendingFunc{
ResolveFn: func(context.Context) (media.Media, error) {
metaItems := make([]any, 0, len(ids))
for _, id := range ids {
metaItems = append(metaItems, map[string]any{"id": id})
}
playlistMeta := map[string]any{
"name": playlistName,
"tracks": map[string]any{"items": metaItems},
}
return media.MediaFunc{RipFn: func(ctx context.Context) error {
if m.Config.Session.CLI.TextOutput {
m.logf("Downloading playlist: %s\n", playlistName)
}
return m.ripPlaylist(ctx, p, source, playlistID, playlistMeta)
}}, nil
},
}
m.Pending = append(m.Pending, pending)
return nil
}
func (m *Main) AddMixedPlaylistByTrackRefs(ctx context.Context, playlistID, playlistName string, refs []PlaylistTrackRef) error { func (m *Main) AddMixedPlaylistByTrackRefs(ctx context.Context, playlistID, playlistName string, refs []PlaylistTrackRef) error {
if strings.TrimSpace(playlistName) == "" { if strings.TrimSpace(playlistName) == "" {
playlistName = playlistID playlistName = playlistID
@@ -346,20 +577,13 @@ func (m *Main) ripTrackCollection(ctx context.Context, p provider.Client, source
if m.Config.Session.Downloads.SourceSubdirectories { if m.Config.Session.Downloads.SourceSubdirectories {
base = filepath.Join(base, jsonutil.TitleCase(source)) base = filepath.Join(base, jsonutil.TitleCase(source))
} }
folder := filepath.Join(base, naming.CleanName(name, naming.Config{RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters, TruncateTo: m.Config.Session.Filepaths.TruncateTo})) folder := m.safeJoin(base, naming.CleanName(name, naming.Config{RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters, TruncateTo: m.Config.Session.Filepaths.TruncateTo}))
tracksMap, ok := meta["tracks"].(map[string]any) tracksMap, ok := meta["tracks"].(map[string]any)
if !ok { if !ok {
return fmt.Errorf("%s missing tracks data", strings.ToLower(kind)) return fmt.Errorf("%s missing tracks data", strings.ToLower(kind))
} }
rawItems := make([]any, 0) rawItems, ok := anySlice(tracksMap["items"])
switch items := tracksMap["items"].(type) { if !ok {
case []any:
rawItems = items
case []map[string]any:
for _, item := range items {
rawItems = append(rawItems, item)
}
default:
return fmt.Errorf("%s tracks missing items", strings.ToLower(kind)) return fmt.Errorf("%s tracks missing items", strings.ToLower(kind))
} }
ids := make([]string, 0, len(rawItems)) ids := make([]string, 0, len(rawItems))
@@ -373,53 +597,15 @@ func (m *Main) ripTrackCollection(ctx context.Context, p provider.Client, source
} }
} }
m.logf("%s: %s (%d tracks)\n", kind, name, len(ids)) m.logf("%s: %s (%d tracks)\n", kind, name, len(ids))
failures := 0 failures := m.runTrackWorkers(ctx, ids, func(pos int, trackID string) error {
runOne := func(i int, trackID string) { opts := ripTrackOptions{albumFolder: folder, index: pos, total: len(ids)}
opts := ripTrackOptions{albumFolder: folder, index: i, total: len(ids)}
if playlistLike { if playlistLike {
opts.forPlaylist = true opts.forPlaylist = true
opts.playlistName = name opts.playlistName = name
opts.playlistPos = i opts.playlistPos = pos
} }
if err := m.ripTrack(ctx, p, source, trackID, "", opts); err != nil { return m.ripTrack(ctx, p, source, trackID, "", opts)
failures++ })
m.logf("track failed: id=%s reason=%v\n", trackID, err)
}
}
if !m.Config.Session.Downloads.Concurrency || m.Config.Session.Downloads.MaxConnections == 1 {
for i, trackID := range ids {
runOne(i+1, trackID)
}
} else {
maxWorkers := m.Config.Session.Downloads.MaxConnections
if maxWorkers <= 0 {
maxWorkers = 6
}
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
var mu sync.Mutex
for i, trackID := range ids {
wg.Add(1)
sem <- struct{}{}
go func(pos int, tid string) {
defer wg.Done()
defer func() { <-sem }()
opts := ripTrackOptions{albumFolder: folder, index: pos, total: len(ids)}
if playlistLike {
opts.forPlaylist = true
opts.playlistName = name
opts.playlistPos = pos
}
if err := m.ripTrack(ctx, p, source, tid, "", opts); err != nil {
mu.Lock()
failures++
m.logf("track failed: id=%s reason=%v\n", tid, err)
mu.Unlock()
}
}(i+1, trackID)
}
wg.Wait()
}
if failures > 0 { if failures > 0 {
m.logf("%s done with %d failed track(s)\n", kind, failures) m.logf("%s done with %d failed track(s)\n", kind, failures)
} }
@@ -440,18 +626,18 @@ func (m *Main) ripVideo(ctx context.Context, p provider.Client, source, videoID
d, err := vp.GetVideoDownloadable(ctx, videoID) d, err := vp.GetVideoDownloadable(ctx, videoID)
if err != nil { if err != nil {
_ = m.Store.MarkFailed(ctx, source, "video", videoID) m.markFailed(ctx, source, "video", videoID)
return fmt.Errorf("id=%s get_video_downloadable: %w", videoID, err) return fmt.Errorf("id=%s get_video_downloadable: %w", videoID, err)
} }
title := titleFromMetadata(meta, videoID) title := titleFromMetadata(meta, videoID)
outPath := m.videoOutputPath(source, videoID, title, d.Extension) outPath := m.videoOutputPath(source, videoID, title, d.Extension)
if err = m.DL.FileVideo(ctx, d.URL, outPath); err != nil { if err = m.DL.FileVideo(ctx, d.URL, outPath); err != nil {
_ = m.Store.MarkFailed(ctx, source, "video", videoID) m.markFailed(ctx, source, "video", videoID)
return fmt.Errorf("id=%s title=%q video download: %w", videoID, title, err) return fmt.Errorf("id=%s title=%q video download: %w", videoID, title, err)
} }
if err = m.Store.MarkDownloaded(ctx, source, videoID); err != nil { if err = m.markDownloaded(ctx, source, videoID); err != nil {
return err return err
} }
return nil return nil
@@ -564,23 +750,11 @@ func extractAlbumIDs(meta map[string]any) []string {
return nil return nil
} }
items := make([]any, 0) var items []any
switch a := albumsObj.(type) { if nested, isMap := albumsObj.(map[string]any); isMap {
case map[string]any: items, _ = anySlice(nested["items"])
switch v := a["items"].(type) { } else {
case []any: items, _ = anySlice(albumsObj)
items = v
case []map[string]any:
for _, it := range v {
items = append(items, it)
}
}
case []any:
items = a
case []map[string]any:
for _, it := range a {
items = append(items, it)
}
} }
out := make([]string, 0, len(items)) out := make([]string, 0, len(items))
@@ -631,11 +805,17 @@ func (m *Main) Rip(ctx context.Context) error {
m.logf("Ripping %d media item(s)\n", len(m.Media)) m.logf("Ripping %d media item(s)\n", len(m.Media))
failures := 0 failures := 0
for _, item := range m.Media { for _, item := range m.Media {
if err := ctx.Err(); err != nil {
return err
}
if err := item.Rip(ctx); err != nil { if err := item.Rip(ctx); err != nil {
failures++ failures++
m.logf("media item failed: %v\n", err) m.logf("media item failed: %v\n", err)
} }
} }
if err := ctx.Err(); err != nil {
return err
}
if failures > 0 { if failures > 0 {
return fmt.Errorf("%d media item(s) failed", failures) return fmt.Errorf("%d media item(s) failed", failures)
} }
@@ -667,14 +847,8 @@ func (m *Main) ripAlbum(ctx context.Context, p provider.Client, source, albumID
if !ok { if !ok {
return fmt.Errorf("album missing tracks data") return fmt.Errorf("album missing tracks data")
} }
rawItems := make([]any, 0) rawItems, ok := anySlice(tracksMap["items"])
if itemsAny, ok := tracksMap["items"].([]any); ok { if !ok {
rawItems = itemsAny
} else if itemsMap, ok := tracksMap["items"].([]map[string]any); ok {
for _, item := range itemsMap {
rawItems = append(rawItems, item)
}
} else {
return fmt.Errorf("album tracks missing items") return fmt.Errorf("album tracks missing items")
} }
trackIDs := make([]string, 0, len(rawItems)) trackIDs := make([]string, 0, len(rawItems))
@@ -703,54 +877,31 @@ func (m *Main) ripAlbum(ctx context.Context, p provider.Client, source, albumID
if discTotal == 0 { if discTotal == 0 {
discTotal = jsonutil.IntFromAny(albumMeta["numberOfVolumes"]) discTotal = jsonutil.IntFromAny(albumMeta["numberOfVolumes"])
} }
// Providers that say their album payload embeds complete track objects let
// us skip one throttled API call per track.
var embeddedMeta map[string]map[string]any
if sd, ok := p.(albumTracksSelfDescribing); ok && sd.AlbumTracksAreComplete() {
embeddedMeta = albumTrackMetadata(albumMeta, rawItems)
}
m.logf("Album: %s (%d tracks)\n", albumTitle, total) m.logf("Album: %s (%d tracks)\n", albumTitle, total)
failures := 0 failures := m.runTrackWorkers(ctx, trackIDs, func(pos int, trackID string) error {
opts := ripTrackOptions{
if !m.Config.Session.Downloads.Concurrency || m.Config.Session.Downloads.MaxConnections == 1 { albumFolder: folder,
for i, trackID := range trackIDs { albumEmbedCover: artRes.EmbedPath,
opts := ripTrackOptions{albumFolder: folder, albumEmbedCover: artRes.EmbedPath, albumArtist: albumArtist, index: i + 1, total: total, albumDiscTotal: discTotal} albumArtist: albumArtist,
if i == 0 { index: pos,
opts.prefetched = prefetched total: total,
} albumDiscTotal: discTotal,
if err := m.ripTrack(ctx, p, source, trackID, "", opts); err != nil { prefetchedMeta: embeddedMeta[trackID],
failures++
m.logf("track failed: id=%s reason=%v\n", trackID, err)
}
} }
if failures > 0 { // The first track's downloadable was already fetched to decide the
m.logf("Album done with %d failed track(s)\n", failures) // album folder's audio values, so reuse it instead of refetching.
if pos == 1 {
opts.prefetched = prefetched
} }
return nil return m.ripTrack(ctx, p, source, trackID, "", opts)
} })
maxWorkers := m.Config.Session.Downloads.MaxConnections
if maxWorkers <= 0 {
maxWorkers = 6
}
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
errCh := make(chan error, len(trackIDs))
for i, trackID := range trackIDs {
wg.Add(1)
sem <- struct{}{}
go func(idx int, tid string) {
defer wg.Done()
defer func() { <-sem }()
opts := ripTrackOptions{albumFolder: folder, albumEmbedCover: artRes.EmbedPath, albumArtist: albumArtist, index: idx, total: total, albumDiscTotal: discTotal}
if idx == 1 {
opts.prefetched = prefetched
}
if err := m.ripTrack(ctx, p, source, tid, "", opts); err != nil {
errCh <- err
}
}(i+1, trackID)
}
wg.Wait()
close(errCh)
for err := range errCh {
failures++
m.logf("track failed: reason=%v\n", err)
}
if failures > 0 { if failures > 0 {
m.logf("Album done with %d failed track(s)\n", failures) m.logf("Album done with %d failed track(s)\n", failures)
} }
@@ -770,7 +921,7 @@ func (m *Main) ripPlaylist(ctx context.Context, p provider.Client, source, playl
if m.Config.Session.Downloads.SourceSubdirectories { if m.Config.Session.Downloads.SourceSubdirectories {
base = filepath.Join(base, jsonutil.TitleCase(source)) base = filepath.Join(base, jsonutil.TitleCase(source))
} }
folder := filepath.Join(base, naming.CleanName(name, naming.Config{ folder := m.safeJoin(base, naming.CleanName(name, naming.Config{
RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters, RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters,
TruncateTo: m.Config.Session.Filepaths.TruncateTo, TruncateTo: m.Config.Session.Filepaths.TruncateTo,
})) }))
@@ -779,16 +930,9 @@ func (m *Main) ripPlaylist(ctx context.Context, p provider.Client, source, playl
if !ok { if !ok {
return fmt.Errorf("playlist missing tracks data") return fmt.Errorf("playlist missing tracks data")
} }
rawItems, ok := tracksMap["items"].([]any) rawItems, ok := anySlice(tracksMap["items"])
if !ok { if !ok {
if itemsMap, ok2 := tracksMap["items"].([]map[string]any); ok2 { return fmt.Errorf("playlist tracks missing items")
rawItems = make([]any, 0, len(itemsMap))
for _, it := range itemsMap {
rawItems = append(rawItems, it)
}
} else {
return fmt.Errorf("playlist tracks missing items")
}
} }
ids := make([]string, 0, len(rawItems)) ids := make([]string, 0, len(rawItems))
@@ -808,52 +952,17 @@ func (m *Main) ripPlaylist(ctx context.Context, p provider.Client, source, playl
total := len(ids) total := len(ids)
m.logf("Playlist: %s (%d tracks)\n", name, total) m.logf("Playlist: %s (%d tracks)\n", name, total)
failures := 0 failures := m.runTrackWorkers(ctx, ids, func(pos int, id string) error {
runOne := func(i int, id string) {
opts := ripTrackOptions{ opts := ripTrackOptions{
albumFolder: folder, albumFolder: folder,
index: i, index: pos,
total: total, total: total,
forPlaylist: true, forPlaylist: true,
playlistName: name, playlistName: name,
playlistPos: i, playlistPos: pos,
} }
if err := m.ripTrack(ctx, p, source, id, "", opts); err != nil { return m.ripTrack(ctx, p, source, id, "", opts)
failures++ })
m.logf("track failed: id=%s reason=%v\n", id, err)
}
}
if !m.Config.Session.Downloads.Concurrency || m.Config.Session.Downloads.MaxConnections == 1 {
for i, id := range ids {
runOne(i+1, id)
}
} else {
maxWorkers := m.Config.Session.Downloads.MaxConnections
if maxWorkers <= 0 {
maxWorkers = 6
}
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
var mu sync.Mutex
for i, id := range ids {
wg.Add(1)
sem <- struct{}{}
go func(pos int, tid string) {
defer wg.Done()
defer func() { <-sem }()
opts := ripTrackOptions{albumFolder: folder, index: pos, total: total, forPlaylist: true, playlistName: name, playlistPos: pos}
if err := m.ripTrack(ctx, p, source, tid, "", opts); err != nil {
mu.Lock()
failures++
m.logf("track failed: id=%s reason=%v\n", tid, err)
mu.Unlock()
}
}(i+1, id)
}
wg.Wait()
}
if failures > 0 { if failures > 0 {
m.logf("Playlist done with %d failed track(s)\n", failures) m.logf("Playlist done with %d failed track(s)\n", failures)
@@ -876,7 +985,7 @@ func (m *Main) ripPlaylistMixed(ctx context.Context, playlistID, name string, re
} }
} }
folder := filepath.Join(m.Config.Session.Downloads.Folder, naming.CleanName(name, naming.Config{ folder := m.safeJoin(m.Config.Session.Downloads.Folder, naming.CleanName(name, naming.Config{
RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters, RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters,
TruncateTo: m.Config.Session.Filepaths.TruncateTo, TruncateTo: m.Config.Session.Filepaths.TruncateTo,
})) }))
@@ -938,6 +1047,9 @@ func (m *Main) requireSourceDownloadAuth(source string) error {
} }
func (m *Main) ripTrack(ctx context.Context, p provider.Client, source, id, fallbackTitle string, opts ripTrackOptions) error { func (m *Main) ripTrack(ctx context.Context, p provider.Client, source, id, fallbackTitle string, opts ripTrackOptions) error {
if err := ctx.Err(); err != nil {
return err
}
if !m.IgnoreDB { if !m.IgnoreDB {
alreadyDownloaded, err := m.Store.IsDownloaded(ctx, source, id) alreadyDownloaded, err := m.Store.IsDownloaded(ctx, source, id)
if err == nil && alreadyDownloaded { if err == nil && alreadyDownloaded {
@@ -950,12 +1062,22 @@ func (m *Main) ripTrack(ctx context.Context, p provider.Client, source, id, fall
} }
} }
meta, err := p.GetMetadata(ctx, id, "track") // A prefetched track object from the album payload saves one API call per
if err != nil { // track. Every call goes through the request-per-minute limiter, so on a
_ = m.Store.MarkFailed(ctx, source, "track", id) // large album this is the difference between the rip being gated by the
return fmt.Errorf("id=%s metadata: %w", id, err) // API and being gated by the network.
meta := opts.prefetchedMeta
var err error
if !trackMetaUsable(meta) {
meta, err = p.GetMetadata(ctx, id, "track")
if err != nil {
m.markFailed(ctx, source, "track", id)
return fmt.Errorf("id=%s metadata: %w", id, err)
}
} }
m.attachLyrics(ctx, p, meta, id)
title := titleFromMetadata(meta, id) title := titleFromMetadata(meta, id)
if title == id && fallbackTitle != "" { if title == id && fallbackTitle != "" {
title = fallbackTitle title = fallbackTitle
@@ -969,7 +1091,7 @@ func (m *Main) ripTrack(ctx context.Context, p provider.Client, source, id, fall
if d == nil { if d == nil {
d, err = p.GetDownloadable(ctx, id, m.qualityForSource(source)) d, err = p.GetDownloadable(ctx, id, m.qualityForSource(source))
if err != nil { if err != nil {
_ = m.Store.MarkFailed(ctx, source, "track", id) m.markFailed(ctx, source, "track", id)
return fmt.Errorf("id=%s title=%q get_downloadable: %w", id, title, err) return fmt.Errorf("id=%s title=%q get_downloadable: %w", id, title, err)
} }
} }
@@ -992,9 +1114,15 @@ func (m *Main) ripTrack(ctx context.Context, p provider.Client, source, id, fall
return m.DL.File(ctx, d.URL, outPath) return m.DL.File(ctx, d.URL, outPath)
} }
if err = downloadOnce(); err != nil { if err = downloadOnce(); err != nil {
// Never spend retries or a provider fallback on a cancelled context:
// otherwise one Ctrl-C costs three doomed attempts per remaining track.
if ctxErr := ctx.Err(); ctxErr != nil {
m.markFailed(ctx, source, "track", id)
return fmt.Errorf("id=%s title=%q download: %w", id, title, err)
}
m.logf("retry: %s (%v)\n", filepath.Base(outPath), err) m.logf("retry: %s (%v)\n", filepath.Base(outPath), err)
if err = downloadOnce(); err != nil { if err = downloadOnce(); err != nil {
if fallbackProvider, ok := p.(trackFallbackDownloader); ok { if fallbackProvider, ok := p.(trackFallbackDownloader); ok && ctx.Err() == nil {
m.logf("fallback: %s via provider backup flow\n", filepath.Base(outPath)) m.logf("fallback: %s via provider backup flow\n", filepath.Base(outPath))
if fbErr := fallbackProvider.DownloadTrackFallback(ctx, id, m.qualityForSource(source), outPath); fbErr == nil { if fbErr := fallbackProvider.DownloadTrackFallback(ctx, id, m.qualityForSource(source), outPath); fbErr == nil {
goto downloaded goto downloaded
@@ -1002,7 +1130,7 @@ func (m *Main) ripTrack(ctx context.Context, p provider.Client, source, id, fall
m.logf("fallback failed: %s (%v)\n", filepath.Base(outPath), fbErr) m.logf("fallback failed: %s (%v)\n", filepath.Base(outPath), fbErr)
} }
} }
_ = m.Store.MarkFailed(ctx, source, "track", id) m.markFailed(ctx, source, "track", id)
return fmt.Errorf("id=%s title=%q download: %w", id, title, err) return fmt.Errorf("id=%s title=%q download: %w", id, title, err)
} }
} }
@@ -1040,24 +1168,24 @@ downloaded:
coverPath = tag.CoverPathForTrack(outPath, opts.albumFolder) coverPath = tag.CoverPathForTrack(outPath, opts.albumFolder)
} }
} }
if err = m.Tagger.TagFLAC(outPath, tagMeta, coverPath); err != nil { if err = m.Tagger.TagFLAC(ctx, outPath, tagMeta, coverPath); err != nil {
if isFFmpegMissingError(err) { if isFFmpegMissingError(err) {
_ = m.Store.MarkFailed(ctx, source, "track", id) m.markFailed(ctx, source, "track", id)
return fmt.Errorf("id=%s title=%q tag: %w", id, title, err) return fmt.Errorf("id=%s title=%q tag: %w", id, title, err)
} }
m.logf("warning: tag failed for %s: %v\n", filepath.Base(outPath), err) m.logf("warning: tag failed for %s: %v\n", filepath.Base(outPath), err)
} }
if m.Config.Session.Conversion.Enabled { if m.Config.Session.Conversion.Enabled {
convertedPath, convErr := convert.Convert(outPath, m.Config.Session.Conversion) convertedPath, convErr := convert.Convert(ctx, outPath, m.Config.Session.Conversion)
if convErr != nil { if convErr != nil {
_ = m.Store.MarkFailed(ctx, source, "track", id) m.markFailed(ctx, source, "track", id)
return fmt.Errorf("id=%s title=%q convert: %w", id, title, convErr) return fmt.Errorf("id=%s title=%q convert: %w", id, title, convErr)
} }
outPath = convertedPath outPath = convertedPath
} }
return m.Store.MarkDownloaded(ctx, source, id) return m.markDownloaded(ctx, source, id)
} }
func (m *Main) qualityForSource(source string) int { func (m *Main) qualityForSource(source string) int {
@@ -1206,7 +1334,7 @@ func (m *Main) albumFolderPath(source, albumID, albumTitle, albumArtist, year st
TruncateTo: m.Config.Session.Filepaths.TruncateTo, TruncateTo: m.Config.Session.Filepaths.TruncateTo,
}) })
return filepath.Join(base, folderName) return m.safeJoin(base, folderName)
} }
func (m *Main) trackOutputPath(source, id, title, ext string, d *provider.Downloadable, trackMeta map[string]any, albumFolder string, albumDiscTotal int) string { func (m *Main) trackOutputPath(source, id, title, ext string, d *provider.Downloadable, trackMeta map[string]any, albumFolder string, albumDiscTotal int) string {
@@ -1279,7 +1407,7 @@ func (m *Main) trackOutputPath(source, id, title, ext string, d *provider.Downlo
RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters, RestrictCharacters: m.Config.Session.Filepaths.RestrictCharacters,
TruncateTo: m.Config.Session.Filepaths.TruncateTo, TruncateTo: m.Config.Session.Filepaths.TruncateTo,
}) })
return filepath.Join(base, fileName+"."+ext) return m.safeJoin(base, fileName+"."+ext)
} }
func (m *Main) videoOutputPath(source, id, title, ext string) string { func (m *Main) videoOutputPath(source, id, title, ext string) string {
@@ -1297,7 +1425,7 @@ func (m *Main) videoOutputPath(source, id, title, ext string) string {
if fileName == "" { if fileName == "" {
fileName = id fileName = id
} }
return filepath.Join(base, fileName+"."+ext) return m.safeJoin(base, fileName+"."+ext)
} }
func titleFromMetadata(meta map[string]any, fallback string) string { func titleFromMetadata(meta map[string]any, fallback string) string {
@@ -1621,11 +1749,11 @@ func extractAlbumArtist(albumMeta map[string]any) string {
} }
} }
if len(names) > 0 { if len(names) > 0 {
return displayArtistNames(names) return jsonutil.DisplayArtistNames(names)
} }
} }
if names := stringSliceFromAny(albumMeta["artist_names"]); len(names) > 0 { if names := stringSliceFromAny(albumMeta["artist_names"]); len(names) > 0 {
return displayArtistNames(names) return jsonutil.DisplayArtistNames(names)
} }
artist := jsonutil.NestedString(albumMeta, "artist", "name") artist := jsonutil.NestedString(albumMeta, "artist", "name")
if artist != "" { if artist != "" {
@@ -1633,16 +1761,3 @@ func extractAlbumArtist(albumMeta map[string]any) string {
} }
return "Unknown" return "Unknown"
} }
func displayArtistNames(names []string) string {
switch len(names) {
case 0:
return ""
case 1:
return names[0]
case 2:
return names[0] + " & " + names[1]
default:
return strings.Join(names[:len(names)-1], ", ") + " & " + names[len(names)-1]
}
}
+585 -2
View File
@@ -2,6 +2,7 @@ package app
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
@@ -9,6 +10,7 @@ import (
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"testing" "testing"
"streamrip-go/internal/audio/tag" "streamrip-go/internal/audio/tag"
@@ -20,13 +22,13 @@ import (
type noopTagger struct{} type noopTagger struct{}
func (n noopTagger) TagFLAC(string, tag.Metadata, string) error { return nil } func (n noopTagger) TagFLAC(context.Context, string, tag.Metadata, string) error { return nil }
type failingTagger struct { type failingTagger struct {
err error err error
} }
func (f failingTagger) TagFLAC(string, tag.Metadata, string) error { return f.err } func (f failingTagger) TagFLAC(context.Context, string, tag.Metadata, string) error { return f.err }
type fakeProvider struct { type fakeProvider struct {
url string url string
@@ -957,3 +959,584 @@ func TestBuildTagMetadataReplayGainFallsBackToDeezerGain(t *testing.T) {
t.Fatalf("track replaygain gain=%q", tags.ReplaygainTrackGain) t.Fatalf("track replaygain gain=%q", tags.ReplaygainTrackGain)
} }
} }
// countingAlbumProvider serves a large album and records how many track
// downloads were attempted, so a cancellation test can assert the rip actually
// stopped instead of grinding through the rest of the queue.
type countingAlbumProvider struct {
url string
trackCount int
mu sync.Mutex
attempts int
cancel context.CancelFunc
cancelAt int
}
func (f *countingAlbumProvider) Source() string { return "qobuz" }
func (f *countingAlbumProvider) Login(context.Context) error { return nil }
func (f *countingAlbumProvider) LoggedIn() bool { return true }
func (f *countingAlbumProvider) Close() error { return nil }
func (f *countingAlbumProvider) Search(context.Context, string, string, int) ([]map[string]any, error) {
return nil, nil
}
func (f *countingAlbumProvider) GetMetadata(_ context.Context, id string, mediaType string) (map[string]any, error) {
if mediaType == "album" {
items := make([]any, 0, f.trackCount)
for i := 0; i < f.trackCount; i++ {
items = append(items, map[string]any{"id": fmt.Sprintf("t%d", i)})
}
return map[string]any{
"title": "Big Album",
"release_date_original": "1977-02-04",
"artist": map[string]any{"name": "Fleetwood Mac"},
"tracks": map[string]any{"items": items},
}, nil
}
// Count metadata fetches as download attempts and trip the cancel once
// enough tracks have been seen.
f.mu.Lock()
f.attempts++
n := f.attempts
f.mu.Unlock()
if f.cancel != nil && n == f.cancelAt {
f.cancel()
}
return map[string]any{
"title": "Track " + id,
"track_number": float64(1),
"performer": map[string]any{"name": "Fleetwood Mac"},
"album": map[string]any{
"title": "Big Album",
"artist": map[string]any{"name": "Fleetwood Mac"},
},
}, nil
}
func (f *countingAlbumProvider) GetDownloadable(_ context.Context, _ string, _ int) (*provider.Downloadable, error) {
return &provider.Downloadable{URL: f.url, Extension: "flac", Source: "qobuz"}, nil
}
func (f *countingAlbumProvider) attemptCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.attempts
}
// TestRipStopsOnCancellation is the regression test for Ctrl-C doing nothing.
// ctx was threaded through the rip but never checked, so cancelling left the
// loops running through every remaining track (and the retry ladder retrying
// on a dead context).
func TestRipStopsOnCancellation(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("fLaC-test-bytes"))
}))
defer ts.Close()
tmp := t.TempDir()
d := config.DefaultConfigData()
d.Downloads.Folder = tmp
d.Downloads.SourceSubdirectories = false
// Serial, so the cancellation point is deterministic.
d.Downloads.Concurrency = false
d.Artwork.Embed = false
d.Artwork.SaveArtwork = false
cfg := &config.Config{File: d, Session: d}
const total = 40
const cancelAfter = 5
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
prov := &countingAlbumProvider{url: ts.URL, trackCount: total, cancel: cancel, cancelAt: cancelAfter}
m := &Main{
Config: cfg,
Providers: map[string]provider.Client{"qobuz": prov},
Store: store.NewDummy(),
DL: download.New(),
Tagger: noopTagger{},
}
if err := m.AddByID(context.Background(), "qobuz", "album", "a1"); err != nil {
t.Fatalf("AddByID() error = %v", err)
}
if err := m.Resolve(context.Background()); err != nil {
t.Fatalf("Resolve() error = %v", err)
}
err := m.Rip(ctx)
if err == nil {
t.Fatal("Rip() returned nil after cancellation, want an error")
}
if !errors.Is(err, context.Canceled) {
t.Errorf("Rip() error = %v, want it to wrap context.Canceled", err)
}
// The rip must stop promptly. Without the cancellation checks it walked
// all 40 tracks; allow a small margin for the in-flight track.
if got := prov.attemptCount(); got > cancelAfter+2 {
t.Errorf("attempted %d track(s) after cancelling at %d; the rip did not stop", got, cancelAfter)
}
}
// TestRipTrackReturnsEarlyOnCancellation pins the funnel check: every worker
// path goes through ripTrack, so one guard there stops all of them.
func TestRipTrackReturnsEarlyOnCancellation(t *testing.T) {
tmp := t.TempDir()
d := config.DefaultConfigData()
d.Downloads.Folder = tmp
cfg := &config.Config{File: d, Session: d}
m := &Main{
Config: cfg,
Providers: map[string]provider.Client{"qobuz": &fakeProvider{}},
Store: store.NewDummy(),
DL: download.New(),
Tagger: noopTagger{},
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
err := m.ripTrack(ctx, &fakeProvider{}, "qobuz", "t1", "", ripTrackOptions{})
if !errors.Is(err, context.Canceled) {
t.Fatalf("ripTrack() error = %v, want context.Canceled", err)
}
}
// TestBookkeepingSurvivesCancellation checks that the failure record still
// lands when the rip was cancelled, which is when it matters most. The store
// writes deliberately use a detached context.
func TestBookkeepingSurvivesCancellation(t *testing.T) {
tmp := t.TempDir()
sqlite, err := store.NewSQLite(filepath.Join(tmp, "db.sqlite"))
if err != nil {
t.Fatal(err)
}
defer func() { _ = sqlite.Close() }()
d := config.DefaultConfigData()
d.Downloads.Folder = tmp
m := &Main{Config: &config.Config{File: d, Session: d}, Store: sqlite, DL: download.New()}
ctx, cancel := context.WithCancel(context.Background())
cancel()
m.markFailed(ctx, "qobuz", "track", "t1")
// Read back with a live context.
failed, err := sqlite.IsFailed(context.Background(), "qobuz", "track", "t1")
if err != nil {
t.Fatal(err)
}
if !failed {
t.Fatal("failure record not written under a cancelled context")
}
}
// countingMetaProvider records how many per-track metadata lookups happen, so
// the album-payload optimization can be asserted rather than assumed.
type countingMetaProvider struct {
url string
complete bool
mu sync.Mutex
trackMeta int
albumMeta int
downloads int
}
func (f *countingMetaProvider) Source() string { return "qobuz" }
func (f *countingMetaProvider) Login(context.Context) error { return nil }
func (f *countingMetaProvider) LoggedIn() bool { return true }
func (f *countingMetaProvider) Close() error { return nil }
func (f *countingMetaProvider) Search(context.Context, string, string, int) ([]map[string]any, error) {
return nil, nil
}
// AlbumTracksAreComplete is only consulted when complete is true; the false
// case still satisfies the interface, which mirrors a provider that opts out.
func (f *countingMetaProvider) AlbumTracksAreComplete() bool { return f.complete }
func (f *countingMetaProvider) GetMetadata(_ context.Context, id string, mediaType string) (map[string]any, error) {
f.mu.Lock()
if mediaType == "album" {
f.albumMeta++
} else {
f.trackMeta++
}
f.mu.Unlock()
if mediaType == "album" {
items := make([]any, 0, 3)
for i := 1; i <= 3; i++ {
items = append(items, map[string]any{
"id": fmt.Sprintf("t%d", i),
"title": fmt.Sprintf("Track %d", i),
"track_number": float64(i),
"media_number": float64(1),
"isrc": fmt.Sprintf("ISRC%d", i),
"copyright": "(C) Label",
"performer": map[string]any{"name": "Fleetwood Mac"},
})
}
return map[string]any{
"title": "Rumours",
"release_date_original": "1977-02-04",
"artist": map[string]any{"name": "Fleetwood Mac"},
"tracks": map[string]any{"items": items},
}, nil
}
return map[string]any{
"title": "Fetched " + id,
"track_number": float64(1),
"performer": map[string]any{"name": "Fleetwood Mac"},
"album": map[string]any{"title": "Rumours"},
}, nil
}
func (f *countingMetaProvider) GetDownloadable(_ context.Context, _ string, _ int) (*provider.Downloadable, error) {
f.mu.Lock()
f.downloads++
f.mu.Unlock()
return &provider.Downloadable{URL: f.url, Extension: "flac", Source: "qobuz"}, nil
}
func (f *countingMetaProvider) counts() (album, track, downloads int) {
f.mu.Lock()
defer f.mu.Unlock()
return f.albumMeta, f.trackMeta, f.downloads
}
func runAlbumRip(t *testing.T, prov *countingMetaProvider) string {
t.Helper()
tmp := t.TempDir()
d := config.DefaultConfigData()
d.Downloads.Folder = tmp
d.Downloads.SourceSubdirectories = false
d.Downloads.Concurrency = false
d.Artwork.Embed = false
d.Artwork.SaveArtwork = false
cfg := &config.Config{File: d, Session: d}
m := &Main{
Config: cfg,
Providers: map[string]provider.Client{"qobuz": prov},
Store: store.NewDummy(),
DL: download.New(),
Tagger: noopTagger{},
}
ctx := context.Background()
if err := m.AddByID(ctx, "qobuz", "album", "a1"); err != nil {
t.Fatalf("AddByID() error = %v", err)
}
if err := m.Resolve(ctx); err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if err := m.Rip(ctx); err != nil {
t.Fatalf("Rip() error = %v", err)
}
return tmp
}
// TestAlbumRipSkipsPerTrackMetadataWhenEmbedded is the performance regression
// test: a provider that embeds complete track objects in its album payload
// must not be asked for each track again. Every such call goes through the
// requests-per-minute limiter, so on a 68-track album the redundant lookups
// were ~68 seconds of pure waiting.
func TestAlbumRipSkipsPerTrackMetadataWhenEmbedded(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("fLaC"))
}))
defer ts.Close()
prov := &countingMetaProvider{url: ts.URL, complete: true}
tmp := runAlbumRip(t, prov)
album, track, _ := prov.counts()
if album != 1 {
t.Errorf("album metadata fetches = %d, want 1", album)
}
if track != 0 {
t.Errorf("per-track metadata fetches = %d, want 0 (album payload is complete)", track)
}
// The embedded titles must be what lands on disk, not the fetched ones.
albumDir := soleSubdir(t, tmp)
for i := 1; i <= 3; i++ {
name := fmt.Sprintf("%02d. Fleetwood Mac - Track %d.flac", i, i)
if _, err := os.Stat(filepath.Join(albumDir, name)); err != nil {
entries, _ := os.ReadDir(albumDir)
var got []string
for _, e := range entries {
got = append(got, e.Name())
}
t.Fatalf("expected %q from the embedded metadata; album dir contains %v", name, got)
}
}
}
// TestAlbumRipFetchesPerTrackMetadataWhenNotEmbedded pins the opt-out path, so
// providers like Tidal (whose track lookup also fetches lyrics) keep working.
func TestAlbumRipFetchesPerTrackMetadataWhenNotEmbedded(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("fLaC"))
}))
defer ts.Close()
prov := &countingMetaProvider{url: ts.URL, complete: false}
runAlbumRip(t, prov)
_, track, _ := prov.counts()
if track != 3 {
t.Errorf("per-track metadata fetches = %d, want 3 (provider opted out)", track)
}
}
// TestAlbumTrackMetadataAttachesParentAlbum covers the one field the live
// Qobuz check showed track/get adds: the nested album object.
func TestAlbumTrackMetadataAttachesParentAlbum(t *testing.T) {
albumMeta := map[string]any{
"title": "Rumours",
"replaygain_album_gain": -7.5,
"tracks": map[string]any{"items": []any{
map[string]any{"id": "t1", "title": "Dreams", "track_number": float64(1)},
}},
}
items, _ := anySlice(albumMeta["tracks"].(map[string]any)["items"])
got := albumTrackMetadata(albumMeta, items)
meta, ok := got["t1"]
if !ok {
t.Fatal("no metadata built for t1")
}
alb, ok := meta["album"].(map[string]any)
if !ok {
t.Fatal("parent album not attached")
}
if alb["title"] != "Rumours" {
t.Errorf("album title = %v", alb["title"])
}
// Album-level ReplayGain is read through trackMeta["album"].
if alb["replaygain_album_gain"] != -7.5 {
t.Errorf("album replaygain lost: %v", alb["replaygain_album_gain"])
}
// The attached album must not carry the whole track list back in.
if _, has := alb["tracks"]; has {
t.Error("attached album still has its track list; that is O(n^2) memory on a large album")
}
}
func TestTrackMetaUsable(t *testing.T) {
tests := []struct {
name string
meta map[string]any
want bool
}{
{"nil", nil, false},
{"empty", map[string]any{}, false},
{"complete", map[string]any{"title": "A", "track_number": float64(1)}, true},
{"no title", map[string]any{"track_number": float64(1)}, false},
{"blank title", map[string]any{"title": " ", "track_number": float64(1)}, false},
{"no track number", map[string]any{"title": "A"}, false},
{"zero track number", map[string]any{"title": "A", "track_number": float64(0)}, false},
{"string track number", map[string]any{"title": "A", "track_number": "2"}, true},
}
for _, tc := range tests {
if got := trackMetaUsable(tc.meta); got != tc.want {
t.Errorf("%s: trackMetaUsable = %v, want %v", tc.name, got, tc.want)
}
}
}
// soleSubdir returns the single directory inside dir, which for an album rip
// is the generated album folder.
func soleSubdir(t *testing.T, dir string) string {
t.Helper()
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
var dirs []string
for _, e := range entries {
if e.IsDir() {
dirs = append(dirs, filepath.Join(dir, e.Name()))
}
}
if len(dirs) != 1 {
t.Fatalf("expected exactly one album folder in %s, got %v", dir, dirs)
}
return dirs[0]
}
// lyricsProvider is a countingMetaProvider that also serves lyrics, to prove
// lyrics still reach the tagger on the album fast path, where the per-track
// metadata call is skipped.
type lyricsProvider struct {
*countingMetaProvider
plain string
synced string
calls int
err error
}
func (f *lyricsProvider) TrackLyrics(context.Context, string) (string, string, error) {
f.mu.Lock()
f.calls++
f.mu.Unlock()
if f.err != nil {
return "", "", f.err
}
return f.plain, f.synced, nil
}
type capturingTagger struct {
mu sync.Mutex
seen []tag.Metadata
}
func (c *capturingTagger) TagFLAC(_ context.Context, _ string, meta tag.Metadata, _ string) error {
c.mu.Lock()
defer c.mu.Unlock()
c.seen = append(c.seen, meta)
return nil
}
func (c *capturingTagger) lyrics() []string {
c.mu.Lock()
defer c.mu.Unlock()
out := make([]string, 0, len(c.seen))
for _, m := range c.seen {
out = append(out, m.Lyrics)
}
return out
}
func runAlbumRipWithTagger(t *testing.T, prov provider.Client, tagger trackTagger, exclude ...string) {
t.Helper()
d := config.DefaultConfigData()
d.Downloads.Folder = t.TempDir()
d.Downloads.Concurrency = false
d.Artwork.Embed = false
d.Artwork.SaveArtwork = false
d.Metadata.Exclude = exclude
cfg := &config.Config{File: d, Session: d}
m := &Main{
Config: cfg,
Providers: map[string]provider.Client{"qobuz": prov},
Store: store.NewDummy(),
DL: download.New(),
Tagger: tagger,
}
ctx := context.Background()
if err := m.AddByID(ctx, "qobuz", "album", "a1"); err != nil {
t.Fatalf("AddByID() error = %v", err)
}
if err := m.Resolve(ctx); err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if err := m.Rip(ctx); err != nil {
t.Fatalf("Rip() error = %v", err)
}
}
func TestAlbumRipAttachesLyricsOnFastPath(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("fLaC"))
}))
defer ts.Close()
prov := &lyricsProvider{
countingMetaProvider: &countingMetaProvider{url: ts.URL, complete: true},
plain: "line one\nline two",
synced: "[00:01.00]line one",
}
tagger := &capturingTagger{}
runAlbumRipWithTagger(t, prov, tagger)
_, trackMeta, _ := prov.counts()
if trackMeta != 0 {
t.Errorf("per-track metadata fetches = %d, want 0 (fast path must be preserved)", trackMeta)
}
if prov.calls != 3 {
t.Errorf("TrackLyrics calls = %d, want 3 (one per track)", prov.calls)
}
got := tagger.lyrics()
if len(got) != 3 {
t.Fatalf("tagged %d tracks, want 3", len(got))
}
for i, l := range got {
// Synced lyrics win over plain, matching the existing behaviour.
if l != "[00:01.00]line one" {
t.Errorf("track %d lyrics = %q", i, l)
}
}
}
func TestAlbumRipSkipsLyricsWhenExcluded(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("fLaC"))
}))
defer ts.Close()
prov := &lyricsProvider{
countingMetaProvider: &countingMetaProvider{url: ts.URL, complete: true},
plain: "should not be fetched",
}
runAlbumRipWithTagger(t, prov, noopTagger{}, "lyrics")
if prov.calls != 0 {
t.Errorf("TrackLyrics calls = %d, want 0 when metadata.exclude contains lyrics", prov.calls)
}
}
// A lyrics failure is cosmetic and must never fail the download.
func TestAlbumRipSurvivesLyricsFailure(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("fLaC"))
}))
defer ts.Close()
prov := &lyricsProvider{
countingMetaProvider: &countingMetaProvider{url: ts.URL, complete: true},
err: errors.New("lyrics service down"),
}
tagger := &capturingTagger{}
runAlbumRipWithTagger(t, prov, tagger)
if n := len(tagger.lyrics()); n != 3 {
t.Fatalf("tagged %d tracks, want 3 despite the lyrics failure", n)
}
for i, l := range tagger.lyrics() {
if l != "" {
t.Errorf("track %d should have no lyrics, got %q", i, l)
}
}
}
// Providers that already attach lyrics in their metadata (Tidal, Deezer) must
// not be asked again.
func TestLyricsNotRefetchedWhenMetadataAlreadyHasThem(t *testing.T) {
d := config.DefaultConfigData()
m := &Main{Config: &config.Config{File: d, Session: d}, DL: download.New()}
prov := &lyricsProvider{countingMetaProvider: &countingMetaProvider{}, plain: "fresh"}
meta := map[string]any{"lyrics_synced": "[00:00.00]already here"}
m.attachLyrics(context.Background(), prov, meta, "t1")
if prov.calls != 0 {
t.Errorf("TrackLyrics calls = %d, want 0 when metadata already carries lyrics", prov.calls)
}
if meta["lyrics_synced"] != "[00:00.00]already here" {
t.Errorf("existing lyrics overwritten: %v", meta["lyrics_synced"])
}
}
+56 -13
View File
@@ -6,10 +6,12 @@ import (
"fmt" "fmt"
"image" "image"
"image/jpeg" "image/jpeg"
_ "image/png"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"golang.org/x/image/draw" "golang.org/x/image/draw"
@@ -31,6 +33,52 @@ var (
tempDirs = map[string]struct{}{} tempDirs = map[string]struct{}{}
) )
var (
folderLocksMu sync.Mutex
folderLocks = map[string]*sync.Mutex{}
)
// lockFolder serializes Prepare calls that target the same album folder and
// returns the unlock function.
//
// ripTrack calls Prepare once per track with the same albumFolder for every
// worker in a playlist rip, and the cover handling below is check-then-write.
// Without this, two workers both saw a missing cover.jpg, both created it, and
// the one that failed removed the file the other had just finished writing.
func lockFolder(folder string) func() {
folderLocksMu.Lock()
mu, ok := folderLocks[folder]
if !ok {
mu = &sync.Mutex{}
folderLocks[folder] = mu
}
folderLocksMu.Unlock()
mu.Lock()
return mu.Unlock
}
// fetchAtomic downloads sourceURL to destPath via a unique temp file in the
// same directory, then renames it into place. Downloading straight to destPath
// meant a failed attempt could delete a file another worker had completed.
func fetchAtomic(ctx context.Context, dl Downloader, sourceURL, destPath string, maxWidth int) bool {
tmpPath := fmt.Sprintf("%s.part-%d-%d", destPath, os.Getpid(), atomic.AddUint64(&tmpCounter, 1))
if err := dl.FileNoProgress(ctx, sourceURL, tmpPath); err != nil {
_ = os.Remove(tmpPath)
return false
}
if maxWidth > 0 {
_ = downscaleImage(tmpPath, maxWidth)
}
if err := os.Rename(tmpPath, destPath); err != nil {
_ = os.Remove(tmpPath)
return false
}
return true
}
var tmpCounter uint64
func Prepare(ctx context.Context, dl Downloader, folder string, albumMeta map[string]any, cfg config.ArtworkConfig, forPlaylist bool) (Result, error) { func Prepare(ctx context.Context, dl Downloader, folder string, albumMeta map[string]any, cfg config.ArtworkConfig, forPlaylist bool) (Result, error) {
saveArtwork := cfg.SaveArtwork saveArtwork := cfg.SaveArtwork
if forPlaylist { if forPlaylist {
@@ -51,30 +99,25 @@ func Prepare(ctx context.Context, dl Downloader, folder string, albumMeta map[st
embedURL = largestURL embedURL = largestURL
} }
// Everything below is check-then-write against shared paths.
unlock := lockFolder(folder)
defer unlock()
result := Result{} result := Result{}
if saveArtwork && largestURL != "" { if saveArtwork && largestURL != "" {
savedPath := filepath.Join(folder, "cover.jpg") savedPath := filepath.Join(folder, "cover.jpg")
if fileExists(savedPath) { if fileExists(savedPath) || fetchAtomic(ctx, dl, largestURL, savedPath, cfg.SavedMaxWidth) {
result.SavedPath = savedPath
} else if err := dl.FileNoProgress(ctx, largestURL, savedPath); err == nil {
if cfg.SavedMaxWidth > 0 {
_ = downscaleImage(savedPath, cfg.SavedMaxWidth)
}
result.SavedPath = savedPath result.SavedPath = savedPath
} }
} }
if cfg.Embed && embedURL != "" { if cfg.Embed && embedURL != "" {
embedDir := sessionEmbedDir(folder) embedDir := sessionEmbedDir(folder)
if err := os.MkdirAll(embedDir, 0o755); err == nil { // 0o700: predictable path under the shared system temp dir.
if err := os.MkdirAll(embedDir, 0o700); err == nil {
registerTempDir(embedDir) registerTempDir(embedDir)
embedPath := filepath.Join(embedDir, embedFilename(embedURL)) embedPath := filepath.Join(embedDir, embedFilename(embedURL))
if fileExists(embedPath) { if fileExists(embedPath) || fetchAtomic(ctx, dl, embedURL, embedPath, cfg.EmbedMaxWidth) {
result.EmbedPath = embedPath
} else if err := dl.FileNoProgress(ctx, embedURL, embedPath); err == nil {
if cfg.EmbedMaxWidth > 0 {
_ = downscaleImage(embedPath, cfg.EmbedMaxWidth)
}
result.EmbedPath = embedPath result.EmbedPath = embedPath
} }
} }
+177
View File
@@ -0,0 +1,177 @@
package artwork
import (
"context"
"errors"
"os"
"path/filepath"
"sync"
"testing"
"time"
"streamrip-go/internal/config"
)
// fakeDownloader models the real downloader's behaviour for this race: the
// first attempt at a given output path succeeds, and any later attempt at that
// same path fails and removes its partial output (downloader.go does
// os.Remove(outputPath) on failure). That is exactly how a losing worker used
// to delete the cover a winning worker had just written.
type fakeDownloader struct {
payload []byte
mu sync.Mutex
seen map[string]bool
delay time.Duration
}
func newFakeDownloader(payload []byte, delay time.Duration) *fakeDownloader {
return &fakeDownloader{payload: payload, seen: map[string]bool{}, delay: delay}
}
func (f *fakeDownloader) File(ctx context.Context, sourceURL, outputPath string) error {
return f.FileNoProgress(ctx, sourceURL, outputPath)
}
func (f *fakeDownloader) FileNoProgress(_ context.Context, _, outputPath string) error {
f.mu.Lock()
first := !f.seen[outputPath]
f.seen[outputPath] = true
f.mu.Unlock()
if !first {
_ = os.Remove(outputPath)
return errors.New("download failed")
}
if err := os.MkdirAll(filepath.Dir(outputPath), 0o700); err != nil {
return err
}
// Widen the window between another worker's existence check and this write.
time.Sleep(f.delay)
return os.WriteFile(outputPath, f.payload, 0o600)
}
func artworkCfg() config.ArtworkConfig {
return config.ArtworkConfig{Embed: true, EmbedSize: "large", SaveArtwork: true}
}
func albumMeta() map[string]any {
return map[string]any{"image": map[string]any{
"original": "https://example.invalid/large.jpg",
"large": "https://example.invalid/large.jpg",
}}
}
// TestPrepareConcurrentSameFolder is the regression test for the cover race:
// many workers calling Prepare for one album folder must leave exactly one
// intact cover.jpg. Run with -race.
func TestPrepareConcurrentSameFolder(t *testing.T) {
folder := t.TempDir()
payload := []byte("jpeg-bytes")
dl := newFakeDownloader(payload, 5*time.Millisecond)
const workers = 16
var wg sync.WaitGroup
results := make([]Result, workers)
for i := 0; i < workers; i++ {
wg.Add(1)
go func(idx int) {
defer wg.Done()
res, err := Prepare(context.Background(), dl, folder, albumMeta(), artworkCfg(), false)
if err != nil {
t.Errorf("Prepare: %v", err)
}
results[idx] = res
}(i)
}
wg.Wait()
saved := filepath.Join(folder, "cover.jpg")
got, err := os.ReadFile(saved)
if err != nil {
t.Fatalf("cover.jpg missing after %d concurrent Prepare calls: %v", workers, err)
}
if string(got) != string(payload) {
t.Fatalf("cover.jpg corrupted: got %q want %q", got, payload)
}
// No .part- temp files may survive.
entries, err := os.ReadDir(folder)
if err != nil {
t.Fatal(err)
}
for _, e := range entries {
if filepath.Ext(e.Name()) != ".jpg" {
t.Errorf("leftover temp artifact in album folder: %s", e.Name())
}
}
for i, res := range results {
if res.SavedPath != saved {
t.Errorf("worker %d: SavedPath = %q want %q", i, res.SavedPath, saved)
}
if res.EmbedPath == "" {
t.Errorf("worker %d: EmbedPath empty", i)
}
}
}
// TestPrepareFailedDownloadKeepsExistingCover covers the specific corruption
// mode: a failing download must not remove a cover that is already on disk.
func TestPrepareFailedDownloadKeepsExistingCover(t *testing.T) {
folder := t.TempDir()
saved := filepath.Join(folder, "cover.jpg")
if err := os.WriteFile(saved, []byte("good-cover"), 0o600); err != nil {
t.Fatal(err)
}
// Mark the saved path as already attempted so any download of it fails.
dl := newFakeDownloader([]byte("new"), 0)
dl.seen[saved] = true
if _, err := Prepare(context.Background(), dl, folder, albumMeta(), artworkCfg(), false); err != nil {
t.Fatalf("Prepare: %v", err)
}
got, err := os.ReadFile(saved)
if err != nil {
t.Fatalf("cover.jpg removed: %v", err)
}
if string(got) != "good-cover" {
t.Fatalf("existing cover overwritten: got %q", got)
}
}
// TestFetchAtomicFailureLeavesNoPartial asserts a failed fetch neither creates
// the destination nor leaves a temp file behind.
func TestFetchAtomicFailureLeavesNoPartial(t *testing.T) {
folder := t.TempDir()
dest := filepath.Join(folder, "cover.jpg")
dl := newFakeDownloader([]byte("x"), 0)
dl.seen[dest+".forced"] = true
// Force failure by pre-marking every temp path is impractical, so use a
// downloader that always fails instead.
always := &failingDownloader{}
if ok := fetchAtomic(context.Background(), always, "https://example.invalid/a.jpg", dest, 0); ok {
t.Fatal("fetchAtomic reported success on a failing download")
}
if _, err := os.Stat(dest); !os.IsNotExist(err) {
t.Fatalf("destination should not exist, stat err = %v", err)
}
entries, err := os.ReadDir(folder)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
t.Fatalf("temp files left behind: %v", entries)
}
}
type failingDownloader struct{}
func (failingDownloader) File(context.Context, string, string) error { return errors.New("nope") }
func (failingDownloader) FileNoProgress(_ context.Context, _, outputPath string) error {
_ = os.WriteFile(outputPath, []byte("partial"), 0o600)
return errors.New("nope")
}
+3 -2
View File
@@ -1,6 +1,7 @@
package convert package convert
import ( import (
"context"
"fmt" "fmt"
"os" "os"
"os/exec" "os/exec"
@@ -27,7 +28,7 @@ var profiles = map[string]profile{
"AAC": {codecLib: "aac", ext: "m4a", lossless: false}, "AAC": {codecLib: "aac", ext: "m4a", lossless: false},
} }
func Convert(path string, cfg config.ConversionConfig) (string, error) { func Convert(ctx context.Context, path string, cfg config.ConversionConfig) (string, error) {
if !cfg.Enabled { if !cfg.Enabled {
return path, nil return path, nil
} }
@@ -45,7 +46,7 @@ func Convert(path string, cfg config.ConversionConfig) (string, error) {
tmpPath := finalPath + ".tmp." + p.ext tmpPath := finalPath + ".tmp." + p.ext
args := buildFFmpegArgs(path, tmpPath, p, cfg) args := buildFFmpegArgs(path, tmpPath, p, cfg)
cmd := exec.Command("ffmpeg", args...) cmd := exec.CommandContext(ctx, "ffmpeg", args...)
output, err := cmd.CombinedOutput() output, err := cmd.CombinedOutput()
if err != nil { if err != nil {
_ = os.Remove(tmpPath) _ = os.Remove(tmpPath)
+2 -1
View File
@@ -1,6 +1,7 @@
package convert package convert
import ( import (
"context"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -86,7 +87,7 @@ func TestConvertKeepsSourceWhenRenameFails(t *testing.T) {
} }
cfg := config.ConversionConfig{Enabled: true, Codec: "ALAC"} cfg := config.ConversionConfig{Enabled: true, Codec: "ALAC"}
out, err := Convert(in, cfg) out, err := Convert(context.Background(), in, cfg)
if err == nil { if err == nil {
t.Fatalf("expected rename failure") t.Fatalf("expected rename failure")
} }
+6 -5
View File
@@ -1,6 +1,7 @@
package tag package tag
import ( import (
"context"
"encoding/binary" "encoding/binary"
"fmt" "fmt"
"os" "os"
@@ -45,17 +46,17 @@ func New() *Tagger {
return &Tagger{} return &Tagger{}
} }
func (t *Tagger) TagFLAC(path string, meta Metadata, coverPath string) error { func (t *Tagger) TagFLAC(ctx context.Context, path string, meta Metadata, coverPath string) error {
if _, err := exec.LookPath("ffmpeg"); err != nil { if _, err := exec.LookPath("ffmpeg"); err != nil {
return fmt.Errorf("ffmpeg not found: %w", err) return fmt.Errorf("ffmpeg not found: %w", err)
} }
ext := strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".") ext := strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")
forceMP4Muxer := shouldForceMP4Muxer(path, ext) forceMP4Muxer := shouldForceMP4Muxer(ctx, path, ext)
tmpPath := taggedTempPath(path) tmpPath := taggedTempPath(path)
runTag := func(cover string) ([]byte, error) { runTag := func(cover string) ([]byte, error) {
args := buildFFmpegArgs(path, tmpPath, meta, cover, ext, forceMP4Muxer) args := buildFFmpegArgs(path, tmpPath, meta, cover, ext, forceMP4Muxer)
cmd := exec.Command("ffmpeg", args...) cmd := exec.CommandContext(ctx, "ffmpeg", args...)
return cmd.CombinedOutput() return cmd.CombinedOutput()
} }
@@ -125,7 +126,7 @@ func buildFFmpegArgs(inputPath, outputPath string, meta Metadata, coverPath, ext
return args return args
} }
func shouldForceMP4Muxer(path, ext string) bool { func shouldForceMP4Muxer(ctx context.Context, path, ext string) bool {
switch strings.TrimPrefix(strings.ToLower(ext), ".") { switch strings.TrimPrefix(strings.ToLower(ext), ".") {
case "m4a", "mp4": case "m4a", "mp4":
default: default:
@@ -134,7 +135,7 @@ func shouldForceMP4Muxer(path, ext string) bool {
if _, err := exec.LookPath("ffprobe"); err != nil { if _, err := exec.LookPath("ffprobe"); err != nil {
return false return false
} }
out, err := exec.Command("ffprobe", "-v", "error", "-select_streams", "a:0", "-show_entries", "stream=codec_name", "-of", "default=nokey=1:noprint_wrappers=1", path).Output() out, err := exec.CommandContext(ctx, "ffprobe", "-v", "error", "-select_streams", "a:0", "-show_entries", "stream=codec_name", "-of", "default=nokey=1:noprint_wrappers=1", path).Output()
if err != nil { if err != nil {
return false return false
} }
+262 -43
View File
@@ -5,6 +5,9 @@ import (
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"sync"
"time"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
) )
@@ -13,8 +16,17 @@ const CurrentConfigVersion = "2.2.0"
var ErrOutdatedConfig = errors.New("config version mismatch") var ErrOutdatedConfig = errors.New("config version mismatch")
// Config holds the on-disk config (File) and the runtime view with CLI
// overrides applied (Session).
//
// mu guards File, Session and the write to Path. Provider clients refresh OAuth
// tokens from inside concurrent download workers, so both the struct fields and
// the config-file write have to be serialized; without this, two workers
// refreshing at once raced each other and issued two concurrent os.WriteFile
// calls against the same path.
type Config struct { type Config struct {
Path string Path string
mu sync.Mutex
File ConfigData File ConfigData
Session ConfigData Session ConfigData
} }
@@ -27,7 +39,6 @@ type ConfigData struct {
Yandex YandexConfig `toml:"yandex"` Yandex YandexConfig `toml:"yandex"`
Beatport BeatportConfig `toml:"beatport"` Beatport BeatportConfig `toml:"beatport"`
Soundcloud SoundcloudConfig `toml:"soundcloud"` Soundcloud SoundcloudConfig `toml:"soundcloud"`
Youtube YoutubeConfig `toml:"youtube"`
Database DatabaseConfig `toml:"database"` Database DatabaseConfig `toml:"database"`
Conversion ConversionConfig `toml:"conversion"` Conversion ConversionConfig `toml:"conversion"`
QobuzFilters QobuzDiscographyFilterConfig `toml:"qobuz_filters"` QobuzFilters QobuzDiscographyFilterConfig `toml:"qobuz_filters"`
@@ -50,13 +61,17 @@ type DownloadsConfig struct {
} }
type QobuzConfig struct { type QobuzConfig struct {
Quality int `toml:"quality"` Quality int `toml:"quality"`
DownloadBooklets bool `toml:"download_booklets"` UseAuthToken bool `toml:"use_auth_token"`
UseAuthToken bool `toml:"use_auth_token"` EmailOrUserID string `toml:"email_or_userid"`
EmailOrUserID string `toml:"email_or_userid"` PasswordOrToken string `toml:"password_or_token"`
PasswordOrToken string `toml:"password_or_token"` AppID string `toml:"app_id"`
AppID string `toml:"app_id"` Secrets []string `toml:"secrets"`
Secrets []string `toml:"secrets"`
// Vendor app credentials for the mobile API fallback. Defaulted in code;
// override here when Qobuz rotates them, instead of waiting for a release.
MobileAppID string `toml:"mobile_app_id"`
MobileAppSecret string `toml:"mobile_app_secret"`
} }
type TidalConfig struct { type TidalConfig struct {
@@ -68,6 +83,10 @@ type TidalConfig struct {
AccessToken string `toml:"access_token"` AccessToken string `toml:"access_token"`
RefreshToken string `toml:"refresh_token"` RefreshToken string `toml:"refresh_token"`
TokenExpiry int64 `toml:"token_expiry"` TokenExpiry int64 `toml:"token_expiry"`
// Vendor OAuth app credentials. Defaulted in code; override on rotation.
AppClientID string `toml:"app_client_id"`
AppClientSecret string `toml:"app_client_secret"`
} }
type DeezerConfig struct { type DeezerConfig struct {
@@ -77,12 +96,21 @@ type DeezerConfig struct {
Email string `toml:"email"` Email string `toml:"email"`
Password string `toml:"password"` Password string `toml:"password"`
RefreshToken string `toml:"refresh_token"` RefreshToken string `toml:"refresh_token"`
// Vendor protocol keys. Defaulted in code; override on rotation.
APIKey string `toml:"api_key"`
GatewayKey string `toml:"gateway_key"`
BlowfishSecret string `toml:"blowfish_secret"`
} }
type YandexConfig struct { type YandexConfig struct {
Quality int `toml:"quality"` Quality int `toml:"quality"`
AccessToken string `toml:"access_token"` AccessToken string `toml:"access_token"`
UserID string `toml:"user_id"` UserID string `toml:"user_id"`
// Vendor request-signing keys. Defaulted in code; override on rotation.
SignKey string `toml:"sign_key"`
LegacyMP3SignSalt string `toml:"legacy_mp3_sign_salt"`
} }
type BeatportConfig struct { type BeatportConfig struct {
@@ -93,18 +121,15 @@ type BeatportConfig struct {
AccessToken string `toml:"access_token"` AccessToken string `toml:"access_token"`
RefreshToken string `toml:"refresh_token"` RefreshToken string `toml:"refresh_token"`
TokenExpiry int64 `toml:"token_expiry"` TokenExpiry int64 `toml:"token_expiry"`
// Vendor OAuth app credential. Defaulted in code; override on rotation.
AppClientID string `toml:"app_client_id"`
} }
// SoundcloudConfig has no client_id/app_version: soundcloud is handled via
// yt-dlp, which supplies its own, so those keys were read by no code.
type SoundcloudConfig struct { type SoundcloudConfig struct {
Quality int `toml:"quality"` Quality int `toml:"quality"`
ClientID string `toml:"client_id"`
AppVersion string `toml:"app_version"`
}
type YoutubeConfig struct {
Quality int `toml:"quality"`
DownloadVideos bool `toml:"download_videos"`
VideoDownloadsFolder string `toml:"video_downloads_folder"`
} }
type DatabaseConfig struct { type DatabaseConfig struct {
@@ -166,8 +191,7 @@ type CLIConfig struct {
} }
type MiscConfig struct { type MiscConfig struct {
Version string `toml:"version"` Version string `toml:"version"`
CheckForUpdates bool `toml:"check_for_updates"`
} }
func Load(path string) (*Config, error) { func Load(path string) (*Config, error) {
@@ -202,34 +226,219 @@ func Load(path string) (*Config, error) {
return nil, fmt.Errorf("%w: need to update from %q to %q", ErrOutdatedConfig, data.Misc.Version, CurrentConfigVersion) return nil, fmt.Errorf("%w: need to update from %q to %q", ErrOutdatedConfig, data.Misc.Version, CurrentConfigVersion)
} }
if err = Validate(&data); err != nil {
return nil, err
}
return &Config{Path: resolvedPath, File: data, Session: cloneConfigData(data)}, nil return &Config{Path: resolvedPath, File: data, Session: cloneConfigData(data)}, nil
} }
func UpgradeOutdated(path string) (string, error) { // Vendor app credentials and protocol keys.
//
// These are the provider apps' own identifiers and signing keys, not user
// secrets, and they are baked in so the tool works out of the box. They are
// mirrored into the config so that when a provider rotates one, a user can fix
// their install with a config edit instead of waiting for a new release.
const (
DefaultQobuzMobileAppID = "312369995"
DefaultQobuzMobileAppSecret = "e79f8b9be485692b0e5f9dd895826368"
DefaultTidalAppClientID = "fX2JxdmntZWK0ixT"
DefaultTidalAppClientSecret = "1Nm5AfDAjxrgJFJbKNWLeAyKGVGmINuXPPLHVXAvxAg="
DefaultDeezerAPIKey = "4VCYIJUCDLOUELGD1V8WBVYBNVDYOXEWSLLZDONGBBDFVXTZJRXPR29JRLQFO6ZE"
DefaultDeezerGatewayKey = "VBK1FSUEXHTSDBJJ"
DefaultDeezerBlowfishSecret = "g4el58wc0zvf9na1"
DefaultYandexSignKey = "kzqU4XhfCaY6B6JTHODeq5"
DefaultYandexLegacyMP3Salt = "XGRlBW9FXlekgbPrRHuSiA"
DefaultBeatportAppClientID = "ryZ8LuyQVPqbK2mBX2Hwt4qSMtnWuTYSqBPO92yQ"
)
// applyVendorDefaults fills in any vendor credential the user has not set, so
// the provider clients never have to nil-check them.
func applyVendorDefaults(data *ConfigData) {
if data.Qobuz.MobileAppID == "" {
data.Qobuz.MobileAppID = DefaultQobuzMobileAppID
}
if data.Qobuz.MobileAppSecret == "" {
data.Qobuz.MobileAppSecret = DefaultQobuzMobileAppSecret
}
if data.Tidal.AppClientID == "" {
data.Tidal.AppClientID = DefaultTidalAppClientID
}
if data.Tidal.AppClientSecret == "" {
data.Tidal.AppClientSecret = DefaultTidalAppClientSecret
}
if data.Deezer.APIKey == "" {
data.Deezer.APIKey = DefaultDeezerAPIKey
}
if data.Deezer.GatewayKey == "" {
data.Deezer.GatewayKey = DefaultDeezerGatewayKey
}
if data.Deezer.BlowfishSecret == "" {
data.Deezer.BlowfishSecret = DefaultDeezerBlowfishSecret
}
if data.Yandex.SignKey == "" {
data.Yandex.SignKey = DefaultYandexSignKey
}
if data.Yandex.LegacyMP3SignSalt == "" {
data.Yandex.LegacyMP3SignSalt = DefaultYandexLegacyMP3Salt
}
if data.Beatport.AppClientID == "" {
data.Beatport.AppClientID = DefaultBeatportAppClientID
}
}
// qualityRange is the accepted quality id range per source on the global 0-4
// scale.
//
// The ceiling is 4 everywhere because each provider maps the scale onto its own
// ladder and saturates at the top: deezer's buildFormatPriority treats anything
// >= 2 as FLAC, and soundcloud ignores the value entirely. Only the floor
// differs: Qobuz and Beatport have no equivalent of tier 0.
var qualityRange = map[string][2]int{
"qobuz": {1, 4},
"tidal": {0, 4},
"deezer": {0, 4},
"soundcloud": {0, 4},
"yandex": {0, 4},
"beatport": {1, 4},
}
// Validate rejects config values that would otherwise fail much later, deep in
// a rip, or be silently patched over at individual call sites. Load previously
// checked nothing but misc.version, so a negative max_connections or a bogus
// conversion codec was accepted and only noticed mid-download.
//
// It normalizes what can be normalized (the conversion codec, which until now
// was only canonicalized when it arrived via --codec) and errors on the rest.
func Validate(data *ConfigData) error {
for source, bounds := range qualityRange {
var q int
switch source {
case "qobuz":
q = data.Qobuz.Quality
case "tidal":
q = data.Tidal.Quality
case "deezer":
q = data.Deezer.Quality
case "soundcloud":
q = data.Soundcloud.Quality
case "yandex":
q = data.Yandex.Quality
case "beatport":
q = data.Beatport.Quality
}
if q < bounds[0] || q > bounds[1] {
return fmt.Errorf("config: %s.quality is %d, must be %d-%d", source, q, bounds[0], bounds[1])
}
}
if data.Downloads.MaxConnections < 1 {
return fmt.Errorf("config: downloads.max_connections is %d, must be >= 1", data.Downloads.MaxConnections)
}
if data.Downloads.RequestsPerMinute == 0 {
return errors.New("config: downloads.requests_per_minute is 0; use a positive value, or a negative value to disable throttling")
}
if data.Filepaths.TruncateTo < 1 {
return fmt.Errorf("config: filepaths.truncate_to is %d, must be >= 1", data.Filepaths.TruncateTo)
}
if data.Conversion.Enabled {
codec, err := NormalizeCodec(data.Conversion.Codec)
if err != nil {
return fmt.Errorf("config: conversion.codec: %w", err)
}
data.Conversion.Codec = codec
}
// artwork.*_max_width uses -1 (and any non-positive value) to mean "do not
// downscale", so there is no lower bound to enforce here.
return nil
}
// NormalizeCodec canonicalizes a conversion codec name. It lives here, rather
// than only in the CLI flag parser, so a codec set in the config file goes
// through the same check.
func NormalizeCodec(raw string) (string, error) {
codec := strings.ToUpper(strings.TrimSpace(raw))
switch codec {
case "ALAC", "FLAC", "MP3", "AAC", "VORBIS":
return codec, nil
case "OGG":
return "VORBIS", nil
default:
return "", fmt.Errorf("unsupported codec %q (expected ALAC, FLAC, OGG, MP3, AAC)", raw)
}
}
// UpgradeOutdated rewrites an outdated config at the current version. It runs
// automatically, without prompting, so it copies the original aside first: the
// rewrite re-serializes the whole file from the parsed struct, which drops any
// key this version does not know about. It returns the config path and the
// backup path.
//
// This is a version stamp, not a migration framework. There are no per-version
// transforms, so a future key rename still needs explicit handling here.
func UpgradeOutdated(path string) (resolved string, backup string, err error) {
resolvedPath, err := resolvePath(path) resolvedPath, err := resolvePath(path)
if err != nil { if err != nil {
return "", err return "", "", err
} }
raw, err := os.ReadFile(resolvedPath) raw, err := os.ReadFile(resolvedPath)
if err != nil { if err != nil {
return "", err return "", "", err
} }
backupPath := fmt.Sprintf("%s.bak-%d", resolvedPath, time.Now().Unix())
if err = os.WriteFile(backupPath, raw, 0o600); err != nil {
return "", "", fmt.Errorf("could not back up %s before upgrading: %w", resolvedPath, err)
}
data := DefaultConfigData() data := DefaultConfigData()
if err = toml.Unmarshal(raw, &data); err != nil { if err = toml.Unmarshal(raw, &data); err != nil {
return "", err return "", "", err
} }
applyRuntimeDefaults(&data) applyRuntimeDefaults(&data)
data.Misc.Version = CurrentConfigVersion data.Misc.Version = CurrentConfigVersion
if err = saveConfigData(resolvedPath, data); err != nil { if err = Validate(&data); err != nil {
return "", err return "", "", err
} }
return resolvedPath, nil if err = saveConfigData(resolvedPath, data); err != nil {
return "", "", err
}
return resolvedPath, backupPath, nil
} }
func (c *Config) SaveFile() error { func (c *Config) SaveFile() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.saveFileLocked()
}
func (c *Config) saveFileLocked() error {
return saveConfigData(c.Path, c.File) return saveConfigData(c.Path, c.File)
} }
// UpdateTokens applies fn to both the on-disk and session copies of the config
// under the config lock, then persists the result. Provider token refresh must
// go through this rather than mutating c.File/c.Session directly, because it
// runs concurrently from download workers.
func (c *Config) UpdateTokens(fn func(file, session *ConfigData)) error {
if fn == nil {
return nil
}
c.mu.Lock()
defer c.mu.Unlock()
fn(&c.File, &c.Session)
// An empty Path means there is no config file backing this Config (tests,
// and callers that loaded defaults only); the in-memory update still has to
// happen, but there is nothing to write.
if strings.TrimSpace(c.Path) == "" {
return nil
}
return c.saveFileLocked()
}
func DefaultConfigData() ConfigData { func DefaultConfigData() ConfigData {
home, _ := os.UserHomeDir() home, _ := os.UserHomeDir()
appDir := defaultAppDir() appDir := defaultAppDir()
@@ -246,9 +455,8 @@ func DefaultConfigData() ConfigData {
VerifySSL: true, VerifySSL: true,
}, },
Qobuz: QobuzConfig{ Qobuz: QobuzConfig{
Quality: 3, Quality: 3,
DownloadBooklets: true, UseAuthToken: false,
UseAuthToken: false,
}, },
Tidal: TidalConfig{ Tidal: TidalConfig{
Quality: 3, Quality: 3,
@@ -269,11 +477,6 @@ func DefaultConfigData() ConfigData {
Soundcloud: SoundcloudConfig{ Soundcloud: SoundcloudConfig{
Quality: 0, Quality: 0,
}, },
Youtube: YoutubeConfig{
Quality: 0,
DownloadVideos: false,
VideoDownloadsFolder: filepath.Join(downloadsFolder, "YouTubeVideos"),
},
Database: DatabaseConfig{ Database: DatabaseConfig{
DownloadsEnabled: true, DownloadsEnabled: true,
DownloadsPath: filepath.Join(appDir, "downloads.db"), DownloadsPath: filepath.Join(appDir, "downloads.db"),
@@ -317,20 +520,25 @@ func DefaultConfigData() ConfigData {
MaxSearchResults: 100, MaxSearchResults: 100,
}, },
Misc: MiscConfig{ Misc: MiscConfig{
Version: CurrentConfigVersion, Version: CurrentConfigVersion,
CheckForUpdates: true,
}, },
} }
applyVendorDefaults(&data)
return data return data
} }
func resolvePath(path string) (string, error) { func resolvePath(path string) (string, error) {
// 0o700: this directory holds the credential-bearing config and the
// download databases.
if path != "" { if path != "" {
return path, os.MkdirAll(filepath.Dir(path), 0o755) if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return "", err
}
return path, nil
} }
appDir := defaultAppDir() appDir := defaultAppDir()
if err := os.MkdirAll(appDir, 0o755); err != nil { if err := os.MkdirAll(appDir, 0o700); err != nil {
return "", err return "", err
} }
return filepath.Join(appDir, "config.toml"), nil return filepath.Join(appDir, "config.toml"), nil
@@ -345,17 +553,24 @@ func defaultAppDir() string {
} }
func saveConfigData(path string, data ConfigData) error { func saveConfigData(path string, data ConfigData) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { // 0o700/0o600: this file holds plaintext provider passwords, Deezer ARL
// cookies and OAuth access/refresh tokens, so it must not be world-readable.
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err return err
} }
b, err := toml.Marshal(data) b, err := toml.Marshal(data)
if err != nil { if err != nil {
return err return err
} }
return os.WriteFile(path, b, 0o644) if err := os.WriteFile(path, b, 0o600); err != nil {
return err
}
// Tighten an existing file that a previous version created as 0o644.
return os.Chmod(path, 0o600)
} }
func applyRuntimeDefaults(data *ConfigData) { func applyRuntimeDefaults(data *ConfigData) {
applyVendorDefaults(data)
home, _ := os.UserHomeDir() home, _ := os.UserHomeDir()
appDir := defaultAppDir() appDir := defaultAppDir()
if data.Downloads.Folder == "" { if data.Downloads.Folder == "" {
@@ -367,9 +582,13 @@ func applyRuntimeDefaults(data *ConfigData) {
if data.Database.FailedDownloadsPath == "" { if data.Database.FailedDownloadsPath == "" {
data.Database.FailedDownloadsPath = filepath.Join(appDir, "failed_downloads.db") data.Database.FailedDownloadsPath = filepath.Join(appDir, "failed_downloads.db")
} }
if data.Youtube.VideoDownloadsFolder == "" { }
data.Youtube.VideoDownloadsFolder = filepath.Join(data.Downloads.Folder, "YouTubeVideos")
} // CloneConfigData returns a deep copy of in. The slice fields must not be
// shared between Config.File and Config.Session, or writing one silently
// mutates the other.
func CloneConfigData(in ConfigData) ConfigData {
return cloneConfigData(in)
} }
func cloneConfigData(in ConfigData) ConfigData { func cloneConfigData(in ConfigData) ConfigData {
+330 -1
View File
@@ -1,10 +1,16 @@
package config package config
import ( import (
"encoding/hex"
"errors" "errors"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"sync"
"testing" "testing"
"github.com/pelletier/go-toml/v2"
) )
func TestDefaultConfigData(t *testing.T) { func TestDefaultConfigData(t *testing.T) {
@@ -64,7 +70,7 @@ func TestUpgradeOutdatedConfig(t *testing.T) {
t.Fatalf("saveConfigData() error = %v", err) t.Fatalf("saveConfigData() error = %v", err)
} }
resolved, err := UpgradeOutdated(path) resolved, backup, err := UpgradeOutdated(path)
if err != nil { if err != nil {
t.Fatalf("UpgradeOutdated() error = %v", err) t.Fatalf("UpgradeOutdated() error = %v", err)
} }
@@ -72,6 +78,28 @@ func TestUpgradeOutdatedConfig(t *testing.T) {
t.Fatalf("resolved path = %q, want %q", resolved, path) t.Fatalf("resolved path = %q, want %q", resolved, path)
} }
// The upgrade re-serializes the whole file and runs without prompting, so
// it must leave the original behind.
if backup == "" {
t.Fatal("UpgradeOutdated() returned no backup path")
}
backupRaw, err := os.ReadFile(backup)
if err != nil {
t.Fatalf("backup not written: %v", err)
}
var backedUp ConfigData
if err := toml.Unmarshal(backupRaw, &backedUp); err != nil {
t.Fatalf("backup is not valid toml: %v", err)
}
if backedUp.Misc.Version != "1.0.0" {
t.Errorf("backup version = %q, want the pre-upgrade 1.0.0", backedUp.Misc.Version)
}
if info, err := os.Stat(backup); err != nil {
t.Fatal(err)
} else if perm := info.Mode().Perm(); perm != 0o600 {
t.Errorf("backup permissions = %04o, want 0600 (it holds the same credentials)", perm)
}
cfg, err := Load(path) cfg, err := Load(path)
if err != nil { if err != nil {
t.Fatalf("Load() after upgrade error = %v", err) t.Fatalf("Load() after upgrade error = %v", err)
@@ -110,3 +138,304 @@ func TestSessionCloneDoesNotAliasSlices(t *testing.T) {
t.Fatalf("file qobuz secrets unexpectedly mutated") t.Fatalf("file qobuz secrets unexpectedly mutated")
} }
} }
func TestDefaultConfigPassesValidation(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
if err := Validate(&d); err != nil {
t.Fatalf("shipped defaults fail validation: %v", err)
}
}
// TestExampleConfigIsValid guards the file users are told to copy.
func TestExampleConfigIsValid(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "..", "config.toml.example"))
if err != nil {
t.Skipf("example config not found: %v", err)
}
data := DefaultConfigData()
if err := toml.Unmarshal(raw, &data); err != nil {
t.Fatalf("config.toml.example does not parse: %v", err)
}
applyRuntimeDefaults(&data)
if err := Validate(&data); err != nil {
t.Fatalf("config.toml.example fails validation: %v", err)
}
if data.Misc.Version != CurrentConfigVersion {
t.Errorf("config.toml.example version = %q, want %q", data.Misc.Version, CurrentConfigVersion)
}
}
func TestValidateRejectsBadValues(t *testing.T) {
tests := []struct {
name string
mutate func(*ConfigData)
want string
}{
{"qobuz quality 0", func(d *ConfigData) { d.Qobuz.Quality = 0 }, "qobuz.quality"},
{"qobuz quality 5", func(d *ConfigData) { d.Qobuz.Quality = 5 }, "qobuz.quality"},
{"beatport quality 0", func(d *ConfigData) { d.Beatport.Quality = 0 }, "beatport.quality"},
{"tidal quality 5", func(d *ConfigData) { d.Tidal.Quality = 5 }, "tidal.quality"},
{"negative quality", func(d *ConfigData) { d.Tidal.Quality = -1 }, "tidal.quality"},
{"max_connections 0", func(d *ConfigData) { d.Downloads.MaxConnections = 0 }, "max_connections"},
{"max_connections negative", func(d *ConfigData) { d.Downloads.MaxConnections = -5 }, "max_connections"},
{"requests_per_minute 0", func(d *ConfigData) { d.Downloads.RequestsPerMinute = 0 }, "requests_per_minute"},
{"truncate_to 0", func(d *ConfigData) { d.Filepaths.TruncateTo = 0 }, "truncate_to"},
{"bogus codec", func(d *ConfigData) {
d.Conversion.Enabled = true
d.Conversion.Codec = "bogus"
}, "conversion.codec"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
tc.mutate(&d)
err := Validate(&d)
if err == nil {
t.Fatalf("Validate() = nil, want an error mentioning %q", tc.want)
}
if !strings.Contains(err.Error(), tc.want) {
t.Fatalf("Validate() = %q, want it to mention %q", err, tc.want)
}
})
}
}
// TestValidateAcceptsDocumentedQualityValues pins the ranges documented in
// config.toml.example. An earlier draft capped deezer at 2 and soundcloud at 1,
// which would have made a config the example itself documents fail to load.
func TestValidateAcceptsDocumentedQualityValues(t *testing.T) {
for q := 0; q <= 4; q++ {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
d.Deezer.Quality = q
d.Soundcloud.Quality = q
d.Tidal.Quality = q
d.Yandex.Quality = q
if err := Validate(&d); err != nil {
t.Errorf("quality %d rejected: %v", q, err)
}
}
// Qobuz and Beatport have no tier 0.
for q := 1; q <= 4; q++ {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
d.Qobuz.Quality = q
d.Beatport.Quality = q
if err := Validate(&d); err != nil {
t.Errorf("qobuz/beatport quality %d rejected: %v", q, err)
}
}
}
func TestValidateAcceptsValidValues(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
// Negative requests_per_minute disables throttling, which is legitimate.
d.Downloads.RequestsPerMinute = -1
d.Soundcloud.Quality = 1
d.Deezer.Quality = 4
d.Qobuz.Quality = 4
if err := Validate(&d); err != nil {
t.Fatalf("Validate() = %v, want nil", err)
}
}
func TestValidateNormalizesCodec(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
d.Conversion.Enabled = true
d.Conversion.Codec = " ogg "
if err := Validate(&d); err != nil {
t.Fatalf("Validate() = %v", err)
}
if d.Conversion.Codec != "VORBIS" {
t.Fatalf("codec = %q, want VORBIS", d.Conversion.Codec)
}
}
func TestNormalizeCodec(t *testing.T) {
for in, want := range map[string]string{
"flac": "FLAC", "FLAC": "FLAC", " mp3 ": "MP3",
"ogg": "VORBIS", "vorbis": "VORBIS", "alac": "ALAC", "aac": "AAC",
} {
got, err := NormalizeCodec(in)
if err != nil {
t.Errorf("NormalizeCodec(%q) err = %v", in, err)
continue
}
if got != want {
t.Errorf("NormalizeCodec(%q) = %q, want %q", in, got, want)
}
}
if _, err := NormalizeCodec("wav"); err == nil {
t.Error("NormalizeCodec(wav) should fail")
}
}
func TestCloneConfigDataDoesNotShareSlices(t *testing.T) {
a := DefaultConfigData()
a.Qobuz.Secrets = []string{"s1", "s2"}
a.Metadata.Exclude = []string{"e1"}
b := CloneConfigData(a)
b.Qobuz.Secrets[0] = "mutated"
b.Metadata.Exclude[0] = "mutated"
if a.Qobuz.Secrets[0] != "s1" {
t.Error("Qobuz.Secrets is shared between copies")
}
if a.Metadata.Exclude[0] != "e1" {
t.Error("Metadata.Exclude is shared between copies")
}
}
func TestSaveFileUsesOwnerOnlyPermissions(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.toml")
cfg := &Config{Path: path, File: DefaultConfigData()}
if err := cfg.SaveFile(); err != nil {
t.Fatalf("SaveFile() = %v", err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
// The file holds plaintext passwords, ARL cookies and OAuth tokens.
if perm := info.Mode().Perm(); perm != 0o600 {
t.Fatalf("config permissions = %04o, want 0600", perm)
}
}
func TestSaveFileTightensExistingWorldReadableFile(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.toml")
if err := os.WriteFile(path, []byte("version = \"x\"\n"), 0o644); err != nil {
t.Fatal(err)
}
cfg := &Config{Path: path, File: DefaultConfigData()}
if err := cfg.SaveFile(); err != nil {
t.Fatal(err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
if perm := info.Mode().Perm(); perm != 0o600 {
t.Fatalf("permissions = %04o, want 0600 after rewrite", perm)
}
}
func TestUpdateTokensIsSafeWithoutAPath(t *testing.T) {
cfg := &Config{File: DefaultConfigData(), Session: DefaultConfigData()}
err := cfg.UpdateTokens(func(file, session *ConfigData) {
file.Deezer.RefreshToken = "rt"
session.Deezer.RefreshToken = "rt"
})
if err != nil {
t.Fatalf("UpdateTokens() = %v", err)
}
// The in-memory update must still happen even with nothing to write to.
if cfg.File.Deezer.RefreshToken != "rt" || cfg.Session.Deezer.RefreshToken != "rt" {
t.Fatal("UpdateTokens did not apply the mutation")
}
}
func TestUpdateTokensPersistsAndIsConcurrencySafe(t *testing.T) {
path := filepath.Join(t.TempDir(), "config.toml")
cfg := &Config{Path: path, File: DefaultConfigData(), Session: DefaultConfigData()}
var wg sync.WaitGroup
for i := 0; i < 16; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
token := fmt.Sprintf("token-%d", n)
if err := cfg.UpdateTokens(func(file, session *ConfigData) {
file.Tidal.AccessToken = token
session.Tidal.AccessToken = token
}); err != nil {
t.Errorf("UpdateTokens: %v", err)
}
}(i)
}
wg.Wait()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var round ConfigData
if err := toml.Unmarshal(raw, &round); err != nil {
t.Fatalf("concurrent writes produced an unparseable config: %v", err)
}
if round.Tidal.AccessToken == "" {
t.Error("no token persisted")
}
}
// TestVendorDefaultsAreApplied guards the Phase-3 move of vendor credentials
// into config: an empty value must resolve to the built-in constant, or the
// providers silently authenticate and sign with empty keys.
func TestVendorDefaultsAreApplied(t *testing.T) {
d := DefaultConfigData()
checks := map[string]string{
"qobuz.mobile_app_id": d.Qobuz.MobileAppID,
"qobuz.mobile_app_secret": d.Qobuz.MobileAppSecret,
"tidal.app_client_id": d.Tidal.AppClientID,
"tidal.app_client_secret": d.Tidal.AppClientSecret,
"deezer.api_key": d.Deezer.APIKey,
"deezer.gateway_key": d.Deezer.GatewayKey,
"deezer.blowfish_secret": d.Deezer.BlowfishSecret,
"yandex.sign_key": d.Yandex.SignKey,
"yandex.legacy_sign_salt": d.Yandex.LegacyMP3SignSalt,
"beatport.app_client_id": d.Beatport.AppClientID,
}
for name, got := range checks {
if got == "" {
t.Errorf("%s is empty in DefaultConfigData()", name)
}
}
// An explicitly empty config must be backfilled too, since that is what a
// user config written before these keys existed looks like.
var blank ConfigData
applyVendorDefaults(&blank)
if blank.Qobuz.MobileAppSecret != DefaultQobuzMobileAppSecret {
t.Error("applyVendorDefaults did not backfill qobuz.mobile_app_secret")
}
if blank.Deezer.BlowfishSecret != DefaultDeezerBlowfishSecret {
t.Error("applyVendorDefaults did not backfill deezer.blowfish_secret")
}
}
func TestVendorDefaultsAreNotOverriddenWhenSet(t *testing.T) {
d := ConfigData{}
d.Tidal.AppClientID = "mine"
d.Deezer.BlowfishSecret = "0123456789abcdef"
applyVendorDefaults(&d)
if d.Tidal.AppClientID != "mine" {
t.Errorf("user value overwritten: %q", d.Tidal.AppClientID)
}
if d.Deezer.BlowfishSecret != "0123456789abcdef" {
t.Errorf("user value overwritten: %q", d.Deezer.BlowfishSecret)
}
}
// TestDeezerBlowfishSecretLength pins the length the key derivation depends
// on: it XORs 16 bytes, so a different length would panic or silently derive a
// wrong key.
func TestDeezerBlowfishSecretLength(t *testing.T) {
if got := len(DefaultDeezerBlowfishSecret); got != 16 {
t.Fatalf("DefaultDeezerBlowfishSecret is %d bytes, want 16", got)
}
}
// TestQobuzMobileAppSecretIsHex pins the format the HKDF step requires.
func TestQobuzMobileAppSecretIsHex(t *testing.T) {
if _, err := hex.DecodeString(DefaultQobuzMobileAppSecret); err != nil {
t.Fatalf("DefaultQobuzMobileAppSecret is not hex: %v", err)
}
}
+198 -253
View File
@@ -7,6 +7,7 @@ import (
"crypto/cipher" "crypto/cipher"
"crypto/md5" "crypto/md5"
"encoding/hex" "encoding/hex"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -21,6 +22,7 @@ import (
"github.com/vbauerster/mpb/v8/decor" "github.com/vbauerster/mpb/v8/decor"
"golang.org/x/term" "golang.org/x/term"
"streamrip-go/internal/config"
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/verbose" "streamrip-go/internal/verbose"
@@ -32,6 +34,10 @@ type Downloader struct {
showProgress bool showProgress bool
progress *mpb.Progress progress *mpb.Progress
barStarted atomic.Int32 barStarted atomic.Int32
// DeezerBlowfishSecret overrides the built-in Deezer protocol secret.
// Set from config by app.New; empty means use the default.
DeezerBlowfishSecret string
} }
// downloadBufferSize sizes HTTP-to-disk copies. 1 MiB cuts read/write syscalls // downloadBufferSize sizes HTTP-to-disk copies. 1 MiB cuts read/write syscalls
@@ -43,10 +49,6 @@ func New() *Downloader {
return NewWithOptions(true, true, 0) return NewWithOptions(true, true, 0)
} }
func NewWithVerifySSL(verifySSL bool) *Downloader {
return NewWithOptions(verifySSL, true, 0)
}
func NewWithOptions(verifySSL bool, showProgress bool, maxConnsPerHost int) *Downloader { func NewWithOptions(verifySSL bool, showProgress bool, maxConnsPerHost int) *Downloader {
forceProgress := strings.EqualFold(os.Getenv("STREAMRIP_GO_FORCE_PROGRESS"), "1") || strings.EqualFold(os.Getenv("STREAMRIP_GO_FORCE_PROGRESS"), "true") forceProgress := strings.EqualFold(os.Getenv("STREAMRIP_GO_FORCE_PROGRESS"), "1") || strings.EqualFold(os.Getenv("STREAMRIP_GO_FORCE_PROGRESS"), "true")
interactive := showProgress && (forceProgress || (term.IsTerminal(int(os.Stderr.Fd())) && strings.ToLower(os.Getenv("TERM")) != "dumb")) interactive := showProgress && (forceProgress || (term.IsTerminal(int(os.Stderr.Fd())) && strings.ToLower(os.Getenv("TERM")) != "dumb"))
@@ -69,11 +71,24 @@ func (d *Downloader) FileVideo(ctx context.Context, sourceURL, outputPath string
return d.file(ctx, sourceURL, outputPath, true, true) return d.file(ctx, sourceURL, outputPath, true, true)
} }
func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputPath, trackID string) error { // streamEncrypted performs the shared scaffolding for an encrypted download:
// create the output directory, GET the URL, check the status, create the output
// file, attach a progress bar, clean up a partial file on failure, verify the
// byte count against Content-Length and fsync.
//
// decode does the format-specific work of turning the response body into plain
// audio on dst and returns the number of bytes written. FileDeezerEncrypted and
// FileYandexEncrypted were ~90% identical before this.
func (d *Downloader) streamEncrypted(
ctx context.Context,
sourceURL, outputPath string,
decode func(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error),
) error {
logDownloadStart(sourceURL, outputPath) logDownloadStart(sourceURL, outputPath)
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil { if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
return err return err
} }
req, err := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL, nil) req, err := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL, nil)
if err != nil { if err != nil {
return err return err
@@ -86,6 +101,7 @@ func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputP
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
return fmt.Errorf("download failed: status=%d", resp.StatusCode) return fmt.Errorf("download failed: status=%d", resp.StatusCode)
} }
out, err := os.Create(outputPath) out, err := os.Create(outputPath)
if err != nil { if err != nil {
return err return err
@@ -98,97 +114,72 @@ func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputP
} }
}() }()
var bar *mpb.Bar bar, finishBar := d.newDownloadBar(outputPath, resp.ContentLength)
if d.ProgressEnabled() { defer func() { finishBar(success) }()
d.barStarted.Store(1)
desc := shortenName(filepath.Base(outputPath), 54)
if resp.ContentLength > 0 {
bar = d.progress.AddBar(
resp.ContentLength,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
decor.Percentage(decor.WCSyncWidthR),
),
mpb.AppendDecorators(
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR),
decor.Name(" | ETA ", decor.WCSyncWidth),
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
} else {
bar = d.progress.AddSpinner(
0,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
),
mpb.AppendDecorators(
decor.CurrentKibiByte("% .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.Elapsed(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
defer bar.SetTotal(-1, true)
}
defer func() {
if !success && bar != nil {
bar.Abort(true)
}
}()
}
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(trackID)) written, err := decode(out, resp.Body, bar)
if err != nil { if err != nil {
return err return err
} }
buf := make([]byte, deezerBFChunkSize) if resp.ContentLength > 0 && written != resp.ContentLength {
dec := make([]byte, deezerBFChunkSize)
chunkIndex := 0
totalRead := int64(0)
for {
n, readErr := io.ReadFull(resp.Body, buf)
if readErr == io.EOF {
break
}
if readErr != nil && readErr != io.ErrUnexpectedEOF {
return readErr
}
chunk := buf[:n]
if chunkIndex%3 == 0 && n == deezerBFChunkSize {
mode := cipher.NewCBCDecrypter(block, deezerBFIV)
mode.CryptBlocks(dec[:n], chunk)
chunk = dec[:n]
}
if _, err = out.Write(chunk); err != nil {
return err
}
totalRead += int64(n)
if bar != nil {
bar.IncrBy(n)
}
chunkIndex++
if readErr == io.ErrUnexpectedEOF {
break
}
}
if resp.ContentLength > 0 && totalRead != resp.ContentLength {
return io.ErrUnexpectedEOF return io.ErrUnexpectedEOF
} }
if err = out.Sync(); err != nil { if err = out.Sync(); err != nil {
return err return err
} }
success = true success = true
return nil return nil
} }
func (d *Downloader) FileYandexEncrypted(ctx context.Context, sourceURL, outputPath, key string) error { // FileDeezerEncrypted downloads a Deezer stream and decrypts it. Deezer
logDownloadStart(sourceURL, outputPath) // encrypts every third 2048-byte chunk with Blowfish-CBC; a short final chunk
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil { // is left as-is.
func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputPath, trackID string) error {
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(d.blowfishSecret(), trackID))
if err != nil {
return err return err
} }
return d.streamEncrypted(ctx, sourceURL, outputPath, func(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error) {
buf := make([]byte, deezerBFChunkSize)
dec := make([]byte, deezerBFChunkSize)
chunkIndex := 0
total := int64(0)
for {
n, readErr := io.ReadFull(src, buf)
if errors.Is(readErr, io.EOF) {
return total, nil
}
if readErr != nil && !errors.Is(readErr, io.ErrUnexpectedEOF) {
return total, readErr
}
chunk := buf[:n]
if chunkIndex%3 == 0 && n == deezerBFChunkSize {
mode := cipher.NewCBCDecrypter(block, deezerBFIV)
mode.CryptBlocks(dec[:n], chunk)
chunk = dec[:n]
}
if _, writeErr := dst.Write(chunk); writeErr != nil {
return total, writeErr
}
total += int64(n)
if bar != nil {
bar.IncrBy(n)
}
chunkIndex++
if errors.Is(readErr, io.ErrUnexpectedEOF) {
return total, nil
}
}
})
}
// FileYandexEncrypted downloads a Yandex stream and decrypts it. Yandex uses
// AES-CTR over the whole body with a zero IV.
func (d *Downloader) FileYandexEncrypted(ctx context.Context, sourceURL, outputPath, key string) error {
keyBytes, err := hex.DecodeString(strings.TrimSpace(key)) keyBytes, err := hex.DecodeString(strings.TrimSpace(key))
if err != nil { if err != nil {
return fmt.Errorf("invalid yandex key: %w", err) return fmt.Errorf("invalid yandex key: %w", err)
@@ -200,102 +191,11 @@ func (d *Downloader) FileYandexEncrypted(ctx context.Context, sourceURL, outputP
if err != nil { if err != nil {
return err return err
} }
req, err := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL, nil)
if err != nil {
return err
}
resp, err := d.http.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("download failed: status=%d", resp.StatusCode)
}
out, err := os.Create(outputPath)
if err != nil {
return err
}
success := false
defer func() {
_ = out.Close()
if !success {
_ = os.Remove(outputPath)
}
}()
var bar *mpb.Bar return d.streamEncrypted(ctx, sourceURL, outputPath, func(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error) {
if d.ProgressEnabled() { stream := cipher.NewCTR(block, make([]byte, aes.BlockSize))
d.barStarted.Store(1) return copyWithProgress(dst, &cipher.StreamReader{S: stream, R: src}, bar)
desc := shortenName(filepath.Base(outputPath), 54) })
if resp.ContentLength > 0 {
bar = d.progress.AddBar(
resp.ContentLength,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
decor.Percentage(decor.WCSyncWidthR),
),
mpb.AppendDecorators(
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR),
decor.Name(" | ETA ", decor.WCSyncWidth),
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
} else {
bar = d.progress.AddSpinner(
0,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
),
mpb.AppendDecorators(
decor.CurrentKibiByte("% .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.Elapsed(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
defer bar.SetTotal(-1, true)
}
defer func() {
if !success && bar != nil {
bar.Abort(true)
}
}()
}
stream := cipher.NewCTR(block, make([]byte, aes.BlockSize))
reader := &cipher.StreamReader{S: stream, R: resp.Body}
buf := make([]byte, downloadBufferSize)
totalWritten := int64(0)
for {
n, readErr := reader.Read(buf)
if n > 0 {
if _, writeErr := out.Write(buf[:n]); writeErr != nil {
return writeErr
}
totalWritten += int64(n)
if bar != nil {
bar.IncrBy(n)
}
}
if readErr != nil {
if readErr == io.EOF {
break
}
return readErr
}
}
if resp.ContentLength > 0 && totalWritten != resp.ContentLength {
return io.ErrUnexpectedEOF
}
if err = out.Sync(); err != nil {
return err
}
success = true
return nil
} }
func (d *Downloader) file(ctx context.Context, sourceURL, outputPath string, allowProgress bool, includeVideo bool) error { func (d *Downloader) file(ctx context.Context, sourceURL, outputPath string, allowProgress bool, includeVideo bool) error {
@@ -338,81 +238,22 @@ func (d *Downloader) file(ctx context.Context, sourceURL, outputPath string, all
} }
}() }()
if d.ProgressEnabled() && allowProgress { var bar *mpb.Bar
d.barStarted.Store(1) finishBar := func(bool) {}
desc := shortenName(filepath.Base(outputPath), 54) if allowProgress {
var bar *mpb.Bar bar, finishBar = d.newDownloadBar(outputPath, resp.ContentLength)
if resp.ContentLength > 0 { }
bar = d.progress.AddBar( defer func() { finishBar(success) }()
resp.ContentLength,
mpb.PrependDecorators( written, err := copyWithProgress(out, reader, bar)
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}), if err != nil {
decor.Percentage(decor.WCSyncWidthR), return err
), }
mpb.AppendDecorators( if resp.ContentLength > 0 && written != resp.ContentLength {
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR), return io.ErrUnexpectedEOF
decor.Name(" | ", decor.WCSyncWidth), }
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR), if err = out.Sync(); err != nil {
decor.Name(" | ETA ", decor.WCSyncWidth), return err
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
} else {
bar = d.progress.AddSpinner(
0,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
),
mpb.AppendDecorators(
decor.CurrentKibiByte("% .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.Elapsed(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
defer bar.SetTotal(-1, true)
}
defer func() {
if !success && bar != nil {
bar.Abort(true)
}
}()
buf := make([]byte, downloadBufferSize)
totalWritten := int64(0)
for {
n, readErr := reader.Read(buf)
if n > 0 {
if _, writeErr := out.Write(buf[:n]); writeErr != nil {
return writeErr
}
totalWritten += int64(n)
bar.IncrBy(n)
}
if readErr != nil {
if readErr == io.EOF {
break
}
return readErr
}
}
if resp.ContentLength > 0 && totalWritten != resp.ContentLength {
return io.ErrUnexpectedEOF
}
if err = out.Sync(); err != nil {
return err
}
} else {
written, copyErr := io.CopyBuffer(out, reader, make([]byte, downloadBufferSize))
if copyErr != nil {
return copyErr
}
if resp.ContentLength > 0 && written != resp.ContentLength {
return io.ErrUnexpectedEOF
}
if err = out.Sync(); err != nil {
return err
}
} }
success = true success = true
@@ -438,6 +279,92 @@ func (d *Downloader) Logf(format string, args ...any) {
fmt.Print(msg) fmt.Print(msg)
} }
// copyWithProgress copies src to dst, reporting progress to bar when it is
// non-nil. It replaces three hand-rolled read/write loops that differed only
// in whether they touched a progress bar.
func copyWithProgress(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error) {
if bar == nil {
return io.CopyBuffer(dst, src, make([]byte, downloadBufferSize))
}
buf := make([]byte, downloadBufferSize)
total := int64(0)
for {
n, readErr := src.Read(buf)
if n > 0 {
if _, writeErr := dst.Write(buf[:n]); writeErr != nil {
return total, writeErr
}
total += int64(n)
bar.IncrBy(n)
}
if readErr != nil {
if errors.Is(readErr, io.EOF) {
return total, nil
}
return total, readErr
}
}
}
// newDownloadBar creates the progress bar for a download: a determinate bar
// when the content length is known, a spinner when it is not. It returns a nil
// bar when progress display is off, so callers can use the result
// unconditionally, plus a finish func to defer.
//
// This block was copy-pasted into all three download paths; the decorator
// stacks had already started to drift apart.
func (d *Downloader) newDownloadBar(outputPath string, contentLength int64) (*mpb.Bar, func(success bool)) {
if !d.ProgressEnabled() {
return nil, func(bool) {}
}
d.barStarted.Store(1)
desc := shortenName(filepath.Base(outputPath), 54)
if contentLength > 0 {
bar := d.progress.AddBar(
contentLength,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
decor.Percentage(decor.WCSyncWidthR),
),
mpb.AppendDecorators(
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR),
decor.Name(" | ETA ", decor.WCSyncWidth),
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
return bar, func(success bool) {
if !success {
bar.Abort(true)
}
}
}
bar := d.progress.AddSpinner(
0,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
),
mpb.AppendDecorators(
decor.CurrentKibiByte("% .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.Elapsed(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
// Order matches the original defers, which ran LIFO: abort first, then
// finalise the spinner total.
return bar, func(success bool) {
if !success {
bar.Abort(true)
}
bar.SetTotal(-1, true)
}
}
// logDownloadStart emits the source URL and destination filename when the // logDownloadStart emits the source URL and destination filename when the
// user passed -v or higher. The transport-level logger covers the same // user passed -v or higher. The transport-level logger covers the same
// requests at -vv, but this line gives a friendlier per-track summary. // requests at -vv, but this line gives a friendlier per-track summary.
@@ -835,10 +762,28 @@ const deezerBFChunkSize = 2048
var deezerBFIV = []byte{0, 1, 2, 3, 4, 5, 6, 7} var deezerBFIV = []byte{0, 1, 2, 3, 4, 5, 6, 7}
func deriveDeezerBlowfishKey(trackID string) []byte { // defaultDeezerBlowfishSecret aliases the single definition in internal/config
// so the literal is not duplicated here.
const defaultDeezerBlowfishSecret = config.DefaultDeezerBlowfishSecret
// blowfishSecret returns the configured Deezer blowfish secret.
//
// The key derivation XORs 16 bytes of it, so a secret of any other length
// would panic; fall back to the built-in value rather than crash on a
// mistyped config entry.
func (d *Downloader) blowfishSecret() string {
if len(d.DeezerBlowfishSecret) == 16 {
return d.DeezerBlowfishSecret
}
if d.DeezerBlowfishSecret != "" {
verbose.Printf(verbose.V, "ignoring deezer.blowfish_secret: got %d bytes, want 16\n", len(d.DeezerBlowfishSecret))
}
return defaultDeezerBlowfishSecret
}
func deriveDeezerBlowfishKey(secret, trackID string) []byte {
sum := md5.Sum([]byte(trackID)) sum := md5.Sum([]byte(trackID))
md5Hex := fmt.Sprintf("%x", sum) md5Hex := fmt.Sprintf("%x", sum)
secret := "g4el58wc0zvf9na1"
key := make([]byte, 16) key := make([]byte, 16)
for i := 0; i < 16; i++ { for i := 0; i < 16; i++ {
key[i] = md5Hex[i] ^ md5Hex[i+16] ^ secret[i] key[i] = md5Hex[i] ^ md5Hex[i+16] ^ secret[i]
+146 -3
View File
@@ -1,17 +1,19 @@
package download package download
import ( import (
"bytes"
"context" "context"
"crypto/aes" "crypto/aes"
"crypto/cipher" "crypto/cipher"
"errors"
"encoding/hex" "encoding/hex"
"errors"
"io" "io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -70,7 +72,7 @@ func TestManifestDetection(t *testing.T) {
func TestDeezerBlowfishKeyDerivation(t *testing.T) { func TestDeezerBlowfishKeyDerivation(t *testing.T) {
trackID := "3135556" trackID := "3135556"
key := deriveDeezerBlowfishKey(trackID) key := deriveDeezerBlowfishKey(defaultDeezerBlowfishSecret, trackID)
if len(key) != 16 { if len(key) != 16 {
t.Fatalf("blowfish key len = %d, want 16", len(key)) t.Fatalf("blowfish key len = %d, want 16", len(key))
} }
@@ -85,7 +87,7 @@ func TestFileDeezerEncrypted(t *testing.T) {
enc := make([]byte, len(plain)) enc := make([]byte, len(plain))
copy(enc, plain) copy(enc, plain)
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(trackID)) block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(defaultDeezerBlowfishSecret, trackID))
if err != nil { if err != nil {
t.Fatalf("cipher error: %v", err) t.Fatalf("cipher error: %v", err)
} }
@@ -367,3 +369,144 @@ func containsArgPair(args []string, key, value string) bool {
} }
return false return false
} }
// newProgressDownloader builds a Downloader with the progress bars actually
// enabled. The bar code paths are otherwise never exercised, because tests do
// not run on a terminal; STREAMRIP_GO_FORCE_PROGRESS is the documented escape
// hatch. Output goes to stderr, which `go test` captures.
func newProgressDownloader(t *testing.T) *Downloader {
t.Helper()
t.Setenv("STREAMRIP_GO_FORCE_PROGRESS", "1")
d := NewWithOptions(true, true, 0)
if !d.ProgressEnabled() {
t.Fatal("expected progress to be enabled")
}
t.Cleanup(d.Close)
return d
}
// TestFileWithProgressBar covers the determinate-bar path (Content-Length
// known) through the shared newDownloadBar/copyWithProgress helpers.
func TestFileWithProgressBar(t *testing.T) {
payload := bytes.Repeat([]byte("abcd"), 4096)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", strconv.Itoa(len(payload)))
_, _ = w.Write(payload)
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "sub", "a.flac")
if err := d.File(context.Background(), ts.URL, out); err != nil {
t.Fatalf("File() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("content mismatch: got %d bytes, want %d", len(got), len(payload))
}
}
// TestFileWithSpinnerUnknownLength covers the spinner path, taken when the
// server does not send a Content-Length.
func TestFileWithSpinnerUnknownLength(t *testing.T) {
payload := bytes.Repeat([]byte("xyz"), 3000)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Transfer-Encoding", "chunked")
_, _ = w.Write(payload)
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "a.flac")
if err := d.File(context.Background(), ts.URL, out); err != nil {
t.Fatalf("File() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("content mismatch: got %d bytes, want %d", len(got), len(payload))
}
}
// TestFileDeezerEncryptedWithProgressBar runs the Deezer decryption through
// the progress-enabled path, so the shared streamEncrypted scaffolding is
// covered with a bar attached as well as without.
func TestFileDeezerEncryptedWithProgressBar(t *testing.T) {
plain := make([]byte, deezerBFChunkSize*4+100)
for i := range plain {
plain[i] = byte((i * 7) % 253)
}
trackID := "123456"
enc := encryptDeezerPayload(t, plain, trackID)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", strconv.Itoa(len(enc)))
_, _ = w.Write(enc)
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "a.mp3")
if err := d.FileDeezerEncrypted(context.Background(), ts.URL, out, trackID); err != nil {
t.Fatalf("FileDeezerEncrypted() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, plain) {
t.Fatal("decrypted output does not match the original plaintext")
}
}
// encryptDeezerPayload applies Deezer's scheme: Blowfish-CBC on every third
// full 2048-byte chunk, everything else left in the clear.
func encryptDeezerPayload(t *testing.T, plain []byte, trackID string) []byte {
t.Helper()
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(defaultDeezerBlowfishSecret, trackID))
if err != nil {
t.Fatal(err)
}
enc := make([]byte, len(plain))
copy(enc, plain)
for i, off := 0, 0; off < len(plain); i, off = i+1, off+deezerBFChunkSize {
end := off + deezerBFChunkSize
if end > len(plain) {
break
}
if i%3 != 0 {
continue
}
mode := cipher.NewCBCEncrypter(block, deezerBFIV)
mode.CryptBlocks(enc[off:end], plain[off:end])
}
return enc
}
// TestProgressBarFailurePathRemovesPartialFile checks that a mid-download
// failure still runs the bar cleanup and removes the partial output, which is
// the behaviour the finishBar(success) defer replaced.
func TestProgressBarFailurePathRemovesPartialFile(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", "100000")
_, _ = w.Write(bytes.Repeat([]byte("a"), 10))
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "a.flac")
if err := d.File(context.Background(), ts.URL, out); err == nil {
t.Fatal("expected a truncated-response error")
}
if _, err := os.Stat(out); !os.IsNotExist(err) {
t.Fatalf("partial file not removed, stat err = %v", err)
}
}
+29
View File
@@ -129,3 +129,32 @@ func TitleCase(s string) string {
} }
return string(r) return string(r)
} }
// DisplayArtistNames joins artist names the way the providers present them:
// "A", "A & B", or "A, B & C". Previously duplicated byte-for-byte in
// internal/app, provider/yandex and provider/beatport.
func DisplayArtistNames(names []string) string {
switch len(names) {
case 0:
return ""
case 1:
return names[0]
case 2:
return names[0] + " & " + names[1]
default:
return strings.Join(names[:len(names)-1], ", ") + " & " + names[len(names)-1]
}
}
// FirstPositiveInt returns the first value greater than zero, or 0 if there is
// none. Useful for picking among optional metadata fields where 0 means
// "absent". Previously duplicated in provider/yandex, provider/beatport and
// cmd/rip.
func FirstPositiveInt(values ...int) int {
for _, v := range values {
if v > 0 {
return v
}
}
return 0
}
+203
View File
@@ -0,0 +1,203 @@
package jsonutil
import "testing"
func TestStringFromAny(t *testing.T) {
tests := []struct {
name string
in any
want string
}{
{"string", "hi", "hi"},
{"int", 7, "7"},
{"int64", int64(8), "8"},
{"float whole", float64(9), "9"},
{"float fractional", 44.1, "44.1"},
{"float no trailing zeroes", float64(100), "100"},
{"nil", nil, ""},
{"bool unsupported", true, ""},
{"map unsupported", map[string]any{}, ""},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := StringFromAny(tc.in); got != tc.want {
t.Fatalf("StringFromAny(%#v) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
func TestIntFromAny(t *testing.T) {
tests := []struct {
in any
want int
}{
{5, 5},
{int32(6), 6},
{int64(7), 7},
{8.9, 8},
{"10", 10},
{" 11 ", 11},
{"not a number", 0},
{nil, 0},
{true, 0},
}
for _, tc := range tests {
if got := IntFromAny(tc.in); got != tc.want {
t.Errorf("IntFromAny(%#v) = %d, want %d", tc.in, got, tc.want)
}
}
}
func TestFloatFromAny(t *testing.T) {
tests := []struct {
in any
want float64
}{
{1.5, 1.5},
{2, 2},
{int64(3), 3},
{"4.5", 0}, // strings are deliberately not parsed here
{nil, 0},
}
for _, tc := range tests {
if got := FloatFromAny(tc.in); got != tc.want {
t.Errorf("FloatFromAny(%#v) = %v, want %v", tc.in, got, tc.want)
}
}
}
func TestBoolFromAny(t *testing.T) {
tests := []struct {
in any
want bool
}{
{true, true},
{false, false},
{"true", true},
{"TRUE", true},
{" yes ", true},
{"1", true},
{"0", false},
{"no", false},
{1, true},
{0, false},
{int64(2), true},
{0.0, false},
{1.5, true},
{nil, false},
}
for _, tc := range tests {
if got := BoolFromAny(tc.in); got != tc.want {
t.Errorf("BoolFromAny(%#v) = %v, want %v", tc.in, got, tc.want)
}
}
}
func TestFirstNonEmpty(t *testing.T) {
if got := FirstNonEmpty("", " ", "found", "later"); got != "found" {
t.Errorf("got %q", got)
}
if got := FirstNonEmpty(" trimmed "); got != "trimmed" {
t.Errorf("got %q, want trimmed", got)
}
if got := FirstNonEmpty("", " "); got != "" {
t.Errorf("got %q, want empty", got)
}
if got := FirstNonEmpty(); got != "" {
t.Errorf("got %q, want empty", got)
}
}
func TestNestedMap(t *testing.T) {
m := map[string]any{"album": map[string]any{"title": "X"}, "n": 1}
if got := NestedMap(m, "album")["title"]; got != "X" {
t.Errorf("got %v", got)
}
if got := NestedMap(m, "missing"); len(got) != 0 {
t.Errorf("missing key should give empty map, got %v", got)
}
if got := NestedMap(m, "n"); len(got) != 0 {
t.Errorf("non-map value should give empty map, got %v", got)
}
}
func TestNestedAnyAndString(t *testing.T) {
m := map[string]any{
"album": map[string]any{
"artist": map[string]any{"name": "Nina Simone", "id": float64(42)},
},
}
if got := NestedString(m, "album", "artist", "name"); got != "Nina Simone" {
t.Errorf("got %q", got)
}
if got := NestedString(m, "album", "artist", "id"); got != "42" {
t.Errorf("numeric id = %q, want 42", got)
}
if got := NestedString(m, "album", "nope", "name"); got != "" {
t.Errorf("missing path = %q, want empty", got)
}
if got := NestedAny(m, "album", "artist", "name"); got != "Nina Simone" {
t.Errorf("got %v", got)
}
if got := NestedAny(m); got == nil {
t.Error("no keys should return the map itself")
}
// Traversing through a non-map must not panic.
if got := NestedAny(m, "album", "artist", "name", "deeper"); got != nil {
t.Errorf("over-deep path = %v, want nil", got)
}
}
func TestTitleCase(t *testing.T) {
tests := []struct{ in, want string }{
{"qobuz", "Qobuz"},
{"Tidal", "Tidal"},
{"", ""},
{"1st", "1st"},
{"ä", "ä"}, // only ASCII is uppercased
}
for _, tc := range tests {
if got := TitleCase(tc.in); got != tc.want {
t.Errorf("TitleCase(%q) = %q, want %q", tc.in, got, tc.want)
}
}
}
func TestDisplayArtistNames(t *testing.T) {
tests := []struct {
in []string
want string
}{
{nil, ""},
{[]string{}, ""},
{[]string{"A"}, "A"},
{[]string{"A", "B"}, "A & B"},
{[]string{"A", "B", "C"}, "A, B & C"},
{[]string{"A", "B", "C", "D"}, "A, B, C & D"},
}
for _, tc := range tests {
if got := DisplayArtistNames(tc.in); got != tc.want {
t.Errorf("DisplayArtistNames(%v) = %q, want %q", tc.in, got, tc.want)
}
}
}
func TestFirstPositiveInt(t *testing.T) {
tests := []struct {
in []int
want int
}{
{nil, 0},
{[]int{0, 0}, 0},
{[]int{0, 5, 7}, 5},
{[]int{-3, 0, 2}, 2},
{[]int{9}, 9},
{[]int{-1}, 0},
}
for _, tc := range tests {
if got := FirstPositiveInt(tc.in...); got != tc.want {
t.Errorf("FirstPositiveInt(%v) = %d, want %d", tc.in, got, tc.want)
}
}
}
+57 -2
View File
@@ -6,6 +6,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"unicode" "unicode"
"unicode/utf8"
) )
type Config struct { type Config struct {
@@ -14,7 +15,7 @@ type Config struct {
} }
var tokenRe = regexp.MustCompile(`\{([a-z_]+)(?::0?(\d+))?\}`) var tokenRe = regexp.MustCompile(`\{([a-z_]+)(?::0?(\d+))?\}`)
var invalidPathRe = regexp.MustCompile(`[<>:"/\\|?*\x00-\x1F]`) var invalidPathRe = regexp.MustCompile(`[<>:"/\\|?*\x00-\x1F\x7F]`)
func FormatTemplate(template string, values map[string]string) string { func FormatTemplate(template string, values map[string]string) string {
return tokenRe.ReplaceAllStringFunc(template, func(m string) string { return tokenRe.ReplaceAllStringFunc(template, func(m string) string {
@@ -35,6 +36,28 @@ func FormatTemplate(template string, values map[string]string) string {
}) })
} }
// maxNameBytes is the per-component limit on ext4, APFS and most other modern
// filesystems. TruncateTo counts runes, so a 120-rune CJK title is ~360 bytes
// and would fail with ENAMETOOLONG without this second, byte-based cap.
const maxNameBytes = 255
// windowsReserved are device names that cannot be used as a path component on
// Windows, with or without an extension.
var windowsReserved = map[string]struct{}{
"CON": {}, "PRN": {}, "AUX": {}, "NUL": {},
"COM1": {}, "COM2": {}, "COM3": {}, "COM4": {}, "COM5": {},
"COM6": {}, "COM7": {}, "COM8": {}, "COM9": {},
"LPT1": {}, "LPT2": {}, "LPT3": {}, "LPT4": {}, "LPT5": {},
"LPT6": {}, "LPT7": {}, "LPT8": {}, "LPT9": {},
}
// CleanName turns a remote title into a single safe path component.
//
// Everything it receives is attacker-controlled: album, playlist and track
// titles come straight from provider APIs, and on SoundCloud and Tidal they are
// user-generated. The result is joined onto the download root by the caller, so
// it must never be "." or ".." (which would escape or collapse into the parent
// directory) and must never contain a separator.
func CleanName(in string, cfg Config) string { func CleanName(in string, cfg Config) string {
s := strings.TrimSpace(in) s := strings.TrimSpace(in)
s = invalidPathRe.ReplaceAllString(s, "_") s = invalidPathRe.ReplaceAllString(s, "_")
@@ -53,13 +76,45 @@ func CleanName(in string, cfg Config) string {
s = string(runes[:cfg.TruncateTo]) s = string(runes[:cfg.TruncateTo])
} }
} }
s = truncateBytes(s, maxNameBytes)
// Trailing dots and spaces are stripped by Windows, which turns "foo." into
// "foo" and makes ".." survive as a traversal component.
s = strings.TrimRight(s, ". ")
s = strings.TrimSpace(s) s = strings.TrimSpace(s)
if s == "" {
if s == "" || isAllDots(s) {
return "Unknown" return "Unknown"
} }
if _, bad := windowsReserved[strings.ToUpper(s)]; bad {
return s + "_"
}
return s return s
} }
// truncateBytes shortens s to at most max bytes without splitting a rune.
func truncateBytes(s string, max int) string {
if len(s) <= max {
return s
}
cut := s[:max]
for len(cut) > 0 && !utf8.ValidString(cut) {
cut = cut[:len(cut)-1]
}
return cut
}
// isAllDots reports whether s consists only of "." characters, i.e. "." or
// ".." or any longer run that some filesystems also treat as relative.
func isAllDots(s string) bool {
for _, ch := range s {
if ch != '.' {
return false
}
}
return len(s) > 0
}
func YearFromDate(date string) string { func YearFromDate(date string) string {
if len(date) >= 4 { if len(date) >= 4 {
prefix := date[:4] prefix := date[:4]
+127 -4
View File
@@ -1,6 +1,10 @@
package naming package naming
import "testing" import (
"path/filepath"
"strings"
"testing"
)
func TestFormatTemplate(t *testing.T) { func TestFormatTemplate(t *testing.T) {
got := FormatTemplate("{tracknumber:02}. {artist} - {title}{explicit}", map[string]string{ got := FormatTemplate("{tracknumber:02}. {artist} - {title}{explicit}", map[string]string{
@@ -15,8 +19,127 @@ func TestFormatTemplate(t *testing.T) {
} }
func TestCleanName(t *testing.T) { func TestCleanName(t *testing.T) {
got := CleanName(" Dreams/Live ", Config{RestrictCharacters: false, TruncateTo: 120}) cfg := Config{RestrictCharacters: false, TruncateTo: 120}
if got != "Dreams_Live" { tests := []struct {
t.Fatalf("got %q", got) name string
in string
want string
}{
{"trims and replaces separator", " Dreams/Live ", "Dreams_Live"},
{"backslash", `AC\DC`, "AC_DC"},
{"colon", "Album: Deluxe", "Album_ Deluxe"},
{"control chars", "Song\x01\x02", "Song__"},
{"del char", "Song\x7f", "Song_"},
{"empty", "", "Unknown"},
{"whitespace only", " ", "Unknown"},
// Path traversal: these must never be returned verbatim, or the
// caller's filepath.Join escapes the download root.
{"dotdot", "..", "Unknown"},
{"single dot", ".", "Unknown"},
{"many dots", "....", "Unknown"},
{"dotdot padded", " .. ", "Unknown"},
{"trailing dot", "Album.", "Album"},
{"trailing dots", "Album...", "Album"},
{"leading dot kept", ".hidden", ".hidden"},
// Windows device names must not be used as a bare component.
{"reserved CON", "CON", "CON_"},
{"reserved nul lowercase", "nul", "nul_"},
{"reserved COM1", "COM1", "COM1_"},
{"not reserved", "CONCERT", "CONCERT"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := CleanName(tc.in, cfg)
if got != tc.want {
t.Fatalf("CleanName(%q) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
// TestCleanNameNeverEscapesRoot is the property that actually matters: whatever
// a provider returns, joining the result onto a root must stay under that root.
func TestCleanNameNeverEscapesRoot(t *testing.T) {
root := "/music"
hostile := []string{
"..", ".", "../..", "../../etc", "..\\..", "/etc/passwd",
"....//....", " .. ", "", "...", "C:\\Windows",
"\x00evil", "a/../../b",
}
for _, in := range hostile {
for _, restrict := range []bool{false, true} {
cfg := Config{RestrictCharacters: restrict, TruncateTo: 120}
name := CleanName(in, cfg)
if strings.ContainsAny(name, `/\`) {
t.Errorf("CleanName(%q) = %q contains a path separator", in, name)
}
if name == "." || name == ".." {
t.Errorf("CleanName(%q) = %q is a relative component", in, name)
}
joined := filepath.Join(root, name)
rel, err := filepath.Rel(root, joined)
if err != nil {
t.Errorf("CleanName(%q) = %q: Rel failed: %v", in, name, err)
continue
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
t.Errorf("CleanName(%q) = %q escapes root: joined=%q rel=%q", in, name, joined, rel)
}
}
}
}
func TestCleanNameTruncation(t *testing.T) {
// TruncateTo counts runes.
got := CleanName(strings.Repeat("a", 200), Config{TruncateTo: 10})
if len(got) != 10 {
t.Fatalf("rune truncation: got %d chars (%q)", len(got), got)
}
// A long multi-byte title must still fit the 255-byte component limit,
// and must remain valid UTF-8.
long := strings.Repeat("日", 200) // 3 bytes each = 600 bytes
got = CleanName(long, Config{TruncateTo: 200})
if len(got) > maxNameBytes {
t.Fatalf("byte truncation: got %d bytes, want <= %d", len(got), maxNameBytes)
}
if !utf8Valid(got) {
t.Fatalf("truncation split a rune: %q", got)
}
}
func utf8Valid(s string) bool {
for _, r := range s {
if r == '\uFFFD' {
return false
}
}
return true
}
func TestCleanNameRestrictCharacters(t *testing.T) {
got := CleanName("Björk – Jóga", Config{RestrictCharacters: true, TruncateTo: 120})
if strings.ContainsAny(got, "ö–ó") {
t.Fatalf("non-ascii survived restrict_characters: %q", got)
}
}
func TestYearFromDate(t *testing.T) {
tests := []struct{ in, want string }{
{"1977-02-04", "1977"},
{"1977", "1977"},
{"", "Unknown"},
{"77", "Unknown"},
{"abcd-01-01", "Unknown"},
{"19x7-01-01", "Unknown"},
}
for _, tc := range tests {
if got := YearFromDate(tc.in); got != tc.want {
t.Errorf("YearFromDate(%q) = %q, want %q", tc.in, got, tc.want)
}
} }
} }
+148
View File
@@ -0,0 +1,148 @@
package netutil
import (
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
)
func TestRedactURLHidesCredentials(t *testing.T) {
tests := []struct {
name string
raw string
contains []string
absent []string
}{
{
name: "user auth token",
raw: "https://api.example.com/track?user_auth_token=sekrit&track_id=42",
contains: []string{"user_auth_token=REDACTED", "track_id=42"},
absent: []string{"sekrit"},
},
{
name: "request signature",
raw: "https://api.example.com/x?request_sig=abc123&app_id=9",
contains: []string{"request_sig=REDACTED", "app_id=9"},
absent: []string{"abc123"},
},
{
name: "several at once",
raw: "https://api.example.com/x?password=p&access_token=a&refresh_token=r&id=1",
contains: []string{"password=REDACTED", "access_token=REDACTED", "refresh_token=REDACTED", "id=1"},
absent: []string{"password=p", "access_token=a", "refresh_token=r"},
},
{
name: "nothing sensitive is left untouched",
raw: "https://api.example.com/track/42?limit=10",
contains: []string{"limit=10", "/track/42"},
},
{
name: "no query string",
raw: "https://api.example.com/track/42",
contains: []string{"https://api.example.com/track/42"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
u, err := url.Parse(tc.raw)
if err != nil {
t.Fatal(err)
}
got := redactURL(u)
for _, want := range tc.contains {
if !strings.Contains(got, want) {
t.Errorf("redactURL(%q) = %q, missing %q", tc.raw, got, want)
}
}
for _, bad := range tc.absent {
if strings.Contains(got, bad) {
t.Errorf("redactURL(%q) = %q, leaked %q", tc.raw, got, bad)
}
}
})
}
}
func TestRedactURLNil(t *testing.T) {
if got := redactURL(nil); got != "" {
t.Errorf("redactURL(nil) = %q, want empty", got)
}
}
func TestIsSensitiveParam(t *testing.T) {
sensitive := []string{
"user_auth_token", "api_token", "access_token", "refresh_token",
"request_sig", "signature", "password", "secret", "token", "code", "auth", "key",
}
for _, name := range sensitive {
if !isSensitiveParam(name) {
t.Errorf("isSensitiveParam(%q) = false, want true", name)
}
}
for _, name := range []string{"track_id", "limit", "offset", "app_id", "quality"} {
if isSensitiveParam(name) {
t.Errorf("isSensitiveParam(%q) = true, want false", name)
}
}
}
func TestNewHTTPClientHonoursVerifySSL(t *testing.T) {
insecure := NewHTTPClient(5*time.Second, false, 4)
tr, ok := insecure.Transport.(*loggingTransport)
if !ok {
t.Fatalf("transport is %T, want *loggingTransport", insecure.Transport)
}
base, ok := tr.base.(*http.Transport)
if !ok {
t.Fatalf("base transport is %T", tr.base)
}
if !base.TLSClientConfig.InsecureSkipVerify {
t.Error("verifySSL=false should set InsecureSkipVerify")
}
secure := NewHTTPClient(5*time.Second, true, 4)
base = secure.Transport.(*loggingTransport).base.(*http.Transport)
if base.TLSClientConfig.InsecureSkipVerify {
t.Error("verifySSL=true must not skip verification")
}
if secure.Timeout != 5*time.Second {
t.Errorf("Timeout = %v, want 5s", secure.Timeout)
}
}
func TestNewHTTPClientZeroTimeoutMeansNoTimeout(t *testing.T) {
// internal/download and the qobuz body copy rely on this: a zero timeout
// must not be replaced with a default, or long track downloads abort.
c := NewHTTPClient(0, true, 0)
if c.Timeout != 0 {
t.Errorf("Timeout = %v, want 0", c.Timeout)
}
}
func TestNewHTTPClientDefaultsMaxConns(t *testing.T) {
c := NewHTTPClient(time.Second, true, 0)
base := c.Transport.(*loggingTransport).base.(*http.Transport)
if base.MaxIdleConnsPerHost != defaultMaxConnsPerHost {
t.Errorf("MaxIdleConnsPerHost = %d, want %d", base.MaxIdleConnsPerHost, defaultMaxConnsPerHost)
}
}
func TestClientPerformsRequest(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusTeapot)
}))
defer ts.Close()
resp, err := NewHTTPClient(5*time.Second, true, 2).Get(ts.URL + "/x?token=sekrit")
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusTeapot {
t.Errorf("status = %d", resp.StatusCode)
}
}
+73 -51
View File
@@ -11,6 +11,7 @@ import (
"net/url" "net/url"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"streamrip-go/internal/config" "streamrip-go/internal/config"
@@ -18,11 +19,11 @@ import (
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit" "streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
) )
const ( const (
baseURL = "https://api.beatport.com/v4" baseURL = "https://api.beatport.com/v4"
clientID = "ryZ8LuyQVPqbK2mBX2Hwt4qSMtnWuTYSqBPO92yQ"
tokenEndpoint = "/auth/o/token/" tokenEndpoint = "/auth/o/token/"
authEndpoint = "/auth/o/authorize/" authEndpoint = "/auth/o/authorize/"
loginEndpoint = "/auth/login/" loginEndpoint = "/auth/login/"
@@ -38,8 +39,13 @@ type Client struct {
http *http.Client http *http.Client
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
baseURL string baseURL string
loggedIn bool
sessionID string sessionID string
// authMu serializes the token refresh / re-login path and guards loggedIn.
// ensureToken is reached from apiRequest, which concurrent download workers
// call, so without this several workers would refresh at once.
authMu sync.Mutex
loggedIn bool
} }
func New(cfg *config.Config) *Client { func New(cfg *config.Config) *Client {
@@ -48,16 +54,22 @@ func New(cfg *config.Config) *Client {
return &Client{ return &Client{
cfg: cfg, cfg: cfg,
http: h, http: h,
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute), limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
baseURL: baseURL, baseURL: baseURL,
} }
} }
func (c *Client) Source() string { return "beatport" } func (c *Client) Source() string { return "beatport" }
func (c *Client) LoggedIn() bool { return c.loggedIn } func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn
}
func (c *Client) Login(ctx context.Context) error { func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
b := &c.cfg.Session.Beatport b := &c.cfg.Session.Beatport
b.Username = strings.TrimSpace(b.Username) b.Username = strings.TrimSpace(b.Username)
b.Password = strings.TrimSpace(b.Password) b.Password = strings.TrimSpace(b.Password)
@@ -82,7 +94,7 @@ func (c *Client) Login(ctx context.Context) error {
} }
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) { func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errNotLoggedIn return nil, errNotLoggedIn
} }
switch mediaType { switch mediaType {
@@ -129,7 +141,7 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
} }
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) { func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errNotLoggedIn return nil, errNotLoggedIn
} }
if limit <= 0 { if limit <= 0 {
@@ -162,7 +174,7 @@ func (c *Client) Search(ctx context.Context, mediaType, query string, limit int)
} }
func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) { func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errNotLoggedIn return nil, errNotLoggedIn
} }
q := beatportQuality(quality) q := beatportQuality(quality)
@@ -296,11 +308,8 @@ func (c *Client) apiRequest(ctx context.Context, method, endpoint string, params
return nil, err return nil, err
} }
if status == http.StatusUnauthorized { if status == http.StatusUnauthorized {
c.cfg.Session.Beatport.TokenExpiry = 0 if err = c.reauthorize(ctx); err != nil {
if err = c.refresh(ctx); err != nil { return nil, err
if loginErr := c.loginPasswordFlow(ctx); loginErr != nil {
return nil, err
}
} }
resp, status, err = c.rawRequest(ctx, method, endpoint, params, payload, contentType, true) resp, status, err = c.rawRequest(ctx, method, endpoint, params, payload, contentType, true)
if err != nil { if err != nil {
@@ -314,6 +323,13 @@ func (c *Client) apiRequest(ctx context.Context, method, endpoint string, params
} }
func (c *Client) ensureToken(ctx context.Context) error { func (c *Client) ensureToken(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.ensureTokenLocked(ctx)
}
// ensureTokenLocked requires c.authMu to be held.
func (c *Client) ensureTokenLocked(ctx context.Context) error {
if strings.TrimSpace(c.cfg.Session.Beatport.AccessToken) != "" && time.Now().Unix()+300 < c.cfg.Session.Beatport.TokenExpiry { if strings.TrimSpace(c.cfg.Session.Beatport.AccessToken) != "" && time.Now().Unix()+300 < c.cfg.Session.Beatport.TokenExpiry {
return nil return nil
} }
@@ -325,6 +341,21 @@ func (c *Client) ensureToken(ctx context.Context) error {
return c.loginPasswordFlow(ctx) return c.loginPasswordFlow(ctx)
} }
// reauthorize forces a token refresh after a 401. Holding authMu means that
// when several workers get a 401 at once only the first re-authenticates and
// the rest reuse the resulting token.
func (c *Client) reauthorize(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
c.cfg.Session.Beatport.TokenExpiry = 0
if err := c.refresh(ctx); err != nil {
if loginErr := c.loginPasswordFlow(ctx); loginErr != nil {
return err
}
}
return nil
}
func (c *Client) loginPasswordFlow(ctx context.Context) error { func (c *Client) loginPasswordFlow(ctx context.Context) error {
b := &c.cfg.Session.Beatport b := &c.cfg.Session.Beatport
if strings.TrimSpace(b.Username) == "" || strings.TrimSpace(b.Password) == "" { if strings.TrimSpace(b.Username) == "" || strings.TrimSpace(b.Password) == "" {
@@ -344,11 +375,11 @@ func (c *Client) loginPasswordFlow(ctx context.Context) error {
if err != nil { if err != nil {
return err return err
} }
return c.issueToken(ctx, map[string]string{"client_id": clientID, "grant_type": "authorization_code", "code": code}) return c.issueToken(ctx, map[string]string{"client_id": c.appClientID(), "grant_type": "authorization_code", "code": code})
} }
func (c *Client) authorize(ctx context.Context) (string, error) { func (c *Client) authorize(ctx context.Context) (string, error) {
params := url.Values{"client_id": []string{clientID}, "response_type": []string{"code"}} params := url.Values{"client_id": []string{c.appClientID()}, "response_type": []string{"code"}}
resp, status, err := c.rawRequest(ctx, http.MethodGet, authEndpoint, params, nil, "", false) resp, status, err := c.rawRequest(ctx, http.MethodGet, authEndpoint, params, nil, "", false)
if err != nil { if err != nil {
return "", err return "", err
@@ -365,7 +396,7 @@ func (c *Client) authorize(ctx context.Context) (string, error) {
func (c *Client) refresh(ctx context.Context) error { func (c *Client) refresh(ctx context.Context) error {
b := &c.cfg.Session.Beatport b := &c.cfg.Session.Beatport
return c.issueToken(ctx, map[string]string{"client_id": clientID, "grant_type": "refresh_token", "refresh_token": b.RefreshToken}) return c.issueToken(ctx, map[string]string{"client_id": c.appClientID(), "grant_type": "refresh_token", "refresh_token": b.RefreshToken})
} }
func (c *Client) issueToken(ctx context.Context, payload map[string]string) error { func (c *Client) issueToken(ctx context.Context, payload map[string]string) error {
@@ -385,15 +416,19 @@ func (c *Client) issueToken(ctx context.Context, payload map[string]string) erro
if expiresIn <= 0 { if expiresIn <= 0 {
expiresIn = 3600 expiresIn = 3600
} }
c.cfg.Session.Beatport.AccessToken = access expiry := time.Now().Unix() + expiresIn
if refresh != "" { if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
c.cfg.Session.Beatport.RefreshToken = refresh session.Beatport.AccessToken = access
if refresh != "" {
session.Beatport.RefreshToken = refresh
}
session.Beatport.TokenExpiry = expiry
file.Beatport.AccessToken = session.Beatport.AccessToken
file.Beatport.RefreshToken = session.Beatport.RefreshToken
file.Beatport.TokenExpiry = session.Beatport.TokenExpiry
}); err != nil {
verbose.Printf(verbose.V, "beatport: could not persist refreshed token: %v\n", err)
} }
c.cfg.Session.Beatport.TokenExpiry = time.Now().Unix() + expiresIn
c.cfg.File.Beatport.AccessToken = c.cfg.Session.Beatport.AccessToken
c.cfg.File.Beatport.RefreshToken = c.cfg.Session.Beatport.RefreshToken
c.cfg.File.Beatport.TokenExpiry = c.cfg.Session.Beatport.TokenExpiry
_ = c.cfg.SaveFile()
return nil return nil
} }
@@ -497,7 +532,7 @@ func (c *Client) normalizeRelease(raw map[string]any, tracks []any) map[string]a
"title": jsonutil.StringFromAny(raw["name"]), "title": jsonutil.StringFromAny(raw["name"]),
"artist": map[string]any{"name": artist}, "artist": map[string]any{"name": artist},
"release_date_original": jsonutil.StringFromAny(raw["new_release_date"]), "release_date_original": jsonutil.StringFromAny(raw["new_release_date"]),
"tracks_count": firstPositiveInt(jsonutil.IntFromAny(raw["track_count"]), len(items)), "tracks_count": jsonutil.FirstPositiveInt(jsonutil.IntFromAny(raw["track_count"]), len(items)),
"maximum_bit_depth": 16, "maximum_bit_depth": 16,
"maximum_sampling_rate": "44.1", "maximum_sampling_rate": "44.1",
"image": imageMap(raw["image"]), "image": imageMap(raw["image"]),
@@ -510,7 +545,7 @@ func (c *Client) normalizeRelease(raw map[string]any, tracks []any) map[string]a
func (c *Client) normalizeTrack(raw map[string]any) map[string]any { func (c *Client) normalizeTrack(raw map[string]any) map[string]any {
release := mapAny(raw["release"]) release := mapAny(raw["release"])
trackArtistNames := artistNames(raw["artists"]) trackArtistNames := artistNames(raw["artists"])
artistName := displayArtistNames(trackArtistNames) artistName := jsonutil.DisplayArtistNames(trackArtistNames)
artistID := firstArtistID(raw["artists"]) artistID := firstArtistID(raw["artists"])
albumArtist := strings.TrimSpace(jsonutil.StringFromAny(release["album_artist_name"])) albumArtist := strings.TrimSpace(jsonutil.StringFromAny(release["album_artist_name"]))
if albumArtist == "" { if albumArtist == "" {
@@ -556,7 +591,7 @@ func (c *Client) normalizeTrackListItem(raw map[string]any) map[string]any {
"id": jsonutil.StringFromAny(raw["id"]), "id": jsonutil.StringFromAny(raw["id"]),
"title": jsonutil.StringFromAny(raw["name"]), "title": jsonutil.StringFromAny(raw["name"]),
"version": jsonutil.StringFromAny(raw["mix_name"]), "version": jsonutil.StringFromAny(raw["mix_name"]),
"artist": map[string]any{"name": displayArtistNames(names)}, "artist": map[string]any{"name": jsonutil.DisplayArtistNames(names)},
"album": map[string]any{"id": jsonutil.NestedString(raw, "release", "id"), "title": jsonutil.NestedString(raw, "release", "name")}, "album": map[string]any{"id": jsonutil.NestedString(raw, "release", "id"), "title": jsonutil.NestedString(raw, "release", "name")},
"track_number": jsonutil.IntFromAny(raw["number"]), "track_number": jsonutil.IntFromAny(raw["number"]),
} }
@@ -613,7 +648,7 @@ func playlistMetadata(raw map[string]any, tracks []any) map[string]any {
"id": jsonutil.StringFromAny(raw["id"]), "id": jsonutil.StringFromAny(raw["id"]),
"name": name, "name": name,
"title": name, "title": name,
"tracks_count": firstPositiveInt(jsonutil.IntFromAny(raw["track_count"]), len(tracks)), "tracks_count": jsonutil.FirstPositiveInt(jsonutil.IntFromAny(raw["track_count"]), len(tracks)),
"image": imageMap(raw["image"]), "image": imageMap(raw["image"]),
"tracks": map[string]any{"items": tracks}, "tracks": map[string]any{"items": tracks},
} }
@@ -676,32 +711,19 @@ func artistNames(v any) []string {
return names return names
} }
func displayArtistNames(names []string) string {
switch len(names) {
case 0:
return ""
case 1:
return names[0]
case 2:
return names[0] + " & " + names[1]
default:
return strings.Join(names[:len(names)-1], ", ") + " & " + names[len(names)-1]
}
}
func (c *Client) releaseArtistName(v any) string { func (c *Client) releaseArtistName(v any) string {
items := sliceAny(v) items := sliceAny(v)
threshold := c.cfg.Session.Beatport.VariousArtistsThreshold threshold := c.cfg.Session.Beatport.VariousArtistsThreshold
if threshold >= 0 && len(items) > threshold { if threshold >= 0 && len(items) > threshold {
return "Various Artists" return "Various Artists"
} }
return displayArtistNames(artistNames(v)) return jsonutil.DisplayArtistNames(artistNames(v))
} }
func (c *Client) releaseArtistNameForTracks(releaseArtists any, tracks []any) string { func (c *Client) releaseArtistNameForTracks(releaseArtists any, tracks []any) string {
common := commonReleaseArtists(releaseArtists, tracks) common := commonReleaseArtists(releaseArtists, tracks)
if len(common) > 0 { if len(common) > 0 {
return displayArtistNames(artistNames(common)) return jsonutil.DisplayArtistNames(artistNames(common))
} }
return c.releaseArtistName(releaseArtists) return c.releaseArtistName(releaseArtists)
} }
@@ -807,15 +829,6 @@ func sliceAny(v any) []any {
return s return s
} }
func firstPositiveInt(vals ...int) int {
for _, v := range vals {
if v > 0 {
return v
}
}
return 0
}
func cloneValues(in url.Values) url.Values { func cloneValues(in url.Values) url.Values {
out := url.Values{} out := url.Values{}
for k, vals := range in { for k, vals := range in {
@@ -823,3 +836,12 @@ func cloneValues(in url.Values) url.Values {
} }
return out return out
} }
// appClientID reads the vendor OAuth app credential from config, falling back
// to the built-in default.
func (c *Client) appClientID() string {
if v := strings.TrimSpace(c.cfg.Session.Beatport.AppClientID); v != "" {
return v
}
return config.DefaultBeatportAppClientID
}
+15 -4
View File
@@ -9,12 +9,13 @@ import (
"time" "time"
"streamrip-go/internal/config" "streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
) )
func newTestClient(t *testing.T, handler http.Handler) (*Client, func()) { func newTestClient(t *testing.T, handler http.Handler) (*Client, func()) {
t.Helper() t.Helper()
ts := httptest.NewServer(handler) ts := httptest.NewServer(handler)
cfg := &config.Config{Session: config.DefaultConfigData(), File: config.DefaultConfigData()} cfg := &config.Config{Session: testConfigData(), File: testConfigData()}
cfg.Session.Beatport.Username = "user" cfg.Session.Beatport.Username = "user"
cfg.Session.Beatport.Password = "pass" cfg.Session.Beatport.Password = "pass"
cfg.Session.Beatport.AccessToken = "token" cfg.Session.Beatport.AccessToken = "token"
@@ -199,7 +200,7 @@ func TestAlbumMetadataUsesArtistPresentOnEveryTrack(t *testing.T) {
} }
func TestReleaseArtistCollapseCanBeDisabled(t *testing.T) { func TestReleaseArtistCollapseCanBeDisabled(t *testing.T) {
cfg := &config.Config{Session: config.DefaultConfigData(), File: config.DefaultConfigData()} cfg := &config.Config{Session: testConfigData(), File: testConfigData()}
cfg.Session.Beatport.VariousArtistsThreshold = -1 cfg.Session.Beatport.VariousArtistsThreshold = -1
c := New(cfg) c := New(cfg)
got := c.releaseArtistName([]any{ got := c.releaseArtistName([]any{
@@ -214,13 +215,13 @@ func TestReleaseArtistCollapseCanBeDisabled(t *testing.T) {
} }
func TestDisplayArtistNamesUsesCreditPunctuation(t *testing.T) { func TestDisplayArtistNamesUsesCreditPunctuation(t *testing.T) {
got := displayArtistNames(artistNames([]any{ got := jsonutil.DisplayArtistNames(artistNames([]any{
map[string]any{"name": "A"}, map[string]any{"name": "A"},
map[string]any{"name": "B"}, map[string]any{"name": "B"},
map[string]any{"name": "C"}, map[string]any{"name": "C"},
})) }))
if got != "A, B & C" { if got != "A, B & C" {
t.Fatalf("displayArtistNames() = %q", got) t.Fatalf("DisplayArtistNames() = %q", got)
} }
} }
@@ -265,3 +266,13 @@ func TestArtistMetadataUsesReleases(t *testing.T) {
t.Fatalf("unexpected album: %+v", album) t.Fatalf("unexpected album: %+v", album)
} }
} }
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+51 -25
View File
@@ -15,6 +15,7 @@ import (
"net/url" "net/url"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"streamrip-go/internal/config" "streamrip-go/internal/config"
@@ -22,6 +23,7 @@ import (
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit" "streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
) )
var ( var (
@@ -31,8 +33,6 @@ var (
mediaURL = "https://media.deezer.com/v1/get_url" mediaURL = "https://media.deezer.com/v1/get_url"
pipeURL = "https://pipe.deezer.com/api" pipeURL = "https://pipe.deezer.com/api"
authURL = "https://auth.deezer.com/login/renew" authURL = "https://auth.deezer.com/login/renew"
apiKey = "4VCYIJUCDLOUELGD1V8WBVYBNVDYOXEWSLLZDONGBBDFVXTZJRXPR29JRLQFO6ZE"
gatewayDec = "VBK1FSUEXHTSDBJJ"
deezerUAPool = []string{ deezerUAPool = []string{
"Deezer/9.0.11.4 (Android; 14; Mobile; us) Xiaomi Redmi Note 7", "Deezer/9.0.11.4 (Android; 14; Mobile; us) Xiaomi Redmi Note 7",
"Deezer/9.0.11.4 (Android; 14; Mobile; us) Samsung SM-G991B", "Deezer/9.0.11.4 (Android; 14; Mobile; us) Samsung SM-G991B",
@@ -47,9 +47,12 @@ var (
) )
type Client struct { type Client struct {
cfg *config.Config cfg *config.Config
http *http.Client http *http.Client
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
// authMu guards loggedIn and the login/refresh path, which download
// workers can reach concurrently.
authMu sync.Mutex
loggedIn bool loggedIn bool
ua string ua string
deviceID string deviceID string
@@ -69,7 +72,7 @@ func New(cfg *config.Config) *Client {
return &Client{ return &Client{
cfg: cfg, cfg: cfg,
http: httpClient, http: httpClient,
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute), limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
ua: randomDeezerUA(), ua: randomDeezerUA(),
deviceID: randomHexN(32), deviceID: randomHexN(32),
arl: strings.TrimSpace(cfg.Session.Deezer.ARL), arl: strings.TrimSpace(cfg.Session.Deezer.ARL),
@@ -82,6 +85,8 @@ func (c *Client) Source() string {
} }
func (c *Client) Login(ctx context.Context) error { func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
c.arl = strings.TrimSpace(c.cfg.Session.Deezer.ARL) c.arl = strings.TrimSpace(c.cfg.Session.Deezer.ARL)
c.sid = "" c.sid = ""
c.jwt = "" c.jwt = ""
@@ -114,6 +119,8 @@ func (c *Client) ensureLaunchSession(ctx context.Context) error {
} }
func (c *Client) LoggedIn() bool { func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn return c.loggedIn
} }
@@ -122,7 +129,7 @@ func (c *Client) Close() error {
} }
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) { func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("deezer client not logged in") return nil, errors.New("deezer client not logged in")
} }
if limit <= 0 { if limit <= 0 {
@@ -151,7 +158,7 @@ func (c *Client) Search(ctx context.Context, mediaType, query string, limit int)
} }
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) { func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("deezer client not logged in") return nil, errors.New("deezer client not logged in")
} }
@@ -471,10 +478,11 @@ func (c *Client) persistRefreshToken() {
if rt == "" { if rt == "" {
return return
} }
c.cfg.Session.Deezer.RefreshToken = rt if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
c.cfg.File.Deezer.RefreshToken = rt file.Deezer.RefreshToken = rt
if strings.TrimSpace(c.cfg.Path) != "" { session.Deezer.RefreshToken = rt
_ = c.cfg.SaveFile() }); err != nil {
verbose.Printf(verbose.V, "deezer: could not persist refreshed token: %v\n", err)
} }
} }
@@ -489,7 +497,7 @@ func (c *Client) loginWithCredentials(ctx context.Context, email, password strin
if err != nil { if err != nil {
return err return err
} }
authToken, err := deriveGatewayAuthToken(mobileToken) authToken, err := deriveGatewayAuthToken(c.gatewayKey(), mobileToken)
if err != nil { if err != nil {
return err return err
} }
@@ -499,7 +507,7 @@ func (c *Client) loginWithCredentials(ctx context.Context, email, password strin
} }
c.sid = jsonutil.FirstNonEmpty(c.sid, sid) c.sid = jsonutil.FirstNonEmpty(c.sid, sid)
encryptedPassword, err := encryptPassword(mobileToken, password) encryptedPassword, err := encryptPassword(c.gatewayKey(), mobileToken, password)
if err != nil { if err != nil {
return err return err
} }
@@ -518,7 +526,7 @@ func (c *Client) loginWithCredentials(ctx context.Context, email, password strin
"password": encryptedPassword, "password": encryptedPassword,
} }
params := url.Values{} params := url.Values{}
params.Set("api_key", apiKey) params.Set("api_key", c.apiKey())
params.Set("sid", c.sid) params.Set("sid", c.sid)
params.Set("method", "mobile_userAuth") params.Set("method", "mobile_userAuth")
params.Set("output", "3") params.Set("output", "3")
@@ -727,7 +735,7 @@ func (c *Client) bootstrapSID(ctx context.Context) (string, error) {
if err != nil { if err != nil {
return "", err return "", err
} }
authToken, err := deriveGatewayAuthToken(mobileToken) authToken, err := deriveGatewayAuthToken(c.gatewayKey(), mobileToken)
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -739,7 +747,7 @@ func (c *Client) mobileAuth(ctx context.Context) (string, error) {
return "", err return "", err
} }
params := url.Values{} params := url.Values{}
params.Set("api_key", apiKey) params.Set("api_key", c.apiKey())
params.Set("output", "3") params.Set("output", "3")
params.Set("method", "mobile_auth") params.Set("method", "mobile_auth")
params.Set("network", randomHexN(32)) params.Set("network", randomHexN(32))
@@ -792,7 +800,7 @@ func (c *Client) apiCheckToken(ctx context.Context, authToken string) (string, e
return "", err return "", err
} }
params := url.Values{} params := url.Values{}
params.Set("api_key", apiKey) params.Set("api_key", c.apiKey())
params.Set("method", "api_checkToken") params.Set("method", "api_checkToken")
params.Set("auth_token", authToken) params.Set("auth_token", authToken)
params.Set("output", "3") params.Set("output", "3")
@@ -855,7 +863,7 @@ func (c *Client) mobileUserAutolog(ctx context.Context) error {
payload["refresh_token"] = strings.TrimSpace(c.refresh) payload["refresh_token"] = strings.TrimSpace(c.refresh)
} }
params := url.Values{} params := url.Values{}
params.Set("api_key", apiKey) params.Set("api_key", c.apiKey())
params.Set("sid", c.sid) params.Set("sid", c.sid)
params.Set("output", "3") params.Set("output", "3")
params.Set("input", "3") params.Set("input", "3")
@@ -1065,8 +1073,10 @@ func (c *Client) refreshLicenseFromPipe(ctx context.Context) error {
return nil return nil
} }
func deriveGatewayAuthToken(mobileToken string) (string, error) { // deriveGatewayAuthToken and friends take gatewayKey explicitly so the key
dec, err := decryptMobileToken(mobileToken) // comes from config rather than a package constant.
func deriveGatewayAuthToken(gatewayKey, mobileToken string) (string, error) {
dec, err := decryptMobileToken(gatewayKey, mobileToken)
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -1082,19 +1092,19 @@ func deriveGatewayAuthToken(mobileToken string) (string, error) {
return hex.EncodeToString(enc), nil return hex.EncodeToString(enc), nil
} }
func decryptMobileToken(mobileToken string) ([]byte, error) { func decryptMobileToken(gatewayKey, mobileToken string) ([]byte, error) {
b, err := hex.DecodeString(strings.TrimSpace(mobileToken)) b, err := hex.DecodeString(strings.TrimSpace(mobileToken))
if err != nil { if err != nil {
return nil, err return nil, err
} }
return aesECBDecrypt([]byte(gatewayDec), b) return aesECBDecrypt([]byte(gatewayKey), b)
} }
func encryptPassword(mobileToken, password string) (string, error) { func encryptPassword(gatewayKey, mobileToken, password string) (string, error) {
if strings.TrimSpace(password) == "" { if strings.TrimSpace(password) == "" {
return "", errors.New("missing deezer password") return "", errors.New("missing deezer password")
} }
dec, err := decryptMobileToken(mobileToken) dec, err := decryptMobileToken(gatewayKey, mobileToken)
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -1503,3 +1513,19 @@ func enrichAlbumImage(meta map[string]any) {
"original": cover, "original": cover,
} }
} }
// apiKey and gatewayKey read the vendor protocol keys from config, falling
// back to the built-in defaults.
func (c *Client) apiKey() string {
if v := strings.TrimSpace(c.cfg.Session.Deezer.APIKey); v != "" {
return v
}
return config.DefaultDeezerAPIKey
}
func (c *Client) gatewayKey() string {
if v := strings.TrimSpace(c.cfg.Session.Deezer.GatewayKey); v != "" {
return v
}
return config.DefaultDeezerGatewayKey
}
+30 -20
View File
@@ -24,7 +24,7 @@ func TestSearchTrack(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -74,7 +74,7 @@ func TestGetMetadataArtistPaginatesAlbums(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -135,7 +135,7 @@ func TestGetMetadataAlbumPaginatesTracks(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -189,7 +189,7 @@ func TestGetMetadataPlaylistPaginatesTracks(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -241,7 +241,7 @@ func TestGetDownloadableNativeCipher(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.ARL = "arl" cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -289,7 +289,7 @@ func TestGetDownloadablePrefersNoneCipherWhenAvailable(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.ARL = "arl" cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -354,7 +354,7 @@ func TestLoginPrefersARLFlowOverRefreshShortcut(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.ARL = "arl" cfgData.Deezer.ARL = "arl"
cfgData.Deezer.RefreshToken = "refresh-token" cfgData.Deezer.RefreshToken = "refresh-token"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -386,7 +386,7 @@ func TestLoginPrefersARLFlowOverRefreshShortcut(t *testing.T) {
} }
func TestGetDownloadableRequiresARL(t *testing.T) { func TestGetDownloadableRequiresARL(t *testing.T) {
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.ARL = "" cfgData.Deezer.ARL = ""
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -409,7 +409,7 @@ func TestGetDownloadableDRMError(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.ARL = "arl" cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -448,7 +448,7 @@ func TestGetTrackTokenPrefersPipeToken(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.jwt = "jwt-token" c.jwt = "jwt-token"
@@ -499,7 +499,7 @@ func TestGetDownloadableUsesPipeTrackToken(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.ARL = "arl" cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
@@ -541,7 +541,7 @@ func TestGetMetadataAddsLyricsFromPipe(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.jwt = "jwt" c.jwt = "jwt"
@@ -594,7 +594,7 @@ func TestLoginWithCredentials(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.Email = "tidal1@alpin.sbs" cfgData.Deezer.Email = "tidal1@alpin.sbs"
cfgData.Deezer.Password = "tidal1@alpin.sbs" cfgData.Deezer.Password = "tidal1@alpin.sbs"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -662,7 +662,7 @@ func TestMobileAuthIncludesAppContextParams(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
origGateway := gatewayURL origGateway := gatewayURL
@@ -703,7 +703,7 @@ func TestMobileUserAutologIncludesRefreshToken(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.sid = "sid123" c.sid = "sid123"
c.userID = "42" c.userID = "42"
@@ -728,7 +728,7 @@ func TestMobileUserAutologIncludesRefreshToken(t *testing.T) {
func testMobileToken(t *testing.T) string { func testMobileToken(t *testing.T) string {
t.Helper() t.Helper()
plain := []byte(strings.Repeat("A", 64) + strings.Repeat("B", 16) + strings.Repeat("C", 16)) plain := []byte(strings.Repeat("A", 64) + strings.Repeat("B", 16) + strings.Repeat("C", 16))
enc, err := aesECBEncrypt([]byte(gatewayDec), plain) enc, err := aesECBEncrypt([]byte(config.DefaultDeezerGatewayKey), plain)
if err != nil { if err != nil {
t.Fatalf("aesECBEncrypt() error = %v", err) t.Fatalf("aesECBEncrypt() error = %v", err)
} }
@@ -748,7 +748,7 @@ func TestLoginWithRefreshToken(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Deezer.RefreshToken = "refresh-token" cfgData.Deezer.RefreshToken = "refresh-token"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -782,7 +782,7 @@ func TestRefreshJWTHTTPError(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.refresh = "refresh-token" c.refresh = "refresh-token"
@@ -802,7 +802,7 @@ func TestRefreshLicenseFromPipeGraphQLError(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.jwt = "jwt-token" c.jwt = "jwt-token"
@@ -848,7 +848,7 @@ func TestRefreshLicenseFromPipeUsesMobileGraphQLShape(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.jwt = "jwt-token" c.jwt = "jwt-token"
@@ -863,3 +863,13 @@ func TestRefreshLicenseFromPipeUsesMobileGraphQLShape(t *testing.T) {
t.Fatalf("license = %q, want license-token", c.license) t.Fatalf("license = %q, want license-token", c.license)
} }
} }
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+256 -42
View File
@@ -30,6 +30,7 @@ import (
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit" "streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
"github.com/vbauerster/mpb/v8" "github.com/vbauerster/mpb/v8"
"github.com/vbauerster/mpb/v8/decor" "github.com/vbauerster/mpb/v8/decor"
@@ -39,8 +40,6 @@ import (
const baseURL = "https://www.qobuz.com/api.json/0.2" const baseURL = "https://www.qobuz.com/api.json/0.2"
const ( const (
mobileAppID = "312369995"
mobileAppSecret = "e79f8b9be485692b0e5f9dd895826368"
mobileUserAgent = "Dalvik/2.1.0 (Linux; U; Android 9; Nexus 6P Build/PQ3A.190801.002) QobuzMobileAndroid/9.7.0.3-b26022717" mobileUserAgent = "Dalvik/2.1.0 (Linux; U; Android 9; Nexus 6P Build/PQ3A.190801.002) QobuzMobileAndroid/9.7.0.3-b26022717"
mobileAppVersion = "9.7.0.3" mobileAppVersion = "9.7.0.3"
mobileSessionProf = "qbz-1" mobileSessionProf = "qbz-1"
@@ -58,11 +57,15 @@ var (
) )
type Client struct { type Client struct {
cfg *config.Config cfg *config.Config
http *http.Client http *http.Client
limiter *ratelimit.Limiter streamHTTP *http.Client
baseURL string limiter *ratelimit.Limiter
fetchCfg func(ctx context.Context) (string, []string, error) baseURL string
fetchCfg func(ctx context.Context) (string, []string, error)
// authMu guards loggedIn and the login/refresh path, which download
// workers can reach concurrently.
authMu sync.Mutex
loggedIn bool loggedIn bool
secret string secret string
uat string uat string
@@ -87,11 +90,16 @@ type mobileFileURL struct {
func New(cfg *config.Config) *Client { func New(cfg *config.Config) *Client {
return &Client{ return &Client{
cfg: cfg, cfg: cfg,
http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections), http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute), // Separate client with no Timeout for streaming a whole track body.
baseURL: baseURL, // http.Client.Timeout bounds the entire request including the body
fetchCfg: nil, // read, so the 30s API client aborted any track that took longer than
// 30s to stream. internal/download does the same for this reason.
streamHTTP: netutil.NewHTTPClient(0, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
baseURL: baseURL,
fetchCfg: nil,
} }
} }
@@ -100,10 +108,14 @@ func (c *Client) Source() string {
} }
func (c *Client) LoggedIn() bool { func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn return c.loggedIn
} }
func (c *Client) Login(ctx context.Context) error { func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
q := &c.cfg.Session.Qobuz q := &c.cfg.Session.Qobuz
q.EmailOrUserID = strings.TrimSpace(q.EmailOrUserID) q.EmailOrUserID = strings.TrimSpace(q.EmailOrUserID)
q.PasswordOrToken = strings.TrimSpace(q.PasswordOrToken) q.PasswordOrToken = strings.TrimSpace(q.PasswordOrToken)
@@ -203,14 +215,18 @@ func (c *Client) refreshAppCredentials(ctx context.Context, q *config.QobuzConfi
return errors.New("qobuz app credential refresh returned no secrets") return errors.New("qobuz app credential refresh returned no secrets")
} }
q.Secrets = append([]string(nil), clean...) q.Secrets = append([]string(nil), clean...)
c.cfg.File.Qobuz.AppID = q.AppID refreshedAppID := q.AppID
c.cfg.File.Qobuz.Secrets = append([]string(nil), clean...) if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
_ = c.cfg.SaveFile() file.Qobuz.AppID = refreshedAppID
file.Qobuz.Secrets = append([]string(nil), clean...)
}); err != nil {
verbose.Printf(verbose.V, "qobuz: could not persist refreshed credentials: %v\n", err)
}
return nil return nil
} }
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) { func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errNotLoggedIn return nil, errNotLoggedIn
} }
if mediaType == "playlist" { if mediaType == "playlist" {
@@ -251,16 +267,8 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
return resp, nil return resp, nil
} }
func (c *Client) GetTrackMetadata(ctx context.Context, id string) (*TrackMetadata, error) {
raw, err := c.GetMetadata(ctx, id, "track")
if err != nil {
return nil, err
}
return ParseTrackMetadata(raw)
}
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) { func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errNotLoggedIn return nil, errNotLoggedIn
} }
if limit <= 0 { if limit <= 0 {
@@ -283,7 +291,7 @@ func (c *Client) Search(ctx context.Context, mediaType, query string, limit int)
} }
func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) { func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errNotLoggedIn return nil, errNotLoggedIn
} }
if quality < 1 || quality > 4 { if quality < 1 || quality > 4 {
@@ -1036,11 +1044,11 @@ func (c *Client) ensureMobileSession(ctx context.Context) error {
func (c *Client) mobileLogin(ctx context.Context, username, password string) error { func (c *Client) mobileLogin(ctx context.Context, username, password string) error {
ts := time.Now().Unix() ts := time.Now().Unix()
params := url.Values{} params := url.Values{}
params.Set("app_id", mobileAppID) params.Set("app_id", c.mobileAppID())
params.Set("username", username) params.Set("username", username)
params.Set("password", password) params.Set("password", password)
params.Set("request_ts", strconv.FormatInt(ts, 10)) params.Set("request_ts", strconv.FormatInt(ts, 10))
params.Set("request_sig", mobileSignRequest("oauth2/login", []kv{{"password", password}, {"username", username}}, ts)) params.Set("request_sig", mobileSignRequest(c.mobileAppSecret(), "oauth2/login", []kv{{"password", password}, {"username", username}}, ts))
reqURL := baseURL + "/oauth2/login?" + params.Encode() reqURL := baseURL + "/oauth2/login?" + params.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil) req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
@@ -1078,9 +1086,9 @@ func (c *Client) mobileLogin(ctx context.Context, username, password string) err
func (c *Client) mobileStartSession(ctx context.Context) error { func (c *Client) mobileStartSession(ctx context.Context) error {
ts := time.Now().Unix() ts := time.Now().Unix()
params := url.Values{} params := url.Values{}
params.Set("app_id", mobileAppID) params.Set("app_id", c.mobileAppID())
params.Set("request_ts", strconv.FormatInt(ts, 10)) params.Set("request_ts", strconv.FormatInt(ts, 10))
params.Set("request_sig", mobileSignRequest("session/start", []kv{{"profile", mobileSessionProf}}, ts)) params.Set("request_sig", mobileSignRequest(c.mobileAppSecret(), "session/start", []kv{{"profile", mobileSessionProf}}, ts))
reqURL := baseURL + "/session/start?" + params.Encode() reqURL := baseURL + "/session/start?" + params.Encode()
form := url.Values{} form := url.Values{}
@@ -1122,12 +1130,12 @@ func (c *Client) mobileStartSession(ctx context.Context) error {
func (c *Client) mobileGetFileURL(ctx context.Context, trackID string, formatID int) (*mobileFileURL, error) { func (c *Client) mobileGetFileURL(ctx context.Context, trackID string, formatID int) (*mobileFileURL, error) {
ts := time.Now().Unix() ts := time.Now().Unix()
params := url.Values{} params := url.Values{}
params.Set("app_id", mobileAppID) params.Set("app_id", c.mobileAppID())
params.Set("track_id", trackID) params.Set("track_id", trackID)
params.Set("format_id", strconv.Itoa(formatID)) params.Set("format_id", strconv.Itoa(formatID))
params.Set("intent", "stream") params.Set("intent", "stream")
params.Set("request_ts", strconv.FormatInt(ts, 10)) params.Set("request_ts", strconv.FormatInt(ts, 10))
params.Set("request_sig", mobileSignRequest("file/url", []kv{{"format_id", strconv.Itoa(formatID)}, {"intent", "stream"}, {"track_id", trackID}}, ts)) params.Set("request_sig", mobileSignRequest(c.mobileAppSecret(), "file/url", []kv{{"format_id", strconv.Itoa(formatID)}, {"intent", "stream"}, {"track_id", trackID}}, ts))
reqURL := baseURL + "/file/url?" + params.Encode() reqURL := baseURL + "/file/url?" + params.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil) req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
if err != nil { if err != nil {
@@ -1159,7 +1167,11 @@ func (c *Client) mobileCopyURLToWriter(ctx context.Context, sourceURL string, ou
return err return err
} }
c.setMobileHeaders(req) c.setMobileHeaders(req)
resp, err := c.http.Do(req) client := c.streamHTTP
if client == nil {
client = c.http
}
resp, err := client.Do(req)
if err != nil { if err != nil {
return err return err
} }
@@ -1245,7 +1257,7 @@ func (c *Client) mobileDownloadSegment(ctx context.Context, sourceURL string) ([
func (c *Client) setMobileHeaders(req *http.Request) { func (c *Client) setMobileHeaders(req *http.Request) {
req.Header.Set("User-Agent", mobileUserAgent) req.Header.Set("User-Agent", mobileUserAgent)
req.Header.Set("X-App-Id", mobileAppID) req.Header.Set("X-App-Id", c.mobileAppID())
req.Header.Set("X-App-Version", mobileAppVersion) req.Header.Set("X-App-Version", mobileAppVersion)
req.Header.Set("X-Device-Platform", "android") req.Header.Set("X-Device-Platform", "android")
req.Header.Set("X-Device-Model", "Nexus 6P") req.Header.Set("X-Device-Model", "Nexus 6P")
@@ -1258,7 +1270,9 @@ func (c *Client) setMobileHeaders(req *http.Request) {
} }
} }
func mobileSignRequest(endpoint string, params []kv, ts int64) string { // mobileSignRequest signs a mobile API request. appSecret comes from config
// (Client.mobileAppSecret) so a rotation can be fixed without a rebuild.
func mobileSignRequest(appSecret, endpoint string, params []kv, ts int64) string {
method := strings.ReplaceAll(endpoint, "/", "") method := strings.ReplaceAll(endpoint, "/", "")
sortKVs(params) sortKVs(params)
var sb strings.Builder var sb strings.Builder
@@ -1268,7 +1282,7 @@ func mobileSignRequest(endpoint string, params []kv, ts int64) string {
sb.WriteString(p.Value) sb.WriteString(p.Value)
} }
sb.WriteString(strconv.FormatInt(ts, 10)) sb.WriteString(strconv.FormatInt(ts, 10))
sb.WriteString(mobileAppSecret) sb.WriteString(appSecret)
h := md5.Sum([]byte(sb.String())) h := md5.Sum([]byte(sb.String()))
return hex.EncodeToString(h[:]) return hex.EncodeToString(h[:])
} }
@@ -1304,7 +1318,17 @@ func (c *Client) mobileDeriveTrackKey(encryptedKey string) ([]byte, error) {
if err != nil { if err != nil {
return nil, fmt.Errorf("decode mobile info: %w", err) return nil, fmt.Errorf("decode mobile info: %w", err)
} }
reader := hkdf.New(func() hash.Hash { return sha256.New() }, hexDecodeOrNil(mobileAppSecret), salt, info) // A non-hex secret used to be swallowed and HKDF keyed on nil, which
// derives a wrong KEK and yields silently corrupt audio rather than an
// error.
secretKey, err := hex.DecodeString(c.mobileAppSecret())
if err != nil {
return nil, fmt.Errorf("decode mobile app secret: %w", err)
}
if len(secretKey) == 0 {
return nil, errors.New("mobile app secret is empty")
}
reader := hkdf.New(func() hash.Hash { return sha256.New() }, secretKey, salt, info)
kek := make([]byte, 16) kek := make([]byte, 16)
if _, err = io.ReadFull(reader, kek); err != nil { if _, err = io.ReadFull(reader, kek); err != nil {
return nil, fmt.Errorf("mobile hkdf derive failed: %w", err) return nil, fmt.Errorf("mobile hkdf derive failed: %w", err)
@@ -1313,11 +1337,6 @@ func (c *Client) mobileDeriveTrackKey(encryptedKey string) ([]byte, error) {
return unwrapQobuzTrackKey(encryptedKey, kek) return unwrapQobuzTrackKey(encryptedKey, kek)
} }
func hexDecodeOrNil(s string) []byte {
b, _ := hex.DecodeString(s)
return b
}
func unwrapQobuzTrackKey(encryptedKey string, kek []byte) ([]byte, error) { func unwrapQobuzTrackKey(encryptedKey string, kek []byte) ([]byte, error) {
parts := strings.SplitN(encryptedKey, ".", 3) parts := strings.SplitN(encryptedKey, ".", 3)
if len(parts) != 3 || parts[0] != mobileSessionProf { if len(parts) != 3 || parts[0] != mobileSessionProf {
@@ -1472,3 +1491,198 @@ func readU24BE(data []byte, off int) uint32 {
} }
return uint32(data[off])<<16 | uint32(data[off+1])<<8 | uint32(data[off+2]) return uint32(data[off])<<16 | uint32(data[off+1])<<8 | uint32(data[off+2])
} }
// mobileAppID and mobileAppSecret read the vendor mobile app credentials from
// config, falling back to the built-in defaults.
func (c *Client) mobileAppID() string {
if v := strings.TrimSpace(c.cfg.Session.Qobuz.MobileAppID); v != "" {
return v
}
return config.DefaultQobuzMobileAppID
}
func (c *Client) mobileAppSecret() string {
if v := strings.TrimSpace(c.cfg.Session.Qobuz.MobileAppSecret); v != "" {
return v
}
return config.DefaultQobuzMobileAppSecret
}
// AlbumTracksAreComplete reports that Qobuz's album/get response embeds track
// objects complete enough to tag from, so internal/app can skip the per-track
// track/get call.
//
// Checked against the live API for a 68-track release: the embedded items
// carry title, version, track_number, media_number, parental_warning, isrc,
// copyright, performer, performers, composer, duration, audio_info,
// maximum_bit_depth, maximum_sampling_rate and release_date_original. The only
// field track/get adds that this app reads is the nested album object, and the
// caller attaches the parent album itself. Everything else track/get returns
// (articles, created_at, has_lyrics, indexed_at) is unused.
func (c *Client) AlbumTracksAreComplete() bool { return true }
// Qobuz serves lyrics in two steps, mirroring its Android app: track/lyricsUrl
// returns a short-lived signed CloudFront URL, and that URL serves the lyrics
// document itself.
//
// Verified against the live API:
//
// GET track/lyricsUrl?track_id=&app_id=&request_ts=&request_sig=
// -> {"track_id","album_id","lyrics_url"}
// GET <lyrics_url>
// -> {"original":{"type","lang","lines":[{"line","start","end"}]},
// "translation_langs":[...],"writers","publishers":[...]}
//
// "type" is plain, lsync (line-synced) or wsync (word-synced); start and end
// are milliseconds. Passing a language parameter returns a translation
// document instead of the original, and only for languages that exist for the
// track, so this deliberately omits it and always takes the original.
const lyricsURLEndpoint = "track/lyricsUrl"
// lyricsExcluded reports whether the user opted out via
// metadata.exclude = ["lyrics"].
func (c *Client) lyricsExcluded() bool {
for _, k := range c.cfg.Session.Metadata.Exclude {
if strings.EqualFold(strings.TrimSpace(k), "lyrics") {
return true
}
}
return false
}
// TrackLyrics returns the plain and synced (LRC) lyrics for a track, or empty
// strings when Qobuz has none. It is exposed separately from GetMetadata so
// that album rips, which reuse the track objects embedded in the album
// payload, can still fetch lyrics without a full per-track metadata call.
func (c *Client) TrackLyrics(ctx context.Context, trackID string) (string, string, error) {
trackID = strings.TrimSpace(trackID)
if trackID == "" || c.lyricsExcluded() {
return "", "", nil
}
if !c.LoggedIn() {
return "", "", errNotLoggedIn
}
requestTS := strconv.FormatInt(time.Now().Unix(), 10)
sigRaw := "tracklyricsUrltrack_id" + trackID + requestTS + c.secret
sum := md5.Sum([]byte(sigRaw))
params := url.Values{}
params.Set("app_id", c.cfg.Session.Qobuz.AppID)
params.Set("track_id", trackID)
params.Set("request_ts", requestTS)
params.Set("request_sig", hex.EncodeToString(sum[:]))
resp, status, err := c.apiRequest(ctx, lyricsURLEndpoint, params, c.authHeaders())
if err != nil {
return "", "", err
}
if status != http.StatusOK {
// A track without lyrics is not an error worth failing a rip over.
verbose.Printf(verbose.V, "qobuz: no lyrics for track %s (status=%d)\n", trackID, status)
return "", "", nil
}
docURL := strings.TrimSpace(jsonutil.StringFromAny(resp["lyrics_url"]))
if docURL == "" {
return "", "", nil
}
return c.fetchLyricsDocument(ctx, trackID, docURL)
}
// fetchLyricsDocument retrieves the signed lyrics document. The URL points at
// a CDN rather than the Qobuz API, so it deliberately bypasses the API rate
// limiter: it is a different host and does not count against the provider's
// request budget.
func (c *Client) fetchLyricsDocument(ctx context.Context, trackID, docURL string) (string, string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, docURL, nil)
if err != nil {
return "", "", err
}
resp, err := c.http.Do(req)
if err != nil {
return "", "", err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
// The signed URL 403s when the document does not exist, which is how
// Qobuz represents "no lyrics in this language".
verbose.Printf(verbose.V, "qobuz: lyrics document for track %s unavailable (status=%d)\n", trackID, resp.StatusCode)
return "", "", nil
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return "", "", err
}
var doc map[string]any
if err = json.Unmarshal(body, &doc); err != nil {
return "", "", fmt.Errorf("qobuz lyrics document: %w", err)
}
plain, synced := parseQobuzLyrics(jsonutil.NestedMap(doc, "original"))
return plain, synced, nil
}
// parseQobuzLyrics turns a lyrics object into plain text and, when the lines
// carry timings, an LRC transcript.
//
// Qobuz separates verses with entries that have an empty line and no start or
// end at all. Those are kept as blank lines in the plain text, so stanzas stay
// readable, but omitted from the LRC: a synced entry needs a timestamp, and
// inventing one would park a blank line at the wrong moment.
func parseQobuzLyrics(obj map[string]any) (string, string) {
lines, ok := obj["lines"].([]any)
if !ok || len(lines) == 0 {
return "", ""
}
// wsync documents are line objects that additionally carry per-word
// timings; the line text and start sit in the same place, so both synced
// variants are handled identically.
kind := strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(obj["type"])))
synced := kind == "lsync" || kind == "wsync"
plainParts := make([]string, 0, len(lines))
lrcParts := make([]string, 0, len(lines))
for _, raw := range lines {
m, ok := raw.(map[string]any)
if !ok {
continue
}
text := strings.TrimSpace(jsonutil.StringFromAny(m["line"]))
start, hasStart := m["start"]
timed := synced && hasStart && start != nil
if text == "" {
if timed {
// A blank line that does carry a timestamp is a display-clear
// marker: it belongs in the LRC, where it blanks the lyric at
// that moment, but not in the plain text.
ms := jsonutil.IntFromAny(start)
lrcParts = append(lrcParts, fmt.Sprintf("[%02d:%05.2f]", ms/60000, float64(ms%60000)/1000.0))
continue
}
// Untimed blank: a verse separator. Keep a single blank line, and
// never a leading one.
if len(plainParts) > 0 && plainParts[len(plainParts)-1] != "" {
plainParts = append(plainParts, "")
}
continue
}
plainParts = append(plainParts, text)
if timed {
ms := jsonutil.IntFromAny(start)
lrcParts = append(lrcParts, fmt.Sprintf("[%02d:%05.2f]%s", ms/60000, float64(ms%60000)/1000.0, text))
}
}
// Drop a trailing separator.
for len(plainParts) > 0 && plainParts[len(plainParts)-1] == "" {
plainParts = plainParts[:len(plainParts)-1]
}
if len(plainParts) == 0 {
return "", ""
}
return strings.Join(plainParts, "\n"), strings.Join(lrcParts, "\n")
}
+229 -4
View File
@@ -208,7 +208,7 @@ func TestGetArtistPagination(t *testing.T) {
func newTestClient(t *testing.T) *Client { func newTestClient(t *testing.T) *Client {
t.Helper() t.Helper()
d := config.DefaultConfigData() d := testConfigData()
d.Qobuz.AppID = "12345" d.Qobuz.AppID = "12345"
cfg := &config.Config{File: d, Session: d} cfg := &config.Config{File: d, Session: d}
return New(cfg) return New(cfg)
@@ -248,7 +248,7 @@ func TestLoginRefreshesAppCredentialsWhenSecretInvalid(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
d := config.DefaultConfigData() d := testConfigData()
d.Qobuz.EmailOrUserID = "user@example.com" d.Qobuz.EmailOrUserID = "user@example.com"
d.Qobuz.PasswordOrToken = "hash" d.Qobuz.PasswordOrToken = "hash"
d.Qobuz.AppID = "old-app" d.Qobuz.AppID = "old-app"
@@ -302,7 +302,7 @@ func TestLoginRetriesAfterRefreshingAppCredentials(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
d := config.DefaultConfigData() d := testConfigData()
d.Qobuz.EmailOrUserID = "user@example.com" d.Qobuz.EmailOrUserID = "user@example.com"
d.Qobuz.PasswordOrToken = "hash" d.Qobuz.PasswordOrToken = "hash"
d.Qobuz.AppID = "old-app" d.Qobuz.AppID = "old-app"
@@ -396,7 +396,7 @@ func qobuzSecretSig(requestTS, secret string) string {
} }
func TestRefreshAppCredentialsRejectsEmptyData(t *testing.T) { func TestRefreshAppCredentialsRejectsEmptyData(t *testing.T) {
d := config.DefaultConfigData() d := testConfigData()
c := New(&config.Config{File: d, Session: d}) c := New(&config.Config{File: d, Session: d})
c.fetchCfg = func(context.Context) (string, []string, error) { c.fetchCfg = func(context.Context) (string, []string, error) {
return "", []string{" "}, nil return "", []string{" "}, nil
@@ -406,3 +406,228 @@ func TestRefreshAppCredentialsRejectsEmptyData(t *testing.T) {
t.Fatalf("expected error for empty refreshed app credentials") t.Fatalf("expected error for empty refreshed app credentials")
} }
} }
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
// newLyricsServer serves the two-step Qobuz lyrics flow: track/lyricsUrl
// returns a URL on this same server, which then serves the document.
func newLyricsServer(t *testing.T, doc any, docStatus int) *httptest.Server {
t.Helper()
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/track/lyricsUrl":
if r.URL.Query().Get("request_sig") == "" {
t.Error("track/lyricsUrl called without a request signature")
}
if r.URL.Query().Get("language") != "" {
t.Error("language must be omitted so the original lyrics are returned, not a translation")
}
_ = json.NewEncoder(w).Encode(map[string]any{
"track_id": r.URL.Query().Get("track_id"),
"album_id": "a1",
"lyrics_url": srv.URL + "/doc.json?Signature=x",
})
case "/doc.json":
if docStatus != http.StatusOK {
w.WriteHeader(docStatus)
_, _ = w.Write([]byte(`<?xml version="1.0"?><Error><Code>AccessDenied</Code></Error>`))
return
}
_ = json.NewEncoder(w).Encode(doc)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
t.Cleanup(srv.Close)
return srv
}
func lyricsClient(t *testing.T, srv *httptest.Server, exclude ...string) *Client {
t.Helper()
d := testConfigData()
d.Metadata.Exclude = exclude
c := New(&config.Config{File: d, Session: d})
c.baseURL = srv.URL
c.loggedIn = true
c.secret = "secret"
return c
}
func TestTrackLyricsLineSynced(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"track_id": "42",
"translation_langs": []any{"fr", "de"},
"original": map[string]any{
"type": "lsync",
"lang": "en",
"lines": []any{
map[string]any{"line": "First line", "start": 5780, "end": 8710},
map[string]any{"line": "", "start": 9000, "end": 9100},
map[string]any{"line": "Second line", "start": 65250, "end": 68000},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatalf("TrackLyrics() error = %v", err)
}
// The timestamped blank is a display-clear marker: it belongs in the LRC
// but not in the readable plain text.
if plain != "First line\nSecond line" {
t.Errorf("plain = %q", plain)
}
want := "[00:05.78]First line\n[00:09.00]\n[01:05.25]Second line"
if lrc != want {
t.Errorf("lrc = %q, want %q", lrc, want)
}
}
func TestTrackLyricsPlainHasNoTimestamps(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "plain",
"lines": []any{
map[string]any{"line": "Just text"},
map[string]any{"line": "More text"},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
if plain != "Just text\nMore text" {
t.Errorf("plain = %q", plain)
}
if lrc != "" {
t.Errorf("plain lyrics must not produce an LRC transcript, got %q", lrc)
}
}
func TestTrackLyricsWordSyncedUsesLineTimings(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "wsync",
"lines": []any{
map[string]any{
"line": "Hello world",
"start": 1500,
"words": []any{
map[string]any{"text": "Hello", "start": 1500, "end": 1800},
map[string]any{"text": "world", "start": 1800, "end": 2100},
},
},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
if plain != "Hello world" {
t.Errorf("plain = %q", plain)
}
if lrc != "[00:01.50]Hello world" {
t.Errorf("lrc = %q", lrc)
}
}
// A missing document 403s on the signed URL; that is how Qobuz says "no
// lyrics", so it must not fail the rip.
func TestTrackLyricsMissingDocumentIsNotAnError(t *testing.T) {
srv := newLyricsServer(t, nil, http.StatusForbidden)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatalf("a missing lyrics document must not be an error, got %v", err)
}
if plain != "" || lrc != "" {
t.Errorf("expected no lyrics, got plain=%q lrc=%q", plain, lrc)
}
}
func TestTrackLyricsRespectsMetadataExclude(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Errorf("no request should be made when lyrics are excluded, got %s", r.URL.Path)
}))
defer srv.Close()
plain, lrc, err := lyricsClient(t, srv, "lyrics").TrackLyrics(context.Background(), "42")
if err != nil || plain != "" || lrc != "" {
t.Fatalf("exclude should short-circuit; got plain=%q lrc=%q err=%v", plain, lrc, err)
}
}
func TestTrackLyricsEmptyIDIsNoOp(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Errorf("no request expected for an empty track id, got %s", r.URL.Path)
}))
defer srv.Close()
if plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), " "); err != nil || plain != "" || lrc != "" {
t.Fatalf("got plain=%q lrc=%q err=%v", plain, lrc, err)
}
}
// TestTrackLyricsKeepsVerseBreaks covers Qobuz's real separator shape: an
// empty line with no start or end. Those keep the plain text in stanzas but
// cannot become LRC entries, since there is no timestamp to use.
func TestTrackLyricsKeepsVerseBreaks(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "lsync",
"lines": []any{
map[string]any{"line": "verse one line one", "start": 1000},
map[string]any{"line": "verse one line two", "start": 2000},
map[string]any{"line": ""},
map[string]any{"line": ""},
map[string]any{"line": "verse two line one", "start": 3000},
map[string]any{"line": ""},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
wantPlain := "verse one line one\nverse one line two\n\nverse two line one"
if plain != wantPlain {
t.Errorf("plain = %q, want %q", plain, wantPlain)
}
wantLRC := "[00:01.00]verse one line one\n[00:02.00]verse one line two\n[00:03.00]verse two line one"
if lrc != wantLRC {
t.Errorf("lrc = %q, want %q", lrc, wantLRC)
}
}
// TestTrackLyricsLeadingSeparatorDropped guards against a lyric tag that
// starts with a blank line.
func TestTrackLyricsLeadingSeparatorDropped(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "plain",
"lines": []any{map[string]any{"line": ""}, map[string]any{"line": "first real line"}},
},
}, http.StatusOK)
plain, _, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
if plain != "first real line" {
t.Errorf("plain = %q, want no leading blank", plain)
}
}
+35 -14
View File
@@ -16,7 +16,9 @@ import (
"streamrip-go/internal/config" "streamrip-go/internal/config"
"streamrip-go/internal/jsonutil" "streamrip-go/internal/jsonutil"
"streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit"
) )
var errUnsupportedMediaType = errors.New("unsupported soundcloud media type") var errUnsupportedMediaType = errors.New("unsupported soundcloud media type")
@@ -26,22 +28,32 @@ var soundcloudSearchBaseURL = "https://soundcloud.com"
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error) type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
type Client struct { type Client struct {
cfg *config.Config cfg *config.Config
bin string
run commandRunner
http *http.Client
limiter *ratelimit.Limiter
mu sync.Mutex
cache map[string]map[string]any
// authMu guards loggedIn. Kept separate from mu, which guards cache, so a
// cache operation can never block a login check.
authMu sync.Mutex
loggedIn bool loggedIn bool
bin string
run commandRunner
http *http.Client
mu sync.Mutex
cache map[string]map[string]any
} }
func New(cfg *config.Config) *Client { func New(cfg *config.Config) *Client {
return &Client{ return &Client{
cfg: cfg, cfg: cfg,
bin: "yt-dlp", bin: "yt-dlp",
run: runCommand, run: runCommand,
http: &http.Client{Timeout: 20 * time.Second}, // Use the shared constructor like every other provider: rolling our own
cache: map[string]map[string]any{}, // client here meant soundcloud silently ignored downloads.verify_ssl and
// downloads.max_connections, and bypassed the transport that logs
// requests at -vv with credentials redacted.
http: netutil.NewHTTPClient(20*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
cache: map[string]map[string]any{},
} }
} }
@@ -50,6 +62,8 @@ func (c *Client) Source() string {
} }
func (c *Client) LoggedIn() bool { func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn return c.loggedIn
} }
@@ -57,12 +71,14 @@ func (c *Client) Login(context.Context) error {
if _, err := exec.LookPath(c.bin); err != nil { if _, err := exec.LookPath(c.bin); err != nil {
return fmt.Errorf("yt-dlp is required for soundcloud downloads/search. install it and ensure it is in $PATH (e.g. pipx install yt-dlp): %w", err) return fmt.Errorf("yt-dlp is required for soundcloud downloads/search. install it and ensure it is in $PATH (e.g. pipx install yt-dlp): %w", err)
} }
c.authMu.Lock()
defer c.authMu.Unlock()
c.loggedIn = true c.loggedIn = true
return nil return nil
} }
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) { func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("soundcloud client not logged in") return nil, errors.New("soundcloud client not logged in")
} }
if limit <= 0 { if limit <= 0 {
@@ -133,6 +149,11 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
} }
req.Header.Set("User-Agent", "Mozilla/5.0") req.Header.Set("User-Agent", "Mozilla/5.0")
if c.limiter != nil {
if err = c.limiter.Wait(ctx); err != nil {
return nil, err
}
}
resp, err := c.http.Do(req) resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -201,7 +222,7 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
} }
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) { func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("soundcloud client not logged in") return nil, errors.New("soundcloud client not logged in")
} }
@@ -273,7 +294,7 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
} }
func (c *Client) GetDownloadable(ctx context.Context, item string, _ int) (*provider.Downloadable, error) { func (c *Client) GetDownloadable(ctx context.Context, item string, _ int) (*provider.Downloadable, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("soundcloud client not logged in") return nil, errors.New("soundcloud client not logged in")
} }
info, err := c.trackInfo(ctx, item) info, err := c.trackInfo(ctx, item)
+16 -6
View File
@@ -14,7 +14,7 @@ import (
) )
func TestGetTrackMetadataAndDownloadable(t *testing.T) { func TestGetTrackMetadataAndDownloadable(t *testing.T) {
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) { c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
@@ -49,7 +49,7 @@ func TestGetTrackMetadataAndDownloadable(t *testing.T) {
} }
func TestGetPlaylistMetadata(t *testing.T) { func TestGetPlaylistMetadata(t *testing.T) {
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) { c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
@@ -81,7 +81,7 @@ func TestGetPlaylistMetadata(t *testing.T) {
} }
func TestSearchTrack(t *testing.T) { func TestSearchTrack(t *testing.T) {
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) { c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
@@ -122,7 +122,7 @@ func TestSearchPlaylist(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.http = ts.Client() c.http = ts.Client()
@@ -167,7 +167,7 @@ func TestSearchPlaylistAcceptsDotsInPath(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true c.loggedIn = true
c.http = ts.Client() c.http = ts.Client()
@@ -203,7 +203,7 @@ func TestSearchPlaylistAcceptsDotsInPath(t *testing.T) {
} }
func TestLoginShowsYtDlpHint(t *testing.T) { func TestLoginShowsYtDlpHint(t *testing.T) {
cfgData := config.DefaultConfigData() cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
c.bin = "definitely-not-a-real-yt-dlp-bin" c.bin = "definitely-not-a-real-yt-dlp-bin"
err := c.Login(context.Background()) err := c.Login(context.Background())
@@ -254,3 +254,13 @@ func TestCanonicalSoundcloudURLAcceptsSubdomain(t *testing.T) {
t.Fatalf("canonical url = %q, want %q", got, "https://soundcloud.com/a/b") t.Fatalf("canonical url = %q, want %q", got, "https://soundcloud.com/a/b")
} }
} }
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+72 -84
View File
@@ -12,6 +12,7 @@ import (
"net/url" "net/url"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"streamrip-go/internal/config" "streamrip-go/internal/config"
@@ -19,6 +20,7 @@ import (
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit" "streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
) )
const ( const (
@@ -26,8 +28,6 @@ const (
lyricsAPIv1 = "https://api.tidal.com/v1" lyricsAPIv1 = "https://api.tidal.com/v1"
openAPIV2 = "https://openapi.tidal.com/v2" openAPIV2 = "https://openapi.tidal.com/v2"
authURL = "https://auth.tidal.com/v1/oauth2" authURL = "https://auth.tidal.com/v1/oauth2"
clientID = "fX2JxdmntZWK0ixT"
clientSec = "1Nm5AfDAjxrgJFJbKNWLeAyKGVGmINuXPPLHVXAvxAg="
tidalRequestAttempts = 3 tidalRequestAttempts = 3
) )
@@ -58,14 +58,17 @@ type Client struct {
baseURL string baseURL string
lyricsAPI string lyricsAPI string
openAPI string openAPI string
loggedIn bool // authMu guards loggedIn and the login/refresh path, which download
// workers can reach concurrently.
authMu sync.Mutex
loggedIn bool
} }
func New(cfg *config.Config) *Client { func New(cfg *config.Config) *Client {
return &Client{ return &Client{
cfg: cfg, cfg: cfg,
http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections), http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute), limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
baseURL: baseURL, baseURL: baseURL,
lyricsAPI: lyricsAPIv1, lyricsAPI: lyricsAPIv1,
openAPI: openAPIV2, openAPI: openAPIV2,
@@ -77,10 +80,14 @@ func (c *Client) Source() string {
} }
func (c *Client) LoggedIn() bool { func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn return c.loggedIn
} }
func (c *Client) Login(ctx context.Context) error { func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
if strings.TrimSpace(c.cfg.Session.Tidal.AccessToken) == "" { if strings.TrimSpace(c.cfg.Session.Tidal.AccessToken) == "" {
return ErrMissingTidalToken return ErrMissingTidalToken
} }
@@ -111,10 +118,10 @@ func (c *Client) Login(ctx context.Context) error {
return fmt.Errorf("tidal login failed: status=%d body=%v", status, resp) return fmt.Errorf("tidal login failed: status=%d body=%v", status, resp)
} }
if v := stringify(resp["countryCode"]); v != "" { if v := jsonutil.StringFromAny(resp["countryCode"]); v != "" {
c.cfg.Session.Tidal.CountryCode = v c.cfg.Session.Tidal.CountryCode = v
} }
if v := stringify(resp["userId"]); v != "" { if v := jsonutil.StringFromAny(resp["userId"]); v != "" {
c.cfg.Session.Tidal.UserID = v c.cfg.Session.Tidal.UserID = v
} }
@@ -137,7 +144,7 @@ func (c *Client) refreshAccessToken(ctx context.Context) error {
} }
form := url.Values{} form := url.Values{}
form.Set("client_id", clientID) form.Set("client_id", c.appClientID())
form.Set("refresh_token", refresh) form.Set("refresh_token", refresh)
form.Set("grant_type", "refresh_token") form.Set("grant_type", "refresh_token")
form.Set("scope", "r_usr+w_usr+w_sub") form.Set("scope", "r_usr+w_usr+w_sub")
@@ -150,33 +157,36 @@ func (c *Client) refreshAccessToken(ctx context.Context) error {
return fmt.Errorf("tidal token refresh failed: status=%d body=%v", status, resp) return fmt.Errorf("tidal token refresh failed: status=%d body=%v", status, resp)
} }
newToken := stringify(resp["access_token"]) newToken := jsonutil.StringFromAny(resp["access_token"])
if newToken == "" { if newToken == "" {
return errors.New("tidal token refresh missing access_token") return errors.New("tidal token refresh missing access_token")
} }
newRefresh := stringify(resp["refresh_token"]) newRefresh := jsonutil.StringFromAny(resp["refresh_token"])
expiresIn := int64(jsonutil.IntFromAny(resp["expires_in"])) expiresIn := int64(jsonutil.IntFromAny(resp["expires_in"]))
if expiresIn <= 0 { if expiresIn <= 0 {
expiresIn = 7 * 24 * 3600 expiresIn = 7 * 24 * 3600
} }
c.cfg.Session.Tidal.AccessToken = newToken
c.cfg.File.Tidal.AccessToken = newToken
if newRefresh != "" {
c.cfg.Session.Tidal.RefreshToken = newRefresh
c.cfg.File.Tidal.RefreshToken = newRefresh
}
expiry := time.Now().Unix() + expiresIn expiry := time.Now().Unix() + expiresIn
c.cfg.Session.Tidal.TokenExpiry = expiry if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
c.cfg.File.Tidal.TokenExpiry = expiry file.Tidal.AccessToken = newToken
_ = c.cfg.SaveFile() session.Tidal.AccessToken = newToken
if newRefresh != "" {
file.Tidal.RefreshToken = newRefresh
session.Tidal.RefreshToken = newRefresh
}
file.Tidal.TokenExpiry = expiry
session.Tidal.TokenExpiry = expiry
}); err != nil {
verbose.Printf(verbose.V, "tidal: could not persist refreshed token: %v\n", err)
}
return nil return nil
} }
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) { func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in") return nil, errors.New("tidal client not logged in")
} }
@@ -251,13 +261,13 @@ func (c *Client) fetchTrackLyrics(ctx context.Context, trackID string) (string,
return "", "" return "", ""
} }
lyrics := strings.TrimSpace(stringify(resp["lyrics"])) lyrics := strings.TrimSpace(jsonutil.StringFromAny(resp["lyrics"]))
lrc := strings.TrimSpace(stringify(resp["subtitles"])) lrc := strings.TrimSpace(jsonutil.StringFromAny(resp["subtitles"]))
return lyrics, lrc return lyrics, lrc
} }
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) { func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in") return nil, errors.New("tidal client not logged in")
} }
if limit <= 0 { if limit <= 0 {
@@ -284,7 +294,7 @@ func (c *Client) Search(ctx context.Context, mediaType, query string, limit int)
} }
func (c *Client) GetDownloadable(ctx context.Context, trackID string, quality int) (*provider.Downloadable, error) { func (c *Client) GetDownloadable(ctx context.Context, trackID string, quality int) (*provider.Downloadable, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in") return nil, errors.New("tidal client not logged in")
} }
if quality < 0 || quality > 4 { if quality < 0 || quality > 4 {
@@ -327,30 +337,6 @@ func (c *Client) GetDownloadable(ctx context.Context, trackID string, quality in
return c.getDownloadableFromTrackManifest(ctx, trackID, quality) return c.getDownloadableFromTrackManifest(ctx, trackID, quality)
} }
func (c *Client) trackSupportsAtmos(ctx context.Context, trackID string) bool {
resp, status, err := c.apiRequest(ctx, "tracks/"+trackID, url.Values{}, c.baseURL)
if err != nil || status != http.StatusOK {
return false
}
if modes, ok := resp["audioModes"].([]any); ok {
for _, mode := range modes {
if strings.Contains(strings.ToUpper(stringify(mode)), "ATMOS") {
return true
}
}
}
if mm, ok := resp["mediaMetadata"].(map[string]any); ok {
if tags, ok := mm["tags"].([]any); ok {
for _, tag := range tags {
if strings.Contains(strings.ToUpper(stringify(tag)), "ATMOS") {
return true
}
}
}
}
return false
}
func (c *Client) getAtmosDownloadable(ctx context.Context, trackID string) (*provider.Downloadable, error) { func (c *Client) getAtmosDownloadable(ctx context.Context, trackID string) (*provider.Downloadable, error) {
var lastErr error var lastErr error
for _, aq := range atmosAudioQualities { for _, aq := range atmosAudioQualities {
@@ -395,18 +381,18 @@ func (c *Client) getAtmosDownloadable(ctx context.Context, trackID string) (*pro
} }
func playbackLooksAtmos(resp map[string]any) bool { func playbackLooksAtmos(resp map[string]any) bool {
if strings.Contains(strings.ToUpper(stringify(resp["audioMode"])), "ATMOS") { if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(resp["audioMode"])), "ATMOS") {
return true return true
} }
if modes, ok := resp["audioModes"].([]any); ok { if modes, ok := resp["audioModes"].([]any); ok {
for _, raw := range modes { for _, raw := range modes {
if strings.Contains(strings.ToUpper(stringify(raw)), "ATMOS") { if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(raw)), "ATMOS") {
return true return true
} }
} }
} }
manifestB64 := stringify(resp["manifest"]) manifestB64 := jsonutil.StringFromAny(resp["manifest"])
if manifestB64 == "" { if manifestB64 == "" {
return false return false
} }
@@ -418,17 +404,17 @@ func playbackLooksAtmos(resp map[string]any) bool {
if err = json.Unmarshal(b, &manifest); err != nil { if err = json.Unmarshal(b, &manifest); err != nil {
return false return false
} }
if strings.Contains(strings.ToUpper(stringify(manifest["audioMode"])), "ATMOS") { if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(manifest["audioMode"])), "ATMOS") {
return true return true
} }
if modes, ok := manifest["audioModes"].([]any); ok { if modes, ok := manifest["audioModes"].([]any); ok {
for _, raw := range modes { for _, raw := range modes {
if strings.Contains(strings.ToUpper(stringify(raw)), "ATMOS") { if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(raw)), "ATMOS") {
return true return true
} }
} }
} }
codec := strings.ToLower(stringify(manifest["codecs"])) codec := strings.ToLower(jsonutil.StringFromAny(manifest["codecs"]))
return strings.Contains(codec, "ec-3") || strings.Contains(codec, "eac3") || strings.Contains(codec, "joc") || strings.Contains(codec, "atmos") return strings.Contains(codec, "ec-3") || strings.Contains(codec, "eac3") || strings.Contains(codec, "joc") || strings.Contains(codec, "atmos")
} }
@@ -512,7 +498,7 @@ func (c *Client) fetchArtistAlbums(ctx context.Context, artistID string) ([]map[
if wrapped, ok := itm["item"].(map[string]any); ok { if wrapped, ok := itm["item"].(map[string]any); ok {
itm = wrapped itm = wrapped
} }
id := stringify(itm["id"]) id := jsonutil.StringFromAny(itm["id"])
if id == "" { if id == "" {
continue continue
} }
@@ -564,14 +550,14 @@ func (c *Client) getDownloadableFromTrackManifestForFormats(ctx context.Context,
if !ok { if !ok {
return nil, errors.New("tidal trackManifests missing attributes") return nil, errors.New("tidal trackManifests missing attributes")
} }
uri := stringify(attrs["uri"]) uri := jsonutil.StringFromAny(attrs["uri"])
if uri == "" { if uri == "" {
return nil, errors.New("tidal trackManifests missing uri") return nil, errors.New("tidal trackManifests missing uri")
} }
attrFormats, _ := attrs["formats"].([]any) attrFormats, _ := attrs["formats"].([]any)
ext := "m4a" ext := "m4a"
for _, f := range attrFormats { for _, f := range attrFormats {
fv := strings.ToUpper(stringify(f)) fv := strings.ToUpper(jsonutil.StringFromAny(f))
if strings.Contains(fv, "FLAC") { if strings.Contains(fv, "FLAC") {
ext = "flac" ext = "flac"
break break
@@ -601,7 +587,7 @@ func formatsForQuality(quality int, preferAtmos bool) []string {
} }
func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*provider.Downloadable, error) { func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*provider.Downloadable, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in") return nil, errors.New("tidal client not logged in")
} }
@@ -618,7 +604,7 @@ func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*pro
return nil, fmt.Errorf("tidal video playbackinfo failed: status=%d", status) return nil, fmt.Errorf("tidal video playbackinfo failed: status=%d", status)
} }
manifestB64 := stringify(resp["manifest"]) manifestB64 := jsonutil.StringFromAny(resp["manifest"])
if manifestB64 == "" { if manifestB64 == "" {
return nil, errors.New("tidal video manifest missing") return nil, errors.New("tidal video manifest missing")
} }
@@ -634,7 +620,7 @@ func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*pro
if !ok || len(urls) == 0 { if !ok || len(urls) == 0 {
return nil, errors.New("tidal video manifest urls missing") return nil, errors.New("tidal video manifest urls missing")
} }
masterURL := stringify(urls[0]) masterURL := jsonutil.StringFromAny(urls[0])
if masterURL == "" { if masterURL == "" {
return nil, errors.New("tidal video master url missing") return nil, errors.New("tidal video master url missing")
} }
@@ -665,7 +651,7 @@ func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*pro
} }
func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadable { func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadable {
manifestB64 := stringify(resp["manifest"]) manifestB64 := jsonutil.StringFromAny(resp["manifest"])
if manifestB64 == "" { if manifestB64 == "" {
return nil return nil
} }
@@ -681,11 +667,11 @@ func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadabl
if !ok || len(urls) == 0 { if !ok || len(urls) == 0 {
return nil return nil
} }
streamURL := stringify(urls[0]) streamURL := jsonutil.StringFromAny(urls[0])
if streamURL == "" { if streamURL == "" {
return nil return nil
} }
codec := strings.ToLower(stringify(manifest["codecs"])) codec := strings.ToLower(jsonutil.StringFromAny(manifest["codecs"]))
ext := "m4a" ext := "m4a"
if strings.Contains(codec, "flac") { if strings.Contains(codec, "flac") {
ext = "flac" ext = "flac"
@@ -696,14 +682,14 @@ func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadabl
if profile.Container == "" { if profile.Container == "" {
profile = tidalAudioProfileFromExtension(ext) profile = tidalAudioProfileFromExtension(ext)
} }
audioQuality := strings.ToUpper(strings.TrimSpace(stringify(resp["audioQuality"]))) audioQuality := strings.ToUpper(strings.TrimSpace(jsonutil.StringFromAny(resp["audioQuality"])))
if audioQuality == "" { if audioQuality == "" {
audioQuality = strings.ToUpper(strings.TrimSpace(stringify(manifest["audioQuality"]))) audioQuality = strings.ToUpper(strings.TrimSpace(jsonutil.StringFromAny(manifest["audioQuality"])))
} }
if audioQuality != "" { if audioQuality != "" {
profile = applyTidalAudioQuality(profile, audioQuality) profile = applyTidalAudioQuality(profile, audioQuality)
} }
if strings.Contains(strings.ToUpper(stringify(resp["audioMode"])), "ATMOS") { if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(resp["audioMode"])), "ATMOS") {
profile = tidalAtmosAudioProfile() profile = tidalAtmosAudioProfile()
} }
return &provider.Downloadable{URL: streamURL, Extension: ext, Source: "tidal", Audio: profile} return &provider.Downloadable{URL: streamURL, Extension: ext, Source: "tidal", Audio: profile}
@@ -712,7 +698,7 @@ func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadabl
func tidalAudioProfileFromFormats(formats []any) provider.AudioProfile { func tidalAudioProfileFromFormats(formats []any) provider.AudioProfile {
best := provider.AudioProfile{} best := provider.AudioProfile{}
for _, raw := range formats { for _, raw := range formats {
f := strings.ToUpper(strings.TrimSpace(stringify(raw))) f := strings.ToUpper(strings.TrimSpace(jsonutil.StringFromAny(raw)))
switch { switch {
case strings.Contains(f, "EAC3") || strings.Contains(f, "JOC") || strings.Contains(f, "ATMOS"): case strings.Contains(f, "EAC3") || strings.Contains(f, "JOC") || strings.Contains(f, "ATMOS"):
return tidalAtmosAudioProfile() return tidalAtmosAudioProfile()
@@ -984,35 +970,20 @@ func (c *Client) apiPost(ctx context.Context, endpoint string, form url.Values,
req.Header.Set("Content-Type", "application/x-www-form-urlencoded") req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("User-Agent", "streamrip-go/0.1") req.Header.Set("User-Agent", "streamrip-go/0.1")
if basicAuth { if basicAuth {
auth := base64.StdEncoding.EncodeToString([]byte(clientID + ":" + clientSec)) auth := base64.StdEncoding.EncodeToString([]byte(c.appClientID() + ":" + c.appClientSecret()))
req.Header.Set("Authorization", "Basic "+auth) req.Header.Set("Authorization", "Basic "+auth)
} }
return req, nil return req, nil
}) })
} }
func stringify(v any) string {
switch t := v.(type) {
case string:
return t
case int:
return strconv.Itoa(t)
case int64:
return strconv.FormatInt(t, 10)
case float64:
return strconv.FormatFloat(t, 'f', -1, 64)
default:
return ""
}
}
func enrichTidalImage(meta map[string]any) { func enrichTidalImage(meta map[string]any) {
if _, ok := meta["image"].(map[string]any); ok { if _, ok := meta["image"].(map[string]any); ok {
return return
} }
cover := stringify(meta["cover"]) cover := jsonutil.StringFromAny(meta["cover"])
if cover == "" { if cover == "" {
cover = stringify(meta["squareImage"]) cover = jsonutil.StringFromAny(meta["squareImage"])
} }
if cover == "" { if cover == "" {
return return
@@ -1031,3 +1002,20 @@ func tidalImageMap(cover string) map[string]any {
"original": base + "/1280x1280.jpg", "original": base + "/1280x1280.jpg",
} }
} }
// appClientID and appClientSecret read the vendor OAuth app credentials from
// config, falling back to the built-in defaults. Config is the source of truth
// so a rotation can be fixed without a rebuild.
func (c *Client) appClientID() string {
if v := strings.TrimSpace(c.cfg.Session.Tidal.AppClientID); v != "" {
return v
}
return config.DefaultTidalAppClientID
}
func (c *Client) appClientSecret() string {
if v := strings.TrimSpace(c.cfg.Session.Tidal.AppClientSecret); v != "" {
return v
}
return config.DefaultTidalAppClientSecret
}
+27 -38
View File
@@ -12,10 +12,11 @@ import (
"testing" "testing"
"streamrip-go/internal/config" "streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
) )
func TestLoginMissingToken(t *testing.T) { func TestLoginMissingToken(t *testing.T) {
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "" cfgData.Tidal.AccessToken = ""
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
err := c.Login(context.Background()) err := c.Login(context.Background())
@@ -37,7 +38,7 @@ func TestSearch(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -73,7 +74,7 @@ func TestGetVideoDownloadable(t *testing.T) {
})) }))
defer server.Close() defer server.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -143,7 +144,7 @@ func TestGetMetadataArtistPaginatesAlbums(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -185,7 +186,7 @@ func TestGetMetadataTrackAddsLyricsAndSyncedLyrics(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "MY" cfgData.Tidal.CountryCode = "MY"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -197,10 +198,10 @@ func TestGetMetadataTrackAddsLyricsAndSyncedLyrics(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetMetadata() err = %v", err) t.Fatalf("GetMetadata() err = %v", err)
} }
if got := stringify(meta["lyrics"]); got != "plain lyrics line" { if got := jsonutil.StringFromAny(meta["lyrics"]); got != "plain lyrics line" {
t.Fatalf("lyrics = %q, want plain lyrics line", got) t.Fatalf("lyrics = %q, want plain lyrics line", got)
} }
if got := stringify(meta["lyrics_synced"]); got != "[00:00.00]plain lyrics line" { if got := jsonutil.StringFromAny(meta["lyrics_synced"]); got != "[00:00.00]plain lyrics line" {
t.Fatalf("lyrics_synced = %q, want synced lrc", got) t.Fatalf("lyrics_synced = %q, want synced lrc", got)
} }
} }
@@ -219,7 +220,7 @@ func TestGetMetadataTrackIgnoresLyricsEndpointFailure(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -257,7 +258,7 @@ func TestAPIRequestRetriesTooManyRequests(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Downloads.RequestsPerMinute = 0 cfgData.Downloads.RequestsPerMinute = 0
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
@@ -274,8 +275,8 @@ func TestAPIRequestRetriesTooManyRequests(t *testing.T) {
if calls != 2 { if calls != 2 {
t.Fatalf("calls = %d, want 2", calls) t.Fatalf("calls = %d, want 2", calls)
} }
if stringify(resp["title"]) != "Song" { if jsonutil.StringFromAny(resp["title"]) != "Song" {
t.Fatalf("title = %q, want Song", stringify(resp["title"])) t.Fatalf("title = %q, want Song", jsonutil.StringFromAny(resp["title"]))
} }
} }
@@ -297,7 +298,7 @@ func TestAPIPostRetriesTooManyRequests(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Downloads.RequestsPerMinute = 0 cfgData.Downloads.RequestsPerMinute = 0
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -311,8 +312,8 @@ func TestAPIPostRetriesTooManyRequests(t *testing.T) {
if calls != 2 { if calls != 2 {
t.Fatalf("calls = %d, want 2", calls) t.Fatalf("calls = %d, want 2", calls)
} }
if stringify(resp["access_token"]) != "fresh-token" { if jsonutil.StringFromAny(resp["access_token"]) != "fresh-token" {
t.Fatalf("access_token = %q, want fresh-token", stringify(resp["access_token"])) t.Fatalf("access_token = %q, want fresh-token", jsonutil.StringFromAny(resp["access_token"]))
} }
} }
@@ -342,7 +343,7 @@ func TestGetDownloadablePrefersAtmosWhenEnabled(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
cfgData.Tidal.PreferAtmos = true cfgData.Tidal.PreferAtmos = true
@@ -377,28 +378,6 @@ func TestPlaybackLooksAtmosFromManifest(t *testing.T) {
} }
} }
func TestTrackSupportsAtmosFromTags(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/tracks/42" {
w.WriteHeader(http.StatusNotFound)
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "audioModes": []any{"STEREO"}, "mediaMetadata": map[string]any{"tags": []any{"LOSSLESS", "DOLBY_ATMOS"}}})
}))
defer ts.Close()
cfgData := config.DefaultConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.baseURL = ts.URL + "/v1"
if !c.trackSupportsAtmos(context.Background(), "42") {
t.Fatalf("expected atmos support from mediaMetadata tags")
}
}
func TestFormatsForQuality(t *testing.T) { func TestFormatsForQuality(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
@@ -448,7 +427,7 @@ func TestGetDownloadableLosslessUsesTrackManifestWhenPlaybackIsAAC(t *testing.T)
})) }))
defer ts.Close() defer ts.Close()
cfgData := config.DefaultConfigData() cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token" cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US" cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData}) c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -473,3 +452,13 @@ func TestGetDownloadableLosslessUsesTrackManifestWhenPlaybackIsAAC(t *testing.T)
t.Fatalf("unexpected audio profile: %+v", d.Audio) t.Fatalf("unexpected audio profile: %+v", d.Audio)
} }
} }
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+59 -71
View File
@@ -18,6 +18,7 @@ import (
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"streamrip-go/internal/config" "streamrip-go/internal/config"
@@ -25,25 +26,26 @@ import (
"streamrip-go/internal/netutil" "streamrip-go/internal/netutil"
"streamrip-go/internal/provider" "streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit" "streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
) )
const ( const (
baseURL = "https://api.music.yandex.net" baseURL = "https://api.music.yandex.net"
desktopClientHeader = "YandexMusicDesktopAppWindows/5.13.2" desktopClientHeader = "YandexMusicDesktopAppWindows/5.13.2"
desktopOrigin = "music-application://desktop" desktopOrigin = "music-application://desktop"
requestAttempts = 3 requestAttempts = 3
desktopWindowsSignKey = "kzqU4XhfCaY6B6JTHODeq5"
legacyMP3SignSalt = "XGRlBW9FXlekgbPrRHuSiA"
defaultEstimatedKbps = 50000
) )
var ErrMissingYandexToken = errors.New("missing yandex access_token") var ErrMissingYandexToken = errors.New("missing yandex access_token")
type Client struct { type Client struct {
cfg *config.Config cfg *config.Config
http *http.Client http *http.Client
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
baseURL string baseURL string
// authMu guards loggedIn and the login/refresh path, which download
// workers can reach concurrently.
authMu sync.Mutex
loggedIn bool loggedIn bool
userID string userID string
} }
@@ -52,7 +54,7 @@ func New(cfg *config.Config) *Client {
return &Client{ return &Client{
cfg: cfg, cfg: cfg,
http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections), http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute), limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
baseURL: baseURL, baseURL: baseURL,
userID: strings.TrimSpace(cfg.Session.Yandex.UserID), userID: strings.TrimSpace(cfg.Session.Yandex.UserID),
} }
@@ -63,10 +65,14 @@ func (c *Client) Source() string {
} }
func (c *Client) LoggedIn() bool { func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn return c.loggedIn
} }
func (c *Client) Login(ctx context.Context) error { func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
if strings.TrimSpace(c.cfg.Session.Yandex.AccessToken) == "" { if strings.TrimSpace(c.cfg.Session.Yandex.AccessToken) == "" {
return ErrMissingYandexToken return ErrMissingYandexToken
} }
@@ -80,16 +86,19 @@ func (c *Client) Login(ctx context.Context) error {
result, _ := resp["result"].(map[string]any) result, _ := resp["result"].(map[string]any)
if uid := strings.TrimSpace(jsonutil.StringFromAny(result["uid"])); uid != "" { if uid := strings.TrimSpace(jsonutil.StringFromAny(result["uid"])); uid != "" {
c.userID = uid c.userID = uid
c.cfg.Session.Yandex.UserID = uid if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
c.cfg.File.Yandex.UserID = uid file.Yandex.UserID = uid
_ = c.cfg.SaveFile() session.Yandex.UserID = uid
}); err != nil {
verbose.Printf(verbose.V, "yandex: could not persist user id: %v\n", err)
}
} }
c.loggedIn = true c.loggedIn = true
return nil return nil
} }
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) { func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("yandex client not logged in") return nil, errors.New("yandex client not logged in")
} }
switch mediaType { switch mediaType {
@@ -107,7 +116,7 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
} }
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) { func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("yandex client not logged in") return nil, errors.New("yandex client not logged in")
} }
if limit <= 0 { if limit <= 0 {
@@ -137,7 +146,7 @@ func (c *Client) Search(ctx context.Context, mediaType, query string, limit int)
} }
func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) { func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) {
if !c.loggedIn { if !c.LoggedIn() {
return nil, errors.New("yandex client not logged in") return nil, errors.New("yandex client not logged in")
} }
trackID, _ := splitTrackRef(item) trackID, _ := splitTrackRef(item)
@@ -168,7 +177,7 @@ func (c *Client) Close() error {
} }
func (c *Client) getTrackMetadata(ctx context.Context, item string) (map[string]any, error) { func (c *Client) getTrackMetadata(ctx context.Context, item string) (map[string]any, error) {
trackRef := canonicalTrackRequestID(item) trackRef := strings.TrimSpace(item)
form := url.Values{} form := url.Values{}
form.Set("trackIds", trackRef) form.Set("trackIds", trackRef)
form.Set("withMixData", "true") form.Set("withMixData", "true")
@@ -334,7 +343,7 @@ func (c *Client) getLegacyDownloadable(ctx context.Context, trackID string) (*pr
if err != nil { if err != nil {
return nil, err return nil, err
} }
directURL, err := legacyDirectURL(xmlInfo) directURL, err := legacyDirectURL(c.legacyMP3Salt(), xmlInfo)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -355,7 +364,7 @@ func (c *Client) getLegacyDownloadable(ctx context.Context, trackID string) (*pr
func (c *Client) getDesktopDownloadable(ctx context.Context, trackID string, quality int, transport string) (*provider.Downloadable, error) { func (c *Client) getDesktopDownloadable(ctx context.Context, trackID string, quality int, transport string) (*provider.Downloadable, error) {
qualityParam, codecs := downloadRequestProfile(quality) qualityParam, codecs := downloadRequestProfile(quality)
sign, ts := yandexDownloadSign(trackID, qualityParam, codecs, transport) sign, ts := yandexDownloadSign(c.signKey(), trackID, qualityParam, codecs, transport)
params := url.Values{} params := url.Values{}
params.Set("trackId", trackID) params.Set("trackId", trackID)
params.Set("codecs", strings.Join(codecs, ",")) params.Set("codecs", strings.Join(codecs, ","))
@@ -421,11 +430,11 @@ func (c *Client) fetchLegacyXML(ctx context.Context, xmlURL string) (*legacyDown
return &out, nil return &out, nil
} }
func legacyDirectURL(info *legacyDownloadInfoXML) (string, error) { func legacyDirectURL(salt string, info *legacyDownloadInfoXML) (string, error) {
if info == nil || info.Host == "" || info.Path == "" || info.TS == "" || info.S == "" { if info == nil || info.Host == "" || info.Path == "" || info.TS == "" || info.S == "" {
return "", errors.New("invalid yandex legacy xml payload") return "", errors.New("invalid yandex legacy xml payload")
} }
sum := md5.Sum([]byte(legacyMP3SignSalt + strings.TrimPrefix(info.Path, "/") + info.S)) sum := md5.Sum([]byte(salt + strings.TrimPrefix(info.Path, "/") + info.S))
return "https://" + info.Host + "/get-mp3/" + hex.EncodeToString(sum[:]) + "/" + info.TS + info.Path, nil return "https://" + info.Host + "/get-mp3/" + hex.EncodeToString(sum[:]) + "/" + info.TS + info.Path, nil
} }
@@ -565,14 +574,14 @@ func normalizeTrack(raw map[string]any, fallbackID string, artistSeparator strin
trackID := canonicalTrackRefFromRaw(raw, fallbackID) trackID := canonicalTrackRefFromRaw(raw, fallbackID)
albumRaw := firstAlbum(raw) albumRaw := firstAlbum(raw)
artistNames := artistNames(raw) artistNames := artistNames(raw)
artistName := displayArtistNames(artistNames) artistName := jsonutil.DisplayArtistNames(artistNames)
albumArtist := joinAlbumArtists(albumRaw, artistSeparator) albumArtist := joinAlbumArtists(albumRaw, artistSeparator)
if albumArtist == "" { if albumArtist == "" {
albumArtist = artistName albumArtist = artistName
} }
trackNumber, discNumber := trackNumbers(albumRaw) trackNumber, discNumber := trackNumbers(albumRaw)
trackTotal := jsonutil.IntFromAny(albumRaw["trackCount"]) trackTotal := jsonutil.IntFromAny(albumRaw["trackCount"])
discTotal := firstPositiveInt( discTotal := jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(raw["volumesCount"]), jsonutil.IntFromAny(raw["volumesCount"]),
jsonutil.IntFromAny(albumRaw["volumesCount"]), jsonutil.IntFromAny(albumRaw["volumesCount"]),
) )
@@ -584,7 +593,7 @@ func normalizeTrack(raw map[string]any, fallbackID string, artistSeparator strin
"tracks_count": trackTotal, "tracks_count": trackTotal,
"media_number": discNumber, "media_number": discNumber,
"numberOfVolumes": discTotal, "numberOfVolumes": discTotal,
"release_date": firstNonEmpty(jsonutil.StringFromAny(raw["releaseDate"]), jsonutil.StringFromAny(albumRaw["releaseDate"])), "release_date": jsonutil.FirstNonEmpty(jsonutil.StringFromAny(raw["releaseDate"]), jsonutil.StringFromAny(albumRaw["releaseDate"])),
"parental_warning": isExplicit(raw), "parental_warning": isExplicit(raw),
"explicit": isExplicit(raw), "explicit": isExplicit(raw),
"source_track_id": jsonutil.StringFromAny(raw["realId"]), "source_track_id": jsonutil.StringFromAny(raw["realId"]),
@@ -672,9 +681,9 @@ func normalizePlaylist(raw map[string]any) map[string]any {
} }
} }
owner, _ := raw["owner"].(map[string]any) owner, _ := raw["owner"].(map[string]any)
ownerName := firstNonEmpty(jsonutil.StringFromAny(owner["name"]), jsonutil.StringFromAny(owner["login"])) ownerName := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(owner["name"]), jsonutil.StringFromAny(owner["login"]))
return map[string]any{ return map[string]any{
"id": firstNonEmpty(jsonutil.StringFromAny(raw["playlistUuid"]), jsonutil.StringFromAny(raw["kind"])), "id": jsonutil.FirstNonEmpty(jsonutil.StringFromAny(raw["playlistUuid"]), jsonutil.StringFromAny(raw["kind"])),
"name": jsonutil.StringFromAny(raw["title"]), "name": jsonutil.StringFromAny(raw["title"]),
"title": jsonutil.StringFromAny(raw["title"]), "title": jsonutil.StringFromAny(raw["title"]),
"description": jsonutil.StringFromAny(raw["description"]), "description": jsonutil.StringFromAny(raw["description"]),
@@ -717,7 +726,7 @@ func normalizeArtistSearchItem(raw map[string]any) map[string]any {
"id": jsonutil.StringFromAny(raw["id"]), "id": jsonutil.StringFromAny(raw["id"]),
"title": jsonutil.StringFromAny(raw["name"]), "title": jsonutil.StringFromAny(raw["name"]),
"name": jsonutil.StringFromAny(raw["name"]), "name": jsonutil.StringFromAny(raw["name"]),
"albums_count": firstPositiveInt( "albums_count": jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(jsonutil.NestedAny(raw, "counts", "directAlbums")), jsonutil.IntFromAny(jsonutil.NestedAny(raw, "counts", "directAlbums")),
jsonutil.IntFromAny(jsonutil.NestedAny(raw, "counts", "tracks")), jsonutil.IntFromAny(jsonutil.NestedAny(raw, "counts", "tracks")),
), ),
@@ -726,8 +735,8 @@ func normalizeArtistSearchItem(raw map[string]any) map[string]any {
func normalizePlaylistSearchItem(raw map[string]any) map[string]any { func normalizePlaylistSearchItem(raw map[string]any) map[string]any {
owner, _ := raw["owner"].(map[string]any) owner, _ := raw["owner"].(map[string]any)
ownerID := firstNonEmpty(jsonutil.StringFromAny(owner["uid"]), jsonutil.StringFromAny(raw["uid"])) ownerID := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(owner["uid"]), jsonutil.StringFromAny(raw["uid"]))
ownerName := firstNonEmpty(jsonutil.StringFromAny(owner["name"]), jsonutil.StringFromAny(owner["login"])) ownerName := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(owner["name"]), jsonutil.StringFromAny(owner["login"]))
return map[string]any{ return map[string]any{
"id": ownerID + ":" + jsonutil.StringFromAny(raw["kind"]), "id": ownerID + ":" + jsonutil.StringFromAny(raw["kind"]),
"title": jsonutil.StringFromAny(raw["title"]), "title": jsonutil.StringFromAny(raw["title"]),
@@ -771,7 +780,7 @@ func firstArtistID(raw map[string]any) string {
} }
func joinArtists(raw map[string]any, artistSeparator string) string { func joinArtists(raw map[string]any, artistSeparator string) string {
return displayArtistNames(artistNames(raw)) return jsonutil.DisplayArtistNames(artistNames(raw))
} }
func artistNames(raw map[string]any) []string { func artistNames(raw map[string]any) []string {
@@ -793,7 +802,7 @@ func joinAlbumArtists(albumRaw map[string]any, artistSeparator string) string {
if len(albumRaw) == 0 { if len(albumRaw) == 0 {
return "" return ""
} }
return displayArtistNames(albumArtistNames(albumRaw)) return jsonutil.DisplayArtistNames(albumArtistNames(albumRaw))
} }
func albumArtistNames(albumRaw map[string]any) []string { func albumArtistNames(albumRaw map[string]any) []string {
@@ -811,19 +820,6 @@ func albumArtistNames(albumRaw map[string]any) []string {
return parts return parts
} }
func displayArtistNames(names []string) string {
switch len(names) {
case 0:
return ""
case 1:
return names[0]
case 2:
return names[0] + " & " + names[1]
default:
return strings.Join(names[:len(names)-1], ", ") + " & " + names[len(names)-1]
}
}
func (c *Client) artistSeparator() string { func (c *Client) artistSeparator() string {
if c.cfg.Session.Metadata.ArtistSeparator == "" { if c.cfg.Session.Metadata.ArtistSeparator == "" {
return "; " return "; "
@@ -837,22 +833,14 @@ func trackNumbers(albumRaw map[string]any) (int, int) {
if trackNumber > 0 { if trackNumber > 0 {
trackNumber++ trackNumber++
} }
return trackNumber, firstPositiveInt(jsonutil.IntFromAny(position["volume"]), 1) return trackNumber, jsonutil.FirstPositiveInt(jsonutil.IntFromAny(position["volume"]), 1)
}
func canonicalTrackRequestID(item string) string {
item = strings.TrimSpace(item)
if item == "" {
return item
}
return item
} }
func canonicalTrackRefFromRaw(raw map[string]any, fallback string) string { func canonicalTrackRefFromRaw(raw map[string]any, fallback string) string {
if strings.Contains(strings.TrimSpace(fallback), ":") { if strings.Contains(strings.TrimSpace(fallback), ":") {
return strings.TrimSpace(fallback) return strings.TrimSpace(fallback)
} }
trackID := firstNonEmpty(jsonutil.StringFromAny(raw["realId"]), jsonutil.StringFromAny(raw["id"])) trackID := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(raw["realId"]), jsonutil.StringFromAny(raw["id"]))
albumID := jsonutil.StringFromAny(firstAlbum(raw)["id"]) albumID := jsonutil.StringFromAny(firstAlbum(raw)["id"])
if trackID == "" { if trackID == "" {
trackID = strings.TrimSpace(fallback) trackID = strings.TrimSpace(fallback)
@@ -919,7 +907,7 @@ func imageMapFromTrack(raw map[string]any, albumRaw map[string]any) map[string]a
} }
func imageMapFromURI(raw map[string]any) map[string]any { func imageMapFromURI(raw map[string]any) map[string]any {
uri := firstNonEmpty(jsonutil.StringFromAny(raw["ogImage"]), jsonutil.StringFromAny(raw["coverUri"])) uri := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(raw["ogImage"]), jsonutil.StringFromAny(raw["coverUri"]))
if uri == "" { if uri == "" {
if cover, ok := raw["cover"].(map[string]any); ok { if cover, ok := raw["cover"].(map[string]any); ok {
uri = jsonutil.StringFromAny(cover["uri"]) uri = jsonutil.StringFromAny(cover["uri"])
@@ -954,9 +942,11 @@ func isExplicit(raw map[string]any) bool {
return jsonutil.BoolFromAny(raw["explicit"]) return jsonutil.BoolFromAny(raw["explicit"])
} }
func yandexDownloadSign(trackID, quality string, codecs []string, transport string) (string, int64) { // yandexDownloadSign signs a download request. signKey comes from config
// (Client.signKey) so a rotation can be fixed without a rebuild.
func yandexDownloadSign(signKey, trackID, quality string, codecs []string, transport string) (string, int64) {
ts := time.Now().Unix() ts := time.Now().Unix()
mac := hmac.New(sha256.New, []byte(desktopWindowsSignKey)) mac := hmac.New(sha256.New, []byte(signKey))
_, _ = mac.Write([]byte(strconv.FormatInt(ts, 10) + trackID + quality + strings.Join(codecs, "") + transport)) _, _ = mac.Write([]byte(strconv.FormatInt(ts, 10) + trackID + quality + strings.Join(codecs, "") + transport))
return strings.TrimRight(base64.StdEncoding.EncodeToString(mac.Sum(nil)), "="), ts return strings.TrimRight(base64.StdEncoding.EncodeToString(mac.Sum(nil)), "="), ts
} }
@@ -1029,20 +1019,18 @@ func waitRetry(ctx context.Context, delay time.Duration) error {
} }
} }
func firstPositiveInt(vals ...int) int { // signKey and legacyMP3Salt read the vendor request-signing keys from config,
for _, v := range vals { // falling back to the built-in defaults.
if v > 0 { func (c *Client) signKey() string {
return v if v := strings.TrimSpace(c.cfg.Session.Yandex.SignKey); v != "" {
} return v
} }
return 0 return config.DefaultYandexSignKey
} }
func firstNonEmpty(vals ...string) string { func (c *Client) legacyMP3Salt() string {
for _, v := range vals { if v := strings.TrimSpace(c.cfg.Session.Yandex.LegacyMP3SignSalt); v != "" {
if strings.TrimSpace(v) != "" { return v
return strings.TrimSpace(v)
}
} }
return "" return config.DefaultYandexLegacyMP3Salt
} }
+14 -11
View File
@@ -15,7 +15,7 @@ import (
) )
func TestYandexDownloadSignMatchesCapturedFormat(t *testing.T) { func TestYandexDownloadSignMatchesCapturedFormat(t *testing.T) {
sign, ts := yandexDownloadSign("32038184", "lossless", []string{"flac", "aac", "he-aac", "mp3", "flac-mp4", "aac-mp4", "he-aac-mp4"}, "raw") sign, ts := yandexDownloadSign(config.DefaultYandexSignKey, "32038184", "lossless", []string{"flac", "aac", "he-aac", "mp3", "flac-mp4", "aac-mp4", "he-aac-mp4"}, "raw")
if ts <= 0 { if ts <= 0 {
t.Fatalf("timestamp = %d", ts) t.Fatalf("timestamp = %d", ts)
} }
@@ -60,7 +60,7 @@ func TestGetDownloadableUsesModernGetFileInfo(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
d := config.DefaultConfigData() d := testConfigData()
d.Downloads.RequestsPerMinute = 0 d.Downloads.RequestsPerMinute = 0
d.Yandex.AccessToken = "token" d.Yandex.AccessToken = "token"
c := New(&config.Config{File: d, Session: d}) c := New(&config.Config{File: d, Session: d})
@@ -132,7 +132,7 @@ func TestGetMetadataTrackUsesModernTracksEndpoint(t *testing.T) {
})) }))
defer ts.Close() defer ts.Close()
d := config.DefaultConfigData() d := testConfigData()
d.Downloads.RequestsPerMinute = 0 d.Downloads.RequestsPerMinute = 0
d.Yandex.AccessToken = "token" d.Yandex.AccessToken = "token"
c := New(&config.Config{File: d, Session: d}) c := New(&config.Config{File: d, Session: d})
@@ -167,15 +167,8 @@ func TestGetMetadataTrackUsesModernTracksEndpoint(t *testing.T) {
} }
} }
func TestDisplayArtistNamesUsesCreditPunctuation(t *testing.T) {
got := displayArtistNames([]string{"A", "B", "C"})
if got != "A, B & C" {
t.Fatalf("displayArtistNames() = %q", got)
}
}
func TestLegacyDirectURLBuildsPlayableMP3URL(t *testing.T) { func TestLegacyDirectURLBuildsPlayableMP3URL(t *testing.T) {
url, err := legacyDirectURL(&legacyDownloadInfoXML{ url, err := legacyDirectURL(config.DefaultYandexLegacyMP3Salt, &legacyDownloadInfoXML{
Host: "example.test", Host: "example.test",
Path: "/abc123", Path: "/abc123",
TS: "1234567890", TS: "1234567890",
@@ -189,3 +182,13 @@ func TestLegacyDirectURLBuildsPlayableMP3URL(t *testing.T) {
t.Fatalf("legacyDirectURL() = %q, want %q", url, want) t.Fatalf("legacyDirectURL() = %q, want %q", url, want)
} }
} }
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+49 -14
View File
@@ -1,3 +1,5 @@
// Package ratelimit provides the API request throttle shared by the provider
// clients.
package ratelimit package ratelimit
import ( import (
@@ -6,20 +8,54 @@ import (
"time" "time"
) )
// Limiter is a token bucket: it enforces a sustained rate of
// requestsPerMinute while allowing up to burst requests to proceed
// immediately when budget has accrued.
//
// It used to space every request exactly interval apart, with no burst. That
// made max_connections meaningless for anything that goes through the API:
// with the default 60 requests/minute, six download workers each needing a
// metadata call and a stream-URL call could only start one track per two
// seconds, so an album appeared to download serially even though the transfers
// themselves were fast.
type Limiter struct { type Limiter struct {
interval time.Duration interval time.Duration
burst int
mu sync.Mutex mu sync.Mutex
// next is the time the next token becomes available. It is allowed to
// trail now by up to (burst-1)*interval, which is what accrues burst
// budget while no requests are being made.
next time.Time next time.Time
} }
func New(requestsPerMinute int) *Limiter { // New returns a limiter allowing requestsPerMinute sustained, with burst
// requests permitted back-to-back. A non-positive requestsPerMinute disables
// throttling entirely.
//
// burst is clamped to at least 1 and to at most one minute's worth of
// requests, so a very low rate cannot be bypassed by a large burst.
func New(requestsPerMinute int, burst int) *Limiter {
if requestsPerMinute <= 0 { if requestsPerMinute <= 0 {
return &Limiter{} return &Limiter{}
} }
return &Limiter{interval: time.Minute / time.Duration(requestsPerMinute)} if burst < 1 {
burst = 1
}
if burst > requestsPerMinute {
burst = requestsPerMinute
}
return &Limiter{
interval: time.Minute / time.Duration(requestsPerMinute),
burst: burst,
}
} }
// Wait blocks until the caller may issue a request, or until ctx is done.
//
// Budget is reserved at call time, so a cancelled Wait still consumes its
// slot. That keeps the sustained rate honest under cancellation, at the cost
// of briefly under-using the budget.
func (l *Limiter) Wait(ctx context.Context) error { func (l *Limiter) Wait(ctx context.Context) error {
if l.interval <= 0 { if l.interval <= 0 {
return nil return nil
@@ -27,22 +63,21 @@ func (l *Limiter) Wait(ctx context.Context) error {
l.mu.Lock() l.mu.Lock()
now := time.Now() now := time.Now()
if l.next.IsZero() { // Accrue budget for idle time, capped at burst tokens.
l.next = now.Add(l.interval) earliest := now.Add(-time.Duration(l.burst-1) * l.interval)
l.mu.Unlock() if l.next.Before(earliest) {
return nil l.next = earliest
} }
wake := l.next wake := l.next
if now.After(wake) {
l.next = now.Add(l.interval)
l.mu.Unlock()
return nil
}
l.next = l.next.Add(l.interval) l.next = l.next.Add(l.interval)
l.mu.Unlock() l.mu.Unlock()
timer := time.NewTimer(time.Until(wake)) delay := time.Until(wake)
if delay <= 0 {
return nil
}
timer := time.NewTimer(delay)
defer timer.Stop() defer timer.Stop()
select { select {
+92 -4
View File
@@ -2,19 +2,25 @@ package ratelimit
import ( import (
"context" "context"
"sync"
"testing" "testing"
"time" "time"
) )
func TestLimiterDisabled(t *testing.T) { func TestLimiterDisabled(t *testing.T) {
l := New(-1) for _, rpm := range []int{-1, 0} {
if err := l.Wait(context.Background()); err != nil { l := New(rpm, 6)
t.Fatalf("Wait() error = %v", err) for i := 0; i < 10; i++ {
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("New(%d).Wait() error = %v", rpm, err)
}
}
} }
} }
func TestLimiterContextCancel(t *testing.T) { func TestLimiterContextCancel(t *testing.T) {
l := New(1) // burst 1 means strict spacing, so the second request must block.
l := New(1, 1)
if err := l.Wait(context.Background()); err != nil { if err := l.Wait(context.Background()); err != nil {
t.Fatalf("first Wait() error = %v", err) t.Fatalf("first Wait() error = %v", err)
} }
@@ -26,3 +32,85 @@ func TestLimiterContextCancel(t *testing.T) {
t.Fatalf("expected context error") t.Fatalf("expected context error")
} }
} }
// TestLimiterAllowsBurst is the regression test for album rips appearing
// serial: with the old strict-spacing limiter, max_connections workers could
// not get their API calls through concurrently, so only one track downloaded
// at a time.
func TestLimiterAllowsBurst(t *testing.T) {
const burst = 6
l := New(60, burst) // the shipped defaults
start := time.Now()
for i := 0; i < burst; i++ {
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("Wait() %d error = %v", i, err)
}
}
elapsed := time.Since(start)
// 60 rpm is one token per second; the burst must not have waited.
if elapsed > 200*time.Millisecond {
t.Fatalf("%d burst requests took %v, want them to proceed immediately", burst, elapsed)
}
}
// TestLimiterEnforcesSustainedRate checks the burst does not become a way to
// exceed the configured rate: once the bucket is drained, requests are spaced
// by the interval again.
func TestLimiterEnforcesSustainedRate(t *testing.T) {
// 6000 rpm = 10ms per token, so this stays fast.
const burst = 3
l := New(6000, burst)
ctx := context.Background()
for i := 0; i < burst; i++ {
if err := l.Wait(ctx); err != nil {
t.Fatal(err)
}
}
// The bucket is drained; the next two must each wait about one interval.
start := time.Now()
for i := 0; i < 2; i++ {
if err := l.Wait(ctx); err != nil {
t.Fatal(err)
}
}
elapsed := time.Since(start)
if elapsed < 15*time.Millisecond {
t.Fatalf("post-burst requests took %v, want them throttled to ~10ms each", elapsed)
}
}
// TestLimiterBurstCappedByRate stops a large max_connections from letting a
// deliberately tiny rate through in one go.
func TestLimiterBurstCappedByRate(t *testing.T) {
l := New(2, 64)
if l.burst != 2 {
t.Fatalf("burst = %d, want it clamped to the rate (2)", l.burst)
}
}
func TestLimiterConcurrentCallersShareTheRate(t *testing.T) {
const burst = 8
l := New(6000, burst)
var wg sync.WaitGroup
start := time.Now()
for i := 0; i < burst; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := l.Wait(context.Background()); err != nil {
t.Errorf("Wait() error = %v", err)
}
}()
}
wg.Wait()
// All within the burst, so none should have blocked meaningfully.
if elapsed := time.Since(start); elapsed > 200*time.Millisecond {
t.Fatalf("%d concurrent burst requests took %v", burst, elapsed)
}
}
+164 -7
View File
@@ -3,23 +3,40 @@ package store
import ( import (
"context" "context"
"database/sql" "database/sql"
"fmt"
"io"
"net/url"
"os"
"strings"
"sync" "sync"
"time"
_ "modernc.org/sqlite" _ "modernc.org/sqlite"
) )
type SQLite struct { type SQLite struct {
db *sql.DB db *sql.DB
mu sync.Mutex path string
mu sync.Mutex
// hasLegacyRows is set when the database was migrated from the pre-source
// schema. Those rows have no source, so IsDownloaded has to consider them
// separately or the entire pre-migration library looks undownloaded.
hasLegacyRows bool
legacyWarned bool
} }
func NewSQLite(path string) (*SQLite, error) { func NewSQLite(path string) (*SQLite, error) {
db, err := sql.Open("sqlite", path) db, err := sql.Open("sqlite", sqliteDSN(path))
if err != nil { if err != nil {
return nil, err return nil, err
} }
// One writer at a time. SQLite allows a single writer anyway, and the
// mutex below already serializes this process; capping the pool keeps the
// driver from opening connections that can only contend with each other.
db.SetMaxOpenConns(1)
s := &SQLite{db: db} s := &SQLite{db: db, path: path}
if err = s.init(); err != nil { if err = s.init(); err != nil {
_ = db.Close() _ = db.Close()
return nil, err return nil, err
@@ -27,6 +44,70 @@ func NewSQLite(path string) (*SQLite, error) {
return s, nil return s, nil
} }
// sqliteDSN builds a file: URI with the pragmas this app needs.
//
// WAL lets a concurrent reader (for example `rip db browse`) run while a rip
// is writing, and busy_timeout makes a contended write wait instead of failing
// immediately with SQLITE_BUSY. The path is passed through url.URL so that a
// config path containing '?' or '#' is not parsed as query parameters.
func sqliteDSN(path string) string {
u := url.URL{Scheme: "file", Opaque: (&url.URL{Path: path}).EscapedPath()}
q := url.Values{}
q.Add("_pragma", "busy_timeout(10000)")
q.Add("_pragma", "journal_mode(WAL)")
q.Add("_pragma", "foreign_keys(on)")
return u.String() + "?" + q.Encode()
}
// schemaVersion is stored in PRAGMA user_version so later migrations can be
// ordered explicitly instead of being inferred from PRAGMA table_info.
const schemaVersion = 1
func (s *SQLite) userVersion() (int, error) {
var v int
if err := s.db.QueryRow(`PRAGMA user_version`).Scan(&v); err != nil {
return 0, err
}
return v, nil
}
func (s *SQLite) setUserVersion(v int) error {
_, err := s.db.Exec(fmt.Sprintf(`PRAGMA user_version = %d`, v))
return err
}
// backupFile copies the database aside before a destructive migration. A
// migration that loses the download history silently costs the user a full
// re-download, so keep a copy they can fall back to.
func (s *SQLite) backupFile() (string, error) {
if strings.TrimSpace(s.path) == "" {
return "", nil
}
src, err := os.Open(s.path)
if err != nil {
if os.IsNotExist(err) {
return "", nil
}
return "", err
}
defer func() { _ = src.Close() }()
dstPath := fmt.Sprintf("%s.bak-%d", s.path, time.Now().Unix())
dst, err := os.OpenFile(dstPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
if err != nil {
return "", err
}
if _, err = io.Copy(dst, src); err != nil {
_ = dst.Close()
_ = os.Remove(dstPath)
return "", err
}
if err = dst.Close(); err != nil {
return "", err
}
return dstPath, nil
}
func (s *SQLite) init() error { func (s *SQLite) init() error {
if err := s.ensureDownloadsSchema(); err != nil { if err := s.ensureDownloadsSchema(); err != nil {
return err return err
@@ -47,6 +128,20 @@ func (s *SQLite) init() error {
} }
} }
// A database migrated by an earlier version has sourceless rows but no
// in-memory flag, so detect them here rather than only at migration time.
if !s.hasLegacyRows {
if n, err := s.legacyRowCount(context.Background()); err == nil && n > 0 {
s.hasLegacyRows = true
}
}
if v, err := s.userVersion(); err == nil && v < schemaVersion {
if err = s.setUserVersion(schemaVersion); err != nil {
return err
}
}
return nil return nil
} }
@@ -94,6 +189,14 @@ func (s *SQLite) ensureDownloadsSchema() error {
return err return err
} }
// Legacy schema: a downloads table with no source column. Back the file up
// first, because the rows below can only be carried over without a source
// and that is not fully reversible.
backupPath, backupErr := s.backupFile()
if backupErr != nil {
return fmt.Errorf("could not back up %s before migrating: %w", s.path, backupErr)
}
tx, err := s.db.Begin() tx, err := s.db.Begin()
if err != nil { if err != nil {
return err return err
@@ -120,12 +223,27 @@ func (s *SQLite) ensureDownloadsSchema() error {
if _, err = tx.Exec(`INSERT OR IGNORE INTO downloads(source, id) SELECT '', id FROM downloads_legacy`); err != nil { if _, err = tx.Exec(`INSERT OR IGNORE INTO downloads(source, id) SELECT '', id FROM downloads_legacy`); err != nil {
return err return err
} }
if _, err = tx.Exec(`DROP TABLE downloads_legacy`); err != nil { // downloads_legacy is deliberately kept. Dropping it used to make the
// migration unrecoverable, and IsDownloaded needs the sourceless rows to
// avoid re-downloading everything the user already has.
if err = tx.Commit(); err != nil {
return err return err
} }
err = tx.Commit() s.hasLegacyRows = true
return err if backupPath != "" {
fmt.Fprintf(os.Stderr, "download database migrated to the source-aware schema; previous copy kept at %s\n", backupPath)
}
return nil
}
// legacyRowCount reports how many pre-migration rows exist. Rows carried over
// from the old schema have an empty source.
func (s *SQLite) legacyRowCount(ctx context.Context) (int, error) {
var n int
err := s.db.QueryRowContext(ctx, `SELECT COUNT(1) FROM downloads WHERE source = ''`).Scan(&n)
return n, err
} }
func (s *SQLite) IsDownloaded(ctx context.Context, source, id string) (bool, error) { func (s *SQLite) IsDownloaded(ctx context.Context, source, id string) (bool, error) {
@@ -137,6 +255,26 @@ func (s *SQLite) IsDownloaded(ctx context.Context, source, id string) (bool, err
if err != nil { if err != nil {
return false, err return false, err
} }
if count > 0 {
return true, nil
}
// Rows migrated from the pre-source schema have an empty source, so an
// exact match on (source, id) misses them and the whole pre-migration
// library would look undownloaded. Matching on id alone can only ever
// over-match across sources, which costs a skipped re-download rather than
// a duplicated library.
if !s.hasLegacyRows {
return false, nil
}
err = s.db.QueryRowContext(ctx, `SELECT COUNT(1) FROM downloads WHERE source = '' AND id = ?`, id).Scan(&count)
if err != nil {
return false, err
}
if count > 0 && !s.legacyWarned {
s.legacyWarned = true
fmt.Fprintf(os.Stderr, "note: matching downloads recorded before sources were tracked; pass --force to re-download them\n")
}
return count > 0, nil return count > 0, nil
} }
@@ -162,6 +300,25 @@ func (s *SQLite) MarkFailed(ctx context.Context, source, mediaType, id string) e
return err return err
} }
// IsFailed reports whether a previous attempt at this item was recorded as
// failed. It complements MarkFailed and is what the retry/repair workflows and
// `rip database browse failed` are built on.
func (s *SQLite) IsFailed(ctx context.Context, source, mediaType, id string) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
var count int
err := s.db.QueryRowContext(
ctx,
`SELECT COUNT(1) FROM failed_downloads WHERE source = ? AND media_type = ? AND id = ?`,
source, mediaType, id,
).Scan(&count)
if err != nil {
return false, err
}
return count > 0, nil
}
func (s *SQLite) Close() error { func (s *SQLite) Close() error {
if s.db == nil { if s.db == nil {
return nil return nil
+195
View File
@@ -2,7 +2,12 @@ package store
import ( import (
"context" "context"
"database/sql"
"fmt"
"os"
"path/filepath" "path/filepath"
"strings"
"sync"
"testing" "testing"
) )
@@ -48,3 +53,193 @@ func TestSQLiteStore(t *testing.T) {
t.Fatalf("MarkFailed() error = %v", err) t.Fatalf("MarkFailed() error = %v", err)
} }
} }
// makeLegacyDB creates a database with the pre-source-column schema and the
// given ids, as a pre-migration install would have on disk.
func makeLegacyDB(t *testing.T, path string, ids ...string) {
t.Helper()
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
if _, err = db.Exec(`CREATE TABLE downloads (id TEXT NOT NULL PRIMARY KEY)`); err != nil {
t.Fatal(err)
}
for _, id := range ids {
if _, err = db.Exec(`INSERT INTO downloads(id) VALUES (?)`, id); err != nil {
t.Fatal(err)
}
}
}
// TestLegacyMigrationPreservesHistory is the regression test for the migration
// that silently wiped the dedupe state: rows were carried over with source=empty
// while IsDownloaded matched source exactly, so every previously downloaded
// track looked new and the whole library was re-downloaded.
func TestLegacyMigrationPreservesHistory(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
path := filepath.Join(dir, "downloads.db")
makeLegacyDB(t, path, "old-1", "old-2")
s, err := NewSQLite(path)
if err != nil {
t.Fatalf("NewSQLite() error = %v", err)
}
defer func() { _ = s.Close() }()
for _, id := range []string{"old-1", "old-2"} {
ok, err := s.IsDownloaded(ctx, "qobuz", id)
if err != nil {
t.Fatalf("IsDownloaded(%q) error = %v", id, err)
}
if !ok {
t.Errorf("pre-migration download %q reported as not downloaded", id)
}
}
ok, err := s.IsDownloaded(ctx, "qobuz", "never-seen")
if err != nil {
t.Fatal(err)
}
if ok {
t.Error("unknown id reported as downloaded")
}
}
// TestLegacyMigrationKeepsBackupAndLegacyTable asserts the migration is
// recoverable: the original file is copied aside and downloads_legacy survives.
func TestLegacyMigrationKeepsBackupAndLegacyTable(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "downloads.db")
makeLegacyDB(t, path, "old-1")
s, err := NewSQLite(path)
if err != nil {
t.Fatalf("NewSQLite() error = %v", err)
}
defer func() { _ = s.Close() }()
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
backups := 0
for _, e := range entries {
if strings.Contains(e.Name(), ".bak-") {
backups++
}
}
if backups != 1 {
t.Errorf("expected exactly 1 backup file, found %d in %v", backups, entries)
}
var n int
if err = s.db.QueryRow(`SELECT COUNT(1) FROM downloads_legacy`).Scan(&n); err != nil {
t.Fatalf("downloads_legacy was dropped: %v", err)
}
if n != 1 {
t.Errorf("downloads_legacy row count = %d, want 1", n)
}
if v, err := s.userVersion(); err != nil || v != schemaVersion {
t.Errorf("user_version = %d (err %v), want %d", v, err, schemaVersion)
}
}
// TestLegacyRowsDetectedOnReopen covers a database already migrated by the
// earlier, lossy code path: the sourceless rows are on disk but nothing in
// this process migrated them.
func TestLegacyRowsDetectedOnReopen(t *testing.T) {
ctx := context.Background()
path := filepath.Join(t.TempDir(), "downloads.db")
makeLegacyDB(t, path, "old-1")
s, err := NewSQLite(path)
if err != nil {
t.Fatal(err)
}
_ = s.Close()
reopened, err := NewSQLite(path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = reopened.Close() }()
if !reopened.hasLegacyRows {
t.Fatal("legacy rows not detected on reopen")
}
ok, err := reopened.IsDownloaded(ctx, "tidal", "old-1")
if err != nil {
t.Fatal(err)
}
if !ok {
t.Error("legacy row not honored after reopen")
}
}
// TestConcurrentAccess exercises the mutex and connection handling under load.
func TestConcurrentAccess(t *testing.T) {
ctx := context.Background()
path := filepath.Join(t.TempDir(), "downloads.db")
s, err := NewSQLite(path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = s.Close() }()
var wg sync.WaitGroup
for i := 0; i < 32; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
id := fmt.Sprintf("t%d", n)
if err := s.MarkDownloaded(ctx, "qobuz", id); err != nil {
t.Errorf("MarkDownloaded: %v", err)
return
}
if _, err := s.IsDownloaded(ctx, "qobuz", id); err != nil {
t.Errorf("IsDownloaded: %v", err)
}
if err := s.MarkFailed(ctx, "qobuz", "track", id); err != nil {
t.Errorf("MarkFailed: %v", err)
}
}(i)
}
wg.Wait()
}
func TestSQLiteDSNPragmasApplied(t *testing.T) {
// A path containing '?' must not be parsed as query parameters.
dir := t.TempDir()
path := filepath.Join(dir, "weird?name#x.db")
s, err := NewSQLite(path)
if err != nil {
t.Fatalf("NewSQLite(%q) = %v", path, err)
}
defer func() { _ = s.Close() }()
var mode string
if err := s.db.QueryRow(`PRAGMA journal_mode`).Scan(&mode); err != nil {
t.Fatal(err)
}
if mode != "wal" {
t.Errorf("journal_mode = %q, want wal", mode)
}
var timeout int
if err := s.db.QueryRow(`PRAGMA busy_timeout`).Scan(&timeout); err != nil {
t.Fatal(err)
}
if timeout != 10000 {
t.Errorf("busy_timeout = %d, want 10000", timeout)
}
if err := s.MarkDownloaded(context.Background(), "qobuz", "a"); err != nil {
t.Fatalf("write to odd path failed: %v", err)
}
}