Author SHA1 Message Date
JorenandClaude Opus 5 d7cc4e639c fix: keep verse breaks in qobuz plain lyrics, handle timestamped blanks
The no-space form after the timestamp is correct and unchanged: LRC is
[mm:ss.xx]text, and the Deezer output already matches. Two real issues behind
that question, though.

Qobuz separates verses with an entry whose line is empty and which has no
start or end at all. Those were dropped entirely, so the plain lyrics tag came
out as one undifferentiated block. They are now kept as single blank lines,
collapsed so a run of separators cannot produce several blanks, and trimmed at
both ends so a tag never begins or ends with one. Checked against the live
track: 41 plain lines with 8 stanza breaks, where before it was 33 flat lines.

They are still excluded from the LRC, which is the correct behaviour rather
than an omission: a synced entry needs a timestamp, and these have none, so
emitting one would park a blank lyric at the wrong moment.

Separately, an empty line that *does* carry a timestamp is a different thing:
a display-clear marker. Qobuz does not appear to emit those, but other lyrics
sources do, and the parser now handles them properly by writing a bare
[mm:ss.xx] entry into the LRC and keeping it out of the plain text.

The LRC itself is unchanged for real data: 33 entries, every one timestamped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 22:05:20 +02:00
JorenandClaude Opus 5 da4add57fe perf: allow request bursts, reuse album track metadata; feat: qobuz lyrics
Two related changes: a 68-track Qobuz album was gated by the API throttle
rather than the network, and Qobuz lyrics are now supported, which is what the
faster album path had to be designed around.

--- perf ---

ratelimit: token bucket instead of fixed spacing
  The limiter spaced every request exactly interval apart with no burst, so at
  the default 60 requests/minute it allowed exactly one request per second no
  matter how many workers ran. Measured: 8 requests took 7s with 1 worker and
  7s with 6, i.e. max_connections had no effect on anything touching the API.
  Since a track needs its metadata and stream URL before downloading, workers
  could only start a track every two seconds, which is why an album looked
  serial even at 57 MiB/s per track.

  New() now takes a burst (providers pass max_connections), idle time accrues
  budget up to that burst, and burst is clamped to one minute's worth of
  requests so a large max_connections cannot bypass a small configured rate.

app: reuse the track objects already in the album payload
  ripAlbum fetched the album, kept only the track ids, then asked the provider
  for every track again, doubling the request count. Providers opt in via
  AlbumTracksAreComplete(); only Qobuz does, because it is only verified there
  and because Tidal does extra work in its track lookup.

  Verified against the live Qobuz API on a 68-track release: the embedded track
  objects already carry title, version, track_number, media_number,
  parental_warning, isrc, copyright, performer, performers, composer, duration,
  audio_info, maximum_bit_depth, maximum_sampling_rate and
  release_date_original. The only field a per-track lookup adds that this app
  reads is the nested album object, so albumTrackMetadata attaches the parent
  album with its own track list stripped, or a 68-track album would hold 68
  copies of itself. trackMetaUsable refetches anyway when title or track number
  is missing, so a thin payload cannot produce a wrongly named file.

  Measured end-to-end through the real app and limiter at the shipped defaults,
  68 tracks: 137 API calls / 2m11s before, 69 calls / 1m3s after.

--- qobuz lyrics ---

Qobuz's main API does not serve lyrics: has_lyrics is only a flag, extra=lyrics
adds nothing, and track/getLyrics does not exist. Its apps use a two-step flow
instead, confirmed live:

  GET track/lyricsUrl?track_id=&app_id=&request_ts=&request_sig=
    -> {"track_id","album_id","lyrics_url"}      (signed CloudFront URL)
  GET <lyrics_url>
    -> {"original":{"type","lang","lines":[{"line","start","end"}]},
        "translation_langs":[...],"writers","publishers":[...]}

type is plain, lsync or wsync; start/end are milliseconds; the line text is
under "line". Passing a language returns a translation instead of the original,
and only for languages that exist for the track (en 403s on an English song),
so the language parameter is deliberately omitted. A 403 on the signed URL is
how Qobuz represents "no lyrics", so it is not treated as an error.

Lyrics are exposed through a TrackLyrics method rather than inside GetMetadata,
because album rips skip the per-track metadata call. That keeps the fast path:
with lyrics on, an album costs the same two provider calls per track it always
did, and the lyrics document itself comes from a CDN that does not consume the
provider's request budget. With metadata.exclude = ["lyrics"] it drops to one
call per track and the full speedup applies.

Existing behaviour is preserved: Tidal and Deezer already attach lyrics in
their metadata, so attachLyrics leaves them alone, and a lyrics failure is
logged and ignored rather than failing a download.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 21:55:48 +02:00
JorenandClaude Opus 5 26fb466afc refactor: finish the jsonutil dedupe; back up config before auto-upgrade
Duplicated helpers
  cmd/rip had its own copies of six helpers that already existed in
  internal/jsonutil: asString (StringFromAny), searchInt (IntFromAny),
  nestedSearchInt and nestedSearchString (NestedAny/NestedString),
  firstNonEmpty (FirstNonEmpty) and firstPositiveInt. firstPositiveInt was
  duplicated a third and fourth time in provider/yandex and provider/beatport,
  so it moves into jsonutil alongside the rest. jsonutil is back to 100%
  coverage.

  searchBool is deliberately left alone, with a comment saying why: it accepts
  only a real bool, and switching it to jsonutil.BoolFromAny would start
  coercing strings and numbers, changing how search results get labelled.

config auto-upgrade is no longer destructive
  UpgradeOutdated re-serializes the whole config from the parsed struct, so any
  key this version does not know about is dropped, and it runs automatically
  from run() without prompting. It now copies the original to
  <path>.bak-<unixtime> (0600, since it holds the same credentials) and
  validates the result before writing. It returns the backup path so the CLI
  can tell the user where it went. The comment states plainly that this is a
  version stamp rather than a migration framework.

cmd/rip tests
  The command bodies are testable now that they are functions returning exit
  codes rather than arms of a switch that called os.Exit. Covers the usage
  errors for every command, the quality floor on `rip id`, config path output,
  the --version flag, and that applyQualityOverride reaches all six sources and
  clamps only the two with a floor. cmd/rip coverage 17% -> 26%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:26:24 +02:00
JorenandClaude Opus 5 b5d4d5688d refactor: consolidate the duplicated download and worker-pool code
internal/download
  The ~40-line progress-bar decorator stack was copy-pasted into three download
  paths and had started to drift; it is now newDownloadBar, which returns a nil
  bar when progress is off (so callers use it unconditionally) plus a finish
  func to defer. The three hand-rolled read/write loops that differed only in
  whether they touched a bar become copyWithProgress.

  FileDeezerEncrypted and FileYandexEncrypted were ~90% identical: mkdir, GET,
  status check, create output, bar, cleanup-on-failure, Content-Length check,
  fsync. That scaffolding is now streamEncrypted, and each function supplies
  only its decryption step. Also switches the io.EOF/io.ErrUnexpectedEOF
  comparisons to errors.Is.

internal/app
  Three copies of "coerce a JSON array that may be []any or []map[string]any"
  become anySlice, which also reports whether the value was an array at all so
  callers can still distinguish missing from empty.

  The worker-pool dispatch existed three times and the copies disagreed: two
  counted failures under a mutex and logged inline with the track id, while the
  third collected errors in a channel and logged them all after wg.Wait()
  without the id. runTrackWorkers unifies them on the inline form, which
  reports failures as they happen and names the track, and keeps the mutex held
  across the log so failure lines cannot interleave.

Tests
  The progress-bar paths had no coverage at all, because progress is disabled
  when not attached to a terminal; four new tests force it on via the existing
  STREAMRIP_GO_FORCE_PROGRESS escape hatch and cover the determinate bar, the
  unknown-length spinner, Deezer decryption with a bar attached, and the
  failure path still removing the partial file. internal/download coverage
  53% -> 63%.

  Three cancellation tests pin the Ctrl-C fix. Verified as genuine regression
  tests by reverting the guards: without them the rip walks all 40 tracks of a
  cancelled album, logs a failure for each, and Rip() returns nil.

Adds store.SQLite.IsFailed, the read complement to MarkFailed, used by the
bookkeeping test to confirm failure records still land under a cancelled
context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:23:24 +02:00
JorenandClaude Opus 5 78d4f0316d refactor: split the command dispatch out of main.go
Completes the restructuring started when main() became run() int. main.go goes
from 1217 lines to 130: global option handling, config loading, signal setup,
and a dispatch switch that delegates to one function per command.

The command bodies move to cmd_rip.go (url, file, id), cmd_search.go (search,
lastfm), cmd_config.go (config, database) and cmd_smoke.go (the developer smoke
commands), each taking (ctx, cfg, gopts, args) and returning an exit code.

They also stop reading the process global. Previously run() did

    os.Args = append([]string{os.Args[0], gopts.command}, gopts.commandArgs...)

and the arms then indexed back into os.Args[2:], os.Args[5:] and so on, so the
already-parsed options were round-tripped through os.Args. The arms now take
the parsed gopts.commandArgs directly, which is a uniform index shift of two
(os.Args[5:] -> args[3:], len(os.Args) < 5 -> len(args) < 3).

No behaviour change. Verified by running each command's argument validation and
help path against the built binary, plus the existing suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:17:24 +02:00
JorenandClaude Opus 5 ff5fb1cdb5 chore: add README, Makefile, CI, lint config and --version
The repo had no README, Makefile, CI, linter config or version string.

- README.md: requirements (ffmpeg/ffprobe, plus yt-dlp for SoundCloud), build,
  the per-source credentials table, usage, the quality scale and the download
  database's migration behaviour. Every claim was checked against the code
  rather than written from memory, which is how the quality-range bug below was
  found. It also records the two deferred refactors (no shared HTTP layer, and
  metadata as map[string]any) so they are documented rather than lore, and
  notes that there is no LICENSE file.
- Makefile: build, install, test, race, cover, fmt, fmt-check, vet, lint,
  check, clean, help. `make build` stamps version/commit/date via -ldflags.
- .github/workflows/ci.yml: gofmt, vet, build, test and test -race, plus a
  lint job. The linter is intentionally unpinned, with a comment: a
  golangci-lint built against an older Go cannot analyse a go1.25 module, which
  is the same reason a locally installed staticcheck built with go1.24 cannot
  run here.
- .golangci.yml: standard set plus bodyclose, errorlint, nilerr and
  govet/copylocks. copylocks matters now that config.Config carries a mutex;
  bodyclose and errorlint target defects this codebase actually has.
- `rip --version` / `-V`, documented in the help output, falling back to the
  VCS revision the toolchain embeds for `go install` builds.

Fixes a bug introduced in the previous commit: Validate capped deezer quality
at 2 and soundcloud at 1, but config.toml.example documents deezer as 2/3/4 and
the global -q flag accepts 0-4, so a documented config would have failed to
load. deezer's buildFormatPriority treats anything >= 2 as FLAC and soundcloud
ignores the value entirely, so the real ceiling is 4 everywhere; only the Qobuz
and Beatport floor of 1 is a genuine constraint. Pinned with a test that walks
the whole documented range.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:14:59 +02:00
JorenandClaude Opus 5 384f32cd67 refactor: move vendor app credentials and signing keys into config
The provider apps' own identifiers and protocol keys were scattered as
constants across six packages. They are not user secrets and stay baked in so
the tool still works out of the box, but they are now mirrored into the config
so that when a provider rotates one, a user can fix their install with a config
edit instead of waiting for a release.

Moved, each with the current value as the default in internal/config:
  qobuz     mobile_app_id, mobile_app_secret
  tidal     app_client_id, app_client_secret
  deezer    api_key, gateway_key, blowfish_secret
  yandex    sign_key, legacy_mp3_sign_salt
  beatport  app_client_id

applyVendorDefaults backfills any empty value, so configs written before these
keys existed keep working and the clients never have to nil-check them. Each
client reads through a small accessor; the package-level signing helpers
(mobileSignRequest, yandexDownloadSign, legacyDirectURL, deriveGatewayAuthToken
and friends, deriveDeezerBlowfishKey) now take the key as a parameter instead
of closing over a constant. The blowfish secret reaches internal/download
through a field on Downloader set by app.New.

The blowfish key derivation XORs 16 bytes of the secret, so a mistyped override
would have panicked; a wrong-length value is now rejected with a warning and
the built-in value is used. Tests pin the length and that the qobuz mobile
secret is valid hex, since the HKDF step requires it.

Also drops soundcloud.client_id and soundcloud.app_version: soundcloud goes
through yt-dlp, which supplies its own, so nothing read them.

config.toml.example documents each new key as "leave empty unless the vendor
rotates theirs".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:11:41 +02:00
JorenandClaude Opus 5 b3b6a2eb1d refactor: delete dead code, dedupe helpers, add tests for untested packages
Dead code
  Removed, each verified to have no non-test caller: promptSearchSelection
  (77 lines, superseded by promptSearchSelectionMenu), AddPlaylistByTrackIDs,
  download.NewWithVerifySSL, qobuz.Client.GetTrackMetadata,
  tidal.Client.trackSupportsAtmos (test-only, and the comment above
  GetDownloadable already documented its removal from the live path) and its
  now-orphaned test, yandex.canonicalTrackRequestID (a no-op returning its
  trimmed input; the call site uses strings.TrimSpace directly) and the unused
  defaultEstimatedKbps constant.

Config keys that did nothing
  qobuz.download_booklets and misc.check_for_updates were declared, defaulted
  and documented in config.toml.example but read by no code. The whole
  [youtube] block went too: there is no YouTube provider. go-toml ignores
  unknown keys, verified, so existing user configs still load.

Duplicated helpers
  tidal.stringify and yandex.firstNonEmpty were byte-identical copies of
  jsonutil.StringFromAny and jsonutil.FirstNonEmpty. displayArtistNames
  existed three times, byte-identical, in internal/app, provider/yandex and
  provider/beatport; it now lives in jsonutil. The CLI's normalizeCodec
  delegates to config.NormalizeCodec so the flag and the config file are
  checked by the same rules.

Tests for packages that had none
  internal/jsonutil (0% -> 100%) and internal/netutil (0% -> 78%), the latter
  covering redactURL, which is a security control, and the zero-timeout
  contract that internal/download and the qobuz body copy depend on.
  internal/config rose to 82% with coverage of the new Validate(), the
  0600 permissions, CloneConfigData not sharing slices, and a concurrent
  UpdateTokens test that exercises the mutex under -race.

Test suite is ~85x faster
  Provider tests built clients with the default 60 requests/minute, so the
  rate limiter slept 1s per HTTP call: deezer took 31s, tidal 11s, qobuz 8s.
  Each provider test package now builds its config through a testConfigData
  helper that disables throttling. Whole suite: ~60s -> 0.7s, and 2.8s
  under -race.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 17:05:55 +02:00
JorenandClaude Opus 5 201aa5daa7 fix: database migration, config validation and provider client defects
Completes the correctness pass.

Download database migration no longer destroys dedupe state
  The pre-source-column migration carried every legacy row over with
  source='' and then dropped downloads_legacy, while IsDownloaded matched
  source exactly. The result was that every previously recorded download
  looked new, the user silently re-downloaded their whole library, and the
  original rows were gone. Now the file is copied to <path>.bak-<unixtime>
  first, downloads_legacy is kept, and IsDownloaded falls back to the
  sourceless rows (over-matching across sources costs a skipped re-download,
  which is the cheaper error). Schema state is tracked in PRAGMA user_version
  so later migrations can be ordered explicitly. Databases already migrated by
  the old code are detected on open.

  Also enables WAL and a 10s busy_timeout and caps the pool at one connection,
  so a concurrent reader no longer gets SQLITE_BUSY with no retry, and builds
  the DSN through url.URL so a path containing '?' is not parsed as query
  parameters.

--quality now applies to every source
  applyGlobalConfigOverrides set only qobuz, tidal, deezer and soundcloud, so
  `rip -q 4 url <beatport-url>` was a silent no-op even though
  qualityForSource reads Yandex and Beatport. All six are set, with the
  per-source floor applied and reported rather than silently ignored.

config: validation and a fixed reset
  Load checked nothing but misc.version, so a negative max_connections or a
  bogus conversion codec was accepted and only surfaced mid-rip. Adds
  Validate() covering per-source quality ranges, max_connections,
  requests_per_minute and truncate_to, and normalizes the conversion codec
  from the file through the same path as --codec. `config reset` assigned one
  value to both Config.File and Config.Session, so their Secrets slices shared
  a backing array; it now takes separate copies.

  Tests cover the shipped defaults and config.toml.example, which is how the
  over-strict artwork max-width bound in the first draft was caught: -1 is the
  documented "do not downscale" sentinel.

soundcloud: use the shared HTTP client
  It was the only provider constructing its own http.Client, so it silently
  ignored downloads.verify_ssl and max_connections and bypassed the transport
  that redacts credentials from -vv logs. It also had no rate limiter at all.

qobuz: mobile fallback could not download a long track
  http.Client.Timeout bounds the whole request including the body read, so the
  30s API client aborted any track that took longer than 30s to stream. The
  body copy now uses a no-timeout client, as internal/download already does.

qobuz: HKDF no longer fails silently
  hexDecodeOrNil discarded its error, so a non-hex app secret keyed HKDF on
  nil, derived a wrong KEK and produced silently corrupt audio instead of an
  error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 16:58:59 +02:00
JorenandClaude Opus 5 c030d1be09 fix: resource leaks, cancellation, races and path traversal
Six correctness bugs that the existing tests could not see. The suite passed
and `go test -race` was clean throughout, because nothing exercised concurrent
token refresh, shared-folder artwork, or hostile metadata.

main(): extract run() int
  main() was a single 1196-line function that registered 20
  `defer mainApp.Close()` handlers and then called os.Exit 150 times, so none
  of them ever ran. Close() is what closes the SQLite handle, flushes the
  progress bars and removes the artwork temp dirs, so every failed rip leaked
  temp directories. main() is now a wrapper around run() int and the command
  bodies return exit codes.

Ctrl-C now stops a rip
  ctx was threaded everywhere but never checked: zero ctx.Err()/ctx.Done()
  in internal/app and internal/download. Cancellation left Rip() iterating
  the whole remaining queue, and the retry ladder retried on a dead context,
  so one Ctrl-C cost three doomed attempts per remaining track. Adds checks to
  the dispatch loops, the worker-spawn loops and the retry ladder, and routes
  the MarkFailed/MarkDownloaded bookkeeping through a detached context so the
  records survive cancellation. Convert and TagFLAC now take a ctx and use
  exec.CommandContext, so a stalled ffmpeg/ffprobe no longer outlives Ctrl-C.

Config race
  Provider token refresh mutated the shared *config.Config and called
  SaveFile() from inside download workers with no lock, racing the struct
  fields and issuing concurrent os.WriteFile against one path. Adds a mutex to
  config.Config plus UpdateTokens(), and routes all five refresh sites through
  it. Each provider client gets an authMu guarding loggedIn and its
  login/refresh path; beatport needed it most, since ensureToken is reached
  from apiRequest.

Artwork cover race
  artwork.Prepare was check-then-write on a path shared by every worker in a
  playlist rip. Two workers both saw a missing cover.jpg, both created it, and
  the loser's cleanup deleted the winner's finished file. Adds a per-folder
  lock and downloads via a unique temp file + rename. The regression test
  fails against the old code with 15 of 16 workers getting no cover.

Path traversal from remote metadata
  naming.CleanName stripped <>:"/\|?* and control chars but not "." , so a
  title of ".." survived into filepath.Join and escaped the download root.
  SoundCloud and Tidal playlist names are user-generated. Now rejects "." and
  "..", trims trailing dots, handles Windows device names, adds DEL to the
  invalid set and caps components at 255 bytes without splitting a rune.
  app.go gains safeJoin as a second line of defence.

Credentials no longer world-readable
  The config holds plaintext provider passwords, Deezer ARL cookies and OAuth
  tokens and was written 0o644. Now 0o600, with 0o700 directories, and an
  existing 0o644 file is tightened on write.

Also: deletes the 9 duplicated *-rip-smoke dev commands (~300 lines fully
subsumed by `rip id`) and the unreachable second dev-help arm; registers
image/png so PNG covers are downscaled instead of being silently saved as
cover.jpg containing PNG bytes; adds the missing .gitignore, which was keeping
a 21MB build artifact one `git add -A` away from the repo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 16:52:58 +02:00
Joren eb7854bac3 fix: use all album artists for folder naming 2026-07-22 22:34:42 +02:00
Joren 5f61b1a3cf fix: use exact qobuz download quality 2026-07-12 21:44:01 +02:00
Joren e336bb96f1 fix: set album track totals 2026-07-12 03:28:27 +02:00
Joren 99c531928e feat: add cli help menus 2026-07-12 00:06:18 +02:00
Joren 150b5b5d85 fix: omit original mix from titles 2026-07-11 23:43:03 +02:00
Joren 80bfbe0ecb feat: infer beatport release artist 2026-07-11 16:40:04 +02:00
Joren 3413de1daa feat: format multi-artist credits 2026-07-11 04:44:54 +02:00
Joren d2fa098d69 feat: add configurable artist separator 2026-07-11 01:21:40 +02:00
Joren 537959b6ec feat: add beatport provider 2026-07-10 19:21:14 +02:00
Joren 2a7d259e9f Merge branch 'feat/yandex-music'
# Conflicts:
#	internal/audio/tag/tagger.go
#	internal/audio/tag/tagger_test.go
#	internal/urlparse/parse_test.go
2026-07-10 18:18:04 +02:00
Joren b65edb4cce yes 2026-06-15 17:18:47 +02:00
Joren 0ae8c7e008 feat yandex desktop downloads 2026-06-10 12:58:04 +02:00
Joren db26a40415 fix qobuz interpreter artist urls 2026-06-10 12:34:37 +02:00
Joren fa39582849 fix tidal playlist metadata and retries 2026-05-21 23:03:27 +02:00
lb-a 3bc965db77 Merge branch 'feat/verbose' 2026-05-07 23:56:42 +02:00
lb-a 3909ba5113 feat: implement second level of verbosity 2026-05-07 18:45:36 +02:00
lb-a 04cc56040b feat: add log hooks 2026-05-07 18:45:05 +02:00
lb-a ef741434cb feat: ad verbose package 2026-05-07 18:42:24 +02:00
Joren ef72aad14e Merge branch 'fix/qobuz-mobile-fallback-eof' 2026-05-05 21:51:19 +02:00
Joren 59b476034e fix qobuz EOF downloads with mobile fallback flow 2026-05-01 17:23:52 +02:00
lb-a 9618108f2a performance: exclude lyrics if they are not enabled 2026-04-30 12:00:58 +02:00
lb-a 7a27845e75 performance: change I/O buffer size 2026-04-29 23:56:05 +02:00
lb-a 945695cea7 performance: tune shared HTTP transport for concurrent CDN downloads 2026-04-29 23:53:51 +02:00
Joren 9e27ba842f fix deezer track pagination and decryption track id resolution 2026-04-26 22:23:23 +02:00
Joren 63e1f20e04 fix tidal lyrics fetch host and synced lyric tagging 2026-04-25 01:24:11 +02:00
Joren 5a6d7926ff Merge remote-tracking branch 'origin/master' into fix/resolved-audio-folder-naming 2026-04-24 18:37:30 +02:00
Joren a06e3fa996 clarify provider quality ladders in config example 2026-04-24 18:36:01 +02:00
Joren b63889f559 refactor deezer session and pipe graphql flow 2026-04-24 15:19:47 +02:00
Joren c29f5415ac fix deezer mobile token and launch parity 2026-04-24 15:11:17 +02:00
Joren 6f6cca19b3 adjust atmos folder bit depth to match delivered stream 2026-04-24 00:45:53 +02:00
Joren 232901f3eb unify folder naming with resolved audio profiles across providers 2026-04-24 00:35:38 +02:00
Joren d5b336ca4e fix tidal lossless quality negotiation and atmos format fallback 2026-04-23 23:53:41 +02:00
Joren c1e89f5876 render manifest ffmpeg progress like standard download bars 2026-04-22 00:40:11 +02:00
Joren 50ca5f564b improve search menu preview and artist release details 2026-04-22 00:16:59 +02:00
Joren 6bc4b3b319 Refactor: comprehensive cleanup and modularization
- Extracted common JSON parsing helpers into internal/jsonutil
- Removed duplicated helper functions from provider packages
- Removed dead code in internal/app/app.go and downloader.go
- Replaced deprecated strings.Title with jsonutil.TitleCase
- Added graceful shutdown with signal handling in main.go
- Split monolithic cmd/rip/main.go into args.go, helpers.go, lastfm.go, search.go
2026-04-21 23:38:41 +02:00
Joren d65dc182f8 harden ffmpeg pipeline failure handling and stream mapping 2026-04-21 23:07:48 +02:00
Joren beb6ce6cbb improve ffmpeg manifest progress rendering 2026-04-21 22:28:37 +02:00
Joren 96348a6a34 harden manifest detection and downloader failure coverage 2026-04-21 20:55:14 +02:00
Joren dfa8095e1d fix lastfm extraction regression and honor no-db 2026-04-21 20:40:28 +02:00
Joren 4c7e6f5792 tighten lastfm parsing and locale url handling 2026-04-21 19:21:32 +02:00
Joren ba97fe85fe harden lastfm playlist parsing and url validation 2026-04-21 19:10:50 +02:00
Joren c67be72869 harden search parsing and qobuz refresh validation 2026-04-21 19:04:33 +02:00
Joren de4e561377 harden qobuz and downloader reliability edge cases 2026-04-21 18:54:10 +02:00
Joren 0161c01a4c add optional tidal atmos preference with immersive fallback 2026-04-21 18:31:58 +02:00
Joren 6dbf6a222d paginate tidal artist album fetches across all pages 2026-04-21 17:33:25 +02:00
Joren 654bed17e2 harden deezer auth errors and mixed playlist preflight 2026-04-21 12:19:23 +02:00
Joren 1246a24749 map deezer gain to replaygain track metadata 2026-04-21 11:51:17 +02:00
Joren 4f86751ff4 paginate artist album fetching for deezer and qobuz 2026-04-21 11:39:04 +02:00
Joren 9ebddc8316 harden deezer auth and lyrics tagging behavior 2026-04-21 11:14:57 +02:00
58 changed files with 16434 additions and 3499 deletions

No files matched your search

+61
View File
@@ -0,0 +1,61 @@
name: CI
on:
push:
branches: [master]
pull_request:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
- name: gofmt
run: |
out="$(gofmt -l .)"
if [ -n "$out" ]; then
echo "These files are not gofmt-clean:"
echo "$out"
exit 1
fi
- name: go vet
run: go vet ./...
- name: Build
run: go build ./...
- name: Test
run: go test ./...
- name: Test (race)
run: go test -race ./...
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
# .golangci.yml uses the v2 config schema, so this needs golangci-lint
# v2.x and action v7+. Left unpinned deliberately: a golangci-lint built
# against an older Go toolchain cannot analyse this module at all (the
# same reason a locally installed staticcheck built with go1.24 fails on
# go.mod's go 1.25). Pin it once a known-good version is confirmed.
- uses: golangci/golangci-lint-action@v7
with:
version: latest
args: --timeout=5m
+21
View File
@@ -0,0 +1,21 @@
# Build output
/rip
/rip.exe
*.test
# Local runtime state
*.db
*.db-wal
*.db-shm
config.toml
*.bak-*
# Scratch files
albums.txt
*.log
coverage.out
# Editor / OS
.DS_Store
.idea/
.vscode/
+39
View File
@@ -0,0 +1,39 @@
version: "2"
run:
timeout: 5m
tests: true
linters:
default: standard
enable:
- bodyclose # unclosed HTTP response bodies: this app makes a lot of requests
- errorlint # == on errors instead of errors.Is; the download code had several
- copyloopvar
- misspell
- nilerr
- unconvert
- usestdlibvars
- whitespace
settings:
govet:
enable:
- copylocks # config.Config carries a mutex and must not be copied
- nilness
staticcheck:
checks:
- all
exclusions:
presets:
- std-error-handling
rules:
# Test helpers legitimately ignore errors on cleanup paths.
- path: _test\.go
linters:
- errcheck
formatters:
enable:
- gofmt
+63
View File
@@ -0,0 +1,63 @@
BINARY := rip
PKG := ./cmd/rip
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
COMMIT ?= $(shell git rev-parse HEAD 2>/dev/null)
DATE ?= $(shell date -u +%Y-%m-%dT%H:%M:%SZ)
LDFLAGS := -X main.version=$(VERSION) -X main.commit=$(COMMIT) -X main.date=$(DATE)
.DEFAULT_GOAL := build
.PHONY: build
build: ## Build the rip binary
go build -ldflags "$(LDFLAGS)" -o $(BINARY) $(PKG)
.PHONY: install
install: ## Install rip into GOBIN
go install -ldflags "$(LDFLAGS)" $(PKG)
.PHONY: test
test: ## Run the test suite
go test ./...
.PHONY: race
race: ## Run the test suite under the race detector
go test -race ./...
.PHONY: cover
cover: ## Run tests and report per-package coverage
go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out | tail -1
.PHONY: fmt
fmt: ## Format all Go files
gofmt -w .
.PHONY: fmt-check
fmt-check: ## Fail if any file is not gofmt-clean
@out="$$(gofmt -l .)"; \
if [ -n "$$out" ]; then echo "not gofmt-clean:"; echo "$$out"; exit 1; fi
.PHONY: vet
vet: ## Run go vet
go vet ./...
.PHONY: lint
lint: ## Run golangci-lint if it is installed
@if command -v golangci-lint >/dev/null 2>&1; then \
golangci-lint run; \
else \
echo "golangci-lint not installed; skipping"; \
echo " go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest"; \
fi
.PHONY: check
check: fmt-check vet test ## Everything CI runs
.PHONY: clean
clean: ## Remove build and coverage artifacts
rm -f $(BINARY) coverage.out
.PHONY: help
help: ## List targets
@grep -hE '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | awk -F':.*?## ' '{printf " %-12s %s\n", $$1, $$2}'
+172
View File
@@ -0,0 +1,172 @@
# streamrip-go
A Go port of [streamrip](https://github.com/nathom/streamrip): a command-line
music downloader for Qobuz, Tidal, Deezer, SoundCloud, Yandex Music and
Beatport. Downloads tracks, albums, playlists, artist discographies and labels,
tags them, optionally embeds cover art and converts the result.
You need your own paid account for any source you use. This tool does not
provide or circumvent access to anything.
## Requirements
- Go 1.25 or newer (to build)
- `ffmpeg` and `ffprobe` — tagging and format conversion
- `yt-dlp` — required for SoundCloud only
```sh
# Arch
sudo pacman -S ffmpeg yt-dlp
# Debian/Ubuntu
sudo apt install ffmpeg && pipx install yt-dlp
# macOS
brew install ffmpeg yt-dlp
```
## Build
```sh
make build # -> ./rip, with version info stamped in
make install # into $(go env GOBIN)
./rip --version
```
Or without make: `go build ./cmd/rip`.
## Configuration
The first run writes a config file and tells you where. It lives at
`$XDG_CONFIG_HOME/streamrip/config.toml` (`~/Library/Application
Support/streamrip/config.toml` on macOS).
```sh
rip config path # print the path
rip config open # open in $EDITOR
rip config reset # restore defaults
```
`config.toml.example` documents every key. The file holds credentials in
plaintext — provider passwords, the Deezer ARL cookie and OAuth tokens — so it
is created `0600`. Keep it that way.
Credentials you need to set, per source:
| Source | Required config |
|------------|-----------------|
| Qobuz | `email_or_userid` + `password_or_token` (md5 of your password), or `use_auth_token = true` with an auth token |
| Tidal | `access_token` / `refresh_token` |
| Deezer | `arl` (recommended), or `email` + `password` |
| Yandex | `access_token` |
| Beatport | `username` + `password` |
| SoundCloud | none; handled via `yt-dlp` |
Lyrics (plain and synced `.lrc`-style) are fetched for Qobuz, Tidal and Deezer.
Each costs one extra API request per track, so `metadata.exclude = ["lyrics"]`
turns them off for a faster rip.
Each provider section also has vendor `app_client_id`-style keys. Leave them
empty: they default to values built into the binary. They exist so that if a
provider rotates one, you can patch your install with a config edit instead of
waiting for a release.
## Usage
```sh
rip url https://play.qobuz.com/album/abc123
rip url https://www.beatport.com/release/example/123 --force
rip id qobuz album abc123
rip search qobuz album "kind of blue"
rip search tidal track "teardrop" --first
rip file urls.txt # one URL per line, or a JSON list of IDs
rip lastfm --source qobuz <lastfm-playlist-url>
rip database downloads # what has already been downloaded
rip database failed
```
Global options (these must come *before* the command):
```
-f, --folder <path> Override downloads folder
-q, --quality <0-4> Override provider quality
-c, --codec <codec> Convert after download: ALAC, FLAC, OGG, MP3, AAC
-ndb, --no-db Ignore the download database
--no-progress Disable progress bars
--no-ssl-verify Disable TLS verification
--config-path <path> Use a custom config file
-v, -vv Verbose logging (-vv also logs HTTP requests)
-V, --version Show version
-h, --help Show help
```
Run `rip help <command>` for per-command help.
### Quality
`-q` is a 0–4 scale mapped onto each provider's own ladder. Qobuz and Beatport
have no tier 0, so `-q 0` is raised to 1 for those and reported on stderr.
| Source | Range | 0 | Top tier |
|------------|-------|---|----------|
| Qobuz | 1–4 | — | FLAC 24-bit up to 192 kHz |
| Tidal | 0–4 | LOW (HE-AAC) | HI_RES_LOSSLESS |
| Deezer | 0–4 | MP3 128 | FLAC (anything ≥ 2) |
| Yandex | 0–4 | LOW | lossless |
| Beatport | 1–4 | — | FLAC 16/44.1 |
| SoundCloud | 0–4 | ignored | provider-defined |
Each provider saturates at the top of its own ladder, so values above its
highest real tier are accepted and behave as that tier.
### Download database
Completed downloads are recorded in a SQLite database and skipped next time.
`--force` / `--ignore-db` overrides this for one run; `-ndb` disables it
entirely. Paths are configurable under `[database]`.
If you are upgrading from a build that predates source tracking, the database
is migrated on first open and a copy is kept alongside it as
`downloads.db.bak-<timestamp>`.
## Development
```sh
make check # gofmt + vet + tests, what CI runs
make test
make race
make cover
make lint # needs golangci-lint
```
The test suite runs in about a second and makes no network requests; provider
tests drive `httptest` servers.
Layout:
```
cmd/rip CLI entry point, one file per command group
internal/app orchestration: queue, resolve, rip
internal/provider/* one client per source, behind provider.Client
internal/download HTTP downloads, progress bars, Deezer/Yandex decryption
internal/audio/{tag,convert} ffmpeg tagging and conversion
internal/{naming,artwork,store,config,netutil,ratelimit,jsonutil,verbose}
```
### Known rough edges
Two deliberate pieces of technical debt, both documented rather than hidden:
- **No shared HTTP layer.** The request/decode/retry sequence is reimplemented
in each provider client, and retry on 429/5xx exists in only two of six. A
`provider/transport` package would remove roughly 700 lines.
- **Metadata is `map[string]any`.** `provider.Client` returns untyped maps, so
each client hand-builds an undeclared "Qobuz-shaped" schema that
`cmd/rip/search.go` then unpicks in a long switch. A typed media model is the
natural next step; `internal/provider/qobuz/model.go` is a started-and-parked
attempt at it.
## License
No license file is present, so this is "all rights reserved" by default. Add
one before distributing. Note that upstream streamrip is GPL-3.0.
+246
View File
@@ -0,0 +1,246 @@
package main
import (
"fmt"
"os"
"strconv"
"strings"
"streamrip-go/internal/config"
)
type smokeOptions struct {
qualitySet bool
quality int
ignoreDB bool
}
type globalOptions struct {
configPath string
folder string
noDB bool
qualitySet bool
quality int
codecSet bool
codec string
noProgress bool
noSSLVerify bool
verbose int
help bool
version bool
command string
commandArgs []string
}
func parseGlobalArgs(args []string) (globalOptions, error) {
opts := globalOptions{}
for i := 0; i < len(args); i++ {
arg := args[i]
if arg == "" {
continue
}
if !strings.HasPrefix(arg, "-") {
opts.command = arg
if i+1 < len(args) {
opts.commandArgs = append([]string(nil), args[i+1:]...)
}
return opts, nil
}
switch {
case isHelpArg(arg):
opts.help = true
return opts, nil
case arg == "--version" || arg == "-V":
opts.version = true
return opts, nil
case arg == "-ndb" || arg == "--no-db":
opts.noDB = true
case arg == "--no-progress":
opts.noProgress = true
case arg == "--no-ssl-verify":
opts.noSSLVerify = true
case arg == "-v" || arg == "--verbose":
if opts.verbose < 2 {
opts.verbose++
}
case arg == "-vv":
if opts.verbose < 2 {
opts.verbose = 2
}
case arg == "-f" || arg == "--folder":
if i+1 >= len(args) {
return globalOptions{}, fmt.Errorf("%s requires a value", arg)
}
opts.folder = strings.TrimSpace(args[i+1])
i++
case strings.HasPrefix(arg, "--folder="):
opts.folder = strings.TrimSpace(strings.TrimPrefix(arg, "--folder="))
case arg == "--config-path":
if i+1 >= len(args) {
return globalOptions{}, fmt.Errorf("--config-path requires a value")
}
opts.configPath = strings.TrimSpace(args[i+1])
i++
case strings.HasPrefix(arg, "--config-path="):
opts.configPath = strings.TrimSpace(strings.TrimPrefix(arg, "--config-path="))
case arg == "-q" || arg == "--quality":
if i+1 >= len(args) {
return globalOptions{}, fmt.Errorf("%s requires a value", arg)
}
q, err := strconv.Atoi(args[i+1])
if err != nil || q < 0 || q > 4 {
return globalOptions{}, fmt.Errorf("invalid quality %q (expected 0-4)", args[i+1])
}
opts.qualitySet = true
opts.quality = q
i++
case strings.HasPrefix(arg, "--quality="):
qRaw := strings.TrimSpace(strings.TrimPrefix(arg, "--quality="))
q, err := strconv.Atoi(qRaw)
if err != nil || q < 0 || q > 4 {
return globalOptions{}, fmt.Errorf("invalid quality %q (expected 0-4)", qRaw)
}
opts.qualitySet = true
opts.quality = q
case arg == "-c" || arg == "--codec":
if i+1 >= len(args) {
return globalOptions{}, fmt.Errorf("%s requires a value", arg)
}
codec, err := normalizeCodec(args[i+1])
if err != nil {
return globalOptions{}, err
}
opts.codecSet = true
opts.codec = codec
i++
case strings.HasPrefix(arg, "--codec="):
codecRaw := strings.TrimSpace(strings.TrimPrefix(arg, "--codec="))
codec, err := normalizeCodec(codecRaw)
if err != nil {
return globalOptions{}, err
}
opts.codecSet = true
opts.codec = codec
default:
return globalOptions{}, fmt.Errorf("unknown global option %q", arg)
}
}
return opts, nil
}
// normalizeCodec delegates to config so the --codec flag and a codec set in
// the config file are validated by the same rules.
func normalizeCodec(raw string) (string, error) {
return config.NormalizeCodec(raw)
}
func applyGlobalConfigOverrides(cfg *config.Config, opts globalOptions) {
if opts.folder != "" {
cfg.Session.Downloads.Folder = opts.folder
}
if opts.noDB {
cfg.Session.Database.DownloadsEnabled = false
}
if opts.qualitySet {
applyQualityOverride(cfg, opts.quality)
}
if opts.codecSet {
cfg.Session.Conversion.Enabled = true
cfg.Session.Conversion.Codec = opts.codec
}
if opts.noProgress {
cfg.Session.CLI.ProgressBars = false
}
if opts.noSSLVerify {
cfg.Session.Downloads.VerifySSL = false
}
}
// minQualityForSource is the lowest quality id each source accepts. Qobuz and
// Beatport have no equivalent of the "lowest" tier that 0 means elsewhere.
var minQualityForSource = map[string]int{
"qobuz": 1,
"beatport": 1,
}
// applyQualityOverride applies a global -q/--quality to every source.
//
// It previously set only qobuz, tidal, deezer and soundcloud, so
// `rip -q 4 url <beatport-url>` was a silent no-op even though
// Main.qualityForSource reads Yandex and Beatport too. Sources with a higher
// floor are clamped rather than erroring, because one global flag has to work
// across a mixed queue.
func applyQualityOverride(cfg *config.Config, quality int) {
clamped := map[string]int{}
for _, source := range []string{"qobuz", "tidal", "deezer", "soundcloud", "yandex", "beatport"} {
q := quality
if min, ok := minQualityForSource[source]; ok && q < min {
q = min
clamped[source] = q
}
switch source {
case "qobuz":
cfg.Session.Qobuz.Quality = q
case "tidal":
cfg.Session.Tidal.Quality = q
case "deezer":
cfg.Session.Deezer.Quality = q
case "soundcloud":
cfg.Session.Soundcloud.Quality = q
case "yandex":
cfg.Session.Yandex.Quality = q
case "beatport":
cfg.Session.Beatport.Quality = q
}
}
for _, source := range []string{"qobuz", "beatport"} {
if q, ok := clamped[source]; ok {
fmt.Fprintf(os.Stderr, "note: %s has no quality %d; using %d for %s\n", source, quality, q, source)
}
}
}
func errorWithActionableHint(err error, opts globalOptions) string {
if err == nil {
return ""
}
msg := err.Error()
if opts.noSSLVerify {
return msg
}
lower := strings.ToLower(msg)
if strings.Contains(lower, "x509") || strings.Contains(lower, "certificate") || strings.Contains(lower, "tls") || strings.Contains(lower, "ssl") {
return msg + " (hint: try again with --no-ssl-verify)"
}
return msg
}
func parseSmokeOptions(args []string, minQuality int, maxQuality int) (smokeOptions, error) {
opts := smokeOptions{}
for _, arg := range args {
switch arg {
case "--force", "--ignore-db":
opts.ignoreDB = true
default:
q, err := parseQuality(arg, minQuality, maxQuality)
if err != nil {
return smokeOptions{}, fmt.Errorf("unknown option %q", arg)
}
opts.quality = q
opts.qualitySet = true
}
}
return opts, nil
}
func parseQuality(raw string, min int, max int) (int, error) {
q, err := strconv.Atoi(raw)
if err != nil {
return 0, err
}
if q < min || q > max {
return 0, fmt.Errorf("quality must be %d-%d, got %d", min, max, q)
}
return q, nil
}
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/config"
"golang.org/x/term"
)
// Commands that inspect local state: config and database.
func cmdConfig(_ context.Context, cfg *config.Config, _ globalOptions, args []string) int {
var err error
if len(args) < 1 {
printCommandHelp(os.Stdout, "config")
return 2
}
switch args[0] {
case "open":
vim := false
for _, arg := range args[1:] {
switch arg {
case "-v", "--vim":
vim = true
default:
fmt.Fprintf(os.Stderr, "option error: unknown option %q\n", arg)
return 2
}
}
fmt.Printf("opening file at %s\n", cfg.Path)
if err = openConfigInEditor(cfg.Path, vim); err != nil {
fmt.Fprintf(os.Stderr, "open config error: %v\n", err)
return 1
}
case "reset":
yes := false
for _, arg := range args[1:] {
switch arg {
case "-y", "--yes":
yes = true
default:
fmt.Fprintf(os.Stderr, "option error: unknown option %q\n", arg)
return 2
}
}
if !yes {
if !term.IsTerminal(int(os.Stdin.Fd())) {
fmt.Fprintln(os.Stderr, "reset requires --yes in non-interactive mode")
return 2
}
ok, askErr := promptYesNo(fmt.Sprintf("Are you sure you want to reset the config file at %s? [y/N]: ", cfg.Path))
if askErr != nil {
fmt.Fprintf(os.Stderr, "prompt error: %v\n", askErr)
return 1
}
if !ok {
fmt.Println("reset aborted")
return 0
}
}
def := config.DefaultConfigData()
// Separate copies: assigning the same value to both halves made
// File.Qobuz.Secrets and Session.Qobuz.Secrets share one backing
// array.
cfg.File = config.CloneConfigData(def)
cfg.Session = config.CloneConfigData(def)
if err = cfg.SaveFile(); err != nil {
fmt.Fprintf(os.Stderr, "reset config error: %v\n", err)
return 1
}
fmt.Printf("reset the config file at %s\n", cfg.Path)
case "path":
if len(args) > 1 {
fmt.Fprintf(os.Stderr, "option error: unexpected argument %q\n", args[1])
return 2
}
fmt.Printf("config path: '%s'\n", cfg.Path)
default:
fmt.Fprintf(os.Stderr, "unknown config command: %s\n", args[0])
return 2
}
return 0
}
func cmdDatabase(_ context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 2 || args[0] != "browse" {
printCommandHelp(os.Stdout, "database")
return 2
}
table := strings.ToLower(strings.TrimSpace(args[1]))
switch table {
case "downloads":
rows, listErr := listDownloadsRows(cfg.Session.Database.DownloadsPath)
if listErr != nil {
fmt.Fprintf(os.Stderr, "database browse error: %v\n", listErr)
return 1
}
fmt.Println("downloads database")
fmt.Println("row id")
for i, id := range rows {
fmt.Printf("%02d %s\n", i, id)
}
case "failed":
rows, listErr := listFailedRows(cfg.Session.Database.FailedDownloadsPath)
if listErr != nil && isNoSuchTableErr(listErr) && cfg.Session.Database.FailedDownloadsPath != cfg.Session.Database.DownloadsPath {
rows, listErr = listFailedRows(cfg.Session.Database.DownloadsPath)
}
if listErr != nil {
fmt.Fprintf(os.Stderr, "database browse error: %v\n", listErr)
return 1
}
fmt.Println("failed downloads database")
fmt.Println("row source media_type id")
for i, row := range rows {
fmt.Printf("%02d %s %s %s\n", i, row.Source, row.MediaType, row.ID)
}
default:
fmt.Fprintf(os.Stderr, "invalid database %q. choose downloads or failed\n", table)
return 2
}
return 0
}
+200
View File
@@ -0,0 +1,200 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/config"
)
// Commands that queue media and rip it: url, file and id.
func cmdURL(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 1 {
printCommandHelp(os.Stdout, "url")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
rawArgs := make([]string, 0, len(args))
ignoreDB := false
for _, arg := range args {
if arg == "--force" || arg == "--ignore-db" {
ignoreDB = true
continue
}
rawArgs = append(rawArgs, arg)
}
mainApp.IgnoreDB = ignoreDB || gopts.noDB
added := 0
for _, raw := range rawArgs {
if addURLToQueue(ctx, mainApp, raw) {
added++
}
}
if added == 0 {
fmt.Println("nothing to rip")
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("url rip complete (%d item(s))\n", added)
return 0
}
func cmdFile(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 1 {
printCommandHelp(os.Stdout, "file")
return 2
}
ignoreDB := false
for _, arg := range args[1:] {
switch arg {
case "--force", "--ignore-db":
ignoreDB = true
default:
fmt.Fprintf(os.Stderr, "option error: unknown option %q\n", arg)
return 2
}
}
content, err := os.ReadFile(args[0])
if err != nil {
fmt.Fprintf(os.Stderr, "read file error: %v\n", err)
return 1
}
idItems, urls, repeated, jsonInput, err := parseFileInput(content)
if err != nil {
fmt.Fprintf(os.Stderr, "file parse error: %v\n", err)
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = ignoreDB || gopts.noDB
added := 0
if jsonInput {
fmt.Printf("detected json file. loading %d item(s)\n", len(idItems))
for _, item := range idItems {
if err = mainApp.AddByID(ctx, item.Source, item.MediaType, item.ID); err != nil {
fmt.Printf("add failed: source=%s type=%s id=%s err=%v\n", item.Source, item.MediaType, item.ID, err)
continue
}
added++
}
} else {
if repeated > 0 {
fmt.Printf("found %d repeated url(s)\n", repeated)
}
fmt.Printf("detected list of urls. loading %d item(s)\n", len(urls))
for _, raw := range urls {
if addURLToQueue(ctx, mainApp, raw) {
added++
}
}
}
if added == 0 {
fmt.Println("nothing to rip")
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("file rip complete (%d item(s))\n", added)
return 0
}
func cmdID(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 3 {
printCommandHelp(os.Stdout, "id")
return 2
}
source := strings.ToLower(strings.TrimSpace(args[0]))
mediaType := strings.ToLower(strings.TrimSpace(args[1]))
itemID := strings.TrimSpace(args[2])
opts, err := parseSmokeOptions(args[3:], 0, 4)
if err != nil {
fmt.Fprintf(os.Stderr, "option error: %v\n", err)
return 2
}
if opts.qualitySet {
switch source {
case "qobuz":
if opts.quality < 1 {
fmt.Fprintf(os.Stderr, "quality error: qobuz quality must be 1-4\n")
return 2
}
cfg.Session.Qobuz.Quality = opts.quality
case "tidal":
cfg.Session.Tidal.Quality = opts.quality
case "yandex":
cfg.Session.Yandex.Quality = opts.quality
case "beatport":
if opts.quality < 1 {
fmt.Fprintf(os.Stderr, "quality error: beatport quality must be 1-4\n")
return 2
}
cfg.Session.Beatport.Quality = opts.quality
}
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = opts.ignoreDB || gopts.noDB
if err = mainApp.AddByID(ctx, source, mediaType, itemID); err != nil {
fmt.Fprintf(os.Stderr, "add error: %v\n", err)
return 1
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("id rip complete: source=%s type=%s id=%s\n", source, mediaType, itemID)
return 0
}
+281
View File
@@ -0,0 +1,281 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/config"
"golang.org/x/term"
)
// Commands that discover media before ripping it: search and lastfm.
func cmdSearch(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
var err error
var source, mediaType string
var sopts searchOptions
if len(args) < 3 {
if !term.IsTerminal(int(os.Stdin.Fd())) {
printCommandHelp(os.Stdout, "search")
return 2
}
source, mediaType, sopts, err = promptSearchInteractive(cfg.Session.CLI.MaxSearchResults)
if err != nil {
fmt.Fprintf(os.Stderr, "search prompt error: %v\n", err)
return 2
}
} else {
source = strings.ToLower(strings.TrimSpace(args[0]))
mediaType = strings.ToLower(strings.TrimSpace(args[1]))
sopts, err = parseSearchArgs(args[2:], cfg.Session.CLI.MaxSearchResults)
}
if err != nil {
fmt.Fprintf(os.Stderr, "search option error: %v\n", err)
return 2
}
if sopts.first && sopts.outputFile != "" {
fmt.Fprintln(os.Stderr, "cannot choose --first and --output-file together")
return 2
}
if !isAllowedSearchSource(source) {
fmt.Fprintf(os.Stderr, "unsupported search source %q\n", source)
return 2
}
if !isAllowedMediaType(mediaType) {
fmt.Fprintf(os.Stderr, "unsupported media type %q\n", mediaType)
return 2
}
if source == "soundcloud" && mediaType != "track" && mediaType != "playlist" {
fmt.Fprintln(os.Stderr, "soundcloud search currently supports media types track and playlist")
return 2
}
if source == "yandex" && mediaType != "track" && mediaType != "album" && mediaType != "playlist" && mediaType != "artist" {
fmt.Fprintln(os.Stderr, "yandex search currently supports media types track, album, playlist, and artist")
return 2
}
if source == "beatport" && mediaType != "track" && mediaType != "album" && mediaType != "label" {
fmt.Fprintln(os.Stderr, "beatport search currently supports media types track, album, and label")
return 2
}
if sopts.query == "" {
fmt.Fprintln(os.Stderr, "search query cannot be empty")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, source)
if err != nil {
fmt.Fprintf(os.Stderr, "%s login error: %v\n", source, err)
return 1
}
pages, err := provider.Search(ctx, mediaType, sopts.query, sopts.limit)
if err != nil {
fmt.Fprintf(os.Stderr, "search error: %v\n", err)
return 1
}
results := normalizeSearchResults(source, mediaType, pages)
if len(results) == 0 {
fmt.Println("no results")
return 0
}
if sopts.outputFile != "" {
fmt.Printf("results: %d\n", len(results))
for i, result := range results {
fmt.Printf("%2d. id=%s | %s\n", i+1, result.ID, result.Title)
}
if err = writeSearchResultsToFile(source, mediaType, results, sopts.outputFile); err != nil {
fmt.Fprintf(os.Stderr, "write results error: %v\n", err)
return 1
}
fmt.Printf("wrote %d results to %s\n", len(results), sopts.outputFile)
return 0
}
if sopts.first {
fmt.Printf("results: %d\n", len(results))
fmt.Printf(" 1. id=%s | %s\n", results[0].ID, results[0].Title)
results = results[:1]
}
if sopts.noDownload {
fmt.Printf("results: %d\n", len(results))
for i, result := range results {
fmt.Printf("%2d. id=%s | %s\n", i+1, result.ID, result.Title)
}
return 0
}
if !sopts.first {
fmt.Printf("results: %d\n", len(results))
}
if sopts.first {
selection := []int{0}
mainApp.IgnoreDB = sopts.ignoreDB || gopts.noDB
skippedDownloaded := 0
added := 0
for _, idx := range selection {
item := results[idx]
if !sopts.ignoreDB {
already, checkErr := mainApp.Store.IsDownloaded(ctx, source, item.ID)
if checkErr == nil && already {
skippedDownloaded++
fmt.Printf("skip (already downloaded): id=%s | %s\n", item.ID, item.Title)
continue
}
}
if err = mainApp.AddByID(ctx, source, mediaType, item.ID); err != nil {
fmt.Printf("add failed: id=%s err=%v\n", item.ID, err)
continue
}
added++
}
if added == 0 {
if skippedDownloaded > 0 {
fmt.Println("selected item was already downloaded (use --force to redownload)")
} else {
fmt.Println("nothing selected to download")
}
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("search download complete (%d item(s))\n", added)
return 0
}
if !term.IsTerminal(int(os.Stdin.Fd())) {
fmt.Println("non-interactive input; use `rip id` to download specific results")
return 0
}
selection, err := promptSearchSelectionMenu(source, mediaType, sopts.query, results)
if err != nil {
fmt.Fprintf(os.Stderr, "selection error: %v\n", err)
return 2
}
if len(selection) == 0 {
fmt.Println("download cancelled")
return 0
}
mainApp.IgnoreDB = sopts.ignoreDB || gopts.noDB
skippedDownloaded := 0
added := 0
for _, idx := range selection {
item := results[idx]
if !sopts.ignoreDB {
already, checkErr := mainApp.Store.IsDownloaded(ctx, source, item.ID)
if checkErr == nil && already {
skippedDownloaded++
fmt.Printf("skip (already downloaded): id=%s | %s\n", item.ID, item.Title)
continue
}
}
if err = mainApp.AddByID(ctx, source, mediaType, item.ID); err != nil {
fmt.Printf("add failed: id=%s err=%v\n", item.ID, err)
continue
}
added++
}
if added == 0 {
if skippedDownloaded > 0 {
fmt.Println("all selected items were already downloaded (use --force to redownload)")
} else {
fmt.Println("nothing selected to download")
}
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %s\n", errorWithActionableHint(err, gopts))
return 1
}
fmt.Printf("search download complete (%d item(s))\n", added)
return 0
}
func cmdLastFM(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
opts, parseErr := parseLastFMArgs(args, cfg.Session.LastFM.Source, cfg.Session.LastFM.FallbackSource)
if parseErr != nil {
fmt.Fprintf(os.Stderr, "lastfm option error: %v\n", parseErr)
printCommandHelp(os.Stdout, "lastfm")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = gopts.noDB
title, tracks, err := fetchLastFMPlaylist(ctx, cfg.Session.Downloads.VerifySSL, opts.PlaylistURL)
if err != nil {
fmt.Fprintf(os.Stderr, "lastfm parse error: %v\n", err)
return 1
}
if len(tracks) == 0 {
fmt.Println("no tracks found in playlist")
return 0
}
fmt.Printf("lastfm playlist: %s (%d tracks)\n", title, len(tracks))
resolvedTracks, err := resolveLastFMTracks(ctx, mainApp, opts, tracks)
if err != nil {
fmt.Fprintf(os.Stderr, "lastfm resolve error: %v\n", err)
return 1
}
if len(resolvedTracks) == 0 {
fmt.Println("no lastfm tracks resolved")
return 0
}
playlistID := fmt.Sprintf("lastfm:%s", strings.ToLower(strings.ReplaceAll(title, " ", "_")))
refs := make([]app.PlaylistTrackRef, 0, len(resolvedTracks))
for _, item := range resolvedTracks {
refs = append(refs, app.PlaylistTrackRef{Source: item.Source, ID: item.ID})
}
if addErr := mainApp.AddMixedPlaylistByTrackRefs(ctx, playlistID, title, refs); addErr != nil {
fmt.Printf("playlist queue failed: err=%v\n", addErr)
fmt.Println("no lastfm playlists queued")
return 0
}
fmt.Printf("queued lastfm playlist: %s (%d tracks)\n", title, len(refs))
if len(refs) == 0 {
fmt.Println("no lastfm playlists queued")
return 0
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %v\n", err)
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %v\n", err)
return 1
}
fmt.Printf("lastfm rip complete (%d track(s) across 1 playlist)\n", len(resolvedTracks))
return 0
}
+310
View File
@@ -0,0 +1,310 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/provider"
)
// Developer smoke commands, listed by "rip dev-help". These exercise a single
// provider call and print the result; they are not part of the supported CLI.
func cmdSoundcloudSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip soundcloud-smoke <soundcloud_url>")
return 2
}
meta, err := fetchSoundcloudOEmbed(ctx, cfg.Session.Downloads.VerifySSL, strings.TrimSpace(args[0]))
if err != nil {
fmt.Fprintf(os.Stderr, "soundcloud smoke error: %v\n", err)
return 1
}
fmt.Printf("soundcloud oembed ok: title=%q author=%q provider=%q\n", jsonutil.StringFromAny(meta["title"]), jsonutil.StringFromAny(meta["author_name"]), jsonutil.StringFromAny(meta["provider_name"]))
return 0
}
func cmdQobuzSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip qobuz-smoke <track_id> [quality]")
return 2
}
opts, err := parseSmokeOptions(args[1:], 1, 4)
if err != nil {
fmt.Fprintf(os.Stderr, "option error: %v\n", err)
return 2
}
if opts.qualitySet {
cfg.Session.Qobuz.Quality = opts.quality
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "qobuz")
if err != nil {
fmt.Fprintf(os.Stderr, "qobuz login error: %v\n", err)
return 1
}
trackID := args[0]
meta, err := provider.GetMetadata(ctx, trackID, "track")
if err != nil {
fmt.Fprintf(os.Stderr, "metadata error: %v\n", err)
return 1
}
title, _ := meta["title"].(string)
d, err := provider.GetDownloadable(ctx, trackID, cfg.Session.Qobuz.Quality)
if err != nil {
fmt.Fprintf(os.Stderr, "downloadable error: %v\n", err)
return 1
}
fmt.Printf("qobuz ok: title=%q quality=%d ext=%s\n", title, cfg.Session.Qobuz.Quality, d.Extension)
fmt.Printf("stream_url=%s\n", d.URL)
return 0
}
func cmdQobuzConvertRipSmoke(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int {
if len(args) < 2 {
fmt.Println("usage: rip qobuz-convert-rip-smoke <track_id> <codec> [quality] [--force|--ignore-db]")
return 2
}
opts, err := parseSmokeOptions(args[2:], 1, 4)
if err != nil {
fmt.Fprintf(os.Stderr, "option error: %v\n", err)
return 2
}
if opts.qualitySet {
cfg.Session.Qobuz.Quality = opts.quality
}
cfg.Session.Conversion.Enabled = true
cfg.Session.Conversion.Codec = strings.ToUpper(strings.TrimSpace(args[1]))
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
mainApp.IgnoreDB = opts.ignoreDB || gopts.noDB
trackID := args[0]
if err = mainApp.AddByID(ctx, "qobuz", "track", trackID); err != nil {
fmt.Fprintf(os.Stderr, "add error: %v\n", err)
return 1
}
if err = mainApp.Resolve(ctx); err != nil {
fmt.Fprintf(os.Stderr, "resolve error: %v\n", err)
return 1
}
if err = mainApp.Rip(ctx); err != nil {
fmt.Fprintf(os.Stderr, "rip error: %v\n", err)
return 1
}
fmt.Printf("qobuz convert rip smoke complete (codec=%s)\n", cfg.Session.Conversion.Codec)
return 0
}
func cmdQobuzSearchSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip qobuz-search-smoke <query>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "qobuz")
if err != nil {
fmt.Fprintf(os.Stderr, "qobuz login error: %v\n", err)
return 1
}
query := strings.Join(args, " ")
pages, err := provider.Search(ctx, "album", query, 10)
if err != nil {
fmt.Fprintf(os.Stderr, "search error: %v\n", err)
return 1
}
for _, page := range pages {
albums, ok := page["albums"].(map[string]any)
if !ok {
continue
}
items, ok := albums["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
item, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(item["id"])
title := jsonutil.StringFromAny(item["title"])
version := jsonutil.StringFromAny(item["version"])
bitDepth := jsonutil.StringFromAny(item["maximum_bit_depth"])
sr := jsonutil.StringFromAny(item["maximum_sampling_rate"])
if version != "" {
title = title + " (" + version + ")"
}
fmt.Printf("album_id=%s | %s | %sB-%skHz\n", id, title, bitDepth, sr)
}
}
return 0
}
func cmdTidalSearchSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip tidal-search-smoke <query>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "tidal")
if err != nil {
fmt.Fprintf(os.Stderr, "tidal login error: %v\n", err)
return 1
}
query := strings.Join(args, " ")
pages, err := provider.Search(ctx, "album", query, 10)
if err != nil {
fmt.Fprintf(os.Stderr, "search error: %v\n", err)
return 1
}
for _, page := range pages {
items, ok := page["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
wrapper, ok := raw.(map[string]any)
if !ok {
continue
}
item, ok := wrapper["item"].(map[string]any)
if !ok {
item = wrapper
}
fmt.Printf("album_id=%s | %s\n", jsonutil.StringFromAny(item["id"]), jsonutil.StringFromAny(item["title"]))
}
}
return 0
}
func cmdTidalMetadataSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 2 {
fmt.Println("usage: rip tidal-metadata-smoke <track|album|playlist|artist> <id>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
provider, err := mainApp.GetLoggedInProvider(ctx, "tidal")
if err != nil {
fmt.Fprintf(os.Stderr, "tidal login error: %v\n", err)
return 1
}
mediaType := args[0]
itemID := args[1]
meta, err := provider.GetMetadata(ctx, itemID, mediaType)
if err != nil {
fmt.Fprintf(os.Stderr, "metadata error: %v\n", err)
return 1
}
title := jsonutil.StringFromAny(meta["title"])
if title == "" {
title = jsonutil.StringFromAny(meta["name"])
}
trackCount := 0
if tracksMap, ok := meta["tracks"].(map[string]any); ok {
if items, ok := tracksMap["items"].([]map[string]any); ok {
trackCount = len(items)
} else if anyItems, ok := tracksMap["items"].([]any); ok {
trackCount = len(anyItems)
}
}
fmt.Printf("tidal metadata ok: type=%s id=%s title=%q tracks=%d\n", mediaType, itemID, title, trackCount)
return 0
}
func cmdTidalVideoSmoke(ctx context.Context, cfg *config.Config, _ globalOptions, args []string) int {
if len(args) < 1 {
fmt.Println("usage: rip tidal-video-smoke <video_id>")
return 2
}
mainApp, err := app.New(cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "app init error: %v\n", err)
return 1
}
defer func() { _ = mainApp.Close() }()
providerClient, err := mainApp.GetLoggedInProvider(ctx, "tidal")
if err != nil {
fmt.Fprintf(os.Stderr, "tidal login error: %v\n", err)
return 1
}
videoProvider, ok := providerClient.(interface {
GetVideoDownloadable(context.Context, string) (*provider.Downloadable, error)
})
if !ok {
fmt.Fprintln(os.Stderr, "tidal provider does not support video downloadable")
return 1
}
videoID := strings.TrimSpace(args[0])
meta, err := providerClient.GetMetadata(ctx, videoID, "video")
if err != nil {
fmt.Fprintf(os.Stderr, "video metadata error: %v\n", err)
return 1
}
d, err := videoProvider.GetVideoDownloadable(ctx, videoID)
if err != nil {
fmt.Fprintf(os.Stderr, "video downloadable error: %v\n", err)
return 1
}
title := jsonutil.StringFromAny(meta["title"])
if title == "" {
title = jsonutil.StringFromAny(meta["name"])
}
fmt.Printf("tidal video ok: id=%s title=%q ext=%s\n", videoID, title, d.Extension)
fmt.Printf("stream_url=%s\n", d.URL)
return 0
}
+248
View File
@@ -0,0 +1,248 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"streamrip-go/internal/config"
)
// testConfig returns a config rooted in a temp dir, so command functions can
// run without touching the user's real config or download folder.
func testConfig(t *testing.T) *config.Config {
t.Helper()
tmp := t.TempDir()
d := config.DefaultConfigData()
d.Downloads.Folder = filepath.Join(tmp, "downloads")
d.Database.DownloadsEnabled = false
d.Database.FailedDownloadsEnabled = false
d.CLI.ProgressBars = false
return &config.Config{Path: filepath.Join(tmp, "config.toml"), File: d, Session: d}
}
// The command functions became testable when main() was split into run() int
// plus one function per command; previously they were arms of a 1196-line
// switch that called os.Exit directly.
func TestCommandsRejectMissingArguments(t *testing.T) {
tests := []struct {
name string
run func(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int
args []string
}{
{"url with no args", cmdURL, nil},
{"file with no args", cmdFile, nil},
{"config with no args", cmdConfig, nil},
{"database with no args", cmdDatabase, nil},
{"database without browse", cmdDatabase, []string{"downloads"}},
{"database browse without table", cmdDatabase, []string{"browse"}},
{"id with too few args", cmdID, []string{"qobuz", "album"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := tc.run(context.Background(), testConfig(t), globalOptions{}, tc.args)
if got != 2 {
t.Fatalf("exit code = %d, want 2 (usage error)", got)
}
})
}
}
func TestCmdDatabaseRejectsUnknownTable(t *testing.T) {
if got := cmdDatabase(context.Background(), testConfig(t), globalOptions{}, []string{"browse", "bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdConfigRejectsUnknownSubcommand(t *testing.T) {
if got := cmdConfig(context.Background(), testConfig(t), globalOptions{}, []string{"bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdConfigPathPrintsPath(t *testing.T) {
cfg := testConfig(t)
out := captureStdout(t, func() {
if got := cmdConfig(context.Background(), cfg, globalOptions{}, []string{"path"}); got != 0 {
t.Errorf("exit code = %d, want 0", got)
}
})
if !strings.Contains(out, cfg.Path) {
t.Fatalf("output %q does not contain the config path %q", out, cfg.Path)
}
}
func TestCmdConfigPathRejectsExtraArgument(t *testing.T) {
if got := cmdConfig(context.Background(), testConfig(t), globalOptions{}, []string{"path", "extra"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdFileReportsMissingFile(t *testing.T) {
missing := filepath.Join(t.TempDir(), "nope.txt")
if got := cmdFile(context.Background(), testConfig(t), globalOptions{}, []string{missing}); got != 1 {
t.Fatalf("exit code = %d, want 1 (read error)", got)
}
}
func TestCmdFileRejectsUnknownOption(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "urls.txt")
if err := os.WriteFile(path, []byte("https://play.qobuz.com/album/abc\n"), 0o600); err != nil {
t.Fatal(err)
}
if got := cmdFile(context.Background(), testConfig(t), globalOptions{}, []string{path, "--bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestCmdURLRejectsInvalidURL(t *testing.T) {
// Nothing resolvable is queued, so this reports "nothing to rip" and exits 0.
out := captureStdout(t, func() {
if got := cmdURL(context.Background(), testConfig(t), globalOptions{}, []string{"not-a-url"}); got != 0 {
t.Errorf("exit code = %d, want 0", got)
}
})
if !strings.Contains(out, "nothing to rip") {
t.Fatalf("output %q should report that nothing was queued", out)
}
}
func TestCmdIDRejectsQualityBelowSourceFloor(t *testing.T) {
// Qobuz has no quality 0.
if got := cmdID(context.Background(), testConfig(t), globalOptions{}, []string{"qobuz", "album", "abc", "0"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
if got := cmdID(context.Background(), testConfig(t), globalOptions{}, []string{"beatport", "album", "abc", "0"}); got != 2 {
t.Fatalf("beatport exit code = %d, want 2", got)
}
}
func TestCmdIDRejectsUnknownOption(t *testing.T) {
if got := cmdID(context.Background(), testConfig(t), globalOptions{}, []string{"qobuz", "album", "abc", "--bogus"}); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
}
func TestSmokeCommandsRejectMissingArguments(t *testing.T) {
tests := []struct {
name string
run func(ctx context.Context, cfg *config.Config, gopts globalOptions, args []string) int
args []string
}{
{"soundcloud", cmdSoundcloudSmoke, nil},
{"qobuz", cmdQobuzSmoke, nil},
{"qobuz convert", cmdQobuzConvertRipSmoke, []string{"id"}},
{"qobuz search", cmdQobuzSearchSmoke, nil},
{"tidal search", cmdTidalSearchSmoke, nil},
{"tidal metadata", cmdTidalMetadataSmoke, []string{"track"}},
{"tidal video", cmdTidalVideoSmoke, nil},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := tc.run(context.Background(), testConfig(t), globalOptions{}, tc.args); got != 2 {
t.Fatalf("exit code = %d, want 2", got)
}
})
}
}
// captureStdout swaps os.Stdout for a pipe while fn runs. The commands print
// user-facing output with fmt.Println rather than taking a writer.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
orig := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
done := make(chan string, 1)
go func() {
var sb strings.Builder
buf := make([]byte, 4096)
for {
n, readErr := r.Read(buf)
if n > 0 {
sb.Write(buf[:n])
}
if readErr != nil {
break
}
}
done <- sb.String()
}()
fn()
_ = w.Close()
os.Stdout = orig
out := <-done
_ = r.Close()
return out
}
func TestVersionOutput(t *testing.T) {
var sb strings.Builder
printVersion(&sb)
out := sb.String()
if !strings.HasPrefix(out, "rip ") {
t.Errorf("version output should start with \"rip \": %q", out)
}
for _, want := range []string{"go:", "target:"} {
if !strings.Contains(out, want) {
t.Errorf("version output missing %q: %q", want, out)
}
}
}
func TestParseGlobalArgsVersionFlag(t *testing.T) {
for _, flag := range []string{"--version", "-V"} {
opts, err := parseGlobalArgs([]string{flag})
if err != nil {
t.Fatalf("parseGlobalArgs(%q) error = %v", flag, err)
}
if !opts.version {
t.Errorf("parseGlobalArgs(%q) did not set version", flag)
}
}
}
func TestApplyQualityOverrideCoversEverySource(t *testing.T) {
cfg := testConfig(t)
applyQualityOverride(cfg, 4)
got := map[string]int{
"qobuz": cfg.Session.Qobuz.Quality,
"tidal": cfg.Session.Tidal.Quality,
"deezer": cfg.Session.Deezer.Quality,
"soundcloud": cfg.Session.Soundcloud.Quality,
"yandex": cfg.Session.Yandex.Quality,
"beatport": cfg.Session.Beatport.Quality,
}
for source, q := range got {
if q != 4 {
t.Errorf("%s quality = %d, want 4; -q must reach every source", source, q)
}
}
}
func TestApplyQualityOverrideClampsSourceFloor(t *testing.T) {
cfg := testConfig(t)
applyQualityOverride(cfg, 0)
if cfg.Session.Qobuz.Quality != 1 {
t.Errorf("qobuz quality = %d, want it clamped to 1", cfg.Session.Qobuz.Quality)
}
if cfg.Session.Beatport.Quality != 1 {
t.Errorf("beatport quality = %d, want it clamped to 1", cfg.Session.Beatport.Quality)
}
if cfg.Session.Tidal.Quality != 0 {
t.Errorf("tidal quality = %d, want 0 preserved", cfg.Session.Tidal.Quality)
}
}
+161
View File
@@ -0,0 +1,161 @@
package main
import (
"fmt"
"io"
"strings"
)
func isHelpArg(arg string) bool {
return arg == "-h" || arg == "--help"
}
func commandWantsHelp(args []string) bool {
for _, arg := range args {
if isHelpArg(arg) {
return true
}
}
return false
}
func printMainHelp(w io.Writer) {
fmt.Fprint(w, `streamrip-go
Usage:
rip [global options] <command> [command options]
rip help [command]
Commands:
url Rip one or more URLs
file Rip URLs or IDs from a file
id Rip by source, media type, and ID
search Search a provider and download selected results
lastfm Import a Last.fm playlist
config Manage configuration
database Inspect download databases
Global options:
-f, --folder <path> Override downloads folder
-q, --quality <0-4> Override provider quality
-c, --codec <codec> Convert after download: ALAC, FLAC, OGG, MP3, AAC
-ndb, --no-db Ignore download database
--no-progress Disable progress bars
--no-ssl-verify Disable TLS verification
--config-path <path> Use a custom config file
-v, -vv Verbose logging
-V, --version Show version
-h, --help Show help
Run 'rip help <command>' for command-specific help.
`)
}
func printCommandHelp(w io.Writer, command string) bool {
switch strings.ToLower(strings.TrimSpace(command)) {
case "url":
fmt.Fprint(w, `Usage:
rip url <url...> [--force|--ignore-db]
Rip one or more provider URLs.
Examples:
rip url https://www.beatport.com/release/example/123
rip url https://play.qobuz.com/album/abc --force
Options:
--force, --ignore-db Redownload even if already in the database
`)
case "file":
fmt.Fprint(w, `Usage:
rip file <path> [--force|--ignore-db]
Rip URLs or JSON ID entries from a file.
Options:
--force, --ignore-db Redownload even if already in the database
`)
case "id":
fmt.Fprint(w, `Usage:
rip id <source> <media-type> <id> [quality] [--force|--ignore-db]
Rip an item by provider ID.
Sources:
qobuz, tidal, deezer, yandex, beatport, soundcloud
Media types:
track, album, playlist, artist, label, chart, video
Options:
quality Override quality for this rip, 0-4
--force, --ignore-db Redownload even if already in the database
`)
case "search":
fmt.Fprint(w, `Usage:
rip search <source> <media-type> <query...> [options]
Search a provider and optionally download selected results.
Sources:
qobuz, tidal, deezer, yandex, beatport, soundcloud
Media types:
track, album, playlist, artist, label, chart, video
Options:
--limit N Maximum results to show
--first Download the first result without prompting
--no-download Show results only
--output-file <path> Write results as JSON
--force, --ignore-db Redownload even if already in the database
`)
case "lastfm":
fmt.Fprint(w, `Usage:
rip lastfm [--source SOURCE] [--fallback-source SOURCE] <playlist_url>
Import a Last.fm playlist and resolve tracks through configured providers.
Options:
--source SOURCE Primary lookup source
--fallback-source SOURCE Fallback lookup source
`)
case "config":
fmt.Fprint(w, `Usage:
rip config <open|reset|path> [options]
Manage configuration.
Commands:
open Open the config file in an editor
reset Reset config to defaults
path Print the config path
Options:
-v, --vim Use Vim for 'rip config open'
-y, --yes Confirm 'rip config reset' without prompting
`)
case "database":
fmt.Fprint(w, `Usage:
rip database browse <downloads|failed>
Inspect local download databases.
`)
case "dev-help":
printDeveloperHelp(w)
default:
return false
}
return true
}
func printDeveloperHelp(w io.Writer) {
fmt.Fprint(w, `Developer smoke commands:
soundcloud-smoke
qobuz-smoke, qobuz-convert-rip-smoke, qobuz-search-smoke
tidal-search-smoke, tidal-metadata-smoke, tidal-video-smoke
The per-media-type rip smoke commands were removed; use the equivalent
rip id <source> <media-type> <id>
`)
}
+235
View File
@@ -0,0 +1,235 @@
package main
import (
"bufio"
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"os/exec"
"regexp"
"runtime"
"strings"
"streamrip-go/internal/app"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/urlparse"
)
type fileIDItem struct {
Source string
MediaType string
ID string
}
type failedRow struct {
Source string
MediaType string
ID string
}
type lastFMOptions struct {
Source string
FallbackSource string
PlaylistURL string
}
type lastFMTrack struct {
Title string
Artist string
}
type resolvedLastFMTrack struct {
Source string
ID string
Query string
}
var (
lastFMTitleTagsRe = regexp.MustCompile(`<a\b[^>]*\btitle=(?:"([^"]+)"|'([^']+)')`)
lastFMDataTrackArtistRe = regexp.MustCompile(`data-track-name=(?:"([^"]+)"|'([^']+)')[^>]*data-artist-name=(?:"([^"]+)"|'([^']+)')`)
lastFMTotalTracksRe = regexp.MustCompile(`data-playlisting-entry-count="(\d+)"`)
lastFMPlaylistTitleRe = regexp.MustCompile(`<h1[^>]*class="[^"]*playlisting-playlist-header-title[^"]*"[^>]*>([^<]+)</h1>`)
lastFMMirrorTitleRe = regexp.MustCompile(`^Title:\s*(.+?)\s+\|`)
lastFMMirrorLinkTextRe = regexp.MustCompile(`\[([^\]]+)\]\(`)
errLastFMInvalidSource = "unsupported source"
)
func addURLToQueue(ctx context.Context, mainApp *app.Main, raw string) bool {
parsed := urlparse.Parse(raw)
if parsed == nil {
fmt.Printf("invalid: %s\n", raw)
return false
}
if parsed.Kind != urlparse.KindGeneric && parsed.Kind != urlparse.KindSoundcloud {
fmt.Printf("not yet supported: %s (kind=%s)\n", raw, parsed.Kind)
return false
}
if parsed.Source != "qobuz" && parsed.Source != "tidal" && parsed.Source != "deezer" && parsed.Source != "yandex" && parsed.Source != "beatport" && parsed.Source != "soundcloud" {
fmt.Printf("provider not yet implemented: source=%s url=%s\n", parsed.Source, raw)
return false
}
if err := mainApp.AddByID(ctx, parsed.Source, parsed.MediaType, parsed.ID); err != nil {
fmt.Printf("add failed: source=%s type=%s id=%s err=%v\n", parsed.Source, parsed.MediaType, parsed.ID, err)
return false
}
return true
}
func parseFileInput(content []byte) ([]fileIDItem, []string, int, bool, error) {
trimmed := strings.TrimSpace(string(content))
if trimmed == "" {
return nil, nil, 0, false, nil
}
var parsed any
if err := json.Unmarshal([]byte(trimmed), &parsed); err == nil {
arr, ok := parsed.([]any)
if !ok {
return nil, nil, 0, true, fmt.Errorf("json input must be an array of objects")
}
items := make([]fileIDItem, 0, len(arr))
for i, raw := range arr {
entry, ok := raw.(map[string]any)
if !ok {
return nil, nil, 0, true, fmt.Errorf("json item %d must be an object", i+1)
}
source := strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(entry["source"])))
mediaType := strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(entry["media_type"])))
if mediaType == "" {
mediaType = strings.ToLower(strings.TrimSpace(jsonutil.StringFromAny(entry["mediaType"])))
}
id := strings.TrimSpace(jsonutil.StringFromAny(entry["id"]))
if source == "" || mediaType == "" || id == "" {
return nil, nil, 0, true, fmt.Errorf("json item %d missing source/media_type/id", i+1)
}
items = append(items, fileIDItem{Source: source, MediaType: mediaType, ID: id})
}
return items, nil, 0, true, nil
}
parts := strings.Fields(trimmed)
if len(parts) == 0 {
return nil, nil, 0, false, nil
}
seen := make(map[string]struct{}, len(parts))
urls := make([]string, 0, len(parts))
repeated := 0
for _, raw := range parts {
if _, ok := seen[raw]; ok {
repeated++
continue
}
seen[raw] = struct{}{}
urls = append(urls, raw)
}
return nil, urls, repeated, false, nil
}
func promptYesNo(prompt string) (bool, error) {
reader := bufio.NewReader(os.Stdin)
fmt.Print(prompt)
line, err := reader.ReadString('\n')
if err != nil {
return false, err
}
line = strings.ToLower(strings.TrimSpace(line))
return line == "y" || line == "yes", nil
}
func openConfigInEditor(path string, vim bool) error {
launch := func(name string, args ...string) error {
cmd := exec.Command(name, args...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Run()
}
if vim {
if p, err := exec.LookPath("nvim"); err == nil {
return launch(p, path)
}
if p, err := exec.LookPath("vim"); err == nil {
return launch(p, path)
}
}
if editor := strings.TrimSpace(os.Getenv("EDITOR")); editor != "" {
parts := strings.Fields(editor)
if len(parts) > 0 {
return launch(parts[0], append(parts[1:], path)...)
}
}
switch runtime.GOOS {
case "darwin":
return launch("open", path)
case "windows":
return launch("cmd", "/c", "start", "", path)
default:
if p, err := exec.LookPath("xdg-open"); err == nil {
return launch(p, path)
}
return fmt.Errorf("could not find an editor (set $EDITOR or install xdg-open)")
}
}
func listDownloadsRows(path string) ([]string, error) {
db, err := sql.Open("sqlite", path)
if err != nil {
return nil, err
}
defer func() { _ = db.Close() }()
rows, err := db.Query(`SELECT id FROM downloads ORDER BY rowid`)
if err != nil {
if isNoSuchTableErr(err) {
return []string{}, nil
}
return nil, err
}
defer func() { _ = rows.Close() }()
out := []string{}
for rows.Next() {
var id string
if err = rows.Scan(&id); err != nil {
return nil, err
}
out = append(out, id)
}
return out, rows.Err()
}
func listFailedRows(path string) ([]failedRow, error) {
db, err := sql.Open("sqlite", path)
if err != nil {
return nil, err
}
defer func() { _ = db.Close() }()
rows, err := db.Query(`SELECT source, media_type, id FROM failed_downloads ORDER BY rowid`)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
out := []failedRow{}
for rows.Next() {
var r failedRow
if err = rows.Scan(&r.Source, &r.MediaType, &r.ID); err != nil {
return nil, err
}
out = append(out, r)
}
return out, rows.Err()
}
func isNoSuchTableErr(err error) bool {
if err == nil {
return false
}
return strings.Contains(strings.ToLower(err.Error()), "no such table")
}
+422
View File
@@ -0,0 +1,422 @@
package main
import (
"context"
"encoding/json"
"fmt"
"html"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"streamrip-go/internal/app"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/netutil"
"streamrip-go/internal/provider"
)
func parseLastFMArgs(args []string, defaultSource, defaultFallback string) (lastFMOptions, error) {
opts := lastFMOptions{Source: strings.ToLower(strings.TrimSpace(defaultSource)), FallbackSource: strings.ToLower(strings.TrimSpace(defaultFallback))}
for i := 0; i < len(args); i++ {
switch args[i] {
case "-s", "--source":
if i+1 >= len(args) {
return lastFMOptions{}, fmt.Errorf("--source requires a value")
}
opts.Source = strings.ToLower(strings.TrimSpace(args[i+1]))
i++
case "-fs", "--fallback-source":
if i+1 >= len(args) {
return lastFMOptions{}, fmt.Errorf("--fallback-source requires a value")
}
opts.FallbackSource = strings.ToLower(strings.TrimSpace(args[i+1]))
i++
default:
if strings.HasPrefix(args[i], "-") {
return lastFMOptions{}, fmt.Errorf("unknown option %q", args[i])
}
if opts.PlaylistURL != "" {
return lastFMOptions{}, fmt.Errorf("unexpected extra argument %q", args[i])
}
opts.PlaylistURL = strings.TrimSpace(args[i])
}
}
if opts.Source == "" {
opts.Source = "qobuz"
}
if opts.PlaylistURL == "" {
return lastFMOptions{}, fmt.Errorf("missing playlist url")
}
if !isValidLastFMPlaylistURL(opts.PlaylistURL) {
return lastFMOptions{}, fmt.Errorf("playlist url must be a last.fm url")
}
if !isAllowedSearchSource(opts.Source) {
return lastFMOptions{}, fmt.Errorf("%s %q", errLastFMInvalidSource, opts.Source)
}
if opts.FallbackSource != "" && !isAllowedSearchSource(opts.FallbackSource) {
return lastFMOptions{}, fmt.Errorf("%s %q", errLastFMInvalidSource, opts.FallbackSource)
}
return opts, nil
}
func isValidLastFMPlaylistURL(raw string) bool {
u, err := url.Parse(strings.TrimSpace(raw))
if err != nil || u == nil || u.Host == "" {
return false
}
s := strings.ToLower(strings.TrimSpace(u.Scheme))
if s != "http" && s != "https" {
return false
}
h := strings.ToLower(strings.TrimPrefix(strings.TrimSpace(u.Host), "www."))
if h != "last.fm" && !strings.HasSuffix(h, ".last.fm") {
return false
}
p := strings.ToLower(strings.TrimSpace(u.Path))
return strings.Contains(p, "/playlists/")
}
func fetchLastFMPlaylist(ctx context.Context, verifySSL bool, playlistURL string) (string, []lastFMTrack, error) {
parsed, err := url.Parse(playlistURL)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return "", nil, fmt.Errorf("invalid playlist url")
}
if !isValidLastFMPlaylistURL(playlistURL) {
return "", nil, fmt.Errorf("invalid playlist url")
}
client := netutil.NewHTTPClient(30*time.Second, verifySSL, 0)
page1, err := fetchLastFMPlaylistPage(ctx, client, parsed, 1)
if err != nil {
return fetchLastFMPlaylistViaMirror(ctx, verifySSL, playlistURL)
}
title, total, err := extractLastFMPlaylistInfo(page1)
if err != nil {
return fetchLastFMPlaylistViaMirror(ctx, verifySSL, playlistURL)
}
tracks := extractLastFMTitleArtistPairs(page1)
if total <= len(tracks) || total <= 50 {
if len(tracks) > total && total > 0 {
tracks = tracks[:total]
}
return title, tracks, nil
}
remaining := total - 50
lastPage := 1 + remaining/50
if remaining%50 != 0 {
lastPage++
}
for page := 2; page <= lastPage; page++ {
body, fetchErr := fetchLastFMPlaylistPage(ctx, client, parsed, page)
if fetchErr != nil {
return "", nil, fetchErr
}
tracks = append(tracks, extractLastFMTitleArtistPairs(body)...)
}
if len(tracks) > total {
tracks = tracks[:total]
}
return title, tracks, nil
}
func fetchLastFMPlaylistViaMirror(ctx context.Context, verifySSL bool, playlistURL string) (string, []lastFMTrack, error) {
client := netutil.NewHTTPClient(30*time.Second, verifySSL, 0)
all := make([]lastFMTrack, 0, 200)
title := ""
for page := 1; page <= 50; page++ {
body, err := fetchLastFMPlaylistMirrorPage(ctx, client, playlistURL, page)
if err != nil {
if page == 1 {
return "", nil, err
}
break
}
pageTitle, tracks := extractLastFMTracksFromMirrorMarkdown(body)
if title == "" && strings.TrimSpace(pageTitle) != "" {
title = pageTitle
}
if len(tracks) == 0 {
break
}
all = append(all, tracks...)
if !strings.Contains(strings.ToLower(body), "show more") {
break
}
}
if len(all) == 0 {
return "", nil, fmt.Errorf("could not parse playlist tracks from last.fm")
}
if strings.TrimSpace(title) == "" {
title = "Last.fm Playlist"
}
return title, all, nil
}
func fetchLastFMPlaylistMirrorPage(ctx context.Context, client *http.Client, playlistURL string, page int) (string, error) {
u, err := url.Parse(playlistURL)
if err != nil {
return "", err
}
if page > 1 {
q := u.Query()
q.Set("page", strconv.Itoa(page))
u.RawQuery = q.Encode()
}
raw := u.String()
raw = strings.TrimPrefix(raw, "https://")
raw = strings.TrimPrefix(raw, "http://")
mirrorURL := "https://r.jina.ai/http://" + raw
req, err := http.NewRequestWithContext(ctx, http.MethodGet, mirrorURL, nil)
if err != nil {
return "", err
}
req.Header.Set("User-Agent", "streamrip-go/0")
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("lastfm mirror request failed: status %d", resp.StatusCode)
}
b, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
return string(b), nil
}
func fetchLastFMPlaylistPage(ctx context.Context, client *http.Client, parsed *url.URL, page int) (string, error) {
u := *parsed
if page > 1 {
q := u.Query()
q.Set("page", strconv.Itoa(page))
u.RawQuery = q.Encode()
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
if err != nil {
return "", err
}
req.Header.Set("User-Agent", "streamrip-go/0")
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("lastfm request failed: status %d", resp.StatusCode)
}
b, err := io.ReadAll(resp.Body)
if err != nil {
return "", err
}
return string(b), nil
}
func extractLastFMPlaylistInfo(page string) (string, int, error) {
titleMatch := lastFMPlaylistTitleRe.FindStringSubmatch(page)
if len(titleMatch) < 2 {
return "", 0, fmt.Errorf("could not parse playlist title")
}
totalMatch := lastFMTotalTracksRe.FindStringSubmatch(page)
if len(totalMatch) < 2 {
return "", 0, fmt.Errorf("could not parse total track count")
}
total, err := strconv.Atoi(totalMatch[1])
if err != nil {
return "", 0, fmt.Errorf("invalid total track count")
}
return html.UnescapeString(strings.TrimSpace(titleMatch[1])), total, nil
}
func extractLastFMTitleArtistPairs(page string) []lastFMTrack {
dataPairs := lastFMDataTrackArtistRe.FindAllStringSubmatch(page, -1)
if len(dataPairs) > 0 {
out := make([]lastFMTrack, 0, len(dataPairs))
for _, m := range dataPairs {
title := html.UnescapeString(strings.TrimSpace(jsonutil.FirstNonEmpty(m[1], m[2])))
artist := html.UnescapeString(strings.TrimSpace(jsonutil.FirstNonEmpty(m[3], m[4])))
if title == "" || artist == "" {
continue
}
out = append(out, lastFMTrack{Title: title, Artist: artist})
}
if len(out) > 0 {
return out
}
}
titles := lastFMTitleTagsRe.FindAllStringSubmatch(page, -1)
out := make([]lastFMTrack, 0, len(titles)/2)
for i := 0; i+1 < len(titles); i += 2 {
titleRaw := strings.TrimSpace(jsonutil.FirstNonEmpty(titles[i][1], titles[i][2]))
artistRaw := strings.TrimSpace(jsonutil.FirstNonEmpty(titles[i+1][1], titles[i+1][2]))
if strings.EqualFold(titleRaw, "Play on YouTube") || strings.EqualFold(artistRaw, "Play on YouTube") {
continue
}
title := html.UnescapeString(titleRaw)
artist := html.UnescapeString(artistRaw)
if title == "" || artist == "" {
continue
}
out = append(out, lastFMTrack{Title: title, Artist: artist})
}
return out
}
func extractLastFMTracksFromMirrorMarkdown(md string) (string, []lastFMTrack) {
lines := strings.Split(strings.ReplaceAll(md, "\r\n", "\n"), "\n")
title := ""
tracks := make([]lastFMTrack, 0, 100)
for _, line := range lines {
line = strings.TrimSpace(line)
if title == "" {
if m := lastFMMirrorTitleRe.FindStringSubmatch(line); len(m) >= 2 {
title = strings.TrimSpace(html.UnescapeString(m[1]))
}
}
if !strings.HasPrefix(line, "|") || !strings.Contains(strings.ToLower(line), "play track") {
continue
}
cols := splitMarkdownTableRow(line)
if len(cols) < 6 {
continue
}
trackName := markdownLinkText(cols[3])
artist := markdownLinkText(cols[4])
if strings.TrimSpace(trackName) == "" || strings.TrimSpace(artist) == "" {
continue
}
tracks = append(tracks, lastFMTrack{Title: html.UnescapeString(strings.TrimSpace(trackName)), Artist: html.UnescapeString(strings.TrimSpace(artist))})
}
return title, tracks
}
func splitMarkdownTableRow(line string) []string {
trimmed := strings.TrimSpace(line)
trimmed = strings.TrimPrefix(trimmed, "|")
trimmed = strings.TrimSuffix(trimmed, "|")
parts := strings.Split(trimmed, "|")
out := make([]string, 0, len(parts))
for _, p := range parts {
out = append(out, strings.TrimSpace(p))
}
return out
}
func markdownLinkText(cell string) string {
m := lastFMMirrorLinkTextRe.FindStringSubmatch(cell)
if len(m) >= 2 {
return m[1]
}
return strings.TrimSpace(cell)
}
func resolveLastFMTracks(ctx context.Context, mainApp *app.Main, opts lastFMOptions, tracks []lastFMTrack) ([]resolvedLastFMTrack, error) {
primary, err := mainApp.GetLoggedInProvider(ctx, opts.Source)
if err != nil {
return nil, fmt.Errorf("%s login error: %w", opts.Source, err)
}
var fallback provider.Client
if opts.FallbackSource != "" && opts.FallbackSource != opts.Source {
fallback, err = mainApp.GetLoggedInProvider(ctx, opts.FallbackSource)
if err != nil {
return nil, fmt.Errorf("%s login error: %w", opts.FallbackSource, err)
}
}
found := 0
failed := 0
resolved := make([]resolvedLastFMTrack, 0, len(tracks))
for i, tr := range tracks {
query := strings.TrimSpace(tr.Title + " " + tr.Artist)
id, source, searchErr := searchLastFMTrack(ctx, opts, primary, fallback, query)
if searchErr != nil {
failed++
fmt.Printf("[%d/%d] search failed: %s (%v)\n", i+1, len(tracks), query, searchErr)
continue
}
if id == "" {
failed++
fmt.Printf("[%d/%d] no result: %s\n", i+1, len(tracks), query)
continue
}
resolved = append(resolved, resolvedLastFMTrack{Source: source, ID: id, Query: query})
found++
fmt.Printf("[%d/%d] found: %s (%s)\n", i+1, len(tracks), query, source)
}
fmt.Printf("lastfm resolve complete: %d found, %d failed\n", found, failed)
return resolved, nil
}
func fetchSoundcloudOEmbed(ctx context.Context, verifySSL bool, trackURL string) (map[string]any, error) {
parsed, err := url.Parse(trackURL)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return nil, fmt.Errorf("invalid soundcloud url")
}
q := url.Values{}
q.Set("format", "json")
q.Set("url", trackURL)
endpoint := "https://soundcloud.com/oembed?" + q.Encode()
client := netutil.NewHTTPClient(20*time.Second, verifySSL, 0)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", "streamrip-go/0.1")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("soundcloud oembed failed: status %d", resp.StatusCode)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
out := map[string]any{}
if err = json.Unmarshal(body, &out); err != nil {
return nil, err
}
return out, nil
}
func searchLastFMTrack(ctx context.Context, opts lastFMOptions, primary provider.Client, fallback provider.Client, query string) (string, string, error) {
pages, err := primary.Search(ctx, "track", query, 1)
if err == nil {
results := normalizeSearchResults(opts.Source, "track", pages)
if len(results) > 0 {
return results[0].ID, opts.Source, nil
}
}
if fallback != nil {
pages, fbErr := fallback.Search(ctx, "track", query, 1)
if fbErr != nil {
if err != nil {
return "", "", fmt.Errorf("primary=%v fallback=%v", err, fbErr)
}
return "", "", fbErr
}
results := normalizeSearchResults(opts.FallbackSource, "track", pages)
if len(results) > 0 {
return results[0].ID, opts.FallbackSource, nil
}
}
if err != nil {
return "", "", err
}
return "", "", nil
}
+72 -2401
View File
File diff suppressed because it is too large. Load diff
+188 -1
View File
@@ -1,7 +1,11 @@
package main
import (
"bytes"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
@@ -91,6 +95,31 @@ func TestParseLastFMArgsOptions(t *testing.T) {
}
}
func TestParseLastFMArgsRejectsNonLastFMURL(t *testing.T) {
_, err := parseLastFMArgs([]string{"https://example.com/user/x/playlists/123"}, "qobuz", "")
if err == nil || !strings.Contains(strings.ToLower(err.Error()), "last.fm") {
t.Fatalf("expected last.fm url validation error, got %v", err)
}
}
func TestIsValidLastFMPlaylistURL(t *testing.T) {
if !isValidLastFMPlaylistURL("https://www.last.fm/user/x/playlists/123") {
t.Fatalf("expected canonical last.fm playlist url to be valid")
}
if !isValidLastFMPlaylistURL("http://last.fm/user/x/playlists/123") {
t.Fatalf("expected http last.fm playlist url to be valid")
}
if isValidLastFMPlaylistURL("ftp://last.fm/user/x/playlists/123") {
t.Fatalf("expected non-http scheme to be invalid")
}
if isValidLastFMPlaylistURL("https://example.com/user/x/playlists/123") {
t.Fatalf("expected non-last.fm host to be invalid")
}
if isValidLastFMPlaylistURL("https://www.last.fm/user/x/library") {
t.Fatalf("expected non-playlist last.fm url to be invalid")
}
}
func TestExtractLastFMPlaylistInfoAndPairs(t *testing.T) {
html := `<h1 class="playlisting-playlist-header-title">Road &amp; Rain</h1>
<div data-playlisting-entry-count="2"></div>
@@ -119,6 +148,49 @@ func TestExtractLastFMPlaylistInfoAndPairs(t *testing.T) {
}
}
func TestExtractLastFMPlaylistInfoFlexibleClass(t *testing.T) {
html := `<h1 id="x" class="foo playlisting-playlist-header-title bar">Road &amp; Rain</h1>
<div data-playlisting-entry-count="1"></div>`
title, total, err := extractLastFMPlaylistInfo(html)
if err != nil {
t.Fatalf("extractLastFMPlaylistInfo() error = %v", err)
}
if title != "Road & Rain" || total != 1 {
t.Fatalf("unexpected parsed values: title=%q total=%d", title, total)
}
}
func TestExtractLastFMTitleArtistPairsSingleQuotes(t *testing.T) {
html := `<a href='/music/a' title='Dreams'></a>
<a href='/music/b' title='Fleetwood Mac'></a>`
pairs := extractLastFMTitleArtistPairs(html)
if len(pairs) != 1 {
t.Fatalf("pairs len = %d, want 1", len(pairs))
}
if pairs[0].Title != "Dreams" || pairs[0].Artist != "Fleetwood Mac" {
t.Fatalf("unexpected pair: %+v", pairs[0])
}
}
func TestExtractLastFMTitleArtistPairsSkipsPlayOnYouTubeNoise(t *testing.T) {
html := `<a href="https://www.youtube.com/watch?v=1" data-track-name="Won&#39;t Forget You" data-artist-name="Shouse" title="Play on YouTube">Play track</a>
<a href="/music/Shouse/_/Won%27t+Forget+You" title="Won&#39;t Forget You"></a>
<a href="/music/Shouse" title="Shouse"></a>
<a href="https://www.youtube.com/watch?v=2" data-track-name="EYES" data-artist-name="The Blaze" title="Play on YouTube">Play track</a>
<a href="/music/The+Blaze/_/EYES" title="EYES"></a>
<a href="/music/The+Blaze" title="The Blaze"></a>`
pairs := extractLastFMTitleArtistPairs(html)
if len(pairs) != 2 {
t.Fatalf("pairs len = %d, want 2", len(pairs))
}
if pairs[0].Title != "Won't Forget You" || pairs[0].Artist != "Shouse" {
t.Fatalf("unexpected first pair: %+v", pairs[0])
}
if pairs[1].Title != "EYES" || pairs[1].Artist != "The Blaze" {
t.Fatalf("unexpected second pair: %+v", pairs[1])
}
}
func TestParseGlobalArgsNoDBBeforeCommand(t *testing.T) {
opts, err := parseGlobalArgs([]string{"-ndb", "url", "https://play.qobuz.com/album/0004228000522"})
if err != nil {
@@ -135,6 +207,58 @@ func TestParseGlobalArgsNoDBBeforeCommand(t *testing.T) {
}
}
func TestParseGlobalArgsHelp(t *testing.T) {
opts, err := parseGlobalArgs([]string{"--help"})
if err != nil {
t.Fatalf("parseGlobalArgs() error = %v", err)
}
if !opts.help {
t.Fatalf("expected help=true")
}
}
func TestCommandWantsHelp(t *testing.T) {
if !commandWantsHelp([]string{"https://example.com", "-h"}) {
t.Fatalf("expected command help")
}
if commandWantsHelp([]string{"https://example.com"}) {
t.Fatalf("did not expect command help")
}
}
func TestMainHelpHidesDeveloperCommands(t *testing.T) {
var buf bytes.Buffer
printMainHelp(&buf)
out := buf.String()
if !strings.Contains(out, "rip help [command]") || !strings.Contains(out, "Commands:") {
t.Fatalf("main help missing expected text: %s", out)
}
if strings.Contains(out, "dev-help") || strings.Contains(out, "smoke") {
t.Fatalf("main help should hide developer commands: %s", out)
}
}
func TestCommandHelpURL(t *testing.T) {
var buf bytes.Buffer
if !printCommandHelp(&buf, "url") {
t.Fatalf("expected url help")
}
out := buf.String()
if !strings.Contains(out, "rip url <url...>") || !strings.Contains(out, "--ignore-db") {
t.Fatalf("url help missing expected text: %s", out)
}
}
func TestPrintCommandHelpRejectsUnknown(t *testing.T) {
var buf bytes.Buffer
if printCommandHelp(&buf, "nope") {
t.Fatalf("unexpected help for unknown command")
}
if buf.Len() != 0 {
t.Fatalf("unexpected output for unknown command: %s", buf.String())
}
}
func TestParseGlobalArgsAllOfficialFlags(t *testing.T) {
opts, err := parseGlobalArgs([]string{
"--config-path", "/tmp/custom.toml",
@@ -156,7 +280,7 @@ func TestParseGlobalArgsAllOfficialFlags(t *testing.T) {
if !opts.noDB || !opts.qualitySet || opts.quality != 3 || !opts.codecSet || opts.codec != "VORBIS" {
t.Fatalf("unexpected quality/codec/db opts: %+v", opts)
}
if !opts.noProgress || !opts.noSSLVerify || !opts.verbose {
if !opts.noProgress || !opts.noSSLVerify || opts.verbose != 1 {
t.Fatalf("unexpected boolean opts: %+v", opts)
}
if opts.command != "search" {
@@ -188,6 +312,17 @@ func TestExtractLastFMTracksFromMirrorMarkdown(t *testing.T) {
}
}
func TestExtractLastFMTracksFromMirrorMarkdownLowercasePlayTrack(t *testing.T) {
md := `Title: My Playlist | user playlists | Last.fm
| Play | Image | Loved | Name | Artist name | Buy | Options | Duration |
| --- | --- | --- | --- | --- | --- | --- | --- |
| [play track](https://x) | [img](https://i) | x | [Song A](https://a) | [Artist A](https://aa) | | | 3:00 |`
_, tracks := extractLastFMTracksFromMirrorMarkdown(md)
if len(tracks) != 1 {
t.Fatalf("tracks len = %d, want 1", len(tracks))
}
}
func TestParseSearchArgsAllowsFirstAndOutputFileButCallerCanReject(t *testing.T) {
opts, err := parseSearchArgs([]string{"q", "--first", "--output-file", "/tmp/out.json"}, 20)
if err != nil {
@@ -198,6 +333,58 @@ func TestParseSearchArgsAllowsFirstAndOutputFileButCallerCanReject(t *testing.T)
}
}
func TestParseSearchArgsRejectsUnknownOption(t *testing.T) {
_, err := parseSearchArgs([]string{"query", "--bogus"}, 20)
if err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("expected unknown option error, got %v", err)
}
}
func TestParseSearchArgsSupportsDoubleDashTerminator(t *testing.T) {
opts, err := parseSearchArgs([]string{"--limit", "10", "--", "--weird", "track"}, 20)
if err != nil {
t.Fatalf("parseSearchArgs() error = %v", err)
}
if opts.limit != 10 {
t.Fatalf("limit = %d, want 10", opts.limit)
}
if opts.query != "--weird track" {
t.Fatalf("query = %q, want %q", opts.query, "--weird track")
}
}
func TestWriteSearchResultsToFileCreatesParentDirectory(t *testing.T) {
tmp := t.TempDir()
out := filepath.Join(tmp, "nested", "search", "results.json")
results := []searchResult{{ID: "1", Title: "Dreams"}}
if err := writeSearchResultsToFile("qobuz", "track", results, out); err != nil {
t.Fatalf("writeSearchResultsToFile() error = %v", err)
}
if _, err := os.Stat(out); err != nil {
t.Fatalf("expected output file, stat error = %v", err)
}
}
func TestNormalizeSearchResultsDedupesByID(t *testing.T) {
pages := []map[string]any{
{"tracks": map[string]any{"items": []any{
map[string]any{"id": "1", "title": "Dreams", "artist": map[string]any{"name": "Fleetwood Mac"}},
map[string]any{"id": "1", "title": "Dreams", "artist": map[string]any{"name": "Fleetwood Mac"}},
}}},
{"tracks": map[string]any{"items": []any{
map[string]any{"id": "2", "title": "Go Your Own Way", "artist": map[string]any{"name": "Fleetwood Mac"}},
map[string]any{"id": "1", "title": "Dreams", "artist": map[string]any{"name": "Fleetwood Mac"}},
}}},
}
results := normalizeSearchResults("qobuz", "track", pages)
if len(results) != 2 {
t.Fatalf("len(results)=%d want 2", len(results))
}
if results[0].ID != "1" || results[1].ID != "2" {
t.Fatalf("unexpected IDs order: %+v", results)
}
}
func TestErrorWithActionableHintForSSL(t *testing.T) {
err := errors.New("x509: certificate signed by unknown authority")
msg := errorWithActionableHint(err, globalOptions{})
+618
View File
@@ -0,0 +1,618 @@
package main
import (
"bufio"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"github.com/AlecAivazis/survey/v2"
"streamrip-go/internal/jsonutil"
)
type searchResult struct {
ID string
Title string
Artist string
Album string
Date string
Releases int
TrackCount int
Explicit bool
}
type searchOptions struct {
query string
limit int
ignoreDB bool
noDownload bool
first bool
outputFile string
}
func parseSearchArgs(args []string, defaultLimit int) (searchOptions, error) {
if defaultLimit <= 0 {
defaultLimit = 20
}
limit := defaultLimit
parts := make([]string, 0, len(args))
ignoreDB := false
noDownload := false
first := false
outputFile := ""
for i := 0; i < len(args); i++ {
if args[i] == "--" {
if i+1 < len(args) {
parts = append(parts, args[i+1:]...)
}
break
}
switch args[i] {
case "--force", "--ignore-db":
ignoreDB = true
continue
case "--no-download":
noDownload = true
continue
case "--first":
first = true
continue
case "--output-file":
if i+1 >= len(args) {
return searchOptions{}, fmt.Errorf("--output-file requires a path")
}
outputFile = strings.TrimSpace(args[i+1])
if outputFile == "" {
return searchOptions{}, fmt.Errorf("--output-file requires a non-empty path")
}
i++
continue
case "--num-results":
if i+1 >= len(args) {
return searchOptions{}, fmt.Errorf("--num-results requires a value")
}
v, err := strconv.Atoi(args[i+1])
if err != nil || v <= 0 {
return searchOptions{}, fmt.Errorf("invalid --num-results value %q", args[i+1])
}
limit = v
i++
continue
}
if args[i] == "--limit" {
if i+1 >= len(args) {
return searchOptions{}, fmt.Errorf("--limit requires a value")
}
v, err := strconv.Atoi(args[i+1])
if err != nil || v <= 0 {
return searchOptions{}, fmt.Errorf("invalid --limit value %q", args[i+1])
}
limit = v
i++
continue
}
if strings.HasPrefix(args[i], "-") {
return searchOptions{}, fmt.Errorf("unknown option %q", args[i])
}
parts = append(parts, args[i])
}
return searchOptions{
query: strings.TrimSpace(strings.Join(parts, " ")),
limit: limit,
ignoreDB: ignoreDB,
noDownload: noDownload,
first: first,
outputFile: outputFile,
}, nil
}
func promptSearchSelectionMenu(source, mediaType, query string, results []searchResult) ([]int, error) {
if len(results) == 0 {
return nil, nil
}
prevTemplate := survey.MultiSelectQuestionTemplate
survey.MultiSelectQuestionTemplate = streamripLikeSearchTemplate
defer func() {
survey.MultiSelectQuestionTemplate = prevTemplate
}()
labels := make([]string, 0, len(results))
labelToIndex := map[string]int{}
for i, r := range results {
label := formatSearchOptionLabel(i+1, mediaType, r)
labels = append(labels, label)
labelToIndex[label] = i
}
pageSize := len(labels)
if pageSize < 15 {
pageSize = 15
}
if pageSize > 30 {
pageSize = 30
}
selected := []string{}
prompt := &survey.MultiSelect{
Message: fmt.Sprintf("Results for %s '%s' from %s", mediaType, query, jsonutil.TitleCase(source)),
Help: "SPACE - select, ENTER - download, ESC - exit",
Options: labels,
Description: func(value string, index int) string {
resultIndex, ok := labelToIndex[value]
if !ok || resultIndex < 0 || resultIndex >= len(results) {
return ""
}
return formatSearchDetails(mediaType, results[resultIndex])
},
PageSize: pageSize,
}
if err := survey.AskOne(prompt, &selected); err != nil {
if strings.Contains(strings.ToLower(err.Error()), "interrupt") {
return nil, nil
}
return nil, err
}
if len(selected) == 0 {
return nil, nil
}
out := make([]int, 0, len(selected))
for _, label := range selected {
if idx, ok := labelToIndex[label]; ok {
out = append(out, idx)
}
}
for i := 1; i < len(out); i++ {
for j := i; j > 0 && out[j] < out[j-1]; j-- {
out[j], out[j-1] = out[j-1], out[j]
}
}
return out, nil
}
const streamripLikeSearchTemplate = `
{{ color "default+hb"}}{{ .Message }}{{color "reset"}}
{{- if .Help }}
{{ .Help }}
{{- end }}
{{- range $ix, $option := .PageEntries }}
{{ if eq $.SelectedIndex $ix }}>{{ else }} {{ end }} [{{ if index $.Checked $option.Index }}x{{ else }} {{ end }}] {{ $option.Value }}
{{- end }}
preview
{{- if gt (len .PageEntries) 0 }}
{{ $current := index .PageEntries .SelectedIndex }}
{{ $.GetDescription $current }}
{{- end }}
`
func writeSearchResultsToFile(source, mediaType string, results []searchResult, path string) error {
type outItem struct {
Source string `json:"source"`
MediaType string `json:"media_type"`
ID string `json:"id"`
Title string `json:"title"`
}
out := make([]outItem, 0, len(results))
for _, r := range results {
out = append(out, outItem{Source: source, MediaType: mediaType, ID: r.ID, Title: r.Title})
}
b, err := json.MarshalIndent(out, "", " ")
if err != nil {
return err
}
dir := filepath.Dir(path)
if dir != "" && dir != "." {
if err = os.MkdirAll(dir, 0o755); err != nil {
return err
}
}
return os.WriteFile(path, b, 0o644)
}
func isAllowedSearchSource(source string) bool {
return source == "qobuz" || source == "tidal" || source == "deezer" || source == "yandex" || source == "beatport" || source == "soundcloud"
}
func isAllowedMediaType(mediaType string) bool {
switch mediaType {
case "track", "album", "playlist", "artist", "label", "video", "chart":
return true
default:
return false
}
}
func promptSearchInteractive(defaultLimit int) (string, string, searchOptions, error) {
reader := bufio.NewReader(os.Stdin)
read := func(prompt string) (string, error) {
fmt.Print(prompt)
line, err := reader.ReadString('\n')
if err != nil {
return "", err
}
return strings.TrimSpace(line), nil
}
for {
source, err := read("Source [qobuz/tidal/deezer/yandex/beatport/soundcloud]: ")
if err != nil {
return "", "", searchOptions{}, err
}
source = strings.ToLower(source)
if !isAllowedSearchSource(source) {
fmt.Println("Invalid source.")
continue
}
mediaType, err := read("Type [track/album/playlist/artist/label/video]: ")
if err != nil {
return "", "", searchOptions{}, err
}
mediaType = strings.ToLower(mediaType)
if !isAllowedMediaType(mediaType) {
fmt.Println("Invalid media type.")
continue
}
if source == "soundcloud" && mediaType != "track" && mediaType != "playlist" {
fmt.Println("SoundCloud search supports track and playlist only.")
continue
}
if source == "yandex" && mediaType != "track" && mediaType != "album" && mediaType != "playlist" && mediaType != "artist" {
fmt.Println("Yandex search supports track, album, playlist, and artist only.")
continue
}
if source == "beatport" && mediaType != "track" && mediaType != "album" && mediaType != "label" {
fmt.Println("Beatport search supports track, album, and label only.")
continue
}
query, err := read("Query: ")
if err != nil {
return "", "", searchOptions{}, err
}
if strings.TrimSpace(query) == "" {
fmt.Println("Query cannot be empty.")
continue
}
limitRaw, err := read(fmt.Sprintf("Limit [%d]: ", defaultLimit))
if err != nil {
return "", "", searchOptions{}, err
}
limit := defaultLimit
if strings.TrimSpace(limitRaw) != "" {
v, convErr := strconv.Atoi(limitRaw)
if convErr != nil || v <= 0 {
fmt.Println("Invalid limit.")
continue
}
limit = v
}
return source, mediaType, searchOptions{query: query, limit: limit}, nil
}
}
func normalizeSearchResults(source, mediaType string, pages []map[string]any) []searchResult {
results := make([]searchResult, 0)
seen := map[string]struct{}{}
appendUnique := func(r searchResult) {
if strings.TrimSpace(r.ID) == "" || strings.TrimSpace(r.Title) == "" {
return
}
key := r.ID
if _, ok := seen[key]; ok {
return
}
seen[key] = struct{}{}
results = append(results, r)
}
for _, page := range pages {
switch source {
case "qobuz":
key := mediaType + "s"
bucket, ok := page[key].(map[string]any)
if !ok {
continue
}
items, ok := bucket["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(itm["id"])
title := jsonutil.StringFromAny(itm["title"])
if title == "" {
title = jsonutil.StringFromAny(itm["name"])
}
if version := jsonutil.StringFromAny(itm["version"]); version != "" {
title += " (" + version + ")"
}
artist := jsonutil.NestedString(itm, "artist", "name")
if artist == "" {
artist = jsonutil.NestedString(itm, "performer", "name")
}
album := jsonutil.NestedString(itm, "album", "title")
trackCount := jsonutil.IntFromAny(itm["tracks_count"])
if trackCount == 0 {
trackCount = jsonutil.IntFromAny(itm["track_count"])
}
explicit := searchBool(itm["parental_warning"])
date := jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(itm["release_date_original"]),
jsonutil.StringFromAny(itm["release_date"]),
jsonutil.StringFromAny(itm["releaseDate"]),
jsonutil.StringFromAny(itm["streamStartDate"]),
jsonutil.NestedString(itm, "album", "release_date_original"),
jsonutil.NestedString(itm, "album", "release_date"),
)
releases := 0
if mediaType == "artist" {
releases = jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(itm["albums_count"]),
jsonutil.IntFromAny(itm["albumsCount"]),
jsonutil.IntFromAny(jsonutil.NestedAny(itm, "albums", "total")),
)
}
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
}
case "tidal":
items, ok := page["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
if wrapped, ok := itm["item"].(map[string]any); ok {
itm = wrapped
}
id := jsonutil.StringFromAny(itm["id"])
title := jsonutil.StringFromAny(itm["title"])
if title == "" {
title = jsonutil.StringFromAny(itm["name"])
}
artist := jsonutil.NestedString(itm, "artist", "name")
if artist == "" {
if artists, ok := itm["artists"].([]any); ok && len(artists) > 0 {
if a0, ok := artists[0].(map[string]any); ok {
artist = jsonutil.StringFromAny(a0["name"])
}
}
}
album := jsonutil.NestedString(itm, "album", "title")
trackCount := jsonutil.IntFromAny(itm["numberOfTracks"])
if trackCount == 0 {
trackCount = jsonutil.IntFromAny(itm["tracks_count"])
}
explicit := searchBool(itm["explicit"])
date := jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(itm["releaseDate"]),
jsonutil.StringFromAny(itm["streamStartDate"]),
jsonutil.StringFromAny(itm["release_date"]),
jsonutil.NestedString(itm, "album", "releaseDate"),
jsonutil.NestedString(itm, "album", "release_date"),
)
releases := 0
if mediaType == "artist" {
releases = jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(itm["numberOfAlbums"]),
jsonutil.IntFromAny(itm["albums_count"]),
jsonutil.IntFromAny(itm["albumsCount"]),
jsonutil.IntFromAny(jsonutil.NestedAny(itm, "albums", "total")),
)
}
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
}
case "deezer":
key := mediaType + "s"
bucket, ok := page[key].(map[string]any)
if !ok {
continue
}
items, ok := bucket["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(itm["id"])
title := jsonutil.StringFromAny(itm["title"])
if title == "" {
title = jsonutil.StringFromAny(itm["name"])
}
artist := jsonutil.NestedString(itm, "artist", "name")
album := jsonutil.NestedString(itm, "album", "title")
trackCount := jsonutil.IntFromAny(itm["nb_tracks"])
explicit := searchBool(itm["explicit_lyrics"])
date := jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(itm["release_date"]),
jsonutil.NestedString(itm, "album", "release_date"),
)
releases := 0
if mediaType == "artist" {
releases = jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(itm["nb_album"]),
jsonutil.IntFromAny(itm["albums_count"]),
jsonutil.IntFromAny(itm["numberOfAlbums"]),
)
}
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
}
case "soundcloud":
items, ok := page["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(itm["id"])
title := jsonutil.StringFromAny(itm["title"])
artist := jsonutil.NestedString(itm, "artist", "name")
trackCount := jsonutil.IntFromAny(itm["tracks_count"])
date := jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(itm["release_date"]),
jsonutil.StringFromAny(itm["display_date"]),
)
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Date: date, TrackCount: trackCount})
}
case "yandex":
items, ok := page["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(itm["id"])
title := jsonutil.StringFromAny(itm["title"])
if title == "" {
title = jsonutil.StringFromAny(itm["name"])
}
artist := jsonutil.NestedString(itm, "artist", "name")
if artist == "" {
artist = jsonutil.NestedString(itm, "performer", "name")
}
album := jsonutil.NestedString(itm, "album", "title")
trackCount := jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(itm["trackCount"]),
jsonutil.IntFromAny(itm["track_count"]),
jsonutil.IntFromAny(itm["tracks_count"]),
)
explicit := searchBool(itm["explicit"])
date := jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(itm["release_date"]),
jsonutil.StringFromAny(itm["releaseDate"]),
jsonutil.NestedString(itm, "album", "release_date"),
jsonutil.NestedString(itm, "album", "releaseDate"),
)
releases := 0
if mediaType == "artist" {
releases = jsonutil.FirstPositiveInt(
jsonutil.IntFromAny(itm["albums_count"]),
jsonutil.IntFromAny(itm["numberOfAlbums"]),
jsonutil.IntFromAny(jsonutil.NestedAny(itm, "albums", "total")),
)
}
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, Releases: releases, TrackCount: trackCount, Explicit: explicit})
}
case "beatport":
items, ok := page["items"].([]any)
if !ok {
continue
}
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
id := jsonutil.StringFromAny(itm["id"])
title := jsonutil.StringFromAny(itm["title"])
if title == "" {
title = jsonutil.StringFromAny(itm["name"])
}
if version := jsonutil.StringFromAny(itm["version"]); version != "" {
title += " (" + version + ")"
}
artist := jsonutil.NestedString(itm, "artist", "name")
album := jsonutil.NestedString(itm, "album", "title")
trackCount := jsonutil.FirstPositiveInt(jsonutil.IntFromAny(itm["tracks_count"]), jsonutil.IntFromAny(itm["track_count"]))
date := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(itm["release_date_original"]), jsonutil.StringFromAny(itm["release_date"]), jsonutil.NestedString(itm, "album", "release_date_original"))
appendUnique(searchResult{ID: id, Title: title, Artist: artist, Album: album, Date: date, TrackCount: trackCount})
}
}
}
return results
}
func formatSearchDetails(mediaType string, r searchResult) string {
date := strings.TrimSpace(r.Date)
if date == "" {
date = "Unknown"
}
lines := []string{}
switch mediaType {
case "album":
lines = append(lines, "Date released:", date)
if r.TrackCount > 0 {
lines = append(lines, "", fmt.Sprintf("%d Tracks", r.TrackCount))
}
case "artist":
if strings.TrimSpace(r.Title) != "" {
lines = append(lines, r.Title)
}
if r.Releases > 0 {
lines = append(lines, "", fmt.Sprintf("%d Releases", r.Releases))
}
case "track":
lines = append(lines, "Released on:", date)
case "playlist":
if r.TrackCount > 0 {
lines = append(lines, fmt.Sprintf("%d Tracks", r.TrackCount))
}
default:
if strings.TrimSpace(r.Title) != "" {
lines = append(lines, r.Title)
}
if strings.TrimSpace(r.Artist) != "" {
lines = append(lines, strings.TrimSpace(r.Artist))
}
if strings.TrimSpace(r.Album) != "" {
lines = append(lines, strings.TrimSpace(r.Album))
}
if r.TrackCount > 0 {
lines = append(lines, fmt.Sprintf("%d Tracks", r.TrackCount))
}
if r.Explicit {
lines = append(lines, "Explicit")
}
}
lines = append(lines, "", "ID: "+r.ID)
return strings.Join(lines, "\n")
}
func formatSearchOptionLabel(index int, mediaType string, r searchResult) string {
title := strings.TrimSpace(r.Title)
if title == "" {
title = "Unknown"
}
artist := strings.TrimSpace(r.Artist)
switch mediaType {
case "artist", "label":
return fmt.Sprintf("%d. %s", index, title)
default:
if artist == "" {
return fmt.Sprintf("%d. %s", index, title)
}
return fmt.Sprintf("%d. %s by %s", index, title, artist)
}
}
// searchBool is deliberately stricter than jsonutil.BoolFromAny: it accepts
// only a real bool. The explicit/parental-warning fields it reads are proper
// booleans in every provider's search payload, and widening this to coerce
// strings and numbers would change how results are labelled.
func searchBool(v any) bool {
b, ok := v.(bool)
return ok && b
}
+87
View File
@@ -0,0 +1,87 @@
package main
import "testing"
func TestFormatSearchDetailsAlbum(t *testing.T) {
r := searchResult{ID: "9399700017915", Date: "1994-02-01", TrackCount: 18}
got := formatSearchDetails("album", r)
want := "Date released:\n1994-02-01\n\n18 Tracks\n\nID: 9399700017915"
if got != want {
t.Fatalf("unexpected album details:\n%s", got)
}
}
func TestFormatSearchDetailsTrackDefaultsDate(t *testing.T) {
r := searchResult{ID: "3083287"}
got := formatSearchDetails("track", r)
want := "Released on:\nUnknown\n\nID: 3083287"
if got != want {
t.Fatalf("unexpected track details:\n%s", got)
}
}
func TestFormatSearchDetailsArtistWithReleases(t *testing.T) {
r := searchResult{ID: "1863897", Title: "Lost Frequencies", Releases: 23}
got := formatSearchDetails("artist", r)
want := "Lost Frequencies\n\n23 Releases\n\nID: 1863897"
if got != want {
t.Fatalf("unexpected artist details:\n%s", got)
}
}
func TestNormalizeSearchResultsQobuzExtractsDate(t *testing.T) {
pages := []map[string]any{
{"albums": map[string]any{"items": []any{
map[string]any{
"id": "1",
"title": "Master of Reality",
"release_date_original": "1971-07-21",
"artist": map[string]any{"name": "Black Sabbath"},
},
}}},
}
results := normalizeSearchResults("qobuz", "album", pages)
if len(results) != 1 {
t.Fatalf("len(results)=%d want 1", len(results))
}
if results[0].Date != "1971-07-21" {
t.Fatalf("date=%q want 1971-07-21", results[0].Date)
}
}
func TestFormatSearchOptionLabelArtistNoUnknownArtistSuffix(t *testing.T) {
r := searchResult{Title: "Lost Frequencies"}
got := formatSearchOptionLabel(1, "artist", r)
want := "1. Lost Frequencies"
if got != want {
t.Fatalf("label=%q want %q", got, want)
}
}
func TestFormatSearchOptionLabelTrackWithArtist(t *testing.T) {
r := searchResult{Title: "Dreams", Artist: "Fleetwood Mac"}
got := formatSearchOptionLabel(2, "track", r)
want := "2. Dreams by Fleetwood Mac"
if got != want {
t.Fatalf("label=%q want %q", got, want)
}
}
func TestNormalizeSearchResultsQobuzArtistExtractsReleases(t *testing.T) {
pages := []map[string]any{
{"artists": map[string]any{"items": []any{
map[string]any{
"id": "1863897",
"name": "Lost Frequencies",
"albums_count": 23,
},
}}},
}
results := normalizeSearchResults("qobuz", "artist", pages)
if len(results) != 1 {
t.Fatalf("len(results)=%d want 1", len(results))
}
if results[0].Releases != 23 {
t.Fatalf("releases=%d want 23", results[0].Releases)
}
}
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"fmt"
"io"
"runtime"
"runtime/debug"
)
// version is overridden at build time:
//
// go build -ldflags "-X main.version=$(git describe --tags --always --dirty)"
//
// It stays "dev" for a plain `go build`. See the Makefile.
var version = "dev"
// commit and date are optional build-time stamps, also set with -X.
var (
commit = ""
date = ""
)
// buildVersion resolves the version string, preferring the ldflags value and
// falling back to the VCS revision the Go toolchain embeds for `go install`.
func buildVersion() string {
v := version
c := commit
if c == "" {
if info, ok := debug.ReadBuildInfo(); ok {
for _, s := range info.Settings {
if s.Key == "vcs.revision" {
c = s.Value
}
if s.Key == "vcs.time" && date == "" {
date = s.Value
}
}
}
}
if len(c) > 12 {
c = c[:12]
}
if c != "" {
v += " (" + c + ")"
}
return v
}
func printVersion(w io.Writer) {
fmt.Fprintf(w, "rip %s\n", buildVersion())
if date != "" {
fmt.Fprintf(w, "built: %s\n", date)
}
fmt.Fprintf(w, "go: %s\n", runtime.Version())
fmt.Fprintf(w, "target: %s/%s\n", runtime.GOOS, runtime.GOARCH)
}
+214
View File
@@ -0,0 +1,214 @@
[downloads]
# Folder where tracks are downloaded to
folder = "/path/to/StreamripDownloads"
# Put Qobuz albums in a 'Qobuz' folder, Tidal albums in 'Tidal', etc.
source_subdirectories = false
# Put tracks in albums with 2+ discs into subfolders named `Disc N`
disc_subdirectories = true
# Download (and convert) tracks concurrently instead of sequentially
concurrency = true
# The maximum number of tracks to download at once
# Set to -1 for no limit
max_connections = 6
# Max number of API requests per source per minute. Set to -1 for no limit.
#
# This is the usual cause of a large album feeling slow: every track needs a
# stream-URL lookup, so a 68-track album is ~69 requests and at 60/minute that
# is a minute of waiting before the audio is even reached. Requests are
# allowed to burst up to max_connections, so raising this is what lets several
# tracks download at once. Raise it if your provider tolerates it.
requests_per_minute = 60
# Verify SSL certificates for API connections
# Set to false only if certificate verification fails (not recommended)
verify_ssl = true
[qobuz]
# Quality ladder (global scale is 0-4, Qobuz supports 1-4):
# 1 = MP3 320, 2 = FLAC 16/44.1, 3 = FLAC 24-bit <=96kHz, 4 = FLAC 24-bit <=192kHz
quality = 3
# Authenticate using Qobuz auth token instead of email/password hash
use_auth_token = false
# If use_auth_token=true, set your user id. Otherwise set your email.
email_or_userid = ""
# If use_auth_token=true, set your auth token. Otherwise set md5(password).
password_or_token = ""
# Managed automatically by streamrip-go
app_id = ""
# Managed automatically by streamrip-go
secrets = []
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Qobuz rotates theirs and you need to
# patch your install without waiting for a new release.
mobile_app_id = ""
mobile_app_secret = ""
[tidal]
# Quality ladder:
# 0 = LOW (HEAACV1), 1 = HIGH (AACLC), 2 = LOSSLESS (FLAC), 3/4 = HI_RES_LOSSLESS when available
# If a tier is unavailable, Tidal may fall back to the best available stream for that track.
quality = 3
# Prefer Dolby Atmos/immersive stream variants when available
# Disabled by default because stereo FLAC is usually preferred
prefer_atmos = false
# Download videos included in supported Tidal media
download_videos = true
# Session values are managed automatically. Do not modify manually.
user_id = ""
country_code = ""
access_token = ""
refresh_token = ""
# Unix timestamp when access_token expires
token_expiry = 0
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Tidal rotates theirs and you need to
# patch your install without waiting for a new release.
app_client_id = ""
app_client_secret = ""
[deezer]
# Quality ladder:
# 0 = MP3_128, 1 = MP3_320, 2/3/4 = FLAC
quality = 2
# If target quality is unavailable, fallback down quality ladder
lower_quality_if_not_available = true
# Deezer ARL cookie (recommended auth method)
arl = ""
# Optional login alternative when ARL is not provided
email = ""
password = ""
# Optional cached Deezer refresh token. Managed automatically when available.
refresh_token = ""
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Deezer rotates theirs and you need to
# patch your install without waiting for a new release.
api_key = ""
gateway_key = ""
# Must be exactly 16 characters if set.
blowfish_secret = ""
[yandex]
# Quality ladder:
# 0 = LOW (HE-AAC/AAC when available), 1 = HIGH (AAC/MP3 192), 2/3/4 = LOSSLESS when available
quality = 2
# OAuth access token for api.music.yandex.net
access_token = ""
# Cached current account uid. Managed automatically when available.
user_id = ""
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Yandex rotates theirs and you need to
# patch your install without waiting for a new release.
sign_key = ""
legacy_mp3_sign_salt = ""
[beatport]
# Quality ladder:
# 1 = medium (AAC 128), 2 = high (AAC 256), 3/4 = lossless (FLAC 16/44.1)
# medium-hls is intentionally not supported.
quality = 3
# If a release lists more artists than this, use "Various Artists" as album artist.
# Set to -1 to disable this collapse.
various_artists_threshold = 3
# Beatport Streaming account credentials
username = ""
password = ""
# Session values are managed automatically. Do not modify manually.
access_token = ""
refresh_token = ""
# Unix timestamp when access_token expires
token_expiry = 0
# Vendor app credentials. Leave empty to use the values built into
# streamrip-go; set them only if Beatport rotates theirs and you need to
# patch your install without waiting for a new release.
app_client_id = ""
[soundcloud]
# Quality is currently provider-defined (keep 0)
quality = 0
[database]
# Track IDs already downloaded are stored here and skipped next time
downloads_enabled = true
downloads_path = "/path/to/.config/streamrip/downloads.db"
# Failed item IDs are stored here for retry/repair workflows
failed_downloads_enabled = true
failed_downloads_path = "/path/to/.config/streamrip/failed_downloads.db"
[conversion]
# Convert tracks after download
enabled = false
# ALAC, FLAC, OGG, MP3, or AAC
codec = "ALAC"
# In Hz. Audio is downsampled when above this rate.
sampling_rate = 48000
# Applied only when source bit depth is higher than this value
bit_depth = 24
# Used for lossy codecs
lossy_bitrate = 320
[qobuz_filters]
# Filter a Qobuz artist discography (best-effort for other sources)
extras = false
repeats = false
non_albums = false
features = false
non_studio_albums = false
non_remaster = false
[artwork]
# Embed artwork in the audio file
embed = true
# thumbnail, small, large, or original
embed_size = "large"
# If > 0, embedded image max(width, height) in pixels
embed_max_width = -1
# Save artwork as separate jpg file
save_artwork = true
# If > 0, saved image max(width, height) in pixels
saved_max_width = -1
[metadata]
# Set ALBUM metadata to playlist name for playlist items
set_playlist_to_album = true
# Use playlist position as tracknumber for playlist items
renumber_playlist_tracks = true
# Separator used when a provider exposes multiple artists as separate values
artist_separator = "; "
# Metadata fields to exclude from tagging.
#
# "lyrics" is worth knowing about: Qobuz, Tidal and Deezer each serve lyrics
# from a separate endpoint, so fetching them costs one extra API request per
# track and is subject to requests_per_minute above. Set
# exclude = ["lyrics"] if you do not want lyrics and would rather have the
# faster rip.
exclude = []
[filepaths]
# Create folders for single tracks using folder_format template
add_singles_to_folder = false
# Available keys: albumartist, title, year, bit_depth, sampling_rate, id, container, codec, quality, bitrate, albumcomposer
folder_format = "{albumartist} - {title} ({year}) [{container}] [{bit_depth}B-{sampling_rate}kHz]"
# Available keys: id, tracknumber, artist, albumartist, composer, title, albumcomposer, explicit
track_format = "{tracknumber:02}. {artist} - {title}{explicit}"
# Restrict filenames to printable ASCII
restrict_characters = false
# Truncate filenames longer than this value
truncate_to = 120
[lastfm]
# Primary source used to resolve Last.fm playlist tracks
source = "qobuz"
# Fallback source when primary lookup fails
fallback_source = ""
[cli]
# Print informational output like "Downloading <album>"
text_output = true
# Show resolve and download progress bars
progress_bars = true
# Max interactive search results displayed
max_search_results = 100
[misc]
# Metadata used for config compatibility checks
version = "2.2.0"
+824 -361
View File
File diff suppressed because it is too large. Load diff
+1000 -10
View File
File diff suppressed because it is too large. Load diff
+56 -13
View File
@@ -6,10 +6,12 @@ import (
"fmt"
"image"
"image/jpeg"
_ "image/png"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"golang.org/x/image/draw"
@@ -31,6 +33,52 @@ var (
tempDirs = map[string]struct{}{}
)
var (
folderLocksMu sync.Mutex
folderLocks = map[string]*sync.Mutex{}
)
// lockFolder serializes Prepare calls that target the same album folder and
// returns the unlock function.
//
// ripTrack calls Prepare once per track with the same albumFolder for every
// worker in a playlist rip, and the cover handling below is check-then-write.
// Without this, two workers both saw a missing cover.jpg, both created it, and
// the one that failed removed the file the other had just finished writing.
func lockFolder(folder string) func() {
folderLocksMu.Lock()
mu, ok := folderLocks[folder]
if !ok {
mu = &sync.Mutex{}
folderLocks[folder] = mu
}
folderLocksMu.Unlock()
mu.Lock()
return mu.Unlock
}
// fetchAtomic downloads sourceURL to destPath via a unique temp file in the
// same directory, then renames it into place. Downloading straight to destPath
// meant a failed attempt could delete a file another worker had completed.
func fetchAtomic(ctx context.Context, dl Downloader, sourceURL, destPath string, maxWidth int) bool {
tmpPath := fmt.Sprintf("%s.part-%d-%d", destPath, os.Getpid(), atomic.AddUint64(&tmpCounter, 1))
if err := dl.FileNoProgress(ctx, sourceURL, tmpPath); err != nil {
_ = os.Remove(tmpPath)
return false
}
if maxWidth > 0 {
_ = downscaleImage(tmpPath, maxWidth)
}
if err := os.Rename(tmpPath, destPath); err != nil {
_ = os.Remove(tmpPath)
return false
}
return true
}
var tmpCounter uint64
func Prepare(ctx context.Context, dl Downloader, folder string, albumMeta map[string]any, cfg config.ArtworkConfig, forPlaylist bool) (Result, error) {
saveArtwork := cfg.SaveArtwork
if forPlaylist {
@@ -51,30 +99,25 @@ func Prepare(ctx context.Context, dl Downloader, folder string, albumMeta map[st
embedURL = largestURL
}
// Everything below is check-then-write against shared paths.
unlock := lockFolder(folder)
defer unlock()
result := Result{}
if saveArtwork && largestURL != "" {
savedPath := filepath.Join(folder, "cover.jpg")
if fileExists(savedPath) {
result.SavedPath = savedPath
} else if err := dl.FileNoProgress(ctx, largestURL, savedPath); err == nil {
if cfg.SavedMaxWidth > 0 {
_ = downscaleImage(savedPath, cfg.SavedMaxWidth)
}
if fileExists(savedPath) || fetchAtomic(ctx, dl, largestURL, savedPath, cfg.SavedMaxWidth) {
result.SavedPath = savedPath
}
}
if cfg.Embed && embedURL != "" {
embedDir := sessionEmbedDir(folder)
if err := os.MkdirAll(embedDir, 0o755); err == nil {
// 0o700: predictable path under the shared system temp dir.
if err := os.MkdirAll(embedDir, 0o700); err == nil {
registerTempDir(embedDir)
embedPath := filepath.Join(embedDir, embedFilename(embedURL))
if fileExists(embedPath) {
result.EmbedPath = embedPath
} else if err := dl.FileNoProgress(ctx, embedURL, embedPath); err == nil {
if cfg.EmbedMaxWidth > 0 {
_ = downscaleImage(embedPath, cfg.EmbedMaxWidth)
}
if fileExists(embedPath) || fetchAtomic(ctx, dl, embedURL, embedPath, cfg.EmbedMaxWidth) {
result.EmbedPath = embedPath
}
}
+177
View File
@@ -0,0 +1,177 @@
package artwork
import (
"context"
"errors"
"os"
"path/filepath"
"sync"
"testing"
"time"
"streamrip-go/internal/config"
)
// fakeDownloader models the real downloader's behaviour for this race: the
// first attempt at a given output path succeeds, and any later attempt at that
// same path fails and removes its partial output (downloader.go does
// os.Remove(outputPath) on failure). That is exactly how a losing worker used
// to delete the cover a winning worker had just written.
type fakeDownloader struct {
payload []byte
mu sync.Mutex
seen map[string]bool
delay time.Duration
}
func newFakeDownloader(payload []byte, delay time.Duration) *fakeDownloader {
return &fakeDownloader{payload: payload, seen: map[string]bool{}, delay: delay}
}
func (f *fakeDownloader) File(ctx context.Context, sourceURL, outputPath string) error {
return f.FileNoProgress(ctx, sourceURL, outputPath)
}
func (f *fakeDownloader) FileNoProgress(_ context.Context, _, outputPath string) error {
f.mu.Lock()
first := !f.seen[outputPath]
f.seen[outputPath] = true
f.mu.Unlock()
if !first {
_ = os.Remove(outputPath)
return errors.New("download failed")
}
if err := os.MkdirAll(filepath.Dir(outputPath), 0o700); err != nil {
return err
}
// Widen the window between another worker's existence check and this write.
time.Sleep(f.delay)
return os.WriteFile(outputPath, f.payload, 0o600)
}
func artworkCfg() config.ArtworkConfig {
return config.ArtworkConfig{Embed: true, EmbedSize: "large", SaveArtwork: true}
}
func albumMeta() map[string]any {
return map[string]any{"image": map[string]any{
"original": "https://example.invalid/large.jpg",
"large": "https://example.invalid/large.jpg",
}}
}
// TestPrepareConcurrentSameFolder is the regression test for the cover race:
// many workers calling Prepare for one album folder must leave exactly one
// intact cover.jpg. Run with -race.
func TestPrepareConcurrentSameFolder(t *testing.T) {
folder := t.TempDir()
payload := []byte("jpeg-bytes")
dl := newFakeDownloader(payload, 5*time.Millisecond)
const workers = 16
var wg sync.WaitGroup
results := make([]Result, workers)
for i := 0; i < workers; i++ {
wg.Add(1)
go func(idx int) {
defer wg.Done()
res, err := Prepare(context.Background(), dl, folder, albumMeta(), artworkCfg(), false)
if err != nil {
t.Errorf("Prepare: %v", err)
}
results[idx] = res
}(i)
}
wg.Wait()
saved := filepath.Join(folder, "cover.jpg")
got, err := os.ReadFile(saved)
if err != nil {
t.Fatalf("cover.jpg missing after %d concurrent Prepare calls: %v", workers, err)
}
if string(got) != string(payload) {
t.Fatalf("cover.jpg corrupted: got %q want %q", got, payload)
}
// No .part- temp files may survive.
entries, err := os.ReadDir(folder)
if err != nil {
t.Fatal(err)
}
for _, e := range entries {
if filepath.Ext(e.Name()) != ".jpg" {
t.Errorf("leftover temp artifact in album folder: %s", e.Name())
}
}
for i, res := range results {
if res.SavedPath != saved {
t.Errorf("worker %d: SavedPath = %q want %q", i, res.SavedPath, saved)
}
if res.EmbedPath == "" {
t.Errorf("worker %d: EmbedPath empty", i)
}
}
}
// TestPrepareFailedDownloadKeepsExistingCover covers the specific corruption
// mode: a failing download must not remove a cover that is already on disk.
func TestPrepareFailedDownloadKeepsExistingCover(t *testing.T) {
folder := t.TempDir()
saved := filepath.Join(folder, "cover.jpg")
if err := os.WriteFile(saved, []byte("good-cover"), 0o600); err != nil {
t.Fatal(err)
}
// Mark the saved path as already attempted so any download of it fails.
dl := newFakeDownloader([]byte("new"), 0)
dl.seen[saved] = true
if _, err := Prepare(context.Background(), dl, folder, albumMeta(), artworkCfg(), false); err != nil {
t.Fatalf("Prepare: %v", err)
}
got, err := os.ReadFile(saved)
if err != nil {
t.Fatalf("cover.jpg removed: %v", err)
}
if string(got) != "good-cover" {
t.Fatalf("existing cover overwritten: got %q", got)
}
}
// TestFetchAtomicFailureLeavesNoPartial asserts a failed fetch neither creates
// the destination nor leaves a temp file behind.
func TestFetchAtomicFailureLeavesNoPartial(t *testing.T) {
folder := t.TempDir()
dest := filepath.Join(folder, "cover.jpg")
dl := newFakeDownloader([]byte("x"), 0)
dl.seen[dest+".forced"] = true
// Force failure by pre-marking every temp path is impractical, so use a
// downloader that always fails instead.
always := &failingDownloader{}
if ok := fetchAtomic(context.Background(), always, "https://example.invalid/a.jpg", dest, 0); ok {
t.Fatal("fetchAtomic reported success on a failing download")
}
if _, err := os.Stat(dest); !os.IsNotExist(err) {
t.Fatalf("destination should not exist, stat err = %v", err)
}
entries, err := os.ReadDir(folder)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
t.Fatalf("temp files left behind: %v", entries)
}
}
type failingDownloader struct{}
func (failingDownloader) File(context.Context, string, string) error { return errors.New("nope") }
func (failingDownloader) FileNoProgress(_ context.Context, _, outputPath string) error {
_ = os.WriteFile(outputPath, []byte("partial"), 0o600)
return errors.New("nope")
}
+26 -5
View File
@@ -1,6 +1,7 @@
package convert
import (
"context"
"fmt"
"os"
"os/exec"
@@ -27,7 +28,7 @@ var profiles = map[string]profile{
"AAC": {codecLib: "aac", ext: "m4a", lossless: false},
}
func Convert(path string, cfg config.ConversionConfig) (string, error) {
func Convert(ctx context.Context, path string, cfg config.ConversionConfig) (string, error) {
if !cfg.Enabled {
return path, nil
}
@@ -45,20 +46,20 @@ func Convert(path string, cfg config.ConversionConfig) (string, error) {
tmpPath := finalPath + ".tmp." + p.ext
args := buildFFmpegArgs(path, tmpPath, p, cfg)
cmd := exec.Command("ffmpeg", args...)
cmd := exec.CommandContext(ctx, "ffmpeg", args...)
output, err := cmd.CombinedOutput()
if err != nil {
_ = os.Remove(tmpPath)
return path, fmt.Errorf("conversion failed: %w: %s", err, string(output))
}
if path != finalPath {
_ = os.Remove(path)
}
if err = os.Rename(tmpPath, finalPath); err != nil {
_ = os.Remove(tmpPath)
return path, err
}
if path != finalPath {
_ = os.Remove(path)
}
return finalPath, nil
}
@@ -72,6 +73,17 @@ func buildFFmpegArgs(inputPath, outputPath string, p profile, cfg config.Convers
"-c:a", p.codecLib,
}
if supportsAttachedPicture(p.ext) {
args = append(args,
"-map", "0:v:0?",
"-c:v", "mjpeg",
"-disposition:v:0", "attached_pic",
)
if p.ext == "mp3" {
args = append(args, "-id3v2_version", "3")
}
}
if p.lossless {
filter := buildLosslessFilter(cfg)
if filter != "" {
@@ -87,6 +99,15 @@ func buildFFmpegArgs(inputPath, outputPath string, p profile, cfg config.Convers
return args
}
func supportsAttachedPicture(ext string) bool {
switch strings.TrimPrefix(strings.ToLower(ext), ".") {
case "flac", "mp3", "m4a", "mp4":
return true
default:
return false
}
}
func buildLosslessFilter(cfg config.ConversionConfig) string {
parts := make([]string, 0, 2)
if cfg.SamplingRate > 0 {
+61
View File
@@ -1,6 +1,9 @@
package convert
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
@@ -28,6 +31,12 @@ func TestBuildFFmpegArgsLossless(t *testing.T) {
if !strings.Contains(joined, "sample_fmts=s16p|s16") {
t.Fatalf("missing bit depth filter args=%s", joined)
}
if !strings.Contains(joined, "-map 0:v:0?") {
t.Fatalf("missing optional cover map args=%s", joined)
}
if !strings.Contains(joined, "-disposition:v:0 attached_pic") {
t.Fatalf("missing attached_pic disposition args=%s", joined)
}
}
func TestBuildFFmpegArgsLossy(t *testing.T) {
@@ -40,4 +49,56 @@ func TestBuildFFmpegArgsLossy(t *testing.T) {
if !strings.Contains(joined, "-b:a 320k") {
t.Fatalf("missing bitrate args=%s", joined)
}
if !strings.Contains(joined, "-id3v2_version 3") {
t.Fatalf("missing id3v2 args=%s", joined)
}
}
func TestBuildFFmpegArgsNoCoverForOpus(t *testing.T) {
cfg := config.ConversionConfig{Enabled: true, Codec: "OPUS", LossyBitrate: 192}
args := buildFFmpegArgs("in.flac", "out.opus", profiles["OPUS"], cfg)
joined := strings.Join(args, " ")
if strings.Contains(joined, "-map 0:v:0?") {
t.Fatalf("unexpected cover map args=%s", joined)
}
}
func TestConvertKeepsSourceWhenRenameFails(t *testing.T) {
tmp := t.TempDir()
in := filepath.Join(tmp, "song.flac")
if err := os.WriteFile(in, []byte("src"), 0o644); err != nil {
t.Fatalf("write input: %v", err)
}
fakeBin := filepath.Join(tmp, "bin")
if err := os.MkdirAll(fakeBin, 0o755); err != nil {
t.Fatalf("mkdir bin: %v", err)
}
fakeFFmpeg := filepath.Join(fakeBin, "ffmpeg")
script := "#!/bin/sh\nout=\"\"\nfor arg in \"$@\"; do out=\"$arg\"; done\n: > \"$out\"\n"
if err := os.WriteFile(fakeFFmpeg, []byte(script), 0o755); err != nil {
t.Fatalf("write fake ffmpeg: %v", err)
}
t.Setenv("PATH", fakeBin)
finalPath := strings.TrimSuffix(in, filepath.Ext(in)) + ".m4a"
if err := os.Mkdir(finalPath, 0o755); err != nil {
t.Fatalf("mkdir final path: %v", err)
}
cfg := config.ConversionConfig{Enabled: true, Codec: "ALAC"}
out, err := Convert(context.Background(), in, cfg)
if err == nil {
t.Fatalf("expected rename failure")
}
if out != in {
t.Fatalf("returned path = %q, want %q", out, in)
}
if _, statErr := os.Stat(in); statErr != nil {
t.Fatalf("expected source to remain, stat err=%v", statErr)
}
tmpPath := finalPath + ".tmp.m4a"
if _, statErr := os.Stat(tmpPath); !os.IsNotExist(statErr) {
t.Fatalf("expected temp output cleanup, stat err=%v", statErr)
}
}
+186 -9
View File
@@ -1,6 +1,8 @@
package tag
import (
"context"
"encoding/binary"
"fmt"
"os"
"os/exec"
@@ -13,13 +15,16 @@ type Metadata struct {
Title string
Album string
Artist string
Artists []string
AlbumArtist string
OmitDiscTags bool
TrackNumber int
DiscNumber int
TrackTotal int
DiscTotal int
Date string
Genre string
InitialKey string
Comment string
Description string
Lyrics string
@@ -41,16 +46,17 @@ func New() *Tagger {
return &Tagger{}
}
func (t *Tagger) TagFLAC(path string, meta Metadata, coverPath string) error {
func (t *Tagger) TagFLAC(ctx context.Context, path string, meta Metadata, coverPath string) error {
if _, err := exec.LookPath("ffmpeg"); err != nil {
return fmt.Errorf("ffmpeg not found: %w", err)
}
ext := strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")
forceMP4Muxer := shouldForceMP4Muxer(ctx, path, ext)
tmpPath := taggedTempPath(path)
runTag := func(cover string) ([]byte, error) {
args := buildFFmpegArgs(path, tmpPath, meta, cover, ext)
cmd := exec.Command("ffmpeg", args...)
args := buildFFmpegArgs(path, tmpPath, meta, cover, ext, forceMP4Muxer)
cmd := exec.CommandContext(ctx, "ffmpeg", args...)
return cmd.CombinedOutput()
}
@@ -68,11 +74,14 @@ func (t *Tagger) TagFLAC(path string, meta Metadata, coverPath string) error {
_ = os.Remove(tmpPath)
return err
}
if err = applyMultiValueFLACTags(path, meta); err != nil {
return err
}
return nil
}
func buildFFmpegArgs(inputPath, outputPath string, meta Metadata, coverPath, ext string) []string {
func buildFFmpegArgs(inputPath, outputPath string, meta Metadata, coverPath, ext string, forceMP4Muxer bool) []string {
args := []string{"-y", "-i", inputPath}
withCover := coverPath != "" && fileExists(coverPath) && supportsAttachedPicture(ext)
if withCover {
@@ -101,11 +110,38 @@ func buildFFmpegArgs(inputPath, outputPath string, meta Metadata, coverPath, ext
}
args = append(args, "-metadata", k+"="+v)
}
if meta.OmitDiscTags {
args = append(args,
"-metadata", "disc=",
"-metadata", "disk=",
"-metadata", "disctotal=",
"-metadata", "totaldiscs=",
)
}
if forceMP4Muxer {
args = append(args, "-f", "mp4")
}
args = append(args, outputPath)
return args
}
func shouldForceMP4Muxer(ctx context.Context, path, ext string) bool {
switch strings.TrimPrefix(strings.ToLower(ext), ".") {
case "m4a", "mp4":
default:
return false
}
if _, err := exec.LookPath("ffprobe"); err != nil {
return false
}
out, err := exec.CommandContext(ctx, "ffprobe", "-v", "error", "-select_streams", "a:0", "-show_entries", "stream=codec_name", "-of", "default=nokey=1:noprint_wrappers=1", path).Output()
if err != nil {
return false
}
return strings.EqualFold(strings.TrimSpace(string(out)), "flac")
}
func taggedTempPath(path string) string {
ext := filepath.Ext(path)
if ext == "" {
@@ -131,6 +167,7 @@ func toTags(meta Metadata) map[string]string {
"album_artist": meta.AlbumArtist,
"date": meta.Date,
"genre": meta.Genre,
"INITIALKEY": meta.InitialKey,
"comment": meta.Comment,
"description": meta.Description,
"lyrics": meta.Lyrics,
@@ -146,11 +183,7 @@ func toTags(meta Metadata) map[string]string {
"source_artist_id": meta.SourceArtistID,
}
if meta.TrackNumber > 0 {
if meta.TrackTotal > 0 {
tags["track"] = fmt.Sprintf("%02d/%02d", meta.TrackNumber, meta.TrackTotal)
} else {
tags["track"] = fmt.Sprintf("%02d", meta.TrackNumber)
}
tags["track"] = fmt.Sprintf("%02d", meta.TrackNumber)
}
if meta.TrackTotal > 0 {
tags["tracktotal"] = strconv.Itoa(meta.TrackTotal)
@@ -168,6 +201,150 @@ func toTags(meta Metadata) map[string]string {
return tags
}
func applyMultiValueFLACTags(path string, meta Metadata) error {
if strings.ToLower(strings.TrimPrefix(filepath.Ext(path), ".")) != "flac" || len(meta.Artists) == 0 {
return nil
}
st, err := os.Stat(path)
if err != nil {
return err
}
data, err := os.ReadFile(path)
if err != nil {
return err
}
updated, err := replaceFLACVorbisComments(data, "ARTISTS", meta.Artists)
if err != nil {
return err
}
if updated == nil {
return nil
}
return os.WriteFile(path, updated, st.Mode())
}
type flacMetadataBlock struct {
isLast bool
blockType byte
data []byte
}
func replaceFLACVorbisComments(data []byte, key string, values []string) ([]byte, error) {
if len(data) < 4 || string(data[:4]) != "fLaC" {
return nil, fmt.Errorf("not a FLAC file")
}
blocks := []flacMetadataBlock{}
pos := 4
vorbisIndex := -1
for {
if pos+4 > len(data) {
return nil, fmt.Errorf("truncated FLAC metadata header")
}
header := data[pos]
blockType := header & 0x7f
length := int(data[pos+1])<<16 | int(data[pos+2])<<8 | int(data[pos+3])
pos += 4
if pos+length > len(data) {
return nil, fmt.Errorf("truncated FLAC metadata block")
}
if blockType == 4 {
vorbisIndex = len(blocks)
}
blocks = append(blocks, flacMetadataBlock{isLast: header&0x80 != 0, blockType: blockType, data: data[pos : pos+length]})
pos += length
if header&0x80 != 0 {
break
}
}
if vorbisIndex < 0 {
return nil, nil
}
commentBlock, err := replaceVorbisCommentValues(blocks[vorbisIndex].data, key, values)
if err != nil {
return nil, err
}
blocks[vorbisIndex].data = commentBlock
out := make([]byte, 0, len(data)+len(commentBlock)-len(blocks[vorbisIndex].data))
out = append(out, data[:4]...)
for _, block := range blocks {
if len(block.data) > 0xffffff {
return nil, fmt.Errorf("FLAC metadata block too large")
}
header := block.blockType
if block.isLast {
header |= 0x80
}
out = append(out, header, byte(len(block.data)>>16), byte(len(block.data)>>8), byte(len(block.data)))
out = append(out, block.data...)
}
out = append(out, data[pos:]...)
return out, nil
}
func replaceVorbisCommentValues(data []byte, key string, values []string) ([]byte, error) {
if len(data) < 8 {
return nil, fmt.Errorf("truncated Vorbis comment block")
}
pos := 0
vendorLength := int(binary.LittleEndian.Uint32(data[pos:]))
pos += 4
if pos+vendorLength+4 > len(data) {
return nil, fmt.Errorf("truncated Vorbis vendor string")
}
vendor := data[pos : pos+vendorLength]
pos += vendorLength
commentCount := int(binary.LittleEndian.Uint32(data[pos:]))
pos += 4
comments := make([][]byte, 0, commentCount+len(values))
for i := 0; i < commentCount; i++ {
if pos+4 > len(data) {
return nil, fmt.Errorf("truncated Vorbis comment length")
}
commentLength := int(binary.LittleEndian.Uint32(data[pos:]))
pos += 4
if pos+commentLength > len(data) {
return nil, fmt.Errorf("truncated Vorbis comment")
}
comment := data[pos : pos+commentLength]
pos += commentLength
if !vorbisCommentKeyEqual(comment, key) {
comments = append(comments, comment)
}
}
for _, value := range values {
value = strings.TrimSpace(value)
if value != "" {
comments = append(comments, []byte(key+"="+value))
}
}
out := make([]byte, 0, len(data))
out = appendUint32LE(out, uint32(len(vendor)))
out = append(out, vendor...)
out = appendUint32LE(out, uint32(len(comments)))
for _, comment := range comments {
out = appendUint32LE(out, uint32(len(comment)))
out = append(out, comment...)
}
return out, nil
}
func vorbisCommentKeyEqual(comment []byte, key string) bool {
idx := strings.IndexByte(string(comment), '=')
if idx < 0 {
return false
}
return strings.EqualFold(string(comment[:idx]), key)
}
func appendUint32LE(out []byte, v uint32) []byte {
return append(out, byte(v), byte(v>>8), byte(v>>16), byte(v>>24))
}
func normalizeCopyright(in string) string {
out := strings.ReplaceAll(in, "(c)", "©")
out = strings.ReplaceAll(out, "(C)", "©")
+136 -3
View File
@@ -1,6 +1,7 @@
package tag
import (
"encoding/binary"
"os"
"path/filepath"
"testing"
@@ -30,6 +31,7 @@ func TestToTagsTotalsAndSourceFields(t *testing.T) {
DiscNumber: 1,
DiscTotal: 2,
ISRC: "USABC1234567",
InitialKey: "Em",
ReplaygainTrackGain: "-7.25 dB",
ReplaygainAlbumGain: "-8.1 dB",
ReplaygainTrackPeak: "0.989",
@@ -37,7 +39,7 @@ func TestToTagsTotalsAndSourceFields(t *testing.T) {
SourcePlatform: "qobuz",
SourceTrackID: "t1",
})
if tags["track"] != "03/12" {
if tags["track"] != "03" {
t.Fatalf("track tag = %q", tags["track"])
}
if tags["disc"] != "1/2" {
@@ -49,6 +51,9 @@ func TestToTagsTotalsAndSourceFields(t *testing.T) {
if tags["isrc"] != "USABC1234567" {
t.Fatalf("isrc missing: %+v", tags)
}
if tags["INITIALKEY"] != "Em" {
t.Fatalf("INITIALKEY missing: %+v", tags)
}
if tags["source_platform"] != "QOBUZ" || tags["source_track_id"] != "t1" {
t.Fatalf("source tags missing: %+v", tags)
}
@@ -60,13 +65,109 @@ func TestToTagsTotalsAndSourceFields(t *testing.T) {
}
}
func TestToTagsArtistList(t *testing.T) {
tags := toTags(Metadata{Artist: "A, B & C", Artists: []string{"A", "B", "C"}})
if tags["artist"] != "A, B & C" {
t.Fatalf("artist tags = %+v", tags)
}
if _, ok := tags["ARTISTS"]; ok {
t.Fatalf("ARTISTS should be written as repeated FLAC comments, got %+v", tags)
}
}
func TestReplaceVorbisCommentValuesWritesRepeatedTags(t *testing.T) {
block := vorbisCommentBlock("vendor", []string{
"ARTIST=A, B & C",
"ARTISTS=A;B;C",
"TITLE=Song",
})
updated, err := replaceVorbisCommentValues(block, "ARTISTS", []string{"A", "B", "C"})
if err != nil {
t.Fatalf("replaceVorbisCommentValues() error = %v", err)
}
comments := readVorbisComments(t, updated)
want := []string{"ARTIST=A, B & C", "TITLE=Song", "ARTISTS=A", "ARTISTS=B", "ARTISTS=C"}
if len(comments) != len(want) {
t.Fatalf("comments=%#v want %#v", comments, want)
}
for i := range want {
if comments[i] != want[i] {
t.Fatalf("comments=%#v want %#v", comments, want)
}
}
}
func TestReplaceFLACVorbisComments(t *testing.T) {
vorbis := vorbisCommentBlock("vendor", []string{"ARTISTS=A;B", "TITLE=Song"})
flac := append([]byte("fLaC"), flacBlockHeader(true, 4, len(vorbis))...)
flac = append(flac, vorbis...)
flac = append(flac, []byte("audio")...)
updated, err := replaceFLACVorbisComments(flac, "ARTISTS", []string{"A", "B"})
if err != nil {
t.Fatalf("replaceFLACVorbisComments() error = %v", err)
}
if string(updated[len(updated)-5:]) != "audio" {
t.Fatalf("audio payload not preserved")
}
length := int(updated[5])<<16 | int(updated[6])<<8 | int(updated[7])
comments := readVorbisComments(t, updated[8:8+length])
want := []string{"TITLE=Song", "ARTISTS=A", "ARTISTS=B"}
if len(comments) != len(want) {
t.Fatalf("comments=%#v want %#v", comments, want)
}
for i := range want {
if comments[i] != want[i] {
t.Fatalf("comments=%#v want %#v", comments, want)
}
}
}
func vorbisCommentBlock(vendor string, comments []string) []byte {
out := []byte{}
out = appendUint32LE(out, uint32(len(vendor)))
out = append(out, vendor...)
out = appendUint32LE(out, uint32(len(comments)))
for _, comment := range comments {
out = appendUint32LE(out, uint32(len(comment)))
out = append(out, comment...)
}
return out
}
func flacBlockHeader(last bool, blockType byte, length int) []byte {
header := blockType
if last {
header |= 0x80
}
return []byte{header, byte(length >> 16), byte(length >> 8), byte(length)}
}
func readVorbisComments(t *testing.T, block []byte) []string {
t.Helper()
pos := 0
vendorLength := int(binary.LittleEndian.Uint32(block[pos:]))
pos += 4 + vendorLength
count := int(binary.LittleEndian.Uint32(block[pos:]))
pos += 4
comments := make([]string, 0, count)
for i := 0; i < count; i++ {
length := int(binary.LittleEndian.Uint32(block[pos:]))
pos += 4
comments = append(comments, string(block[pos:pos+length]))
pos += length
}
return comments
}
func TestBuildFFmpegArgsWithCover(t *testing.T) {
tmp := t.TempDir()
cover := filepath.Join(tmp, "cover.jpg")
if err := os.WriteFile(cover, []byte("x"), 0o644); err != nil {
t.Fatalf("write cover: %v", err)
}
args := buildFFmpegArgs("in.flac", "out.flac", Metadata{Title: "x"}, cover, "flac")
args := buildFFmpegArgs("in.flac", "out.flac", Metadata{Title: "x"}, cover, "flac", false)
foundInput2 := false
foundAttach := false
for i := 0; i < len(args)-1; i++ {
@@ -88,7 +189,7 @@ func TestBuildFFmpegArgsSkipsCoverForUnsupportedContainer(t *testing.T) {
if err := os.WriteFile(cover, []byte("x"), 0o644); err != nil {
t.Fatalf("write cover: %v", err)
}
args := buildFFmpegArgs("in.opus", "out.opus", Metadata{Title: "x"}, cover, "opus")
args := buildFFmpegArgs("in.opus", "out.opus", Metadata{Title: "x"}, cover, "opus", false)
for i := 0; i < len(args)-1; i++ {
if args[i] == "-i" && args[i+1] == cover {
t.Fatalf("unexpected cover input for opus: %v", args)
@@ -96,6 +197,38 @@ func TestBuildFFmpegArgsSkipsCoverForUnsupportedContainer(t *testing.T) {
}
}
func TestBuildFFmpegArgsClearsDiscTagsWhenRequested(t *testing.T) {
args := buildFFmpegArgs("in.flac", "out.flac", Metadata{Title: "x", OmitDiscTags: true}, "", "flac", false)
want := map[string]bool{
"disc=": false,
"disk=": false,
"disctotal=": false,
"totaldiscs=": false,
}
for i := 0; i < len(args)-1; i++ {
if args[i] == "-metadata" {
if _, ok := want[args[i+1]]; ok {
want[args[i+1]] = true
}
}
}
for tag, found := range want {
if !found {
t.Fatalf("missing clear tag %q in args: %v", tag, args)
}
}
}
func TestBuildFFmpegArgsForcesMP4Muxer(t *testing.T) {
args := buildFFmpegArgs("in.m4a", "out.m4a", Metadata{Title: "x"}, "", "m4a", true)
for i := 0; i < len(args)-1; i++ {
if args[i] == "-f" && args[i+1] == "mp4" {
return
}
}
t.Fatalf("missing forced mp4 muxer args: %v", args)
}
func TestTaggedTempPathPreservesExtension(t *testing.T) {
if got := taggedTempPath("/tmp/song.flac"); got != "/tmp/song.tmp.flac" {
t.Fatalf("taggedTempPath(flac)=%q", got)
+294 -47
View File
@@ -5,6 +5,9 @@ import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/pelletier/go-toml/v2"
)
@@ -13,8 +16,17 @@ const CurrentConfigVersion = "2.2.0"
var ErrOutdatedConfig = errors.New("config version mismatch")
// Config holds the on-disk config (File) and the runtime view with CLI
// overrides applied (Session).
//
// mu guards File, Session and the write to Path. Provider clients refresh OAuth
// tokens from inside concurrent download workers, so both the struct fields and
// the config-file write have to be serialized; without this, two workers
// refreshing at once raced each other and issued two concurrent os.WriteFile
// calls against the same path.
type Config struct {
Path string
mu sync.Mutex
File ConfigData
Session ConfigData
}
@@ -24,8 +36,9 @@ type ConfigData struct {
Qobuz QobuzConfig `toml:"qobuz"`
Tidal TidalConfig `toml:"tidal"`
Deezer DeezerConfig `toml:"deezer"`
Yandex YandexConfig `toml:"yandex"`
Beatport BeatportConfig `toml:"beatport"`
Soundcloud SoundcloudConfig `toml:"soundcloud"`
Youtube YoutubeConfig `toml:"youtube"`
Database DatabaseConfig `toml:"database"`
Conversion ConversionConfig `toml:"conversion"`
QobuzFilters QobuzDiscographyFilterConfig `toml:"qobuz_filters"`
@@ -48,43 +61,75 @@ type DownloadsConfig struct {
}
type QobuzConfig struct {
Quality int `toml:"quality"`
DownloadBooklets bool `toml:"download_booklets"`
UseAuthToken bool `toml:"use_auth_token"`
EmailOrUserID string `toml:"email_or_userid"`
PasswordOrToken string `toml:"password_or_token"`
AppID string `toml:"app_id"`
Secrets []string `toml:"secrets"`
Quality int `toml:"quality"`
UseAuthToken bool `toml:"use_auth_token"`
EmailOrUserID string `toml:"email_or_userid"`
PasswordOrToken string `toml:"password_or_token"`
AppID string `toml:"app_id"`
Secrets []string `toml:"secrets"`
// Vendor app credentials for the mobile API fallback. Defaulted in code;
// override here when Qobuz rotates them, instead of waiting for a release.
MobileAppID string `toml:"mobile_app_id"`
MobileAppSecret string `toml:"mobile_app_secret"`
}
type TidalConfig struct {
Quality int `toml:"quality"`
DownloadVideos bool `toml:"download_videos"`
PreferAtmos bool `toml:"prefer_atmos"`
UserID string `toml:"user_id"`
CountryCode string `toml:"country_code"`
AccessToken string `toml:"access_token"`
RefreshToken string `toml:"refresh_token"`
TokenExpiry int64 `toml:"token_expiry"`
// Vendor OAuth app credentials. Defaulted in code; override on rotation.
AppClientID string `toml:"app_client_id"`
AppClientSecret string `toml:"app_client_secret"`
}
type DeezerConfig struct {
Quality int `toml:"quality"`
LowerQualityIfNotAvailable bool `toml:"lower_quality_if_not_available"`
ARL string `toml:"arl"`
UseDeezloader bool `toml:"use_deezloader"`
DeezloaderWarnings bool `toml:"deezloader_warnings"`
Email string `toml:"email"`
Password string `toml:"password"`
RefreshToken string `toml:"refresh_token"`
// Vendor protocol keys. Defaulted in code; override on rotation.
APIKey string `toml:"api_key"`
GatewayKey string `toml:"gateway_key"`
BlowfishSecret string `toml:"blowfish_secret"`
}
type YandexConfig struct {
Quality int `toml:"quality"`
AccessToken string `toml:"access_token"`
UserID string `toml:"user_id"`
// Vendor request-signing keys. Defaulted in code; override on rotation.
SignKey string `toml:"sign_key"`
LegacyMP3SignSalt string `toml:"legacy_mp3_sign_salt"`
}
type BeatportConfig struct {
Quality int `toml:"quality"`
VariousArtistsThreshold int `toml:"various_artists_threshold"`
Username string `toml:"username"`
Password string `toml:"password"`
AccessToken string `toml:"access_token"`
RefreshToken string `toml:"refresh_token"`
TokenExpiry int64 `toml:"token_expiry"`
// Vendor OAuth app credential. Defaulted in code; override on rotation.
AppClientID string `toml:"app_client_id"`
}
// SoundcloudConfig has no client_id/app_version: soundcloud is handled via
// yt-dlp, which supplies its own, so those keys were read by no code.
type SoundcloudConfig struct {
Quality int `toml:"quality"`
ClientID string `toml:"client_id"`
AppVersion string `toml:"app_version"`
}
type YoutubeConfig struct {
Quality int `toml:"quality"`
DownloadVideos bool `toml:"download_videos"`
VideoDownloadsFolder string `toml:"video_downloads_folder"`
Quality int `toml:"quality"`
}
type DatabaseConfig struct {
@@ -122,6 +167,7 @@ type ArtworkConfig struct {
type MetadataConfig struct {
SetPlaylistToAlbum bool `toml:"set_playlist_to_album"`
RenumberPlaylistTracks bool `toml:"renumber_playlist_tracks"`
ArtistSeparator string `toml:"artist_separator"`
Exclude []string `toml:"exclude"`
}
@@ -145,8 +191,7 @@ type CLIConfig struct {
}
type MiscConfig struct {
Version string `toml:"version"`
CheckForUpdates bool `toml:"check_for_updates"`
Version string `toml:"version"`
}
func Load(path string) (*Config, error) {
@@ -181,34 +226,219 @@ func Load(path string) (*Config, error) {
return nil, fmt.Errorf("%w: need to update from %q to %q", ErrOutdatedConfig, data.Misc.Version, CurrentConfigVersion)
}
if err = Validate(&data); err != nil {
return nil, err
}
return &Config{Path: resolvedPath, File: data, Session: cloneConfigData(data)}, nil
}
func UpgradeOutdated(path string) (string, error) {
// Vendor app credentials and protocol keys.
//
// These are the provider apps' own identifiers and signing keys, not user
// secrets, and they are baked in so the tool works out of the box. They are
// mirrored into the config so that when a provider rotates one, a user can fix
// their install with a config edit instead of waiting for a new release.
const (
DefaultQobuzMobileAppID = "312369995"
DefaultQobuzMobileAppSecret = "e79f8b9be485692b0e5f9dd895826368"
DefaultTidalAppClientID = "fX2JxdmntZWK0ixT"
DefaultTidalAppClientSecret = "1Nm5AfDAjxrgJFJbKNWLeAyKGVGmINuXPPLHVXAvxAg="
DefaultDeezerAPIKey = "4VCYIJUCDLOUELGD1V8WBVYBNVDYOXEWSLLZDONGBBDFVXTZJRXPR29JRLQFO6ZE"
DefaultDeezerGatewayKey = "VBK1FSUEXHTSDBJJ"
DefaultDeezerBlowfishSecret = "g4el58wc0zvf9na1"
DefaultYandexSignKey = "kzqU4XhfCaY6B6JTHODeq5"
DefaultYandexLegacyMP3Salt = "XGRlBW9FXlekgbPrRHuSiA"
DefaultBeatportAppClientID = "ryZ8LuyQVPqbK2mBX2Hwt4qSMtnWuTYSqBPO92yQ"
)
// applyVendorDefaults fills in any vendor credential the user has not set, so
// the provider clients never have to nil-check them.
func applyVendorDefaults(data *ConfigData) {
if data.Qobuz.MobileAppID == "" {
data.Qobuz.MobileAppID = DefaultQobuzMobileAppID
}
if data.Qobuz.MobileAppSecret == "" {
data.Qobuz.MobileAppSecret = DefaultQobuzMobileAppSecret
}
if data.Tidal.AppClientID == "" {
data.Tidal.AppClientID = DefaultTidalAppClientID
}
if data.Tidal.AppClientSecret == "" {
data.Tidal.AppClientSecret = DefaultTidalAppClientSecret
}
if data.Deezer.APIKey == "" {
data.Deezer.APIKey = DefaultDeezerAPIKey
}
if data.Deezer.GatewayKey == "" {
data.Deezer.GatewayKey = DefaultDeezerGatewayKey
}
if data.Deezer.BlowfishSecret == "" {
data.Deezer.BlowfishSecret = DefaultDeezerBlowfishSecret
}
if data.Yandex.SignKey == "" {
data.Yandex.SignKey = DefaultYandexSignKey
}
if data.Yandex.LegacyMP3SignSalt == "" {
data.Yandex.LegacyMP3SignSalt = DefaultYandexLegacyMP3Salt
}
if data.Beatport.AppClientID == "" {
data.Beatport.AppClientID = DefaultBeatportAppClientID
}
}
// qualityRange is the accepted quality id range per source on the global 0-4
// scale.
//
// The ceiling is 4 everywhere because each provider maps the scale onto its own
// ladder and saturates at the top: deezer's buildFormatPriority treats anything
// >= 2 as FLAC, and soundcloud ignores the value entirely. Only the floor
// differs: Qobuz and Beatport have no equivalent of tier 0.
var qualityRange = map[string][2]int{
"qobuz": {1, 4},
"tidal": {0, 4},
"deezer": {0, 4},
"soundcloud": {0, 4},
"yandex": {0, 4},
"beatport": {1, 4},
}
// Validate rejects config values that would otherwise fail much later, deep in
// a rip, or be silently patched over at individual call sites. Load previously
// checked nothing but misc.version, so a negative max_connections or a bogus
// conversion codec was accepted and only noticed mid-download.
//
// It normalizes what can be normalized (the conversion codec, which until now
// was only canonicalized when it arrived via --codec) and errors on the rest.
func Validate(data *ConfigData) error {
for source, bounds := range qualityRange {
var q int
switch source {
case "qobuz":
q = data.Qobuz.Quality
case "tidal":
q = data.Tidal.Quality
case "deezer":
q = data.Deezer.Quality
case "soundcloud":
q = data.Soundcloud.Quality
case "yandex":
q = data.Yandex.Quality
case "beatport":
q = data.Beatport.Quality
}
if q < bounds[0] || q > bounds[1] {
return fmt.Errorf("config: %s.quality is %d, must be %d-%d", source, q, bounds[0], bounds[1])
}
}
if data.Downloads.MaxConnections < 1 {
return fmt.Errorf("config: downloads.max_connections is %d, must be >= 1", data.Downloads.MaxConnections)
}
if data.Downloads.RequestsPerMinute == 0 {
return errors.New("config: downloads.requests_per_minute is 0; use a positive value, or a negative value to disable throttling")
}
if data.Filepaths.TruncateTo < 1 {
return fmt.Errorf("config: filepaths.truncate_to is %d, must be >= 1", data.Filepaths.TruncateTo)
}
if data.Conversion.Enabled {
codec, err := NormalizeCodec(data.Conversion.Codec)
if err != nil {
return fmt.Errorf("config: conversion.codec: %w", err)
}
data.Conversion.Codec = codec
}
// artwork.*_max_width uses -1 (and any non-positive value) to mean "do not
// downscale", so there is no lower bound to enforce here.
return nil
}
// NormalizeCodec canonicalizes a conversion codec name. It lives here, rather
// than only in the CLI flag parser, so a codec set in the config file goes
// through the same check.
func NormalizeCodec(raw string) (string, error) {
codec := strings.ToUpper(strings.TrimSpace(raw))
switch codec {
case "ALAC", "FLAC", "MP3", "AAC", "VORBIS":
return codec, nil
case "OGG":
return "VORBIS", nil
default:
return "", fmt.Errorf("unsupported codec %q (expected ALAC, FLAC, OGG, MP3, AAC)", raw)
}
}
// UpgradeOutdated rewrites an outdated config at the current version. It runs
// automatically, without prompting, so it copies the original aside first: the
// rewrite re-serializes the whole file from the parsed struct, which drops any
// key this version does not know about. It returns the config path and the
// backup path.
//
// This is a version stamp, not a migration framework. There are no per-version
// transforms, so a future key rename still needs explicit handling here.
func UpgradeOutdated(path string) (resolved string, backup string, err error) {
resolvedPath, err := resolvePath(path)
if err != nil {
return "", err
return "", "", err
}
raw, err := os.ReadFile(resolvedPath)
if err != nil {
return "", err
return "", "", err
}
backupPath := fmt.Sprintf("%s.bak-%d", resolvedPath, time.Now().Unix())
if err = os.WriteFile(backupPath, raw, 0o600); err != nil {
return "", "", fmt.Errorf("could not back up %s before upgrading: %w", resolvedPath, err)
}
data := DefaultConfigData()
if err = toml.Unmarshal(raw, &data); err != nil {
return "", err
return "", "", err
}
applyRuntimeDefaults(&data)
data.Misc.Version = CurrentConfigVersion
if err = saveConfigData(resolvedPath, data); err != nil {
return "", err
if err = Validate(&data); err != nil {
return "", "", err
}
return resolvedPath, nil
if err = saveConfigData(resolvedPath, data); err != nil {
return "", "", err
}
return resolvedPath, backupPath, nil
}
func (c *Config) SaveFile() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.saveFileLocked()
}
func (c *Config) saveFileLocked() error {
return saveConfigData(c.Path, c.File)
}
// UpdateTokens applies fn to both the on-disk and session copies of the config
// under the config lock, then persists the result. Provider token refresh must
// go through this rather than mutating c.File/c.Session directly, because it
// runs concurrently from download workers.
func (c *Config) UpdateTokens(fn func(file, session *ConfigData)) error {
if fn == nil {
return nil
}
c.mu.Lock()
defer c.mu.Unlock()
fn(&c.File, &c.Session)
// An empty Path means there is no config file backing this Config (tests,
// and callers that loaded defaults only); the in-memory update still has to
// happen, but there is nothing to write.
if strings.TrimSpace(c.Path) == "" {
return nil
}
return c.saveFileLocked()
}
func DefaultConfigData() ConfigData {
home, _ := os.UserHomeDir()
appDir := defaultAppDir()
@@ -225,28 +455,28 @@ func DefaultConfigData() ConfigData {
VerifySSL: true,
},
Qobuz: QobuzConfig{
Quality: 3,
DownloadBooklets: true,
UseAuthToken: false,
Quality: 3,
UseAuthToken: false,
},
Tidal: TidalConfig{
Quality: 3,
DownloadVideos: true,
PreferAtmos: false,
},
Deezer: DeezerConfig{
Quality: 2,
LowerQualityIfNotAvailable: true,
UseDeezloader: true,
DeezloaderWarnings: true,
},
Yandex: YandexConfig{
Quality: 2,
},
Beatport: BeatportConfig{
Quality: 3,
VariousArtistsThreshold: 3,
},
Soundcloud: SoundcloudConfig{
Quality: 0,
},
Youtube: YoutubeConfig{
Quality: 0,
DownloadVideos: false,
VideoDownloadsFolder: filepath.Join(downloadsFolder, "YouTubeVideos"),
},
Database: DatabaseConfig{
DownloadsEnabled: true,
DownloadsPath: filepath.Join(appDir, "downloads.db"),
@@ -271,6 +501,7 @@ func DefaultConfigData() ConfigData {
Metadata: MetadataConfig{
SetPlaylistToAlbum: true,
RenumberPlaylistTracks: true,
ArtistSeparator: "; ",
Exclude: []string{},
},
Filepaths: FilepathsConfig{
@@ -289,20 +520,25 @@ func DefaultConfigData() ConfigData {
MaxSearchResults: 100,
},
Misc: MiscConfig{
Version: CurrentConfigVersion,
CheckForUpdates: true,
Version: CurrentConfigVersion,
},
}
applyVendorDefaults(&data)
return data
}
func resolvePath(path string) (string, error) {
// 0o700: this directory holds the credential-bearing config and the
// download databases.
if path != "" {
return path, os.MkdirAll(filepath.Dir(path), 0o755)
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return "", err
}
return path, nil
}
appDir := defaultAppDir()
if err := os.MkdirAll(appDir, 0o755); err != nil {
if err := os.MkdirAll(appDir, 0o700); err != nil {
return "", err
}
return filepath.Join(appDir, "config.toml"), nil
@@ -317,17 +553,24 @@ func defaultAppDir() string {
}
func saveConfigData(path string, data ConfigData) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
// 0o700/0o600: this file holds plaintext provider passwords, Deezer ARL
// cookies and OAuth access/refresh tokens, so it must not be world-readable.
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err
}
b, err := toml.Marshal(data)
if err != nil {
return err
}
return os.WriteFile(path, b, 0o644)
if err := os.WriteFile(path, b, 0o600); err != nil {
return err
}
// Tighten an existing file that a previous version created as 0o644.
return os.Chmod(path, 0o600)
}
func applyRuntimeDefaults(data *ConfigData) {
applyVendorDefaults(data)
home, _ := os.UserHomeDir()
appDir := defaultAppDir()
if data.Downloads.Folder == "" {
@@ -339,9 +582,13 @@ func applyRuntimeDefaults(data *ConfigData) {
if data.Database.FailedDownloadsPath == "" {
data.Database.FailedDownloadsPath = filepath.Join(appDir, "failed_downloads.db")
}
if data.Youtube.VideoDownloadsFolder == "" {
data.Youtube.VideoDownloadsFolder = filepath.Join(data.Downloads.Folder, "YouTubeVideos")
}
}
// CloneConfigData returns a deep copy of in. The slice fields must not be
// shared between Config.File and Config.Session, or writing one silently
// mutates the other.
func CloneConfigData(in ConfigData) ConfigData {
return cloneConfigData(in)
}
func cloneConfigData(in ConfigData) ConfigData {
+330 -1
View File
@@ -1,10 +1,16 @@
package config
import (
"encoding/hex"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"github.com/pelletier/go-toml/v2"
)
func TestDefaultConfigData(t *testing.T) {
@@ -64,7 +70,7 @@ func TestUpgradeOutdatedConfig(t *testing.T) {
t.Fatalf("saveConfigData() error = %v", err)
}
resolved, err := UpgradeOutdated(path)
resolved, backup, err := UpgradeOutdated(path)
if err != nil {
t.Fatalf("UpgradeOutdated() error = %v", err)
}
@@ -72,6 +78,28 @@ func TestUpgradeOutdatedConfig(t *testing.T) {
t.Fatalf("resolved path = %q, want %q", resolved, path)
}
// The upgrade re-serializes the whole file and runs without prompting, so
// it must leave the original behind.
if backup == "" {
t.Fatal("UpgradeOutdated() returned no backup path")
}
backupRaw, err := os.ReadFile(backup)
if err != nil {
t.Fatalf("backup not written: %v", err)
}
var backedUp ConfigData
if err := toml.Unmarshal(backupRaw, &backedUp); err != nil {
t.Fatalf("backup is not valid toml: %v", err)
}
if backedUp.Misc.Version != "1.0.0" {
t.Errorf("backup version = %q, want the pre-upgrade 1.0.0", backedUp.Misc.Version)
}
if info, err := os.Stat(backup); err != nil {
t.Fatal(err)
} else if perm := info.Mode().Perm(); perm != 0o600 {
t.Errorf("backup permissions = %04o, want 0600 (it holds the same credentials)", perm)
}
cfg, err := Load(path)
if err != nil {
t.Fatalf("Load() after upgrade error = %v", err)
@@ -110,3 +138,304 @@ func TestSessionCloneDoesNotAliasSlices(t *testing.T) {
t.Fatalf("file qobuz secrets unexpectedly mutated")
}
}
func TestDefaultConfigPassesValidation(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
if err := Validate(&d); err != nil {
t.Fatalf("shipped defaults fail validation: %v", err)
}
}
// TestExampleConfigIsValid guards the file users are told to copy.
func TestExampleConfigIsValid(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "..", "config.toml.example"))
if err != nil {
t.Skipf("example config not found: %v", err)
}
data := DefaultConfigData()
if err := toml.Unmarshal(raw, &data); err != nil {
t.Fatalf("config.toml.example does not parse: %v", err)
}
applyRuntimeDefaults(&data)
if err := Validate(&data); err != nil {
t.Fatalf("config.toml.example fails validation: %v", err)
}
if data.Misc.Version != CurrentConfigVersion {
t.Errorf("config.toml.example version = %q, want %q", data.Misc.Version, CurrentConfigVersion)
}
}
func TestValidateRejectsBadValues(t *testing.T) {
tests := []struct {
name string
mutate func(*ConfigData)
want string
}{
{"qobuz quality 0", func(d *ConfigData) { d.Qobuz.Quality = 0 }, "qobuz.quality"},
{"qobuz quality 5", func(d *ConfigData) { d.Qobuz.Quality = 5 }, "qobuz.quality"},
{"beatport quality 0", func(d *ConfigData) { d.Beatport.Quality = 0 }, "beatport.quality"},
{"tidal quality 5", func(d *ConfigData) { d.Tidal.Quality = 5 }, "tidal.quality"},
{"negative quality", func(d *ConfigData) { d.Tidal.Quality = -1 }, "tidal.quality"},
{"max_connections 0", func(d *ConfigData) { d.Downloads.MaxConnections = 0 }, "max_connections"},
{"max_connections negative", func(d *ConfigData) { d.Downloads.MaxConnections = -5 }, "max_connections"},
{"requests_per_minute 0", func(d *ConfigData) { d.Downloads.RequestsPerMinute = 0 }, "requests_per_minute"},
{"truncate_to 0", func(d *ConfigData) { d.Filepaths.TruncateTo = 0 }, "truncate_to"},
{"bogus codec", func(d *ConfigData) {
d.Conversion.Enabled = true
d.Conversion.Codec = "bogus"
}, "conversion.codec"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
tc.mutate(&d)
err := Validate(&d)
if err == nil {
t.Fatalf("Validate() = nil, want an error mentioning %q", tc.want)
}
if !strings.Contains(err.Error(), tc.want) {
t.Fatalf("Validate() = %q, want it to mention %q", err, tc.want)
}
})
}
}
// TestValidateAcceptsDocumentedQualityValues pins the ranges documented in
// config.toml.example. An earlier draft capped deezer at 2 and soundcloud at 1,
// which would have made a config the example itself documents fail to load.
func TestValidateAcceptsDocumentedQualityValues(t *testing.T) {
for q := 0; q <= 4; q++ {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
d.Deezer.Quality = q
d.Soundcloud.Quality = q
d.Tidal.Quality = q
d.Yandex.Quality = q
if err := Validate(&d); err != nil {
t.Errorf("quality %d rejected: %v", q, err)
}
}
// Qobuz and Beatport have no tier 0.
for q := 1; q <= 4; q++ {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
d.Qobuz.Quality = q
d.Beatport.Quality = q
if err := Validate(&d); err != nil {
t.Errorf("qobuz/beatport quality %d rejected: %v", q, err)
}
}
}
func TestValidateAcceptsValidValues(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
// Negative requests_per_minute disables throttling, which is legitimate.
d.Downloads.RequestsPerMinute = -1
d.Soundcloud.Quality = 1
d.Deezer.Quality = 4
d.Qobuz.Quality = 4
if err := Validate(&d); err != nil {
t.Fatalf("Validate() = %v, want nil", err)
}
}
func TestValidateNormalizesCodec(t *testing.T) {
d := DefaultConfigData()
applyRuntimeDefaults(&d)
d.Conversion.Enabled = true
d.Conversion.Codec = " ogg "
if err := Validate(&d); err != nil {
t.Fatalf("Validate() = %v", err)
}
if d.Conversion.Codec != "VORBIS" {
t.Fatalf("codec = %q, want VORBIS", d.Conversion.Codec)
}
}
func TestNormalizeCodec(t *testing.T) {
for in, want := range map[string]string{
"flac": "FLAC", "FLAC": "FLAC", " mp3 ": "MP3",
"ogg": "VORBIS", "vorbis": "VORBIS", "alac": "ALAC", "aac": "AAC",
} {
got, err := NormalizeCodec(in)
if err != nil {
t.Errorf("NormalizeCodec(%q) err = %v", in, err)
continue
}
if got != want {
t.Errorf("NormalizeCodec(%q) = %q, want %q", in, got, want)
}
}
if _, err := NormalizeCodec("wav"); err == nil {
t.Error("NormalizeCodec(wav) should fail")
}
}
func TestCloneConfigDataDoesNotShareSlices(t *testing.T) {
a := DefaultConfigData()
a.Qobuz.Secrets = []string{"s1", "s2"}
a.Metadata.Exclude = []string{"e1"}
b := CloneConfigData(a)
b.Qobuz.Secrets[0] = "mutated"
b.Metadata.Exclude[0] = "mutated"
if a.Qobuz.Secrets[0] != "s1" {
t.Error("Qobuz.Secrets is shared between copies")
}
if a.Metadata.Exclude[0] != "e1" {
t.Error("Metadata.Exclude is shared between copies")
}
}
func TestSaveFileUsesOwnerOnlyPermissions(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.toml")
cfg := &Config{Path: path, File: DefaultConfigData()}
if err := cfg.SaveFile(); err != nil {
t.Fatalf("SaveFile() = %v", err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
// The file holds plaintext passwords, ARL cookies and OAuth tokens.
if perm := info.Mode().Perm(); perm != 0o600 {
t.Fatalf("config permissions = %04o, want 0600", perm)
}
}
func TestSaveFileTightensExistingWorldReadableFile(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.toml")
if err := os.WriteFile(path, []byte("version = \"x\"\n"), 0o644); err != nil {
t.Fatal(err)
}
cfg := &Config{Path: path, File: DefaultConfigData()}
if err := cfg.SaveFile(); err != nil {
t.Fatal(err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
if perm := info.Mode().Perm(); perm != 0o600 {
t.Fatalf("permissions = %04o, want 0600 after rewrite", perm)
}
}
func TestUpdateTokensIsSafeWithoutAPath(t *testing.T) {
cfg := &Config{File: DefaultConfigData(), Session: DefaultConfigData()}
err := cfg.UpdateTokens(func(file, session *ConfigData) {
file.Deezer.RefreshToken = "rt"
session.Deezer.RefreshToken = "rt"
})
if err != nil {
t.Fatalf("UpdateTokens() = %v", err)
}
// The in-memory update must still happen even with nothing to write to.
if cfg.File.Deezer.RefreshToken != "rt" || cfg.Session.Deezer.RefreshToken != "rt" {
t.Fatal("UpdateTokens did not apply the mutation")
}
}
func TestUpdateTokensPersistsAndIsConcurrencySafe(t *testing.T) {
path := filepath.Join(t.TempDir(), "config.toml")
cfg := &Config{Path: path, File: DefaultConfigData(), Session: DefaultConfigData()}
var wg sync.WaitGroup
for i := 0; i < 16; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
token := fmt.Sprintf("token-%d", n)
if err := cfg.UpdateTokens(func(file, session *ConfigData) {
file.Tidal.AccessToken = token
session.Tidal.AccessToken = token
}); err != nil {
t.Errorf("UpdateTokens: %v", err)
}
}(i)
}
wg.Wait()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var round ConfigData
if err := toml.Unmarshal(raw, &round); err != nil {
t.Fatalf("concurrent writes produced an unparseable config: %v", err)
}
if round.Tidal.AccessToken == "" {
t.Error("no token persisted")
}
}
// TestVendorDefaultsAreApplied guards the Phase-3 move of vendor credentials
// into config: an empty value must resolve to the built-in constant, or the
// providers silently authenticate and sign with empty keys.
func TestVendorDefaultsAreApplied(t *testing.T) {
d := DefaultConfigData()
checks := map[string]string{
"qobuz.mobile_app_id": d.Qobuz.MobileAppID,
"qobuz.mobile_app_secret": d.Qobuz.MobileAppSecret,
"tidal.app_client_id": d.Tidal.AppClientID,
"tidal.app_client_secret": d.Tidal.AppClientSecret,
"deezer.api_key": d.Deezer.APIKey,
"deezer.gateway_key": d.Deezer.GatewayKey,
"deezer.blowfish_secret": d.Deezer.BlowfishSecret,
"yandex.sign_key": d.Yandex.SignKey,
"yandex.legacy_sign_salt": d.Yandex.LegacyMP3SignSalt,
"beatport.app_client_id": d.Beatport.AppClientID,
}
for name, got := range checks {
if got == "" {
t.Errorf("%s is empty in DefaultConfigData()", name)
}
}
// An explicitly empty config must be backfilled too, since that is what a
// user config written before these keys existed looks like.
var blank ConfigData
applyVendorDefaults(&blank)
if blank.Qobuz.MobileAppSecret != DefaultQobuzMobileAppSecret {
t.Error("applyVendorDefaults did not backfill qobuz.mobile_app_secret")
}
if blank.Deezer.BlowfishSecret != DefaultDeezerBlowfishSecret {
t.Error("applyVendorDefaults did not backfill deezer.blowfish_secret")
}
}
func TestVendorDefaultsAreNotOverriddenWhenSet(t *testing.T) {
d := ConfigData{}
d.Tidal.AppClientID = "mine"
d.Deezer.BlowfishSecret = "0123456789abcdef"
applyVendorDefaults(&d)
if d.Tidal.AppClientID != "mine" {
t.Errorf("user value overwritten: %q", d.Tidal.AppClientID)
}
if d.Deezer.BlowfishSecret != "0123456789abcdef" {
t.Errorf("user value overwritten: %q", d.Deezer.BlowfishSecret)
}
}
// TestDeezerBlowfishSecretLength pins the length the key derivation depends
// on: it XORs 16 bytes, so a different length would panic or silently derive a
// wrong key.
func TestDeezerBlowfishSecretLength(t *testing.T) {
if got := len(DefaultDeezerBlowfishSecret); got != 16 {
t.Fatalf("DefaultDeezerBlowfishSecret is %d bytes, want 16", got)
}
}
// TestQobuzMobileAppSecretIsHex pins the format the HKDF step requires.
func TestQobuzMobileAppSecretIsHex(t *testing.T) {
if _, err := hex.DecodeString(DefaultQobuzMobileAppSecret); err != nil {
t.Fatalf("DefaultQobuzMobileAppSecret is not hex: %v", err)
}
}
+596 -99
View File
@@ -3,8 +3,11 @@ package download
import (
"bufio"
"context"
"crypto/aes"
"crypto/cipher"
"crypto/md5"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
@@ -19,7 +22,9 @@ import (
"github.com/vbauerster/mpb/v8/decor"
"golang.org/x/term"
"streamrip-go/internal/config"
"streamrip-go/internal/netutil"
"streamrip-go/internal/verbose"
"golang.org/x/crypto/blowfish"
)
@@ -29,20 +34,25 @@ type Downloader struct {
showProgress bool
progress *mpb.Progress
barStarted atomic.Int32
// DeezerBlowfishSecret overrides the built-in Deezer protocol secret.
// Set from config by app.New; empty means use the default.
DeezerBlowfishSecret string
}
// downloadBufferSize sizes HTTP-to-disk copies. 1 MiB cuts read/write syscalls
// ~32x vs Go's default 32 KiB io.Copy buffer, which matters for multi-MB FLAC
// streams off CDNs that can sustain high per-connection throughput.
const downloadBufferSize = 1 << 20
func New() *Downloader {
return NewWithOptions(true, true)
return NewWithOptions(true, true, 0)
}
func NewWithVerifySSL(verifySSL bool) *Downloader {
return NewWithOptions(verifySSL, true)
}
func NewWithOptions(verifySSL bool, showProgress bool) *Downloader {
func NewWithOptions(verifySSL bool, showProgress bool, maxConnsPerHost int) *Downloader {
forceProgress := strings.EqualFold(os.Getenv("STREAMRIP_GO_FORCE_PROGRESS"), "1") || strings.EqualFold(os.Getenv("STREAMRIP_GO_FORCE_PROGRESS"), "true")
interactive := showProgress && (forceProgress || (term.IsTerminal(int(os.Stderr.Fd())) && strings.ToLower(os.Getenv("TERM")) != "dumb"))
d := &Downloader{http: netutil.NewHTTPClient(0, verifySSL), showProgress: interactive}
d := &Downloader{http: netutil.NewHTTPClient(0, verifySSL, maxConnsPerHost), showProgress: interactive}
if interactive {
d.progress = mpb.New(mpb.WithWidth(40), mpb.WithOutput(os.Stderr))
}
@@ -61,10 +71,24 @@ func (d *Downloader) FileVideo(ctx context.Context, sourceURL, outputPath string
return d.file(ctx, sourceURL, outputPath, true, true)
}
func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputPath, trackID string) error {
// streamEncrypted performs the shared scaffolding for an encrypted download:
// create the output directory, GET the URL, check the status, create the output
// file, attach a progress bar, clean up a partial file on failure, verify the
// byte count against Content-Length and fsync.
//
// decode does the format-specific work of turning the response body into plain
// audio on dst and returns the number of bytes written. FileDeezerEncrypted and
// FileYandexEncrypted were ~90% identical before this.
func (d *Downloader) streamEncrypted(
ctx context.Context,
sourceURL, outputPath string,
decode func(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error),
) error {
logDownloadStart(sourceURL, outputPath)
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL, nil)
if err != nil {
return err
@@ -77,18 +101,105 @@ func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputP
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("download failed: status=%d", resp.StatusCode)
}
encrypted, err := io.ReadAll(resp.Body)
out, err := os.Create(outputPath)
if err != nil {
return err
}
plain, err := decryptDeezerBFCBCStripe(encrypted, trackID)
success := false
defer func() {
_ = out.Close()
if !success {
_ = os.Remove(outputPath)
}
}()
bar, finishBar := d.newDownloadBar(outputPath, resp.ContentLength)
defer func() { finishBar(success) }()
written, err := decode(out, resp.Body, bar)
if err != nil {
return err
}
return os.WriteFile(outputPath, plain, 0o644)
if resp.ContentLength > 0 && written != resp.ContentLength {
return io.ErrUnexpectedEOF
}
if err = out.Sync(); err != nil {
return err
}
success = true
return nil
}
// FileDeezerEncrypted downloads a Deezer stream and decrypts it. Deezer
// encrypts every third 2048-byte chunk with Blowfish-CBC; a short final chunk
// is left as-is.
func (d *Downloader) FileDeezerEncrypted(ctx context.Context, sourceURL, outputPath, trackID string) error {
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(d.blowfishSecret(), trackID))
if err != nil {
return err
}
return d.streamEncrypted(ctx, sourceURL, outputPath, func(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error) {
buf := make([]byte, deezerBFChunkSize)
dec := make([]byte, deezerBFChunkSize)
chunkIndex := 0
total := int64(0)
for {
n, readErr := io.ReadFull(src, buf)
if errors.Is(readErr, io.EOF) {
return total, nil
}
if readErr != nil && !errors.Is(readErr, io.ErrUnexpectedEOF) {
return total, readErr
}
chunk := buf[:n]
if chunkIndex%3 == 0 && n == deezerBFChunkSize {
mode := cipher.NewCBCDecrypter(block, deezerBFIV)
mode.CryptBlocks(dec[:n], chunk)
chunk = dec[:n]
}
if _, writeErr := dst.Write(chunk); writeErr != nil {
return total, writeErr
}
total += int64(n)
if bar != nil {
bar.IncrBy(n)
}
chunkIndex++
if errors.Is(readErr, io.ErrUnexpectedEOF) {
return total, nil
}
}
})
}
// FileYandexEncrypted downloads a Yandex stream and decrypts it. Yandex uses
// AES-CTR over the whole body with a zero IV.
func (d *Downloader) FileYandexEncrypted(ctx context.Context, sourceURL, outputPath, key string) error {
keyBytes, err := hex.DecodeString(strings.TrimSpace(key))
if err != nil {
return fmt.Errorf("invalid yandex key: %w", err)
}
if len(keyBytes) != 16 {
return fmt.Errorf("invalid yandex key length: %d", len(keyBytes))
}
block, err := aes.NewCipher(keyBytes)
if err != nil {
return err
}
return d.streamEncrypted(ctx, sourceURL, outputPath, func(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error) {
stream := cipher.NewCTR(block, make([]byte, aes.BlockSize))
return copyWithProgress(dst, &cipher.StreamReader{S: stream, R: src}, bar)
})
}
func (d *Downloader) file(ctx context.Context, sourceURL, outputPath string, allowProgress bool, includeVideo bool) error {
logDownloadStart(sourceURL, outputPath)
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
return err
}
@@ -119,48 +230,33 @@ func (d *Downloader) file(ctx context.Context, sourceURL, outputPath string, all
if err != nil {
return err
}
defer func() { _ = out.Close() }()
success := false
defer func() {
_ = out.Close()
if !success {
_ = os.Remove(outputPath)
}
}()
if d.ProgressEnabled() && allowProgress && resp.ContentLength > 0 {
d.barStarted.Store(1)
desc := shortenName(filepath.Base(outputPath), 54)
bar := d.progress.AddBar(
resp.ContentLength,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
decor.Percentage(decor.WCSyncWidthR),
),
mpb.AppendDecorators(
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR),
decor.Name(" | ETA ", decor.WCSyncWidth),
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
buf := make([]byte, 256*1024)
for {
n, readErr := reader.Read(buf)
if n > 0 {
if _, writeErr := out.Write(buf[:n]); writeErr != nil {
return writeErr
}
bar.IncrBy(n)
}
if readErr != nil {
if readErr == io.EOF {
break
}
return readErr
}
}
} else {
if _, err = io.Copy(out, reader); err != nil {
return err
}
var bar *mpb.Bar
finishBar := func(bool) {}
if allowProgress {
bar, finishBar = d.newDownloadBar(outputPath, resp.ContentLength)
}
defer func() { finishBar(success) }()
written, err := copyWithProgress(out, reader, bar)
if err != nil {
return err
}
if resp.ContentLength > 0 && written != resp.ContentLength {
return io.ErrUnexpectedEOF
}
if err = out.Sync(); err != nil {
return err
}
success = true
return nil
}
@@ -183,6 +279,102 @@ func (d *Downloader) Logf(format string, args ...any) {
fmt.Print(msg)
}
// copyWithProgress copies src to dst, reporting progress to bar when it is
// non-nil. It replaces three hand-rolled read/write loops that differed only
// in whether they touched a progress bar.
func copyWithProgress(dst io.Writer, src io.Reader, bar *mpb.Bar) (int64, error) {
if bar == nil {
return io.CopyBuffer(dst, src, make([]byte, downloadBufferSize))
}
buf := make([]byte, downloadBufferSize)
total := int64(0)
for {
n, readErr := src.Read(buf)
if n > 0 {
if _, writeErr := dst.Write(buf[:n]); writeErr != nil {
return total, writeErr
}
total += int64(n)
bar.IncrBy(n)
}
if readErr != nil {
if errors.Is(readErr, io.EOF) {
return total, nil
}
return total, readErr
}
}
}
// newDownloadBar creates the progress bar for a download: a determinate bar
// when the content length is known, a spinner when it is not. It returns a nil
// bar when progress display is off, so callers can use the result
// unconditionally, plus a finish func to defer.
//
// This block was copy-pasted into all three download paths; the decorator
// stacks had already started to drift apart.
func (d *Downloader) newDownloadBar(outputPath string, contentLength int64) (*mpb.Bar, func(success bool)) {
if !d.ProgressEnabled() {
return nil, func(bool) {}
}
d.barStarted.Store(1)
desc := shortenName(filepath.Base(outputPath), 54)
if contentLength > 0 {
bar := d.progress.AddBar(
contentLength,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
decor.Percentage(decor.WCSyncWidthR),
),
mpb.AppendDecorators(
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR),
decor.Name(" | ETA ", decor.WCSyncWidth),
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
return bar, func(success bool) {
if !success {
bar.Abort(true)
}
}
}
bar := d.progress.AddSpinner(
0,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
),
mpb.AppendDecorators(
decor.CurrentKibiByte("% .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.Elapsed(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
// Order matches the original defers, which ran LIFO: abort first, then
// finalise the spinner total.
return bar, func(success bool) {
if !success {
bar.Abort(true)
}
bar.SetTotal(-1, true)
}
}
// logDownloadStart emits the source URL and destination filename when the
// user passed -v or higher. The transport-level logger covers the same
// requests at -vv, but this line gives a friendlier per-track summary.
func logDownloadStart(sourceURL, outputPath string) {
if !verbose.Enabled(verbose.V) {
return
}
verbose.Printf(verbose.V, "download %s -> %s\n", sourceURL, filepath.Base(outputPath))
}
func shortenName(name string, max int) string {
if max <= 0 {
return name
@@ -202,24 +394,179 @@ func (d *Downloader) streamManifestWithFFmpeg(ctx context.Context, sourceURL, ou
return fmt.Errorf("ffmpeg not found for manifest stream: %w", err)
}
args := buildFFmpegStreamArgs(sourceURL, outputPath, includeVideo)
if !d.ProgressEnabled() {
cmd := exec.CommandContext(ctx, "ffmpeg", args...)
output, err := cmd.CombinedOutput()
if err != nil {
_ = os.Remove(outputPath)
return fmt.Errorf("ffmpeg stream copy failed: %w: %s", err, string(output))
}
return nil
}
d.barStarted.Store(1)
desc := shortenName(filepath.Base(outputPath), 54)
spinner := d.progress.AddSpinner(
0,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
),
mpb.AppendDecorators(
decor.CurrentKibiByte("% .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.Elapsed(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
cmd := exec.CommandContext(ctx, "ffmpeg", args...)
stderr, err := cmd.StderrPipe()
if err != nil {
spinner.SetTotal(-1, true)
return fmt.Errorf("ffmpeg stream setup failed: %w", err)
}
if err = cmd.Start(); err != nil {
spinner.SetTotal(-1, true)
return fmt.Errorf("ffmpeg stream setup failed: %w", err)
}
var ffOut strings.Builder
scanDone := make(chan scanState, 1)
go func() {
state := scanState{}
shownBytes := int64(0)
barTotalBytes := int64(0)
newEstimatedBar := func(totalBytes int64) *mpb.Bar {
return d.progress.AddBar(
totalBytes,
mpb.PrependDecorators(
decor.Name(desc+" ", decor.WC{W: 56, C: decor.DSyncWidth | decor.DindentRight}),
decor.Percentage(decor.WCSyncWidthR),
),
mpb.AppendDecorators(
decor.CountersKibiByte("% .1f / % .1f", decor.WCSyncWidthR),
decor.Name(" | ", decor.WCSyncWidth),
decor.AverageSpeed(decor.SizeB1024(0), "% .1f", decor.WCSyncWidthR),
decor.Name(" | ETA ", decor.WCSyncWidth),
decor.AverageETA(decor.ET_STYLE_GO, decor.WCSyncWidthR),
),
mpb.BarRemoveOnComplete(),
)
}
var bar *mpb.Bar
scanner := bufio.NewScanner(stderr)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for scanner.Scan() {
line := scanner.Text()
if ffOut.Len() < 256*1024 {
ffOut.WriteString(line)
ffOut.WriteByte('\n')
}
if totalMS, ok := parseFFmpegDurationLine(line); ok && totalMS > 0 {
state.totalMS = totalMS
}
if bitrateBPS, ok := parseFFmpegDurationBitrateBPS(line); ok && bitrateBPS > 0 {
state.bitrateBPS = bitrateBPS
}
if bitrateBPS, ok := parseFFmpegProgressBitrateBPS(line); ok && bitrateBPS > 0 {
state.bitrateBPS = bitrateBPS
}
if totalSize, ok := parseFFmpegTotalSize(line); ok && totalSize > state.currentBytes {
state.currentBytes = totalSize
}
if currentMS, ok := parseFFmpegOutTime(line); ok {
if currentMS > state.currentMS {
state.currentMS = currentMS
}
}
if bar == nil {
estimatedTotal := int64(0)
if state.totalMS > 0 && state.bitrateBPS > 0 {
estimatedTotal = estimateTotalBytesFromBitrate(state.totalMS, state.bitrateBPS)
}
if estimatedTotal == 0 {
estimatedTotal = estimateTotalBytesFromProgress(state.totalMS, state.currentMS, state.currentBytes)
}
if estimatedTotal > 0 {
if estimatedTotal < state.currentBytes {
estimatedTotal = state.currentBytes
}
barTotalBytes = estimatedTotal
spinner.SetTotal(-1, true)
bar = newEstimatedBar(barTotalBytes)
}
}
if state.currentBytes > shownBytes {
delta := state.currentBytes - shownBytes
if bar != nil {
bar.IncrBy(int(delta))
if state.currentBytes > barTotalBytes {
barTotalBytes = state.currentBytes
bar.SetTotal(barTotalBytes, false)
}
} else {
spinner.IncrBy(int(delta))
}
shownBytes = state.currentBytes
}
}
if bar != nil {
if shownBytes < barTotalBytes {
bar.IncrBy(int(barTotalBytes - shownBytes))
}
bar.SetTotal(barTotalBytes, true)
} else {
spinner.SetTotal(-1, true)
}
state.currentBytes = shownBytes
state.scanErr = scanner.Err()
scanDone <- state
}()
waitErr := cmd.Wait()
state := <-scanDone
if waitErr != nil {
_ = os.Remove(outputPath)
if out := strings.TrimSpace(ffOut.String()); out != "" {
return fmt.Errorf("ffmpeg stream copy failed: %w: %s", waitErr, out)
}
return fmt.Errorf("ffmpeg stream copy failed: %w", waitErr)
}
if state.scanErr != nil {
return fmt.Errorf("ffmpeg stream output read failed: %w", state.scanErr)
}
return nil
}
func buildFFmpegStreamArgs(sourceURL, outputPath string, includeVideo bool) []string {
args := []string{
"-y",
"-protocol_whitelist", "file,http,https,tcp,tls,crypto,data",
"-i", sourceURL,
}
if includeVideo {
args = append(args, "-map", "0")
args = append(args,
"-map", "0:v:0?",
"-map", "0:a:0?",
)
} else {
args = append(args, "-map", "0:a:0")
}
args = append(args, "-c", "copy", outputPath)
args = append(args, "-c", "copy", "-hide_banner", "-nostats", "-progress", "pipe:2", outputPath)
return args
}
cmd := exec.CommandContext(ctx, "ffmpeg", args...)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("ffmpeg stream copy failed: %w: %s", err, string(output))
}
return nil
type scanState struct {
totalMS int64
currentMS int64
bitrateBPS int64
currentBytes int64
scanErr error
}
func isManifestResponse(contentType string, peek []byte) bool {
@@ -228,7 +575,7 @@ func isManifestResponse(contentType string, peek []byte) bool {
return true
}
s := strings.TrimSpace(strings.ToLower(string(peek)))
if strings.HasPrefix(s, "<?xml") && strings.Contains(s, "<mpd") {
if (strings.HasPrefix(s, "<?xml") && strings.Contains(s, "<mpd")) || strings.HasPrefix(s, "<mpd") {
return true
}
if strings.HasPrefix(s, "#extm3u") {
@@ -237,59 +584,209 @@ func isManifestResponse(contentType string, peek []byte) bool {
return false
}
func parseFFmpegDurationLine(line string) (int64, bool) {
idx := strings.Index(line, "Duration:")
if idx < 0 {
return 0, false
}
raw := strings.TrimSpace(line[idx+len("Duration:"):])
if raw == "" {
return 0, false
}
if comma := strings.Index(raw, ","); comma >= 0 {
raw = raw[:comma]
}
return parseClockDurationMS(raw)
}
func parseFFmpegOutTime(line string) (int64, bool) {
if !strings.HasPrefix(line, "out_time=") {
return 0, false
}
raw := strings.TrimSpace(strings.TrimPrefix(line, "out_time="))
if raw == "" || strings.EqualFold(raw, "N/A") {
return 0, false
}
return parseClockDurationMS(raw)
}
func parseFFmpegTotalSize(line string) (int64, bool) {
if !strings.HasPrefix(line, "total_size=") {
return 0, false
}
raw := strings.TrimSpace(strings.TrimPrefix(line, "total_size="))
if raw == "" || strings.EqualFold(raw, "N/A") {
return 0, false
}
v, err := strconv.ParseInt(raw, 10, 64)
if err != nil || v < 0 {
return 0, false
}
return v, true
}
func parseFFmpegDurationBitrateBPS(line string) (int64, bool) {
idx := strings.Index(strings.ToLower(line), "bitrate:")
if idx < 0 {
return 0, false
}
raw := strings.TrimSpace(line[idx+len("bitrate:"):])
if raw == "" {
return 0, false
}
return parseFFmpegBitrateBPS(raw)
}
func parseFFmpegProgressBitrateBPS(line string) (int64, bool) {
if !strings.HasPrefix(line, "bitrate=") {
return 0, false
}
raw := strings.TrimSpace(strings.TrimPrefix(line, "bitrate="))
if raw == "" || strings.EqualFold(raw, "N/A") {
return 0, false
}
return parseFFmpegBitrateBPS(raw)
}
func parseFFmpegBitrateBPS(raw string) (int64, bool) {
raw = strings.TrimSpace(raw)
if raw == "" || strings.EqualFold(raw, "N/A") {
return 0, false
}
parts := strings.Fields(strings.ToLower(raw))
if len(parts) == 0 {
return 0, false
}
numStr := parts[0]
unit := ""
if len(parts) > 1 {
unit = parts[1]
} else {
i := 0
for i < len(numStr) {
c := numStr[i]
if (c >= '0' && c <= '9') || c == '.' {
i++
continue
}
break
}
if i < len(numStr) {
unit = numStr[i:]
numStr = numStr[:i]
}
}
num, err := strconv.ParseFloat(strings.TrimSpace(numStr), 64)
if err != nil || num <= 0 {
return 0, false
}
unit = strings.TrimSpace(strings.Trim(unit, ","))
mult := float64(1)
switch unit {
case "kb/s", "kbits/s", "kbit/s", "kbps":
mult = 1000
case "mb/s", "mbits/s", "mbit/s", "mbps":
mult = 1000 * 1000
case "gb/s", "gbits/s", "gbit/s", "gbps":
mult = 1000 * 1000 * 1000
case "b/s", "bit/s", "bits/s", "":
mult = 1
default:
return 0, false
}
bps := int64(num * mult)
if bps <= 0 {
return 0, false
}
return bps, true
}
func estimateTotalBytesFromBitrate(totalMS, bitrateBPS int64) int64 {
if totalMS <= 0 || bitrateBPS <= 0 {
return 0
}
return (totalMS * bitrateBPS) / 8000
}
func estimateTotalBytesFromProgress(totalMS, currentMS, currentBytes int64) int64 {
if totalMS <= 0 || currentMS <= 0 || currentBytes <= 0 {
return 0
}
if currentMS > totalMS {
currentMS = totalMS
}
est := (currentBytes * totalMS) / currentMS
if est < currentBytes {
est = currentBytes
}
return est
}
func parseClockDurationMS(raw string) (int64, bool) {
parts := strings.Split(strings.TrimSpace(raw), ":")
if len(parts) != 3 {
return 0, false
}
hours, err := strconv.ParseInt(parts[0], 10, 64)
if err != nil || hours < 0 {
return 0, false
}
minutes, err := strconv.ParseInt(parts[1], 10, 64)
if err != nil || minutes < 0 {
return 0, false
}
secPart := parts[2]
secRaw, fracRaw, hasFrac := strings.Cut(secPart, ".")
seconds, err := strconv.ParseInt(secRaw, 10, 64)
if err != nil || seconds < 0 {
return 0, false
}
ms := ((hours*60+minutes)*60 + seconds) * 1000
if hasFrac {
if len(fracRaw) > 3 {
fracRaw = fracRaw[:3]
}
for len(fracRaw) < 3 {
fracRaw += "0"
}
fracMS, fracErr := strconv.ParseInt(fracRaw, 10, 64)
if fracErr != nil || fracMS < 0 {
return 0, false
}
ms += fracMS
}
return ms, true
}
const deezerBFChunkSize = 2048
var deezerBFIV = []byte{0, 1, 2, 3, 4, 5, 6, 7}
func decryptDeezerBFCBCStripe(in []byte, trackID string) ([]byte, error) {
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(trackID))
if err != nil {
return nil, err
// defaultDeezerBlowfishSecret aliases the single definition in internal/config
// so the literal is not duplicated here.
const defaultDeezerBlowfishSecret = config.DefaultDeezerBlowfishSecret
// blowfishSecret returns the configured Deezer blowfish secret.
//
// The key derivation XORs 16 bytes of it, so a secret of any other length
// would panic; fall back to the built-in value rather than crash on a
// mistyped config entry.
func (d *Downloader) blowfishSecret() string {
if len(d.DeezerBlowfishSecret) == 16 {
return d.DeezerBlowfishSecret
}
out := make([]byte, len(in))
for i := 0; i*deezerBFChunkSize < len(in); i++ {
start := i * deezerBFChunkSize
end := start + deezerBFChunkSize
if end > len(in) {
end = len(in)
}
chunk := in[start:end]
if i%3 == 0 && len(chunk) == deezerBFChunkSize {
dec := make([]byte, len(chunk))
mode := cipher.NewCBCDecrypter(block, deezerBFIV)
mode.CryptBlocks(dec, chunk)
copy(out[start:end], dec)
} else {
copy(out[start:end], chunk)
}
if d.DeezerBlowfishSecret != "" {
verbose.Printf(verbose.V, "ignoring deezer.blowfish_secret: got %d bytes, want 16\n", len(d.DeezerBlowfishSecret))
}
return out, nil
return defaultDeezerBlowfishSecret
}
func deriveDeezerBlowfishKey(trackID string) []byte {
func deriveDeezerBlowfishKey(secret, trackID string) []byte {
sum := md5.Sum([]byte(trackID))
md5Hex := fmt.Sprintf("%x", sum)
secret := "g4el58wc0zvf9na1"
key := make([]byte, 16)
for i := 0; i < 16; i++ {
key[i] = md5Hex[i] ^ md5Hex[i+16] ^ secret[i]
}
return key
}
func normalizeDeezerTrackID(raw string) string {
trimmed := strings.TrimSpace(raw)
if trimmed == "" {
return ""
}
if _, err := strconv.Atoi(trimmed); err == nil {
return trimmed
}
parts := strings.Split(strings.Trim(trimmed, "/"), "/")
for i := len(parts) - 1; i >= 0; i-- {
if _, err := strconv.Atoi(parts[i]); err == nil {
return parts[i]
}
}
return trimmed
}
+439 -13
View File
@@ -1,19 +1,28 @@
package download
import (
"bytes"
"context"
"crypto/aes"
"crypto/cipher"
"encoding/hex"
"errors"
"io"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"golang.org/x/crypto/blowfish"
)
func TestDownloaderHasNoClientTimeout(t *testing.T) {
d := NewWithOptions(true, false)
d := NewWithOptions(true, false, 0)
if d.http.Timeout != 0 {
t.Fatalf("http timeout = %v, want 0 (no global timeout)", d.http.Timeout)
}
@@ -44,9 +53,15 @@ func TestManifestDetection(t *testing.T) {
if !isManifestResponse("application/dash+xml", []byte("x")) {
t.Fatalf("expected dash content-type to be manifest")
}
if !isManifestResponse("application/vnd.apple.mpegurl", []byte("x")) {
t.Fatalf("expected mpegurl content-type to be manifest")
}
if !isManifestResponse("application/octet-stream", []byte("<?xml version='1.0'?><MPD></MPD>")) {
t.Fatalf("expected MPD XML body to be manifest")
}
if !isManifestResponse("application/octet-stream", []byte("<MPD type='static'></MPD>")) {
t.Fatalf("expected MPD body without xml prolog to be manifest")
}
if !isManifestResponse("text/plain", []byte("#EXTM3U\n#EXT-X-VERSION:3")) {
t.Fatalf("expected HLS body to be manifest")
}
@@ -55,32 +70,443 @@ func TestManifestDetection(t *testing.T) {
}
}
func TestNormalizeDeezerTrackID(t *testing.T) {
if got := normalizeDeezerTrackID("https://www.deezer.com/track/3135556"); got != "3135556" {
t.Fatalf("normalize track id = %q, want 3135556", got)
func TestDeezerBlowfishKeyDerivation(t *testing.T) {
trackID := "3135556"
key := deriveDeezerBlowfishKey(defaultDeezerBlowfishSecret, trackID)
if len(key) != 16 {
t.Fatalf("blowfish key len = %d, want 16", len(key))
}
}
func TestDecryptDeezerBFCBCStripe(t *testing.T) {
func TestFileDeezerEncrypted(t *testing.T) {
trackID := "3135556"
plain := make([]byte, deezerBFChunkSize*2)
plain := make([]byte, deezerBFChunkSize+777)
for i := range plain {
plain[i] = byte(i % 251)
plain[i] = byte((i * 7) % 251)
}
enc := make([]byte, len(plain))
copy(enc, plain)
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(trackID))
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(defaultDeezerBlowfishSecret, trackID))
if err != nil {
t.Fatalf("cipher error: %v", err)
}
cbc := cipher.NewCBCEncrypter(block, deezerBFIV)
cbc.CryptBlocks(enc[:deezerBFChunkSize], enc[:deezerBFChunkSize])
dec, err := decryptDeezerBFCBCStripe(enc, trackID)
if err != nil {
t.Fatalf("decrypt error: %v", err)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(enc)
}))
defer ts.Close()
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "a.flac")
if err = d.FileDeezerEncrypted(context.Background(), ts.URL, out, trackID); err != nil {
t.Fatalf("FileDeezerEncrypted() error = %v", err)
}
if len(dec) != len(plain) || string(dec) != string(plain) {
t.Fatalf("decrypted data mismatch")
got, err := os.ReadFile(out)
if err != nil {
t.Fatalf("ReadFile() error = %v", err)
}
if string(got) != string(plain) {
t.Fatalf("decrypted file mismatch")
}
}
func TestFileYandexEncrypted(t *testing.T) {
plain := make([]byte, 8192+333)
for i := range plain {
plain[i] = byte((i * 11) % 251)
}
keyHex := "00112233445566778899aabbccddeeff"
key, err := hex.DecodeString(keyHex)
if err != nil {
t.Fatalf("DecodeString() error = %v", err)
}
block, err := aes.NewCipher(key)
if err != nil {
t.Fatalf("NewCipher() error = %v", err)
}
enc := make([]byte, len(plain))
copy(enc, plain)
stream := cipher.NewCTR(block, make([]byte, aes.BlockSize))
stream.XORKeyStream(enc, enc)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(enc)
}))
defer ts.Close()
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "a.m4a")
if err = d.FileYandexEncrypted(context.Background(), ts.URL, out, keyHex); err != nil {
t.Fatalf("FileYandexEncrypted() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatalf("ReadFile() error = %v", err)
}
if string(got) != string(plain) {
t.Fatalf("decrypted file mismatch")
}
}
func TestDownloaderFileTruncatedResponseRemovesPartialFile(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", "10")
_, _ = w.Write([]byte("abc"))
}))
defer ts.Close()
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "a.bin")
err := d.File(context.Background(), ts.URL, out)
if err == nil || !errors.Is(err, io.ErrUnexpectedEOF) {
t.Fatalf("expected unexpected EOF, got %v", err)
}
if _, statErr := os.Stat(out); !errors.Is(statErr, os.ErrNotExist) {
t.Fatalf("expected no partial file, stat err=%v", statErr)
}
}
func TestFileDeezerEncryptedTruncatedResponseRemovesPartialFile(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", "4096")
_, _ = w.Write([]byte("short"))
}))
defer ts.Close()
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "a.flac")
err := d.FileDeezerEncrypted(context.Background(), ts.URL, out, "3135556")
if err == nil || !errors.Is(err, io.ErrUnexpectedEOF) {
t.Fatalf("expected unexpected EOF, got %v", err)
}
if _, statErr := os.Stat(out); !errors.Is(statErr, os.ErrNotExist) {
t.Fatalf("expected no partial file, stat err=%v", statErr)
}
}
func TestFileDeezerEncryptedBadStatus(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusForbidden)
}))
defer ts.Close()
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "a.flac")
err := d.FileDeezerEncrypted(context.Background(), ts.URL, out, "3135556")
if err == nil || !strings.Contains(err.Error(), "status=403") {
t.Fatalf("expected status error, got %v", err)
}
}
func TestFileYandexEncryptedBadKey(t *testing.T) {
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "a.m4a")
err := d.FileYandexEncrypted(context.Background(), "https://example.com/file", out, "abcd")
if err == nil || !strings.Contains(err.Error(), "invalid yandex key length") {
t.Fatalf("expected key length error, got %v", err)
}
}
func TestDownloaderFileContextCancellationRemovesPartialFile(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/octet-stream")
_, _ = w.Write([]byte("partial-data"))
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
<-r.Context().Done()
}))
defer ts.Close()
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "x", "cancel.bin")
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Millisecond)
defer cancel()
err := d.File(ctx, ts.URL, out)
if err == nil {
t.Fatalf("expected context cancellation error")
}
if _, statErr := os.Stat(out); !errors.Is(statErr, os.ErrNotExist) {
t.Fatalf("expected no partial file, stat err=%v", statErr)
}
}
func TestStreamManifestWithFFmpegMissing(t *testing.T) {
d := NewWithOptions(true, false, 0)
t.Setenv("PATH", "")
err := d.streamManifestWithFFmpeg(context.Background(), "https://example.com/live.m3u8", filepath.Join(t.TempDir(), "out.m4a"), false)
if err == nil || !strings.Contains(strings.ToLower(err.Error()), "ffmpeg not found") {
t.Fatalf("expected ffmpeg missing error, got %v", err)
}
}
func TestStreamManifestWithFFmpegFailureRemovesPartialFile(t *testing.T) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
t.Skip("ffmpeg not installed")
}
d := NewWithOptions(true, false, 0)
out := filepath.Join(t.TempDir(), "out.m4a")
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
err := d.streamManifestWithFFmpeg(ctx, "https://127.0.0.1:1/unreachable.m3u8", out, false)
if err == nil || !strings.Contains(err.Error(), "ffmpeg stream copy failed") {
t.Fatalf("expected ffmpeg failure error, got %v", err)
}
if _, statErr := os.Stat(out); !errors.Is(statErr, os.ErrNotExist) {
t.Fatalf("expected no partial file after ffmpeg failure, stat err=%v", statErr)
}
}
func TestParseFFmpegDurationLine(t *testing.T) {
totalMS, ok := parseFFmpegDurationLine(" Duration: 00:04:52.57, start: 0.000000, bitrate: 975 kb/s")
if !ok {
t.Fatalf("expected duration parse to succeed")
}
if want := int64(292570); totalMS != want {
t.Fatalf("unexpected duration ms: got=%d want=%d", totalMS, want)
}
}
func TestParseFFmpegDurationBitrateBPS(t *testing.T) {
bps, ok := parseFFmpegDurationBitrateBPS(" Duration: 00:04:52.57, start: 0.000000, bitrate: 975 kb/s")
if !ok {
t.Fatalf("expected bitrate parse to succeed")
}
if want := int64(975000); bps != want {
t.Fatalf("unexpected bitrate: got=%d want=%d", bps, want)
}
}
func TestParseFFmpegProgressBitrateBPS(t *testing.T) {
bps, ok := parseFFmpegProgressBitrateBPS("bitrate=1706.8kbits/s")
if !ok {
t.Fatalf("expected progress bitrate parse to succeed")
}
if want := int64(1706800); bps != want {
t.Fatalf("unexpected bitrate: got=%d want=%d", bps, want)
}
}
func TestParseFFmpegTotalSize(t *testing.T) {
size, ok := parseFFmpegTotalSize("total_size=1234567")
if !ok {
t.Fatalf("expected total_size parse to succeed")
}
if want := int64(1234567); size != want {
t.Fatalf("unexpected total_size: got=%d want=%d", size, want)
}
}
func TestParseFFmpegOutTime(t *testing.T) {
currentMS, ok := parseFFmpegOutTime("out_time=00:01:02.340000")
if !ok {
t.Fatalf("expected out_time parse to succeed")
}
if want := int64(62340); currentMS != want {
t.Fatalf("unexpected out_time ms: got=%d want=%d", currentMS, want)
}
}
func TestParseClockDurationMSInvalid(t *testing.T) {
if _, ok := parseClockDurationMS("bad"); ok {
t.Fatalf("expected invalid duration to fail")
}
if _, ok := parseClockDurationMS("00:12"); ok {
t.Fatalf("expected short duration to fail")
}
}
func TestEstimateTotalBytesFromBitrate(t *testing.T) {
total := estimateTotalBytesFromBitrate(10000, 1600000)
if want := int64(2000000); total != want {
t.Fatalf("unexpected estimate from bitrate: got=%d want=%d", total, want)
}
}
func TestEstimateTotalBytesFromProgress(t *testing.T) {
total := estimateTotalBytesFromProgress(10000, 2500, 500000)
if want := int64(2000000); total != want {
t.Fatalf("unexpected estimate from progress: got=%d want=%d", total, want)
}
}
func TestBuildFFmpegStreamArgsAudioOnly(t *testing.T) {
args := buildFFmpegStreamArgs("https://example.com/master.m3u8", "/tmp/out.m4a", false)
if !containsArgPair(args, "-map", "0:a:0") {
t.Fatalf("expected audio map in args: %v", args)
}
if containsArgPair(args, "-map", "0:v:0?") {
t.Fatalf("did not expect video map in audio-only args: %v", args)
}
if containsArgPair(args, "-map", "0") {
t.Fatalf("did not expect broad map=0 in args: %v", args)
}
}
func TestBuildFFmpegStreamArgsIncludeVideo(t *testing.T) {
args := buildFFmpegStreamArgs("https://example.com/master.m3u8", "/tmp/out.mp4", true)
if !containsArgPair(args, "-map", "0:v:0?") {
t.Fatalf("expected video map in args: %v", args)
}
if !containsArgPair(args, "-map", "0:a:0?") {
t.Fatalf("expected audio map in args: %v", args)
}
if containsArgPair(args, "-map", "0") {
t.Fatalf("did not expect broad map=0 in args: %v", args)
}
}
func containsArgPair(args []string, key, value string) bool {
for i := 0; i+1 < len(args); i++ {
if args[i] == key && args[i+1] == value {
return true
}
}
return false
}
// newProgressDownloader builds a Downloader with the progress bars actually
// enabled. The bar code paths are otherwise never exercised, because tests do
// not run on a terminal; STREAMRIP_GO_FORCE_PROGRESS is the documented escape
// hatch. Output goes to stderr, which `go test` captures.
func newProgressDownloader(t *testing.T) *Downloader {
t.Helper()
t.Setenv("STREAMRIP_GO_FORCE_PROGRESS", "1")
d := NewWithOptions(true, true, 0)
if !d.ProgressEnabled() {
t.Fatal("expected progress to be enabled")
}
t.Cleanup(d.Close)
return d
}
// TestFileWithProgressBar covers the determinate-bar path (Content-Length
// known) through the shared newDownloadBar/copyWithProgress helpers.
func TestFileWithProgressBar(t *testing.T) {
payload := bytes.Repeat([]byte("abcd"), 4096)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", strconv.Itoa(len(payload)))
_, _ = w.Write(payload)
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "sub", "a.flac")
if err := d.File(context.Background(), ts.URL, out); err != nil {
t.Fatalf("File() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("content mismatch: got %d bytes, want %d", len(got), len(payload))
}
}
// TestFileWithSpinnerUnknownLength covers the spinner path, taken when the
// server does not send a Content-Length.
func TestFileWithSpinnerUnknownLength(t *testing.T) {
payload := bytes.Repeat([]byte("xyz"), 3000)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Transfer-Encoding", "chunked")
_, _ = w.Write(payload)
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "a.flac")
if err := d.File(context.Background(), ts.URL, out); err != nil {
t.Fatalf("File() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("content mismatch: got %d bytes, want %d", len(got), len(payload))
}
}
// TestFileDeezerEncryptedWithProgressBar runs the Deezer decryption through
// the progress-enabled path, so the shared streamEncrypted scaffolding is
// covered with a bar attached as well as without.
func TestFileDeezerEncryptedWithProgressBar(t *testing.T) {
plain := make([]byte, deezerBFChunkSize*4+100)
for i := range plain {
plain[i] = byte((i * 7) % 253)
}
trackID := "123456"
enc := encryptDeezerPayload(t, plain, trackID)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", strconv.Itoa(len(enc)))
_, _ = w.Write(enc)
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "a.mp3")
if err := d.FileDeezerEncrypted(context.Background(), ts.URL, out, trackID); err != nil {
t.Fatalf("FileDeezerEncrypted() error = %v", err)
}
got, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, plain) {
t.Fatal("decrypted output does not match the original plaintext")
}
}
// encryptDeezerPayload applies Deezer's scheme: Blowfish-CBC on every third
// full 2048-byte chunk, everything else left in the clear.
func encryptDeezerPayload(t *testing.T, plain []byte, trackID string) []byte {
t.Helper()
block, err := blowfish.NewCipher(deriveDeezerBlowfishKey(defaultDeezerBlowfishSecret, trackID))
if err != nil {
t.Fatal(err)
}
enc := make([]byte, len(plain))
copy(enc, plain)
for i, off := 0, 0; off < len(plain); i, off = i+1, off+deezerBFChunkSize {
end := off + deezerBFChunkSize
if end > len(plain) {
break
}
if i%3 != 0 {
continue
}
mode := cipher.NewCBCEncrypter(block, deezerBFIV)
mode.CryptBlocks(enc[off:end], plain[off:end])
}
return enc
}
// TestProgressBarFailurePathRemovesPartialFile checks that a mid-download
// failure still runs the bar cleanup and removes the partial output, which is
// the behaviour the finishBar(success) defer replaced.
func TestProgressBarFailurePathRemovesPartialFile(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", "100000")
_, _ = w.Write(bytes.Repeat([]byte("a"), 10))
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
}))
defer ts.Close()
d := newProgressDownloader(t)
out := filepath.Join(t.TempDir(), "a.flac")
if err := d.File(context.Background(), ts.URL, out); err == nil {
t.Fatal("expected a truncated-response error")
}
if _, err := os.Stat(out); !os.IsNotExist(err) {
t.Fatalf("partial file not removed, stat err = %v", err)
}
}
+160
View File
@@ -0,0 +1,160 @@
// Package jsonutil provides shared helpers for working with untyped JSON
// values (map[string]any) that come from API responses across all providers.
package jsonutil
import (
"strconv"
"strings"
)
// StringFromAny converts a dynamic JSON value to a string.
// Numeric types are formatted without trailing zeroes.
func StringFromAny(v any) string {
switch t := v.(type) {
case string:
return t
case int:
return strconv.Itoa(t)
case int64:
return strconv.FormatInt(t, 10)
case float64:
return strconv.FormatFloat(t, 'f', -1, 64)
default:
return ""
}
}
// IntFromAny converts a dynamic JSON value to an int.
// Handles int, int64, float64, and string types.
func IntFromAny(v any) int {
switch t := v.(type) {
case int:
return t
case int32:
return int(t)
case int64:
return int(t)
case float64:
return int(t)
case string:
i, _ := strconv.Atoi(strings.TrimSpace(t))
return i
default:
return 0
}
}
// FloatFromAny converts a dynamic JSON value to a float64.
func FloatFromAny(v any) float64 {
switch t := v.(type) {
case float64:
return t
case int:
return float64(t)
case int64:
return float64(t)
default:
return 0
}
}
// BoolFromAny converts a dynamic JSON value to a bool.
// Supports bool, string ("true"/"1"/"yes"), and numeric types.
func BoolFromAny(v any) bool {
switch t := v.(type) {
case bool:
return t
case string:
l := strings.ToLower(strings.TrimSpace(t))
return l == "1" || l == "true" || l == "yes"
case int:
return t != 0
case int64:
return t != 0
case float64:
return t != 0
default:
return false
}
}
// FirstNonEmpty returns the first string in items that is non-empty after trimming.
func FirstNonEmpty(items ...string) string {
for _, item := range items {
if strings.TrimSpace(item) != "" {
return strings.TrimSpace(item)
}
}
return ""
}
// NestedMap returns the value at m[key] as a map[string]any.
// Returns an empty map if the key is missing or the value is not a map.
func NestedMap(m map[string]any, key string) map[string]any {
v, ok := m[key].(map[string]any)
if !ok {
return map[string]any{}
}
return v
}
// NestedAny traverses a chain of map keys and returns the final value.
func NestedAny(v map[string]any, keys ...string) any {
cur := any(v)
for _, key := range keys {
m, ok := cur.(map[string]any)
if !ok {
return nil
}
cur = m[key]
}
return cur
}
// NestedString traverses a chain of map keys and returns the final value as a string.
func NestedString(v map[string]any, keys ...string) string {
return StringFromAny(NestedAny(v, keys...))
}
// TitleCase capitalises the first rune of s. This is a simple ASCII replacement
// for the deprecated strings.Title function, suitable for source names like
// "qobuz" → "Qobuz".
func TitleCase(s string) string {
if s == "" {
return s
}
r := []rune(s)
if r[0] >= 'a' && r[0] <= 'z' {
r[0] -= 'a' - 'A'
}
return string(r)
}
// DisplayArtistNames joins artist names the way the providers present them:
// "A", "A & B", or "A, B & C". Previously duplicated byte-for-byte in
// internal/app, provider/yandex and provider/beatport.
func DisplayArtistNames(names []string) string {
switch len(names) {
case 0:
return ""
case 1:
return names[0]
case 2:
return names[0] + " & " + names[1]
default:
return strings.Join(names[:len(names)-1], ", ") + " & " + names[len(names)-1]
}
}
// FirstPositiveInt returns the first value greater than zero, or 0 if there is
// none. Useful for picking among optional metadata fields where 0 means
// "absent". Previously duplicated in provider/yandex, provider/beatport and
// cmd/rip.
func FirstPositiveInt(values ...int) int {
for _, v := range values {
if v > 0 {
return v
}
}
return 0
}
+203
View File
@@ -0,0 +1,203 @@
package jsonutil
import "testing"
func TestStringFromAny(t *testing.T) {
tests := []struct {
name string
in any
want string
}{
{"string", "hi", "hi"},
{"int", 7, "7"},
{"int64", int64(8), "8"},
{"float whole", float64(9), "9"},
{"float fractional", 44.1, "44.1"},
{"float no trailing zeroes", float64(100), "100"},
{"nil", nil, ""},
{"bool unsupported", true, ""},
{"map unsupported", map[string]any{}, ""},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := StringFromAny(tc.in); got != tc.want {
t.Fatalf("StringFromAny(%#v) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
func TestIntFromAny(t *testing.T) {
tests := []struct {
in any
want int
}{
{5, 5},
{int32(6), 6},
{int64(7), 7},
{8.9, 8},
{"10", 10},
{" 11 ", 11},
{"not a number", 0},
{nil, 0},
{true, 0},
}
for _, tc := range tests {
if got := IntFromAny(tc.in); got != tc.want {
t.Errorf("IntFromAny(%#v) = %d, want %d", tc.in, got, tc.want)
}
}
}
func TestFloatFromAny(t *testing.T) {
tests := []struct {
in any
want float64
}{
{1.5, 1.5},
{2, 2},
{int64(3), 3},
{"4.5", 0}, // strings are deliberately not parsed here
{nil, 0},
}
for _, tc := range tests {
if got := FloatFromAny(tc.in); got != tc.want {
t.Errorf("FloatFromAny(%#v) = %v, want %v", tc.in, got, tc.want)
}
}
}
func TestBoolFromAny(t *testing.T) {
tests := []struct {
in any
want bool
}{
{true, true},
{false, false},
{"true", true},
{"TRUE", true},
{" yes ", true},
{"1", true},
{"0", false},
{"no", false},
{1, true},
{0, false},
{int64(2), true},
{0.0, false},
{1.5, true},
{nil, false},
}
for _, tc := range tests {
if got := BoolFromAny(tc.in); got != tc.want {
t.Errorf("BoolFromAny(%#v) = %v, want %v", tc.in, got, tc.want)
}
}
}
func TestFirstNonEmpty(t *testing.T) {
if got := FirstNonEmpty("", " ", "found", "later"); got != "found" {
t.Errorf("got %q", got)
}
if got := FirstNonEmpty(" trimmed "); got != "trimmed" {
t.Errorf("got %q, want trimmed", got)
}
if got := FirstNonEmpty("", " "); got != "" {
t.Errorf("got %q, want empty", got)
}
if got := FirstNonEmpty(); got != "" {
t.Errorf("got %q, want empty", got)
}
}
func TestNestedMap(t *testing.T) {
m := map[string]any{"album": map[string]any{"title": "X"}, "n": 1}
if got := NestedMap(m, "album")["title"]; got != "X" {
t.Errorf("got %v", got)
}
if got := NestedMap(m, "missing"); len(got) != 0 {
t.Errorf("missing key should give empty map, got %v", got)
}
if got := NestedMap(m, "n"); len(got) != 0 {
t.Errorf("non-map value should give empty map, got %v", got)
}
}
func TestNestedAnyAndString(t *testing.T) {
m := map[string]any{
"album": map[string]any{
"artist": map[string]any{"name": "Nina Simone", "id": float64(42)},
},
}
if got := NestedString(m, "album", "artist", "name"); got != "Nina Simone" {
t.Errorf("got %q", got)
}
if got := NestedString(m, "album", "artist", "id"); got != "42" {
t.Errorf("numeric id = %q, want 42", got)
}
if got := NestedString(m, "album", "nope", "name"); got != "" {
t.Errorf("missing path = %q, want empty", got)
}
if got := NestedAny(m, "album", "artist", "name"); got != "Nina Simone" {
t.Errorf("got %v", got)
}
if got := NestedAny(m); got == nil {
t.Error("no keys should return the map itself")
}
// Traversing through a non-map must not panic.
if got := NestedAny(m, "album", "artist", "name", "deeper"); got != nil {
t.Errorf("over-deep path = %v, want nil", got)
}
}
func TestTitleCase(t *testing.T) {
tests := []struct{ in, want string }{
{"qobuz", "Qobuz"},
{"Tidal", "Tidal"},
{"", ""},
{"1st", "1st"},
{"ä", "ä"}, // only ASCII is uppercased
}
for _, tc := range tests {
if got := TitleCase(tc.in); got != tc.want {
t.Errorf("TitleCase(%q) = %q, want %q", tc.in, got, tc.want)
}
}
}
func TestDisplayArtistNames(t *testing.T) {
tests := []struct {
in []string
want string
}{
{nil, ""},
{[]string{}, ""},
{[]string{"A"}, "A"},
{[]string{"A", "B"}, "A & B"},
{[]string{"A", "B", "C"}, "A, B & C"},
{[]string{"A", "B", "C", "D"}, "A, B, C & D"},
}
for _, tc := range tests {
if got := DisplayArtistNames(tc.in); got != tc.want {
t.Errorf("DisplayArtistNames(%v) = %q, want %q", tc.in, got, tc.want)
}
}
}
func TestFirstPositiveInt(t *testing.T) {
tests := []struct {
in []int
want int
}{
{nil, 0},
{[]int{0, 0}, 0},
{[]int{0, 5, 7}, 5},
{[]int{-3, 0, 2}, 2},
{[]int{9}, 9},
{[]int{-1}, 0},
}
for _, tc := range tests {
if got := FirstPositiveInt(tc.in...); got != tc.want {
t.Errorf("FirstPositiveInt(%v) = %d, want %d", tc.in, got, tc.want)
}
}
}
+57 -2
View File
@@ -6,6 +6,7 @@ import (
"strconv"
"strings"
"unicode"
"unicode/utf8"
)
type Config struct {
@@ -14,7 +15,7 @@ type Config struct {
}
var tokenRe = regexp.MustCompile(`\{([a-z_]+)(?::0?(\d+))?\}`)
var invalidPathRe = regexp.MustCompile(`[<>:"/\\|?*\x00-\x1F]`)
var invalidPathRe = regexp.MustCompile(`[<>:"/\\|?*\x00-\x1F\x7F]`)
func FormatTemplate(template string, values map[string]string) string {
return tokenRe.ReplaceAllStringFunc(template, func(m string) string {
@@ -35,6 +36,28 @@ func FormatTemplate(template string, values map[string]string) string {
})
}
// maxNameBytes is the per-component limit on ext4, APFS and most other modern
// filesystems. TruncateTo counts runes, so a 120-rune CJK title is ~360 bytes
// and would fail with ENAMETOOLONG without this second, byte-based cap.
const maxNameBytes = 255
// windowsReserved are device names that cannot be used as a path component on
// Windows, with or without an extension.
var windowsReserved = map[string]struct{}{
"CON": {}, "PRN": {}, "AUX": {}, "NUL": {},
"COM1": {}, "COM2": {}, "COM3": {}, "COM4": {}, "COM5": {},
"COM6": {}, "COM7": {}, "COM8": {}, "COM9": {},
"LPT1": {}, "LPT2": {}, "LPT3": {}, "LPT4": {}, "LPT5": {},
"LPT6": {}, "LPT7": {}, "LPT8": {}, "LPT9": {},
}
// CleanName turns a remote title into a single safe path component.
//
// Everything it receives is attacker-controlled: album, playlist and track
// titles come straight from provider APIs, and on SoundCloud and Tidal they are
// user-generated. The result is joined onto the download root by the caller, so
// it must never be "." or ".." (which would escape or collapse into the parent
// directory) and must never contain a separator.
func CleanName(in string, cfg Config) string {
s := strings.TrimSpace(in)
s = invalidPathRe.ReplaceAllString(s, "_")
@@ -53,13 +76,45 @@ func CleanName(in string, cfg Config) string {
s = string(runes[:cfg.TruncateTo])
}
}
s = truncateBytes(s, maxNameBytes)
// Trailing dots and spaces are stripped by Windows, which turns "foo." into
// "foo" and makes ".." survive as a traversal component.
s = strings.TrimRight(s, ". ")
s = strings.TrimSpace(s)
if s == "" {
if s == "" || isAllDots(s) {
return "Unknown"
}
if _, bad := windowsReserved[strings.ToUpper(s)]; bad {
return s + "_"
}
return s
}
// truncateBytes shortens s to at most max bytes without splitting a rune.
func truncateBytes(s string, max int) string {
if len(s) <= max {
return s
}
cut := s[:max]
for len(cut) > 0 && !utf8.ValidString(cut) {
cut = cut[:len(cut)-1]
}
return cut
}
// isAllDots reports whether s consists only of "." characters, i.e. "." or
// ".." or any longer run that some filesystems also treat as relative.
func isAllDots(s string) bool {
for _, ch := range s {
if ch != '.' {
return false
}
}
return len(s) > 0
}
func YearFromDate(date string) string {
if len(date) >= 4 {
prefix := date[:4]
+127 -4
View File
@@ -1,6 +1,10 @@
package naming
import "testing"
import (
"path/filepath"
"strings"
"testing"
)
func TestFormatTemplate(t *testing.T) {
got := FormatTemplate("{tracknumber:02}. {artist} - {title}{explicit}", map[string]string{
@@ -15,8 +19,127 @@ func TestFormatTemplate(t *testing.T) {
}
func TestCleanName(t *testing.T) {
got := CleanName(" Dreams/Live ", Config{RestrictCharacters: false, TruncateTo: 120})
if got != "Dreams_Live" {
t.Fatalf("got %q", got)
cfg := Config{RestrictCharacters: false, TruncateTo: 120}
tests := []struct {
name string
in string
want string
}{
{"trims and replaces separator", " Dreams/Live ", "Dreams_Live"},
{"backslash", `AC\DC`, "AC_DC"},
{"colon", "Album: Deluxe", "Album_ Deluxe"},
{"control chars", "Song\x01\x02", "Song__"},
{"del char", "Song\x7f", "Song_"},
{"empty", "", "Unknown"},
{"whitespace only", " ", "Unknown"},
// Path traversal: these must never be returned verbatim, or the
// caller's filepath.Join escapes the download root.
{"dotdot", "..", "Unknown"},
{"single dot", ".", "Unknown"},
{"many dots", "....", "Unknown"},
{"dotdot padded", " .. ", "Unknown"},
{"trailing dot", "Album.", "Album"},
{"trailing dots", "Album...", "Album"},
{"leading dot kept", ".hidden", ".hidden"},
// Windows device names must not be used as a bare component.
{"reserved CON", "CON", "CON_"},
{"reserved nul lowercase", "nul", "nul_"},
{"reserved COM1", "COM1", "COM1_"},
{"not reserved", "CONCERT", "CONCERT"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := CleanName(tc.in, cfg)
if got != tc.want {
t.Fatalf("CleanName(%q) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
// TestCleanNameNeverEscapesRoot is the property that actually matters: whatever
// a provider returns, joining the result onto a root must stay under that root.
func TestCleanNameNeverEscapesRoot(t *testing.T) {
root := "/music"
hostile := []string{
"..", ".", "../..", "../../etc", "..\\..", "/etc/passwd",
"....//....", " .. ", "", "...", "C:\\Windows",
"\x00evil", "a/../../b",
}
for _, in := range hostile {
for _, restrict := range []bool{false, true} {
cfg := Config{RestrictCharacters: restrict, TruncateTo: 120}
name := CleanName(in, cfg)
if strings.ContainsAny(name, `/\`) {
t.Errorf("CleanName(%q) = %q contains a path separator", in, name)
}
if name == "." || name == ".." {
t.Errorf("CleanName(%q) = %q is a relative component", in, name)
}
joined := filepath.Join(root, name)
rel, err := filepath.Rel(root, joined)
if err != nil {
t.Errorf("CleanName(%q) = %q: Rel failed: %v", in, name, err)
continue
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
t.Errorf("CleanName(%q) = %q escapes root: joined=%q rel=%q", in, name, joined, rel)
}
}
}
}
func TestCleanNameTruncation(t *testing.T) {
// TruncateTo counts runes.
got := CleanName(strings.Repeat("a", 200), Config{TruncateTo: 10})
if len(got) != 10 {
t.Fatalf("rune truncation: got %d chars (%q)", len(got), got)
}
// A long multi-byte title must still fit the 255-byte component limit,
// and must remain valid UTF-8.
long := strings.Repeat("日", 200) // 3 bytes each = 600 bytes
got = CleanName(long, Config{TruncateTo: 200})
if len(got) > maxNameBytes {
t.Fatalf("byte truncation: got %d bytes, want <= %d", len(got), maxNameBytes)
}
if !utf8Valid(got) {
t.Fatalf("truncation split a rune: %q", got)
}
}
func utf8Valid(s string) bool {
for _, r := range s {
if r == '\uFFFD' {
return false
}
}
return true
}
func TestCleanNameRestrictCharacters(t *testing.T) {
got := CleanName("Björk – Jóga", Config{RestrictCharacters: true, TruncateTo: 120})
if strings.ContainsAny(got, "ö–ó") {
t.Fatalf("non-ascii survived restrict_characters: %q", got)
}
}
func TestYearFromDate(t *testing.T) {
tests := []struct{ in, want string }{
{"1977-02-04", "1977"},
{"1977", "1977"},
{"", "Unknown"},
{"77", "Unknown"},
{"abcd-01-01", "Unknown"},
{"19x7-01-01", "Unknown"},
}
for _, tc := range tests {
if got := YearFromDate(tc.in); got != tc.want {
t.Errorf("YearFromDate(%q) = %q, want %q", tc.in, got, tc.want)
}
}
}
+87 -2
View File
@@ -3,18 +3,103 @@ package netutil
import (
"crypto/tls"
"net/http"
"net/url"
"time"
"streamrip-go/internal/verbose"
)
func NewHTTPClient(timeout time.Duration, verifySSL bool) *http.Client {
const defaultMaxConnsPerHost = 16
// NewHTTPClient builds an *http.Client whose transport is tuned for the
// concurrent download workloads this app issues against single CDN hosts.
//
// maxConnsPerHost caps idle keep-alive sockets per host; pass <= 0 to use a
// sensible default. The downloader and provider clients should pass the
// configured concurrency so keep-alive sockets aren't evicted between workers.
func NewHTTPClient(timeout time.Duration, verifySSL bool, maxConnsPerHost int) *http.Client {
if maxConnsPerHost <= 0 {
maxConnsPerHost = defaultMaxConnsPerHost
}
transport := http.DefaultTransport.(*http.Transport).Clone()
if transport.TLSClientConfig == nil {
transport.TLSClientConfig = &tls.Config{}
}
transport.TLSClientConfig.InsecureSkipVerify = !verifySSL
transport.MaxIdleConnsPerHost = maxConnsPerHost
if maxIdle := maxConnsPerHost * 4; maxIdle > transport.MaxIdleConns {
transport.MaxIdleConns = maxIdle
}
if transport.MaxIdleConns < 100 {
transport.MaxIdleConns = 100
}
transport.MaxConnsPerHost = 0
transport.IdleConnTimeout = 90 * time.Second
transport.WriteBufferSize = 64 * 1024
transport.ReadBufferSize = 64 * 1024
transport.ForceAttemptHTTP2 = true
return &http.Client{
Timeout: timeout,
Transport: transport,
Transport: &loggingTransport{base: transport},
}
}
// loggingTransport emits one verbose line per HTTP request when verbose
// level >= VV. The check is per-call so toggling the level at runtime
// affects subsequent requests without rebuilding clients.
type loggingTransport struct {
base http.RoundTripper
}
func (t *loggingTransport) RoundTrip(req *http.Request) (*http.Response, error) {
if !verbose.Enabled(verbose.VV) {
return t.base.RoundTrip(req)
}
start := time.Now()
resp, err := t.base.RoundTrip(req)
elapsed := time.Since(start).Round(time.Millisecond)
target := redactURL(req.URL)
if err != nil {
verbose.Printf(verbose.VV, "http %s %s -> error %v (%s)\n", req.Method, target, err, elapsed)
return resp, err
}
verbose.Printf(verbose.VV, "http %s %s -> %d (%s)\n", req.Method, target, resp.StatusCode, elapsed)
return resp, err
}
// redactURL hides values for query parameters that commonly carry
// credentials so -vv output is safe to paste in an issue.
func redactURL(u *url.URL) string {
if u == nil {
return ""
}
if u.RawQuery == "" {
return u.String()
}
q := u.Query()
redacted := false
for k := range q {
if isSensitiveParam(k) {
q.Set(k, "REDACTED")
redacted = true
}
}
if !redacted {
return u.String()
}
cp := *u
cp.RawQuery = q.Encode()
return cp.String()
}
func isSensitiveParam(name string) bool {
switch name {
case "user_auth_token", "api_token", "access_token", "refresh_token",
"request_sig", "signature", "password", "secret", "token", "code", "auth", "key":
return true
}
return false
}
+148
View File
@@ -0,0 +1,148 @@
package netutil
import (
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
)
func TestRedactURLHidesCredentials(t *testing.T) {
tests := []struct {
name string
raw string
contains []string
absent []string
}{
{
name: "user auth token",
raw: "https://api.example.com/track?user_auth_token=sekrit&track_id=42",
contains: []string{"user_auth_token=REDACTED", "track_id=42"},
absent: []string{"sekrit"},
},
{
name: "request signature",
raw: "https://api.example.com/x?request_sig=abc123&app_id=9",
contains: []string{"request_sig=REDACTED", "app_id=9"},
absent: []string{"abc123"},
},
{
name: "several at once",
raw: "https://api.example.com/x?password=p&access_token=a&refresh_token=r&id=1",
contains: []string{"password=REDACTED", "access_token=REDACTED", "refresh_token=REDACTED", "id=1"},
absent: []string{"password=p", "access_token=a", "refresh_token=r"},
},
{
name: "nothing sensitive is left untouched",
raw: "https://api.example.com/track/42?limit=10",
contains: []string{"limit=10", "/track/42"},
},
{
name: "no query string",
raw: "https://api.example.com/track/42",
contains: []string{"https://api.example.com/track/42"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
u, err := url.Parse(tc.raw)
if err != nil {
t.Fatal(err)
}
got := redactURL(u)
for _, want := range tc.contains {
if !strings.Contains(got, want) {
t.Errorf("redactURL(%q) = %q, missing %q", tc.raw, got, want)
}
}
for _, bad := range tc.absent {
if strings.Contains(got, bad) {
t.Errorf("redactURL(%q) = %q, leaked %q", tc.raw, got, bad)
}
}
})
}
}
func TestRedactURLNil(t *testing.T) {
if got := redactURL(nil); got != "" {
t.Errorf("redactURL(nil) = %q, want empty", got)
}
}
func TestIsSensitiveParam(t *testing.T) {
sensitive := []string{
"user_auth_token", "api_token", "access_token", "refresh_token",
"request_sig", "signature", "password", "secret", "token", "code", "auth", "key",
}
for _, name := range sensitive {
if !isSensitiveParam(name) {
t.Errorf("isSensitiveParam(%q) = false, want true", name)
}
}
for _, name := range []string{"track_id", "limit", "offset", "app_id", "quality"} {
if isSensitiveParam(name) {
t.Errorf("isSensitiveParam(%q) = true, want false", name)
}
}
}
func TestNewHTTPClientHonoursVerifySSL(t *testing.T) {
insecure := NewHTTPClient(5*time.Second, false, 4)
tr, ok := insecure.Transport.(*loggingTransport)
if !ok {
t.Fatalf("transport is %T, want *loggingTransport", insecure.Transport)
}
base, ok := tr.base.(*http.Transport)
if !ok {
t.Fatalf("base transport is %T", tr.base)
}
if !base.TLSClientConfig.InsecureSkipVerify {
t.Error("verifySSL=false should set InsecureSkipVerify")
}
secure := NewHTTPClient(5*time.Second, true, 4)
base = secure.Transport.(*loggingTransport).base.(*http.Transport)
if base.TLSClientConfig.InsecureSkipVerify {
t.Error("verifySSL=true must not skip verification")
}
if secure.Timeout != 5*time.Second {
t.Errorf("Timeout = %v, want 5s", secure.Timeout)
}
}
func TestNewHTTPClientZeroTimeoutMeansNoTimeout(t *testing.T) {
// internal/download and the qobuz body copy rely on this: a zero timeout
// must not be replaced with a default, or long track downloads abort.
c := NewHTTPClient(0, true, 0)
if c.Timeout != 0 {
t.Errorf("Timeout = %v, want 0", c.Timeout)
}
}
func TestNewHTTPClientDefaultsMaxConns(t *testing.T) {
c := NewHTTPClient(time.Second, true, 0)
base := c.Transport.(*loggingTransport).base.(*http.Transport)
if base.MaxIdleConnsPerHost != defaultMaxConnsPerHost {
t.Errorf("MaxIdleConnsPerHost = %d, want %d", base.MaxIdleConnsPerHost, defaultMaxConnsPerHost)
}
}
func TestClientPerformsRequest(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusTeapot)
}))
defer ts.Close()
resp, err := NewHTTPClient(5*time.Second, true, 2).Get(ts.URL + "/x?token=sekrit")
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusTeapot {
t.Errorf("status = %d", resp.StatusCode)
}
}
+847
View File
@@ -0,0 +1,847 @@
package beatport
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"time"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/netutil"
"streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
)
const (
baseURL = "https://api.beatport.com/v4"
tokenEndpoint = "/auth/o/token/"
authEndpoint = "/auth/o/authorize/"
loginEndpoint = "/auth/login/"
)
var (
errMissingCredentials = errors.New("missing beatport credentials")
errNotLoggedIn = errors.New("beatport client not logged in")
)
type Client struct {
cfg *config.Config
http *http.Client
limiter *ratelimit.Limiter
baseURL string
sessionID string
// authMu serializes the token refresh / re-login path and guards loggedIn.
// ensureToken is reached from apiRequest, which concurrent download workers
// call, so without this several workers would refresh at once.
authMu sync.Mutex
loggedIn bool
}
func New(cfg *config.Config) *Client {
h := netutil.NewHTTPClient(40*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections)
h.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }
return &Client{
cfg: cfg,
http: h,
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
baseURL: baseURL,
}
}
func (c *Client) Source() string { return "beatport" }
func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn
}
func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
b := &c.cfg.Session.Beatport
b.Username = strings.TrimSpace(b.Username)
b.Password = strings.TrimSpace(b.Password)
if b.Username == "" || b.Password == "" {
return errMissingCredentials
}
if strings.TrimSpace(b.AccessToken) != "" && time.Now().Unix()+300 < b.TokenExpiry {
c.loggedIn = true
return nil
}
if strings.TrimSpace(b.RefreshToken) != "" {
if err := c.refresh(ctx); err == nil {
c.loggedIn = true
return nil
}
}
if err := c.loginPasswordFlow(ctx); err != nil {
return err
}
c.loggedIn = true
return nil
}
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.LoggedIn() {
return nil, errNotLoggedIn
}
switch mediaType {
case "track":
track, err := c.getMap(ctx, "/catalog/tracks/"+url.PathEscape(strings.TrimSpace(item))+"/", nil)
if err != nil {
return nil, err
}
if relID := jsonutil.NestedString(track, "release", "id"); relID != "" {
if rel, relErr := c.getRelease(ctx, relID); relErr == nil {
track["release"] = rel
}
}
return c.normalizeTrack(track), nil
case "album":
return c.getAlbumMetadata(ctx, item)
case "playlist":
return c.getPlaylistMetadata(ctx, item)
case "chart":
return c.getChartMetadata(ctx, strings.TrimPrefix(strings.TrimSpace(item), "chart:"))
case "artist":
artist, err := c.getMap(ctx, "/catalog/artists/"+url.PathEscape(strings.TrimSpace(item))+"/", nil)
if err != nil {
return nil, err
}
releases, err := c.getPaginated(ctx, "/catalog/releases/", url.Values{"artist_id": []string{strings.TrimSpace(item)}})
if err != nil {
return nil, err
}
return c.releaseCollectionMetadata(artist, releases), nil
case "label":
label, err := c.getMap(ctx, "/catalog/labels/"+url.PathEscape(strings.TrimSpace(item))+"/", nil)
if err != nil {
return nil, err
}
tracks, err := c.getTrackCollection(ctx, url.Values{"label_id": []string{strings.TrimSpace(item)}})
if err != nil {
return nil, err
}
return c.collectionMetadata(label, tracks), nil
default:
return nil, fmt.Errorf("unsupported beatport media type %q", mediaType)
}
}
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.LoggedIn() {
return nil, errNotLoggedIn
}
if limit <= 0 {
limit = 25
}
var key string
switch mediaType {
case "track":
key = "tracks"
case "album":
key = "releases"
case "label":
key = "labels"
default:
return nil, fmt.Errorf("unsupported beatport search media type %q", mediaType)
}
params := url.Values{}
params.Set("q", query)
params.Set("order_by", "-publish_date")
params.Set("is_available_for_streaming", "true")
resp, err := c.getMap(ctx, "/catalog/search/", params)
if err != nil {
return nil, err
}
items := sliceAny(resp[key])
if limit < len(items) {
items = items[:limit]
}
return []map[string]any{{"items": c.normalizeSearchItems(mediaType, items)}}, nil
}
func (c *Client) GetDownloadable(ctx context.Context, item string, quality int) (*provider.Downloadable, error) {
if !c.LoggedIn() {
return nil, errNotLoggedIn
}
q := beatportQuality(quality)
params := url.Values{"quality": []string{q}}
resp, err := c.getMap(ctx, "/catalog/tracks/"+url.PathEscape(strings.TrimSpace(item))+"/download/", params)
if err != nil {
return nil, err
}
location := strings.TrimSpace(jsonutil.StringFromAny(resp["location"]))
if location == "" {
return nil, errors.New("beatport download response missing location")
}
streamQuality := strings.TrimSpace(jsonutil.StringFromAny(resp["stream_quality"]))
profile, ext := audioProfile(q, streamQuality)
return &provider.Downloadable{URL: location, Extension: ext, Source: "beatport", TrackID: strings.TrimSpace(item), Audio: profile}, nil
}
func (c *Client) Close() error { return nil }
func (c *Client) getAlbumMetadata(ctx context.Context, id string) (map[string]any, error) {
release, err := c.getRelease(ctx, id)
if err != nil {
return nil, err
}
tracks, err := c.getReleaseTracks(ctx, id)
if err != nil {
return nil, err
}
return c.normalizeRelease(release, tracks), nil
}
func (c *Client) getPlaylistMetadata(ctx context.Context, id string) (map[string]any, error) {
if strings.HasPrefix(strings.TrimSpace(id), "chart:") {
return c.getChartMetadata(ctx, strings.TrimPrefix(strings.TrimSpace(id), "chart:"))
}
playlist, err := c.getMap(ctx, "/catalog/playlists/"+url.PathEscape(strings.TrimSpace(id))+"/", nil)
if err != nil {
return nil, err
}
items, err := c.getPaginated(ctx, "/catalog/playlists/"+url.PathEscape(strings.TrimSpace(id))+"/tracks/", nil)
if err != nil {
return nil, err
}
tracks := make([]any, 0, len(items))
for _, raw := range items {
m, ok := raw.(map[string]any)
if !ok {
continue
}
if track, ok := m["track"].(map[string]any); ok {
tracks = append(tracks, c.normalizeTrackListItem(track))
}
}
return playlistMetadata(playlist, tracks), nil
}
func (c *Client) getChartMetadata(ctx context.Context, id string) (map[string]any, error) {
chart, err := c.getMap(ctx, "/catalog/charts/"+url.PathEscape(strings.TrimSpace(id))+"/", nil)
if err != nil {
return nil, err
}
tracks, err := c.getPaginated(ctx, "/catalog/charts/"+url.PathEscape(strings.TrimSpace(id))+"/tracks/", nil)
if err != nil {
return nil, err
}
normalized := make([]any, 0, len(tracks))
for _, raw := range tracks {
if track, ok := raw.(map[string]any); ok {
normalized = append(normalized, c.normalizeTrackListItem(track))
}
}
return playlistMetadata(chart, normalized), nil
}
func (c *Client) getRelease(ctx context.Context, id string) (map[string]any, error) {
return c.getMap(ctx, "/catalog/releases/"+url.PathEscape(strings.TrimSpace(id))+"/", nil)
}
func (c *Client) getReleaseTracks(ctx context.Context, id string) ([]any, error) {
items, err := c.getPaginated(ctx, "/catalog/releases/"+url.PathEscape(strings.TrimSpace(id))+"/tracks/", nil)
if err != nil {
return nil, err
}
for _, raw := range items {
if track, ok := raw.(map[string]any); ok {
track["release"] = map[string]any{"id": id}
}
}
return items, nil
}
func (c *Client) getTrackCollection(ctx context.Context, params url.Values) ([]any, error) {
return c.getPaginated(ctx, "/catalog/tracks/", params)
}
func (c *Client) getPaginated(ctx context.Context, endpoint string, params url.Values) ([]any, error) {
if params == nil {
params = url.Values{}
} else {
params = cloneValues(params)
}
out := make([]any, 0)
for page := 1; ; page++ {
params.Set("page", strconv.Itoa(page))
resp, err := c.getMap(ctx, endpoint, params)
if err != nil {
return nil, err
}
out = append(out, sliceAny(resp["results"])...)
if strings.TrimSpace(jsonutil.StringFromAny(resp["next"])) == "" {
break
}
}
return out, nil
}
func (c *Client) getMap(ctx context.Context, endpoint string, params url.Values) (map[string]any, error) {
resp, err := c.apiRequest(ctx, http.MethodGet, endpoint, params, nil, "")
if err != nil {
return nil, err
}
return resp, nil
}
func (c *Client) apiRequest(ctx context.Context, method, endpoint string, params url.Values, payload any, contentType string) (map[string]any, error) {
if err := c.ensureToken(ctx); err != nil {
return nil, err
}
resp, status, err := c.rawRequest(ctx, method, endpoint, params, payload, contentType, true)
if err != nil {
return nil, err
}
if status == http.StatusUnauthorized {
if err = c.reauthorize(ctx); err != nil {
return nil, err
}
resp, status, err = c.rawRequest(ctx, method, endpoint, params, payload, contentType, true)
if err != nil {
return nil, err
}
}
if status < 200 || status >= 300 {
return nil, fmt.Errorf("beatport request failed: status=%d body=%v", status, resp)
}
return resp, nil
}
func (c *Client) ensureToken(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.ensureTokenLocked(ctx)
}
// ensureTokenLocked requires c.authMu to be held.
func (c *Client) ensureTokenLocked(ctx context.Context) error {
if strings.TrimSpace(c.cfg.Session.Beatport.AccessToken) != "" && time.Now().Unix()+300 < c.cfg.Session.Beatport.TokenExpiry {
return nil
}
if strings.TrimSpace(c.cfg.Session.Beatport.RefreshToken) != "" {
if err := c.refresh(ctx); err == nil {
return nil
}
}
return c.loginPasswordFlow(ctx)
}
// reauthorize forces a token refresh after a 401. Holding authMu means that
// when several workers get a 401 at once only the first re-authenticates and
// the rest reuse the resulting token.
func (c *Client) reauthorize(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
c.cfg.Session.Beatport.TokenExpiry = 0
if err := c.refresh(ctx); err != nil {
if loginErr := c.loginPasswordFlow(ctx); loginErr != nil {
return err
}
}
return nil
}
func (c *Client) loginPasswordFlow(ctx context.Context) error {
b := &c.cfg.Session.Beatport
if strings.TrimSpace(b.Username) == "" || strings.TrimSpace(b.Password) == "" {
return errMissingCredentials
}
loginResp, status, err := c.rawRequest(ctx, http.MethodPost, loginEndpoint, nil, map[string]string{"username": b.Username, "password": b.Password}, "application/json", false)
if err != nil {
return err
}
if status < 200 || status >= 300 {
return fmt.Errorf("beatport login failed: status=%d body=%v", status, loginResp)
}
if c.sessionID == "" {
return errors.New("beatport login response missing sessionid cookie")
}
code, err := c.authorize(ctx)
if err != nil {
return err
}
return c.issueToken(ctx, map[string]string{"client_id": c.appClientID(), "grant_type": "authorization_code", "code": code})
}
func (c *Client) authorize(ctx context.Context) (string, error) {
params := url.Values{"client_id": []string{c.appClientID()}, "response_type": []string{"code"}}
resp, status, err := c.rawRequest(ctx, http.MethodGet, authEndpoint, params, nil, "", false)
if err != nil {
return "", err
}
if status != http.StatusFound {
return "", fmt.Errorf("beatport authorize failed: status=%d body=%v", status, resp)
}
code := strings.TrimSpace(jsonutil.StringFromAny(resp["code"]))
if code == "" {
return "", errors.New("beatport authorize redirect missing code")
}
return code, nil
}
func (c *Client) refresh(ctx context.Context) error {
b := &c.cfg.Session.Beatport
return c.issueToken(ctx, map[string]string{"client_id": c.appClientID(), "grant_type": "refresh_token", "refresh_token": b.RefreshToken})
}
func (c *Client) issueToken(ctx context.Context, payload map[string]string) error {
resp, status, err := c.rawRequest(ctx, http.MethodPost, tokenEndpoint, nil, payload, "application/x-www-form-urlencoded", false)
if err != nil {
return err
}
if status < 200 || status >= 300 {
return fmt.Errorf("beatport token request failed: status=%d body=%v", status, resp)
}
access := strings.TrimSpace(jsonutil.StringFromAny(resp["access_token"]))
refresh := strings.TrimSpace(jsonutil.StringFromAny(resp["refresh_token"]))
if access == "" {
return errors.New("beatport token response missing access_token")
}
expiresIn := int64(jsonutil.IntFromAny(resp["expires_in"]))
if expiresIn <= 0 {
expiresIn = 3600
}
expiry := time.Now().Unix() + expiresIn
if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
session.Beatport.AccessToken = access
if refresh != "" {
session.Beatport.RefreshToken = refresh
}
session.Beatport.TokenExpiry = expiry
file.Beatport.AccessToken = session.Beatport.AccessToken
file.Beatport.RefreshToken = session.Beatport.RefreshToken
file.Beatport.TokenExpiry = session.Beatport.TokenExpiry
}); err != nil {
verbose.Printf(verbose.V, "beatport: could not persist refreshed token: %v\n", err)
}
return nil
}
func (c *Client) rawRequest(ctx context.Context, method, endpoint string, params url.Values, payload any, contentType string, auth bool) (map[string]any, int, error) {
if c.limiter != nil {
if err := c.limiter.Wait(ctx); err != nil {
return nil, 0, err
}
}
var body io.Reader
if payload != nil {
switch contentType {
case "application/json":
b, err := json.Marshal(payload)
if err != nil {
return nil, 0, err
}
body = bytes.NewReader(b)
case "application/x-www-form-urlencoded":
vals := url.Values{}
for k, v := range payload.(map[string]string) {
vals.Set(k, v)
}
body = strings.NewReader(vals.Encode())
default:
return nil, 0, fmt.Errorf("unsupported beatport content type %q", contentType)
}
}
u, err := url.Parse(strings.TrimRight(c.baseURL, "/") + endpoint)
if err != nil {
return nil, 0, err
}
if len(params) > 0 {
u.RawQuery = params.Encode()
}
req, err := http.NewRequestWithContext(ctx, method, u.String(), body)
if err != nil {
return nil, 0, err
}
req.Header.Set("Accept", "application/json")
req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36")
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
if auth && strings.TrimSpace(c.cfg.Session.Beatport.AccessToken) != "" {
req.Header.Set("Authorization", "Bearer "+strings.TrimSpace(c.cfg.Session.Beatport.AccessToken))
}
if endpoint == authEndpoint && c.sessionID != "" {
req.Header.Set("Cookie", "sessionid="+c.sessionID)
}
resp, err := c.http.Do(req)
if err != nil {
return nil, 0, err
}
defer func() { _ = resp.Body.Close() }()
if endpoint == loginEndpoint {
for _, cookie := range resp.Cookies() {
if cookie.Name == "sessionid" {
c.sessionID = cookie.Value
break
}
}
}
if endpoint == authEndpoint && resp.StatusCode == http.StatusFound {
loc := resp.Header.Get("Location")
parsed, parseErr := url.Parse(loc)
if parseErr == nil && strings.TrimSpace(parsed.Query().Get("code")) != "" {
return map[string]any{"code": parsed.Query().Get("code")}, resp.StatusCode, nil
}
return map[string]any{}, resp.StatusCode, nil
}
var out map[string]any
if err = json.NewDecoder(resp.Body).Decode(&out); err != nil {
if errors.Is(err, io.EOF) {
return map[string]any{}, resp.StatusCode, nil
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil, resp.StatusCode, err
}
return map[string]any{}, resp.StatusCode, nil
}
return out, resp.StatusCode, nil
}
func (c *Client) normalizeRelease(raw map[string]any, tracks []any) map[string]any {
id := jsonutil.StringFromAny(raw["id"])
artist := c.releaseArtistNameForTracks(raw["artists"], tracks)
trackRelease := cloneMap(raw)
trackRelease["album_artist_name"] = artist
items := make([]any, 0, len(tracks))
for _, entry := range tracks {
track, ok := entry.(map[string]any)
if !ok {
continue
}
track["release"] = trackRelease
items = append(items, c.normalizeTrack(track))
}
return map[string]any{
"id": id,
"title": jsonutil.StringFromAny(raw["name"]),
"artist": map[string]any{"name": artist},
"release_date_original": jsonutil.StringFromAny(raw["new_release_date"]),
"tracks_count": jsonutil.FirstPositiveInt(jsonutil.IntFromAny(raw["track_count"]), len(items)),
"maximum_bit_depth": 16,
"maximum_sampling_rate": "44.1",
"image": imageMap(raw["image"]),
"tracks": map[string]any{"items": items},
"upc": jsonutil.StringFromAny(raw["upc"]),
"label": raw["label"],
}
}
func (c *Client) normalizeTrack(raw map[string]any) map[string]any {
release := mapAny(raw["release"])
trackArtistNames := artistNames(raw["artists"])
artistName := jsonutil.DisplayArtistNames(trackArtistNames)
artistID := firstArtistID(raw["artists"])
albumArtist := strings.TrimSpace(jsonutil.StringFromAny(release["album_artist_name"]))
if albumArtist == "" {
albumArtist = c.releaseArtistName(release["artists"])
}
if albumArtist == "" {
albumArtist = artistName
}
genreName := jsonutil.NestedString(raw, "genre", "name")
if sub := jsonutil.NestedString(raw, "sub_genre", "name"); sub != "" {
genreName = sub
}
date := jsonutil.FirstNonEmpty(jsonutil.StringFromAny(release["new_release_date"]), jsonutil.StringFromAny(raw["publish_date"]))
track := map[string]any{
"id": jsonutil.StringFromAny(raw["id"]),
"title": jsonutil.StringFromAny(raw["name"]),
"version": jsonutil.StringFromAny(raw["mix_name"]),
"artist": map[string]any{"id": artistID, "name": artistName},
"performer": map[string]any{"id": artistID, "name": artistName},
"artist_names": trackArtistNames,
"album": map[string]any{"id": jsonutil.StringFromAny(release["id"]), "title": jsonutil.StringFromAny(release["name"]), "artist": map[string]any{"name": albumArtist}, "image": imageMap(release["image"]), "release_date_original": date},
"track_number": jsonutil.IntFromAny(raw["number"]),
"tracks_count": jsonutil.IntFromAny(release["track_count"]),
"release_date_original": date,
"genre": map[string]any{"name": genreName},
"isrc": jsonutil.StringFromAny(raw["isrc"]),
"bpm": jsonutil.IntFromAny(raw["bpm"]),
"source_track_id": jsonutil.StringFromAny(raw["id"]),
"source_album_id": jsonutil.StringFromAny(release["id"]),
"source_artist_id": artistID,
"maximum_bit_depth": 16,
"maximum_sampling_rate": "44.1",
}
if keyName := jsonutil.NestedString(raw, "key", "name"); keyName != "" {
track["key"] = keyName
}
return track
}
func (c *Client) normalizeTrackListItem(raw map[string]any) map[string]any {
names := artistNames(raw["artists"])
return map[string]any{
"id": jsonutil.StringFromAny(raw["id"]),
"title": jsonutil.StringFromAny(raw["name"]),
"version": jsonutil.StringFromAny(raw["mix_name"]),
"artist": map[string]any{"name": jsonutil.DisplayArtistNames(names)},
"album": map[string]any{"id": jsonutil.NestedString(raw, "release", "id"), "title": jsonutil.NestedString(raw, "release", "name")},
"track_number": jsonutil.IntFromAny(raw["number"]),
}
}
func (c *Client) collectionMetadata(raw map[string]any, tracks []any) map[string]any {
items := make([]any, 0, len(tracks))
for _, entry := range tracks {
track, ok := entry.(map[string]any)
if !ok {
continue
}
items = append(items, c.normalizeTrackListItem(track))
}
return map[string]any{
"id": jsonutil.StringFromAny(raw["id"]),
"name": jsonutil.StringFromAny(raw["name"]),
"title": jsonutil.StringFromAny(raw["name"]),
"tracks_count": len(items),
"tracks": map[string]any{"items": items},
}
}
func (c *Client) releaseCollectionMetadata(raw map[string]any, releases []any) map[string]any {
items := make([]any, 0, len(releases))
for _, entry := range releases {
release, ok := entry.(map[string]any)
if !ok {
continue
}
id := strings.TrimSpace(jsonutil.StringFromAny(release["id"]))
if id == "" {
continue
}
items = append(items, map[string]any{
"id": id,
"title": jsonutil.StringFromAny(release["name"]),
"artist": map[string]any{"name": c.releaseArtistName(release["artists"])},
"tracks_count": jsonutil.IntFromAny(release["track_count"]),
"release_date_original": jsonutil.StringFromAny(release["new_release_date"]),
})
}
return map[string]any{
"id": jsonutil.StringFromAny(raw["id"]),
"name": jsonutil.StringFromAny(raw["name"]),
"title": jsonutil.StringFromAny(raw["name"]),
"albums": map[string]any{"items": items},
}
}
func playlistMetadata(raw map[string]any, tracks []any) map[string]any {
name := jsonutil.StringFromAny(raw["name"])
return map[string]any{
"id": jsonutil.StringFromAny(raw["id"]),
"name": name,
"title": name,
"tracks_count": jsonutil.FirstPositiveInt(jsonutil.IntFromAny(raw["track_count"]), len(tracks)),
"image": imageMap(raw["image"]),
"tracks": map[string]any{"items": tracks},
}
}
func (c *Client) normalizeSearchItems(mediaType string, items []any) []any {
out := make([]any, 0, len(items))
for _, raw := range items {
m, ok := raw.(map[string]any)
if !ok {
continue
}
switch mediaType {
case "track":
out = append(out, c.normalizeTrackListItem(m))
case "album":
out = append(out, map[string]any{"id": jsonutil.StringFromAny(m["id"]), "title": jsonutil.StringFromAny(m["name"]), "artist": map[string]any{"name": c.releaseArtistName(m["artists"])}, "tracks_count": jsonutil.IntFromAny(m["track_count"]), "release_date_original": jsonutil.StringFromAny(m["new_release_date"])})
case "label":
out = append(out, map[string]any{"id": jsonutil.StringFromAny(m["id"]), "title": jsonutil.StringFromAny(m["name"]), "name": jsonutil.StringFromAny(m["name"])})
}
}
return out
}
func beatportQuality(q int) string {
switch q {
case 1:
return "medium"
case 2:
return "high"
default:
return "lossless"
}
}
func audioProfile(quality, streamQuality string) (provider.AudioProfile, string) {
s := strings.ToLower(strings.TrimSpace(streamQuality))
switch {
case strings.Contains(s, "flac") || quality == "lossless":
return provider.AudioProfile{Container: "FLAC", Codec: "FLAC", Quality: "LOSSLESS", BitDepth: 16, SamplingRate: "44.1"}, "flac"
case strings.Contains(s, "256") || quality == "high":
return provider.AudioProfile{Container: "M4A", Codec: "AACLC", Quality: "HIGH", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 256}, "m4a"
default:
return provider.AudioProfile{Container: "M4A", Codec: "AACLC", Quality: "LOW", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 128}, "m4a"
}
}
func artistNames(v any) []string {
items := sliceAny(v)
names := make([]string, 0, len(items))
for _, raw := range items {
m, ok := raw.(map[string]any)
if !ok {
continue
}
if name := strings.TrimSpace(jsonutil.StringFromAny(m["name"])); name != "" {
names = append(names, name)
}
}
return names
}
func (c *Client) releaseArtistName(v any) string {
items := sliceAny(v)
threshold := c.cfg.Session.Beatport.VariousArtistsThreshold
if threshold >= 0 && len(items) > threshold {
return "Various Artists"
}
return jsonutil.DisplayArtistNames(artistNames(v))
}
func (c *Client) releaseArtistNameForTracks(releaseArtists any, tracks []any) string {
common := commonReleaseArtists(releaseArtists, tracks)
if len(common) > 0 {
return jsonutil.DisplayArtistNames(artistNames(common))
}
return c.releaseArtistName(releaseArtists)
}
func commonReleaseArtists(releaseArtists any, tracks []any) []any {
items := sliceAny(releaseArtists)
if len(items) == 0 || len(tracks) == 0 {
return nil
}
counts := map[string]int{}
trackCount := 0
for _, rawTrack := range tracks {
track, ok := rawTrack.(map[string]any)
if !ok {
continue
}
trackCount++
seen := map[string]bool{}
for _, rawArtist := range sliceAny(track["artists"]) {
if key := artistIdentity(rawArtist); key != "" {
seen[key] = true
}
}
for key := range seen {
counts[key]++
}
}
if trackCount == 0 {
return nil
}
common := make([]any, 0)
for _, rawArtist := range items {
if key := artistIdentity(rawArtist); key != "" && counts[key] == trackCount {
common = append(common, rawArtist)
}
}
return common
}
func artistIdentity(raw any) string {
m, ok := raw.(map[string]any)
if !ok {
return ""
}
if id := strings.TrimSpace(jsonutil.StringFromAny(m["id"])); id != "" {
return "id:" + id
}
if name := strings.TrimSpace(jsonutil.StringFromAny(m["name"])); name != "" {
return "name:" + strings.ToLower(name)
}
return ""
}
func cloneMap(in map[string]any) map[string]any {
out := make(map[string]any, len(in))
for k, v := range in {
out[k] = v
}
return out
}
func firstArtistID(v any) string {
items := sliceAny(v)
if len(items) == 0 {
return ""
}
m, _ := items[0].(map[string]any)
return jsonutil.StringFromAny(m["id"])
}
func imageMap(v any) map[string]any {
m := mapAny(v)
dynamic := strings.TrimSpace(jsonutil.StringFromAny(m["dynamic_uri"]))
uri := strings.TrimSpace(jsonutil.StringFromAny(m["uri"]))
if dynamic != "" {
return map[string]any{
"original": strings.ReplaceAll(dynamic, "{w}x{h}", "1400x1400"),
"extralarge": strings.ReplaceAll(dynamic, "{w}x{h}", "1000x1000"),
"large": strings.ReplaceAll(dynamic, "{w}x{h}", "500x500"),
"small": strings.ReplaceAll(dynamic, "{w}x{h}", "250x250"),
"thumbnail": strings.ReplaceAll(dynamic, "{w}x{h}", "100x100"),
}
}
if uri != "" {
return map[string]any{"original": uri, "large": uri}
}
return nil
}
func mapAny(v any) map[string]any {
m, _ := v.(map[string]any)
if m == nil {
return map[string]any{}
}
return m
}
func sliceAny(v any) []any {
s, ok := v.([]any)
if !ok {
return nil
}
return s
}
func cloneValues(in url.Values) url.Values {
out := url.Values{}
for k, vals := range in {
out[k] = append([]string(nil), vals...)
}
return out
}
// appClientID reads the vendor OAuth app credential from config, falling back
// to the built-in default.
func (c *Client) appClientID() string {
if v := strings.TrimSpace(c.cfg.Session.Beatport.AppClientID); v != "" {
return v
}
return config.DefaultBeatportAppClientID
}
+278
View File
@@ -0,0 +1,278 @@
package beatport
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
)
func newTestClient(t *testing.T, handler http.Handler) (*Client, func()) {
t.Helper()
ts := httptest.NewServer(handler)
cfg := &config.Config{Session: testConfigData(), File: testConfigData()}
cfg.Session.Beatport.Username = "user"
cfg.Session.Beatport.Password = "pass"
cfg.Session.Beatport.AccessToken = "token"
cfg.Session.Beatport.TokenExpiry = time.Now().Add(time.Hour).Unix()
c := New(cfg)
c.baseURL = ts.URL
c.http = ts.Client()
c.loggedIn = true
return c, ts.Close
}
func writeJSON(t *testing.T, w http.ResponseWriter, v any) {
t.Helper()
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(v); err != nil {
t.Fatalf("encode json: %v", err)
}
}
func TestGetDownloadableMapsQuality(t *testing.T) {
c, closeServer := newTestClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/catalog/tracks/42/download/" {
t.Fatalf("path = %q", r.URL.Path)
}
if got := r.URL.Query().Get("quality"); got != "high" {
t.Fatalf("quality = %q, want high", got)
}
writeJSON(t, w, map[string]any{"location": "https://cdn.example/42.m4a", "stream_quality": ".256k.aac.mp4"})
}))
defer closeServer()
d, err := c.GetDownloadable(context.Background(), "42", 2)
if err != nil {
t.Fatalf("GetDownloadable() error = %v", err)
}
if d.URL != "https://cdn.example/42.m4a" || d.Extension != "m4a" || d.Audio.BitrateKbps != 256 || d.Audio.Quality != "HIGH" {
t.Fatalf("unexpected downloadable: %+v", d)
}
}
func TestAlbumMetadataNormalizesReleaseTracks(t *testing.T) {
c, closeServer := newTestClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/catalog/releases/7/":
writeJSON(t, w, map[string]any{
"id": 7,
"name": "Release Name",
"new_release_date": "2024-01-02",
"track_count": 1,
"artists": []any{map[string]any{"id": 10, "name": "Album Artist"}},
"image": map[string]any{"dynamic_uri": "https://img.example/{w}x{h}.jpg"},
})
case "/catalog/releases/7/tracks/":
writeJSON(t, w, map[string]any{
"next": nil,
"results": []any{map[string]any{
"id": 42,
"name": "Track Name",
"mix_name": "Original Mix",
"number": 1,
"artists": []any{map[string]any{"id": 11, "name": "Track Artist"}},
"genre": map[string]any{"name": "House"},
"isrc": "USABC1234567",
}},
})
default:
t.Fatalf("unexpected path %q", r.URL.Path)
}
}))
defer closeServer()
meta, err := c.GetMetadata(context.Background(), "7", "album")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
if meta["title"] != "Release Name" || meta["release_date_original"] != "2024-01-02" {
t.Fatalf("unexpected album meta: %+v", meta)
}
tracks := meta["tracks"].(map[string]any)["items"].([]any)
if len(tracks) != 1 {
t.Fatalf("tracks len = %d", len(tracks))
}
track := tracks[0].(map[string]any)
if track["id"] != "42" || track["title"] != "Track Name" || track["version"] != "Original Mix" {
t.Fatalf("unexpected track meta: %+v", track)
}
album := track["album"].(map[string]any)
if album["title"] != "Release Name" {
t.Fatalf("unexpected track album: %+v", album)
}
}
func TestAlbumMetadataCollapsesManyReleaseArtists(t *testing.T) {
c, closeServer := newTestClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/catalog/releases/7/":
writeJSON(t, w, map[string]any{
"id": 7,
"name": "Compilation",
"track_count": 2,
"artists": []any{
map[string]any{"id": 1, "name": "A"},
map[string]any{"id": 2, "name": "B"},
map[string]any{"id": 3, "name": "C"},
map[string]any{"id": 4, "name": "D"},
},
})
case "/catalog/releases/7/tracks/":
writeJSON(t, w, map[string]any{
"next": nil,
"results": []any{
map[string]any{"id": 42, "name": "Track A", "number": 1, "artists": []any{map[string]any{"id": 1, "name": "A"}}},
map[string]any{"id": 43, "name": "Track B", "number": 2, "artists": []any{map[string]any{"id": 2, "name": "B"}}},
},
})
default:
t.Fatalf("unexpected path %q", r.URL.Path)
}
}))
defer closeServer()
meta, err := c.GetMetadata(context.Background(), "7", "album")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
artist := meta["artist"].(map[string]any)
if artist["name"] != "Various Artists" {
t.Fatalf("album artist = %q, want Various Artists", artist["name"])
}
track := meta["tracks"].(map[string]any)["items"].([]any)[0].(map[string]any)
album := track["album"].(map[string]any)
albumArtist := album["artist"].(map[string]any)
if albumArtist["name"] != "Various Artists" {
t.Fatalf("track album artist = %q, want Various Artists", albumArtist["name"])
}
}
func TestAlbumMetadataUsesArtistPresentOnEveryTrack(t *testing.T) {
c, closeServer := newTestClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/catalog/releases/7/":
writeJSON(t, w, map[string]any{
"id": 7,
"name": "Artist Album",
"track_count": 3,
"artists": []any{
map[string]any{"id": 1, "name": "Main"},
map[string]any{"id": 2, "name": "Guest A"},
map[string]any{"id": 3, "name": "Guest B"},
map[string]any{"id": 4, "name": "Guest C"},
},
})
case "/catalog/releases/7/tracks/":
writeJSON(t, w, map[string]any{
"next": nil,
"results": []any{
map[string]any{"id": 42, "name": "Track A", "number": 1, "artists": []any{map[string]any{"id": 1, "name": "Main"}, map[string]any{"id": 2, "name": "Guest A"}}},
map[string]any{"id": 43, "name": "Track B", "number": 2, "artists": []any{map[string]any{"id": 1, "name": "Main"}, map[string]any{"id": 3, "name": "Guest B"}}},
map[string]any{"id": 44, "name": "Track C", "number": 3, "artists": []any{map[string]any{"id": 1, "name": "Main"}, map[string]any{"id": 4, "name": "Guest C"}}},
},
})
default:
t.Fatalf("unexpected path %q", r.URL.Path)
}
}))
defer closeServer()
meta, err := c.GetMetadata(context.Background(), "7", "album")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
artist := meta["artist"].(map[string]any)
if artist["name"] != "Main" {
t.Fatalf("album artist = %q, want Main", artist["name"])
}
track := meta["tracks"].(map[string]any)["items"].([]any)[0].(map[string]any)
album := track["album"].(map[string]any)
albumArtist := album["artist"].(map[string]any)
if albumArtist["name"] != "Main" {
t.Fatalf("track album artist = %q, want Main", albumArtist["name"])
}
}
func TestReleaseArtistCollapseCanBeDisabled(t *testing.T) {
cfg := &config.Config{Session: testConfigData(), File: testConfigData()}
cfg.Session.Beatport.VariousArtistsThreshold = -1
c := New(cfg)
got := c.releaseArtistName([]any{
map[string]any{"name": "A"},
map[string]any{"name": "B"},
map[string]any{"name": "C"},
map[string]any{"name": "D"},
})
if got != "A, B, C & D" {
t.Fatalf("releaseArtistName() = %q", got)
}
}
func TestDisplayArtistNamesUsesCreditPunctuation(t *testing.T) {
got := jsonutil.DisplayArtistNames(artistNames([]any{
map[string]any{"name": "A"},
map[string]any{"name": "B"},
map[string]any{"name": "C"},
}))
if got != "A, B & C" {
t.Fatalf("DisplayArtistNames() = %q", got)
}
}
func TestArtistMetadataUsesReleases(t *testing.T) {
c, closeServer := newTestClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/catalog/artists/908468/":
writeJSON(t, w, map[string]any{"id": 908468, "name": "D'ort"})
case "/catalog/releases/":
if got := r.URL.Query().Get("artist_id"); got != "908468" {
t.Fatalf("artist_id = %q", got)
}
writeJSON(t, w, map[string]any{
"next": nil,
"results": []any{map[string]any{
"id": 123,
"name": "An Artisan's Exhibition",
"new_release_date": "2026-01-01",
"track_count": 10,
"artists": []any{map[string]any{"id": 908468, "name": "D'ort"}},
}},
})
default:
t.Fatalf("unexpected path %q", r.URL.Path)
}
}))
defer closeServer()
meta, err := c.GetMetadata(context.Background(), "908468", "artist")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
if meta["name"] != "D'ort" {
t.Fatalf("artist name = %q", meta["name"])
}
albums := meta["albums"].(map[string]any)["items"].([]any)
if len(albums) != 1 {
t.Fatalf("albums len = %d", len(albums))
}
album := albums[0].(map[string]any)
if album["id"] != "123" || album["title"] != "An Artisan's Exhibition" || album["tracks_count"] != 10 {
t.Fatalf("unexpected album: %+v", album)
}
}
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
File diff suppressed because it is too large. Load diff
+756 -4
View File
@@ -2,12 +2,15 @@ package deezer
import (
"context"
"encoding/hex"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/config"
)
@@ -21,7 +24,7 @@ func TestSearchTrack(t *testing.T) {
}))
defer ts.Close()
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
@@ -38,12 +41,199 @@ func TestSearchTrack(t *testing.T) {
}
}
func TestGetMetadataArtistPaginatesAlbums(t *testing.T) {
callCount := 0
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/artist/9":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 9, "name": "Lost Frequencies"})
case "/artist/9/albums":
callCount++
index := r.URL.Query().Get("index")
limit := r.URL.Query().Get("limit")
if limit != "100" {
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{"error": map[string]any{"message": "bad limit"}})
return
}
switch index {
case "0":
items := make([]any, 0, 100)
for i := 0; i < 100; i++ {
items = append(items, map[string]any{"id": i + 1, "title": "Album"})
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": items, "total": 101})
case "100":
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]any{"id": 101, "title": "Album 101"}}, "total": 101})
default:
w.WriteHeader(http.StatusBadRequest)
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
origBase := baseURL
baseURL = ts.URL
defer func() { baseURL = origBase }()
meta, err := c.GetMetadata(context.Background(), "9", "artist")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
albumsObj, _ := meta["albums"].(map[string]any)
items, _ := albumsObj["items"].([]any)
if len(items) != 101 {
t.Fatalf("albums len = %d, want 101", len(items))
}
if got := strings.TrimSpace(jsonutil.StringFromAny(meta["name"])); got != "Lost Frequencies" {
t.Fatalf("artist name = %q, want Lost Frequencies", got)
}
if callCount != 2 {
t.Fatalf("call count = %d, want 2", callCount)
}
}
func TestGetMetadataAlbumPaginatesTracks(t *testing.T) {
callCount := 0
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/album/46514392":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 46514392, "title": "Clouseau30", "tracks": map[string]any{"data": []any{}}})
case "/album/46514392/tracks":
callCount++
index := r.URL.Query().Get("index")
limit := r.URL.Query().Get("limit")
if limit != "100" {
w.WriteHeader(http.StatusBadRequest)
return
}
switch index {
case "0":
items := make([]any, 0, 100)
for i := 0; i < 100; i++ {
items = append(items, map[string]any{"id": i + 1, "title": "T"})
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": items, "total": 105})
case "100":
items := make([]any, 0, 5)
for i := 0; i < 5; i++ {
items = append(items, map[string]any{"id": 101 + i, "title": "T"})
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": items, "total": 105})
default:
w.WriteHeader(http.StatusBadRequest)
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
origBase := baseURL
baseURL = ts.URL
defer func() { baseURL = origBase }()
meta, err := c.GetMetadata(context.Background(), "46514392", "album")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
tracksObj, _ := meta["tracks"].(map[string]any)
items, _ := tracksObj["items"].([]any)
if len(items) != 105 {
t.Fatalf("tracks len = %d, want 105", len(items))
}
if callCount != 2 {
t.Fatalf("track page call count = %d, want 2", callCount)
}
}
func TestGetMetadataPlaylistPaginatesTracks(t *testing.T) {
callCount := 0
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/playlist/123":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 123, "title": "Mix", "tracks": map[string]any{"data": []any{}}})
case "/playlist/123/tracks":
callCount++
index := r.URL.Query().Get("index")
limit := r.URL.Query().Get("limit")
if limit != "100" {
w.WriteHeader(http.StatusBadRequest)
return
}
switch index {
case "0":
items := make([]any, 0, 100)
for i := 0; i < 100; i++ {
items = append(items, map[string]any{"id": i + 1, "title": "T"})
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": items, "total": 101})
case "100":
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]any{"id": 101, "title": "T"}}, "total": 101})
default:
w.WriteHeader(http.StatusBadRequest)
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
origBase := baseURL
baseURL = ts.URL
defer func() { baseURL = origBase }()
meta, err := c.GetMetadata(context.Background(), "123", "playlist")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
tracksObj, _ := meta["tracks"].(map[string]any)
items, _ := tracksObj["items"].([]any)
if len(items) != 101 {
t.Fatalf("tracks len = %d, want 101", len(items))
}
if callCount != 2 {
t.Fatalf("track page call count = %d, want 2", callCount)
}
}
func TestGetDownloadableNativeCipher(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/track/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "title": "X", "track_token": "tt"})
case "/media":
if got := strings.TrimSpace(r.Header.Get("Accept-Charset")); got != "UTF-8" {
t.Fatalf("accept-charset = %q, want UTF-8", got)
}
var payload map[string]any
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
media, _ := payload["media"].([]any)
if len(media) != 1 {
t.Fatalf("media length = %d, want 1", len(media))
}
entry, _ := media[0].(map[string]any)
formats, _ := entry["formats"].([]any)
if len(formats) != 6 {
t.Fatalf("formats length = %d, want 6", len(formats))
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]any{"errors": []any{}, "media": []any{map[string]any{"cipher": map[string]any{"type": "BF_CBC_STRIPE"}, "format": "FLAC", "sources": []any{map[string]any{"url": "https://cdn.example/file"}}}}}}})
default:
w.WriteHeader(http.StatusNotFound)
@@ -51,20 +241,24 @@ func TestGetDownloadableNativeCipher(t *testing.T) {
}))
defer ts.Close()
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.arl = "arl"
c.license = "license"
c.jwt = "jwt"
origBase := baseURL
origMedia := mediaURL
origPipe := pipeURL
baseURL = ts.URL
mediaURL = ts.URL + "/media"
pipeURL = ts.URL + "/pipe"
defer func() {
baseURL = origBase
mediaURL = origMedia
pipeURL = origPipe
}()
d, err := c.GetDownloadable(context.Background(), "42", 2)
@@ -74,10 +268,125 @@ func TestGetDownloadableNativeCipher(t *testing.T) {
if d.Cipher != "BF_CBC_STRIPE" || d.Extension != "flac" || d.TrackID != "42" {
t.Fatalf("unexpected downloadable: %+v", d)
}
if d.Audio.Container != "FLAC" || d.Audio.Quality != "LOSSLESS" {
t.Fatalf("unexpected audio profile: %+v", d.Audio)
}
}
func TestGetDownloadablePrefersNoneCipherWhenAvailable(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/track/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "title": "X", "track_token": "tt"})
case "/media":
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]any{"errors": []any{}, "media": []any{
map[string]any{"cipher": map[string]any{"type": "BF_CBC_STRIPE"}, "format": "FLAC", "sources": []any{map[string]any{"url": "https://cdn.example/bf"}}},
map[string]any{"cipher": map[string]any{"type": "NONE"}, "format": "FLAC", "sources": []any{map[string]any{"url": "https://cdn.example/plain"}}},
}}}})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.arl = "arl"
c.license = "license"
c.jwt = "jwt"
origBase := baseURL
origMedia := mediaURL
origPipe := pipeURL
baseURL = ts.URL
mediaURL = ts.URL + "/media"
pipeURL = ts.URL + "/pipe"
defer func() {
baseURL = origBase
mediaURL = origMedia
pipeURL = origPipe
}()
d, err := c.GetDownloadable(context.Background(), "42", 2)
if err != nil {
t.Fatalf("GetDownloadable() error = %v", err)
}
if d.Cipher != "NONE" || d.URL != "https://cdn.example/plain" {
t.Fatalf("expected NONE cipher source, got %+v", d)
}
}
func TestExtractTrackIDFromMediaURL(t *testing.T) {
url := "https://f-cdnt-stream.dzcdn.net/media/1/9/6/4/8/2552667002/64821d6a2007e90768fa0300b508fcf4.flac?hdnea=x"
if got := extractTrackIDFromMediaURL(url); got != "2552667002" {
t.Fatalf("extractTrackIDFromMediaURL() = %q, want 2552667002", got)
}
}
func TestLoginPrefersARLFlowOverRefreshShortcut(t *testing.T) {
mobileToken := testMobileToken(t)
refreshCalled := false
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/web":
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"USER_ID": "42", "JWT": "jwt-from-web", "refresh_token": "refresh-from-web", "license_token": "license-from-web"}})
case "/gateway":
switch r.URL.Query().Get("method") {
case "mobile_auth":
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"TOKEN": mobileToken}})
case "api_checkToken":
_ = json.NewEncoder(w).Encode(map[string]any{"results": "sid123"})
case "mobile_userAutolog":
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"JWT": "jwt-from-autolog", "license_token": "license-from-autolog", "refresh_token": "refresh-from-autolog"}})
default:
w.WriteHeader(http.StatusNotFound)
}
case "/pipe":
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"tokens": map[string]any{"mediaServiceLicenseToken": map[string]any{"token": "license-from-pipe"}}}})
case "/renew":
refreshCalled = true
w.WriteHeader(http.StatusInternalServerError)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Deezer.ARL = "arl"
cfgData.Deezer.RefreshToken = "refresh-token"
c := New(&config.Config{File: cfgData, Session: cfgData})
origGateway := gatewayURL
origWeb := webGWLight
origPipe := pipeURL
origAuth := authURL
gatewayURL = ts.URL + "/gateway"
webGWLight = ts.URL + "/web"
pipeURL = ts.URL + "/pipe"
authURL = ts.URL + "/renew"
defer func() {
gatewayURL = origGateway
webGWLight = origWeb
pipeURL = origPipe
authURL = origAuth
}()
if err := c.Login(context.Background()); err != nil {
t.Fatalf("Login() error = %v", err)
}
if refreshCalled {
t.Fatalf("expected ARL launch flow without refresh shortcut")
}
if c.license != "license-from-pipe" {
t.Fatalf("license = %q, want license-from-pipe", c.license)
}
}
func TestGetDownloadableRequiresARL(t *testing.T) {
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
cfgData.Deezer.ARL = ""
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
@@ -100,20 +409,24 @@ func TestGetDownloadableDRMError(t *testing.T) {
}))
defer ts.Close()
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.arl = "arl"
c.license = "license"
c.jwt = "jwt"
origBase := baseURL
origMedia := mediaURL
origPipe := pipeURL
baseURL = ts.URL
mediaURL = ts.URL + "/media"
pipeURL = ts.URL + "/pipe"
defer func() {
baseURL = origBase
mediaURL = origMedia
pipeURL = origPipe
}()
_, err := c.GetDownloadable(context.Background(), "42", 2)
@@ -121,3 +434,442 @@ func TestGetDownloadableDRMError(t *testing.T) {
t.Fatalf("expected drm error, got %v", err)
}
}
func TestGetTrackTokenPrefersPipeToken(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/pipe":
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"track": map[string]any{"media": map[string]any{"token": map[string]any{"payload": "pipe-track-token"}}}}})
case "/track/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "track_token": "api-track-token"})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.jwt = "jwt-token"
origBase := baseURL
origPipe := pipeURL
baseURL = ts.URL
pipeURL = ts.URL + "/pipe"
defer func() {
baseURL = origBase
pipeURL = origPipe
}()
token, mediaID, err := c.getTrackToken(context.Background(), "42")
if err != nil {
t.Fatalf("getTrackToken() error = %v", err)
}
if token != "pipe-track-token" {
t.Fatalf("token = %q, want pipe-track-token", token)
}
if mediaID != "" {
t.Fatalf("mediaID = %q, want empty when pipe media id missing", mediaID)
}
}
func TestGetDownloadableUsesPipeTrackToken(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/track/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "title": "X", "track_token": "api-track-token"})
case "/pipe":
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"track": map[string]any{"media": map[string]any{"token": map[string]any{"payload": "pipe-track-token"}}}}})
case "/media":
var payload map[string]any
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
tokens, _ := payload["track_tokens"].([]any)
if len(tokens) == 0 || strings.TrimSpace(jsonutil.StringFromAny(tokens[0])) != "pipe-track-token" {
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]any{"errors": []any{map[string]any{"code": 2004, "message": "The track country differs from the license."}}, "media": []any{}}}})
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]any{"errors": []any{}, "media": []any{map[string]any{"cipher": map[string]any{"type": "BF_CBC_STRIPE"}, "format": "FLAC", "sources": []any{map[string]any{"url": "https://cdn.example/file"}}}}}}})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Deezer.ARL = "arl"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.arl = "arl"
c.license = "license"
c.jwt = "jwt"
origBase := baseURL
origMedia := mediaURL
origPipe := pipeURL
baseURL = ts.URL
mediaURL = ts.URL + "/media"
pipeURL = ts.URL + "/pipe"
defer func() {
baseURL = origBase
mediaURL = origMedia
pipeURL = origPipe
}()
d, err := c.GetDownloadable(context.Background(), "42", 2)
if err != nil {
t.Fatalf("GetDownloadable() error = %v", err)
}
if d.URL != "https://cdn.example/file" || d.Extension != "flac" {
t.Fatalf("unexpected downloadable: %+v", d)
}
}
func TestGetMetadataAddsLyricsFromPipe(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/track/1141668":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 1141668, "title": "In Da Club", "artist": map[string]any{"name": "50 Cent"}})
case "/pipe":
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"track": map[string]any{"lyrics": map[string]any{"text": "Go, go, go\nGo shawty", "synchronizedLines": []any{map[string]any{"line": "Go, go, go", "milliseconds": 0}, map[string]any{"line": "Go shawty", "milliseconds": 4280}}}}}})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.jwt = "jwt"
origBase := baseURL
origPipe := pipeURL
baseURL = ts.URL
pipeURL = ts.URL + "/pipe"
defer func() {
baseURL = origBase
pipeURL = origPipe
}()
meta, err := c.GetMetadata(context.Background(), "1141668", "track")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
if !strings.Contains(jsonutil.StringFromAny(meta["lyrics"]), "Go shawty") {
t.Fatalf("expected lyrics text, got %q", jsonutil.StringFromAny(meta["lyrics"]))
}
if !strings.Contains(jsonutil.StringFromAny(meta["lyrics_synced"]), "[00:00.00]Go, go, go") {
t.Fatalf("expected synced lyrics, got %q", jsonutil.StringFromAny(meta["lyrics_synced"]))
}
}
func TestLoginWithCredentials(t *testing.T) {
mobileToken := testMobileToken(t)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/gateway" {
w.WriteHeader(http.StatusNotFound)
return
}
switch r.URL.Query().Get("method") {
case "mobile_auth":
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"TOKEN": mobileToken}})
case "api_checkToken":
_ = json.NewEncoder(w).Encode(map[string]any{"results": "sid123"})
case "mobile_userAuth":
var payload map[string]any
_ = json.NewDecoder(r.Body).Decode(&payload)
if strings.TrimSpace(jsonutil.StringFromAny(payload["mail"])) == "" || strings.TrimSpace(jsonutil.StringFromAny(payload["password"])) == "" {
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{"error": map[string]any{"message": "missing creds"}})
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"ARL": "arl-token", "JWT": "jwt-token", "refresh_token": "refresh-token", "license_token": "license-token", "USER_ID": "42"}})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Deezer.Email = "tidal1@alpin.sbs"
cfgData.Deezer.Password = "tidal1@alpin.sbs"
c := New(&config.Config{File: cfgData, Session: cfgData})
origGateway := gatewayURL
gatewayURL = ts.URL + "/gateway"
defer func() { gatewayURL = origGateway }()
if err := c.Login(context.Background()); err != nil {
t.Fatalf("Login() error = %v", err)
}
if !c.loggedIn {
t.Fatalf("expected logged in client")
}
if c.arl != "arl-token" {
t.Fatalf("arl = %q, want arl-token", c.arl)
}
if c.jwt != "jwt-token" {
t.Fatalf("jwt = %q, want jwt-token", c.jwt)
}
if c.refresh != "refresh-token" {
t.Fatalf("refresh = %q, want refresh-token", c.refresh)
}
if c.license != "license-token" {
t.Fatalf("license = %q, want license-token", c.license)
}
if c.cfg.Session.Deezer.RefreshToken != "refresh-token" {
t.Fatalf("session refresh token = %q", c.cfg.Session.Deezer.RefreshToken)
}
if c.cfg.File.Deezer.RefreshToken != "refresh-token" {
t.Fatalf("file refresh token = %q", c.cfg.File.Deezer.RefreshToken)
}
}
func TestMobileAuthIncludesAppContextParams(t *testing.T) {
mobileToken := testMobileToken(t)
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/gateway" {
w.WriteHeader(http.StatusNotFound)
return
}
if r.URL.Query().Get("method") != "mobile_auth" {
w.WriteHeader(http.StatusBadRequest)
return
}
if got := strings.TrimSpace(r.URL.Query().Get("version")); got != deezerAppVersion {
t.Fatalf("version = %q, want %q", got, deezerAppVersion)
}
if got := strings.TrimSpace(r.URL.Query().Get("lang")); got != deezerAppLang {
t.Fatalf("lang = %q, want %q", got, deezerAppLang)
}
if got := strings.TrimSpace(r.URL.Query().Get("buildId")); got != deezerBuildID {
t.Fatalf("buildId = %q, want %q", got, deezerBuildID)
}
if got := strings.TrimSpace(r.URL.Query().Get("screenWidth")); got != deezerScreenW {
t.Fatalf("screenWidth = %q, want %q", got, deezerScreenW)
}
if got := strings.TrimSpace(r.URL.Query().Get("screenHeight")); got != deezerScreenH {
t.Fatalf("screenHeight = %q, want %q", got, deezerScreenH)
}
if got := strings.TrimSpace(r.URL.Query().Get("uniq_id")); got == "" {
t.Fatalf("uniq_id is empty")
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"TOKEN": mobileToken}})
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
origGateway := gatewayURL
gatewayURL = ts.URL + "/gateway"
defer func() { gatewayURL = origGateway }()
token, err := c.mobileAuth(context.Background())
if err != nil {
t.Fatalf("mobileAuth() error = %v", err)
}
if token != mobileToken {
t.Fatalf("token = %q, want %q", token, mobileToken)
}
}
func TestMobileUserAutologIncludesRefreshToken(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/gateway" {
w.WriteHeader(http.StatusNotFound)
return
}
if method := r.URL.Query().Get("method"); method != "mobile_userAutolog" {
w.WriteHeader(http.StatusBadRequest)
return
}
if got := strings.TrimSpace(r.URL.Query().Get("arl")); got != "" {
t.Fatalf("unexpected arl query parameter: %q", got)
}
var payload map[string]any
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
if got := strings.TrimSpace(jsonutil.StringFromAny(payload["refresh_token"])); got != "refresh-token" {
t.Fatalf("refresh_token payload = %q, want refresh-token", got)
}
_ = json.NewEncoder(w).Encode(map[string]any{"results": map[string]any{"JWT": "jwt-token", "license_token": "license-token"}})
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.sid = "sid123"
c.userID = "42"
c.arl = "arl-token"
c.refresh = "refresh-token"
origGateway := gatewayURL
gatewayURL = ts.URL + "/gateway"
defer func() { gatewayURL = origGateway }()
if err := c.mobileUserAutolog(context.Background()); err != nil {
t.Fatalf("mobileUserAutolog() error = %v", err)
}
if c.jwt != "jwt-token" {
t.Fatalf("jwt = %q, want jwt-token", c.jwt)
}
if c.license != "license-token" {
t.Fatalf("license = %q, want license-token", c.license)
}
}
func testMobileToken(t *testing.T) string {
t.Helper()
plain := []byte(strings.Repeat("A", 64) + strings.Repeat("B", 16) + strings.Repeat("C", 16))
enc, err := aesECBEncrypt([]byte(config.DefaultDeezerGatewayKey), plain)
if err != nil {
t.Fatalf("aesECBEncrypt() error = %v", err)
}
return hex.EncodeToString(enc)
}
func TestLoginWithRefreshToken(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/renew":
_ = json.NewEncoder(w).Encode(map[string]any{"jwt": "jwt-token", "refresh_token": "refresh-token-2"})
case "/pipe":
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"tokens": map[string]any{"mediaServiceLicenseToken": map[string]any{"token": "license-token"}}}})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Deezer.RefreshToken = "refresh-token"
c := New(&config.Config{File: cfgData, Session: cfgData})
origAuth := authURL
origPipe := pipeURL
authURL = ts.URL + "/renew"
pipeURL = ts.URL + "/pipe"
defer func() {
authURL = origAuth
pipeURL = origPipe
}()
if err := c.Login(context.Background()); err != nil {
t.Fatalf("Login() error = %v", err)
}
if !c.loggedIn {
t.Fatalf("expected logged in client")
}
if c.jwt != "jwt-token" || c.license != "license-token" {
t.Fatalf("unexpected jwt/license: jwt=%q license=%q", c.jwt, c.license)
}
if c.cfg.Session.Deezer.RefreshToken != "refresh-token-2" {
t.Fatalf("session refresh token = %q", c.cfg.Session.Deezer.RefreshToken)
}
}
func TestRefreshJWTHTTPError(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
_ = json.NewEncoder(w).Encode(map[string]any{"error": map[string]any{"message": "bad refresh"}})
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.refresh = "refresh-token"
origAuth := authURL
authURL = ts.URL
defer func() { authURL = origAuth }()
err := c.refreshJWT(context.Background())
if err == nil || !strings.Contains(strings.ToLower(err.Error()), "status=401") {
t.Fatalf("expected http status error, got %v", err)
}
}
func TestRefreshLicenseFromPipeGraphQLError(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"errors": []any{map[string]any{"message": "token expired"}}})
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.jwt = "jwt-token"
origPipe := pipeURL
pipeURL = ts.URL
defer func() { pipeURL = origPipe }()
err := c.refreshLicenseFromPipe(context.Background())
if err == nil || !strings.Contains(strings.ToLower(err.Error()), "token expired") {
t.Fatalf("expected graphql error, got %v", err)
}
}
func TestRefreshLicenseFromPipeUsesMobileGraphQLShape(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := strings.TrimSpace(r.Header.Get("Authorization")); got != "Bearer jwt-token" {
t.Fatalf("authorization = %q, want Bearer jwt-token", got)
}
if got := r.Header.Get("Accept"); !strings.Contains(got, "application/graphql-response+json") {
t.Fatalf("accept header = %q", got)
}
if got := strings.TrimSpace(r.Header.Get("Accept-Language")); got != "en-US" {
t.Fatalf("accept-language = %q, want en-US", got)
}
var payload map[string]any
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
op := strings.TrimSpace(jsonutil.StringFromAny(payload["operationName"]))
if op != "KmpMpMediaServiceLicenseToken" {
t.Fatalf("operationName = %q, want KmpMpMediaServiceLicenseToken", op)
}
ext, _ := payload["extensions"].(map[string]any)
clientLib, _ := ext["clientLibrary"].(map[string]any)
if got := strings.TrimSpace(jsonutil.StringFromAny(clientLib["name"])); got != "apollo-kotlin" {
t.Fatalf("clientLibrary.name = %q, want apollo-kotlin", got)
}
if got := strings.TrimSpace(jsonutil.StringFromAny(clientLib["version"])); got != "4.4.2" {
t.Fatalf("clientLibrary.version = %q, want 4.4.2", got)
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"tokens": map[string]any{"mediaServiceLicenseToken": map[string]any{"token": "license-token"}}}})
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.jwt = "jwt-token"
origPipe := pipeURL
pipeURL = ts.URL
defer func() { pipeURL = origPipe }()
if err := c.refreshLicenseFromPipe(context.Background()); err != nil {
t.Fatalf("refreshLicenseFromPipe() error = %v", err)
}
if c.license != "license-token" {
t.Fatalf("license = %q, want license-token", c.license)
}
}
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+11
View File
@@ -7,7 +7,18 @@ type Downloadable struct {
Extension string
Source string
Cipher string
Key string
TrackID string
Audio AudioProfile
}
type AudioProfile struct {
Container string
Codec string
Quality string
BitDepth int
SamplingRate string
BitrateKbps int
}
type Client interface {
File diff suppressed because it is too large. Load diff
+460 -1
View File
@@ -2,6 +2,8 @@ package qobuz
import (
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"net/http"
"net/http/httptest"
@@ -157,9 +159,56 @@ func TestGetLabelPagination(t *testing.T) {
}
}
func TestGetArtistPagination(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
offset := r.URL.Query().Get("offset")
if offset == "" {
offset = "0"
}
resp := map[string]any{}
switch offset {
case "0":
resp = map[string]any{
"albums_count": 620,
"albums": map[string]any{"items": makeItems(0, 500)},
}
case "500":
resp = map[string]any{"albums": map[string]any{"items": makeItems(500, 620)}}
default:
w.WriteHeader(http.StatusNotFound)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "not found"})
return
}
_ = json.NewEncoder(w).Encode(resp)
}))
defer ts.Close()
c := newTestClient(t)
c.loggedIn = true
c.baseURL = ts.URL
raw, err := c.GetMetadata(context.Background(), "artist-id", "artist")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
albums, ok := mapValue(raw["albums"])
if !ok {
t.Fatalf("albums missing")
}
items, ok := albums["items"].([]any)
if !ok {
t.Fatalf("items missing")
}
if len(items) != 620 {
t.Fatalf("len(items) = %d, want 620", len(items))
}
}
func newTestClient(t *testing.T) *Client {
t.Helper()
d := config.DefaultConfigData()
d := testConfigData()
d.Qobuz.AppID = "12345"
cfg := &config.Config{File: d, Session: d}
return New(cfg)
@@ -172,3 +221,413 @@ func makeItems(start, end int) []map[string]any {
}
return items
}
func TestLoginRefreshesAppCredentialsWhenSecretInvalid(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/user/login":
_ = json.NewEncoder(w).Encode(map[string]any{"user_auth_token": "uat-token"})
case "/track/getFileUrl":
if r.Header.Get("X-App-Id") != "new-app" {
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "bad app"})
return
}
tsValue := r.URL.Query().Get("request_ts")
sig := r.URL.Query().Get("request_sig")
if sig != qobuzSecretSig(tsValue, "good-secret") {
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "bad secret"})
return
}
w.WriteHeader(http.StatusUnauthorized)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "ok secret"})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
d := testConfigData()
d.Qobuz.EmailOrUserID = "user@example.com"
d.Qobuz.PasswordOrToken = "hash"
d.Qobuz.AppID = "old-app"
d.Qobuz.Secrets = []string{"bad-secret"}
cfg := &config.Config{File: d, Session: d}
c := New(cfg)
c.baseURL = ts.URL
c.fetchCfg = func(context.Context) (string, []string, error) {
return "new-app", []string{"good-secret"}, nil
}
if err := c.Login(context.Background()); err != nil {
t.Fatalf("Login() error = %v", err)
}
if !c.loggedIn {
t.Fatalf("expected logged-in client")
}
if c.secret != "good-secret" {
t.Fatalf("secret = %q, want good-secret", c.secret)
}
if c.cfg.Session.Qobuz.AppID != "new-app" {
t.Fatalf("session app id = %q", c.cfg.Session.Qobuz.AppID)
}
}
func TestLoginRetriesAfterRefreshingAppCredentials(t *testing.T) {
loginCalls := 0
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/user/login":
loginCalls++
if r.URL.Query().Get("app_id") != "new-app" {
w.WriteHeader(http.StatusUnauthorized)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "expired app id"})
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"user_auth_token": "uat-token"})
case "/track/getFileUrl":
tsValue := r.URL.Query().Get("request_ts")
sig := r.URL.Query().Get("request_sig")
if r.Header.Get("X-App-Id") == "new-app" && sig == qobuzSecretSig(tsValue, "good-secret") {
w.WriteHeader(http.StatusUnauthorized)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "ok secret"})
return
}
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{"message": "bad secret"})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
d := testConfigData()
d.Qobuz.EmailOrUserID = "user@example.com"
d.Qobuz.PasswordOrToken = "hash"
d.Qobuz.AppID = "old-app"
d.Qobuz.Secrets = []string{"old-secret"}
cfg := &config.Config{File: d, Session: d}
c := New(cfg)
c.baseURL = ts.URL
c.fetchCfg = func(context.Context) (string, []string, error) {
return "new-app", []string{"good-secret"}, nil
}
if err := c.Login(context.Background()); err != nil {
t.Fatalf("Login() error = %v", err)
}
if loginCalls < 2 {
t.Fatalf("expected login retry after refresh, calls=%d", loginCalls)
}
}
func TestQobuzDownloadExtension(t *testing.T) {
tests := []struct {
name string
resp map[string]any
quality int
url string
want string
}{
{name: "from url flac", resp: map[string]any{}, quality: 1, url: "https://cdn.example/a.flac?token=1", want: "flac"},
{name: "from url mp3", resp: map[string]any{}, quality: 4, url: "https://cdn.example/a.mp3", want: "mp3"},
{name: "from mime type", resp: map[string]any{"mime_type": "audio/flac"}, quality: 1, url: "https://cdn.example/stream", want: "flac"},
{name: "from format id", resp: map[string]any{"format_id": float64(5)}, quality: 4, url: "https://cdn.example/stream", want: "mp3"},
{name: "fallback quality", resp: map[string]any{}, quality: 3, url: "https://cdn.example/stream", want: "flac"},
}
for _, tt := range tests {
if got := qobuzDownloadExtension(tt.resp, tt.quality, tt.url); got != tt.want {
t.Fatalf("%s: qobuzDownloadExtension()=%q want %q", tt.name, got, tt.want)
}
}
}
func TestGetDownloadableUsesReturnedURLExtension(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/track/getFileUrl" {
w.WriteHeader(http.StatusNotFound)
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"url": "https://cdn.example/track.mp3?token=abc"})
}))
defer ts.Close()
c := newTestClient(t)
c.loggedIn = true
c.secret = "secret"
c.baseURL = ts.URL
d, err := c.GetDownloadable(context.Background(), "19512574", 4)
if err != nil {
t.Fatalf("GetDownloadable() error = %v", err)
}
if d.Extension != "mp3" {
t.Fatalf("extension = %q, want mp3", d.Extension)
}
if d.Audio.Container != "MP3" || d.Audio.Codec != "MP3" {
t.Fatalf("unexpected audio profile: %+v", d.Audio)
}
}
func TestQobuzAudioProfileUsesExactReturnedQuality(t *testing.T) {
profile := qobuzAudioProfile(map[string]any{
"format_id": float64(7),
"bit_depth": float64(24),
"sampling_rate": float64(44.1),
}, 4, "flac")
if profile.BitDepth != 24 || profile.SamplingRate != "44.1" || profile.Quality != "HI_RES" {
t.Fatalf("unexpected profile: %+v", profile)
}
}
func TestQobuzAudioProfileAvoidsGuessingHiResSampleRate(t *testing.T) {
profile := qobuzAudioProfile(map[string]any{"format_id": float64(7)}, 4, "flac")
if profile.BitDepth != 24 || profile.SamplingRate != "" || profile.Quality != "HI_RES" {
t.Fatalf("unexpected profile: %+v", profile)
}
}
func qobuzSecretSig(requestTS, secret string) string {
raw := "trackgetFileUrlformat_id27intentstreamtrack_id19512574" + requestTS + secret
hash := md5.Sum([]byte(raw))
return hex.EncodeToString(hash[:])
}
func TestRefreshAppCredentialsRejectsEmptyData(t *testing.T) {
d := testConfigData()
c := New(&config.Config{File: d, Session: d})
c.fetchCfg = func(context.Context) (string, []string, error) {
return "", []string{" "}, nil
}
err := c.refreshAppCredentials(context.Background(), &c.cfg.Session.Qobuz)
if err == nil {
t.Fatalf("expected error for empty refreshed app credentials")
}
}
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
// newLyricsServer serves the two-step Qobuz lyrics flow: track/lyricsUrl
// returns a URL on this same server, which then serves the document.
func newLyricsServer(t *testing.T, doc any, docStatus int) *httptest.Server {
t.Helper()
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/track/lyricsUrl":
if r.URL.Query().Get("request_sig") == "" {
t.Error("track/lyricsUrl called without a request signature")
}
if r.URL.Query().Get("language") != "" {
t.Error("language must be omitted so the original lyrics are returned, not a translation")
}
_ = json.NewEncoder(w).Encode(map[string]any{
"track_id": r.URL.Query().Get("track_id"),
"album_id": "a1",
"lyrics_url": srv.URL + "/doc.json?Signature=x",
})
case "/doc.json":
if docStatus != http.StatusOK {
w.WriteHeader(docStatus)
_, _ = w.Write([]byte(`<?xml version="1.0"?><Error><Code>AccessDenied</Code></Error>`))
return
}
_ = json.NewEncoder(w).Encode(doc)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
t.Cleanup(srv.Close)
return srv
}
func lyricsClient(t *testing.T, srv *httptest.Server, exclude ...string) *Client {
t.Helper()
d := testConfigData()
d.Metadata.Exclude = exclude
c := New(&config.Config{File: d, Session: d})
c.baseURL = srv.URL
c.loggedIn = true
c.secret = "secret"
return c
}
func TestTrackLyricsLineSynced(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"track_id": "42",
"translation_langs": []any{"fr", "de"},
"original": map[string]any{
"type": "lsync",
"lang": "en",
"lines": []any{
map[string]any{"line": "First line", "start": 5780, "end": 8710},
map[string]any{"line": "", "start": 9000, "end": 9100},
map[string]any{"line": "Second line", "start": 65250, "end": 68000},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatalf("TrackLyrics() error = %v", err)
}
// The timestamped blank is a display-clear marker: it belongs in the LRC
// but not in the readable plain text.
if plain != "First line\nSecond line" {
t.Errorf("plain = %q", plain)
}
want := "[00:05.78]First line\n[00:09.00]\n[01:05.25]Second line"
if lrc != want {
t.Errorf("lrc = %q, want %q", lrc, want)
}
}
func TestTrackLyricsPlainHasNoTimestamps(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "plain",
"lines": []any{
map[string]any{"line": "Just text"},
map[string]any{"line": "More text"},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
if plain != "Just text\nMore text" {
t.Errorf("plain = %q", plain)
}
if lrc != "" {
t.Errorf("plain lyrics must not produce an LRC transcript, got %q", lrc)
}
}
func TestTrackLyricsWordSyncedUsesLineTimings(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "wsync",
"lines": []any{
map[string]any{
"line": "Hello world",
"start": 1500,
"words": []any{
map[string]any{"text": "Hello", "start": 1500, "end": 1800},
map[string]any{"text": "world", "start": 1800, "end": 2100},
},
},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
if plain != "Hello world" {
t.Errorf("plain = %q", plain)
}
if lrc != "[00:01.50]Hello world" {
t.Errorf("lrc = %q", lrc)
}
}
// A missing document 403s on the signed URL; that is how Qobuz says "no
// lyrics", so it must not fail the rip.
func TestTrackLyricsMissingDocumentIsNotAnError(t *testing.T) {
srv := newLyricsServer(t, nil, http.StatusForbidden)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatalf("a missing lyrics document must not be an error, got %v", err)
}
if plain != "" || lrc != "" {
t.Errorf("expected no lyrics, got plain=%q lrc=%q", plain, lrc)
}
}
func TestTrackLyricsRespectsMetadataExclude(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Errorf("no request should be made when lyrics are excluded, got %s", r.URL.Path)
}))
defer srv.Close()
plain, lrc, err := lyricsClient(t, srv, "lyrics").TrackLyrics(context.Background(), "42")
if err != nil || plain != "" || lrc != "" {
t.Fatalf("exclude should short-circuit; got plain=%q lrc=%q err=%v", plain, lrc, err)
}
}
func TestTrackLyricsEmptyIDIsNoOp(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Errorf("no request expected for an empty track id, got %s", r.URL.Path)
}))
defer srv.Close()
if plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), " "); err != nil || plain != "" || lrc != "" {
t.Fatalf("got plain=%q lrc=%q err=%v", plain, lrc, err)
}
}
// TestTrackLyricsKeepsVerseBreaks covers Qobuz's real separator shape: an
// empty line with no start or end. Those keep the plain text in stanzas but
// cannot become LRC entries, since there is no timestamp to use.
func TestTrackLyricsKeepsVerseBreaks(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "lsync",
"lines": []any{
map[string]any{"line": "verse one line one", "start": 1000},
map[string]any{"line": "verse one line two", "start": 2000},
map[string]any{"line": ""},
map[string]any{"line": ""},
map[string]any{"line": "verse two line one", "start": 3000},
map[string]any{"line": ""},
},
},
}, http.StatusOK)
plain, lrc, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
wantPlain := "verse one line one\nverse one line two\n\nverse two line one"
if plain != wantPlain {
t.Errorf("plain = %q, want %q", plain, wantPlain)
}
wantLRC := "[00:01.00]verse one line one\n[00:02.00]verse one line two\n[00:03.00]verse two line one"
if lrc != wantLRC {
t.Errorf("lrc = %q, want %q", lrc, wantLRC)
}
}
// TestTrackLyricsLeadingSeparatorDropped guards against a lyric tag that
// starts with a blank line.
func TestTrackLyricsLeadingSeparatorDropped(t *testing.T) {
srv := newLyricsServer(t, map[string]any{
"original": map[string]any{
"type": "plain",
"lines": []any{map[string]any{"line": ""}, map[string]any{"line": "first real line"}},
},
}, http.StatusOK)
plain, _, err := lyricsClient(t, srv).TrackLyrics(context.Background(), "42")
if err != nil {
t.Fatal(err)
}
if plain != "first real line" {
t.Errorf("plain = %q, want no leading blank", plain)
}
}
+110 -137
View File
@@ -10,13 +10,15 @@ import (
"net/url"
"os/exec"
"regexp"
"strconv"
"strings"
"sync"
"time"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/netutil"
"streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit"
)
var errUnsupportedMediaType = errors.New("unsupported soundcloud media type")
@@ -26,22 +28,32 @@ var soundcloudSearchBaseURL = "https://soundcloud.com"
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
type Client struct {
cfg *config.Config
cfg *config.Config
bin string
run commandRunner
http *http.Client
limiter *ratelimit.Limiter
mu sync.Mutex
cache map[string]map[string]any
// authMu guards loggedIn. Kept separate from mu, which guards cache, so a
// cache operation can never block a login check.
authMu sync.Mutex
loggedIn bool
bin string
run commandRunner
http *http.Client
mu sync.Mutex
cache map[string]map[string]any
}
func New(cfg *config.Config) *Client {
return &Client{
cfg: cfg,
bin: "yt-dlp",
run: runCommand,
http: &http.Client{Timeout: 20 * time.Second},
cache: map[string]map[string]any{},
cfg: cfg,
bin: "yt-dlp",
run: runCommand,
// Use the shared constructor like every other provider: rolling our own
// client here meant soundcloud silently ignored downloads.verify_ssl and
// downloads.max_connections, and bypassed the transport that logs
// requests at -vv with credentials redacted.
http: netutil.NewHTTPClient(20*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
cache: map[string]map[string]any{},
}
}
@@ -50,6 +62,8 @@ func (c *Client) Source() string {
}
func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn
}
@@ -57,12 +71,14 @@ func (c *Client) Login(context.Context) error {
if _, err := exec.LookPath(c.bin); err != nil {
return fmt.Errorf("yt-dlp is required for soundcloud downloads/search. install it and ensure it is in $PATH (e.g. pipx install yt-dlp): %w", err)
}
c.authMu.Lock()
defer c.authMu.Unlock()
c.loggedIn = true
return nil
}
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("soundcloud client not logged in")
}
if limit <= 0 {
@@ -102,14 +118,14 @@ func (c *Client) searchTracks(ctx context.Context, query string, limit int) ([]m
if id == "" {
continue
}
artist := strings.TrimSpace(stringFromAny(m["uploader"]))
artist := strings.TrimSpace(jsonutil.StringFromAny(m["uploader"]))
if artist == "" {
artist = strings.TrimSpace(stringFromAny(m["channel"]))
artist = strings.TrimSpace(jsonutil.StringFromAny(m["channel"]))
}
artistID := strings.TrimSpace(firstNonEmpty(stringFromAny(m["uploader_id"]), stringFromAny(m["channel_id"])))
artistID := strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(m["uploader_id"]), jsonutil.StringFromAny(m["channel_id"])))
item := map[string]any{
"id": id,
"title": stringFromAny(m["title"]),
"title": jsonutil.StringFromAny(m["title"]),
"artist": map[string]any{
"name": artist,
},
@@ -117,7 +133,7 @@ func (c *Client) searchTracks(ctx context.Context, query string, limit int) ([]m
if artistID != "" {
item["artist"] = map[string]any{"name": artist, "id": artistID}
}
if trackID := strings.TrimSpace(stringFromAny(m["id"])); trackID != "" {
if trackID := strings.TrimSpace(jsonutil.StringFromAny(m["id"])); trackID != "" {
item["source_track_id"] = trackID
}
items = append(items, item)
@@ -133,6 +149,11 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
}
req.Header.Set("User-Agent", "Mozilla/5.0")
if c.limiter != nil {
if err = c.limiter.Wait(ctx); err != nil {
return nil, err
}
}
resp, err := c.http.Do(req)
if err != nil {
return nil, err
@@ -146,7 +167,7 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
return nil, err
}
re := regexp.MustCompile(`/[A-Za-z0-9_-]+/sets/[A-Za-z0-9_-]+`)
re := regexp.MustCompile(`/(?:[A-Za-z0-9._-]+)/sets/(?:[A-Za-z0-9._%~-]+)`)
paths := re.FindAllString(string(body), -1)
if len(paths) == 0 {
return []map[string]any{}, nil
@@ -163,17 +184,17 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
if infoErr != nil {
continue
}
title := strings.TrimSpace(stringFromAny(info["title"]))
title := strings.TrimSpace(jsonutil.StringFromAny(info["title"]))
if title == "" {
title = strings.Trim(strings.ReplaceAll(path, "/", " "), " ")
}
artist := strings.TrimSpace(firstNonEmpty(stringFromAny(info["uploader"]), stringFromAny(info["channel"])))
artistID := strings.TrimSpace(firstNonEmpty(stringFromAny(info["uploader_id"]), stringFromAny(info["channel_id"])))
artist := strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(info["uploader"]), jsonutil.StringFromAny(info["channel"])))
artistID := strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(info["uploader_id"]), jsonutil.StringFromAny(info["channel_id"])))
trackCount := 0
if entries := asAnySlice(info["entries"]); len(entries) > 0 {
trackCount = len(entries)
}
canonical := firstNonEmpty(canonicalSoundcloudURL(info), playlistURL)
canonical := jsonutil.FirstNonEmpty(canonicalSoundcloudURL(info), playlistURL)
item := map[string]any{
"id": canonical,
"title": title,
@@ -183,10 +204,10 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
if artistID != "" {
item["artist"] = map[string]any{"name": artist, "id": artistID}
}
if pid := strings.TrimSpace(stringFromAny(info["id"])); pid != "" {
if pid := strings.TrimSpace(jsonutil.StringFromAny(info["id"])); pid != "" {
item["source_playlist_id"] = pid
}
if thumb := strings.TrimSpace(stringFromAny(info["thumbnail"])); thumb != "" {
if thumb := strings.TrimSpace(jsonutil.StringFromAny(info["thumbnail"])); thumb != "" {
item["image"] = soundcloudImageMap(thumb)
}
items = append(items, item)
@@ -201,7 +222,7 @@ func (c *Client) searchPlaylists(ctx context.Context, query string, limit int) (
}
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("soundcloud client not logged in")
}
@@ -228,15 +249,15 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
continue
}
track := map[string]any{"id": id}
if trackID := strings.TrimSpace(stringFromAny(entry["id"])); trackID != "" {
if trackID := strings.TrimSpace(jsonutil.StringFromAny(entry["id"])); trackID != "" {
track["source_track_id"] = trackID
}
if title := strings.TrimSpace(stringFromAny(entry["title"])); title != "" {
if title := strings.TrimSpace(jsonutil.StringFromAny(entry["title"])); title != "" {
track["title"] = title
}
if artist := strings.TrimSpace(firstNonEmpty(stringFromAny(entry["uploader"]), stringFromAny(entry["channel"]))); artist != "" {
if artist := strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(entry["uploader"]), jsonutil.StringFromAny(entry["channel"]))); artist != "" {
artistMap := map[string]any{"name": artist}
if artistID := strings.TrimSpace(firstNonEmpty(stringFromAny(entry["uploader_id"]), stringFromAny(entry["channel_id"]))); artistID != "" {
if artistID := strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(entry["uploader_id"]), jsonutil.StringFromAny(entry["channel_id"]))); artistID != "" {
artistMap["id"] = artistID
}
track["artist"] = artistMap
@@ -244,23 +265,23 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
track["track_number"] = i + 1
tracks = append(tracks, track)
}
name := strings.TrimSpace(stringFromAny(root["title"]))
name := strings.TrimSpace(jsonutil.StringFromAny(root["title"]))
if name == "" {
name = "SoundCloud Playlist"
}
meta := map[string]any{
"id": firstNonEmpty(canonicalSoundcloudURL(root), item),
"id": jsonutil.FirstNonEmpty(canonicalSoundcloudURL(root), item),
"name": name,
"description": strings.TrimSpace(stringFromAny(root["description"])),
"description": strings.TrimSpace(jsonutil.StringFromAny(root["description"])),
"tracks": map[string]any{"items": tracks},
}
if pid := strings.TrimSpace(stringFromAny(root["id"])); pid != "" {
if pid := strings.TrimSpace(jsonutil.StringFromAny(root["id"])); pid != "" {
meta["source_playlist_id"] = pid
}
if artist := strings.TrimSpace(firstNonEmpty(stringFromAny(root["uploader"]), stringFromAny(root["channel"]))); artist != "" {
if artist := strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(root["uploader"]), jsonutil.StringFromAny(root["channel"]))); artist != "" {
meta["artist"] = map[string]any{"name": artist}
}
if thumb := strings.TrimSpace(stringFromAny(root["thumbnail"])); thumb != "" {
if thumb := strings.TrimSpace(jsonutil.StringFromAny(root["thumbnail"])); thumb != "" {
meta["image"] = soundcloudImageMap(thumb)
}
if entries := asAnySlice(root["entries"]); len(entries) > 0 {
@@ -273,28 +294,45 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
}
func (c *Client) GetDownloadable(ctx context.Context, item string, _ int) (*provider.Downloadable, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("soundcloud client not logged in")
}
info, err := c.trackInfo(ctx, item)
if err != nil {
return nil, err
}
streamURL := strings.TrimSpace(stringFromAny(info["url"]))
streamURL := strings.TrimSpace(jsonutil.StringFromAny(info["url"]))
if streamURL == "" {
return nil, errors.New("yt-dlp output missing url (track may be unavailable or region-restricted)")
}
ext := strings.TrimSpace(stringFromAny(info["ext"]))
ext := strings.TrimSpace(jsonutil.StringFromAny(info["ext"]))
if ext == "" {
ext = "m4a"
}
return &provider.Downloadable{URL: streamURL, Extension: ext, Source: "soundcloud"}, nil
return &provider.Downloadable{URL: streamURL, Extension: ext, Source: "soundcloud", Audio: soundcloudAudioProfile(ext)}, nil
}
func (c *Client) Close() error {
return nil
}
func soundcloudAudioProfile(ext string) provider.AudioProfile {
switch strings.ToLower(strings.TrimSpace(ext)) {
case "mp3":
return provider.AudioProfile{Container: "MP3", Codec: "MP3", Quality: "LOSSY", BitDepth: 16, SamplingRate: "44.1"}
case "flac":
return provider.AudioProfile{Container: "FLAC", Codec: "FLAC", Quality: "LOSSLESS", BitDepth: 16, SamplingRate: "44.1"}
case "m4a", "aac":
return provider.AudioProfile{Container: "M4A", Codec: "AAC", Quality: "LOSSY", BitDepth: 16, SamplingRate: "44.1"}
default:
container := strings.ToUpper(strings.TrimSpace(ext))
if container == "" {
container = "M4A"
}
return provider.AudioProfile{Container: container, Codec: container, Quality: "LOSSY", BitDepth: 16, SamplingRate: "44.1"}
}
}
func (c *Client) trackInfo(ctx context.Context, item string) (map[string]any, error) {
if strings.TrimSpace(item) == "" {
return nil, errors.New("empty soundcloud item")
@@ -337,36 +375,36 @@ func (c *Client) playlistInfo(ctx context.Context, item string) (map[string]any,
}
func trackMetadataFromInfo(id string, info map[string]any) map[string]any {
canonicalID := firstNonEmpty(canonicalSoundcloudURL(info), id)
publisher := nestedMap(info, "publisher_metadata")
title := strings.TrimSpace(stringFromAny(info["title"]))
canonicalID := jsonutil.FirstNonEmpty(canonicalSoundcloudURL(info), id)
publisher := jsonutil.NestedMap(info, "publisher_metadata")
title := strings.TrimSpace(jsonutil.StringFromAny(info["title"]))
if title == "" {
title = canonicalID
}
albumTitle := strings.TrimSpace(stringFromAny(publisher["album_title"]))
albumTitle := strings.TrimSpace(jsonutil.StringFromAny(publisher["album_title"]))
if albumTitle == "" {
albumTitle = strings.TrimSpace(stringFromAny(info["album"]))
albumTitle = strings.TrimSpace(jsonutil.StringFromAny(info["album"]))
}
if albumTitle == "" {
albumTitle = title
}
artistName := strings.TrimSpace(stringFromAny(info["artist"]))
artistName := strings.TrimSpace(jsonutil.StringFromAny(info["artist"]))
if artistName == "" {
artistName = strings.TrimSpace(stringFromAny(publisher["artist"]))
artistName = strings.TrimSpace(jsonutil.StringFromAny(publisher["artist"]))
}
if artistName == "" {
artistName = strings.TrimSpace(stringFromAny(info["uploader"]))
artistName = strings.TrimSpace(jsonutil.StringFromAny(info["uploader"]))
}
if artistName == "" {
artistName = strings.TrimSpace(stringFromAny(info["channel"]))
artistName = strings.TrimSpace(jsonutil.StringFromAny(info["channel"]))
}
artistID := strings.TrimSpace(firstNonEmpty(
stringFromAny(info["uploader_id"]),
stringFromAny(info["channel_id"]),
stringFromAny(nestedMap(info, "user")["id"]),
artistID := strings.TrimSpace(jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(info["uploader_id"]),
jsonutil.StringFromAny(info["channel_id"]),
jsonutil.StringFromAny(jsonutil.NestedMap(info, "user")["id"]),
))
trackNum := intFromAny(info["track_number"])
trackNum := jsonutil.IntFromAny(info["track_number"])
if trackNum <= 0 {
trackNum = 1
}
@@ -378,26 +416,26 @@ func trackMetadataFromInfo(id string, info map[string]any) map[string]any {
"artist": map[string]any{"name": artistName, "id": artistID},
"performer": map[string]any{"name": artistName, "id": artistID},
"album": map[string]any{
"id": firstNonEmpty(strings.TrimSpace(stringFromAny(info["album"])), canonicalID),
"id": jsonutil.FirstNonEmpty(strings.TrimSpace(jsonutil.StringFromAny(info["album"])), canonicalID),
"title": albumTitle,
"artist": map[string]any{"name": artistName, "id": artistID},
},
"description": strings.TrimSpace(stringFromAny(info["description"])),
"genre": strings.TrimSpace(stringFromAny(info["genre"])),
"isrc": strings.TrimSpace(stringFromAny(info["isrc"])),
"label": strings.TrimSpace(firstNonEmpty(stringFromAny(info["label"]), stringFromAny(info["label_name"]))),
"copyright": strings.TrimSpace(stringFromAny(publisher["p_line"])),
"release_date": strings.TrimSpace(firstNonEmpty(
stringFromAny(info["created_at"]),
stringFromAny(info["release_date"]),
stringFromAny(info["upload_date"]),
"description": strings.TrimSpace(jsonutil.StringFromAny(info["description"])),
"genre": strings.TrimSpace(jsonutil.StringFromAny(info["genre"])),
"isrc": strings.TrimSpace(jsonutil.StringFromAny(info["isrc"])),
"label": strings.TrimSpace(jsonutil.FirstNonEmpty(jsonutil.StringFromAny(info["label"]), jsonutil.StringFromAny(info["label_name"]))),
"copyright": strings.TrimSpace(jsonutil.StringFromAny(publisher["p_line"])),
"release_date": strings.TrimSpace(jsonutil.FirstNonEmpty(
jsonutil.StringFromAny(info["created_at"]),
jsonutil.StringFromAny(info["release_date"]),
jsonutil.StringFromAny(info["upload_date"]),
)),
}
if trackID := strings.TrimSpace(stringFromAny(info["id"])); trackID != "" {
if trackID := strings.TrimSpace(jsonutil.StringFromAny(info["id"])); trackID != "" {
meta["source_track_id"] = trackID
}
if boolFromAny(publisher["explicit"]) || intFromAny(info["age_limit"]) >= 18 {
if jsonutil.BoolFromAny(publisher["explicit"]) || jsonutil.IntFromAny(info["age_limit"]) >= 18 {
meta["explicit"] = true
}
@@ -405,11 +443,11 @@ func trackMetadataFromInfo(id string, info map[string]any) map[string]any {
delete(meta, "release_date")
}
if thumb := strings.TrimSpace(stringFromAny(info["thumbnail"])); thumb != "" {
if thumb := strings.TrimSpace(jsonutil.StringFromAny(info["thumbnail"])); thumb != "" {
meta["image"] = soundcloudImageMap(thumb)
}
if strings.TrimSpace(stringFromAny(info["album"])) == "" && strings.TrimSpace(stringFromAny(publisher["album_title"])) == "" {
if strings.TrimSpace(jsonutil.StringFromAny(info["album"])) == "" && strings.TrimSpace(jsonutil.StringFromAny(publisher["album_title"])) == "" {
meta["album"] = map[string]any{
"id": canonicalID,
"title": title,
@@ -417,7 +455,7 @@ func trackMetadataFromInfo(id string, info map[string]any) map[string]any {
}
}
if durationSec := intFromAny(info["duration"]); durationSec > 0 {
if durationSec := jsonutil.IntFromAny(info["duration"]); durationSec > 0 {
meta["duration"] = durationSec
}
@@ -426,7 +464,7 @@ func trackMetadataFromInfo(id string, info map[string]any) map[string]any {
func canonicalSoundcloudURL(info map[string]any) string {
for _, key := range []string{"webpage_url", "original_url", "url"} {
raw := strings.TrimSpace(stringFromAny(info[key]))
raw := strings.TrimSpace(jsonutil.StringFromAny(info[key]))
if raw == "" {
continue
}
@@ -435,10 +473,11 @@ func canonicalSoundcloudURL(info map[string]any) string {
continue
}
host := strings.ToLower(strings.TrimPrefix(u.Host, "www."))
if host != "soundcloud.com" {
if host != "soundcloud.com" && !strings.HasSuffix(host, ".soundcloud.com") {
continue
}
u.Scheme = "https"
u.Host = "soundcloud.com"
u.RawQuery = ""
u.Fragment = ""
u.Path = strings.TrimSuffix(u.Path, "/")
@@ -477,72 +516,6 @@ func asAnySlice(v any) []any {
return items
}
func stringFromAny(v any) string {
switch t := v.(type) {
case string:
return t
case int:
return strconv.Itoa(t)
case int64:
return strconv.FormatInt(t, 10)
case float64:
return strconv.FormatFloat(t, 'f', -1, 64)
default:
return ""
}
}
func intFromAny(v any) int {
switch t := v.(type) {
case int:
return t
case int64:
return int(t)
case float64:
return int(t)
case string:
i, _ := strconv.Atoi(strings.TrimSpace(t))
return i
default:
return 0
}
}
func firstNonEmpty(items ...string) string {
for _, item := range items {
if strings.TrimSpace(item) != "" {
return strings.TrimSpace(item)
}
}
return ""
}
func nestedMap(m map[string]any, key string) map[string]any {
v, ok := m[key].(map[string]any)
if !ok {
return map[string]any{}
}
return v
}
func boolFromAny(v any) bool {
switch t := v.(type) {
case bool:
return t
case string:
l := strings.ToLower(strings.TrimSpace(t))
return l == "1" || l == "true" || l == "yes"
case int:
return t != 0
case int64:
return t != 0
case float64:
return t != 0
default:
return false
}
}
func soundcloudImageMap(raw string) map[string]any {
base := strings.TrimSpace(raw)
if base == "" {
+90 -23
View File
@@ -8,11 +8,13 @@ import (
"strings"
"testing"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/config"
)
func TestGetTrackMetadataAndDownloadable(t *testing.T) {
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
@@ -27,11 +29,11 @@ func TestGetTrackMetadataAndDownloadable(t *testing.T) {
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
if stringFromAny(meta["title"]) != "Lean On" {
t.Fatalf("title = %q, want Lean On", stringFromAny(meta["title"]))
if jsonutil.StringFromAny(meta["title"]) != "Lean On" {
t.Fatalf("title = %q, want Lean On", jsonutil.StringFromAny(meta["title"]))
}
if stringFromAny(meta["id"]) != "https://soundcloud.com/a/b" {
t.Fatalf("id = %q, want canonical soundcloud url", stringFromAny(meta["id"]))
if jsonutil.StringFromAny(meta["id"]) != "https://soundcloud.com/a/b" {
t.Fatalf("id = %q, want canonical soundcloud url", jsonutil.StringFromAny(meta["id"]))
}
d, err := c.GetDownloadable(context.Background(), "https://soundcloud.com/a/b", 0)
@@ -41,10 +43,13 @@ func TestGetTrackMetadataAndDownloadable(t *testing.T) {
if d.URL != "https://cdn.example/audio.m4a" || d.Extension != "m4a" {
t.Fatalf("unexpected downloadable: %+v", d)
}
if d.Audio.Container != "M4A" || d.Audio.Codec != "AAC" {
t.Fatalf("unexpected audio profile: %+v", d.Audio)
}
}
func TestGetPlaylistMetadata(t *testing.T) {
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
@@ -59,8 +64,8 @@ func TestGetPlaylistMetadata(t *testing.T) {
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
if stringFromAny(meta["name"]) != "Road Trip" {
t.Fatalf("name = %q, want Road Trip", stringFromAny(meta["name"]))
if jsonutil.StringFromAny(meta["name"]) != "Road Trip" {
t.Fatalf("name = %q, want Road Trip", jsonutil.StringFromAny(meta["name"]))
}
tracksMap, ok := meta["tracks"].(map[string]any)
if !ok {
@@ -70,13 +75,13 @@ func TestGetPlaylistMetadata(t *testing.T) {
if len(items) != 2 {
t.Fatalf("playlist items len = %d, want 2", len(items))
}
if stringFromAny(meta["id"]) != "https://soundcloud.com/a/sets/road-trip" {
t.Fatalf("playlist id not canonical: %q", stringFromAny(meta["id"]))
if jsonutil.StringFromAny(meta["id"]) != "https://soundcloud.com/a/sets/road-trip" {
t.Fatalf("playlist id not canonical: %q", jsonutil.StringFromAny(meta["id"]))
}
}
func TestSearchTrack(t *testing.T) {
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
@@ -102,8 +107,8 @@ func TestSearchTrack(t *testing.T) {
if !ok {
t.Fatalf("expected first item map")
}
if stringFromAny(item0["id"]) != "https://soundcloud.com/a/b" {
t.Fatalf("track search id not canonical: %q", stringFromAny(item0["id"]))
if jsonutil.StringFromAny(item0["id"]) != "https://soundcloud.com/a/b" {
t.Fatalf("track search id not canonical: %q", jsonutil.StringFromAny(item0["id"]))
}
}
@@ -117,7 +122,7 @@ func TestSearchPlaylist(t *testing.T) {
}))
defer ts.Close()
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.http = ts.Client()
@@ -147,13 +152,58 @@ func TestSearchPlaylist(t *testing.T) {
if !ok {
t.Fatalf("expected first item map")
}
if stringFromAny(item0["id"]) != "https://soundcloud.com/a/sets/road-trip" {
t.Fatalf("playlist search id not canonical: %q", stringFromAny(item0["id"]))
if jsonutil.StringFromAny(item0["id"]) != "https://soundcloud.com/a/sets/road-trip" {
t.Fatalf("playlist search id not canonical: %q", jsonutil.StringFromAny(item0["id"]))
}
}
func TestSearchPlaylistAcceptsDotsInPath(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/search/sets" {
_, _ = w.Write([]byte(`<html><body><a href="/artist.name/sets/road.trip">x</a></body></html>`))
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer ts.Close()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.http = ts.Client()
origBase := soundcloudSearchBaseURL
soundcloudSearchBaseURL = ts.URL
defer func() { soundcloudSearchBaseURL = origBase }()
c.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
joined := strings.Join(args, " ")
if strings.Contains(joined, "https://soundcloud.com/artist.name/sets/road.trip") {
return []byte(`{"title":"Road Trip","uploader":"User","entries":[{"webpage_url":"https://soundcloud.com/a/t1"}]}`), nil
}
return nil, fmt.Errorf("unexpected args: %v", args)
}
pages, err := c.Search(context.Background(), "playlist", "road trip", 5)
if err != nil {
t.Fatalf("Search() error = %v", err)
}
if len(pages) != 1 {
t.Fatalf("pages len = %d, want 1", len(pages))
}
items := asAnySlice(pages[0]["items"])
if len(items) != 1 {
t.Fatalf("items len = %d, want 1", len(items))
}
item0, ok := items[0].(map[string]any)
if !ok {
t.Fatalf("expected first item map")
}
if jsonutil.StringFromAny(item0["id"]) != "https://soundcloud.com/artist.name/sets/road.trip" {
t.Fatalf("playlist search id not canonical: %q", jsonutil.StringFromAny(item0["id"]))
}
}
func TestLoginShowsYtDlpHint(t *testing.T) {
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
c := New(&config.Config{File: cfgData, Session: cfgData})
c.bin = "definitely-not-a-real-yt-dlp-bin"
err := c.Login(context.Background())
@@ -176,18 +226,18 @@ func TestTrackMetadataIncludesExplicitAndISRC(t *testing.T) {
"thumbnail": "https://img",
"upload_date": "20240101",
})
if stringFromAny(meta["isrc"]) != "US123" {
t.Fatalf("isrc = %q, want US123", stringFromAny(meta["isrc"]))
if jsonutil.StringFromAny(meta["isrc"]) != "US123" {
t.Fatalf("isrc = %q, want US123", jsonutil.StringFromAny(meta["isrc"]))
}
explicit, _ := meta["explicit"].(bool)
if !explicit {
t.Fatalf("expected explicit=true")
}
if stringFromAny(meta["source_track_id"]) != "9876" {
t.Fatalf("source_track_id = %q, want 9876", stringFromAny(meta["source_track_id"]))
if jsonutil.StringFromAny(meta["source_track_id"]) != "9876" {
t.Fatalf("source_track_id = %q, want 9876", jsonutil.StringFromAny(meta["source_track_id"]))
}
if stringFromAny(nestedMap(meta, "album")["title"]) != "T" {
t.Fatalf("album title mismatch: %#v", nestedMap(meta, "album"))
if jsonutil.StringFromAny(jsonutil.NestedMap(meta, "album")["title"]) != "T" {
t.Fatalf("album title mismatch: %#v", jsonutil.NestedMap(meta, "album"))
}
}
@@ -197,3 +247,20 @@ func TestCanonicalSoundcloudURL(t *testing.T) {
t.Fatalf("canonical url = %q, want %q", got, "https://soundcloud.com/a/b")
}
}
func TestCanonicalSoundcloudURLAcceptsSubdomain(t *testing.T) {
got := canonicalSoundcloudURL(map[string]any{"webpage_url": "https://m.soundcloud.com/a/b/?si=x#frag"})
if got != "https://soundcloud.com/a/b" {
t.Fatalf("canonical url = %q, want %q", got, "https://soundcloud.com/a/b")
}
}
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+534 -161
View File
@@ -12,20 +12,23 @@ import (
"net/url"
"strconv"
"strings"
"sync"
"time"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
"streamrip-go/internal/netutil"
"streamrip-go/internal/provider"
"streamrip-go/internal/ratelimit"
"streamrip-go/internal/verbose"
)
const (
baseURL = "https://api.tidalhifi.com/v1"
openAPIV2 = "https://openapi.tidal.com/v2"
authURL = "https://auth.tidal.com/v1/oauth2"
clientID = "fX2JxdmntZWK0ixT"
clientSec = "1Nm5AfDAjxrgJFJbKNWLeAyKGVGmINuXPPLHVXAvxAg="
baseURL = "https://api.tidalhifi.com/v1"
lyricsAPIv1 = "https://api.tidal.com/v1"
openAPIV2 = "https://openapi.tidal.com/v2"
authURL = "https://auth.tidal.com/v1/oauth2"
tidalRequestAttempts = 3
)
var qualityMap = map[int]string{
@@ -36,30 +39,39 @@ var qualityMap = map[int]string{
4: "HI_RES_LOSSLESS",
}
var qualityToFormat = map[int]string{
0: "HEAACV1",
1: "AACLC",
2: "FLAC",
3: "FLAC_HIRES",
4: "FLAC_HIRES",
var qualityToFormats = map[int][]string{
0: {"HEAACV1"},
1: {"HEAACV1", "AACLC"},
2: {"HEAACV1", "AACLC", "FLAC"},
3: {"HEAACV1", "AACLC", "FLAC", "FLAC_HIRES"},
4: {"HEAACV1", "AACLC", "FLAC", "FLAC_HIRES"},
}
var atmosAudioQualities = []string{"HI_RES_LOSSLESS", "HI_RES", "LOSSLESS", "HIGH"}
var ErrMissingTidalToken = errors.New("missing tidal access_token")
type Client struct {
cfg *config.Config
http *http.Client
limiter *ratelimit.Limiter
baseURL string
cfg *config.Config
http *http.Client
limiter *ratelimit.Limiter
baseURL string
lyricsAPI string
openAPI string
// authMu guards loggedIn and the login/refresh path, which download
// workers can reach concurrently.
authMu sync.Mutex
loggedIn bool
}
func New(cfg *config.Config) *Client {
return &Client{
cfg: cfg,
http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute),
baseURL: baseURL,
cfg: cfg,
http: netutil.NewHTTPClient(30*time.Second, cfg.Session.Downloads.VerifySSL, cfg.Session.Downloads.MaxConnections),
limiter: ratelimit.New(cfg.Session.Downloads.RequestsPerMinute, cfg.Session.Downloads.MaxConnections),
baseURL: baseURL,
lyricsAPI: lyricsAPIv1,
openAPI: openAPIV2,
}
}
@@ -68,10 +80,14 @@ func (c *Client) Source() string {
}
func (c *Client) LoggedIn() bool {
c.authMu.Lock()
defer c.authMu.Unlock()
return c.loggedIn
}
func (c *Client) Login(ctx context.Context) error {
c.authMu.Lock()
defer c.authMu.Unlock()
if strings.TrimSpace(c.cfg.Session.Tidal.AccessToken) == "" {
return ErrMissingTidalToken
}
@@ -102,10 +118,10 @@ func (c *Client) Login(ctx context.Context) error {
return fmt.Errorf("tidal login failed: status=%d body=%v", status, resp)
}
if v := stringify(resp["countryCode"]); v != "" {
if v := jsonutil.StringFromAny(resp["countryCode"]); v != "" {
c.cfg.Session.Tidal.CountryCode = v
}
if v := stringify(resp["userId"]); v != "" {
if v := jsonutil.StringFromAny(resp["userId"]); v != "" {
c.cfg.Session.Tidal.UserID = v
}
@@ -128,7 +144,7 @@ func (c *Client) refreshAccessToken(ctx context.Context) error {
}
form := url.Values{}
form.Set("client_id", clientID)
form.Set("client_id", c.appClientID())
form.Set("refresh_token", refresh)
form.Set("grant_type", "refresh_token")
form.Set("scope", "r_usr+w_usr+w_sub")
@@ -141,33 +157,36 @@ func (c *Client) refreshAccessToken(ctx context.Context) error {
return fmt.Errorf("tidal token refresh failed: status=%d body=%v", status, resp)
}
newToken := stringify(resp["access_token"])
newToken := jsonutil.StringFromAny(resp["access_token"])
if newToken == "" {
return errors.New("tidal token refresh missing access_token")
}
newRefresh := stringify(resp["refresh_token"])
expiresIn := int64(intFromAny(resp["expires_in"]))
newRefresh := jsonutil.StringFromAny(resp["refresh_token"])
expiresIn := int64(jsonutil.IntFromAny(resp["expires_in"]))
if expiresIn <= 0 {
expiresIn = 7 * 24 * 3600
}
c.cfg.Session.Tidal.AccessToken = newToken
c.cfg.File.Tidal.AccessToken = newToken
if newRefresh != "" {
c.cfg.Session.Tidal.RefreshToken = newRefresh
c.cfg.File.Tidal.RefreshToken = newRefresh
}
expiry := time.Now().Unix() + expiresIn
c.cfg.Session.Tidal.TokenExpiry = expiry
c.cfg.File.Tidal.TokenExpiry = expiry
_ = c.cfg.SaveFile()
if err := c.cfg.UpdateTokens(func(file, session *config.ConfigData) {
file.Tidal.AccessToken = newToken
session.Tidal.AccessToken = newToken
if newRefresh != "" {
file.Tidal.RefreshToken = newRefresh
session.Tidal.RefreshToken = newRefresh
}
file.Tidal.TokenExpiry = expiry
session.Tidal.TokenExpiry = expiry
}); err != nil {
verbose.Printf(verbose.V, "tidal: could not persist refreshed token: %v\n", err)
}
return nil
}
func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[string]any, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in")
}
@@ -201,13 +220,54 @@ func (c *Client) GetMetadata(ctx context.Context, item, mediaType string) (map[s
if album, ok := resp["album"].(map[string]any); ok {
enrichTidalImage(album)
}
// Lyrics live on a separate endpoint, so fetching them costs an extra
// rate-limited roundtrip per track. Users who don't embed lyrics can
// opt out via metadata.exclude = ["lyrics"].
if !c.lyricsExcluded() {
if lyrics, lrc := c.fetchTrackLyrics(ctx, item); lyrics != "" || lrc != "" {
if lyrics != "" {
resp["lyrics"] = lyrics
}
if lrc != "" {
resp["lyrics_synced"] = lrc
}
}
}
}
return resp, nil
}
func (c *Client) lyricsExcluded() bool {
for _, k := range c.cfg.Session.Metadata.Exclude {
if strings.EqualFold(strings.TrimSpace(k), "lyrics") {
return true
}
}
return false
}
func (c *Client) fetchTrackLyrics(ctx context.Context, trackID string) (string, string) {
params := url.Values{}
params.Set("deviceType", "PHONE")
params.Set("locale", "en_US")
params.Set("platform", "ANDROID")
resp, status, err := c.apiRequest(ctx, "tracks/"+url.PathEscape(strings.TrimSpace(trackID))+"/lyrics", params, c.lyricsAPI)
if err != nil {
return "", ""
}
if status != http.StatusOK {
return "", ""
}
lyrics := strings.TrimSpace(jsonutil.StringFromAny(resp["lyrics"]))
lrc := strings.TrimSpace(jsonutil.StringFromAny(resp["subtitles"]))
return lyrics, lrc
}
func (c *Client) Search(ctx context.Context, mediaType, query string, limit int) ([]map[string]any, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in")
}
if limit <= 0 {
@@ -234,13 +294,22 @@ func (c *Client) Search(ctx context.Context, mediaType, query string, limit int)
}
func (c *Client) GetDownloadable(ctx context.Context, trackID string, quality int) (*provider.Downloadable, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in")
}
if quality < 0 || quality > 4 {
quality = c.cfg.Session.Tidal.Quality
}
if c.cfg.Session.Tidal.PreferAtmos {
// No tracks/{id} pre-check: getAtmosDownloadable already validates
// each candidate response via playbackLooksAtmos and falls back
// through the format-specific trackManifests paths.
if d, _ := c.getAtmosDownloadable(ctx, trackID); d != nil {
return d, nil
}
}
params := url.Values{}
params.Set("audioquality", qualityMap[quality])
params.Set("playbackmode", "STREAM")
@@ -252,6 +321,15 @@ func (c *Client) GetDownloadable(ctx context.Context, trackID string, quality in
}
if status == http.StatusOK {
if d := downloadableFromPlaybackManifest(resp); d != nil {
// Tidal's playbackinfo sometimes returns an m4a (HIGH/AAC)
// stream even when LOSSLESS+ was requested. There is no
// lossless m4a tier, so retry via the openAPI v2 manifest
// before settling for the downgrade.
if quality >= 2 && d.Extension == "m4a" {
if strict, strictErr := c.getDownloadableFromTrackManifest(ctx, trackID, quality); strictErr == nil && strict != nil {
return strict, nil
}
}
return d, nil
}
}
@@ -259,6 +337,87 @@ func (c *Client) GetDownloadable(ctx context.Context, trackID string, quality in
return c.getDownloadableFromTrackManifest(ctx, trackID, quality)
}
func (c *Client) getAtmosDownloadable(ctx context.Context, trackID string) (*provider.Downloadable, error) {
var lastErr error
for _, aq := range atmosAudioQualities {
params := url.Values{}
params.Set("audioquality", aq)
params.Set("playbackmode", "STREAM")
params.Set("assetpresentation", "FULL")
params.Set("immersiveaudio", "true")
resp, status, err := c.apiRequest(ctx, "tracks/"+trackID+"/playbackinfopostpaywall", params, c.baseURL)
if err != nil {
lastErr = err
continue
}
if status != http.StatusOK {
lastErr = fmt.Errorf("tidal atmos playbackinfo failed: status=%d", status)
continue
}
if !playbackLooksAtmos(resp) {
continue
}
if d := downloadableFromPlaybackManifest(resp); d != nil {
return d, nil
}
}
if d, err := c.getDownloadableFromTrackManifestForFormat(ctx, trackID, "EAC3_JOC"); err == nil {
return d, nil
} else if err != nil {
lastErr = err
}
if d, err := c.getDownloadableFromTrackManifestForFormat(ctx, trackID, "DOLBY_ATMOS"); err == nil {
return d, nil
} else if err != nil {
lastErr = err
}
if d, err := c.getDownloadableFromTrackManifestForFormat(ctx, trackID, "SONY_360RA"); err == nil {
return d, nil
} else if err != nil {
lastErr = err
}
return nil, lastErr
}
func playbackLooksAtmos(resp map[string]any) bool {
if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(resp["audioMode"])), "ATMOS") {
return true
}
if modes, ok := resp["audioModes"].([]any); ok {
for _, raw := range modes {
if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(raw)), "ATMOS") {
return true
}
}
}
manifestB64 := jsonutil.StringFromAny(resp["manifest"])
if manifestB64 == "" {
return false
}
b, err := base64.StdEncoding.DecodeString(manifestB64)
if err != nil {
return false
}
manifest := map[string]any{}
if err = json.Unmarshal(b, &manifest); err != nil {
return false
}
if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(manifest["audioMode"])), "ATMOS") {
return true
}
if modes, ok := manifest["audioModes"].([]any); ok {
for _, raw := range modes {
if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(raw)), "ATMOS") {
return true
}
}
}
codec := strings.ToLower(jsonutil.StringFromAny(manifest["codecs"]))
return strings.Contains(codec, "ec-3") || strings.Contains(codec, "eac3") || strings.Contains(codec, "joc") || strings.Contains(codec, "atmos")
}
func (c *Client) Close() error {
return nil
}
@@ -310,46 +469,72 @@ func (c *Client) fetchArtistAlbums(ctx context.Context, artistID string) ([]map[
out := make([]map[string]any, 0)
seen := map[string]struct{}{}
for _, p := range paths {
resp, status, err := c.apiRequest(ctx, p.path, p.params, c.baseURL)
if err != nil {
return nil, err
}
if status != http.StatusOK {
return nil, fmt.Errorf("tidal artist albums failed: status=%d", status)
}
items, _ := resp["items"].([]any)
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
offset := 0
for {
params := url.Values{}
for k, values := range p.params {
for _, v := range values {
params.Add(k, v)
}
}
if wrapped, ok := itm["item"].(map[string]any); ok {
itm = wrapped
params.Set("offset", strconv.Itoa(offset))
resp, status, err := c.apiRequest(ctx, p.path, params, c.baseURL)
if err != nil {
return nil, err
}
id := stringify(itm["id"])
if id == "" {
continue
if status != http.StatusOK {
return nil, fmt.Errorf("tidal artist albums failed: status=%d offset=%d", status, offset)
}
if _, dup := seen[id]; dup {
continue
items, _ := resp["items"].([]any)
if len(items) == 0 {
break
}
seen[id] = struct{}{}
out = append(out, itm)
for _, raw := range items {
itm, ok := raw.(map[string]any)
if !ok {
continue
}
if wrapped, ok := itm["item"].(map[string]any); ok {
itm = wrapped
}
id := jsonutil.StringFromAny(itm["id"])
if id == "" {
continue
}
if _, dup := seen[id]; dup {
continue
}
seen[id] = struct{}{}
out = append(out, itm)
}
if len(items) < 100 {
break
}
offset += 100
}
}
return out, nil
}
func (c *Client) getDownloadableFromTrackManifest(ctx context.Context, trackID string, quality int) (*provider.Downloadable, error) {
format := qualityToFormat[quality]
formats := formatsForQuality(quality, c.cfg.Session.Tidal.PreferAtmos)
return c.getDownloadableFromTrackManifestForFormats(ctx, trackID, formats)
}
func (c *Client) getDownloadableFromTrackManifestForFormat(ctx context.Context, trackID, format string) (*provider.Downloadable, error) {
return c.getDownloadableFromTrackManifestForFormats(ctx, trackID, []string{format})
}
func (c *Client) getDownloadableFromTrackManifestForFormats(ctx context.Context, trackID string, formats []string) (*provider.Downloadable, error) {
params := url.Values{}
params.Set("manifestType", "MPEG_DASH")
params.Set("formats", format)
params.Set("formats", strings.Join(formats, ","))
params.Set("uriScheme", "HTTPS")
params.Set("usage", "PLAYBACK")
params.Set("adaptive", "false")
resp, status, err := c.apiRequest(ctx, "trackManifests/"+trackID, params, openAPIV2)
resp, status, err := c.apiRequest(ctx, "trackManifests/"+trackID, params, c.openAPI)
if err != nil {
return nil, err
}
@@ -365,24 +550,44 @@ func (c *Client) getDownloadableFromTrackManifest(ctx context.Context, trackID s
if !ok {
return nil, errors.New("tidal trackManifests missing attributes")
}
uri := stringify(attrs["uri"])
uri := jsonutil.StringFromAny(attrs["uri"])
if uri == "" {
return nil, errors.New("tidal trackManifests missing uri")
}
formats, _ := attrs["formats"].([]any)
attrFormats, _ := attrs["formats"].([]any)
ext := "m4a"
for _, f := range formats {
if strings.Contains(strings.ToUpper(stringify(f)), "FLAC") {
for _, f := range attrFormats {
fv := strings.ToUpper(jsonutil.StringFromAny(f))
if strings.Contains(fv, "FLAC") {
ext = "flac"
break
}
if strings.Contains(fv, "EAC3") || strings.Contains(fv, "ATMOS") || strings.Contains(fv, "JOC") {
ext = "mka"
}
}
return &provider.Downloadable{URL: uri, Extension: ext, Source: "tidal"}, nil
profile := tidalAudioProfileFromFormats(attrFormats)
if profile.Container == "" {
profile = tidalAudioProfileFromExtension(ext)
}
return &provider.Downloadable{URL: uri, Extension: ext, Source: "tidal", Audio: profile}, nil
}
func formatsForQuality(quality int, preferAtmos bool) []string {
base, ok := qualityToFormats[quality]
if !ok {
base = qualityToFormats[0]
}
out := append([]string(nil), base...)
if preferAtmos {
out = append(out, "EAC3_JOC")
}
return out
}
func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*provider.Downloadable, error) {
if !c.loggedIn {
if !c.LoggedIn() {
return nil, errors.New("tidal client not logged in")
}
@@ -399,7 +604,7 @@ func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*pro
return nil, fmt.Errorf("tidal video playbackinfo failed: status=%d", status)
}
manifestB64 := stringify(resp["manifest"])
manifestB64 := jsonutil.StringFromAny(resp["manifest"])
if manifestB64 == "" {
return nil, errors.New("tidal video manifest missing")
}
@@ -415,7 +620,7 @@ func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*pro
if !ok || len(urls) == 0 {
return nil, errors.New("tidal video manifest urls missing")
}
masterURL := stringify(urls[0])
masterURL := jsonutil.StringFromAny(urls[0])
if masterURL == "" {
return nil, errors.New("tidal video master url missing")
}
@@ -446,7 +651,7 @@ func (c *Client) GetVideoDownloadable(ctx context.Context, videoID string) (*pro
}
func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadable {
manifestB64 := stringify(resp["manifest"])
manifestB64 := jsonutil.StringFromAny(resp["manifest"])
if manifestB64 == "" {
return nil
}
@@ -462,16 +667,132 @@ func downloadableFromPlaybackManifest(resp map[string]any) *provider.Downloadabl
if !ok || len(urls) == 0 {
return nil
}
streamURL := stringify(urls[0])
streamURL := jsonutil.StringFromAny(urls[0])
if streamURL == "" {
return nil
}
codec := strings.ToLower(stringify(manifest["codecs"]))
codec := strings.ToLower(jsonutil.StringFromAny(manifest["codecs"]))
ext := "m4a"
if strings.Contains(codec, "flac") {
ext = "flac"
} else if strings.Contains(codec, "ec-3") || strings.Contains(codec, "eac3") || strings.Contains(codec, "joc") || strings.Contains(codec, "atmos") {
ext = "mka"
}
return &provider.Downloadable{URL: streamURL, Extension: ext, Source: "tidal"}
profile := tidalAudioProfileFromCodec(codec)
if profile.Container == "" {
profile = tidalAudioProfileFromExtension(ext)
}
audioQuality := strings.ToUpper(strings.TrimSpace(jsonutil.StringFromAny(resp["audioQuality"])))
if audioQuality == "" {
audioQuality = strings.ToUpper(strings.TrimSpace(jsonutil.StringFromAny(manifest["audioQuality"])))
}
if audioQuality != "" {
profile = applyTidalAudioQuality(profile, audioQuality)
}
if strings.Contains(strings.ToUpper(jsonutil.StringFromAny(resp["audioMode"])), "ATMOS") {
profile = tidalAtmosAudioProfile()
}
return &provider.Downloadable{URL: streamURL, Extension: ext, Source: "tidal", Audio: profile}
}
func tidalAudioProfileFromFormats(formats []any) provider.AudioProfile {
best := provider.AudioProfile{}
for _, raw := range formats {
f := strings.ToUpper(strings.TrimSpace(jsonutil.StringFromAny(raw)))
switch {
case strings.Contains(f, "EAC3") || strings.Contains(f, "JOC") || strings.Contains(f, "ATMOS"):
return tidalAtmosAudioProfile()
case strings.Contains(f, "FLAC_HIRES"):
best = provider.AudioProfile{Container: "FLAC", Codec: "FLAC", Quality: "HI_RES_LOSSLESS", BitDepth: 24}
case strings.Contains(f, "FLAC"):
if best.Container == "" {
best = provider.AudioProfile{Container: "FLAC", Codec: "FLAC", Quality: "LOSSLESS", BitDepth: 16, SamplingRate: "44.1"}
}
case strings.Contains(f, "AACLC"):
if best.Container == "" {
best = provider.AudioProfile{Container: "M4A", Codec: "AACLC", Quality: "HIGH", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 320}
}
case strings.Contains(f, "HEAAC"):
if best.Container == "" {
best = provider.AudioProfile{Container: "M4A", Codec: "HEAACV1", Quality: "LOW", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 96}
}
}
}
return best
}
func tidalAudioProfileFromCodec(codec string) provider.AudioProfile {
c := strings.ToLower(strings.TrimSpace(codec))
switch {
case strings.Contains(c, "ec-3") || strings.Contains(c, "eac3") || strings.Contains(c, "joc") || strings.Contains(c, "atmos"):
return tidalAtmosAudioProfile()
case strings.Contains(c, "flac"):
return provider.AudioProfile{Container: "FLAC", Codec: "FLAC", Quality: "LOSSLESS", BitDepth: 16, SamplingRate: "44.1"}
case strings.Contains(c, "mp4a.40.5") || strings.Contains(c, "mp4a.40.29"):
return provider.AudioProfile{Container: "M4A", Codec: "HEAACV1", Quality: "LOW", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 96}
case strings.Contains(c, "mp4a") || strings.Contains(c, "aac"):
return provider.AudioProfile{Container: "M4A", Codec: "AACLC", Quality: "HIGH", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 320}
default:
return provider.AudioProfile{}
}
}
func tidalAtmosAudioProfile() provider.AudioProfile {
return provider.AudioProfile{Container: "MKA", Codec: "EAC3_JOC", Quality: "ATMOS", BitDepth: 32, SamplingRate: "48"}
}
func tidalAudioProfileFromExtension(ext string) provider.AudioProfile {
switch strings.ToLower(strings.TrimSpace(ext)) {
case "flac":
return provider.AudioProfile{Container: "FLAC", Codec: "FLAC", Quality: "LOSSLESS", BitDepth: 16, SamplingRate: "44.1"}
case "mka":
return tidalAtmosAudioProfile()
case "m4a":
return provider.AudioProfile{Container: "M4A", Codec: "AACLC", Quality: "HIGH", BitDepth: 16, SamplingRate: "44.1", BitrateKbps: 320}
default:
container := strings.ToUpper(strings.TrimSpace(ext))
if container == "" {
container = "M4A"
}
return provider.AudioProfile{Container: container, Codec: container}
}
}
func applyTidalAudioQuality(profile provider.AudioProfile, audioQuality string) provider.AudioProfile {
aq := strings.ToUpper(strings.TrimSpace(audioQuality))
if aq == "" {
return profile
}
profile.Quality = aq
switch aq {
case "HI_RES", "HI_RES_LOSSLESS":
if strings.EqualFold(profile.Container, "FLAC") {
if profile.BitDepth < 24 {
profile.BitDepth = 24
}
}
case "LOSSLESS":
if strings.EqualFold(profile.Container, "FLAC") {
if profile.BitDepth == 0 {
profile.BitDepth = 16
}
if profile.SamplingRate == "" {
profile.SamplingRate = "44.1"
}
}
case "HIGH":
if strings.EqualFold(profile.Container, "M4A") && profile.BitrateKbps == 0 {
profile.BitrateKbps = 320
}
case "LOW":
if strings.EqualFold(profile.Container, "M4A") {
profile.Codec = "HEAACV1"
if profile.BitrateKbps == 0 {
profile.BitrateKbps = 96
}
}
}
return profile
}
func bestHLSVariantURL(masterURL, playlist string) string {
@@ -509,11 +830,111 @@ func resolvePlaylistURL(baseRaw, refRaw string) string {
return baseURL.ResolveReference(refURL).String()
}
func (c *Client) apiRequest(ctx context.Context, path string, params url.Values, base string) (map[string]any, int, error) {
if err := c.limiter.Wait(ctx); err != nil {
return nil, 0, err
}
func shouldRetryStatus(status int) bool {
return status == http.StatusTooManyRequests || status >= http.StatusInternalServerError
}
func retryDelay(retryAfter string, attempt int) time.Duration {
retryAfter = strings.TrimSpace(retryAfter)
if retryAfter != "" {
if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(retryAfter); err == nil {
if delay := time.Until(when); delay > 0 {
return delay
}
return 0
}
}
return time.Duration(attempt+1) * 500 * time.Millisecond
}
func waitRetry(ctx context.Context, delay time.Duration) error {
if delay <= 0 {
return nil
}
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func parseAPIResponseBody(body []byte, status int) (map[string]any, error) {
out := map[string]any{}
if len(body) == 0 {
return out, nil
}
if err := json.Unmarshal(body, &out); err != nil {
if status < http.StatusOK || status >= http.StatusMultipleChoices {
if raw := strings.TrimSpace(string(body)); raw != "" {
out["raw"] = raw
}
return out, nil
}
return nil, err
}
return out, nil
}
func readAPIResponse(resp *http.Response) (map[string]any, int, string, error) {
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, resp.StatusCode, resp.Header.Get("Retry-After"), err
}
parsed, err := parseAPIResponseBody(body, resp.StatusCode)
return parsed, resp.StatusCode, resp.Header.Get("Retry-After"), err
}
func (c *Client) doJSONWithRetry(ctx context.Context, newRequest func() (*http.Request, error)) (map[string]any, int, error) {
var lastStatus int
for attempt := 0; attempt < tidalRequestAttempts; attempt++ {
if err := c.limiter.Wait(ctx); err != nil {
return nil, 0, err
}
req, err := newRequest()
if err != nil {
return nil, 0, err
}
resp, err := c.http.Do(req)
if err != nil {
if attempt+1 < tidalRequestAttempts {
if waitErr := waitRetry(ctx, retryDelay("", attempt)); waitErr != nil {
return nil, 0, waitErr
}
continue
}
return nil, 0, err
}
parsed, status, retryAfter, err := readAPIResponse(resp)
lastStatus = status
if err != nil {
if attempt+1 < tidalRequestAttempts {
if waitErr := waitRetry(ctx, retryDelay(retryAfter, attempt)); waitErr != nil {
return nil, 0, waitErr
}
continue
}
return nil, status, err
}
if shouldRetryStatus(status) && attempt+1 < tidalRequestAttempts {
if waitErr := waitRetry(ctx, retryDelay(retryAfter, attempt)); waitErr != nil {
return nil, 0, waitErr
}
continue
}
return parsed, status, nil
}
return map[string]any{}, lastStatus, nil
}
func (c *Client) apiRequest(ctx context.Context, path string, params url.Values, base string) (map[string]any, int, error) {
if params == nil {
params = url.Values{}
}
@@ -529,89 +950,40 @@ func (c *Client) apiRequest(ctx context.Context, path string, params url.Values,
reqURL += "?" + params.Encode()
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
if err != nil {
return nil, 0, err
}
req.Header.Set("Authorization", "Bearer "+c.cfg.Session.Tidal.AccessToken)
req.Header.Set("User-Agent", "streamrip-go/0.1")
resp, err := c.http.Do(req)
if err != nil {
return nil, 0, err
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, resp.StatusCode, err
}
parsed := map[string]any{}
if len(body) > 0 {
if err = json.Unmarshal(body, &parsed); err != nil {
return nil, resp.StatusCode, err
return c.doJSONWithRetry(ctx, func() (*http.Request, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
if err != nil {
return nil, err
}
}
return parsed, resp.StatusCode, nil
req.Header.Set("Authorization", "Bearer "+c.cfg.Session.Tidal.AccessToken)
req.Header.Set("User-Agent", "streamrip-go/0.1")
return req, nil
})
}
func (c *Client) apiPost(ctx context.Context, endpoint string, form url.Values, basicAuth bool) (map[string]any, int, error) {
if err := c.limiter.Wait(ctx); err != nil {
return nil, 0, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewBufferString(form.Encode()))
if err != nil {
return nil, 0, err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("User-Agent", "streamrip-go/0.1")
if basicAuth {
auth := base64.StdEncoding.EncodeToString([]byte(clientID + ":" + clientSec))
req.Header.Set("Authorization", "Basic "+auth)
}
resp, err := c.http.Do(req)
if err != nil {
return nil, 0, err
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, resp.StatusCode, err
}
out := map[string]any{}
if len(body) > 0 {
if err = json.Unmarshal(body, &out); err != nil {
return nil, resp.StatusCode, err
return c.doJSONWithRetry(ctx, func() (*http.Request, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewBufferString(form.Encode()))
if err != nil {
return nil, err
}
}
return out, resp.StatusCode, nil
}
func stringify(v any) string {
switch t := v.(type) {
case string:
return t
case int:
return strconv.Itoa(t)
case int64:
return strconv.FormatInt(t, 10)
case float64:
return strconv.FormatFloat(t, 'f', -1, 64)
default:
return ""
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("User-Agent", "streamrip-go/0.1")
if basicAuth {
auth := base64.StdEncoding.EncodeToString([]byte(c.appClientID() + ":" + c.appClientSecret()))
req.Header.Set("Authorization", "Basic "+auth)
}
return req, nil
})
}
func enrichTidalImage(meta map[string]any) {
if _, ok := meta["image"].(map[string]any); ok {
return
}
cover := stringify(meta["cover"])
cover := jsonutil.StringFromAny(meta["cover"])
if cover == "" {
cover = stringify(meta["squareImage"])
cover = jsonutil.StringFromAny(meta["squareImage"])
}
if cover == "" {
return
@@ -631,18 +1003,19 @@ func tidalImageMap(cover string) map[string]any {
}
}
func intFromAny(v any) int {
switch t := v.(type) {
case int:
return t
case int64:
return int(t)
case float64:
return int(t)
case string:
i, _ := strconv.Atoi(t)
return i
default:
return 0
// appClientID and appClientSecret read the vendor OAuth app credentials from
// config, falling back to the built-in defaults. Config is the source of truth
// so a rotation can be fixed without a rebuild.
func (c *Client) appClientID() string {
if v := strings.TrimSpace(c.cfg.Session.Tidal.AppClientID); v != "" {
return v
}
return config.DefaultTidalAppClientID
}
func (c *Client) appClientSecret() string {
if v := strings.TrimSpace(c.cfg.Session.Tidal.AppClientSecret); v != "" {
return v
}
return config.DefaultTidalAppClientSecret
}
+367 -3
View File
@@ -6,13 +6,17 @@ import (
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"reflect"
"strconv"
"testing"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
)
func TestLoginMissingToken(t *testing.T) {
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = ""
c := New(&config.Config{File: cfgData, Session: cfgData})
err := c.Login(context.Background())
@@ -34,7 +38,7 @@ func TestSearch(t *testing.T) {
}))
defer ts.Close()
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -70,7 +74,7 @@ func TestGetVideoDownloadable(t *testing.T) {
}))
defer server.Close()
cfgData := config.DefaultConfigData()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
@@ -98,3 +102,363 @@ func TestBestHLSVariantURLFallsBackToMaster(t *testing.T) {
t.Fatalf("url = %q, want %q", got, master)
}
}
func TestGetMetadataArtistPaginatesAlbums(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/sessions":
_ = json.NewEncoder(w).Encode(map[string]any{"countryCode": "US", "userId": 123})
case "/v1/artists/9":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 9, "name": "Artist X"})
case "/v1/artists/9/albums":
offset, _ := strconv.Atoi(r.URL.Query().Get("offset"))
filter := r.URL.Query().Get("filter")
if filter == "" {
if offset == 0 {
items := make([]any, 0, 100)
for i := 0; i < 100; i++ {
items = append(items, map[string]any{"id": i + 1})
}
_ = json.NewEncoder(w).Encode(map[string]any{"items": items})
return
}
if offset == 100 {
_ = json.NewEncoder(w).Encode(map[string]any{"items": []any{map[string]any{"id": 101}}})
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"items": []any{}})
return
}
if filter == "EPSANDSINGLES" {
if offset == 0 {
_ = json.NewEncoder(w).Encode(map[string]any{"items": []any{map[string]any{"id": 101}, map[string]any{"id": 102}}})
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"items": []any{}})
return
}
w.WriteHeader(http.StatusBadRequest)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.baseURL = ts.URL + "/v1"
if err := c.Login(context.Background()); err != nil {
t.Fatalf("login err = %v", err)
}
meta, err := c.GetMetadata(context.Background(), "9", "artist")
if err != nil {
t.Fatalf("GetMetadata() err = %v", err)
}
albumsObj, _ := meta["albums"].(map[string]any)
items, _ := albumsObj["items"].([]map[string]any)
if len(items) != 102 {
t.Fatalf("albums len = %d, want 102", len(items))
}
}
func TestGetMetadataTrackAddsLyricsAndSyncedLyrics(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/tracks/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "title": "Song", "album": map[string]any{"id": 10, "title": "Album"}})
case "/v1/tracks/42/lyrics":
q := r.URL.Query()
if q.Get("deviceType") != "PHONE" || q.Get("locale") != "en_US" || q.Get("platform") != "ANDROID" || q.Get("countryCode") != "MY" {
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{"error": "bad query"})
return
}
_ = json.NewEncoder(w).Encode(map[string]any{
"lyrics": "plain lyrics line",
"subtitles": "[00:00.00]plain lyrics line",
})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "MY"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.baseURL = ts.URL + "/v1"
c.lyricsAPI = ts.URL + "/v1"
meta, err := c.GetMetadata(context.Background(), "42", "track")
if err != nil {
t.Fatalf("GetMetadata() err = %v", err)
}
if got := jsonutil.StringFromAny(meta["lyrics"]); got != "plain lyrics line" {
t.Fatalf("lyrics = %q, want plain lyrics line", got)
}
if got := jsonutil.StringFromAny(meta["lyrics_synced"]); got != "[00:00.00]plain lyrics line" {
t.Fatalf("lyrics_synced = %q, want synced lrc", got)
}
}
func TestGetMetadataTrackIgnoresLyricsEndpointFailure(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/tracks/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "title": "Song"})
case "/v1/tracks/42/lyrics":
w.WriteHeader(http.StatusNotFound)
_ = json.NewEncoder(w).Encode(map[string]any{"error": "not found"})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.baseURL = ts.URL + "/v1"
c.lyricsAPI = ts.URL + "/v1"
meta, err := c.GetMetadata(context.Background(), "42", "track")
if err != nil {
t.Fatalf("GetMetadata() err = %v", err)
}
if _, ok := meta["lyrics"]; ok {
t.Fatalf("did not expect lyrics when endpoint fails")
}
if _, ok := meta["lyrics_synced"]; ok {
t.Fatalf("did not expect lyrics_synced when endpoint fails")
}
}
func TestAPIRequestRetriesTooManyRequests(t *testing.T) {
calls := 0
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/tracks/42" {
w.WriteHeader(http.StatusNotFound)
return
}
calls++
if calls == 1 {
w.Header().Set("Retry-After", "0")
w.WriteHeader(http.StatusTooManyRequests)
_, _ = w.Write([]byte("slow down"))
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "title": "Song"})
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Downloads.RequestsPerMinute = 0
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.baseURL = ts.URL + "/v1"
resp, status, err := c.apiRequest(context.Background(), "tracks/42", nil, c.baseURL)
if err != nil {
t.Fatalf("apiRequest() err = %v", err)
}
if status != http.StatusOK {
t.Fatalf("status = %d, want %d", status, http.StatusOK)
}
if calls != 2 {
t.Fatalf("calls = %d, want 2", calls)
}
if jsonutil.StringFromAny(resp["title"]) != "Song" {
t.Fatalf("title = %q, want Song", jsonutil.StringFromAny(resp["title"]))
}
}
func TestAPIPostRetriesTooManyRequests(t *testing.T) {
calls := 0
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/token" {
w.WriteHeader(http.StatusNotFound)
return
}
calls++
if calls == 1 {
w.Header().Set("Retry-After", "0")
w.WriteHeader(http.StatusTooManyRequests)
_, _ = w.Write([]byte("slow down"))
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"access_token": "fresh-token"})
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Downloads.RequestsPerMinute = 0
c := New(&config.Config{File: cfgData, Session: cfgData})
resp, status, err := c.apiPost(context.Background(), ts.URL+"/token", url.Values{"grant_type": []string{"refresh_token"}}, false)
if err != nil {
t.Fatalf("apiPost() err = %v", err)
}
if status != http.StatusOK {
t.Fatalf("status = %d, want %d", status, http.StatusOK)
}
if calls != 2 {
t.Fatalf("calls = %d, want 2", calls)
}
if jsonutil.StringFromAny(resp["access_token"]) != "fresh-token" {
t.Fatalf("access_token = %q, want fresh-token", jsonutil.StringFromAny(resp["access_token"]))
}
}
func TestGetDownloadablePrefersAtmosWhenEnabled(t *testing.T) {
var calls []string
allImmersive := true
var ts *httptest.Server
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/tracks/42":
_ = json.NewEncoder(w).Encode(map[string]any{"id": 42, "audioModes": []any{"DOLBY_ATMOS", "STEREO"}, "mediaMetadata": map[string]any{"tags": []any{"LOSSLESS", "DOLBY_ATMOS"}}})
case "/v1/tracks/42/playbackinfopostpaywall":
if r.URL.Query().Get("immersiveaudio") != "true" {
allImmersive = false
}
aq := r.URL.Query().Get("audioquality")
calls = append(calls, aq)
manifest := map[string]any{"urls": []string{ts.URL + "/stereo.m3u8"}, "codecs": "flac"}
if aq == "HI_RES" {
manifest = map[string]any{"urls": []string{ts.URL + "/atmos.m3u8"}, "codecs": "ec-3", "audioMode": "DOLBY_ATMOS"}
}
b, _ := json.Marshal(manifest)
_ = json.NewEncoder(w).Encode(map[string]any{"manifest": base64.StdEncoding.EncodeToString(b), "audioMode": manifest["audioMode"]})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
cfgData.Tidal.PreferAtmos = true
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.baseURL = ts.URL + "/v1"
d, err := c.GetDownloadable(context.Background(), "42", 3)
if err != nil {
t.Fatalf("GetDownloadable() err = %v", err)
}
if d.URL != ts.URL+"/atmos.m3u8" {
t.Fatalf("url = %q, want %q", d.URL, ts.URL+"/atmos.m3u8")
}
if d.Extension != "mka" {
t.Fatalf("extension = %q, want mka", d.Extension)
}
if len(calls) < 2 || calls[0] != "HI_RES_LOSSLESS" || calls[1] != "HI_RES" {
t.Fatalf("unexpected audioquality call order: %+v", calls)
}
if !allImmersive {
t.Fatalf("expected immersiveaudio=true on Atmos probing calls")
}
}
func TestPlaybackLooksAtmosFromManifest(t *testing.T) {
manifest := map[string]any{"urls": []string{"https://cdn.example/stream.m3u8"}, "codecs": "ec-3"}
b, _ := json.Marshal(manifest)
resp := map[string]any{"manifest": base64.StdEncoding.EncodeToString(b)}
if !playbackLooksAtmos(resp) {
t.Fatalf("expected atmos detection from manifest codec")
}
}
func TestFormatsForQuality(t *testing.T) {
tests := []struct {
name string
q int
atmos bool
wants []string
}{
{name: "low", q: 0, wants: []string{"HEAACV1"}},
{name: "high", q: 1, wants: []string{"HEAACV1", "AACLC"}},
{name: "lossless", q: 2, wants: []string{"HEAACV1", "AACLC", "FLAC"}},
{name: "hires", q: 4, wants: []string{"HEAACV1", "AACLC", "FLAC", "FLAC_HIRES"}},
{name: "atmos adds eac3", q: 2, atmos: true, wants: []string{"HEAACV1", "AACLC", "FLAC", "EAC3_JOC"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := formatsForQuality(tc.q, tc.atmos)
if !reflect.DeepEqual(got, tc.wants) {
t.Fatalf("formatsForQuality(%d, atmos=%v) = %#v, want %#v", tc.q, tc.atmos, got, tc.wants)
}
})
}
}
func TestGetDownloadableLosslessUsesTrackManifestWhenPlaybackIsAAC(t *testing.T) {
var gotFormats string
var ts *httptest.Server
ts = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/tracks/42/playbackinfopostpaywall":
manifest := map[string]any{"urls": []string{ts.URL + "/aac.m3u8"}, "codecs": "mp4a.40.2"}
b, _ := json.Marshal(manifest)
_ = json.NewEncoder(w).Encode(map[string]any{"manifest": base64.StdEncoding.EncodeToString(b)})
case "/v2/trackManifests/42":
gotFormats = r.URL.Query().Get("formats")
_ = json.NewEncoder(w).Encode(map[string]any{
"data": map[string]any{
"attributes": map[string]any{
"uri": ts.URL + "/song.flac",
"formats": []any{"FLAC"},
},
},
})
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer ts.Close()
cfgData := testConfigData()
cfgData.Tidal.AccessToken = "token"
cfgData.Tidal.CountryCode = "US"
c := New(&config.Config{File: cfgData, Session: cfgData})
c.loggedIn = true
c.baseURL = ts.URL + "/v1"
c.openAPI = ts.URL + "/v2"
d, err := c.GetDownloadable(context.Background(), "42", 2)
if err != nil {
t.Fatalf("GetDownloadable() err = %v", err)
}
if gotFormats != "HEAACV1,AACLC,FLAC" {
t.Fatalf("formats query = %q, want %q", gotFormats, "HEAACV1,AACLC,FLAC")
}
if d.URL != ts.URL+"/song.flac" {
t.Fatalf("url = %q, want %q", d.URL, ts.URL+"/song.flac")
}
if d.Extension != "flac" {
t.Fatalf("extension = %q, want flac", d.Extension)
}
if d.Audio.Container != "FLAC" || d.Audio.Quality != "LOSSLESS" {
t.Fatalf("unexpected audio profile: %+v", d.Audio)
}
}
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
File diff suppressed because it is too large. Load diff
+194
View File
@@ -0,0 +1,194 @@
package yandex
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"streamrip-go/internal/config"
"streamrip-go/internal/jsonutil"
)
func TestYandexDownloadSignMatchesCapturedFormat(t *testing.T) {
sign, ts := yandexDownloadSign(config.DefaultYandexSignKey, "32038184", "lossless", []string{"flac", "aac", "he-aac", "mp3", "flac-mp4", "aac-mp4", "he-aac-mp4"}, "raw")
if ts <= 0 {
t.Fatalf("timestamp = %d", ts)
}
if strings.TrimSpace(sign) == "" {
t.Fatalf("decoded sign is empty")
}
if strings.Contains(sign, "=") {
t.Fatalf("sign unexpectedly contains base64 padding: %q", sign)
}
if strings.Contains(sign, " ") {
t.Fatalf("sign unexpectedly contains space: %q", sign)
}
}
func TestGetDownloadableUsesModernGetFileInfo(t *testing.T) {
var gotPath string
var gotQuery url.Values
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
gotQuery = r.URL.Query()
if r.URL.Path == "/account/about" {
_ = json.NewEncoder(w).Encode(map[string]any{"result": map[string]any{"uid": "123"}})
return
}
if r.URL.Path != "/get-file-info" {
w.WriteHeader(http.StatusNotFound)
return
}
_ = json.NewEncoder(w).Encode(map[string]any{
"result": map[string]any{
"downloadInfo": map[string]any{
"trackId": "32038184",
"quality": "lossless",
"codec": "flac-mp4",
"transport": "encraw",
"key": "00112233445566778899aabbccddeeff",
"bitrate": 0,
"url": "https://strm.example/music-v2/crypt/x/flac-mp4",
},
},
})
}))
defer ts.Close()
d := testConfigData()
d.Downloads.RequestsPerMinute = 0
d.Yandex.AccessToken = "token"
c := New(&config.Config{File: d, Session: d})
c.baseURL = ts.URL
c.loggedIn = true
dl, err := c.GetDownloadable(context.Background(), "32038184:1683700", 2)
if err != nil {
t.Fatalf("GetDownloadable() error = %v", err)
}
if gotPath != "/get-file-info" {
t.Fatalf("path = %q, want /get-file-info", gotPath)
}
if gotQuery.Get("trackId") != "32038184" {
t.Fatalf("trackId = %q, want 32038184", gotQuery.Get("trackId"))
}
if gotQuery.Get("quality") != "lossless" {
t.Fatalf("quality = %q, want lossless", gotQuery.Get("quality"))
}
if gotQuery.Get("transports") != "encraw" {
t.Fatalf("transports = %q, want encraw", gotQuery.Get("transports"))
}
if dl.Extension != "m4a" {
t.Fatalf("extension = %q, want m4a", dl.Extension)
}
if dl.Audio.Codec != "FLAC" || dl.Audio.Quality != "LOSSLESS" {
t.Fatalf("unexpected audio profile: %+v", dl.Audio)
}
if dl.TrackID != "32038184" {
t.Fatalf("track id = %q, want 32038184", dl.TrackID)
}
if dl.Cipher != "AES_CTR" || dl.Key == "" {
t.Fatalf("expected yandex cipher metadata, got cipher=%q key=%q", dl.Cipher, dl.Key)
}
}
func TestGetMetadataTrackUsesModernTracksEndpoint(t *testing.T) {
var gotMethod string
var gotPath string
var gotBody string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotMethod = r.Method
gotPath = r.URL.Path
if r.URL.Path != "/tracks" {
w.WriteHeader(http.StatusNotFound)
return
}
body, _ := io.ReadAll(r.Body)
gotBody = string(body)
_ = json.NewEncoder(w).Encode(map[string]any{
"result": []map[string]any{{
"id": "9442712",
"realId": "9442712",
"title": "Nightcall",
"artists": []map[string]any{{"id": "1433871", "name": "Kavinsky"}, {"id": "42", "name": "Lovefoxxx"}},
"albums": []map[string]any{{
"id": "1000856",
"title": "OutRun",
"releaseDate": "2013-02-25T00:00:00+04:00",
"trackCount": 13,
"artists": []map[string]any{{"id": "1433871", "name": "Kavinsky"}, {"id": "42", "name": "Lovefoxxx"}},
"trackPosition": map[string]any{
"index": 0,
"volume": 1,
},
}},
}},
})
}))
defer ts.Close()
d := testConfigData()
d.Downloads.RequestsPerMinute = 0
d.Yandex.AccessToken = "token"
c := New(&config.Config{File: d, Session: d})
c.baseURL = ts.URL
c.loggedIn = true
meta, err := c.GetMetadata(context.Background(), "9442712:1000856", "track")
if err != nil {
t.Fatalf("GetMetadata() error = %v", err)
}
if gotMethod != http.MethodPost || gotPath != "/tracks" {
t.Fatalf("unexpected request: %s %s", gotMethod, gotPath)
}
if !strings.Contains(gotBody, "trackIds=9442712%3A1000856") {
t.Fatalf("body = %q", gotBody)
}
if meta["id"] != "9442712:1000856" {
t.Fatalf("id = %v", meta["id"])
}
if album, _ := meta["album"].(map[string]any); jsonutil.StringFromAny(album["title"]) != "OutRun" {
t.Fatalf("unexpected album: %+v", album)
}
if artist := jsonutil.NestedString(meta, "artist", "name"); artist != "Kavinsky & Lovefoxxx" {
t.Fatalf("artist = %q", artist)
}
if albumArtist := jsonutil.NestedString(meta, "album", "artist", "name"); albumArtist != "Kavinsky & Lovefoxxx" {
t.Fatalf("album artist = %q", albumArtist)
}
artists, _ := meta["artist_names"].([]string)
if strings.Join(artists, ";") != "Kavinsky;Lovefoxxx" {
t.Fatalf("artist_names = %#v", artists)
}
}
func TestLegacyDirectURLBuildsPlayableMP3URL(t *testing.T) {
url, err := legacyDirectURL(config.DefaultYandexLegacyMP3Salt, &legacyDownloadInfoXML{
Host: "example.test",
Path: "/abc123",
TS: "1234567890",
S: "tailxyz",
})
if err != nil {
t.Fatalf("legacyDirectURL() error = %v", err)
}
want := "https://example.test/get-mp3/248c1c6ff5daf481560d3bd9f24e8058/1234567890/abc123"
if url != want {
t.Fatalf("legacyDirectURL() = %q, want %q", url, want)
}
}
// testConfigData returns defaults with request throttling disabled.
//
// At the default 60 requests/minute the rate limiter sleeps 1s per request,
// which accounted for most of this suite's runtime (deezer alone was 31s).
func testConfigData() config.ConfigData {
d := config.DefaultConfigData()
d.Downloads.RequestsPerMinute = -1
return d
}
+49 -14
View File
@@ -1,3 +1,5 @@
// Package ratelimit provides the API request throttle shared by the provider
// clients.
package ratelimit
import (
@@ -6,20 +8,54 @@ import (
"time"
)
// Limiter is a token bucket: it enforces a sustained rate of
// requestsPerMinute while allowing up to burst requests to proceed
// immediately when budget has accrued.
//
// It used to space every request exactly interval apart, with no burst. That
// made max_connections meaningless for anything that goes through the API:
// with the default 60 requests/minute, six download workers each needing a
// metadata call and a stream-URL call could only start one track per two
// seconds, so an album appeared to download serially even though the transfers
// themselves were fast.
type Limiter struct {
interval time.Duration
burst int
mu sync.Mutex
mu sync.Mutex
// next is the time the next token becomes available. It is allowed to
// trail now by up to (burst-1)*interval, which is what accrues burst
// budget while no requests are being made.
next time.Time
}
func New(requestsPerMinute int) *Limiter {
// New returns a limiter allowing requestsPerMinute sustained, with burst
// requests permitted back-to-back. A non-positive requestsPerMinute disables
// throttling entirely.
//
// burst is clamped to at least 1 and to at most one minute's worth of
// requests, so a very low rate cannot be bypassed by a large burst.
func New(requestsPerMinute int, burst int) *Limiter {
if requestsPerMinute <= 0 {
return &Limiter{}
}
return &Limiter{interval: time.Minute / time.Duration(requestsPerMinute)}
if burst < 1 {
burst = 1
}
if burst > requestsPerMinute {
burst = requestsPerMinute
}
return &Limiter{
interval: time.Minute / time.Duration(requestsPerMinute),
burst: burst,
}
}
// Wait blocks until the caller may issue a request, or until ctx is done.
//
// Budget is reserved at call time, so a cancelled Wait still consumes its
// slot. That keeps the sustained rate honest under cancellation, at the cost
// of briefly under-using the budget.
func (l *Limiter) Wait(ctx context.Context) error {
if l.interval <= 0 {
return nil
@@ -27,22 +63,21 @@ func (l *Limiter) Wait(ctx context.Context) error {
l.mu.Lock()
now := time.Now()
if l.next.IsZero() {
l.next = now.Add(l.interval)
l.mu.Unlock()
return nil
// Accrue budget for idle time, capped at burst tokens.
earliest := now.Add(-time.Duration(l.burst-1) * l.interval)
if l.next.Before(earliest) {
l.next = earliest
}
wake := l.next
if now.After(wake) {
l.next = now.Add(l.interval)
l.mu.Unlock()
return nil
}
l.next = l.next.Add(l.interval)
l.mu.Unlock()
timer := time.NewTimer(time.Until(wake))
delay := time.Until(wake)
if delay <= 0 {
return nil
}
timer := time.NewTimer(delay)
defer timer.Stop()
select {
+92 -4
View File
@@ -2,19 +2,25 @@ package ratelimit
import (
"context"
"sync"
"testing"
"time"
)
func TestLimiterDisabled(t *testing.T) {
l := New(-1)
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("Wait() error = %v", err)
for _, rpm := range []int{-1, 0} {
l := New(rpm, 6)
for i := 0; i < 10; i++ {
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("New(%d).Wait() error = %v", rpm, err)
}
}
}
}
func TestLimiterContextCancel(t *testing.T) {
l := New(1)
// burst 1 means strict spacing, so the second request must block.
l := New(1, 1)
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("first Wait() error = %v", err)
}
@@ -26,3 +32,85 @@ func TestLimiterContextCancel(t *testing.T) {
t.Fatalf("expected context error")
}
}
// TestLimiterAllowsBurst is the regression test for album rips appearing
// serial: with the old strict-spacing limiter, max_connections workers could
// not get their API calls through concurrently, so only one track downloaded
// at a time.
func TestLimiterAllowsBurst(t *testing.T) {
const burst = 6
l := New(60, burst) // the shipped defaults
start := time.Now()
for i := 0; i < burst; i++ {
if err := l.Wait(context.Background()); err != nil {
t.Fatalf("Wait() %d error = %v", i, err)
}
}
elapsed := time.Since(start)
// 60 rpm is one token per second; the burst must not have waited.
if elapsed > 200*time.Millisecond {
t.Fatalf("%d burst requests took %v, want them to proceed immediately", burst, elapsed)
}
}
// TestLimiterEnforcesSustainedRate checks the burst does not become a way to
// exceed the configured rate: once the bucket is drained, requests are spaced
// by the interval again.
func TestLimiterEnforcesSustainedRate(t *testing.T) {
// 6000 rpm = 10ms per token, so this stays fast.
const burst = 3
l := New(6000, burst)
ctx := context.Background()
for i := 0; i < burst; i++ {
if err := l.Wait(ctx); err != nil {
t.Fatal(err)
}
}
// The bucket is drained; the next two must each wait about one interval.
start := time.Now()
for i := 0; i < 2; i++ {
if err := l.Wait(ctx); err != nil {
t.Fatal(err)
}
}
elapsed := time.Since(start)
if elapsed < 15*time.Millisecond {
t.Fatalf("post-burst requests took %v, want them throttled to ~10ms each", elapsed)
}
}
// TestLimiterBurstCappedByRate stops a large max_connections from letting a
// deliberately tiny rate through in one go.
func TestLimiterBurstCappedByRate(t *testing.T) {
l := New(2, 64)
if l.burst != 2 {
t.Fatalf("burst = %d, want it clamped to the rate (2)", l.burst)
}
}
func TestLimiterConcurrentCallersShareTheRate(t *testing.T) {
const burst = 8
l := New(6000, burst)
var wg sync.WaitGroup
start := time.Now()
for i := 0; i < burst; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := l.Wait(context.Background()); err != nil {
t.Errorf("Wait() error = %v", err)
}
}()
}
wg.Wait()
// All within the burst, so none should have blocked meaningfully.
if elapsed := time.Since(start); elapsed > 200*time.Millisecond {
t.Fatalf("%d concurrent burst requests took %v", burst, elapsed)
}
}
+164 -7
View File
@@ -3,23 +3,40 @@ package store
import (
"context"
"database/sql"
"fmt"
"io"
"net/url"
"os"
"strings"
"sync"
"time"
_ "modernc.org/sqlite"
)
type SQLite struct {
db *sql.DB
mu sync.Mutex
db *sql.DB
path string
mu sync.Mutex
// hasLegacyRows is set when the database was migrated from the pre-source
// schema. Those rows have no source, so IsDownloaded has to consider them
// separately or the entire pre-migration library looks undownloaded.
hasLegacyRows bool
legacyWarned bool
}
func NewSQLite(path string) (*SQLite, error) {
db, err := sql.Open("sqlite", path)
db, err := sql.Open("sqlite", sqliteDSN(path))
if err != nil {
return nil, err
}
// One writer at a time. SQLite allows a single writer anyway, and the
// mutex below already serializes this process; capping the pool keeps the
// driver from opening connections that can only contend with each other.
db.SetMaxOpenConns(1)
s := &SQLite{db: db}
s := &SQLite{db: db, path: path}
if err = s.init(); err != nil {
_ = db.Close()
return nil, err
@@ -27,6 +44,70 @@ func NewSQLite(path string) (*SQLite, error) {
return s, nil
}
// sqliteDSN builds a file: URI with the pragmas this app needs.
//
// WAL lets a concurrent reader (for example `rip db browse`) run while a rip
// is writing, and busy_timeout makes a contended write wait instead of failing
// immediately with SQLITE_BUSY. The path is passed through url.URL so that a
// config path containing '?' or '#' is not parsed as query parameters.
func sqliteDSN(path string) string {
u := url.URL{Scheme: "file", Opaque: (&url.URL{Path: path}).EscapedPath()}
q := url.Values{}
q.Add("_pragma", "busy_timeout(10000)")
q.Add("_pragma", "journal_mode(WAL)")
q.Add("_pragma", "foreign_keys(on)")
return u.String() + "?" + q.Encode()
}
// schemaVersion is stored in PRAGMA user_version so later migrations can be
// ordered explicitly instead of being inferred from PRAGMA table_info.
const schemaVersion = 1
func (s *SQLite) userVersion() (int, error) {
var v int
if err := s.db.QueryRow(`PRAGMA user_version`).Scan(&v); err != nil {
return 0, err
}
return v, nil
}
func (s *SQLite) setUserVersion(v int) error {
_, err := s.db.Exec(fmt.Sprintf(`PRAGMA user_version = %d`, v))
return err
}
// backupFile copies the database aside before a destructive migration. A
// migration that loses the download history silently costs the user a full
// re-download, so keep a copy they can fall back to.
func (s *SQLite) backupFile() (string, error) {
if strings.TrimSpace(s.path) == "" {
return "", nil
}
src, err := os.Open(s.path)
if err != nil {
if os.IsNotExist(err) {
return "", nil
}
return "", err
}
defer func() { _ = src.Close() }()
dstPath := fmt.Sprintf("%s.bak-%d", s.path, time.Now().Unix())
dst, err := os.OpenFile(dstPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
if err != nil {
return "", err
}
if _, err = io.Copy(dst, src); err != nil {
_ = dst.Close()
_ = os.Remove(dstPath)
return "", err
}
if err = dst.Close(); err != nil {
return "", err
}
return dstPath, nil
}
func (s *SQLite) init() error {
if err := s.ensureDownloadsSchema(); err != nil {
return err
@@ -47,6 +128,20 @@ func (s *SQLite) init() error {
}
}
// A database migrated by an earlier version has sourceless rows but no
// in-memory flag, so detect them here rather than only at migration time.
if !s.hasLegacyRows {
if n, err := s.legacyRowCount(context.Background()); err == nil && n > 0 {
s.hasLegacyRows = true
}
}
if v, err := s.userVersion(); err == nil && v < schemaVersion {
if err = s.setUserVersion(schemaVersion); err != nil {
return err
}
}
return nil
}
@@ -94,6 +189,14 @@ func (s *SQLite) ensureDownloadsSchema() error {
return err
}
// Legacy schema: a downloads table with no source column. Back the file up
// first, because the rows below can only be carried over without a source
// and that is not fully reversible.
backupPath, backupErr := s.backupFile()
if backupErr != nil {
return fmt.Errorf("could not back up %s before migrating: %w", s.path, backupErr)
}
tx, err := s.db.Begin()
if err != nil {
return err
@@ -120,12 +223,27 @@ func (s *SQLite) ensureDownloadsSchema() error {
if _, err = tx.Exec(`INSERT OR IGNORE INTO downloads(source, id) SELECT '', id FROM downloads_legacy`); err != nil {
return err
}
if _, err = tx.Exec(`DROP TABLE downloads_legacy`); err != nil {
// downloads_legacy is deliberately kept. Dropping it used to make the
// migration unrecoverable, and IsDownloaded needs the sourceless rows to
// avoid re-downloading everything the user already has.
if err = tx.Commit(); err != nil {
return err
}
err = tx.Commit()
return err
s.hasLegacyRows = true
if backupPath != "" {
fmt.Fprintf(os.Stderr, "download database migrated to the source-aware schema; previous copy kept at %s\n", backupPath)
}
return nil
}
// legacyRowCount reports how many pre-migration rows exist. Rows carried over
// from the old schema have an empty source.
func (s *SQLite) legacyRowCount(ctx context.Context) (int, error) {
var n int
err := s.db.QueryRowContext(ctx, `SELECT COUNT(1) FROM downloads WHERE source = ''`).Scan(&n)
return n, err
}
func (s *SQLite) IsDownloaded(ctx context.Context, source, id string) (bool, error) {
@@ -137,6 +255,26 @@ func (s *SQLite) IsDownloaded(ctx context.Context, source, id string) (bool, err
if err != nil {
return false, err
}
if count > 0 {
return true, nil
}
// Rows migrated from the pre-source schema have an empty source, so an
// exact match on (source, id) misses them and the whole pre-migration
// library would look undownloaded. Matching on id alone can only ever
// over-match across sources, which costs a skipped re-download rather than
// a duplicated library.
if !s.hasLegacyRows {
return false, nil
}
err = s.db.QueryRowContext(ctx, `SELECT COUNT(1) FROM downloads WHERE source = '' AND id = ?`, id).Scan(&count)
if err != nil {
return false, err
}
if count > 0 && !s.legacyWarned {
s.legacyWarned = true
fmt.Fprintf(os.Stderr, "note: matching downloads recorded before sources were tracked; pass --force to re-download them\n")
}
return count > 0, nil
}
@@ -162,6 +300,25 @@ func (s *SQLite) MarkFailed(ctx context.Context, source, mediaType, id string) e
return err
}
// IsFailed reports whether a previous attempt at this item was recorded as
// failed. It complements MarkFailed and is what the retry/repair workflows and
// `rip database browse failed` are built on.
func (s *SQLite) IsFailed(ctx context.Context, source, mediaType, id string) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
var count int
err := s.db.QueryRowContext(
ctx,
`SELECT COUNT(1) FROM failed_downloads WHERE source = ? AND media_type = ? AND id = ?`,
source, mediaType, id,
).Scan(&count)
if err != nil {
return false, err
}
return count > 0, nil
}
func (s *SQLite) Close() error {
if s.db == nil {
return nil
+195
View File
@@ -2,7 +2,12 @@ package store
import (
"context"
"database/sql"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
@@ -48,3 +53,193 @@ func TestSQLiteStore(t *testing.T) {
t.Fatalf("MarkFailed() error = %v", err)
}
}
// makeLegacyDB creates a database with the pre-source-column schema and the
// given ids, as a pre-migration install would have on disk.
func makeLegacyDB(t *testing.T, path string, ids ...string) {
t.Helper()
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
if _, err = db.Exec(`CREATE TABLE downloads (id TEXT NOT NULL PRIMARY KEY)`); err != nil {
t.Fatal(err)
}
for _, id := range ids {
if _, err = db.Exec(`INSERT INTO downloads(id) VALUES (?)`, id); err != nil {
t.Fatal(err)
}
}
}
// TestLegacyMigrationPreservesHistory is the regression test for the migration
// that silently wiped the dedupe state: rows were carried over with source=empty
// while IsDownloaded matched source exactly, so every previously downloaded
// track looked new and the whole library was re-downloaded.
func TestLegacyMigrationPreservesHistory(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
path := filepath.Join(dir, "downloads.db")
makeLegacyDB(t, path, "old-1", "old-2")
s, err := NewSQLite(path)
if err != nil {
t.Fatalf("NewSQLite() error = %v", err)
}
defer func() { _ = s.Close() }()
for _, id := range []string{"old-1", "old-2"} {
ok, err := s.IsDownloaded(ctx, "qobuz", id)
if err != nil {
t.Fatalf("IsDownloaded(%q) error = %v", id, err)
}
if !ok {
t.Errorf("pre-migration download %q reported as not downloaded", id)
}
}
ok, err := s.IsDownloaded(ctx, "qobuz", "never-seen")
if err != nil {
t.Fatal(err)
}
if ok {
t.Error("unknown id reported as downloaded")
}
}
// TestLegacyMigrationKeepsBackupAndLegacyTable asserts the migration is
// recoverable: the original file is copied aside and downloads_legacy survives.
func TestLegacyMigrationKeepsBackupAndLegacyTable(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "downloads.db")
makeLegacyDB(t, path, "old-1")
s, err := NewSQLite(path)
if err != nil {
t.Fatalf("NewSQLite() error = %v", err)
}
defer func() { _ = s.Close() }()
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
backups := 0
for _, e := range entries {
if strings.Contains(e.Name(), ".bak-") {
backups++
}
}
if backups != 1 {
t.Errorf("expected exactly 1 backup file, found %d in %v", backups, entries)
}
var n int
if err = s.db.QueryRow(`SELECT COUNT(1) FROM downloads_legacy`).Scan(&n); err != nil {
t.Fatalf("downloads_legacy was dropped: %v", err)
}
if n != 1 {
t.Errorf("downloads_legacy row count = %d, want 1", n)
}
if v, err := s.userVersion(); err != nil || v != schemaVersion {
t.Errorf("user_version = %d (err %v), want %d", v, err, schemaVersion)
}
}
// TestLegacyRowsDetectedOnReopen covers a database already migrated by the
// earlier, lossy code path: the sourceless rows are on disk but nothing in
// this process migrated them.
func TestLegacyRowsDetectedOnReopen(t *testing.T) {
ctx := context.Background()
path := filepath.Join(t.TempDir(), "downloads.db")
makeLegacyDB(t, path, "old-1")
s, err := NewSQLite(path)
if err != nil {
t.Fatal(err)
}
_ = s.Close()
reopened, err := NewSQLite(path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = reopened.Close() }()
if !reopened.hasLegacyRows {
t.Fatal("legacy rows not detected on reopen")
}
ok, err := reopened.IsDownloaded(ctx, "tidal", "old-1")
if err != nil {
t.Fatal(err)
}
if !ok {
t.Error("legacy row not honored after reopen")
}
}
// TestConcurrentAccess exercises the mutex and connection handling under load.
func TestConcurrentAccess(t *testing.T) {
ctx := context.Background()
path := filepath.Join(t.TempDir(), "downloads.db")
s, err := NewSQLite(path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = s.Close() }()
var wg sync.WaitGroup
for i := 0; i < 32; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
id := fmt.Sprintf("t%d", n)
if err := s.MarkDownloaded(ctx, "qobuz", id); err != nil {
t.Errorf("MarkDownloaded: %v", err)
return
}
if _, err := s.IsDownloaded(ctx, "qobuz", id); err != nil {
t.Errorf("IsDownloaded: %v", err)
}
if err := s.MarkFailed(ctx, "qobuz", "track", id); err != nil {
t.Errorf("MarkFailed: %v", err)
}
}(i)
}
wg.Wait()
}
func TestSQLiteDSNPragmasApplied(t *testing.T) {
// A path containing '?' must not be parsed as query parameters.
dir := t.TempDir()
path := filepath.Join(dir, "weird?name#x.db")
s, err := NewSQLite(path)
if err != nil {
t.Fatalf("NewSQLite(%q) = %v", path, err)
}
defer func() { _ = s.Close() }()
var mode string
if err := s.db.QueryRow(`PRAGMA journal_mode`).Scan(&mode); err != nil {
t.Fatal(err)
}
if mode != "wal" {
t.Errorf("journal_mode = %q, want wal", mode)
}
var timeout int
if err := s.db.QueryRow(`PRAGMA busy_timeout`).Scan(&timeout); err != nil {
t.Fatal(err)
}
if timeout != 10000 {
t.Errorf("busy_timeout = %d, want 10000", timeout)
}
if err := s.MarkDownloaded(context.Background(), "qobuz", "a"); err != nil {
t.Fatalf("write to odd path failed: %v", err)
}
}
+122 -4
View File
@@ -45,17 +45,75 @@ func Parse(raw string) *ParsedURL {
switch {
case isQobuzHost(host):
return parseQobuz(raw, parts)
case isYandexHost(host):
return parseYandex(raw, parts)
case isBeatportHost(host):
return parseBeatport(raw, parts)
case isTidalHost(host):
return parseTidal(raw, parts)
case isDeezerHost(host):
return parseDeezer(raw, parts)
case isSoundcloudHost(host):
return parseSoundcloud(raw, parts)
return parseSoundcloud(raw, host, parts)
default:
return nil
}
}
func parseBeatport(raw string, parts []string) *ParsedURL {
if len(parts) < 2 {
return nil
}
if len(parts[0]) == 2 {
parts = parts[1:]
}
if len(parts) > 0 && parts[0] == "catalog" {
parts = parts[1:]
}
if len(parts) < 2 {
return nil
}
mediaType := ""
idIndex := 1
switch parts[0] {
case "track", "tracks":
mediaType = "track"
if parts[0] == "track" {
idIndex = 2
}
case "release", "releases":
mediaType = "album"
if parts[0] == "release" {
idIndex = 2
}
case "library":
if len(parts) < 3 || (parts[1] != "playlists" && parts[1] != "playlist") {
return nil
}
mediaType = "playlist"
idIndex = 2
case "playlists":
mediaType = "playlist"
idIndex = 2
case "chart", "playlist":
mediaType = "chart"
idIndex = 2
case "artist":
mediaType = "artist"
idIndex = 2
case "label":
mediaType = "label"
idIndex = 2
default:
return nil
}
if idIndex >= len(parts) || strings.TrimSpace(parts[idIndex]) == "" {
return nil
}
return &ParsedURL{OriginalURL: raw, Source: "beatport", MediaType: mediaType, ID: parts[idIndex], Kind: KindGeneric}
}
func parseQobuz(raw string, parts []string) *ParsedURL {
if len(parts) < 2 {
return nil
@@ -69,6 +127,9 @@ func parseQobuz(raw string, parts []string) *ParsedURL {
}
mediaType := parts[0]
if mediaType == "interpreter" {
mediaType = "artist"
}
if !isSupportedMedia(mediaType) {
return nil
}
@@ -80,11 +141,54 @@ func parseQobuz(raw string, parts []string) *ParsedURL {
return &ParsedURL{OriginalURL: raw, Source: "qobuz", MediaType: mediaType, ID: id, Kind: KindGeneric}
}
func parseYandex(raw string, parts []string) *ParsedURL {
if len(parts) < 2 {
return nil
}
switch parts[0] {
case "track":
if len(parts) != 2 || strings.TrimSpace(parts[1]) == "" {
return nil
}
return &ParsedURL{OriginalURL: raw, Source: "yandex", MediaType: "track", ID: parts[1], Kind: KindGeneric}
case "album":
if len(parts) == 2 && strings.TrimSpace(parts[1]) != "" {
return &ParsedURL{OriginalURL: raw, Source: "yandex", MediaType: "album", ID: parts[1], Kind: KindGeneric}
}
if len(parts) == 4 && parts[2] == "track" && strings.TrimSpace(parts[1]) != "" && strings.TrimSpace(parts[3]) != "" {
return &ParsedURL{OriginalURL: raw, Source: "yandex", MediaType: "track", ID: parts[3] + ":" + parts[1], Kind: KindGeneric}
}
case "artist":
if len(parts) != 2 || strings.TrimSpace(parts[1]) == "" {
return nil
}
return &ParsedURL{OriginalURL: raw, Source: "yandex", MediaType: "artist", ID: parts[1], Kind: KindGeneric}
case "users":
if len(parts) == 4 && parts[2] == "playlists" && strings.TrimSpace(parts[1]) != "" && strings.TrimSpace(parts[3]) != "" {
return &ParsedURL{OriginalURL: raw, Source: "yandex", MediaType: "playlist", ID: parts[1] + ":" + parts[3], Kind: KindGeneric}
}
case "playlists":
if len(parts) == 2 && strings.TrimSpace(parts[1]) != "" {
return &ParsedURL{OriginalURL: raw, Source: "yandex", MediaType: "playlist", ID: parts[1], Kind: KindGeneric}
}
}
return nil
}
func parseTidal(raw string, parts []string) *ParsedURL {
if len(parts) < 2 {
return nil
}
if isLangToken(parts[0]) {
parts = parts[1:]
}
if len(parts) < 2 {
return nil
}
if parts[0] == "browse" {
parts = parts[1:]
}
@@ -128,14 +232,20 @@ func parseDeezer(raw string, parts []string) *ParsedURL {
return &ParsedURL{OriginalURL: raw, Source: "deezer", MediaType: mediaType, ID: id, Kind: KindGeneric}
}
func parseSoundcloud(raw string, parts []string) *ParsedURL {
func parseSoundcloud(raw, host string, parts []string) *ParsedURL {
if len(parts) < 1 {
return nil
}
if host == "on.soundcloud.com" {
return &ParsedURL{OriginalURL: raw, Source: "soundcloud", MediaType: "track", ID: raw, Kind: KindSoundcloud}
}
mediaType := "track"
if len(parts) >= 3 && parts[1] == "sets" {
mediaType = "playlist"
} else if len(parts) < 2 || parts[1] == "sets" {
return nil
}
return &ParsedURL{OriginalURL: raw, Source: "soundcloud", MediaType: mediaType, ID: raw, Kind: KindSoundcloud}
@@ -164,12 +274,20 @@ func isQobuzHost(host string) bool {
return host == "qobuz.com" || host == "open.qobuz.com" || host == "play.qobuz.com"
}
func isYandexHost(host string) bool {
return host == "music.yandex.ru" || host == "music.yandex.com" || host == "music.yandex.kz" || host == "music.yandex.by"
}
func isBeatportHost(host string) bool {
return host == "beatport.com" || host == "api.beatport.com"
}
func isTidalHost(host string) bool {
return host == "tidal.com" || host == "open.tidal.com" || host == "listen.tidal.com"
}
func isDeezerHost(host string) bool {
return host == "deezer.com"
return host == "deezer.com" || strings.HasSuffix(host, ".deezer.com")
}
func isSoundcloudHost(host string) bool {
@@ -178,7 +296,7 @@ func isSoundcloudHost(host string) bool {
func isSupportedMedia(mediaType string) bool {
switch mediaType {
case "album", "track", "playlist", "artist", "label", "video":
case "album", "track", "playlist", "artist", "label", "video", "chart":
return true
default:
return false
+102 -7
View File
@@ -27,14 +27,85 @@ func TestQobuzAlbumURL(t *testing.T) {
}
}
func TestTidalTrackURL(t *testing.T) {
url := "https://tidal.com/browse/track/3083287"
result := Parse(url)
if result == nil {
t.Fatalf("expected parsed url")
func TestQobuzInterpreterURLParsesAsArtist(t *testing.T) {
inputs := []string{
"https://www.qobuz.com/us-en/interpreter/odezenne/739874",
"https://play.qobuz.com/artist/739874",
}
if result.Source != "tidal" || result.MediaType != "track" || result.ID != "3083287" {
t.Fatalf("unexpected parse result: %+v", result)
for _, input := range inputs {
result := Parse(input)
if result == nil {
t.Fatalf("expected parsed url for %q", input)
}
if result.Source != "qobuz" || result.MediaType != "artist" || result.ID != "739874" {
t.Fatalf("unexpected parse result for %q: %+v", input, result)
}
}
}
func TestYandexURLs(t *testing.T) {
tests := []struct {
url string
mediaType string
id string
}{
{url: "https://music.yandex.ru/track/9442712", mediaType: "track", id: "9442712"},
{url: "https://music.yandex.ru/album/1000856", mediaType: "album", id: "1000856"},
{url: "https://music.yandex.ru/album/1000856/track/9442712", mediaType: "track", id: "9442712:1000856"},
{url: "https://music.yandex.ru/artist/1433871", mediaType: "artist", id: "1433871"},
{url: "https://music.yandex.ru/users/yandexmusic/playlists/1635", mediaType: "playlist", id: "yandexmusic:1635"},
{url: "https://music.yandex.ru/playlists/4ae45ac1-0972-734f-8537-769490399170", mediaType: "playlist", id: "4ae45ac1-0972-734f-8537-769490399170"},
}
for _, tc := range tests {
result := Parse(tc.url)
if result == nil {
t.Fatalf("expected parse for %q", tc.url)
}
if result.Source != "yandex" || result.MediaType != tc.mediaType || result.ID != tc.id {
t.Fatalf("unexpected parse result for %q: %+v", tc.url, result)
}
}
}
func TestBeatportURLs(t *testing.T) {
tests := []struct {
url string
mediaType string
id string
}{
{url: "https://www.beatport.com/track/strobe/1696999", mediaType: "track", id: "1696999"},
{url: "https://www.beatport.com/release/random-album/12345", mediaType: "album", id: "12345"},
{url: "https://www.beatport.com/library/playlists/67890", mediaType: "playlist", id: "67890"},
{url: "https://www.beatport.com/chart/some-chart/111", mediaType: "chart", id: "111"},
{url: "https://www.beatport.com/artist/deadmau5/24078", mediaType: "artist", id: "24078"},
{url: "https://www.beatport.com/label/mau5trap/1234", mediaType: "label", id: "1234"},
{url: "https://api.beatport.com/v4/catalog/tracks/1696999/", mediaType: "track", id: "1696999"},
}
for _, tc := range tests {
result := Parse(tc.url)
if result == nil {
t.Fatalf("expected parse for %q", tc.url)
}
if result.Source != "beatport" || result.MediaType != tc.mediaType || result.ID != tc.id {
t.Fatalf("unexpected parse result for %q: %+v", tc.url, result)
}
}
}
func TestTidalTrackURL(t *testing.T) {
inputs := []string{
"https://tidal.com/browse/track/3083287",
"https://tidal.com/us/browse/track/3083287",
"https://tidal.com/us/track/3083287",
}
for _, url := range inputs {
result := Parse(url)
if result == nil {
t.Fatalf("expected parsed url for %q", url)
}
if result.Source != "tidal" || result.MediaType != "track" || result.ID != "3083287" {
t.Fatalf("unexpected parse result for %q: %+v", url, result)
}
}
}
@@ -93,6 +164,7 @@ func TestURLWithLanguageCode(t *testing.T) {
"https://www.qobuz.com/gb-en/album/name/id123456",
"https://www.deezer.com/en/track/4195713",
"https://www.deezer.com/fr/track/4195713",
"https://m.deezer.com/en/track/4195713",
}
for _, input := range inputs {
if result := Parse(input); result == nil {
@@ -101,6 +173,17 @@ func TestURLWithLanguageCode(t *testing.T) {
}
}
func TestDeezerMobileHostURL(t *testing.T) {
url := "https://m.deezer.com/track/4195713"
result := Parse(url)
if result == nil {
t.Fatalf("expected parsed url")
}
if result.Source != "deezer" || result.MediaType != "track" || result.ID != "4195713" {
t.Fatalf("unexpected parse result: %+v", result)
}
}
func TestSoundcloudURL(t *testing.T) {
inputs := []string{
"https://soundcloud.com/artist-name/track-name",
@@ -118,3 +201,15 @@ func TestSoundcloudURL(t *testing.T) {
}
}
}
func TestSoundcloudProfileURLIsNotTrack(t *testing.T) {
if result := Parse("https://soundcloud.com/artist-name"); result != nil {
t.Fatalf("expected nil for profile url, got %+v", result)
}
}
func TestSoundcloudSetsRootWithoutPlaylistSlugInvalid(t *testing.T) {
if result := Parse("https://soundcloud.com/artist-name/sets"); result != nil {
t.Fatalf("expected nil for sets root url, got %+v", result)
}
}
+68
View File
@@ -0,0 +1,68 @@
// Package verbose provides a process-wide verbosity level and a pluggable
// log sink so verbose output integrates with the downloader's progress bars.
//
// Level meaning:
//
// 0 (Off) - no extra output
// 1 (V) - log per-track CDN URLs from the downloader
// 2 (VV) - additionally log every outbound HTTP request via the
// netutil-wrapped transport (covers all provider API calls
// and downloads)
package verbose
import (
"fmt"
"os"
"sync/atomic"
)
const (
Off byte = 0
V byte = 1
VV byte = 2
)
var (
level atomic.Uint32
sink atomic.Pointer[func(string)]
)
// SetLevel clamps and stores the verbosity level. Pass 0 to disable.
func SetLevel(l int) {
if l < 0 {
l = 0
}
if l > int(VV) {
l = int(VV)
}
level.Store(uint32(l))
}
func Level() byte { return byte(level.Load()) }
func Enabled(l byte) bool { return Level() >= l }
// SetSink installs a writer for verbose output. Use this to route logs
// through the downloader so they don't tear progress bars. Pass nil to
// fall back to stderr.
func SetSink(fn func(string)) {
if fn == nil {
sink.Store(nil)
return
}
sink.Store(&fn)
}
// Printf emits a line at the given level if verbosity is enabled. The
// caller is responsible for including a trailing newline.
func Printf(l byte, format string, args ...any) {
if !Enabled(l) {
return
}
msg := fmt.Sprintf(format, args...)
if p := sink.Load(); p != nil {
(*p)(msg)
return
}
_, _ = fmt.Fprint(os.Stderr, msg)
}