Big, Bad iTunes

iTunes is a fantastic application. It is well designed, has a great feature set, and has excellent usability. Well... mostly. I have a rather large library (27,000+ songs), stored on a remote drive, and this is where iTunes starts looking a little pale.

I'm running iTunes 7.3.2 (6) on Mac OS X 10.4.10, the latest and greatest at the time of this writing. I've been updating religiously for years now, hoping these problems would finally go away. I've even tried submitting suggestions to Apple, but that avenue seems to be a black hole. So welcome to my blog! I hope you'll chime in with suggestions, or at least, lend some solidarity.

My Setup

I have a ReadyNAS 600 drive with 650GB of disk space in RAID5 configuration exporting a CIFS share. The NAS has a GIG-E ethernet connection to an AirPort Express connected to my stereo. The AirPort Express is strictly used as an endpoint for digital audio -- it's wireless capabilities are disabled. The GIG-E switch is also connected to a Buffalo WHR-HP-G54 wireless router running DD-WRT v23 SP2. I use my MacBook Pro to connect to this wireless to access the NAS share and AirTunes service.

The NAS stores about 233GB of music files, along with my "iTunes Library" file. I could store the library file on my laptop, but I sometimes run iTunes from another computer in my house, and I like keeping it with the music. (At the time of this writing, my "iTunes Library" file size is 42.1MB.) The music files are all mp3s, encoded mostly as 192kbps or greater.

I suppose my setup is more complicated than most, because I also have the files on the NAS drive organized into different logical groupings, stored in separate folders (e.g. mp3 rips of my CDs, tracks I've downloaded, tracks from friends). That means I do not enable "Keep iTunes Music folder organized."

Here's a list of some of the problems I've encountered with this setup:

Slowness

1. Dropping a folder of new music into iTunes to load the track info (meta-data) for those tracks into my library can take anywhere from 15 min to an hour (or more!) depending on how many/how large the files are. There's definitely something really wrong here, and I suspect that iTunes is writing and rewriting its "iTunes Library" file over and over, for each track imported. I realize that the tracks are making a round trip between the NAS drive and my laptop over my 802.11G wireless network, but it's much much faster to manually do a "Get Info" for each track, navigate to it in the "Choose a File" dialog, and let it re-read the meta-data that way.

2. Pasting album artwork into a CD's worth of tracks can also take a very long time. Again, I suspect that the tracks are making a round trip from the NAS to the laptop, the artwork is getting inserted, and the tracks are written back out, all over wireless.

3. Scrolling slowness when viewing with album artwork. Scrolling performance is very poor with a library of this size, no matter what view I use (even List View). iTunes seems to be fetching track info and artwork on the fly, or at the very least, checking whether the tracks exist. I notice that sometimes scrolling stops and the cursor spins before the exclamation marks appear.

4. Progress dialog, when updating, is usually blank. While you're waiting around for all this library update activity to complete, a progress dialog appears. However, the dialog hardly ever repaints. Also it can annoying pop up over other applications long after you switched out of iTunes. Often the only way to get it out of your way is to temporarily switch back to iTunes, then back to your other application, and the dialog will float to the bottom.

5. All library updates should execute in background thread. iTunes seems to be able to play tracks and update the library at the same time, but it can't let the user interact with it while it's updating the library. And when library updates take and extraordinary long time as they do in my case, this can become very annoying. iTunes could be vastly improved if all updates were queued up and conducted by a background thread.

6. Current song's audio should be sufficiently (fully) buffered if the song's track info is to be updated. While iTunes does seem to be able to update the library at the same time it's playing a track, it gets into trouble when it comes to updating the track that's currently playing. The playing will stop (sometimes for minutes) while the track info is updated, and then it will resume again. Again, with queued background updates, this could be completely avoided. Or if iTunes could first completely buffer the audio before updating the track, it could avoid stopping playback.

Bad synchronization

7. Missing files. For a while I had some tracks on my laptop's local hard drive, and some on the share, and I had my iTunes Library file on my laptop. This turned into a real nightmare for several reasons. First, if you ever run iTunes with just the local music (e.g. from work where you can't access the share), and if you play tracks randomly, iTunes will inevitably pick a track to play that doesn't exist, and then flag it as missing. However, when you get connected to the share again later, iTunes doesn't seem to realize that the tracks are back. Instead it asks you if you'd like to locate them. This can be a slow and tedious process, because when the file chooser dialog comes up, you have to wait for it to display thousands of folders as you wade through your library.

8. "The iTunes Library file cannot be saved. The disk could not be read from or written to. (Do not warn me again)" -- I agree Apple... please do not warn me again! I get this when I stupidly forget to quit iTunes before putting my laptop to sleep. When I get up the next morning and start it up again, it pops up this dialog. Even if I reconnect to the share it still insists that the library is inaccessible. I can play tracks that are stored on the share by clicking on them in the iTunes interface, but every few seconds the dialog will keep popping up, insisting that the library cannot be found. And worst of all, all my library modifications from the previous evening will be lost!

Difficulty creating work-arounds

9. Bad XML. At one point I thought I could fix all my library problems by writing scripts that read and modify the "iTunes Library.xml" file that iTunes exports. This file contains a machine-parsable description of all the meta-data in the library. This XML file contains <key> and <dict> tags and <dict> entries in turn include <key> and value tags:

<key>7296</key>
<dict>
<key>Track ID</key><integer>7296</integer>
<key>Name</key><string>Novacane</string>
<key>Artist</key><string>Beck</string>
...
</dict>

The problem is that when you try to parse (e.g. from perl), you don't end up with what you might expect. The parser will return 2 lists for the above fragment of XML -- a list of the keys, and a list of the <dict> objects. And each <dict> object will contain another list of all the keys, and a list of the values. So there's no way to match a key to a given <dict> object or value! Parsing iTunes XML requires you to parse and preserve the order of the <key> and <dict> elements as they occur in the file.

Perhaps the XML should be more like this:

<key name="7296">
<dict>
  <key name="Track ID"><integer>7296</integer></key>
  <key name="Name"><string>Novacane</string></key>
  <key name="Artist"><string>Beck</string></key>
  ...
</dict>
</key>

10. Album Artwork folder. This is another design abomination. Whenever iTunes fetches artwork for you, it places it in the Album Artwork directory instead of with the tracks themselves. Moreover, the Album Artwork folder will contain sub-folders with names like "EE0BF336E7869369". If you study the "iTunes Library.xml" file, you will see these correspond to a "Library Persistent ID" property. But why make this unique to the library? If I create another library but include the same songs, the artwork will not be found. Why can't iTunes just look for a "cover.jpg" file in the album's directory? This would be fast to save because it would involve modifying the mp3 files, and it would be easier for users to understand and modify.

Other annoyances

11. "Compilation" (my pet peeve.) I don't mean that I don't like compilations, but Apple's nomenclature here is completely misleading, and leads to tons of bad meta-data. You must understand that when iTunes says "Compilation", it really means "Various Artists". It's a way of making sure tracks group together by album name alone, instead of by album and artist names together.

I can't tell you how many albums I've downloaded the track list for, only to find that it's mis-categorized in this way. Here are a few guesses as to where things go wrong:

• Single-artist album, but the tracks are a collection of the artist's previous work, e.g. a Greatest Hits album. This confuses the person uploading the track information, and they click "compilation" even though the disc is not by various artists.
• Classical music performance, performed by one symphony and one conductor, but the works are those of a handful of classical composers. Again the "compilation" flag gets set. (I suppose this type of disc could be considered to be by various artists, but for the sake of consistent tagging in iTunes, I think it's better to consider the artist to be the symphony and/or conductor, so that these show up in the artist column of the iTunes browser.)

Conversely, often times people to set the compilation flag before uploading track information. That means that if you're the unlucky user that ends up downloading this track information when ripping your disc, the tracks end up getting strewn all over the place (by the various artist names, instead of collected under the Compilations folder).

And unfortunately, if you have your iTunes set up to "Import CD and Eject" (as I do), you won't find out about all this badness until you go to play the album some time later. It sure would be nice if the CD view had an indication of whether the compilation flag was set, and if it also showed you whether the artwork was present or not. That would give you a chance of fixing them before ripping.

12. Album Artist. What is that? It's an evil hack (as far as I can tell) whose sole purpose in life is to make up for the abuse of the Compilation flag. If you fail to have the Compilation flag set, then your album's tracks will still be grouped together in the Album View if the Album Artist field is set. However, something I did at one point (perhaps it was building a new iTunes library out of just one of my several library folders, and telling iTunes to keep that folder organized) caused the Album Artist field to be set to the name of the Album for a number of compilations. Then the "keep organized" setting moved all these tracks to folders named by the albums under my top-level music directory. What a mess to get cleaned up!

Recommendations to Apple

• optimize iTunes Library management
• send just the deltas to the (remote) drive, don't rewrite the entire file
• update on a periodic basis (auto-save), using a background thread
• improve scrolling performance
• perform artwork fetch on background thread
• don't check for existence of tracks until they are to be played
• reconnect to iTunes Library if remote drive appears to have become disconnected
• always re-check existence of tracks (with exclamation marks) before asking the user to locate them
• update progress dialog more frequently
• or eliminate it altogether by performing updates in the background
• perform better handling of track info updates with respect to the currently playing track
• don't let the music stop!
• try a few heuristics to find missing tracks before asking the user to locate them
• if another track from the same album can be found, then look in that album's folder
• look for the album under compilations, if not marked as a compilation, or
• if it's supposed to be under compilations, look for it under the artist's name
• look for truncated album, artist and track names
• improve export library XML format
• change format to maintain proper key/value pair association
• export all fields from user interface, including My Rating
• eliminate Album Artwork folder in favor of storing artwork alongside tracks in the track's folder
• rename "Compilation" flag to "Various Artists"
• infer Various Artists when importing by looking at the Artist field for each track
• clarify the meaning of "Album Artist"
• if it's supposed to be the DJ, compiler of the CD, or conductor/symphony, then maybe it should be displayed in the Artist column of the iTunes browser if it is present

Thanks for listening! I'm expecting big improvements in the next release!