Ripping

XLD for Mac also uses AccurateRip. I haven't noticed that it compares your CD/DVD drive performance in the same way EAC and dbPoweramp do.

Nice and simple write-up Vincent. I can't resist correcting an OT point though

Vincent Kars said:

This is a common design decision, e.g.
- Streaming audio uses UDP instead of TCP/IP as a protocol.
TCP/IP is strict, UDP is not. UPD allows for dropping packages but requires less bandwidth.

Click to expand...

TCP and IP use (almost) identical amount of bandwidth as protocols. That is not the reason UDP is preferred for streaming. Let me expand.

UDP and TCP are two types of communication a computer can establish with another computer (or device) to send data back and forth. If you send a message using TCP, the networking code attempts to make sure that the data gets to the other side. If something corrupts or drops data, TCP notices that and resends the data until the receiver acknowledges that it has gotten it (this is responsible for an extra bit of data that TCP takes). TCP and UDP exist in all systems that support communication these days.

UDP on the other hand doesn't care if the receiver gets the data. It sends the information onto your network and then it doesn't care after that.

When you listen to an Internet Radio or watch a Netflix movie, you are streaming such data from the server to your computer. One desires to deliver all the data from the source to the target or else, you lose some of your audio or video.

So at first, you might think that TCP is the best choice. After all it makes sure the data gets to the other side. Alas, that is not the case. Imagine a scenario where I send a frame of video and it gets lost over the network to my phone. TCP will wait a while and if it doesn't hear back from the receiver that it has got that data, it will retransmit. On a slow network such cellular, by the time you send the retransmitted data, the player may have moved on to the next frame of video. As such, it doesn't have any use for the old frame. The TCP retransmission therefore, wastes network traffic sending the old frame without accomplishing anything.

If the system uses UDP, then there is no automatic retransmission. The client will only ask for data that it "thinks" will arrive in time for it to be able to use it. Since the player knows more about the specific application here (streaming of audio/video), it can make better choices than "dumb" TCP software could.

So net, net, UDP is used to optimize system response and not because it automatically reduces network traffic.

Now, back to regular programming.

I posted a overview of issues and recommendations for ripping a couple of months back on AVGuide. There is a fair amount of overlap/reinforcement with Vincent's first post, but there is also some additional info, so I'll reproduce it here and hope it's useful to more folks than just me

-----------------------------------------------------------------------------


OK, ripping. Doing this right can greatly increase your enjoyment of your music collection; not getting it right can greatly increase your frustration

There are three things you want to accomplish when ripping. First, you want to be sure the data gets read from the CD without errors. Second, you want the data to be stored in a lossless or uncompressed format – you want to make sure none of the audio data is “thrown away”. Third – and most difficult – you want to make sure the music is “tagged” with data that is accurate, and sufficiently comprehensive enough, that you can search, navigate, and discover your collection.

Here is where it is useful, although not absolutely necessary, to know what software solution you’ll be using. For the sake of simplicity, let’s assume that your options are iTunes on Mac OSx, or J River Media Center on Windows. iTunes on Windows is not bit perfect because it doesn’t support the necessary audio driver protocols (unless you’re using the proprietary audio pipeline on the Blue Smoke Black Box), so that is out. Note that you can run Windows on all of the more recent Macs, should you have one laying around. (As an aside, you can also run OSx on a “PC” – see hackintosh.com – but that is technically illegal and irrelevant to our discussion).

Moving on to the first of our three goals, there are basically two approaches to ensuring data is accurately read from the CD. The first approach is to read the entire CD twice, compare the two reads, and assume the read was accurate if the two reads are identical. This is the approach used in the “secure” ripping modes of iTunes, J River, and most other integrated media libraries. The other approach, used by the commercial ripping program dbPowerAmp and the freeware Exact Audio Copy, is to calculate a unique fingerprint (called a CRC) from the data that is read, and then compare that fingerprint to an online database of fingerprints from all other users who have ripped the same recording. In this way, most CDs need only be read once to guarantee accurate data, saving a lot of time if you’re ripping thousands of recordings.

The second goal is easily accomplished, but requires an upfront decision. Fortunately, that decision can be fairly easily changed later. There are three categories of audio formats to think about: uncompressed, lossless compression, and lossy compression. Uncompressed is pretty self-explanatory: the data is basically copied bit-for-bit from the CD file system to the computer file system. Primary uncompressed formats are AIFF and WAV. Lossless compression is like a zip file: all of the audio data is preserved, but it is stored in a more compact form (typically 40-50% smaller). Primary lossless formats are FLAC, WMA Lossless, and Apple Lossless. Lossy compression further reduces the file sizes by trying to anticipate what the ear is most sensitive to, and essentially throwing out the less important audio data. There are many lossy formats, but the two most prominent are MP3 and AAC (Apple).

Any of the uncompressed formats, and any of the lossless formats, can be converted from one to the other with no loss of audio data, using free or inexpensive software tools. However, the WAV format has no provision for tagging data, so there is no reason to use that – use AIFF instead, which is just a wrapper around WAV that supports tagging. You can also convert from any of the lossless or uncompressed formats to any of the lossy formats – but the reverse is obviously not true! Once data is thrown away, it’s gone, gone, gone.

Given the extremely low cost of external data storage these days, there is no reason to make your initial rips of your CDs to a lossy format. If you want smaller files for use on your iPod or smart phone, you can use one of those conversion tools I mentioned to create a separate library of compressed files from your lossless main library – that’s what I do. So what should your primary format of choice be? Either AIFF, or one of the lossless formats. I have never been able to discern an audible difference between AIFF or the lossless formats; some people say they can. What I will say is that, given that the data is the same, the only differences must come from hardware issues stimulated by different software activity. Playing back a lossless file slightly increases the load on the processor, since the lossless compression must be undone before the data is fed to the audio pipeline. On the other hand, file system access or network access will be reduced with lossless files. As per my discussion previously on this thread, I prefer not to make strategic choices on software or audio formats based on the potential flaws of a particular hardware implementation. I use the FLAC lossless format, because it is an open format not owned by Microsoft or Apple, and therefore not subject to arbitrary changes for business reasons. Also, most high-resolution music downloads are now in that format, and I prefer the consistency. The file storage space saved by a lossless format is not that significant to me, but since all of my playback is over the network, the network bandwidth savings are significant, given the other members of my family streaming HD video and playing games. If you’re going to be using iTunes/Mac, Apple Lossless or AIFF are your only choices; you can use a free program called Max to convert FLAC downloads to those formats.

On to our third goal – accurate and sufficiently comprehensive tagging. Another term often used, instead of “tags”, is “metadata” – data about the music. Without metadata, all you have is a collection of audio files, which would obviously be very difficult to browse and search. The goal of good metadata is to enable to view your music collection from different perspectives, discover relationships, find music that matches your mood, stumble across old favorites you haven’t played in a long time, and do this all much more easily than searching many shelves full of discs. So first, then, your metadata must be accurate and consistent. Your playback software will think that “The Beatles” and “The Beetles” are two different artists. And if you’re searching for “Eleanor Rigby”, the song name tag had better not be “Alanor Rigby”. Second, your metadata must be sufficiently comprehensive. This is definitely a subjective judgment. For instance, if you have a collection of a few hundred recordings of mostly “mainstream” Pop or Rock music, having good Artist, Album, and Song tags may be enough for you. On the other hand, having additional metadata available, like the “Style” data you find on the AMG or Last.FM websites, can help you find, organize, and understand that kind of music in a different way. And if you have a significant percentage of Classical music in your collection, Artist/Album/Song just won’t cut it. Composer, Conductor, Sub-Genre or Work Type (Symphony, Concerto, Quartet, etc), Era, Performer, Release date, etc. are a major aid to navigating and enjoying a Classical collection. To some extent, Jazz is the same way. This is one area where your decision on playback software has major ramifications – iTunes simply doesn’t recognize many of these less often used tags, and allows only Artist, Album, Composer, Genre, and Groupings to be used in the “browser”. J River can be configured to recognize any tag, and allows you to define multiple different “views” which allow browsing by any combination of tags. Since I have close to 1000 Classical CDs, J River is a no-brainer for me.

So, how to get good metadata? Well, metadata only comes from one of two sources: either manual entry, or a metadata database. I’ll leave it to you to contemplate the horrors of manually typing in accurate data for tens of thousands of audio tracks – though there are folks apparently patient and dedicated enough to do it. Fortunately, there are a number of these metadata databases available, which use the number and duration of the tracks on a CD to create a (hopefully) unique fingerprint that can be used to locate the metadata for a particular recording. All of the programs used for ripping use one or more of these databases to “look up” and automatically enter metadata for a CD at the beginning of the ripping process, and allow you to edit the data manually if desired. The quality of these databases, though, varies widely – in terms of the number of CDs in the database, in terms of the accuracy of the data, and in terms of the number of different kinds metadata tags supported.

The major metadata databases are GraceNote, freeDB, MusicBrainz, YADB, AMG, GD3, and SonataDB. Gracenote is the only DB used by iTunes. YADB is the database used by J River. freeDB and MusicBrainz are free, and used by a couple of different rippers. AMG, GD3, and SonataDB are commercial databases requiring license fees to use, and are therefore not available with any of the free software choices.

Gracenote, freeDB, MusicBrainz, and YADB all rely heavily or exclusively on user-generated data, with all of the possibilities for error and inconsistency that implies. When I first started ripping my library using iTunes 5 years back, the crappy quality and spotty coverage of the Gracenote data drove me crazy. Gracenote has improved a great deal since then, but is still well short of the best commercial DBs, in my opinion.

AMG, GD3, and SonataDB – the other commercial alternatives – all rely primarily on in-house data entry and quality control, and are therefore much more consistent, although none of them is completely error-free. They also all have support for a broader range of tags. AMG is generally considered the gold standard for metadata; go to the AMG website, search for a couple of your favorites Albums or Artists, and observe the comprehensive range of information in the AMG database. Remember all those rave reviews of the Sooloos user interface? One of the things you’re paying for with a Sooloos is a license to the full AMG database, without which that user interface couldn’t and wouldn’t exist in anything like the same form.

So to my bottom line: if you want the best metadata outside of Sooloos, combined with fast secure ripping and support for essentially any output format, the software of choice is dbPowerAmp. dbPA uses the clever method I described earlier to mostly eliminate dual rips for secure ripping, it supports any of the output formats I mentioned earlier along with many others, and it comes with a batch conversion utility to automatically convert files from one format to another. Best of all, the Reference version allows you to mix and match metadata from five different DBs: freeDB, MusicBrainz, GD3, SonataDB, and a subset of AMG. It costs $38, and the interface takes a little while to get used to, but it’s worth the investment many times over in my opinion. There is nothing else out there remotely comparable in total. Unfortunately, it’s Windows-only. Like I said, if you’re going to be using iTunes, with primarily mainstream, popular music, the iTunes ripper may be an acceptable alternative.

This turned into quite a sermon! Hope some folks out there find it useful.

Edit - errata pointed out by other sources:

- There is a Mac/OSx ripping program, called XLD, which uses the AccurateRip technique described above to validate accurate rips without a second pass, same as EAC and dbPowerAmp.

- With manual intervention, you can use iTunes to output bit-perfect audio on Windows. You have to set the bit rate and depth of both QuickTime and Windows audio to match that of the source file. This is much less user-friendly than the automatic switching provided by some other Windows-based media players that can use different audio interface protocols like ASIO or WASAPI, but it can definitely be done.

That's a superb write-up Scott. I usually good at finding something to fault on such write ups but you are leaving me with next to nothing .

So maybe I can a bit of "insider" data. When we created Windows Media Player, we went searching for a source for metadata. The gold standard then was cddb (gracenotes). Problem was what you mentioned: a lot of user contributed data which at the time had a lot of quality issues including people putting foul language and such in there as a joke. So we decided to license AMG. As you correctly state, AMG database was superb and included cool things like discography. Problem was that their database was good for pop music and modern ones at that. Put in a classical or international music in it and it had nothing.

Our solution was to have people enter those. The system would then search to see if X number of people put in essentially the same data. Once there, a human would review and if correct, would add it to a secondary database Microsoft maintained. This gave us a way to augment AMG data but until we got it built, we took a heavy beating compared to cddb that others used. Over time though, we managed to augment AGM well enough and the complaints went away.

One last thing: lossly compressed audio has unpredictable data rate. Depending on what you feed them as far as audio samples, it is possible for a lossless encoder to actually explode the data larger than its input! When that happens, the lossless encoder simply passes the input data as is. This means your peak data rate is the same as the uncompressed data even though the average might be half as much.

Amir,

Tried to post a response earlier, but it didn't show up. Hopefully this one will, and will not be redundant

I was the CTO of Autodesk, Inc. from 99-05, and spent quite a bit of time in Redmond working with Microsoft folk. Don't remember you, but it's a big place, and the overlap between CAD and Audio isn't big I live in the Portland, OR area and am interested in this PNWAS thing - hadn't heard of that before.

Interesting observation re: AMG. In ripping my own collection with dbPowerAmp two years ago, I noted that AMG was usually the most accurate database, especially with Classical, but not always. Close attention to the metadata still paid big dividends in terms of not having to repair metadata errors later on. This is why I can't endorse the idea of ripping CDs as a "background" process, either formally or informally - one significant error in metadata can take you much more than the CD ripping time to find and fix. You have to be paying significant attention.

The other thing I'd note is that ripping is a big time investment regardless. I found that feeding two high-speed CD drives, while still keeping up with metadata validation, removing/replacing CDs, etc, was about my limit. This meant that average ripping time per CD, using dbPowerAmp AccurateRip mode, was between 3 and 4 minutes per CD. Do the math, and that means 5-6 ten hour days per 1000 CDs. Getting it right the first time is obviously a priority.

For those looking to save themselves such pain, I'd take a look at:

http://readytoplaysecure.com/

They use dbPowerAmp for ripping, and claim to have extensive processes to guarantee accurate rips and good metadata. If it was my collection, though, I'd send them 100 representative CDs as a low-cost trial before trusting them with my whole collection.

amirm said:

One last thing: lossly [ed ScottB: assumed "losslessly"] compressed audio has unpredictable data rate. Depending on what you feed them as far as audio samples, it is possible for a lossless encoder to actually explode the data larger than its input! When that happens, the lossless encoder simply passes the input data as is. This means your peak data rate is the same as the uncompressed data even though the average might be half as much.

Click to expand...

I did not know that. Anecdotally, I always thought my FLAC files were smaller than their sources, but I certainly didn't do a CDFS file size to FLAC file size comparison on each individual file, so how would I know? Overall, my library computes to less than 300 MB of FLAC per CD, which is clearly some data savings considering that most of those are very well-filled Classical CDs. But on an individual track-by-track basis, haven't done those comparisons.

Oh, track by track would not be revealing in that regard. To be clear, I was not speaking of the file size. I was speaking of instantaneous data rates of the file. Across an entire song, you will be hard pressed to find no compression. However, if you plotted the data rate of the file, you will see highly variable data rate including peaks that would show no compression but those moments don't last throughout the file.

Thanks Vincent for the corrections/additions.

The MusicCHI suite looks interesting, particularly the tagger and the "clean" database of composers/performers. In other respects, it appears to roughly duplicate the functionality of my current favorite, J River (e.g. "flavors" appears roughly similar to custom views in JRMC). I may give it a whirl; if so, I'll post my impressions.

Hi

Call it inertia. I have downloaded JRMC but remain attached to the simplicity and customization-ability of foobar plus it's entirely free ... Is there a way to have foobar emulate what JR does with the tagging? Quite interesting by the way although I would like to know if that can be done somehow automatically. IOW having the CD or the rip to retrieve the info on conductors, soloists, Orchestra, etc.

Oh, and BTW, The Well Tempered Computer is a very good resource. Thanks for doing it.

I particularly like your description of ripping:

Ripping - transferring CD's to the hard disk.
In general this requires 2 stages:
- ripping your entire CD collection
- re-ripping it because now you have found out how to do it

Click to expand...

Hilariously and painfully true.

Hi

I got this piece of information from Vincent's very useful and well organized Web Site in his signature " The Well Tempered Computer:

For those who haven't ripped their CDs (and DVDs) yet and find it a chore there are additional solutions:

Free: Vortexbox available for download at Vortexbox.org ... Once build .. You turn it on and push a CD or DVD in it .. It rips it in both flac and mp3 ... on the box HDD .. You can get fancy and use external Drive with it and you can have use it as your music server ...
Can be as low cost as one wish .. Get an old PC , almost any will do PII or P4 etc ... buy some higher capacity drive ( 1 TB is less than $60 these days to repeat this "ad infinitum" ) with a good CD/DVD drive. Download the free software and install it .. You put the CD in the drive Vortexbox rips it and if you have install yout unit properly an Internet connection will allow it to do more .. The only thing I find missing is the ability to use the Accurip service.. It uses MusicBrainz but I don't know how reliable that is compared to AccurateRip available with the low cost DBPowerAmp...

non-free: Or if one is not inclined to build a server and just want to get with the ripping .. There is the much more expensive solution of Ripnas servers which is a dedicated ripping machine all that you do is to push a CD in it and about 5 minutes later it is accurately ripped, in fact as accurately as it is possible. It cost a lot more, about $1400 . Works with Apple and Windows ... It does more than just rip and can be used as server itself ...

Thanks to Scott, Amir and others for their very technical descriptions of ripping, although, as non-technologist, I am sure that I am missing some of their points. As a Mac user with a dedicated Mac Mini for serving music, are there any special settings that I need to insure when using iTunes or XLD for ripping and do you recommend an external optical drive or is the internal good enough provided that the bit check works out?