ThinkPad X200 Tablet Save Extra 5%Audiophile MP3 players, by the numbers
When Jasmine and I evaluate MP3 players for CNET reviews, we always try to spend a few sentences describing any noticeable audio performance characteristics we detect during our subjective testing. We'll play around with all of the gadget's different EQ and sound enhancement options, listen back on our reference headphones, and run through a playlist of familiar music. We're only human, however, and hearing loss, ear wax, head congestion, and hangovers can skew our perceptions of audio quality from day to day. Thankfully, we have Eric Franklin.
Are you the kind of portable audio purist who wouldn't think twice about dropping $150 on a portable heaphone amplifier for your MP3 player?
(Credit: HeadRoom)Eric works in our CNET Labs and tests the audio quality and battery life of our MP3 players as part of his job. Unlike Jasmine and myself, Eric never actually listens to the MP3 players we review. Instead, he reformats the player, makes sure the latest firmware is installed, and transfers over a few white noise and sine wave audio files. Finally, he connects a Audio Precision ATS-2 Audio Analyzer to the headphone output of the MP3 player to reveal details such as the MP3 player's signal-to-noise ratio (SNR), total harmonic distortion (THD+N), frequency response deviation, maximum power output, and stereo crosstalk.
We've been testing our MP3 players this way for more than a year, but you wouldn't know it from reading our reviews because we've never included the data. Here's why:
Most MP3 players spit out the same results during testing, especially when you consider that a variance of +/- 5dB in each testing measure is indistinguishable to the human ear. Most of the variances we see between players aren't worth mentioning.
When we talk about "audio quality," most people assume we mean how pleasing something sounds to your ear. Unfortunately, our tests don't measure how pleasant an MP3 player audio chip sounds, but how accurately it performs. Paradoxically, not all accurate players sound great, and some of our favorite-sounding MP3 players have a few technical shortcomings.
I expect to hear some backlash on this point, but I'll put it out there anyway. These are "portable" music players. They are used as a means to shoehorn music into all the circumstances of your life that are the least ideal for appreciating the finer nuances of recorded audio. When you're on the bus, or at the gym, or driving in your car, the ambient noises all around you mask enough frequencies to make our lab measurements inconsequential. Personally, I think it's much more practical to know that an MP3 player has enough EQ and sound enhancement options to make a bass line cut through subway noise, than to know how accurately a given player reproduces frequencies in the inaudible 22kHz range. If we were testing home hi-fis, that's one thing, but portable audio players are a different beast.
That said, I'm an audio nerd at heart, and when I pored over the past year's audio test results, I couldn't help but share the high scores. As an extended version of the product comparison I posted last week, I've included the charts and full results of our top performers.
1. Creative Zen
The Creative Zen took the top score in our CNET Labs audio test results.
(Credit: Corinne Schulze/CNET Networks)I was not expecting a Creative MP3 player to come out on top of our list of cleanest-sounding MP3 players (much less, the top two). Don't get me wrong; Jasmine and I have long sung the praises of the excellent EQ and sound enhancement features Creative builds into their MP3 players. You just don't expect an MP3 player that's capable of boosting low frequencies to skull-rattling levels to also be the player with the most desirable audio traits for EQ-eschewing purists. I guess it shouldn't have come as a surprise that one of the leading manufacturers of computer audio cards would know a thing or two about digital audio performance. The Zen didn't have the flattest frequency response of the bunch, but it had an unmatched triple threat score of a 1.47dB frequency response deviation average, -82.27dB THD+N average, and a -83.62dB SNR. The Zen also gets bonus points for having its THD+N and SNR levels so close to one another.
The Creative Zen offered a frequency response deviation of -1.47dB during testing, with a very gradual drop under 60Hz and over 22kHz.
The Creative Zen Stone Plus delivers clean sound in a small, affordable package.
Creative's Zen Stone Plus couldn't quite strike the same overall balance as its bigger brother, but for less than $60, who can complain? With a -0.65dB frequency response deviation average, -78.95dB THD+N, and a -84.75dB SNR, the Stone Plus arguably performed better than the Zen depending on which measure you place more weight in. Still, we feel a difference of 4dB in the total harmonic distortion reading between the Zen and the Stone means more than the 1dB differences in FRD and SNR.
Then again, what's 4dB if you're saving $50 over the Zen?
The Creative Zen Stone Plus scored better than its big brother during our frequency response deviation test, scoring an FRD average of just -0.65dB. While the majority of frequencies skimmed right along the 0dB midline, sloping at the upper and lower (inaudible) frequency ranges is a little more pronounced than the larger Zen player.
(Credit: CNET Networks)
The mighty iPod Classic, in third place?
The iPod sound quality debate has followed the king of MP3 players from the very beginning. Love it or hate it, I had my money on Apple to win for the best technical sound quality. With an frequency response deviation of -1.56dB, a THD+N of -69.26dB, and an SNR of -84.42dB, the iPod Classic is nothing to turn up your nose at when it comes to clean audio reproduction. Other iPod models came close, but the Classic ruled roost among Apple's offerings. Support for the Apple Lossless music format, up to 160Gb of storage, and a hackable line-output through the dock connection, give the Classic other advantages over the competition.
Among all four models of the iPod, the iPod Classic performed the best during testing, scoring an average frequency response deviation rating of -1.56dB. While the lower frequencies roll off down below 40Hz, the upper ranges stay steady up to 22kHz.
(Credit: CNET Networks)
The Insignia Pilot? We were as surprised as you are.
We pride ourselves on not judging a book by its cover, but admittedly, our expectations were fairly low for Best Buy's in-house brand Insignia Pilot MP3 player. We couldn't have been more dumbfounded when the Pilot NS-4V24 Bluetooth MP3 player gave an unbelievably flat frequency response, with a deviation average of just -0.44dB. Its total harmonic distortion average came in at a decent -62.38dB, while its SNR impressed us with an average of -87.3 dB--the best signal-to-noise ratio we have on record. Plus, it has stereo Bluetooth audio streaming built-in, and support for MP3, WAV, WMA, WMAPro, Audible, Ogg Vorbis, and Protected WMA file formats.
Out of nowhere comes the Insignia Pilot MP3 player with an impressive frequency response deviation average of -0.44dB. Sure, it dips at the inaudible extremes, but between 90Hz and 20kHz the reading stays glued to the 0dB midline.
(Credit: CNET Networks)
SanDisk Sansa Clip
Like the Insignia Pilot, we were not expecting much from the Sansa Clip. Why on earth would SanDisk pack one of their best sound chips into an MP3 player costing less than $50? The Clip's -1.36dB frequency response deviation average is slightly better than the iPod Classic, and a closer inspection of the graph reveals absolutely no frequency dips between the audible range of 20Hz-22kHz. A total harmonic distortion score of -65.25dB and an SNR of -84.78dB makes the Clip a top pick for budget audiophiles who have already put themselves in debt with their $500 headphones.
SanDisk MP3 players are not known for their audio quality, but the superaffordable Sansa Clip is a notable exception. With a frequency response deviation average of -1.36dB that plots steady from 20Hz-22kHz and an admirable signal to noise ratio, the Clip can hold its own.
(Credit: CNET Networks)
Sony makes the cut, with a little performance enhancement.
I had Eric test the Sony NWZ-S718F with its noise-cancellation feature switched off, and its DSEE high-frequency enhancement feature turned on and turned off. It turns out that the Walkman performed better with DSEE engaged, giving us the supreme frequency response deviation reading of -0.28dB (or -0.77 with DSEE off). Throw in a THD+N rating of -63.57dB and an SNR of -83.25dB and Sony just slips out of the top five. Subjectively speaking, however, the NWZ-S718F is at the top of my audio-lover list. The noise-canceling headphones integrated into the NWZ-S718F allow you to hear more detail in your music under real world conditions.
The Editors' Choice award-winning Sony Walkman NWZ-S718F with its DSEE high-frequency enhancement setting turned on. Ironically, when the setting was active, frequencies above 20kHz appear to be completely shelved. The active DSEE setting produced a better overall frequency response deviation average of -0.28dB, however, with DSEE disengaged the frequency response curve is continuous up through 22kHz.
(Credit: CNET Networks)Down, but not out...the Zune
There are only six slots in our CNET product comparisons, so I didn't have room to fit the Microsoft Zune. The 80Gb Zune scored a -1.03dB on frequency response, and an excellent -84.99dB on SNR, but had relatively average scores on total harmonic distortion, grabbing a -59.25dB. Arguably, the Zune's high capacity and support for WMA Lossless music playback might be more important to audio purists than a few dB of THD.
The Microsoft Zune nearly broke the top list, especially with its frequency response deviation average of -1.03.
(Credit: CNET Networks)The world is not flat
It's been fun to geek out on all these numbers, but ultimately I don't recommend anyone buy an MP3 player based on stats alone. My best piece of advice: if audio quality is important to you, trust your ears over anything else. Your perception of what sounds good or bad is unique to you, shaped over years of listening habits and a personal musical taste.
For instance, one of our favorite-sounding MP3 players, the Cowon D2, tested poorly because it has a musically flattering EQ curve engineered into it. So what? It sounds great, supports all kinds of lossless audio formats, and has one of the best EQ sections you can buy on a portable device.
To give some perspective, here's the frequency response chart of the Cowon D2, an MP3 player whose sound we can't get enough of. Many manufacturers (not just Cowon) deliberately engineer their MP3 players to sweeten certain frequencies. Being flat isn't everything.
(Credit: CNET Networks)So don't miss the forest for the trees, folks. Music is supposed to be fun, not math, and the best part of owning an MP3 player is weaving music into your messy, noisy life. That said, I hope I've given those unapologetically obsessed portable audiophiles something to chew on. If you'd like to see more of this kind of analysis in our formal MP3 player reviews, let me here you in the comments section. Also, let me know if there's another audio performance measure beyond frequency response, THD+N, or SNR, that you really want our labs to test for.
Donald Bell is CNET Reviews' senior editor for MP3 players and portable audio, and one half of the MP3 Insider blog and weekly podcast. He also likes getting his hands dirty with digital audio tools for musicians and DJs.
Donald Bell is an electronic musician, a veteran record store employee, and a fearless hardware hacker. He's also CNET's Senior Editor for MP3 and digital audio.
Jasmine France is CNET's resident digital audio doyenne, writing and editing product reviews, crave blogs, and feature stories on all things MP3. And if you need advice on headphones, she's your girl.
?Paradoxically, not all accurate players sound great, and some of our favorite-sounding MP3 players have a few technical shortcomings.? Nonsense. Accurate MP3 players should all have excellent sound quality.
?My best piece of advice: if audio quality is important to you, trust your ears over anything else. Your perception of what sounds good or bad is unique to you, shaped over years of listening habits and a personal musical taste.? Beware of the audio equivalent of the placebo effect.
?For instance, one of our favorite-sounding MP3 players, the Cowon D2, tested poorly because it has a musically flattering EQ curve engineered into it.? For best sound quality, frequency response should be 0 dB from 20 Hz to 20 kHz. If users want to modify the frequency response, then they should look for an MP3 player with a good nine-band equaliser.
?If you'd like to see more of this kind of analysis in our formal MP3 player reviews, let me here [sic] you in the comments section. Also, let me know if there's another audio performance measure beyond frequency response, THD+N, or SNR that you really want our labs to test for.? Every MP3-player review should come with laboratory measurements. You should also test for channel separation; 20 dB is enough, which should be very easy to achieve with just about every MP3 player ever manufactured. For the signal-to-noise ratio, I would like to see A-weighting used.
Here are two websites every reviewer of audio equipment should be very familiar with before they start reviewing audio equipment:
http://www.provide.net/~djcarlst/abx_plac.htm
http://www.theaudiocritic.com/
Reality is that the perceived frequency response varies MUCH more due to how tightly coupled the headphones are to your ears. For example, adding a little extra pressure with a finger can boost the bass below 100Hz more than 10dB easily and likewise if they are too lose you can experience drop in bass.
More importantly in my opinion is to judge the audio quality in terms of its transparency and accuracy. But such parameters are usually also more a matter of the headphones than the player.
Some times you can hear clipping when you apply equalization e.g. if you add bass then the bass starts clipping too early because the DSP algorithms or amplifier are saturating. Ideally the sound is exactly as you want it to be using the particular headphones you intend to use - without any special audio processing mode enabled.
Then you will hear that MP3 players it is still not a choice for listeners...
Also, might be worth saying that these results only apply to driving an amp since they are unloaded. Try again with 16/32/60 ohm headphones hooked up through a splitter to get real world results.
-
by mmonsor
October 27, 2008 9:19 AM PDT
- Just another comment: I know my 'mp3' player is just a portable device, but I have my whole music collection on it and it is my primary listening vehicle. I have to say that my old Rio Karma with my Shure $500 ear buds, for the most part, sound at least as good and often better than my $3k two channel stereo system. I don't get that 'feel the music on the surface of your skin feeling' and sometimes miss the big bass, but for clarity, it's close. I am, frankly, looking for a device that I can put in my pocket that will give me everything I want as far as music goes. I want that 98db signal to noise ratio and .001% THD in my 'MP3' player and am glad to see you publishing some numbers. I hope, with the numbers going public, the manufacturers will get to work.
-
Like this
Reply to this comment
-
(16 Comments)