Format Listening Tests:
CD, MD (ATRAC 4.5 & 3.0), MP3, VQF, RM

Guy Churchill ([email protected])
Fri, 19 Mar 1999 16:12:22 +0800

Initial impressions and final report on A/B listening tests comparing
CD, MD (ATRAC 4.5 & 3.0), MP3 , VQF, RM.

NOTICE and DISCLAIMER. Everything expressed here is *my* opinion. Processes were dealt with as scientifically as possible (though some may want to dispute that), but listening is purely subjective. You may happen to find different results than me, I accept that, but I do not believe you can truly tell the difference between each format unless you conduct an experiment like this one. The disc is available to anyone else who wants to make up their own mind on the subject. But bear in mind however that I am not likely to entertain criticisms such as "you should have used co-ax, because it is better" or "Mitsubishi CDRs are superior to Kodak CDRs", or "You should be using a full Krell system as the listening equipment", etc etc etc.

Introduction

At the beginning of 1999 I requested suggestions for songs to include in an audio compression listening test. I finally ended up choosing "Helene" by Roch Voisine and "The Carpenter" by Thursday Diva. Both are very clear DDD recordings, and contain a mix of simple and complex music, varying in frequency content. I finished preparing the recordings a few weeks later and performed initial listening tests. Subsequently I used 2 identical CD players to perform A/B listening tests.

Encodings

Six encodings of each track were considered:

All processes remained in the digital domain (optically cabled for those interested in the optical vs. coax debate).

Equipment used for recording:

Equipment used for playback

CD player and DAC used for initial test listening CD players used for A/B switching tests. Accessories and cables.

Listening tests, Part I

Initial impressions (based on basic listening, skipping from track to track, not A/B switching).

CD

As it should be -- it is the original after all!

ATRAC 4.5

Unable to distinguishable with accuracy from the original at this stage, more listening needed (perhaps A/B switching will show something up). Both extreme high and low frequencies appear to be intact. No loss in db (digital sound meter peaked at the exact same level).

ATRAC 3.0

Small decrease in imaging, "space" around instruments not as pronounced, sound is more 2 dimensional, i.e. lacking some depth. Both extreme high and low frequencies still appear to be intact. Loss in dB was less then 1dB (digital sound meter peaked at just below the same level, less then 1% diff.).

MP3

Impressively good but instantly recognisable, decrease in image, decay rates far too fast, highs washed out. Lows still there but lack "oomph" when compared to CD

VQF

Again, impressively good but instantly recognisable, sounds better then MP3 (IMHO) but uses less bandwidth. Highs are brighter and lows similar to MP3 . VQF is basically the basis for MP4.

RealAudio

Worst of the bunch, handles the "Carpenter" track OK but the male vocals on Helene are really poor. Loss in dB and high and low frequencies, compressed imaging.

Listening tests, Part II

Results of A/B listening tests conducted with identical CD players & interconnects.

Listening Methodology

Having 2 identical CD players meant that a single remote could be used to control both simultaneously, allowing easy navigation to tracks of different formats. I had an accomplice start the two CD players on different tracks, then I simply switched audio sources to make comparisons (the A/B switching was also done via remote and the time delay incurred when switching sources was barely noticeable). With this setup I could cause both machines to skip backwards to the start of the track, or fast forward at the same rate, and then go to the next track. Using this method I was able to conduct the tests in essentially "blind" fashion. I propose that if anyone else wants to listen to the test CD, that they first compare the tracks without foreknowledge of which track is which. After they have made their comparison I will release the specific track information to them.

The first objective was to isolate parts of the track that could theoretically sound different in each format. I found these by carefully listening to the track as a whole and marking down points of interest (or music that could cause anomalies). Using the same remote I could skip back on both CD players simultaneously and re-listen to each section as many times as I liked. Each format could then be compared to the other and later a random selection of tracks was selected.

CD

The CD original was certainly the best, not only was the entire frequency range still intact, the imaging was right and the individual sounds had life, clarity and position.

ATRAC 4.5

ATRAC 4.5 was initially only distinguishable via A/B switching and with about 10 listenings between the two. The areas where it was different were in decay rates, depth, & point percussion. These may not be the terms normally used, so it is important that I explain what I mean by each one.

ATRAC 3

ATRAC 3 was easily distinguished. In comparison to 4.5 it is quite compressed image wise -- it sounds flat and lifeless. Against CD, ATRAC 3.0 is certainly poorer -- it is muffled and lifeless (there is that word again). Though I challenge anyone to pick ATRAC 3.0 just by listening to it (without A/B switching), A/B switching gave a clear result, and anyone should be able to tell the difference.

MP3

MP3 was the disappointment I expected it to be. This does not mean it has no application - for the PC, MP3 is ideal. It has a good compression factor, allows easy transfer over the Internet, and sets new standards in audio compression (I said new - not better). MP3 is instantly recognised without an A/B switching test. It introduces some distortion to highs in particular. Also, in complex passages the instruments get lost - you cannot follow each one - it is as if the encoder could not work out what parts to keep and which to throw away. This hypothesis was confirmed when I went back and recorded the same songs at higher MP3 bitrates (128kbps was the one I used initially, as it is the most widely used today). At 256kbps and 320kbps there were some remarkable improvements.

I do believe MP3 will stay mainly in the computing field, having said that have a look at the self contained MP3 car audio storage system at empeg.com. This is one application that certainly has promise (it would be even better if it accepted MP3 encoded CDs, as you get 150 to 180 MP3s(!) on a single disc). For car applications, background music, or played over typical speakers found on most PC systems, MP3 is fine. As a serious audio storage medium to be used with quality equipment however, I cannot recommend it at all.

VQF

Despite the even lower bitrate, Yamaha has come up with a great little format. Once again however, not even A/B switching was needed to distinguish it. It suffers from all the problems of ATRAC 3.0 and MP3. How does it stack up against MP3? Remarkably well. I was impressed at the frequency range it held. My success rate at picking MP3 from VQF was only just better then chance (about 65%) which leaves me to wonder, was it just chance? I think I will need to conduct this experiment in the future and compare MP3 to VQF again. At a size 30% smaller then MP3 I will give it the nod (but it is not a widely accepted format and the encoder is very slow -- 2 big negatives in my book).

RealAudio

I am wondering why I even tested this, it was by far the worst, like listening to the radio (actually, radio is better). Its voice audio application on the Internet is where it will remain, since that is what it was designed for (and because that is all its audio quality is good for). Bringing it into this test was not a fair challenge.

Conclusion

ATRAC 4.5 is remarkably good, and for applications like portables and car stereos I am more then happy with its performance (I have been using an MZ-R3 since 1995 and also listen with an MZ-E25). The A/B listening test was the only way of picking up the differences, and only certain passages of music tripped it up. For home recording of tapes, radio and CDs it is perfectly fine and without high quality partnering equipment it would be impossible to tell it from CD. Incidentally, I did some tests to compare different CD players and DACs, and when playing the same track I could hear differences - not all CD players and DACs are the same (DACs appear to make the major difference).

Even though ATRAC 4.5 has far surpassed the sound quality of ATRAC 1, MiniDisc made an impressive debut in 1992. The convenience of the MD format far outweighs tape, and MD's lack of acceptance in the early years was not an issue about quality, it was a combination of Sony's poor strategy at promoting the product, high equipment and disc prices, DCC confusion, and a DAT fraternity defending the little territory they had. And as MD changed the face of consumer audio recording, MP3 has done likewise with computer audio recording and storage. But as MP3 quality stands now, its place remains firmly in PC and network applications (not to mention the problems brought on by copyright issues). VQF and RealAudio are further audio compression algorithms, with their own strenghts and weaknesses.

The next project I have planned is to compare the audio quality of Sony's and Sharp's ATRAC.

This concludes my article. Once again I invite anyone (golden-ears especially) to listen to these recordings and report your findings on the real differences. If you are interested, please contact me at [email protected] and arrangements can be made.

Appendix: CD Selections

Artist: Thursday Diva, Album: Follow Me, Track 9: The Carpenter, Digital Music Productions 1995, Ref: CD-509

Artist: Roch Voisine, Album: Helene, Track 1: Helene, Les Disques Star Inc. 1989, Ref: STR-CD-8014

Return to the MiniDisc Community Page.