 | Voice over IP |
 | |
Voice over Internet Protocol (VoIP) phone services boast unquestionably more features than what you get with a traditional phone service. Voice quality is another matter, however. In order to provide you with information about how phone calls sound with a particular VoIP service, CNET Labs has developed a battery of tests to complement the anecdotal tests performed by the reviewer. With a VoIP service, you are eschewing your regular phone line in favor of sending voice traffic over your broadband Internet connection. Some services are more adept at directing both voice and data traffic over the same network. To test how well a VoIP service maintains this balance, we place numerous calls while uploading and downloading files on our network, and we make calls while the network is otherwise quiet, as well. Throughout our tests, we listen to hear how voice quality is affected on both ends of the line, and we monitor the throughput during our data transfers.
Test environment
We test VoIP services using the telephone adapter (TA) that comes bundled with the VoIP service subscription. The TA is tested on a residential broadband cable connection: Time Warner Cable of New York City's Road Runner service, which features throughputs of 5Mbps downstream and 384Kbps upstream. We install the TA as per the service provider's instructions. Some TAs connect directly to the broadband modem, while others connect to an available Ethernet connection on a router. The router we use for testing is a Linksys WRT54G. Some TAs include integrated routers, which connect directly to the broadband modem--in these cases we use the TA's integrated router instead of our Linksys WRT54G.
All test calls are made between the VoIP service being tested and an analog (landline) phone connection. The telephone instruments used during testing (on both ends of our calls) are wired telephones. This ensures the cleanest possible connection by minimizing outside variables that could adversely impact audio quality.
As broadband Internet connection throughputs can fluctuate, we test the quality of the upstream and downstream connections before we conduct our formal VoIP testing. If the broadband connection's throughput appears to be adversely affected, we postpone our testing session until throughput levels are acceptable. We avoid testing during known periods of high Internet usage, such as weekdays between the hours of 4 p.m. and 6 p.m. ET.
For our tests that involve generating network traffic in the form of data uploads and downloads, we use a desktop system configured with a 2.8GHz P4 CPU, 512MB of DDR memory, an Intel 865PE chipset, a Western Digital Caviar SE 80GB 7,200rpm EIDE hard drive, an AGP-based Nvidia GeForce FX 5200 64MB graphics card; the system runs Windows XP Professional SP2. Unless a particular test involves generating network traffic other than that of the VoIP calls, the PC stays powered off. During such tests, the only devices permitted to transmit and receive network traffic are the TA, the broadband modem, and the router. We refer to this as baseline conditions.
Voice-quality tests
To establish the voice quality of the VoIP service and the bundled TA under baseline conditions, we make and receive numerous telephone calls using the VoIP service. The primary criterion we look for is the level of clarity of the voices on both ends of the calls--essentially, we ask ourselves how close the audio quality comes to matching that of a typical analog phone connection. We also listen for any echoes, clipping of words, tone or volume modulations, and any noticeable latency. Additionally, we also gauge how perceptible any background noise is and under what conditions it is most noticeable.
Most residential and small-business broadband connections don't have enough upstream throughput to support both voice and data packets simultaneously. The end result is that under those circumstances when you are sending large amounts of data from your PC (such as uploading photographs to an online photo-finishing service) while using the VoIP service, the audio quality of your call will probably be adversely affected. Since most broadband connections have a high enough downstream throughput, you'll likely hear the person on the other end just fine, but the person you're talking to will have great difficulty hearing you intelligibly: whole words and sentences will drop out. Therefore, we also evaluate how much the VoIP audio quality degrades during data uploads, and we test whether it is impacted at all by data downloads.
The voice-quality rating we provide is a weighted average of how good the VoIP service's audio sounds under baseline conditions, as well as during data uploads and data downloads. Baseline conditions are given the highest weight of 66 percent; audio quality during data uploads and data downloads each factor 17 percent of the weightings. The scale for the voice-quality ratings is from 0 to 10, with a perfect score of 10 equaling our reference analog connection.
File-transfer tests
The data uploads are conducted by downloading a 5.51MB MP3 file from Download.com. The data uploads are conducted by uploading a 1.37MB JPEG photo up to Kodak EasyShare Gallery. We time how long it takes to perform these file transfers when a VoIP call is active and compare that to how long it takes to perform these file transfers when there isn't a VoIP call taking place. Any degradation of data throughput while VoIP calls are active is an indication that a TA is at least somewhat successful in giving priority to the voice packets in an attempt to minimize the loss of audio quality--this is commonly referred to as Quality of Service (QoS). Typically, the slower you see your data throughput drop during VoIP calls--especially the upstream data throughput--the better the TA and the VoIP service are at prioritizing the voice packets and the less likely you are to experience any degradation of audio quality. Because of the limited upstream bandwidth of most residential and small-business broadband connections, this is a necessary compromise. Unfortunately, most TAs and VoIP services are not yet sophisticated enough to find the right balance, with the most common result being a significant degradation of audio quality during data uploads.
Inband-ringing tests
We also test how long it takes to connect a dialed phone number. We time how long it takes to connect a call from the moment the last digit is pressed to the moment we hear ringing. We compare the VoIP services against each other and use the 2-second connection time of our analog reference line as our reference point.
Test environment
We test VoIP services using the telephone adapter (TA) that comes bundled with the VoIP service subscription. The TA is tested on a residential broadband cable connection: Time Warner Cable of New York City's Road Runner service, which features throughputs of 5Mbps downstream and 384Kbps upstream. We install the TA as per the service provider's instructions. Some TAs connect directly to the broadband modem, while others connect to an available Ethernet connection on a router. The router we use for testing is a Linksys WRT54G. Some TAs include integrated routers, which connect directly to the broadband modem--in these cases we use the TA's integrated router instead of our Linksys WRT54G.
All test calls are made between the VoIP service being tested and an analog (landline) phone connection. The telephone instruments used during testing (on both ends of our calls) are wired telephones. This ensures the cleanest possible connection by minimizing outside variables that could adversely impact audio quality.
As broadband Internet connection throughputs can fluctuate, we test the quality of the upstream and downstream connections before we conduct our formal VoIP testing. If the broadband connection's throughput appears to be adversely affected, we postpone our testing session until throughput levels are acceptable. We avoid testing during known periods of high Internet usage, such as weekdays between the hours of 4 p.m. and 6 p.m. ET.
For our tests that involve generating network traffic in the form of data uploads and downloads, we use a desktop system configured with a 2.8GHz P4 CPU, 512MB of DDR memory, an Intel 865PE chipset, a Western Digital Caviar SE 80GB 7,200rpm EIDE hard drive, an AGP-based Nvidia GeForce FX 5200 64MB graphics card; the system runs Windows XP Professional SP2. Unless a particular test involves generating network traffic other than that of the VoIP calls, the PC stays powered off. During such tests, the only devices permitted to transmit and receive network traffic are the TA, the broadband modem, and the router. We refer to this as baseline conditions.
Voice-quality tests
To establish the voice quality of the VoIP service and the bundled TA under baseline conditions, we make and receive numerous telephone calls using the VoIP service. The primary criterion we look for is the level of clarity of the voices on both ends of the calls--essentially, we ask ourselves how close the audio quality comes to matching that of a typical analog phone connection. We also listen for any echoes, clipping of words, tone or volume modulations, and any noticeable latency. Additionally, we also gauge how perceptible any background noise is and under what conditions it is most noticeable.
Most residential and small-business broadband connections don't have enough upstream throughput to support both voice and data packets simultaneously. The end result is that under those circumstances when you are sending large amounts of data from your PC (such as uploading photographs to an online photo-finishing service) while using the VoIP service, the audio quality of your call will probably be adversely affected. Since most broadband connections have a high enough downstream throughput, you'll likely hear the person on the other end just fine, but the person you're talking to will have great difficulty hearing you intelligibly: whole words and sentences will drop out. Therefore, we also evaluate how much the VoIP audio quality degrades during data uploads, and we test whether it is impacted at all by data downloads.
The voice-quality rating we provide is a weighted average of how good the VoIP service's audio sounds under baseline conditions, as well as during data uploads and data downloads. Baseline conditions are given the highest weight of 66 percent; audio quality during data uploads and data downloads each factor 17 percent of the weightings. The scale for the voice-quality ratings is from 0 to 10, with a perfect score of 10 equaling our reference analog connection.
File-transfer tests
The data uploads are conducted by downloading a 5.51MB MP3 file from Download.com. The data uploads are conducted by uploading a 1.37MB JPEG photo up to Kodak EasyShare Gallery. We time how long it takes to perform these file transfers when a VoIP call is active and compare that to how long it takes to perform these file transfers when there isn't a VoIP call taking place. Any degradation of data throughput while VoIP calls are active is an indication that a TA is at least somewhat successful in giving priority to the voice packets in an attempt to minimize the loss of audio quality--this is commonly referred to as Quality of Service (QoS). Typically, the slower you see your data throughput drop during VoIP calls--especially the upstream data throughput--the better the TA and the VoIP service are at prioritizing the voice packets and the less likely you are to experience any degradation of audio quality. Because of the limited upstream bandwidth of most residential and small-business broadband connections, this is a necessary compromise. Unfortunately, most TAs and VoIP services are not yet sophisticated enough to find the right balance, with the most common result being a significant degradation of audio quality during data uploads.
Inband-ringing tests
We also test how long it takes to connect a dialed phone number. We time how long it takes to connect a call from the moment the last digit is pressed to the moment we hear ringing. We compare the VoIP services against each other and use the 2-second connection time of our analog reference line as our reference point.
