Our regular testing procedure uses two different sets of benchmark tests, focusing on several key areas of desktop performance. BAPCo's SysMark 2004 measures a system's overall application performance using real-world office-productivity and content-creation software. To determine a system's high-end 3D graphics capabilities, we run the popular Half-Life 2 and Doom 3 games at settings designed to push a PC's graphics subsystem to its limits (in a measurement to which gamers can relate: frames per second). And to give you an idea of a system's gaming performance with less demanding material, we also run these games at more conservative settings.
For systems that contain multiple-core processors, we run additional tests that focus on multimedia applications and multitasking. These are the areas that multiple-core processors are designed to excel at above and beyond their single-core brethren.
If a system does not come with a restore disc or a partition, we back up the system's hard disk before changing any settings or installing our benchmarks. We use a variety of tools to back up a system's hard disk, including Symantec Ghost Corporate Edition, Acronis True Image 8.0 Corporate Workstation, and PowerQuest's PartitionMagic 8.0.
We use a Dell PowerEdge 4600 file server to store the hard disk images of the systems we test. Additionally, we store our benchmark test files on the server and download the files from the server to the test systems. Our Dell PowerEdge 4600 uses two 2GHz Intel Xeon processors, 2GB of ECC DDR SDRAM, an integrated Dell Perc 3/Di RAID Controller, four Fujitsu Ultra160 hot-pluggable SCSI hard drives set to RAID 5 and providing 50GB of storage, two Seagate Ultra160 hot-pluggable SCSI hard drives set to RAID 1 as the boot partition, four 300W hot-pluggable redundant power supplies, integrated Fast Ethernet and Gigabit Ethernet network adapters, and Microsoft Windows 2000 Server.
Before running our tests, we disable all applications that launch at start-up and uninstall any programs that conflict with our tests, including apps that will be installed and used by the benchmarks. We set all power-management settings for maximum performance, with no components allowed to power down. We run Windows Update, then turn off automatic updating. We also disable Windows' System Restore, Error Reporting, Remote Assistance, and Remote Desktop. We run all tests with the desktop unattached to any network and with wireless networking (if any) disabled.
Occasionally, some desktops arrive for testing at CNET Labs with nonstandard system settings--such as overclocking of the CPU, GPU, or graphics memory--that can potentially affect the performance. As long as the system manufacturer can demonstrate that the test system's configuration is identical to how it is shipped to paying customers, we test systems with these nonstandard settings and specifically mention these settings in the review.
|BAPCo SysMark 2004|
The SysMark 2004 benchmark measures a system's overall performance using off-the-shelf applications. SysMark comprises two workload scenarios, reflecting usage models of mainstream business users ("office productivity") and Web designers ("Internet content creation"). The benchmark runs multiple applications simultaneously, continually switching between applications, with some tasks running in the background.
- The office-productivity scenario runs the following applications: Adobe Acrobat 5.0.5, Microsoft Access 2002 SP2, Microsoft Excel 2002 SP2, Microsoft Internet Explorer 6 SP1, Microsoft Outlook 2002 SP2, Microsoft PowerPoint 2002 SP2, Microsoft Word 2002 SP2, Network Associates McAfee VirusScan 7.0, ScanSoft Dragon NaturallySpeaking 6.0 Preferred, and WinZip Computing WinZip 8.1. File compression and virus scanning runs in the background.
- The Internet-content-creation scenario runs the following applications: Adobe After Effects 5.5, Adobe Photoshop 7.01, Adobe Premiere 6.5, Discreet 3ds Max 5.1, Macromedia Dreamweaver MX, Macromedia Flash MX, Microsoft Windows Media Encoder 9.0 series, Network Associates McAfee VirusScan 7.0, and WinZip Computing WinZip 8.1. Video encoding, WinZip, and VirusScan run in the background.
SysMark's scoring methodology is broken down into groups or subcategories, such as 3D creation and Web publication (we do not report these scores but use them for our analysis). SysMark determines each of these subcategory ratings by dividing the sum of the total time to perform all of the operations in that subcategory on the calibration system by the total time to perform these same operations on the test system, then multiplying by 100. SysMark's calibration platform is as follows: a 2GHz Pentium 4, Intel's D845GBV motherboard chipset, 512MB of 266MHz DDR SDRAM, an ATI Radeon 9700 Pro graphics card with 64MB of DDR memory, an ATA/100 hard drive using the NTFS file system, Windows XP Professional SP1, Windows' display properties set to 1,024x768 and 32-bit color at 75Hz, and Windows' power scheme set to Always On.
The office-productivity and Internet-content-creation scenario ratings, which we report, are generated by taking the geometric mean of the respective subcategory ratings. Finally, the SysMark 2004 rating, which we also report, is the geometric mean of the Internet-content-creation and office-productivity scenario ratings; higher scores are better. A rating of 100 means the test system's SysMark performance equals that of the calibration platform. Scores above or below 100 indicate performance relative to the calibration platform.
Depending on the class of the system, we may report only the office-productivity or Internet-content-creation portions of SysMark.
We run SysMark with Windows' display properties set to a resolution of 1,024x768 and a 32-bit color depth. (Systems with CRT monitors have the refresh rate set to 85Hz; systems with LCDs are set to 60Hz.)
After installing the game, we download and install the latest updates from Valve. We then make the following settings within the game:
- Combat difficulty: Easy
- Enabled developer console (~): Enabled
- Aspect ratio: Normal (4:3)
- Display mode: Full screen
- Model detail: High
- Texture detail: High
- Water detail: Reflect all
- Shadow detail: High
- Shader detail: High
- Wait for vertical sync: Disabled
- Video quality: High quality
- Full screen: Yes
- High-quality special effects: Yes
- Enable shadows: Yes
- Enable specular: Yes
- Enable bump maps: Yes
- Vertical sync: No
|Multimedia performance tests|
Increasing the clock speed of a CPU is not the only way to eke out faster performance from a processor. In fact, both AMD and Intel are moving away from the gigahertz war and focusing their attention on producing processors with multiple cores. This approach spreads the processing power around to multiple processing cores, so that multiple simultaneous tasks can execute quicker. Multimedia applications, in particular, are well suited for such a scenario. However, for a single application to take advantage of a multicore environment, the application must be designed specifically to be multithreaded. Unfortunately, not many of today's mainstream multimedia applications are multithreaded and therefore are not specifically optimized to take advantage of multicore technology. As multicore processors become more prevalent in mainstream systems, the availability of mainstream multithreaded applications should increase significantly.
Systems that utilize multicore processors are put through a few additional paces on our tests. We use a collection of multithreaded and nonmultithreaded multimedia applications to determine how well multicore systems perform multimedia-type workloads, such as ripping MP3 files and encoding video files. We also include a multitasking test where two different applications are working simultaneously in order to see how well a multicore system can handle such a workload.
We use McAfee VirusScan 9 and DivX 6.1.1 together to test the multitasking capabilities of a multicore system. Using the latest updates and VirusScan's default settings, we scan 42.1GB worth of files on the system's hard drive. While the virus scan takes place in the background, Dr. DivX converts a 2-minute, 1-second VOB file, also stored on the system's hard drive, to a high-quality DivX file. The Dr. DivX video encoding takes place in the foreground. We start the virus scan first, then start the video encoding immediately after, timing how long it takes for the system to perform the completion of all tasks from start to finish.
Using our own custom Action file, we time how long it takes for Adobe Photoshop CS to execute the Action file on a collection of 15 JPEG and TIFF files, ranging in size from 1.83MB to 49.2MB. The Action file represents a Web-image production work flow, by performing such tasks as converting color bit-depths, resizing images, applying a number of Photoshop's built-in filters, and exporting the resulting images as moderately compressed JPEG files.
Using iTunes version 220.127.116.11, we time how long it takes to rip 19 audio tracks to 192Kbps MP3 files. So as to better isolate the system's CPU and eliminate the optical drive as a potential bottleneck, we cheat a bit with this test: instead of ripping directly from an audio CD, we rip from WAV files already stored on the system's hard drive.
Sorenson Squeeze video-encoding test
Using our own custom project file, we time how long it takes Sorenson Squeeze 4.0.301.11 to convert a 30-second DV AVI file to MPEG-2 and MPEG-4 files. Our source AVI file was originally captured from a DV camera and its attributes are:
- Frame size: 720x480 pixels
- Video data rate: 1,536Kbps
- Frame rate: 29fps
- Audio: 16-bit
For the MPEG-2 file creation, we use Sorenson Squeeze's default DVD_NTSC_Lg selection.
For the MPEG-4 file creation, we use Sorenson Squeeze's default 384K_Stream selection, with the following deviations:
- Frame size: 480x360
- Video data rate: 1,000Kbps
- Audio data rate: 128Kbps
- Audio sample rate: 44100
- Audio Channels: stereo
- Video method: Sorenson 2-Pass VBP
In addition to the benchmark tests mentioned above, we run additional tests that are designed to evaluate the performance of the CPU, the memory, and the hard disk subsystems of a Windows desktop. If any of the results from these additional tests are especially relevant to a particular system, we comment on these findings in the review. We are constantly evaluating new tools to assist us in this process. At present, the benchmark we use for our anecdotal testing is SiSoftware Sandra 2005.