A "back door" in computing terms is a method that hackers use to circumvent a system's authentication features and gain access without being detected. Usually this involves taking advantage of bugs in the built-in sharing services and OS features, but it also can happen if a user inadvertently installs some malware that provides a path around the system's security.
Anytime you start a sharing service on your computer, be it for files, screen sharing, chatting, or printers, you are technically opening a door for a client application running on remote system to connect and change or use aspects of your system. For instance, when you enable Web sharing, then a Web browser client on another computer can connect and read HTML Web pages that the server has made available. These sharing services run as background tasks and usually first authenticate and authorize users and connections based on the system's security measures (recognizing accounts and permissions limits).
While these services are built to be legitimate and productive features of an OS, bugs in them may provide a route that hackers can take to open back doors in the system's security and gain access to the system. These holes are rare and are usually patched quickly by Apple or legitimate third-party developers when found, but besides taking advantage of built-in services one method that hackers can use is to trick users into installing a malware service that runs hidden in the background and allows the hacker entry into the system.
A new Trojan horse security threat has recently surfaced that has been described as a backdoor Trojan for OS X. The malware allows an attacker to connect to a system using a client application and perform tasks like shutting it down, restarting it, creating files on the desktop, opening URLs in a Web client, requesting administrative passwords, and messaging the current user.
As with any Trojan horse program, the user inadvertently installs the application thinking it is a legitimate package, but instead of being a standalone program that alters configurations (like the DNSChanger Trojan) or sends data to remote systems (like a botnet hack), this Trojan installs a server on your system that allows a hacker to connect and administer the system with a small remote client program (called a remote administration tool, or RAT). The hacker enters your IP address into his client, connects to the malware service installed on your system, and can then send remote commands to your system using the RAT system.
This malware is very similar to the age-old NetBus and Sub7 RATs for Windows, and can ultimately be characterized more as a prank application than anything else (though there are legitimate RAT services, including Apple's Server Admin tools). Nevertheless, it does still pose a security risk if installed because users can be tricked into supplying their administrative passwords to the hacker, among other things.
Does this change the nature of OS X security? Absolutely not, and given the measures required to install and enable this threat, it is ultimately a very low risk. While there is always concern that OS X's security features can be circumvented and result in malware being automatically installed, so far this malware, as with most other Trojans, requires you to manually run an installer to load a separate standalone program. The supplied OS features and services are not touched and their security measures are left intact.
Because you need to install the program to put your system at risk, the simplest and easiest way to avoid it and other similar Trojan threats is to never run an installer or other program unless you know exactly where it came from. Depending on your familiarity with computers this may be difficult to tell, so your next-best bet is to keep away from underground Web sites and any online deals that seem to be too good to be true, especially if the sites require a program to be installed to view their offerings.
If you are uncertain about your ability to identify hack attempts, install a malware scanner and have it watch your Downloads folder so any new files added to this folder are immediately scanned. Also go to Safari's General settings and uncheck the option to open "safe" files after downloading, and set your browsers to download files into the watched downloads folder. Some antivirus tools have on-access scanning features, but these are likely not yet necessary for OS X and may cause compatibility and performance problems. Therefore, disable these features unless you specifically need them, and then set them up to watch or manually scan a common Downloads folder. See this article for a list of antivirus software recommendations for OS X.
In addition to scanning with a robust and updated malware scanner, there are other ways to protect your system. Because this malware appears to be a standard client-server program that uses basic IP connectivity, even if it is running on your system it will be nearly impossible for a hacker to use if your system is behind a properly configured network firewall. Most modern home and workplace routers have robust NAT firewalls with numerous extra security features (such as stealth modes and flood detection), so be sure your network is protected by one. Additionally, check your router and disable any unused ports and DMZ hosting.
Lastly, be sure to enable the OS X firewall and regularly clear the list of applications allowed through the firewall (found in the Security system preferences in the Firewall tab). This will ensure you only allow the programs you currently use through the firewall, and are notified of other, less common ones that might be requesting network access. In addition, while the built-in firewall blocks incoming traffic, it does not block outgoing traffic, so you might consider installing a program like Little Snitch to detect when applications on your system send information out to the Internet.