Kurt Seifried, [email protected]
As you've probably heard by now, Apache.org was broken into last week, and the system was compromised. How this happened is much more interesting then your usual "known hole in an old piece of software" scenario (although this was the final nail in the coffin for Apache.org).
Most break-ins we hear about are relatively straightforward -- someone was running an unpatched copy of IIS 5.0 or they forgot to upgrade their FTP server to a recent version. These types of problems are relatively easy to deal with, you keep software up to date, but they are impossible to completely eliminate. At some point you have to run software to make the system useful, and even the most heavily audited and secured system will have bugs in it's software. It is inevitable that at some point an attacker will discover a problem and use it, or release code publicly allowing anyone to exploit the flaw, and software vendors will react, provide a fix, but this still needs to be applied by system administrators.
However, an old issue is becoming increasingly problematic as more machines are placed online. The problem is trust.
From dictionary.com we get several definitions for "trust", depending on usage, but the one I will be dealing with primarily is this:
Firm reliance on the integrity, ability, or character of a person or thing.
An example of this would be developers logging into the primary Apache.org development machine from remote locations. The SSH protocol is used to secure these connections with strong encryption, which provides a tunnel between the two communicating machines. Furthermore, it is assumed that the end developer's machine is secure, and that there are no keystroke loggers running, or items like KeyGhost hooked up to the machine. Herein lies a problem. More and more people are using machines that are not always secure or should not be considered "trusted".
The number of publicly available terminals in libraries, educational computer labs, cafe's and other places has exploded in the last few years. The vast majority of these machines are not very well secured, ranging from Linux machines in a private cubicle (where LILO was not locked down) to windows machines that will cheerfully boot from a floppy disk.
While breaking into these systems and logging passwords is probably not going to help you break into a specific site, if you throw out a large enough net you will catch something of interest eventually. This is especially true for more populous systems such as ISP shell servers and university servers (which are notorious for being poorly secured). Once you have a shell account on a system it becomes much easier to exploit any security flaws, local or remote.
The number of passwords being entered into insecure machines and sent to insecure machines will only continue to grow, and once an attacker gains administrative or root access on a machine it is trivial to log all passwords (using modified versions of ssh clients and daemons for example). Harvesting passwords in this manner is relatively easy (choose poorly secured and maintained systems in a foreign country) and it only gets easier with time, since many people have a tendency to use the same password for multiple systems.
If you see someone logging into a server with the password "NBd72ts" chances are probably good that it's the same password to log into the machine they came from. This process can even be automated by a savvy attacker -- software loads onto a system, breaks root with any number of well known local exploits, then attempts to connect back to machines people log in from, using their passwords, breaks root with any number of well known local exploits, and so on and so forth. Sending passwords back to it's master is not to hard either, by encrypting them and posting them to newsgroups, public mailing lists, online website and so forth where the attacker can easily collect them anonymously.
An attacker "broke in" using a developer's password and then managed to exploit a security hole in the SSH server that had accidentally gone unfixed (the administrator had created new SSH binaries but failed to install them properly). This developer's password was harvested from SourceForge, which the attacker had broken into. So there were three main elements that contributed to the incident:
And several lesser elements:
Brian is President of the Apache Software Foundation, and responsible for the infrastructure.
Kurt: Do you currently have guidelines, or plan to implement any, with respect to security on developer machines? For example TAMU used to require people to run cops and pass all its checks before they would allow machines to go online.
Brian: Since we only have one box (today), the guidelines have always been "do what you can to keep it secure" -- and nothing more formal. The default FreeBSD security auditing programs have been pretty successful in detecting hacks; there is always more we can do, though.
Kurt: And what about the workstations and systems used by developers on their end?
Brian: Nope, no standards; we trust people to do the Right Thing, but now are getting large enough that defining that might be worthwhile. Someone is working on a security standards for developers document, this should probably be added.
Kurt: What is/are the biggest problem(s) with removing people's shell access to the servers, what solutions are being implemented or looked at?
Brian: It's simply that so many of the tools we use today presume shell access is easy to give, and are thus designed to work best with it. For example, CVS, updating web pages, and managing mailing lists, all (at this point) are all managed by editing files locally. There are non-shell solutions to each, but getting them all to work together consistently (so you have one view of who a developer is, etc) is a huge challenge - one my company has been focusing on with our Tigris project (www.tigris.org). I'd say of all of them CVS is probably the biggest pain, but there are solutions.
Kurt: Are you considering the use of one time passwords, tokens or smartcards for shell access to the server?
Brian: Have not considered that, but could. In fact that would even make logging in from keystroke-capturing cybercafes safe, wouldn't it? Good idea. Wish there was a way to make it optional, though making it optional is probably missing the point.
I guess this is the eternal struggle between ease of use and security; sometimes we lean too far towards the ease-of-use side, and get burned. There's also the cost-to-admin issue - since all resources available to Apache are donations of time and effort, the simplest solution is usually the one taken. Again, not always the wisest choice.
Kurt: With the increase in public and semi-public systems (like SourceForge and ISP servers) do you think there will be more attacks like this one?
Brian: If I were a cracker and had gotten root on a box, I'd certainly want to be able to get root on more, and compromising sshd & ssh is probably the best way to at least get a shell these days, and usually from a shell there's at least one or two root compromises available. So yeah, this is probably going to increase (we'll probably be dealing with the aftereffects of all those hacked passwords for awhile anyways), and finding ways to avoid needing shells (or better containing people who use shells, e.g. using jails or capabilities) is probably going to be a pretty fruitful area of security research.
The most obvious solution would be to remove shell access where possible, so that even if an attacker manages to grab a password, exploiting the system is much more difficult.
Unfortunately, this is the only easy answer, and not available to everyone. If you must provide shell access (because you are selling it, for example) the best thing to do would be to lock down the system as much as possible, removing as many setuid and setgid binaries is a good start. Restricting what users can do through a restrictive shell, chroot'ing, subdomain, tight file permissions and so forth is also recommended and can limit any damage. Programs like tripwire can monitor binaries such as ssh and sshd, so if an attacker replaces them with trojan'ed versions you will hopefully be notified promptly assuming of course the attacker doesn't intercept the alert and quash it. Of course you need to pay attention to your alerts and act upon them quickly otherwise they will do no good. Reducing the reliance upon passwords will also help, however most solutions such as tokens, smartcards, biometric authentication and so on tend to be relatively expensive. As long as people can remotely access a system without restrictions (such as using tcp_wrappers to restrict which IP's and domains they can log in from) you essentially trust every machine on the Internet.
Reference links
Information about the hack, from Apache.org - http://www.apache.org/info/20010519-hack.html
Last updated 4/10/2001
Copyright Kurt Seifried 2001