Running lights out management without putting your organization’s lights out permanently
LOM is a potent technology which has its uses; however it also poses some potential risks which every enterprise must be aware of.
By Philip Lieberman.
Intelligent Platform Management Interface (IPMI) technology underpins lights out management (LOM) in IT departments around the world. LOM allows an IT administrator or IT security manager to manipulate and manage servers using remote control - even switching on the machines when they are ‘off’.
LOM usually provides access to the keyboard, video, and mouse over LAN. The so-called ‘lights out’ cards are used by many manufacturers of computer servers and corporate workstations. Dell sells them as Dell Remote Access Cards (DRAC), HP refers to them as Integrated Lights Out cards (ILOs). No matter what the label, it is a potent technology which has its uses. However it also poses some potential threats which every enterprise must be aware of.
Even though each vendor has a different implementation and name for their lights out management at a high level they all offer the same basic level of features.
I used to work in an office where we called the LOM process, "working from the beach" and we used to deliberately design our systems so that we could work on them as if we were literally on the beach. Since the organization we worked for had servers at every point in the globe, which we were supposed to manage, we operated on the principle that a server in West Ealing should get the same love and attention we lavished on the server in Iceland.
Now it was easier to drop into the server in West Ealing than it was on the latter in Iceland, but the principle for us was the same. We would rather work from the beach and our beloved organization gave us the ability to do so easily. The technology was great - instead of multiple devices, each with its own IP address and management interface, now we had a single port with a single IP that you can use to manage everything. Heaven!
Our need to make our jobs easier became our mother of invention and what had started as a practical step to ensure that we could administrate far-off servers became something which we started to rely on for almost every server. Unfortunately, as many organizations find to their cost long after staff who set up LOM have moved on, this process has its downsides if it’s not managed properly.
IPMI allows admin staff, while sitting at their own terminals anywhere in the world, to log in and take control of a server and perform tasks as if they were actually standing right in front of the screen. Staff can turn it on, turn it off, and interact with it. Most usefully of all they can manage the BIOS of the machine, install software on it, insert a CD, form a network share, and boot up from that. Almost anything an interactive user with physical access to the box could do, a remote user can do. These lights out management devices are providing keyboard, video, and mouse (KVM) over LAN functionality.
As long as you are the right person, and you know what you’re doing, it’s a great aptitude to have. However, put it into the wrong hands and you have an entirely different situation.
IPMI – How secure are you anyway?
Recently reports highlight that IPMI may have some fundamental flaws if it is not installed and managed properly and that, maybe, hackers could use it to infiltrate the network even if the device is turned off.
The bad guys have known about IPMI for years and to think that they won’t have used this back door entrance into the enterprise shows a lack of imagination. The hackers now find it easy to locate these devices on a network using free tools such as IPMIPing or a port scanner looking for port UDP 623.
The fact is that IPMI is a clear and present danger to every organization that installs this technology without rigorous controls and effective management of passwords and access rights.
It is not only the malicious insider that can do damage, all those who have been sacked from the company could retain this back door access to the company’s vital secrets simply if passwords are not changed regularly, as could a person who left on good terms and is now simply a curious outsider.
Seal up the gaps now!
No one is suggesting ditching this useful technology because it has a downside. That would be akin to suggesting we stop using mobile devices because they often get stolen
The danger is not that you have IPMI, or that the machine remains plugged in to a power socket. The problem is how you manage the credentials and access to these machines — because remember you may have thousands of them — and one security policy is worth a thousand breached machines.
One major mistake many organizations make is to leave the device with its default passwords unchanged. It is common knowledge that Dell delivers its DRAC cards with the default account ‘root’, and the password ‘calvin’. Unfortunately most companies leave the default password as is. It’s a fair assumption that hackers know this as well. For SuperMicro server, the IPMI credentials are ‘ADMIN’ for the username and ‘ADMIN’ for the password. If this knowledge alone doesn't make you change your passwords then the information that these lists of default passwords are easily available on the Internet should make you do so.
The HP users amongst you will be sitting there thinking they can be applauded as at least the old industry veteran sets a random password on each device. However, if it then never gets changed it still poses a risk from current or ex-employees with access to that password. For HP ILO, the default username is ‘administrator’ and the default password is printed on the device – again unless it is changed.
Where it goes wrong...
The main failure of most organizations, and this isn’t just confined to IPMI but to many high-tech solutions, is that organizations do not control the credentials for these devices adequately. If everyone uses the same log-in details then there is absolutely no accountability and you will never know who used the password that is accessible to everyone.
In areas where the regulators are not completely asleep at the wheel, they're beginning to understand the reality of the risks posed by these ‘super user’ accounts. They are beginning to stipulate that organizations cannot leave these systems with the factory default setting, and managing the credentials of the IPMI devices is becoming a necessary requirement for all regulatory frameworks. However, the fact is that most companies don't admit their lack of compliance nor their corporate failure to properly manage these devices.
Failure to manage devices such as these can put you in a big regulatory mess when your auditor discovers that your systems are wide-open to manipulation even when they are switched off. However, this often doesn’t stop enterprises carrying on as if they knew nothing about the dangers. Even companies such as Sony are finding it hard to regain their reputations in the face of massive data breaches and huge losses.
This is also an important factor to be considered if you are using cloud providers. When you are looking at the service level agreements (SLAs), look closely at what they say about their own critical infrastructure because most cloud vendors extensively use lights-out management. If you are a cloud customer, you may be shocked at what you find when you ask your cloud vendor how they manage the credentials in the IPMI devices.
IPMI is a sound technology - it just needs to be secure. Rather than operating a scorched earth IT policy organisations need to:
The most secure situation to engineer is to make sure that when someone wants to get access to a machine that the request is checked and the individual authorised. For access to highly sensitive systems or crucial adjustments, another individual should have to approve the request.
Having granted access to the device (and therefore disclosing the password), a session timer should be used that terminates the validity of the credentials after a set period. The final piece is that all of these elements should be fully audited.
You must manage all passwords for IPMI devices thoroughly with a tried and tested process. You must ensure that you have a process in place to change the password of an IPMI device if it has been disclosed to anyone and the specific need for access (i.e. repair, patch, etc) has been completed. This means that IPMI credentials should be changed within hours of their disclosure.
Changing the default password is okay, but if everyone in the relevant department has a spreadsheet with the new values for the lights-out devices, you may never know who had access or who did what, and never find out the reason for the change, no matter how malicious. Many organizations which use IPMI devices may not be aware that there are solutions to provide randomization of these credentials automatically as well as providing an easy, attributable, time-limited and controlled access these devices.
IPMI is a fundamentally sound technology, but a fundamentally dangerous one in the hands of an IT department which does not have the correct security policies in place. As long as you ensure that you have these policies and practices in place it will continue to be a valuable tool for your organization.
Author: Philip Lieberman, Lieberman Software. For more information on Lieberman Software and its solutions, please visit www.liebsoft.com
•Date: 9th Oct 2012 • World •Type: Article • Topic: ISM