Typing in login names and passwords has, for many years, been the most common form of authentication in IT environments with normal protection requirements. Alternatives such as tokens, smart cards, electronic cards, and various types of biometrics have not changed this. There is virtually no alternative to passwords: Low implementation costs, sufficiently high user acceptance, and the relative rarity of significant security incidents suggest that passwords are unlikely to become extinct in the near future.

Emergency password management covers situations in which third parties not directly involved in service operations need access to systems under exceptional circumstances to prevent greater damage. For example, the objective could be for a 24/7 security team to access compromised servers, even if the system administrator is not available.

In Case of Emergency

The classic solution to this problem is a list of passwords in a sealed envelope that is deposited in a vault and handed over when an emergency occurs. Conventional solutions like this, however, do not scale adequately: Now larger organizations and data centers need to deposit not just a handful but dozens or hundreds of passwords, and regular password changes are required not only for password policies based on ISO/IEC 27001 but also, for example, in case of staff changes.

Maintaining a stored list of passwords thus evolves from a subjectively annoying chore to an objective time waster. Migrating emergency password management to a centralized, server-based software solution that can be used from any workstation offers many benefits but also incurs many security risks and needs to be well considered because of its importance.

This article examines the opportunities and risks and derives specific selection criteria for centralized password management products. The practical implementation is discussed using the Leibniz data center (LRZ) as an example. The LRZ is the central IT service provider for Munich universities and colleges; it provides services to approximately 130,000 university staff and students and operates the Munich Scientific Network with around 110,000 devices.

To manage the mass of passwords for servers and services operated at the LRZ, the commercial password manager Password Safe Enterprise Edition (PSE) by Mateso was introduced. PSE quite elegantly meets many requirements but also has some weaknesses, which we hope will be fixed in future versions.

Why Centralized Password Management?

Although many regular users are rather careless with their passwords – think of the infamous yellow sticky notes on monitors – most administrators take care to uphold their secrecy. Many admins would prefer not to share passwords with others and manage them exclusively offline, for example, using a password list on a piece of paper that they carry around in a wallet or lock away in some item of office furniture.

With some effort, both the number of passwords to manage and the need to share passwords can be minimized. For example, SSH keypairs replace passwords for authentication in remote login scenarios on Linux servers. Personal, non-administrative identifiers in combination with the use of sudo on Linux, or UAC on Windows, make shared knowledge of the root or administrator password unnecessary.

The problem of how to define emergency passwords still persists; every organization wants to continue managing its servers even if the usual administrator is not available. Added to that is the necessity of password-sharing for services that do not support role authorization models with multiple administrators.

Many administrators use personal password management tools like KeePass or KeePassX, where all of their passwords are stored in a file and encrypted with a personal master password, so that no one else can access it. With this approach, passwords can be easily entered via copy and paste, which avoids the media gap resulting from writing passwords on paper. Many tools also directly start an SSH or RDP connection with automatic password entry; the process is thus similar to storing passwords in a web browser.

Doubts About a Central Control Center

When it comes to depositing passwords centrally in an organization-wide database, there is often much skepticism, and it is not entirely unjustified. Offline variants, such as the password list in a sealed envelope in a vault, are still often well accepted. In practice, however, the vault solution has many disadvantages. For example, it cannot readily be known who has viewed which password; even complete password lists may have been copied.

Another question is whether the normal administrator has been notified of access to his or her password and with what kind of delay. Additionally, access is only possible to the vault if the key owner is available, which is not always the case and ultimately only shifts the administrator’s problem. And, if passwords must be changed regularly, the vault simply becomes a paper file, thereby adding administrative overhead.

Software Solutions at an Advantage

In contrast, a centralized software-based solution offers a number of advantages: The stored passwords can be accessed by the authorized party at any time and without a media gap; logging and reporting show who was given which passwords and when; and it is easier to modify a stored password digitally than with paper, envelopes, and a central vault.

However, it is precisely this assumed ease of access to the content of a software vault that fuels the concerns of the administrators using it. What happens if someone can simply retrieve all the passwords because of a vulnerability in the vault software? Does capturing a single administrative password mean the attacker can grab the entire database? Such new risks must be considered when implementing centralized password management tools. Of course, the software vendor is also aware of these concerns and can address them in organization-wide centralized solutions.

The desire to be informed reliably whether someone is given access to a password means it’s logical for password management to be client-server based: The password management software must be a trusted intermediate layer, which cannot be circumvented by someone, for example, copying the entire password database and using it for their own purposes. This also means that you must be able to trust the administrators of the central password management services and that the functionality of this service must be maintained even under adverse circumstances.

When you take a closer look at the relevant software products, the high-availability/redundancy features are what separate the wheat from the chaff. When considering PSE, for example, it is possible automatically to mirror a database with all your data on a second server.

Guess My Name

Because it is undesirable for all passwords to be viewed by all users of the software, it is necessary to introduce an appropriately granular authorization concept. Many software solutions, including PSE, essentially provide the permission levels “read” and “write,” wherein setting both permissions is equivalent to full access.

Often, other permissions govern, for example, printing, deleting, or exporting contacts via a dedicated client software, as well as the authority to assign permissions, as shown in Figure 1.

Figure 1: Managing authorization levels in the PSE system.

However, one typically desired permission level is often missing: “read and alert.” This level also must be implemented in some other way depending on the product and its philosophy.

This authorization level is crucial for emergency password management. Users who need to access their passwords in an emergency are not entitled to do so in everyday operation. Therefore, “read” would give them too much leeway, contradicting the principle that users should be assigned only minimum rights. The “read and alert” level restricts permissions to the extent that, although read access is still possible at any time, notification is sent to those responsible. To minimize the risk of further abuse, it should be an option to enforce a two-pairs-of-eyes principle for certain password entries or for specific user groups.

Is the Seal Intact?

Formerly, the integrity of a seal indicated whether a message had been opened and thus whether the content had been read by unauthorized persons en route. The same principle is used by some password management systems to make it clear whether passwords have been displayed or used. To do so, you apply a virtual seal to a password entry. The password manager then registers access to the password and only returns the contents of the password entry if the seal is broken by the requester (Figure 2).

Figure 2: Seal-protecting a password entry in PSE.

In password managers with a sophisticated authorization system, such as PSE, you can specify which users are allowed to break the seal or which users are entitled to edit password entries that are under seal protection. This property is an important requirement when a regular password change is required. If this functionality were not present, every password change would mean first breaking the seal to change the password and then reapplying the seal with all its permission levels.

Finally, the question arises as to how much of the password manager’s functionality should be deployed on the server system and how much on the users’ systems. Approaches range from keeping the database and the authorization check on the server side to examples in which the entire program logic is in the hands of a web application.

Tasks such as securing availability and managing authorizations must be executed on the server side. A client-based implementation of the remaining functionalities makes it easier to customize the behavior of a password management solution, whereas a web application can be used by different operating systems and devices. Examples of possible behavioral adaptation include automatic opening of SSH or RDP connections, without needing to display the password to the user. Client-based solutions generally allow for the creation of individual settings for each user.

Notifications

Notifications are an essential requirement for access to passwords in emergency password management. Password management solutions that have been developed explicitly for multiuser mode sometimes do not adequately implement this feature. For example, messages related to broken seals are sent only internally in PSE. This means you only see a notification if you are using the program for password management. In general, however, the software is only launched when needed; timely notification of the person responsible is not guaranteed.

Although it is possible to forward all messages sent in PSE to exactly one fixed predetermined email address, a more sensible approach would be to send these email messages to the administrators responsible for the password entries that have been accessed. The Python script shown in Listing 1 does this by reading email from a fixed IMAP account, parsing the messages, and forwarding them to the intended recipient.

Listing 1: Forwarding PSE Messages 

001 #!/usr/bin/python
002
003 import imaplib
004 import email
005 import email.mime.text
006 import re
007 import smtplib
008 import pdb
009 import logging
010
011 HOSTNAME = ''
012 USER = ''
013 PASS = ''
014 MAILFROM = ''
015 MAILOUT = ''
016 ADSDOMAIN = ''
017 FOLDER = "INBOX"
018 FOLDERPROCESSED = "INBOX/processed"
019 FOLDERADMIN = "INBOX/admin"
020 RERECIPIENT = re.compile("^Recipient: (?P<recipient>.+) \(.+$")
021 RESUBJECT = re.compile("^Subject: (?P<subject>.+)$")
022 REUID = re.compile('\d+ \(UID (?P<uid>\d+)\)')
023
024 def send_mail(recipient, subject, text):
025   logging.debug("Sending mail to %s" % recipient )
026   mail = email.mime.text.MIMEText(text, 'html', 'iso-8859-1')
027   mail["Subject"] = subject
028   mail["From"] = MAILFROM
029   mail["To"] = recipient
030   # reduced by error handling
031   smtp = smtplib.SMTP(MAILOUT, port='587')
032   smtp.ehlo()
033   smtp.starttls()
034   smtp.login(USER, PASS)
035   smtp.sendmail(MAILFROM, recipient, mail.as_string())
036   smtp.quit()
037   logging.info("Sent mail to %s" % recipient )
038
039 def move_message(imap, msgid, targetfolder):
040   try:
041     logging.debug("Moving Message %s to folder %s" % ( msgid, targetfolder ))
042     resp, data = imap.fetch(msgid, '(UID)')
043     match = REUID.match(data[0])
044     msguid = match.group('uid')
045     logging.debug("Found UID %s" % ( msguid ))
046     result = imap.uid('COPY', msguid, targetfolder)
047     if result[0] == "OK":
048       mov, data = imap.uid('STORE', msguid , '+FLAGS', '(\Deleted)')
049       logging.debug("Moved Message %s to folder %s" % ( msgid, targetfolder ))
050     else:
051       logging.warn("Moving message failed, result=%s" % str(result))
052   except Exception, e:
053     logging.warn("Exception in move_message: %s" % e)
054
055 def resolve_recipient(user):
056   return "%s@%s" % ( user, ADSDOMAIN)
057
058 def main():
059   logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(message)s')
060   imap = imaplib.IMAP4_SSL(HOSTNAME)
061   logging.debug("Connected to server %s" % HOSTNAME)
062   imap.login(USER, PASS)
063   logging.debug("Logged in as %s" % USER)
064   imap.select(FOLDER)
065   logging.debug("Changed to folder %s" % FOLDER)
066   resp, msglist = imap.search(None, "ALL")
067   if msglist == ['']:
068     logging.info("No messages found")
069     imap.logout()
070     return 0
071
072   logging.debug("Found %d messages" % len(msglist[0].split(' ')))
073   for msgnum in msglist[0].split(' '):
074     msgnum = int(msgnum)
075     logging.debug("Getting body for ID %d " % msgnum)
076     resp, ((msguid, msgbody), msgflags) = imap.fetch(msgnum, '(RFC822)')
077     try:
078       mimeobj = email.message_from_string(msgbody)
079     except:
080       logging.warn("Could not decode msgnum %d, skipping") % msgnum
081       continue
082
083     # Try parsing the message
084     # Valid mails
085     # - have Subject "New internal password manager message"
086
087     subject = mimeobj["Subject"]
088     if subject == 'New internal password manager message':
089       logging.debug("Message %d has my subject, processing" % msgnum)
090
091       payload = mimeobj.get_payload()
092       recipient = None
093       subject = None
094
095       for line in payload.split("<br>\r\n"):
096         match = RERECIPIENT.match(line)
097         if match:
098           recipient = match.group('recipient')
099
100         match = RESUBJECT.match(line)
101         if match:
102           subject = match.group('subject')
103
104       if recipient and subject:
105         logging.info("Found message to %s: Subject %s" % ( recipient, subject ))
106         if recipient == 'Administrator':
107           move_message(imap, msgnum, FOLDERADMIN)
108           continue
109
110         # Resolve recipient username to a email address
111         rrecipient = resolve_recipient(recipient)
112         logging.debug("Resolved %s to %s" % ( recipient, rrecipient ) )
113         send_mail(rrecipient, subject, payload)
114         move_message(imap, msgnum, FOLDERPROCESSED)
115
116   imap.expunge()
117   imap.logout()
118
119 if __name__ == '__main__':
120   main()

Additionally, the retrieved email messages are archived in a special folder. The script relies on the fact that the email sent by the PSE to a fixed destination address has a predefined structure and thus can be easily parsed to determine the desired target address for the notification.

Security Concerns

Many security concerns voiced by users can be attributed to a subjective feeling of security. For example, you can expect users to be reluctant to store an encrypted and sealed password in a server-based database, where it is exposed to many potential hazards, such as inadequate encryption, a backdoor, a bug in the product, or an attack that provides the ability to launch a remote exploit against the stored data.

In addition to these subjective security concerns, however, administrators must be quite aware that a central place where the majority of passwords and other access permissions are stored really does represent an attractive target for potential attackers. For this reason, it is important to investigate beforehand whether the selected software solution, and the system on which it is operated, meet your own security requirements.

Because you cannot, with any reasonable overhead, investigate closed source solutions for potential vulnerabilities or intentional backdoors, it is helpful to investigate the manufacturer’s security track record. Even open source solutions are usually not free of vulnerabilities, though; some projects are so extensive they cannot be analyzed sufficiently just by reading the code. On the other hand, organization-wide password management solutions are a specialized field that is rarely backed by large and exceptionally active open source communities. Thus, it is also advisable to discover what kind of response times the developers have exhibited in cases of disclosures in the past and what kinds of vulnerabilities needed to be fixed.

In their product descriptions, many manufacturers state that password entries are encrypted with the latest and most secure cryptographic algorithms and then stored in the database. Proving this assertion is, however, very difficult or even impossible. Even in open source solutions, validating the implementation takes a great deal of effort.

Note that the encrypted storage of password entries is only one of several important safety modules. At least as important is the reliable implementation of authorization management and authentication options that reflect your protection needs. For example, two-factor authentication prevents an attacker who has shoulder-surfed a user’s password from accessing all of that user’s password entries.

Deployment and Operation

In general, you will need to integrate the new software solution into your infrastructure. Thus, one important requirement for the password management solution is that you can import existing passwords into the database easily and without too much overhead. During everyday operation, automating the process of loading recently changed passwords into the database can make life much easier.

This aspect, however, is not so easy to implement. For example, it is impossible to script PSE directly, but you can use a CSV import to bring externally created password lists into the tool. However, these are just imported in their current state and with the default permissions for that location. If you want to seal the passwords or assign some other permissions, you need to do this manually – at least in PSE. Similar software products often lack a feature for editing imported passwords automatically.

Life is also easier if the password management software can perform regularly required password changes itself. On one hand, you need a random password generator that can be adapted to the organization-wide policies regarding password complexity. On the other hand, the software must log in to the managed systems and change the existing password, which requires some complex scripting options. For example, changing a root password on a Linux server via an SSH login is not entirely trivial if the root user cannot log in directly using SSH, and you need to use a non-privileged account in combination with sudo to do this.

To keep track of potentially hundreds of stored password entries, structuring options are needed; in most products, these are based on groups using an approach that’s comparable to creating folder structures on filesystems. Using these options to organize password entries intuitively and making it easy to find them again can be the subject of long discussions. The organizational structure, broken down by division into departments and operating groups, however, can serve as a basis. Figure 3 shows some services at the LRZ as an example.

Figure 3: A folder structure in PSE for passwords and a new password entry.

Alternatively, you could use a service catalog to which the individual servers are assigned. You might already have naming conventions for DNS entries or structuring for other systems, such as an organization-wide Active Directory that users are familiar with. In any case, the password management software should offer an appropriate search function; ideally, you should be able to define several logical views without needing to store password entries multiple times.

Guidelines for Permissions

The default permissions for new password entries also should be preconfigured. For example, you might want to grant all members of the operating team full access and grant the security team read permission for all new password entries in the “Mail Server” section. It should be possible to import the necessary group permissions, including read identifiers from Active Directory, so you can manage group memberships in one place. Automated sealing prevents race conditions when creating new password entries and keeps this important step from being forgotten.

Permissions should be of a temporary nature for situations in which someone assumes the responsibilities of another; otherwise, the risk is that someone assigned temporary privileges actually keeps them forever. As with any other software, two further selection criteria must be considered: The password management tool must be easy to operate, so you need to find a compromise between efficiency for power users who are constantly working with the tool and intuitive controls for casual users who might only need to deposit their changed passwords once a quarter. Additionally, the total cost of ownership for the password management solution – including hardware, software, administration, ongoing operation, and commercial support – must be appropriate to your organizational framework and objectives.

Conclusions

Organization-wide, centralized password management has many advantages, but it often meets with skepticism from administrators who are asked to deposit their credentials. The number of passwords that need to be accessible to several people can be minimized, but completely avoiding password sharing is difficult.

The transition from purely personal password management or offline methods, such as sealed envelopes in a vault, to an organization-wide password management tool must be well planned.

The biggest advantage of server-based solutions is reliable alerting for the person responsible if passwords are accessed in exceptional circumstances. However, you should not underestimate the overhead involved, for example, in configuring the folder structure, permissions, and default email alerts.