Elasticsearch Custom realm for Kerberos

Shield is the official security plugin for Elasticsearch. Since version 2.0 it supports custom realms which offer the possibility to add support for arbitrary authentication and authorization mechanisms. Codecentric AG has developed a custom realm for Shield to support seamless integration into a pre-existing Kerberos infrastructure.

“Custom realms make it easy to integrate Shield with a wide range of advanced or custom authentication sources” said Steve Kearns, Senior Director of Product Management at Elastic. “The Kerberos realm created by codecentric is a great example of what is possible and how to build a proper integration with Shield.”

Shield Realms

A realm implements how authentication and authorization is done against a specific backend system. Authentication is the first step and it validates a user by its credentials. If validation is successful then authorization occurs which associates roles with the previously authenticated user. Later on privileges will be mapped to these roles and each action the user wants to perform is checked against the mapped privileges.

Shield comes with four built-in realms:

• esusers This realm authenticates and authorizes against a file which lists the users, their hashed credentials and their roles.
• LDAP Authentication and authorization is done against an LDAP (Lightweight Directory Protocol) server.
• Active Directory Special kind of the LDAP realm which is predefined and optimized for Active Directory.
• PKI A realm implementation which can authenticate users by an X.509 certificate. A Public-Key-Infrastructure (PKI) and usage of SSL/TLS is mandatory to make this realm work.

A realm is an authentication mechanism which authenticates users and maps them to roles; Realms are invoked upon access to Elasticsearch on the transport protocol layer as well as on HTTP/REST layer. It defines therefore also the HTTP authentication method which is in the first three cases above HTTP basic authentication. To make this really secure the usage of HTTPS is mandatory. Otherwise the passwords are transmitted in cleartext over the network.

The PKI realm does not use basic authentication, instead the authentication is done on the SSL/TLS layer (beneath the application protocol). It works via two-way SSL where not only the server has to authenticate itself to the client but also the client authenticates itself to the server. 

If those four realms were not sufficient then you’ll have to write your own realm implementation. The next section shows how to do this based on the Kerberos realm implementation.

Kerberos Custom Realm

To write your own Shield custom realm you have to implement a few classes defined by the generic public Shield realm API.

First let’s gain a fundamental understanding on how the authentication flow works. A client requests access to Elasticsearch either via the transport protocol or the HTTP protocol. This access request will be mapped to a Transport Message or a Rest Request (depending on the protocol the client utilizes). From this we have to extract an Authentication Token. How exactly we achieve this or how exactly the token looks like is up to the underlying authentication mechanism we implement. For Kerberos the token is transmitted via a header in the HTTP request and consists of byte sequences defined by the Kerberos protocol. So extraction of the token is easy, we just get the value as a base64 encoded string of the Authorization header and decode the bytes back.

In a second step this Authentication Token needs to be validated. And again, the details on how to do this depend on the underlying authentication mechanism. For Kerberos we send the token to the Kerberos server and receive either a successful or failed authentication message (simplified). In case of success we extract the username from the token and put a user object into the context of the request. In case of a failure (if there is no token to extract or validation of the token fails) we throw a special exception. This exception is handled by the Authentication Failure Handler which forwards the failure to the client. The exception handled by the Authentication Failure Handler can contain additional commands for the client like “no authentication token, pls. send one” or “authentication token not complete, pls. send the rest of the token”. Once the token is successfully validated the user object will be put into the context what makes this request an authenticated request. In addition roles can also be added to the user to let Shield later perform authorization for that particular user.

A widespread authentication method found within large enterprises is Kerberos. Main reasons for that are its native support through Active Directory and its Single-Sign-On capabilities. Another important feature is that it allows secure authentication across untrusted networks. This means especially that SSL/TLS is not mandatory, because no plain passwords (or no passwords at all) are transmitted. To sum it up, Kerberos is a failure-resistant, ticket-based and distributed authentication service which can authenticate clients (human users as well as machines) across insecure networks. The main component of a Kerberos infrastructure is the “Key Distribution Center” (KDC). This server issues special tickets (so called Ticket Granting Tickets or TGT) to clients if they can prove their identity. Once a client receives a TGT it stores this ticket locally and for the lifetime of this ticket the client can generate service specific tickets (without entering credentials again) from this ticket to access services (for example Elasticsearch).

Kerberos was originally not designed to work together with HTTP, but luckily there is an extension to the Kerberos protocol called “SPNEGO” to make it work via HTTP. The SPNEGO protocol defines how to transmit tickets by HTTP headers. Kerberos and SPNEGO are supported by the Java Runtime through JAAS and the GSSAPI layer so there was no need to implement the Kerberos protocol or communication with KDC ourselves.

The codecentric Kerberos realm secures the HTTP-/REST layer as well as the transport protocol and enables Single-Sign-On. The plugin is free of charge and is available as Open-Source software under the Apache 2 license.

Installation is done via the Elasticsearch plugin manager:

bin/plugin install file:///Downloads/elasticsearch-shield-kerberos-realm-2.1.0.zip

Although Kerberos and its configuration is not easy the configuration of the Kerberos plugin is fairly simple because all the JAAS internals and most of the Kerberos internals are hidden.

For a working installation you have to configure three properties in elasticsearch.yml:

shield.authc.realms.cc-kerberos.acceptor_keytab_path: /path/to/server.keytab (1)
shield.authc.realms.cc-kerberos.acceptor_principal: HTTP/localhost@REALM.COM (2)
de.codecentric.realm.cc-kerberos.krb5.file_path: /etc/krb5.conf (3)

1. Kerberos Keytab file which holds the identity of the Elasticsearch service
2. The name (principal) of the Elasticsearch service
3. Path to the krb5.conf file

All this information is typically provided by the system administrator of your company.

To test the plugin without a working Kerberos infrastructure just refer to https://goo.gl/KdqDRt for a working demo.

Conclusion

With the Kerberos realm for Shield companies are now able to seamlessly integrate Elasticsearch and their pre-existing Kerberos infrastructure and enable Single-Sign-On for all users.

Custom realms are a powerful new feature of Shield which makes it possible for enterprises to connect Elasticsearch to their legacy or self-developed authentication systems or to integrate Elasticsearch with other standard protocols like OAuth or SAML.

Download

https://github.com/codecentric/elasticsearch-shield-kerberos-realm/releases

References

https://github.com/codecentric/elasticsearch-shield-kerberos-realm
https://github.com/salyh/elasticsearch-beyond-basicauthentication

https://www.elastic.co/guide/en/shield/current/custom-realms.html
https://speakerdeck.com/salyh/elasticsearch-2-security-beyond-basic-authentication
http://goo.gl/OgXuhl
https://tools.ietf.org/html/rfc4120
https://tools.ietf.org/html/rfc4178
https://tools.ietf.org/html/rfc4559