Overview

Apache Knox is a reverse proxy that simplifies security in front of a Kerberos secured Apache Hadoop cluster and other related components. Knox can be extended with custom services to support authenticating components that aren’t originally shipped with a release. One of the services we wanted Apache Knox support for was Apache Livy, a REST API for interacting with Apache Spark.

Why use an Apache Knox service for Apache Livy?

Apache Knox simplifies deployments with multiple REST services since the authentication can be handled in a single location. Knox also significantly simplifies end user interactions since they don’t need to deal with Kerberos authentication. Apache Livy, when configured with Kerberos, is hard to use and interact with. Apache Knox makes this simple but supporting basic authentication via LDAP as well as other authentication mechanisms.

Adding and using the Apache Knox service for Apache Livy

@bernhard-42 created a Hortonworks Community Connection post showing that it was possible to add Apache Livy to Knox. KNOX-842 was created in January 2017 to get Apache Knox to support Apache Livy as part of a release. In September 2017, I worked with @westeras to incorporate Livy into our Knox server. We were able to test the patch uploaded by @JeffRodriguez successfully ensuring that Kerberos authentication worked correctly. In December 2017, Apache Knox 0.14.0 was released supporting the initial version of Apache Livy support.

What is next?

There are a few minor issues with the initial version of Apache Livy support. Most of these are easily worked around and are for future improvement.

You can help out by attaching a patch or providing feedback to the Apache Knox community.