Security

ENTERPRISE

Securing your service

DC/OS Apache Kafka Security

-The DC/OS Apache Kafka service supports Kafka’s native transport encryption, authentication, and authorization mechanisms. The service provides automation and orchestration to simplify the use of these important features.

-A good overview of these features can be found here, and Kafka’s security documentation can be found here.

NOTE: These security features are only available on DC/OS Enterprise 1.10 and later.

Provisioning a service account

This section describes how to configure DC/OS access for Apache Kafka. Depending on your security mode, Apache Kafka may require service authentication for access to DC/OS.

Security mode Service Account
Disabled Not available
Permissive Optional
Strict Required

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created using the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

NOTE: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (for example, kafka) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d <description> kafka

You can verify your new service account using the following command.

dcos security org service-accounts show kafka

Create a Secret

Create a secret (kafka/<secret-name>) with your service account and private key specified (<private-key>.pem).

NOTE: If you store your secret in a path that matches the service name, for example, service name and secret path are both kafka, then only the service named kafka can access it.

Permissive

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> kafka/<secret-name>

Strict

dcos security secrets create-sa-secret --strict <private-key>.pem <service-account-id> kafka/<secret-name>

You can list the secrets with this command:

dcos security secrets list /

Create and Assign Permissions

Use the following curl commands to rapidly provision the Apache Kafka service account with the required permissions.

NOTE: Any forward slash ("/") in a resource must be replaced with `%252F` before it can be passed in a curl command.

When using the API to manage permissions, you must first create the permissions and then assign them. Sometimes, the permission may already exist. In this case, the API returns an informative message. You can regard this as a confirmation and continue to the next command.

  1. Create the permission.

IMPORTANT: These commands use the default Apache Kafka role value of kafka-role. If you are running multiple instances of Apache Kafka, replace the instances of kafka-role with the correct name (<name>-role). For example, if you have a Apache Kafka instance named kafka2, you would replace each role value in the code samples to kafka2-role.

Permissive

Run these commands with your service account name (<service-account-id>) specified.

curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody \
-d '{"description":"Allows Linux user nobody to execute tasks"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:kafka-role \
-d '{"description":"Controls the ability of kafka-role to register as a framework with the Mesos master"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:kafka-role \
-d '{"description":"Controls the ability of kafka-role to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:kafka-role \
-d '{"description":"Controls the ability of kafka-role to access volumes"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
-H 'Content-Type: application/json'

Strict

Run these commands with your service account name (<service-account-id>) specified.

curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:kafka-role \
-d '{"description":"Controls the ability of kafka-role to register as a framework with the Mesos master"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:kafka-role \
-d '{"description":"Controls the ability of kafka-role to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:kafka-role \
-d '{"description":"Controls the ability of kafka-role to access volumes"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
-H 'Content-Type: application/json'
  1. Grant the permissions and the allowed actions to the service account using the following commands.

    IMPORTANT: These commands use the default Apache Kafka role value of kafka-role. If you are running multiple instances of Apache Kafka, replace the instances of kafka-role with the correct name (-role). For example, if you have a Apache Kafka instance named kafka2, you would replace each role value in the code samples to kafka2-role.

    Run these commands with your service account name (<service-account-id>) specified.

    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:kafka-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:kafka-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:kafka-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id>/users/<service-account-id>/delete
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id>/users/<service-account-id>/delete
    

Transport Encryption

With transport encryption enabled, DC/OS Apache Kafka will automatically deploy all nodes with the correct configuration to encrypt communication via SSL. The nodes will communicate securely between themselves using SSL.

The service uses the DC/OS CA to generate the SSL artifacts that it uses to secure the service. Any client that trusts the DC/OS CA will consider the service’s certificates valid.

NOTE: Enabling transport encryption is **required** to use [SSL authentication](#ssl-authentication) for [authentication](#authentication), but is optional for [Kerberos authentication](#kerberos-authentication).

Prerequisites

Configure Transport Encryption

Set up the service account

Grant the service account the correct permissions.

  • In DC/OS 1.10, the required permission is dcos:superuser full.
  • In DC/OS 1.11 and later, the required permissions are:
dcos:secrets:default:/<service name>/* full
dcos:secrets:list:default:/<service name> read
dcos:adminrouter:ops:ca:rw full
dcos:adminrouter:ops:ca:ro full

where <service name> is the name of the service to be installed.

Run the following DC/OS Enterprise CLI commands to set permissions for the service account on a strict cluster:

dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:task:app_id:<service/name> create
dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:reservation:principal:dev_hdfs create
dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:volume:principal:dev_hdfs create

Install the service

Install the DC/OS Apache Kafka service including the following options in addition to your own:

{
    "service": {
        "service_account": "<your service account name>",
        "service_account_secret": "<full path of service secret>",
        "security": {
            "transport_encryption": {
                "enabled": true, "allow_plaintext": <true|false default false>

            }
        }
    }
}

NOTE: It is possible to update a running DC/OS Apache Kafka service to enable transport encryption after initial installation, but the service may be unavailable during the transition. Additionally, your Kafka clients will need to be reconfigured unless `service.security.transport_encryption.allow_plaintext` is set to true.

Verify Transport Encryption Enabled

After service deployment completes, check the list of Kafka endpoints for the endpoint broker-tls. If service.security.transport_encryption.allow_plaintext is true, then the broker endpoint will also be available.

Transport Encryption for Clients

With Transport Encryption enabled, service clients will need to be configured to use the DC/OS CA bundle to verify the connections they make to the service. Consult your client’s documentation for trusting a CA and configure your client appropriately.

Authentication

DC/OS Apache Kafka supports two authentication mechanisms, SSL and Kerberos. The two are supported independently and may not be combined. If both SSL and Kerberos authentication are enabled, the service will use Kerberos authentication.

NOTE: Kerberos authentication can, however, be combined with transport encryption.

Kerberos Authentication

Kerberos authentication relies on a central authority to verify that Kafka clients (be it broker, consumer, or producer) are who they say they are. DC/OS Apache Kafka integrates with your existing Kerberos infrastructure to verify the identity of clients.

Prerequisites

  • The hostname and port of a KDC reachable from your DC/OS cluster
  • Sufficient access to the KDC to create Kerberos principals
  • Sufficient access to the KDC to retrieve a keytab for the generated principals
  • The DC/OS Enterprise CLI
  • DC/OS Superuser permissions

Configure Kerberos Authentication

Create principals

The DC/OS Apache Kafka service requires a Kerberos principal for each broker to be deployed. Each principal must be of the form

<service primary>/kafka-<broker index>-broker.<service subdomain>.autoip.dcos.thisdcos.directory@<service realm>

with:

  • service primary = service.security.kerberos.primary
  • broker index = 0 up to brokers.count - 1
  • service subdomain = service.name with all/'s removed
  • service realm = service.security.kerberos.realm

For example, if installing with these options:

{
    "service": {
        "name": "a/good/example",
        "security": {
            "kerberos": {
                "primary": "example",
                "realm": "EXAMPLE"
            }
        }
    },
    "brokers": {
        "count": 3
    }
}

then the principals to create would be:

example/kafka-0-broker.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE
example/kafka-1-broker.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE
example/kafka-2-broker.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE
Active Directory

Microsoft Active Directory can be used as a Kerberos KDC. Doing so requires creating a mapping between Active Directory users and Kerberos principals.

The utility ktpass can be used to both create a keytab from Active Directory and generate the mapping at the same time.

The mapping can, however, be created manually. For a Kerberos principal like <primary>/<host>@<REALM>, the Active Directory user should have its servicePrincipalName and userPrincipalName attributes set to,

servicePrincipalName = <primary>/<host>
userPrincipalName = <primary>/<host>@<REALM>

For example, with the Kerberos principal example&#x2F;kafka-0-broker.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE, then the correct mapping would be,

servicePrincipalName = example&#x2F;kafka-0-broker.agoodexample.autoip.dcos.thisdcos.directory
userPrincipalName = example&#x2F;kafka-0-broker.agoodexample.autoip.dcos.thisdcos.directory@EXAMPLE

If either mapping is incorrect or not present, the service will fail to authenticate that Principal. The symptom in the Kerberos debug logs will be an error of the form

KRBError:
sTime is Wed Feb 07 03:22:47 UTC 2018 1517973767000
suSec is 697984
error code is 6
error Message is Client not found in Kerberos database
sname is krbtgt/AD.MESOSPHERE.COM@AD.MESOSPHERE.COM
msgType is 30

when the userPrincipalName is set incorrectly, and an error of the form

KRBError:
sTime is Wed Feb 07 03:44:57 UTC 2018 1517975097000
suSec is 128465
error code is 7
error Message is Server not found in Kerberos database
sname is kafka/kafka-1-broker.confluent-kafka.autoip.dcos.thisdcos.directory@AD.MESOSPHERE.COM
msgType is 30

when the servicePrincipalName is set incorrectly.

Place Service Keytab in DC/OS Secret Store

The DC/OS Apache Kafka service uses a keytab containing all node principals (service keytab). After creating the principals above, generate the service keytab making sure to include all the node principals. This will be stored as a secret in the DC/OS Secret Store.

NOTE: DC/OS 1.10 does not support adding binary secrets directly to the secret store, only text files are supported. Instead, first base64 encode the file, and save it to the secret store as /desired/path/__dcos_base64__secret_name. The DC/OS security modules will handle decoding the file when it is used by the service.

The service keytab should be stored at service/path/name/service.keytab. As noted above. for DC/OS 1.10, it would be __dcos_base64__service.keytab), where service/path/name matches the path and name of the service. For example, if installing with the options

{
    "service": {
        "name": "a/good/example"
    }
}

then the service keytab should be stored at a/good/example/service.keytab.

Documentation for adding a file to the secret store can be found here.

NOTE: Secrets access is controlled by DC/OS Spaces, which function like namespaces. Any secret in the same DC/OS Space as the service will be accessible by the service. However, matching the two paths is the most secure option. Additionally the secret name service.keytab is a convention and not a requirement.

Install the Service

Install the DC/OS Apache Kafka service with the following options in addition to your own:

{
    "service": {
        "security": {
            "kerberos": {
                "enabled": true,
                "enabled_for_zookeeper": <true|false default false>,
                "kdc": {
                    "hostname": "<kdc host>",
                    "port": <kdc port>
                },
                "primary": "<service primary default kafka>",
                "realm": "<realm>",
                "keytab_secret": "<path to keytab secret>",
                "debug": <true|false default false>
            }
        }
    }
}

NOTE: If `service.kerberos.enabled_for_zookeeper` is set to true, then the additional setting `kafka.kafka_zookeeper_uri` must be configured to point at a kerberized Apache Zookeeper as follows:

{
    "kafka": {
        "kafka_zookeeper_uri": <list of zookeeper hosts>
    }
}

The DC/OS Apache Zookeeper service (kafka-zookeeper package) is intended for this purpose and supports Kerberos.

NOTE: It is possible to enable Kerberos after initial installation but the service may be unavailable during the transition. Additionally, your Kafka clients will need to be reconfigured.

SSL Authentication

SSL authentication requires that all clients be they brokers, producers, or consumers present a valid certificate from which their identity can be derived. DC/OS Apache Kafka uses the CN of the SSL certificate as the principal for a given client. For example, from the certificate CN=bob@example.com,OU=,O=Example,L=London,ST=London,C=GB the principal bob@example.com will be extracted.

Prerequisites

Install the Service

Install the DC/OS Apache Kafka service with the following options in addition to your own:

{
    "service": {
        "service_account": "<service-account>",
        "service_account_secret": "<secret path>",
        "security": {
            "transport_encryption": {
                "enabled": true
            },
            "ssl_authentication": {
                "enabled": true
            }
        }
    }
}

NOTE: It is possible to enable SSL authentication after initial installation, but the service may be unavailable during the transition. Additionally, your Kafka clients will need to be reconfigured.

Authenticating a Client

To authenticate a client against DC/OS Apache Kafka, you will need to configure it to use a certificate signed by the DC/OS CA. After generating a certificate signing request, you can issue it to the DC/OS CA by calling the API <dcos-cluster>/ca/api/v2/sign. Using curl the request would look like:

$ curl -X POST \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" \
    <dcos-cluster>/ca/api/v2/sign \
    -d '{"certificate_request": "<json-encoded-value-of-request.csr>"}'

The <json-encoded-value-of-request.csr> field represents the content of the csr file as a single line, where new lines are replaced with \n.

$ curl -X POST \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" \
    <dcos-cluster>/ca/api/v2/sign \
    -d '{"certificate_request": ""-----BEGIN CERTIFICATE REQUEST-----\nMIIC<snipped for brevity>o39lBi1w=\n-----END CERTIFICATE REQUEST-----\n""}'

The response will contain a signed public certificate. More information on DC/OS CA API can be found here.

Authorization

The DC/OS Apache Kafka service supports Kafka’s ACL-based authorization system. To use Kafka’s ACLs, either SSL or Kerberos authentication must be enabled as detailed above.

Enable Authorization

Prerequisites

  • Completion of either SSL or Kerberos authentication above.

Install the Service

Install the DC/OS Apache Kafka service with the following options in addition to your own (remember, either SSL authentication or Kerberos must be enabled):

{
    "service": {
        "security": {
            "authorization": {
                "enabled": true,
                "super_users": "<list of super users>",
                "allow_everyone_if_no_acl_found": <true|false default false>
            }
        }
    }
}

service.security.authorization.super_users should be set to a semi-colon delimited list of principals to treat as super users (all permissions). The format of the list is User:<user1>;User:<user2>;.... Using Kerberos authentication, the “user” value is the Kerberos primary, and for SSL authentication the “user” value is the CN of the certificate. The Kafka brokers themselves are automatically designated as super users.

NOTE: It is possible to enable Authorization after initial installation, but the service may be unavailable during the transition. Additionally, Kafka clients may fail to function if they do not have the correct ACLs assigned to their principals. During the transition `service.security.authorization.allow_everyone_if_no_acl_found` can be set to `true` to prevent clients from being failing until their ACLs can be set correctly. After the transition, `service.security.authorization.allow_everyone_if_no_acl_found` should be reversed to `false`

Securely Exposing DC/OS Apache Kafka Outside the Cluster.

Both transport encryption and Kerberos are tightly coupled to the DNS hosts of the Kafka brokers. As such, exposing a secure Apache Kafka service outside of the cluster requires additional setup.

Broker to Client Connection

To expose a secure Apache Kafka service outside of the cluster, any client connecting to it must be able to access all brokers of the service via the IP address assigned to the broker. This IP address will be one of: an IP address on a virtual network or the IP address of the agent the broker is running on.

Forwarding DNS and Custom Domain

Every DC/OS cluster has a unique cryptographic ID which can be used to forward DNS queries to that Cluster. To securely expose the service outside the cluster, external clients must have an upstream resolver configured to forward DNS queries to the DC/OS cluster of the service as described here.

With only forwarding configured, DNS entries within the DC/OS cluster will be resolvable at <task-domain>.autoip.dcos.<cryptographic-id>.dcos.directory. However, if you configure a DNS alias, you can use a custom domain. For example, <task-domain>.cluster-1.acmeco.net. In either case, the DC/OS Apache Kafka service will need to be installed with an additional security option:

{
    "service": {
        "security": {
            "custom_domain": "<custom-domain>"
        }
    }
}

where <custom-domain> is one of autoip.dcos.<cryptographic-id>.dcos.directory or your organization specific domain (e.g., cluster-1.acmeco.net).

As a concrete example, using the custom domain of cluster-1.acmeco.net the broker 0 task would have a host of kafka-0-broker.<service-name>.cluster-1.acmeco.net.

Kerberos Principal Changes

Transport encryption alone does not require any additional changes. Endpoint discovery will work as normal, and clients will be able to connect securely with the custom domain as long as they are configured as described here.

Kerberos, however, does require slightly different configuration. As noted in the section Create Principals, the principals of the service depend on the hostname of the service. When creating the Kerberos principals, be sure to use the correct domain.

For example, if you install with the following settings:

{
    "service": {
        "name": "a/good/example",
        "security": {
            "custom_domain": "cluster-1.example.net",
            "kerberos": {
                "primary": "example",
                "realm": "EXAMPLE"
            }
        }
    },
    "brokers": {
        "count": 3
    }
}

The principals to create are as follows:

example/kafka-0-broker.agoodexample.cluster-1.example.net@EXAMPLE
example/kafka-1-broker.agoodexample.cluster-1.example.net@EXAMPLE
example/kafka-2-broker.agoodexample.cluster-1.example.net@EXAMPLE