Configuring DC/OS Access for Confluent Kafka

ENTERPRISE

This topic describes how to configure DC/OS access for Confluent Kafka. Depending on your security mode, Confluent Kafka requires service authentication for access to DC/OS.

Security mode Service Account
Disabled Not available
Permissive Optional
Strict Required

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created uses the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

Tip: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (<service-account-id>) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d "Confluent Kafka service account" <service-account-id>

Tip: You can verify your new service account using the following command.

dcos security org service-accounts show <service-account-id>

Create a Secret

Create a secret (confluent-kafka/<secret-name>) with your service account (<service-account-id>) and private key specified (<private-key>.pem).

Tip: If you store your secret in a path that matches the service name (e.g. service name and secret path are confluent-kafka), then only the service named confluent-kafka can access it.

Permissive

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> confluent-kafka/<secret-name>

Strict

dcos security secrets create-sa-secret --strict <private-key>.pem <service-account-id> confluent-kafka/<secret-name>

Tip: You can list the secrets with this command:

dcos security secrets list /

Create and Assign Permissions Use the following curl commands to rapidly provision the Confluent Kafka service account with the required permissions.

Tips:

  • Any / character in a resource must be replaced with %252F before it can be passed in a curl command.
  • When using the API to manage permissions, you must first create the permissions and then assign them. Sometimes, the permission may already exist. In this case, the API returns an informative message. You can regard this as a confirmation and continue to the next command.
  1. Create the permission.

    Important: These commands use the default Confluent Kafka role value of confluent-kafka-role. If you’re running multiple instances of Confluent Kafka, replace the instances of confluent-kafka-role with the correct name (<name>-role). For example, if you have a Confluent Kafka instance named confluent-kafka2, you would replace each role value in the code samples to confluent-kafka2-role.

    Permissive

    Run these commands with your service account name (<service-account-id>) specified.

    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody \
    -d '{"description":"Allows Linux user nobody to execute tasks"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:confluent-kafka-role \
    -d '{"description":"Controls the ability of confluent-kafka-role to register as a framework with the Mesos master"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:confluent-kafka-role \
    -d '{"description":"Controls the ability of confluent-kafka-role to reserve resources"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:confluent-kafka-role \
    -d '{"description":"Controls the ability of confluent-kafka-role to access volumes"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
    -d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
    -d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
    -H 'Content-Type: application/json'
    

    Strict

    Run these commands with your service account name (<service-account-id>) specified.

    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:confluent-kafka-role \
    -d '{"description":"Controls the ability of confluent-kafka-role to register as a framework with the Mesos master"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:confluent-kafka-role \
    -d '{"description":"Controls the ability of confluent-kafka-role to reserve resources"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:confluent-kafka-role \
    -d '{"description":"Controls the ability of confluent-kafka-role to access volumes"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
    -d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
    -H 'Content-Type: application/json'
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
    -d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
    -H 'Content-Type: application/json'
    
  2. Grant the permissions and the allowed actions to the service account using the following commands.

    Run these commands with your service account name (<service-account-id>) specified.

    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:confluent-kafka-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:confluent-kafka-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:confluent-kafka-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id>/users/<service-account-id>/delete
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id>/users/<service-account-id>/delete
    

Create a Configuration File

Create a custom configuration file that will be used to install Confluent Kafka and save as config.json.

Specify the service account (<service-account-id>) and secret path (confluent-kafka/<secret-name>).

{
  "service": {
    "service_account": "<service-account-id>",
    "service_account_secret": "confluent-kafka/<secret-name>"
  }
}

Install Confluent Kafka

Install Confluent Kafka with this command:

dcos package install --options=config.json confluent-kafka