Security

DC/OS NiFi Service encryption, authentication, and authorization

DC/OS NiFi Security

The DC/OS NiFi service supports DC/OS NiFi’s native transport encryption, authentication, and authorization mechanisms. The service provides automation and orchestration to simplify the usage of these important features.

A good overview of these features can be found here.

Transport Encryption and Kerberos Authentication

With transport encryption enabled, DC/OS NiFi will automatically deploy all nodes with the correct configuration to encrypt communication via SSL. The nodes will communicate securely between themselves using SSL. SSL authentication requires that all DC/OS NiFi Nodes present a valid certificate from which their identity can be derived for communicating between themselves. DC/OS NiFi uses the CN of the SSL certificate as the principal for a given Node. For example, CN=nifi-0-node.demonifi, O=“Mesosphere, Inc”, L=San Francisco, ST=CA, C=US.

The service uses the DC/OS CA to generate the SSL artifacts that it uses to secure the service. Any client that trusts the DC/OS CA will consider the service’s certificates valid.

NOTE: Enabling transport encryption is required to use SSL authentication for authentication, but is optional for Kerberos authentication.

Prerequisites

  • A DC/OS Service Account with a secret stored in the DC/OS Secret Store.
  • DC/OS superuser permissions for modifying the permissions of the Service Account.

Configure Transport Encryption

NOTE: A complete guide to Configuring DC/OS Access for DC/OS NiFi can be found below.

Set up the service account

Grant the service account the correct permissions.

  • In DC/OS 1.10, the required permission is dcos:superuser full.
  • In DC/OS 1.11 and later, the required permissions are:
dcos:secrets:default:/<service name>/* full
dcos:secrets:list:default:/<service name> read
dcos:adminrouter:ops:ca:rw full
dcos:adminrouter:ops:ca:ro full

where <service name> is the name of the service to be installed.

Run the following DC/OS Enterprise CLI commands to set permissions for the service account on a strict cluster:

dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:task:app_id:<service/name> create
dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:reservation:principal:dev_hdfs create
dcos security org users grant ${SERVICE_ACCOUNT} dcos:mesos:master:volume:principal:dev_hdfs create

Transport Encryption for Clients

Transport Encryption for Clients

With Transport Encryption enabled, service clients will need to be configured to use the DC/OS CA bundle to verify the connections they make to the service. Consult your client’s documentation for trusting a CA and configure your client appropriately.

Authentication

DC/OS NiFi supports two authentication mechanisms, SSL and Kerberos. The two are supported independently and may not be combined. If both SSL and Kerberos authentication are enabled, the service will use Kerberos authentication.

NOTE: Kerberos authentication can, however, be combined with transport encryption.

CA based authentication between nodes

DC/OS NiFi requires certificate based authentication between nodes. DC/OS NiFi uses the CN of the SSL certificate as the principal for a given Node. For example, CN=nifi-0-node.demonifi, O=“Mesosphere, Inc”, L=San Francisco, ST=CA, C=US.

Kerberos Authentication for End Users

Kerberos authentication relies on a central authority to verify that DC/OS NiFi clients are who they say they are. DC/OS NiFi integrates with your existing Kerberos infrastructure to verify the identity of clients.

Prerequisites

  • The hostname and port of a Key Distribution Center (KDC) reachable from your DC/OS cluster
  • Sufficient access to the KDC to create Kerberos principals
  • Sufficient access to the KDC to retrieve a keytab for the generated principals
  • The DC/OS Enterprise CLI
  • DC/OS Superuser permissions

Configure Kerberos Authentication

1. Create principals

The DC/OS NiFi service requires a Kerberos principal for the service principal and user principal. Each principal must be of the form

nifiprincipal@<service realm>
nifiadmin@<service realm>

2. Place Service Keytab in DC/OS Secret Store

The DC/OS NiFi service uses a keytab containing the above service and user principals (service keytab). After creating the principals above, generate the service keytab, making sure to include all the node principals. This will be stored as a secret in the DC/OS Secret Store by name __dcos_base64__secret_name. The DC/OS security modules will handle decoding the file when it is used by the service.

Create secret named “nifiadmin_kerberos_secret” for password of Kerberos User Principal: nifiadmin

Documentation for adding a file to the secret store can be found here.

NOTE: Secrets access is controlled by DC/OS Spaces, which function like namespaces. Any secret in the same DC/OS Space as the service will be accessible by the service.

3. Install the Service

Install the DC/OS NiFi service with the following options in addition to your own:

{
 "service": {
  "name": "/demo/nifi",
   "security": {
   "kerberos": {
        "kdc": {
       "hostname": "kdc.marathon.autoip.dcos.thisdcos.directory",
       "port": 2500
     },
     "keytab_secret": "__dcos_base64___keytab",
     "primary": "nifi",
     "realm": "LOCAL",
     "service_principal": "nifiprincipal@LOCAL",
     "user_principal": "nifiadmin@LOCAL",
     "user_principal_keytab": "nifiadmin_kerberos_secret"
   },
   "kerberos_tls": {
     "enable": true
   }
 },
 "service_account": "dcos_nifi",
 "service_account_secret": "dcos_nifi_secret",
 "virtual_network_enabled": true,
 "virtual_network_name": "dcos",
 }
}

Configuring DC/OS Access for DC/OS NiFi

Provisioning a service account

This section describes how to configure DC/OS access for NiFi. Depending on your security mode, NiFi may require service authentication for access to DC/OS.

Security mode Service Account
Disabled Not available
Permissive Optional
Strict Required

If you install a service in permissive mode and do not specify a service account, Metronome and Marathon will act as if requests made by this service are made by an account with the superuser permission.

Prerequisites:

Create a Key Pair

In this step, a 2048-bit RSA public-private key pair is created using the Enterprise DC/OS CLI.

Create a public-private key pair and save each value into a separate file within the current directory.

dcos security org service-accounts keypair <private-key>.pem <public-key>.pem

NOTE: You can use the DC/OS Secret Store to secure the key pair.

Create a Service Account

From a terminal prompt, create a new service account (for example, nifi) containing the public key (<your-public-key>.pem).

dcos security org service-accounts create -p <your-public-key>.pem -d <description> nifi

You can verify your new service account using the following command.

dcos security org service-accounts show nifi

Create a Secret

Create a secret (nifi/<secret-name>) with your service account and private key specified (<private-key>.pem).

NOTE: If you store your secret in a path that matches the service name, for example, service name and secret path are both nifi, then only the service named nifi can access it.

Permissive

dcos security secrets create-sa-secret <private-key>.pem <service-account-id> nifi/<secret-name>

Strict

dcos security secrets create-sa-secret --strict <private-key>.pem <service-account-id> nifi/<secret-name>

You can list the secrets with this command:

dcos security secrets list /

Create and Assign Permissions

Use the following curl commands to rapidly provision the NiFi service account with the required permissions.

NOTE: Any forward slash ("/") in a resource must be replaced with `%252F` before it can be passed in a curl command.

When using the API to manage permissions, you must first create the permissions and then assign them. Sometimes, the permission may already exist. In this case, the API returns an informative message. You can regard this as a confirmation and continue to the next command.

  1. Create the permission.

IMPORTANT: These commands use the default NiFi role value of nifi-role. If you are running multiple instances of NiFi, replace the instances of nifi-role with the correct name (<name>-role). For example, if you have a NiFi instance named nifi2, you would replace each role value in the code samples to nifi2-role.

Permissive

Run these commands with your service account name (<service-account-id>) specified.

curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody \
-d '{"description":"Allows Linux user nobody to execute tasks"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:nifi-role \
-d '{"description":"Controls the ability of nifi-role to register as a framework with the Mesos master"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:nifi-role \
-d '{"description":"Controls the ability of nifi-role to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:nifi-role \
-d '{"description":"Controls the ability of nifi-role to access volumes"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
-H 'Content-Type: application/json'

Strict

Run these commands with your service account name (<service-account-id>) specified.

curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:nifi-role \
-d '{"description":"Controls the ability of nifi-role to register as a framework with the Mesos master"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:nifi-role \
-d '{"description":"Controls the ability of nifi-role to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:nifi-role \
-d '{"description":"Controls the ability of nifi-role to access volumes"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to reserve resources"}' \
-H 'Content-Type: application/json'
curl -X PUT --cacert dcos-ca.crt \
-H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id> \
-d '{"description":"Controls the ability of <service-account-id> to access volumes"}' \
-H 'Content-Type: application/json'
  1. Grant the permissions and the allowed actions to the service account using the following commands.

    IMPORTANT: These commands use the default NiFi role value of nifi-role. If you are running multiple instances of NiFi, replace the instances of nifi-role with the correct name (-role). For example, if you have a NiFi instance named nifi2, you would replace each role value in the code samples to nifi2-role.

    Run these commands with your service account name (<service-account-id>) specified.

    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:framework:role:nifi-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:role:nifi-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:role:nifi-role/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:task:user:nobody/users/<service-account-id>/create
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:reservation:principal:<service-account-id>/users/<service-account-id>/delete
    curl -X PUT --cacert dcos-ca.crt \
    -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/acs/api/v1/acls/dcos:mesos:master:volume:principal:<service-account-id>/users/<service-account-id>/delete