Limitations

Some limitations of the DC/OS Elastic service

Configuration via elasticsearch.yml and/or Elastic APIs

Elasticsearch provides two ways of updating settings: persistent (through elasticsearch.yml file) and transient (through Elasticsearch Cluster Update Settings API). The service’s Configuration Options are carried over to the tasks’ elasticsearch.yml file automatically. Out-of-band configuration changes (either via Elasticsearch’s Cluster Update Settings API or externally modifying elasticsearch.yml files) will not persist in case of a restart, failure recovery, or upgrade.

Kibana configured with X-Pack Security enabled

If Kibana is configured with kibana.elasticsearch_xpack_security_enabled set to true the default DC/OS service link (https://<cluster-url>/service/<kibana-service-name>) will not work. This is due to a change in how Kibana deals with Authorization HTTP headers starting in version 6.3.

As a workaround, you should be able to expose Kibana using EdgeLB by following the following how-to: Expose Kibana using EdgeLB.

Out-of-band configuration

Out-of-band configuration modifications are not supported. The service’s core responsibility is to deploy and maintain the service with a specified configuration. In order to do this, the service assumes that it has ownership of task configuration. If an end-user makes modifications to individual tasks through out-of-band configuration operations, the service will override those modifications at a later time. For example:

  • If a task crashes, it will be restarted with the configuration known to the scheduler, not one modified out-of-band.
  • If a configuration update is initiated, all out-of-band modifications will be overwritten during the rolling update.

Scaling

To prevent accidental data loss, the service does not support reducing the number of pods.

Disk changes

To prevent accidental data loss from reallocation, the service does not support changing volume requirements after initial deployment.

Best-effort installation

If your cluster does not have enough resources to deploy the service as requested, the initial deployment will not complete until either those resources are available or until you reinstall the service with corrected resource requirements. Similarly, scale-outs following initial deployment will not complete if the cluster does not have the needed available resources to complete the scale-out.

Virtual networks

When the service is deployed on a virtual network, the service may not be switched to host networking without a full re-installation. The same is true for attempting to switch from host to virtual networking.

Task Environment Variables

Each service task has some number of environment variables, which are used to configure the task. These environment variables are set by the service scheduler. While it is possible to use these environment variables in adhoc scripts (e.g. via dcos task exec), the name of a given environment variable may change between versions of a service and should not be considered a public API of the service.

Zones

If the service is deployed with a Zone constraint it may not be removed after initial installation.

Additionally, if the service was deployed without a Zone constraint, it may not have one added after initial installation.

Regions

The service cannot be moved between regions. It is also not possible to update the service to use a region if initially installed without one.

Upgrades and configuration updates

Upgrades and rolling configuration updates do not wait for a cluster green health status. During deployment and upgrades, the serial strategy does not wait for the Elasticsearch cluster to reach green health before proceeding to the next node.

Security

Elasticsearch’s built-in authentication mechanisms (realms) cannot currently be configured through service configuration options (e.g. on package installs or service updates). However, since the native realm is enabled by default by Elasticsearch it’s possible to configure it through the security APIs. These APIs include both authentication and authorization mechanisms.

Transport Encryption

Toggling Transport Encryption requires doing a full-cluster restart. This is an Elasticsearch limitation. The service’s default update strategy is a rolling upgrade (serial). If you change its configuration to enable or disable transport encryption, nodes that have been configured with TLS will be unable to communicate with nodes configured with unencrypted networking, and vice-versa. A full-cluster restart is required, using the parallel update strategy. Make sure you have backups, and plan for some downtime. Afterwards, you should set the update strategy back to serial for future updates.