D2iQDC/OS Documentation

DC/OS Service Docs

View All

Cluster Management

DC/OS Confluent ZooKeeper is an automated service that makes it easy to deploy and manage Confluent ZooKeeper on DC/OS for the purposes of running the DC/OS Apache Kafka and DC/OS Confluent Kafka Services. ZooKeeper is a centralized service for maintaining configuration and naming information, as well as providing distributed synchronization and group services.

DC/OS Apache ZooKeeper is an automated service that makes it easy to deploy and manage Apache ZooKeeper on DC/OS for the purposes of running the DC/OS Apache Kafka and DC/OS Confluent Kafka Services. ZooKeeper is a centralized service for maintaining configuration and naming information, as well as providing distributed synchronization and group services.

Container Orchestration

The DC/OS Kubernetes package makes it easy to deploy and manage Kubernetes on DC/OS, eliminating nearly all of the complexity traditionally associated with setting up and managing a secure and highly-available cluster of Kubernetes nodes.

Continuous Delivery

Run your continuous integration, automated testing, and continuous delivery jobs at scale with Jenkins for DC/OS. Instead of the static partitions so typical of other Jenkins clusters, Jenkins for DC/OS can create and destroy agents as demand increases and decreases. With multiple Jenkins masters sharing a single pool of compute resources, you can achieve much more efficient and resilient automations.

Spinnaker

    ENTERPRISE

    COMMUNITY

Spinnaker is an open-source, multi-cloud continuous delivery platform that helps you release software changes with high velocity and confidence. Spinnaker provides two core sets of features: application management and application deployment. You use Spinnaker’s application management features to view and manage your cloud resources. Spinnaker’s application deployment features are used to construct and manage continuous delivery workflows.

Data Processing

DC/OS Data Science Engine is the next-generation web-based user interface for Project Jupyter. It is an extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

NiFi

    COMMUNITY

Apache NiFi is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation and system mediation logic. Apache NiFi has a web-based user interface for design, control, feedback, and monitoring of dataflows. It is highly configurable along several dimensions of quality of service, such as loss-tolerant versus guaranteed delivery, low latency versus high throughput and priority-based queuing. Apache NiFi provides fine-grained data provenance for all data received, forked, joined cloned, modified, sent and ultimately dropped upon reaching its configured end-state.

Apache Spark is a fast and general-purpose cluster computing system for big data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including: Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Spark Streaming for stream processing.

JupyterLab is the next-generation web-based user interface for Project Jupyter. It is an extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

Data Services

Elasticsearch is a distributed, multitenant-capable, full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch clusters are highly available, fault tolerant, and durable. Multiple Elasticsearch clusters can be installed on DC/OS and managed independently, so you can offer Elasticsearch as a managed service to your organization with or without the commercial X-Pack capabilities.

Databases

DC/OS Apache Cassandra is an automated service that makes it easy to deploy and manage Apache Cassandra on DC/OS. Apache Cassandra is a distributed NoSQL database offering high availability, fault tolerance and scalability across data centers.

Couchbase

    COMMUNITY

DC/OS Couchbase service is an autonomous service that makes it easy to deploy and operate Couchbase Server and the Couchbase Sync Gateway on Mesosphere DC/OS.

DataStax Enterprise (DSE) Service is an automated service that makes it easy to deploy and manage DataStax Enterprise clusters on Mesosphere DC/OS, eliminating nearly all of the complexity traditionally associated with managing a DataStax cluster. DataStax Enterprise helps customers of all sizes build and run cloud-native applications at epic scale.

Percona Server for MongoDB is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB® Community Server that includes enterprise-grade features and functionality.

Deploy and manage Percona XtraDB Cluster on Mesosphere DC/OS

Messaging Queues

Confluent Kafka

    ENTERPRISE

Confluent Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. DC/OS Confluent Kafka gives you direct access to the Confluent Kafka API so that existing producers and consumers can interoperate. You can configure and install DC/OS Confluent Kafka in moments. Multiple Confluent Kafka clusters can be installed on DC/OS and managed independently, so you can offer Confluent Kafka as a managed service to your organization.

Apache Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. DC/OS Kafka gives you direct access to the Kafka API so that existing producers and consumers can interoperate. You can configure and install DC/OS Kafka in moments. Multiple Kafka clusters can be installed on DC/OS and managed independently, so you can offer Kafka as a managed service to your organization.

Monitoring

The DC/OS Monitoring service makes it easy to monitor DC/OS components and your services on the DC/OS cluster. It can be configured to automatically load Grafana dashboard and alert configurations from Git repositories. It also ships with a set of default Grafana dashboards for monitoring DC/OS itself. The service can be configured to automatically load Alertmanager configuration from a Git repository.

Prometheus

    COMMUNITY

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Prometheus is designed for reliability, to be the system you go to during an outage to allow you to quickly diagnose problems. It is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Each Prometheus server is standalone, not depending on network storage or other remote services.

Networking

Edge-LB

    ENTERPRISE

One of the most important ways you can manage cluster operations is through efficient load balancing of access requests and workload processing. Load balancing improves the performance, reliability, and network efficiency for web-based properties, applications, databases, and other services by distributing workload across multiple servers. By introducing a load balancer like Edge-LB, you can distribute the traffic for all services that run on a DC/OS Enterprise cluster.

Marathon provides a meta-framework for scheduling, container orchestration, and load balancing as part of the Mesosphere DC/OS platform. Marathon load balancer (Marathon-LB) is a proxy server and load balancer for TCP, HTTP, and HTTPS requests based on HAProxy open-source software.

Storage

DC/OS Storage Service (DSS) is a service that manages volumes, volume profiles, volume providers, and storage devices in the DC/OS cluster. The DSS serves as the brain of the storage support system by collecting storage related information from various components in the cluster, keeping track of their states, and acting on user requests.

DC/OS Apache HDFS is a managed service that makes it easy to deploy and manage an HA (High Availability) Apache HDFS cluster on Mesosphere DC/OS. Apache HDFS (Hadoop Distributed File System) is an open source distributed file system based on Google’s GFS (Google File System) paper. It is a replicated and distributed file system interface for use with “big data” and “fast data” applications.

Minio

    COMMUNITY

MinIO Object Storage is a high performance, distributed object storage server, designed for large-scale data infrastructure. It is an ideal S3-compatible replacement for Hadoop HDFS and scale out NAS for machine learning and other big data workloads by employing cloud native technologies and disaggregating the compute and storage layer to create highly efficient and scalable object storage solutions.