Twister2

Twister2

  • Getting Started
  • Docs
  • Tutorial
  • AI
  • Examples
  • Contribute
  • Download
  • Configurations
  • Java Docs
  • GitHub
  • Blog

›Deployment

Compiling

  • Overview
  • Linux
  • MacOS
  • Maven Artifacts

App Development

  • API Overview
  • Developing Applications
  • Streaming Jobs
  • Batch Jobs

APIs

  • Worker API
  • Data API
  • Compute API
  • Operator API
  • Windowing API
  • Storm API
  • Apache Beam
  • Python API

Deployment

  • Job Submit
  • Standalone
  • Docker
  • Kubernetes
  • Minikube
  • Mesos
  • Nomad
  • Slurm
  • Dashboard
  • Logging
  • Configurations

Concepts

  • Overview
  • Architecture
  • Operators
  • Task System
  • Data Access

Resources

  • Publications

Twister2 Configurations

A detailed description of all the configuration parameters supported by Twister2 can be found in Twister2 Configurations.

Configuration Folder

Twister2 is configured using set of YAML files. These configurations are related to different components of Twister2. All the configurations can be found inside the conf folder of Twister2 distribution. Below shows the sample structure of the configuration folder

Inside there you'll find several folders. Below table describes these folders along with the configurations inside them.

DirectoryDescription
commonCommon configurations to all the resource schedulers
standaloneStandalone deployment mode specific configurations
kubernetesKubernetes based deployment configurations
mesosMesos based deployment configurations
nomadNomad based deployment configurations
slurmSlurm based deployment configuration
dashboardUI Configurations

When reading configurations, Twister2 first looks at the common folder. Then it looks at deployment specific folder for configurations. One can specify the same configuration parameter in a file in common folder as well as deployment specific folder. The configurations read from the deployment specific folder takes precedent over common configurations. If the same parameter is specifed in two files under the same folder the order is not specified.

Configuration Files

In common and deployment specific folders we can find the following files.

Config FileDescription
core.yamlCore services related configurations, job master, thread pools etc
resource.yamlResource provisioning and job submit related configurations
network.yamlNetwork and parallel operator related configurations
task.yamlCompute API related configurations including, task scheduler and executor
data.yamlData access related configurations
checkpoint.yamlCheck-pointing configurations

A detailed description of the configurations can be found in the following guide.

Twister2 Configurations.

← LoggingOverview →
  • Configuration Folder
  • Configuration Files
Twister2
Docs
Getting Started (Quickstart)Guides (Programming Guides)
Community
Stack OverflowProject Chat
More
BlogGitHubStar
Copyright © 2020 Indiana University