Twister2

Twister2

  • Getting Started
  • Docs
  • Tutorial
  • AI
  • Examples
  • Contribute
  • Download
  • Configurations
  • Java Docs
  • GitHub
  • Blog

›Twister2Flow

AI

  • Overview

Twister2Flow

  • Overview on Twister2Flow

Machine Learning

  • Overview on Machine Learning
  • K-Means
  • SVM

Deep Learning

  • Overview on Deep Learning
  • Pytorch Integration

Twister2Flow

Designing a dataflow pipeline to support high performance deep learning in Twister2. This is an experimental version of the initial High Performance Deep Learning Connect of Twister2.

Supporting Dataflow Operations

  1. File Systems oriented dataflow operations
  2. Remote Memory Access (Work In Progress)

Supporting Frameworks

  1. Twister2 (JVM Oriented Big Data Toolkit)
  2. Pytorch (Deep Learning Library)
  3. Python3

Pre-Requisites

  1. Twister2 Python Installation. See more.
  2. Pytorch Installation from Source See more.
  3. Install PyArrow pip3 install pyarrow
  4. Install requests pip3 install requests
  5. Install mlxtend pip3 install mlxtend
  6. Add Twister2 to Path
export TWISTER2_HOME=<path-to-twister2-binaries>
export PATH=$TWISTER2_HOME/bin:$PATH

Install

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps twister2flow-test

Bootstrap Task Executor

python3 bootstrap/PytorchJobSubmitter.py\ 
    --script /home/vibhatha/github/forks/twister2/deeplearning/pytorch/src/main/python/PytorchMnistDist.py\ 
    --executor /home/vibhatha/venv/ENV37/bin/python3\ 
    --parallelism 4 --hostfile hostfile

Example Dataflow

from twister2flow.twister2.pipeline import PipelineGraph
from twister2flow.twister2.task.Twister2Task import Twister2Task
from twister2flow.twister2.task.PytorchTask import PytorchTask
from twister2flow.twister2.task.PythonTask import PythonTask

plg = PipelineGraph.PipelineGraph(name="UNNAMED_TASK")

download_task = PythonTask(name="download_task")
download_task.set_command("python3")
download_task.set_script_path(script_path="MnistDownload.py")
download_task.set_exec_path(exec_path=None)

twister2_task = Twister2Task(name="t2_task")
twister2_task.set_command("twister2 submit standalone python")
twister2_task.set_script_path(script_path="Twister2PytorchMnist.py")
twister2_task.set_exec_path(exec_path=None)

pytorch_task = PytorchTask(name="pytorch_task")
pytorch_task.set_command()
pytorch_task.set_script_path(script_path="PytorchMnistDist.py")
pytorch_task.set_exec_path(exec_path=None)

plg.add_task(download_task)
plg.add_task(twister2_task)
plg.add_task(pytorch_task)


print(str(plg))

plg.execute()

Running example

python3 examples/Twister2Flow.py
← OverviewOverview on Machine Learning →
  • Supporting Dataflow Operations
  • Supporting Frameworks
  • Pre-Requisites
  • Install
  • Bootstrap Task Executor
  • Example Dataflow
  • Running example
Twister2
Docs
Getting Started (Quickstart)Guides (Programming Guides)
Community
Stack OverflowProject Chat
More
BlogGitHubStar
Copyright © 2020 Indiana University