Tobias created FLINK-959:
----------------------------
Summary: Automated bare-metal deployment of FLINK on Amazon EC2 and OpenStack instances
Key: FLINK-959
URL:
https://issues.apache.org/jira/browse/FLINK-959 Project: Flink
Issue Type: New Feature
Affects Versions: pre-apache-0.5
Reporter: Tobias
Assignee: Tobias
Fix For: pre-apache-0.5
This python script does start Amazon ec2|OpenStack instances to install java+hadoop and configure hdfs/yarn via puppet. In order to run FLINK on top of hadoop YARN.
In order to install java and hadoop binaries are downloaded by the script and handed over to puppet for automated provisioning.
User-data scripts are used to install puppet (only debian) on the master and slave instances. Accordingly security groups are created and configured.
The master instance then starts a self configuration process, so that the puppet modules are set up according to the cluster structure.
The master detects if hadoop YARN web interface is accessible and waits for all expected nodes to be up and running. Then a stratosphere yarn session is started. Taskmanager and Jobmanager memory allocations are set up in the instances.cfg.
Notes:
- Configuration reserves 600mb for the operating system and allocates the rest for the YARN node.
- The Flink web interface is not accessible because the yarn.web.proxy throws a NullpointerException
- Only runs on Debian derivatives because it uses apt-get
- Tested with ubuntu-13.08
- FLINK is still named Stratosphere
Code at:
https://github.com/tobwiens/StratopshereBareMetalProvPuppet--
This message was sent by Atlassian JIRA
(v6.2#6252)
Locked
