Flink分布式standalone部署方式(第二种方式)
接著上面一節中介紹的,瀏覽器中輸入地址http://swarm-manager:8088/cluster,輸出如下:
拿到application_1569070146084_0001,然后在控制臺先kill掉:./yarn application -kill application_1569070146084_0001
Run a single Flink job on YARN
之前介紹了在一個Hadoop Yarn 環境中啟動一個Flink集群,此外,還可以僅僅通過執行一個任務來啟動一個flink
./bin/flink run -m yarn-cluster -yn 1 ./examples/batch/WordCount.jar
其中-m yarn-cluster表示使用yarn集群,-yn 1表示taskmanager的數量。
這個任務提交后,很快就可以運行完成:
自定義一個任務,并提交到Flink集群中
寫一個最簡單的代碼
public class JavaHDFSBatchApp {public static void main(String[] args) throws Exception {ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();DataSource<String> dataSource = env.readTextFile("hdfs://swarm-manager:9000/LICENSE-2.0.txt");dataSource.print();} }在原有的maven基礎上修改pom.xml文件,指定main class:
<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>3.0.0</version><executions><!-- Run shade goal on package phase --><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><artifactSet><excludes><exclude>org.apache.flink:force-shading</exclude><exclude>com.google.code.findbugs:jsr305</exclude><exclude>org.slf4j:*</exclude><exclude>log4j:*</exclude></excludes></artifactSet><filters><filter><!-- Do not copy the signatures in the META-INF folder.Otherwise, this might cause SecurityExceptions when using the JAR. --><artifact>*:*</artifact><excludes><exclude>META-INF/*.SF</exclude><exclude>META-INF/*.DSA</exclude><exclude>META-INF/*.RSA</exclude></excludes></filter></filters><transformers><transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"><mainClass>com.vincent.course08.JavaHDFSBatchApp</mainClass></transformer></transformers></configuration></execution></executions></plugin>maven install生成文件springboot-flink-train-1.0-shaded.jar,導入到/home/iie4bu/app/flink-1.8.2/examples/batch下,然后運行命令:./bin/flink run -m yarn-cluster -yn 1 ./examples/batch/springboot-flink-train-1.0-shaded.jar
總結
以上是生活随笔為你收集整理的Flink分布式standalone部署方式(第二种方式)的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: Flink分布式standalone部署
- 下一篇: Apache Flink 零基础入门(二