πFlow is an easy to use, powerful big data pipeline system. Try PiFlow v0.6 with: http://piflow.cstcloud.cn/piflow-web/

Table of Contents

• Features
• Architecture
• Requirements
• Getting Started
• PiFlow Docker
• Use Interface

Features

• Easy to use
  • provide a WYSIWYG web interface to configure data flow
  • monitor data flow status
  • check the logs of data flow
  • provide checkpoints
• Strong scalability:
  • Support customized development of data processing components
• Superior performance
  • based on distributed computing engine Spark
• Powerful
  • 100+ data processing components available
  • include spark、mllib、hadoop、hive、hbase、solr、redis、memcache、elasticSearch、jdbc、mongodb、http、ftp、xml、csv、json，etc.

Architecture

Requirements

• JDK 1.8
• Scala-2.11.8
• Apache Maven 3.1.0 or newer
• Git Client (used during build process by 'bower' plugin)
• Spark-2.1.0、 Spark-2.2.0、 Spark-2.3.0
• Hadoop-2.6.0

Getting Started

To Build:

• install external package

      mvn install:install-file -Dfile=/.../piflow/piflow-bundle/lib/spark-xml_2.11-0.4.2.jar -DgroupId=com.databricks -DartifactId=spark-xml_2.11 -Dversion=0.4.2 -Dpackaging=jar
      mvn install:install-file -Dfile=/.../piflow/piflow-bundle/lib/java_memcached-release_2.6.6.jar -DgroupId=com.memcached -DartifactId=java_memcached-release -Dversion=2.6.6 -Dpackaging=jar
      mvn install:install-file -Dfile=/.../piflow/piflow-bundle/lib/ojdbc6-11.2.0.3.jar -DgroupId=oracle -DartifactId=ojdbc6 -Dversion=11.2.0.3 -Dpackaging=jar
      mvn install:install-file -Dfile=/.../piflow/piflow-bundle/lib/edtftpj.jar -DgroupId=ftpClient -DartifactId=edtftp -Dversion=1.0.0 -Dpackaging=jar
  

• mvn clean package -Dmaven.test.skip=true

      [INFO] Replacing original artifact with shaded artifact.
      [INFO] Reactor Summary:
      [INFO]
      [INFO] piflow-project ..................................... SUCCESS [  4.369 s]
      [INFO] piflow-core ........................................ SUCCESS [01:23 min]
      [INFO] piflow-configure ................................... SUCCESS [ 12.418 s]
      [INFO] piflow-bundle ...................................... SUCCESS [02:15 min]
      [INFO] piflow-server ...................................... SUCCESS [02:05 min]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 06:01 min
      [INFO] Finished at: 2020-05-21T15:22:58+08:00
      [INFO] Final Memory: 118M/691M
      [INFO] ------------------------------------------------------------------------
  

Run Piflow Server：

• run piflow server on intellij:

  • edit config.properties
  • build piflow to generate piflow-server-0.9.jar
  • main class is cn.piflow.api.Main(remember to set SPARK_HOME)

• run piflow server by release version:

  • download piflow.tar.gz:
    https://github.com/cas-bigdatalab/piflow/releases/download/v0.5/piflow.tar.gz
    https://github.com/cas-bigdatalab/piflow/releases/download/v0.6/piflow-server-v0.6.tar.gz
    https://github.com/cas-bigdatalab/piflow/releases/download/v0.7/piflow-server-v0.7.tar.gz

  • unzip piflow.tar.gz:
    tar -zxvf piflow.tar.gz

  • edit config.properties

  • run start.sh、stop.sh、 restart.sh、 status.sh

• how to configure config.properties

  #spark and yarn config
  spark.master=yarn
  spark.deploy.mode=cluster
  
  #hdfs default file system
  fs.defaultFS=hdfs://10.0.86.191:9000
  
  #yarn resourcemanager.hostname
  yarn.resourcemanager.hostname=10.0.86.191
  
  #if you want to use hive, set hive metastore uris
  #hive.metastore.uris=thrift://10.0.88.71:9083
  
  #show data in log, set 0 if you do not want to show data in logs
  data.show=10
  
  #server port
  server.port=8002
  
  #h2db port
  h2.port=50002
  

Run Piflow Web：

• https://github.com/cas-bigdatalab/piflow-web

Use with command line：

• command line

  flow config example

      
        {
          "flow":{
          "name":"test",
          "uuid":"1234",
          "checkpoint":"Merge",
          "stops":[
          {
            "uuid":"1111",
            "name":"XmlParser",
            "bundle":"cn.piflow.bundle.xml.XmlParser",
            "properties":{
                "xmlpath":"hdfs://10.0.86.89:9000/xjzhu/dblp.mini.xml",
                "rowTag":"phdthesis"
            }
          },
          {
            "uuid":"2222",
            "name":"SelectField",
            "bundle":"cn.piflow.bundle.common.SelectField",
            "properties":{
                "schema":"title,author,pages"
            }
      },
      {
        "uuid":"3333",
        "name":"PutHiveStreaming",
        "bundle":"cn.piflow.bundle.hive.PutHiveStreaming",
        "properties":{
            "database":"sparktest",
            "table":"dblp_phdthesis"
        }
      },
      {
        "uuid":"4444",
        "name":"CsvParser",
        "bundle":"cn.piflow.bundle.csv.CsvParser",
        "properties":{
            "csvPath":"hdfs://10.0.86.89:9000/xjzhu/phdthesis.csv",
            "header":"false",
            "delimiter":",",
            "schema":"title,author,pages"
        }
      },
      {
        "uuid":"555",
        "name":"Merge",
        "bundle":"cn.piflow.bundle.common.Merge",
        "properties":{
          "inports":"data1,data2"
        }
      },
      {
        "uuid":"666",
        "name":"Fork",
        "bundle":"cn.piflow.bundle.common.Fork",
        "properties":{
          "outports":"out1,out2,out3"
        }
      },
      {
        "uuid":"777",
        "name":"JsonSave",
        "bundle":"cn.piflow.bundle.json.JsonSave",
        "properties":{
          "jsonSavePath":"hdfs://10.0.86.89:9000/xjzhu/phdthesis.json"
        }
      },
      {
        "uuid":"888",
        "name":"CsvSave",
        "bundle":"cn.piflow.bundle.csv.CsvSave",
        "properties":{
          "csvSavePath":"hdfs://10.0.86.89:9000/xjzhu/phdthesis_result.csv",
          "header":"true",
          "delimiter":","
        }
      }
    ],
    "paths":[
      {
        "from":"XmlParser",
        "outport":"",
        "inport":"",
        "to":"SelectField"
      },
      {
        "from":"SelectField",
        "outport":"",
        "inport":"data1",
        "to":"Merge"
      },
      {
        "from":"CsvParser",
        "outport":"",
        "inport":"data2",
        "to":"Merge"
      },
      {
        "from":"Merge",
        "outport":"",
        "inport":"",
        "to":"Fork"
      },
      {
        "from":"Fork",
        "outport":"out1",
        "inport":"",
        "to":"PutHiveStreaming"
      },
      {
        "from":"Fork",
        "outport":"out2",
        "inport":"",
        "to":"JsonSave"
      },
      {
        "from":"Fork",
        "outport":"out3",
        "inport":"",
        "to":"CsvSave"
      }
    ]
  }
  
  }
  
  

  - flow config example
  • curl -0 -X POST http://10.0.86.191:8002/flow/start -H "Content-type: application/json" -d 'this is your flow json'

docker-started

• pull piflow images
  docker pull registry.cn-hangzhou.aliyuncs.com/cnic_piflow/piflow:v0.6.1

• show docker images
  docker images

• run a container with piflow imageID ， all services run automatically
  docker run --name piflow-v0.6 -it [imageID]

• please visit "containerip:6001/piflow-web", it may take a while

• if somethings goes wrong, all the application are in /opt folder，

use-interface

• Login:

  

• Flow list:

  

• Create flow:

  

• Configure flow:

  

• Load flow:

  

• Monitor flow:

  

• Flow logs:

  

• Group list:

  

• Configure group:

  

• Monitor group:

  

• Process List:

  

• Template List:

  

• Save Template:

  

Welcome to join PiFlow User Group! Contact US
Name:吴老师
Mobile Phone：18910263390
WeChat：18910263390
Email: wzs@cnic.cn
QQ Group：1003489545