yystopf
/
piflow
forked from opensci/piflow
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
298 Commits 16 Branches 0 Tags 88 MiB
Commit Graph

298 Commits

Author SHA1 Message Date
xiaoxiao 763644cfda Merge branch 'master' of https://github.com/cas-bigdatalab/piflow 2018-10-10 10:49:04 +08:00
xiaoxiao 9550f60d70 add NaiveBayesTraining Stop 2018-10-10 10:42:21 +08:00
judy0131 ab040dad25 add logs for application 2018-10-09 10:36:08 +08:00
judy0131 de2efc9a24 fix bug: getGroups api guava version conflict 2018-10-08 17:56:00 +08:00
judy0131 f374264dac fix bug: getGroups(user defined stop must implement interface "getGroup") 2018-10-08 14:10:41 +08:00
judy0131 f2510a4285 set appname 2018-10-08 13:46:12 +08:00
judy0131 f577b6ad79 run application on yarn by SparkLauncher(Now checkpoint.path is fixed, we can not read this config in config.properties, because yarn will find config.properties in path /opt/hadoop-2.6.0/tmp/nm-local-dir/usercache/root/appcache/appId/containerId/conf/config.properties) 2018-10-08 13:26:00 +08:00
judy0131 8b5fb4e619 fix bug 2018-09-30 16:38:04 +08:00
judy0131 3ebd4e1758 Merge remote-tracking branch 'origin/master' 2018-09-29 18:03:32 +08:00
judy0131 b34dec8110 support multi sparkContext to run more than one flow at the same time(but only in standalone mode, yarn still have problems) 2018-09-29 18:03:18 +08:00
judy0131 1825eb07ff support multi sparkContext to run more than one flow at the same time(but only in standalone mode, yarn still have problems) 2018-09-29 17:53:30 +08:00
xiaoxiao a987093467 fix the bug with data consumption from kafka 2018-09-28 16:14:04 +08:00
xiaoxiao 1b63b70c4d alter getGroup method and add authorEmail for stops 2018-09-28 13:31:27 +08:00
xiaoxiao 952f0cefea add ftp group(LoadFromFtp and UploadToFtp) 2018-09-28 13:24:32 +08:00
xiaoxiao 97e2600fb8 fix a bug for ReadFromKafka Stop 2018-09-28 12:47:07 +08:00
xiaoxiao caf919bae5 fix a bug 2018-09-28 10:50:37 +08:00
xiaoxiao 38cd7de567 fix a bug for ReadFromKafka stop 2018-09-28 10:40:57 +08:00
xiaoxiao 54b7a8509b Merge branch 'master' of https://github.com/cas-bigdatalab/piflow 2018-09-27 12:23:48 +08:00
xiaoxiao 9fd6688dd7 add redis and kafka group 2018-09-27 12:20:07 +08:00
xiaoxiao 7ae7e6a888 add redis group(WriteToRedis and ReadFromRedis) 2018-09-27 12:16:24 +08:00
judy0131 dcd8777558 add bundle for getStopInfo 2018-09-27 11:31:24 +08:00
judy0131 6804e91a59 fix bug: can not find user defined Configurable Stop 2018-09-27 11:23:25 +08:00
judy0131 9102bf97bb fix bug: can not find user defined Configurable Stop 2018-09-26 17:49:32 +08:00
judy0131 fd2ef07acd add stop info api 2018-09-25 18:07:34 +08:00
judy0131 4c3385ea9f add stop groups api 2018-09-25 17:35:11 +08:00
judy0131 e2b2df0418 1.modify ConfigurableStop getGroup api 
2.modify findAllConfigurableStop
3.add findAllGroups
 2018-09-25 16:31:55 +08:00
judy0131 08079e1985 add wait process terminate Runnable 2018-09-17 15:13:36 +08:00
judy0131 e6c4b22e5a add stop's author 2018-09-17 14:10:51 +08:00
judy0131 c142c03529 add get stop properties api 2018-09-14 12:58:50 +08:00
judy0131 dd06ac6a43 add comment 2018-09-13 10:35:41 +08:00
judy0131 11a594d20c add yarn.jars config to solve the problem -->"Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME",which makes the spark on yarn job run very slow! 2018-09-13 10:16:22 +08:00
judy0131 6b72da8518 1.return immediately when start flow, do not wait 
2.fix stop flow bug
 2018-09-12 16:42:33 +08:00
judy0131 66b3175ef7 fix bug 2018-09-11 17:13:05 +08:00
judy0131 ac8ad1a92e Merge branch 'master' of https://github.com/cas-bigdatalab/piflow 2018-09-11 14:53:54 +08:00
judy0131 06f9a772dc remove core-site.xml hdfs-site.xml hive-site.xml yarn-site.xml 2018-09-11 14:51:24 +08:00
xiaoxiao bb2ddc60d3 Merge branch 'master' of https://github.com/cas-bigdatalab/piflow 2018-09-11 14:24:50 +08:00
xiaoxiao 3732d0bcef add ftp group(LoadFromFtp and UploadToFtp) 2018-09-11 14:11:46 +08:00
judy0131 674de714c6 return applicationID when start the flow 2018-09-10 16:26:55 +08:00
judy0131 60e4f3310e run on yarn 2018-09-10 15:26:51 +08:00
judy0131 da88183668 get flow info 2018-09-10 10:58:42 +08:00
judy0131 88471237ad fix bug: can not run flow when package code into piflow.jar 2018-09-06 17:09:35 +08:00
judy0131 2ab05fccdd 1.modify piflow module,delete piflow-conf 2.deploy jar on server 2018-09-05 17:54:21 +08:00
lj044500 e4d51735cc XmlTesst 2018-09-04 10:48:35 +08:00
lj044500 ed1c4e47fa add folder xml file dispose 2018-09-04 10:46:01 +08:00
lj044500 1250dee09f 文件夹中所有得.xml文件处理 2018-09-04 10:40:06 +08:00
judy0131 508f0639a9 rename piflow-api to piflow-server;add getFlowInfo Api 2018-09-04 09:53:19 +08:00
judy0131 c347710084 fix bug 2018-09-03 14:50:55 +08:00
judy0131 cf32e770dc clean code 2018-09-03 13:14:33 +08:00
judy0131 6db6d54e56 add httpClient 2018-08-31 13:07:44 +08:00
xiaoxiao 3285bf602b Merge branch 'master' of https://github.com/cas-bigdatalab/piflow 2018-08-30 15:21:34 +08:00