[jira] [Created] (FLINK-5976) refactoring remove duplicate Tokenizer in flink-test

[Shang Yuanchun (Jira)]

liuyuzhong7 created FLINK-5976:
----------------------------------

             Summary: refactoring remove duplicate Tokenizer in flink-test
                 Key: FLINK-5976
                 URL: https://issues.apache.org/jira/browse/FLINK-5976
             Project: Flink
          Issue Type: Improvement
          Components: Examples
    Affects Versions: 2.0.0
            Reporter: liuyuzhong7
            Priority: Minor
             Fix For: 2.0.0


There are some duplicate code like this in flink-test, I think refactor this will be better.
```
public final class Tokenizer implements FlatMapFunction<String, Tuple2<String, Integer>> {

        @Override
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
                // normalize and split the line
                String[] tokens = value.toLowerCase().split("\\W+");

                // emit the pairs
                for (String token : tokens) {
                        if (token.length() > 0) {
                                out.collect(new Tuple2<String, Integer>(token, 1));
                        }
                }
        }
}

```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)