How to Join a dimension table in flink sql

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to Join a dimension table in flink sql

XiangWei Huang
Hi all,
   Is it possible to  join records read from a kafka stream with one(or
more) dimension tables which are saved as mysql table using flink streaming
sql.
Reply | Threaded
Open this post in threaded view
|

Re: How to Join a dimension table in flink sql

Lin Li
Hi XiangWei,
Currently one could implement a udtf to achieve that. (see
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/udfs.
html#table-functions)
BTW, this udtf should carefully dealing with the connections to database
server(s),  and enable caching if possible can reduce server's pressure.

2018-01-15 18:11 GMT+08:00 XiangWei Huang <[hidden email]>:

> Hi all,
>    Is it possible to  join records read from a kafka stream with one(or
> more) dimension tables which are saved as mysql table using flink streaming
> sql.
>
Reply | Threaded
Open this post in threaded view
|

Re: How to Join a dimension table in flink sql

xingcanc
In reply to this post by XiangWei Huang
Hi XiangWei,

Unfortunately, it's not possible for the Flink SQL API to join a stream
with a common dataset now. As a workaround, if the table is just a tiny
one, you can achieve a inner/left outer join with the user defined table
functions
<https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/sql.html#joins>.
Note that due to some reasons, the UDTF left outer join cannot support
arbitrary conditions now.

Hope that helps.

Best,

Xingcan

On 15/01/2018 6:11 PM, XiangWei Huang wrote:
> Hi all,
> Is it possible to  join records read from a kafka stream with one(or
> more) dimension tableswhich are saved as mysql table using flink
> streaming sql.