Hi All,
I'm new to Flink and was looking for a JDBC stream sink connector, and didn't see one in flink-streaming-connectors. Is there one somewhere else, or is there one currently in development. If not could I pick up a ticket to add it? Thanks, Tim |
Hi Timothy,
you can use the `JDBCOutputFormat` in combination with the `DataStream.writeUsingOutputFormat` method. However, this won't give you exactly once guarantees since the output formats don't take part in the checkpointing mechanism. Currently, Chesnay is working on a generic framework for almost exactly once sinks. Maybe you can sync with him. You'll find the corresponding JIRA ticket here [1]. [1] https://issues.apache.org/jira/browse/FLINK-3332 Cheers, Till On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas < [hidden email]> wrote: > Hi All, > > I'm new to Flink and was looking for a JDBC stream sink connector, and > didn't see one in flink-streaming-connectors. Is there one somewhere else, > or is there one currently in development. If not could I pick up a ticket > to add it? > > Thanks, > Tim > |
Hi Timothy!
I would first just write a stream data sink based on the JDBCOutputFormat, and then let's look into making it exactly-once. For JDBC with support for transactions, the basic idea would be to start a transaction, insert elements, commit when a checkpoint is confirmed, and start a new transaction. On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <[hidden email]> wrote: > Hi Timothy, > > you can use the `JDBCOutputFormat` in combination with the > `DataStream.writeUsingOutputFormat` method. However, this won't give you > exactly once guarantees since the output formats don't take part in the > checkpointing mechanism. > > Currently, Chesnay is working on a generic framework for almost exactly > once sinks. Maybe you can sync with him. You'll find the corresponding JIRA > ticket here [1]. > > [1] https://issues.apache.org/jira/browse/FLINK-3332 > > Cheers, > Till > > > On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas < > [hidden email]> wrote: > > > Hi All, > > > > I'm new to Flink and was looking for a JDBC stream sink connector, and > > didn't see one in flink-streaming-connectors. Is there one somewhere > else, > > or is there one currently in development. If not could I pick up a ticket > > to add it? > > > > Thanks, > > Tim > > > |
I'm actually working on a JDBC streaming connector with transactions.
On 26.02.2016 10:33, Stephan Ewen wrote: > Hi Timothy! > > I would first just write a stream data sink based on the JDBCOutputFormat, > and then let's look into making it exactly-once. > > For JDBC with support for transactions, the basic idea would be to start a > transaction, insert elements, commit when a checkpoint is confirmed, and > start a new transaction. > > On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <[hidden email]> > wrote: > >> Hi Timothy, >> >> you can use the `JDBCOutputFormat` in combination with the >> `DataStream.writeUsingOutputFormat` method. However, this won't give you >> exactly once guarantees since the output formats don't take part in the >> checkpointing mechanism. >> >> Currently, Chesnay is working on a generic framework for almost exactly >> once sinks. Maybe you can sync with him. You'll find the corresponding JIRA >> ticket here [1]. >> >> [1] https://issues.apache.org/jira/browse/FLINK-3332 >> >> Cheers, >> Till >> >> >> On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas < >> [hidden email]> wrote: >> >>> Hi All, >>> >>> I'm new to Flink and was looking for a JDBC stream sink connector, and >>> didn't see one in flink-streaming-connectors. Is there one somewhere >> else, >>> or is there one currently in development. If not could I pick up a ticket >>> to add it? >>> >>> Thanks, >>> Tim >>> |
Thanks everyone. I'll take a look at JDBCOutputFormat, and FLINK-3332
<https://issues.apache.org/jira/browse/FLINK-3332> along with other related tickets. Tim On Fri, Feb 26, 2016 at 1:55 AM, Chesnay Schepler <[hidden email]> wrote: > I'm actually working on a JDBC streaming connector with transactions. > > > On 26.02.2016 10:33, Stephan Ewen wrote: > >> Hi Timothy! >> >> I would first just write a stream data sink based on the JDBCOutputFormat, >> and then let's look into making it exactly-once. >> >> For JDBC with support for transactions, the basic idea would be to start a >> transaction, insert elements, commit when a checkpoint is confirmed, and >> start a new transaction. >> >> On Fri, Feb 26, 2016 at 10:28 AM, Till Rohrmann <[hidden email]> >> wrote: >> >> Hi Timothy, >>> >>> you can use the `JDBCOutputFormat` in combination with the >>> `DataStream.writeUsingOutputFormat` method. However, this won't give you >>> exactly once guarantees since the output formats don't take part in the >>> checkpointing mechanism. >>> >>> Currently, Chesnay is working on a generic framework for almost exactly >>> once sinks. Maybe you can sync with him. You'll find the corresponding >>> JIRA >>> ticket here [1]. >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-3332 >>> >>> Cheers, >>> Till >>> >>> >>> On Fri, Feb 26, 2016 at 1:23 AM, Timothy Farkas < >>> [hidden email]> wrote: >>> >>> Hi All, >>>> >>>> I'm new to Flink and was looking for a JDBC stream sink connector, and >>>> didn't see one in flink-streaming-connectors. Is there one somewhere >>>> >>> else, >>> >>>> or is there one currently in development. If not could I pick up a >>>> ticket >>>> to add it? >>>> >>>> Thanks, >>>> Tim >>>> >>>> > |
Free forum by Nabble | Edit this page |