Thanks Jie for driving this discussion. I also prefer a pluggable
delegation token provider. And I think users can use configurations to
specify the hive conf path, similar to how users specify a hive catalog.
On Wed, Jan 13, 2021 at 4:51 PM Till Rohrmann <
[hidden email]> wrote:
> Hi Jie Wang,
>
> thanks for starting this discussion. To me the SPI approach sounds better
> because it is not as brittle as using reflection. Concerning the
> configuration, we could think about introducing some Hive specific
> configuration options which allow us to specify these paths. How are other
> projects which integrate with Hive are solving this problem?
>
> Cheers,
> Till
>
> On Tue, Jan 12, 2021 at 4:13 PM 王 杰 <
[hidden email]> wrote:
>
>> Hi everyone,
>>
>> Currently, Hive delegation token is not obtained when Flink submits the
>> application in Yarn mode using kinit way. The ticket is
>>
https://issues.apache.org/jira/browse/FLINK-20714. I'd like to start a
>> discussion about how to support this feature.
>>
>> Maybe we have two options:
>> 1. Using a reflection way to construct a Hive client to obtain the token,
>> just same as the org.apache.flink.yarn.Utils.obtainTokenForHBase
>> implementation.
>> 2. Introduce a pluggable delegation provider via SPI. Delegation provider
>> could be placed in connector related code, so reflection is not needed and
>> is more extendable.
>>
>>
>>
>> Both options have to handle how to specify the HiveConf to use. In Hive
>> connector, user could specify both hiveConfDir and hadoopConfDir when
>> creating HiveCatalog. The hadoopConfDir may not the same as the Hadoop
>> configuration in HadoopModule.
>>
>> Looking forward to your suggestions.
>>
>> --
>> Best regards!
>> Jie Wang
>>
>>
--
Best regards!
Rui Li
Locked
