[Bug]: Kerberos authentication failure

[shmilygkd]

What happened?

When using the External Catalog type, Mixed-Iceberg, Mixed-Hive, Iceberg table formats, and Kerberos authentication method, after creating a hive_catalog with Hive service tickets and keytab files, if you create an Iceberg table in that Catalog and a specific database and insert data, subsequent queries will prompt a client authentication failure. It will look like this:

Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:754)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:709)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
... 44 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
... 47 more

After I restarted the Amoro service using ams.sh restart, querying the iceberg table data worked normally. However, if I continued to click the query button, it would throw the authentication exception mentioned above again.

Interestingly, every time I restarted the service, the first query would succeed, but the N query would fail; then after restarting the service again, the first query would succeed, and the N query would fail...

Affects Versions

0.7.1

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS

How to reproduce

No response

Relevant log output

No response

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[klion26]

It seems that AMS threw the error when querying from the web front, could you please share the whole log, thanks.

[jerui]

I also encountered the same problem, and the error log is as follows:

[jerui]

[org.apache.hadoop.ipc.Client] [] - Exception encountered while connecting to the server xxxxxxxx03/ip:端口
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795) ~[hadoop-common-3.4.0.jar:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1473) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1426) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258) ~[hadoop-common-3.4.0.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139) ~[hadoop-common-3.4.0.jar:?]
at com.sun.proxy.$Proxy61.getBlockLocations(Unknown Source) ~[?:?]

[jerui]

The front-end logs are as follows:

2024/12/06 19:16:00 prepare execute statement, line:1
2024/12/06 19:16:00 select * from dbb.cdrs
2024/12/06 19:16:00 meet exception during execution.
2024/12/06 19:16:00 org.apache.iceberg.exceptions.RuntimeIOException: Failed to get block locations for path: hdfs:///houser/tablespace/managed/hive/dbb.db/cdrs/data/create_time_day=2024-12-06/00120-6-e6f2443a-9344-4083-983a-0fcb79b5d9d9-00001.parquet
at org.apache.iceberg.hadoop.HadoopInputFile.getBlockLocations(HadoopInputFile.java:217)
at org.apache.iceberg.hadoop.Util.blockLocations(Util.java:111)
at org.apache.iceberg.hadoop.Util.blockLocations(Util.java:84)
at org.apache.iceberg.spark.source.SparkInputPartition.(SparkInputPartition.java:62)
at org.apache.iceberg.spark.source.SparkBatch.lambda$planInputPartitions$0(SparkBatch.java:90)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:69)
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: DestHost:destPort xxxxxxxx:8010 , LocalHost:localPort xxxxxxxx06/ip:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at sun.reflect.GeneratedConstructorAccessor92.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:948)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:923)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1588)
at org.apache.hadoop.ipc.Client.call(Client.java:1529)
at org.apache.hadoop.ipc.Client.call(Client.java:1426)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
at com.sun.proxy.$Proxy60.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.lambda$getBlockLocations$0(ClientNamenodeProtocolTranslatorPB.java:340)
at org.apache.hadoop.ipc.internal.ShadedProtobufHelper.ipc(ShadedProtobufHelper.java:160)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:340)
at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:437)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:170)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
at com.sun.proxy.$Proxy61.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:931)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:920)
at org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:977)
at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:289)
at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:286)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:296)
at org.apache.iceberg.hadoop.HadoopInputFile.getBlockLocations(HadoopInputFile.java:210)
... 12 more
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:754)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:709)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
... 36 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
... 39 more

[zhangwl9]

Has the Kerberos ticket expired (after more than 7 days)? If you execute kinit -kt xx xx and then perform the same operation again, will you still encounter the same issue

[jerui]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。