Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Catalog support only one table format now #3379

Open
1 of 2 tasks
jun7891 opened this issue Dec 23, 2024 · 0 comments · May be fixed by #3381
Open
1 of 2 tasks

[Bug]: Catalog support only one table format now #3379

jun7891 opened this issue Dec 23, 2024 · 0 comments · May be fixed by #3381
Labels
type:bug Something isn't working

Comments

@jun7891
Copy link

jun7891 commented Dec 23, 2024

What happened?

An error occurred when creating a database using the catalog terminal that supports multiple table formats. Similarly, the same error will also appear when using the InternalCatalogBuilder in the datastream to load the catalog that supports multiple table formats.(iceberg and mixed-iceberg).

Affects Versions

0.7.1-incubating

What table formats are you seeing the problem on?

Iceberg, Mixed-Iceberg

What engines are you seeing the problem on?

AMS, Flink

How to reproduce

285e04b0964e2bf60a54313edc4f66f
014b4a55189e02860c840126575bcc9
image

Relevant log output

java.lang.IllegalStateException: failed when load catalog mixed_catalog
	at org.apache.amoro.mixed.CatalogLoader.loadCatalog(CatalogLoader.java:183)
	at org.apache.amoro.mixed.CatalogLoader.load(CatalogLoader.java:64)
	at org.apache.amoro.spark.mixed.MixedSparkCatalogBase.initialize(MixedSparkCatalogBase.java:89)
	at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:60)
	at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$catalog$1(CatalogManager.scala:53)
	at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
	at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:53)
	at org.apache.spark.sql.connector.catalog.LookupCatalog$CatalogAndNamespace$.unapply(LookupCatalog.scala:86)
	at org.apache.spark.sql.catalyst.analysis.ResolveCatalogs$$anonfun$apply$1.applyOrElse(ResolveCatalogs.scala:33)
	at org.apache.spark.sql.catalyst.analysis.ResolveCatalogs$$anonfun$apply$1.applyOrElse(ResolveCatalogs.scala:32)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:170)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:168)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:164)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$4(AnalysisHelper.scala:175)
	at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1228)
	at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1227)
	at org.apache.spark.sql.catalyst.plans.logical.SetCatalogAndNamespace.mapChildren(v2Commands.scala:752)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$1(AnalysisHelper.scala:175)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning(AnalysisHelper.scala:168)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDownWithPruning$(AnalysisHelper.scala:164)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDownWithPruning(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsWithPruning(AnalysisHelper.scala:99)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsWithPruning$(AnalysisHelper.scala:96)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsWithPruning(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators(AnalysisHelper.scala:76)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators$(AnalysisHelper.scala:75)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.analysis.ResolveCatalogs.apply(ResolveCatalogs.scala:32)
	at org.apache.spark.sql.catalyst.analysis.ResolveCatalogs.apply(ResolveCatalogs.scala:28)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:211)
	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	at scala.collection.immutable.List.foldLeft(List.scala:91)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:208)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:200)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:231)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:227)
	at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:173)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:227)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:188)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:179)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
	at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:179)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:212)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:211)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:76)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:185)
	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
	at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:184)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:76)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:66)
	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
	at org.apache.amoro.server.terminal.local.LocalTerminalSession.executeStatement(LocalTerminalSession.java:72)
	at org.apache.amoro.server.terminal.TerminalSessionContext$ExecutionTask.executeStatement(TerminalSessionContext.java:302)
	at org.apache.amoro.server.terminal.TerminalSessionContext$ExecutionTask.execute(TerminalSessionContext.java:266)
	at org.apache.amoro.server.terminal.TerminalSessionContext$ExecutionTask.lambda$get$0(TerminalSessionContext.java:227)
	at org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:256)
	at org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:229)
	at org.apache.amoro.server.terminal.TerminalSessionContext$ExecutionTask.get(TerminalSessionContext.java:209)
	at org.apache.amoro.server.terminal.TerminalSessionContext$ExecutionTask.get(TerminalSessionContext.java:184)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: Catalog support only one table format now.
	at org.apache.amoro.shade.guava32.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
	at org.apache.amoro.mixed.CatalogLoader.catalogImpl(CatalogLoader.java:86)
	at org.apache.amoro.mixed.CatalogLoader.createCatalog(CatalogLoader.java:201)
	at org.apache.amoro.mixed.CatalogLoader.loadCatalog(CatalogLoader.java:175)
	... 80 more


Exception in thread "main" java.lang.IllegalStateException: failed when load catalog mixed_catalog
	at org.apache.amoro.mixed.CatalogLoader.loadCatalog(CatalogLoader.java:183)
	at org.apache.amoro.mixed.CatalogLoader.load(CatalogLoader.java:64)
	at org.apache.amoro.flink.InternalCatalogBuilder.createMixedFormatCatalog(InternalCatalogBuilder.java:47)
	at org.apache.amoro.flink.InternalCatalogBuilder.build(InternalCatalogBuilder.java:65)
	at org.example.IcebergTest.main(IcebergTest.java:13)
Caused by: java.lang.IllegalArgumentException: Catalog support only one table format now.
	at org.apache.amoro.shade.guava32.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
	at org.apache.amoro.mixed.CatalogLoader.catalogImpl(CatalogLoader.java:86)
	at org.apache.amoro.mixed.CatalogLoader.createCatalog(CatalogLoader.java:201)
	at org.apache.amoro.mixed.CatalogLoader.loadCatalog(CatalogLoader.java:175)
	... 4 more

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct
@jun7891 jun7891 added the type:bug Something isn't working label Dec 23, 2024
@ihadoop ihadoop linked a pull request Dec 25, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[AMORO-3379][amoro-web] only one catalog can be selected ihadoop/amoro
1 participant
@jun7891