Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add $transactions system table to Delta Lake #24292

Merged
merged 2 commits into from
Dec 5, 2024

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Nov 28, 2024

Description

This table will help us debug issues in transaction logs.
I didn't document it because it's mainly for debug purpose and I expect most users don't use it in normal situations.

trino> SELECT * FROM delta.tpch."region$transactions";

 version |                                                                                                                                                                                      transaction                                       >
---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
       0 | [ {                                                                                                                                                                                                                                    >
         |   "commitInfo" : {                                                                                                                                                                                                                     >
         |     "version" : 0,                                                                                                                                                                                                                     >
         |     "timestamp" : 1733023200751,                                                                                                                                                                                                       >
         |     "userId" : "user",                                                                                                                                                                                                                 >
         |     "userName" : "user",                                                                                                                                                                                                               >
         |     "operation" : "CREATE TABLE AS SELECT",                                                                                                                                                                                            >
         |     "operationParameters" : {                                                                                                                                                                                                          >
         |       "queryId" : "20241201_032000_00022_p82tv"                                                                                                                                                                                        >
         |     },                                                                                                                                                                                                                                 >
         |     "clusterId" : "trino-testversion-db504efc-1f50-4f2d-9605-a57eda620b9d",                                                                                                                                                            >
         |     "readVersion" : 0,                                                                                                                                                                                                                 >
         |     "isolationLevel" : "WriteSerializable",                                                                                                                                                                                            >
         |     "isBlindAppend" : true                                                                                                                                                                                                             >
         |   }                                                                                                                                                                                                                                    >
         | }, {                                                                                                                                                                                                                                   >
         |   "protocol" : {                                                                                                                                                                                                                       >
         |     "minReaderVersion" : 1,                                                                                                                                                                                                            >
         |     "minWriterVersion" : 2                                                                                                                                                                                                             >
         |   }                                                                                                                                                                                                                                    >
         | }, {                                                                                                                                                                                                                                   >
         |   "metaData" : {                                                                                                                                                                                                                       >
         |     "id" : "ccb5b6f1-4efd-46b8-a212-4c3d5dc8fec2",                                                                                                                                                                                     >
         |     "format" : {                                                                                                                                                                                                                       >
         |       "provider" : "parquet",                                                                                                                                                                                                          >
         |       "options" : { }                                                                                                                                                                                                                  >
         |     },                                                                                                                                                                                                                                 >
         |     "schemaString" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"regionkey\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"comment\",\"t>
         |     "partitionColumns" : [ ],                                                                                                                                                                                                          >
         |     "configuration" : {                                                                                                                                                                                                                >
         |       "delta.enableDeletionVectors" : "false"                                                                                                                                                                                          >
         |     },                                                                                                                                                                                                                                 >
         |     "createdTime" : 1733023200751                                                                                                                                                                                                      >
         |   }                                                                                                                                                                                                                                    >
         | }, {                                                                                                                                                                                                                                   >
         |   "add" : {                                                                                                                                                                                                                            >
         |     "path" : "20241201_032000_00022_p82tv_614227e8-71ab-4b9a-a8f4-f0f166fc11e0",                                                                                                                                                       >
         |     "partitionValues" : { },                                                                                                                                                                                                           >
         |     "size" : 937,                                                                                                                                                                                                                      >
         |     "modificationTime" : 1733023200743,                                                                                                                                                                                                >
         |     "dataChange" : true,                                                                                                                                                                                                               >
         |     "stats" : "{\"numRecords\":5,\"minValues\":{\"regionkey\":0,\"name\":\"AFRICA\",\"comment\":\"ges. thinly even pinto beans ca\"},\"maxValues\":{\"regionkey\":4,\"name\":\"MIDDLE EAST\",\"comment\":\"uickly special accounts cajo>
         |     "tags" : { }                                                                                                                                                                                                                       >
         |   }                                                                                                                                                                                                                                    >
         | } ]

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Nov 28, 2024
@github-actions github-actions bot added the delta-lake Delta Lake connector label Nov 28, 2024
@ebyhr ebyhr force-pushed the ebi/delta-history branch from 961c9f4 to a03d3f6 Compare November 28, 2024 04:01
@findinpath findinpath requested a review from mdesmet November 29, 2024 05:05
@ebyhr ebyhr force-pushed the ebi/delta-history branch from a03d3f6 to 78ef751 Compare December 1, 2024 03:20
@ebyhr ebyhr changed the title Add hidden $entries column to Delta history table Add $transactions system table to Delta Lake Dec 1, 2024
@ebyhr ebyhr force-pushed the ebi/delta-history branch from 78ef751 to c16cc73 Compare December 4, 2024 13:19
@ebyhr ebyhr force-pushed the ebi/delta-history branch from c16cc73 to d802427 Compare December 4, 2024 13:33
@ebyhr ebyhr force-pushed the ebi/delta-history branch from d802427 to f0581e4 Compare December 5, 2024 02:11
@ebyhr ebyhr requested a review from pajaks December 5, 2024 02:11
@ebyhr ebyhr merged commit 62d77f7 into trinodb:master Dec 5, 2024
22 checks passed
@ebyhr ebyhr deleted the ebi/delta-history branch December 5, 2024 03:56
@github-actions github-actions bot added this to the 467 milestone Dec 5, 2024
@mosabua
Copy link
Member

mosabua commented Dec 5, 2024

We still have to at least add it to the release notes.. and I think we also have to document it.

@mosabua
Copy link
Member

mosabua commented Dec 5, 2024

Confirmed with @martint when talking about the release notes.. if its useful for us as users and developers .. its useful for others.. so we should document.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed delta-lake Delta Lake connector
Development

Successfully merging this pull request may close these issues.

5 participants