Forced start of the history archiver or deleting workflow history immediately

[ceearrashee]

Hello everyone!

Background: My team created a domain with retentions day equal for one year more than two years ago. With such a long time, our database (we use MySQL database) has grown to a vast size, has become expensive to maintain. After discussing the problem with the team, we decided that we do not need such a prolonged storage period for the history of the execution of various workflows. A history of 30 days is quite enough. When trying to change the retention period, I ran into the problem of cluster naming, but I successfully solved it in the end.

The retention period has changed from 365 days to 30 days, but the size of the table with the node_history has not changed even after a few weeks. I thought that the task of starting archiving or deleting old workflows would begin after some time, but nothing happened. How I understood that nothing happened:

1. I counted the number of records in the table - it stays the same;
2. I have performed a table optimization operation (shrink) - table size did not change.

Colleagues, tell me how to start a forced history cleanup.
As a target: I want to remove old workflow histories to reduce the Cadence node_history table to reduce space used by the Cadence.

[longquanzheng]

There is a command cadence --do samples-domain adm wf refresh-tasks -w <> to refresh background tasks. But looks like it doesn't work for closed workflows. I think we should fix that.

cc @yycptt

see below comments from @yycptt about fixing the command for closed workflows

[longquanzheng]

I have verified that admin workflow refresh-tasks command can be used to force start of history archival for open workflows.

Steps:

1. Use a domain retention >0 to start a workflow. Remember the workflowID and runID. e.g.: cadence --do samples-domain wf start --wt wt --tl tl --et 300
2. Update the domain retention to 0 (so that workflow will be archived and deleted immediately when closed)e.g.:cadence --do samples-domain d up --rd 0
3. Run the command to refresh tasks for the workflow e.g.:cadence --do samples-domain adm wf refresh-tasks -w 00bd01db-5797-488f-b0dc-e34d90665e78
4. Terminate the workflows e.g.:cadence --do samples-domain wf terminate -w 00bd01db-5797-488f-b0dc-e34d90665e78
5. Show the history from archival e.g.: cadence --do samples-domain wf show -w <> -r <> Note that RunID is required when getting history from archival.

Final results:

cadence --do samples-domain wf show -w 00bd01db-5797-488f-b0dc-e34d90665e78 -r 1bc87bb4-61dc-4c4b-9b7b-7c182a85d058
  1  WorkflowExecutionStarted     {WorkflowType:{Name:wt}, TaskList:{Name:tl},
                                   Input:[], ExecutionStartToCloseTimeoutSeconds:300,
                                   TaskStartToCloseTimeoutSeconds:10,
                                   ContinuedFailureDetails:[], LastCompletionResult:[],
                                   OriginalExecutionRunId:1bc87bb4-61dc-4c4b-9b7b-7c182a85d058,
                                   Identity:cadence-cli@IT-USA-25920,
                                   FirstExecutionRunId:1bc87bb4-61dc-4c4b-9b7b-7c182a85d058,
                                   Attempt:0, FirstDecisionTaskBackoffSeconds:0}
  2  DecisionTaskScheduled        {TaskList:{Name:tl},
                                   StartToCloseTimeoutSeconds:10,
                                   Attempt:0}
  3  WorkflowExecutionTerminated  {Details:[],
                                   Identity:53163@IT-USA-25920@}
History Source: History Archival

[ceearrashee]

Maybe I have a problem with DB growth cos I have: History Source: Default Storage.

[longquanzheng]

Yes, that’s why we will let you use refresh task command to force workflow to be archived

[longquanzheng]

please ignore my below comment because @yycptt has provided a fix to refresh-tasks command for closed workflow

Another idea for closed workflow: before we fix the refresh tasks command for closed workflows, you can use reset. After you have updated the retention to zero, reset the closed workflow to last decision completed and let the workflow complete again. Then you should be able to archive it.
A disadvantage is that requires workflow worker still be able to close the workflows that were closed long time ago

[ceearrashee]

Did I understand correctly that only active workflows would move into the archive?

Currently, I have more than 25000 closed workflows (completed, terminated, failed). And many of them are different, and they use different workers, task lists, and flow lengths.

How can I do it automatically, cos to do it manually is too expensive?
On attached screenshot is a part of my zoo.

[longquanzheng]

Did I understand correctly that only active workflows would move into the archive?

After the fix below, closed workflows can be archived as well.

[yycptt]

I have sent out a PR (#4472) for fixing workflow refresh for closed workflows. So after domain retention is updated, refresh the workflow will archive the workflow if the new retention has passed.

Here's an example:

➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain d re 
Domain samples-domain successfully registered.
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain d desc
Name: samples-domain
UUID: db54542b-47bb-4e92-a82e-438bf1712b24
Description: 
OwnerEmail: 
DomainData: map[]
Status: REGISTERED
RetentionInDays: 3
EmitMetrics: true
IsGlobal(XDC)Domain: true
ActiveClusterName: cluster0
Clusters: cluster0
HistoryArchivalStatus: ENABLED
HistoryArchivalURI: file:///tmp/cadence_archival/development
VisibilityArchivalStatus: ENABLED
VisibilityArchivalURI: file:///tmp/cadence_vis_archival/development
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain wf start --tl random-tl --wid random-wf-id-2 --wt random-wf-type --et 60 --dt 60                  
Started Workflow Id: random-wf-id-2, run Id: 826b421a-179c-433c-b572-9398dd5b1db3
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain wf term --wid random-wf-id-2  Terminate workflow succeeded.
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain wf show --wid random-wf-id-2
  1  WorkflowExecutionStarted     {WorkflowType:{Name:random-wf-type},                          
                                   TaskList:{Name:random-tl}, Input:[],                          
                                   ExecutionStartToCloseTimeoutSeconds:60,                       
                                   TaskStartToCloseTimeoutSeconds:60,                            
                                   ContinuedFailureDetails:[], LastCompletionResult:[],          
                                   OriginalExecutionRunId:826b421a-179c-433c-b572-9398dd5b1db3,  
                                   Identity:cadence-cli@ycyang-C02XW41WJGH6,                     
                                   FirstExecutionRunId:826b421a-179c-433c-b572-9398dd5b1db3,     
                                   Attempt:0, FirstDecisionTaskBackoffSeconds:0}                 
  2  DecisionTaskScheduled        {TaskList:{Name:random-tl},                                   
                                   StartToCloseTimeoutSeconds:60,                                
                                   Attempt:0}                                                    
  3  WorkflowExecutionTerminated  {Details:[],                                                  
                                   Identity:12126@ycyang-C02XW41WJGH6@}                          
History Source: Default Storage
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain d u --rd 0
Domain samples-domain successfully updated.
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain d desc    
Name: samples-domain
UUID: db54542b-47bb-4e92-a82e-438bf1712b24
Description: 
OwnerEmail: 
DomainData: map[]
Status: REGISTERED
RetentionInDays: 0
EmitMetrics: true
IsGlobal(XDC)Domain: true
ActiveClusterName: cluster0
Clusters: cluster0
HistoryArchivalStatus: ENABLED
HistoryArchivalURI: file:///tmp/cadence_archival/development
VisibilityArchivalStatus: ENABLED
VisibilityArchivalURI: file:///tmp/cadence_vis_archival/development
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain adm wf rt --wid random-wf-id-2
Refresh workflow task succeeded.
➜  cadence (fix-close-workflow-refresh) ✔ ./cadence --do samples-domain wf show --wid random-wf-id-2 --rid 826b421a-179c-433c-b572-9398dd5b1db3
  1  WorkflowExecutionStarted     {WorkflowType:{Name:random-wf-type},                          
                                   TaskList:{Name:random-tl}, Input:[],                          
                                   ExecutionStartToCloseTimeoutSeconds:60,                       
                                   TaskStartToCloseTimeoutSeconds:60,                            
                                   ContinuedFailureDetails:[], LastCompletionResult:[],          
                                   OriginalExecutionRunId:826b421a-179c-433c-b572-9398dd5b1db3,  
                                   Identity:cadence-cli@ycyang-C02XW41WJGH6,                     
                                   FirstExecutionRunId:826b421a-179c-433c-b572-9398dd5b1db3,     
                                   Attempt:0, FirstDecisionTaskBackoffSeconds:0}                 
  2  DecisionTaskScheduled        {TaskList:{Name:random-tl},                                   
                                   StartToCloseTimeoutSeconds:60,                                
                                   Attempt:0}                                                    
  3  WorkflowExecutionTerminated  {Details:[],                                                  
                                   Identity:12126@ycyang-C02XW41WJGH6@}                          
History Source: History Archival

[yycptt]

Did I understand correctly that only active workflows would move into the archive?

History (the workflow history) archival happens after your workflow closes + the domain retention period (the period when the workflow is closed).

Visibility (the list of workflow records you see in webUI) archival happens after the workflow is closed. So between the retention period visibility records are available from both default storage and archival storage. But this is an implementation detail, visibility records in archival storage should only be checked after the retention period. We may change when visibility archival is done in the future.

Archival won't happen to any open workflows.

[longquanzheng]

Thanks to @yycptt fix to the command. Now we have a good solution to your problem.

A few questions before I provided a detailed instructions for you:

1. Have you enabled the archival feature? https://cadenceworkflow.io/docs/concepts/archival . Note that the dev setup of using local file system for archival wouldn’t work for production. Unless you have a distributed file system, you need to use AWS S3 or GCP blob store.

2. What is your current server version? I can provide a patch on your current version so that you don’t need to worry about upgrading

[ceearrashee]

Hello!
I try to enable archival, but it doesn't work for me.

/app # ./cadence d up --huri file:///tmp/cadence_archival/staging --has enabled
Domain paser successfully updated.
/app # ./cadence d desc
Name: paser
UUID: dcc71fdb-44eb-41f7-940d-96df1c1e59d3
Description: 
OwnerEmail: 
DomainData: map[]
Status: REGISTERED
RetentionInDays: 30
EmitMetrics: false
ActiveClusterName: active
Clusters: active
HistoryArchivalStatus: DISABLED
VisibilityArchivalStatus: DISABLED
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+

[longquanzheng]

No, we are not using the archival features. Is the archival feature required to remove history under retention?

It is not. So your question title mislead me. If you don't need to archive, you can simple use the admin command to delete the workflow entity and history from DB:
see cadence admin workflow delete —help

You need to pass the DB address/port/password/workflowID to delete a workflow

[longquanzheng]

I try to enable archival, but it doesn't work for me.

File base archival will work for a cluster only if you have a distributed file system. Otherwise you have to enable it via AWS S3 or GCP cloud blob store.

However, seems like you don't want to archive the history. You just need to delete them. See my above command to use cadence admin workflow delete command to achieve it.

[ceearrashee]

This time I got confused.
I understand correct: After fixing the error mentioned above, the workflows in the final state (Completed, Canceled, Terminated, Failed) would be automatically deleted if they were got those states early days than the current date minus the retention period?

[longquanzheng]

That’s correct. You can go that way too. But you need to wait for the fix to be released, maybe a patch release.

The alternative is easier — using admin command to delete them right away.

The reason I thought we need to fix the error is that you said you wanted to archive the closed workflows. The refresh command will delete the closed workflows if passing retention and then archive them. If you don’t need to archive then probably we don’t have to fix the refresh command.

Lmk which route you want to go. I can still release the patch version for you.

[longquanzheng]

Also , CLI has a command to hard delete the workflow entity and history without archiving:

see “cadence admin workflow delete —help”