Copying files results in Command failed with code 123

[peteror]

I'm running a couple of fairly simple shell scripts using Digdag. Most of these involve copying files from a source dir to my user's home directory, then doing some operations via shell scripts (sed, transformation by calling some Java code, etc.). This works well.

However, as soon as I'd like to copy some processed files to any directory outside the my user's home dir (which is the same user Digdag runs as) I get this:

2021-07-15 18:39:40.426 +0300 [INFO] (660901@[0:test]+test+step1) io.digdag.core.agent.OperatorManager: sh>: find /home/auser/test/files -maxdepth 1 -type f -print0 | xargs -0 cp -t /data/transfer/staging/
cp: cannot create regular file ‘/data/transfer/staging/done202107151807926.log’: Permission denied
2021-07-15 18:39:40.434 +0300 [ERROR] (660901@[0:test]+test+step1) io.digdag.core.agent.OperatorManager: Task failed with unexpected error: Command failed with code 123
java.lang.RuntimeException: Command failed with code 123
	at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runCode(ShOperatorFactory.java:121)
	at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:88)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:365)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:298)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:151)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:77)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:149)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:132)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.MultiThreadAgent.lambda$run$0(MultiThreadAgent.java:132)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

I can run the same command from the command line (same user) and it works without an issue. I tried running the command from Digdag directly:

_export:
 sh:
  shell: ["/usr/bin/bash"]

+step1:
 sh>: find /home/auser/test/files -maxdepth 1 -type f -print0 | xargs -0 cp -t /data/transfer/staging/

+teardown:
 echo>: finish ${session_time}

or by calling a shell script, with the same result.

What am I doing wrong?

[hiroyuki-sato]

Hello, @peteror

I'll try to reproduce your issue in my environment.
Could you tell me the following?

• Digdag version:
• Java version: java -version
• OS version
• Digdag config (remove sensitive information)
• Error message (Digdag run result)
• Server mode or local mode
• Sample files. (/home/auser/test/files directory layouts)