Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

incremental backup failed: Bitmap does not exist #896

Open
meilihao opened this issue Nov 29, 2023 · 2 comments
Open

incremental backup failed: Bitmap does not exist #896

meilihao opened this issue Nov 29, 2023 · 2 comments
Labels
bug

Comments

@meilihao
Copy link

Affected version

oVirt Engine: ovirt-engine-4.5.4-1.el8.noarch
Host OS Variant: ovirt-node-ng-installer-4.5.4-2022120615.el8.iso
VDSM version: vdsm-4.50.3.4-1.el8.x86_64

Describe the bug

op log:

# --- backup_vm.py is from python-ovirt-engine-sdk4-4.6.1/examples
# ./backup_vm.py -c engine1 --debug full --backup-dir bak_dir 95c839b9-04c6-4f9e-b113-83e0aefcddb1
[   0.0 ] Starting full backup for VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1'
[   1.3 ] Waiting until backup '05d48f54-abed-4a7b-9273-4bc58902f671' is ready
[  15.6 ] Created checkpoint '79af917a-b113-45e7-bc10-ead951260373'
[  15.6 ] Downloading full backup for disk '198668cf-7f01-467a-badc-85396034c1de'
[  15.6 ] Creating backup file 'bak_dir/20231129174020.79af917a-b113-45e7-bc10-ead951260373.198668cf-7f01-467a-badc-85396034c1de.full.qcow2'
[  16.9 ] Image transfer 'd692b395-8110-4230-a6f3-d03966e558ec' is ready
[ 100% ] 10.00 GiB, 58.70 s, 174.46 MiB/s                                      
[  75.6 ] Finalizing image transfer
[  82.8 ] Download completed successfully
[  82.8 ] Finalizing backup
[ 159.2 ] Full backup '05d48f54-abed-4a7b-9273-4bc58902f671' completed successfully
# ./backup_vm.py -c engine1 --debug start --from-checkpoint-uuid 79af917a-b113-45e7-bc10-ead951260373 95c839b9-04c6-4f9e-b113-83e0aefcddb1
[   0.0 ] Starting incremental backup since checkpoint '79af917a-b113-45e7-bc10-ead951260373' for VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1'
[   1.1 ] Waiting until backup '304dbd1b-1074-4d3d-8c9c-d09b1e117cdf' is ready
[  14.3 ] Created checkpoint 'f8453825-e0aa-484b-90c9-c759be8b222b'
[  14.4 ] Backup '304dbd1b-1074-4d3d-8c9c-d09b1e117cdf' is ready
05:43:32 chen@fyhdesktop29 examples ±|main ✗|→ ./backup_vm.py -c engine1 --debug download --backup-uuid 304dbd1b-1074-4d3d-8c9c-d09b1e117cdf --incremental 95c839b9-04c6-4f9e-b113-83e0aefcddb1
[   0.0 ] Downloading VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1' disks
[   0.6 ] Downloading incremental backup for disk '198668cf-7f01-467a-badc-85396034c1de'
[   0.6 ] Creating backup file './20231129174405.f8453825-e0aa-484b-90c9-c759be8b222b.198668cf-7f01-467a-badc-85396034c1de.incremental.qcow2'
Traceback (most recent call last):
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 542, in <module>
    main()
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 145, in main
    args.command(args)
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 227, in cmd_download
    download_backup(connection, backup, args, incremental=args.incremental)
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 406, in download_backup
    download_disk(
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 427, in download_disk
    transfer = imagetransfer.create_transfer(
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/helpers/imagetransfer.py", line 200, in create_transfer
    transfer = transfers_service.add(transfer)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/services.py", line 14109, in add
    return self._internal_add(image_transfer, headers, query, wait)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 232, in _internal_add
    return future.wait() if wait else future
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 55, in wait
    return self._code(response)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 229, in callback
    self._check_fault(response)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 132, in _check_fault
    self._raise_error(response, body)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 118, in _raise_error
    raise error
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400.

engine log:

2023-11-29 17:24:40,116+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [8dab67a2-29c3-455b-a63b-5a75fdc8199d] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_SUCCESS(68), Snapshot 'Auto-generated for Backup VM' creation for VM 'test' has been completed.
2023-11-29 17:24:40,150+08 INFO  [org.ovirt.engine.core.bll.storage.backup.HybridBackupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [8dab67a2-29c3-455b-a63b-5a75fdc8199d] Change VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1' backup '304dbd1b-1074-4d3d-8c9c-d09b1e117cdf' phase from 'STARTING' to 'READY'
2023-11-29 17:24:40,511+08 INFO  [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-146) [63e5ef3b] User admin@ovirt@internalkeycloak-authz with profile [internalsso] successfully logged out
2023-11-29 17:24:40,530+08 INFO  [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-154) [307fc464] Running command: TerminateSessionsForTokenCommand internal: true.
2023-11-29 17:24:40,530+08 INFO  [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-154) [307fc464] Not removing session 'e7AHKd/znvdsye0WvjEPKXix5z/qwkBVDaXo4XHsxno7x22lB3HVOYrWgCRAHa08k+LIeUNu1g1KoPpYBP/YCA==', session has running commands for user 'admin@ovirt@internalkeycloak-authz'.
2023-11-29 17:25:04,158+08 INFO  [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-42) [] Not removing session 'e7AHKd/znvdsye0WvjEPKXix5z/qwkBVDaXo4XHsxno7x22lB3HVOYrWgCRAHa08k+LIeUNu1g1KoPpYBP/YCA==', session has running commands for user 'admin@ovirt@internalkeycloak-authz'.
2023-11-29 17:25:07,370+08 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] Setting new tasks map. The map contains now 1 tasks
2023-11-29 17:25:12,431+08 INFO  [org.ovirt.engine.core.sso.service.ExternalOIDCService] (default task-145) [] User admin@ovirt@internalkeycloak-authz with profile [internalsso] successfully logged into external OP with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access
2023-11-29 17:25:12,518+08 INFO  [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-145) [56312054] Running command: CreateUserSessionCommand internal: false.
2023-11-29 17:25:12,567+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [56312054] EVENT_ID: USER_VDC_LOGIN(30), User admin@ovirt@internalkeycloak-authz connecting from '192.168.88.236' using session 'jXndZzjdDxXOJNPeQbUBDrbcn1IID2sOs5baDTAWJcFX3tNyw03UFu+Ekc/SXDo85djAGO2yhjfjyW9biVRenA==' logged in.
2023-11-29 17:25:12,788+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-11-29 17:25:12,811+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Running command: TransferDiskImageCommand internal: false. Entities affected :  ID: 198668cf-7f01-467a-badc-85396034c1de Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER
2023-11-29 17:25:12,811+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Creating ImageTransfer entity for command 'b06937e2-da39-4685-86e5-634b0cdbdb3d', proxyEnabled: true
2023-11-29 17:25:12,814+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Starting image transfer: ImageTransfer:{id='b06937e2-da39-4685-86e5-634b0cdbdb3d', phase='Initializing', type='Download', active='false', lastUpdated='Wed Nov 29 17:25:12 CST 2023', message='null', vdsId='null', diskId='null', imagedTicketId='null', proxyUri='null', bytesSent='null', bytesTotal='10737418240', clientInactivityTimeout='60', timeoutPolicy='cancel', imageFormat='RAW', transferClientType='Transfer via API', shallow='false'}
2023-11-29 17:25:12,816+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Successfully added Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26') for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d'
2023-11-29 17:25:12,833+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, PrepareImageVDSCommand(HostName = node151, PrepareImageVDSCommandParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d'}), log id: 6d2edf4f
2023-11-29 17:25:12,980+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 6d2edf4f
2023-11-29 17:25:12,996+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, StartNbdServerVDSCommand(HostName = node151, NbdServerVDSParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', serverId='b06937e2-da39-4685-86e5-634b0cdbdb3d', storageDomainId='3eab4802-18dd-4a4d-a140-ac35edca735a', imageId='198668cf-7f01-467a-badc-85396034c1de', volumeId='c0586bf2-e409-4e4e-98e1-f62ef967da26', readonly='true', discard='true', detectZeroes='true', backingChain='true', bitmap='79af917a-b113-45e7-bc10-ead951260373'}), log id: 7a95154c
2023-11-29 17:25:13,028+08 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Unexpected return value: Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"]
2023-11-29 17:25:13,028+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Failed in 'StartNbdServerVDS' method
2023-11-29 17:25:13,028+08 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Unexpected return value: Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"]
2023-11-29 17:25:13,032+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM node151 command StartNbdServerVDS failed: Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"
2023-11-29 17:25:13,032+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand' return value 'NbdServerURLReturn:{status='Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"]'}'
2023-11-29 17:25:13,032+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] HostName = node151
2023-11-29 17:25:13,033+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Command 'StartNbdServerVDSCommand(HostName = node151, NbdServerVDSParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', serverId='b06937e2-da39-4685-86e5-634b0cdbdb3d', storageDomainId='3eab4802-18dd-4a4d-a140-ac35edca735a', imageId='198668cf-7f01-467a-badc-85396034c1de', volumeId='c0586bf2-e409-4e4e-98e1-f62ef967da26', readonly='true', discard='true', detectZeroes='true', backingChain='true', bitmap='79af917a-b113-45e7-bc10-ead951260373'})' execution failed: VDSGenericException: VDSErrorException: Failed to StartNbdServerVDS, error = Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}", code = 947
2023-11-29 17:25:13,033+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, StartNbdServerVDSCommand, return: , log id: 7a95154c
2023-11-29 17:25:13,033+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Failed to start NBD server for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d': {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StartNbdServerVDS, error = Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}", code = 947 (Failed with error unexpected and code 16)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2121)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:1084)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:681)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:518)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2010)
	at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140)
	at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:424)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387)
	at jdk.internal.reflect.GeneratedMethodAccessor860.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at [email protected]//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at [email protected]//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92)
	at [email protected]//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107)
	at [email protected]//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126)
	at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
	at jdk.internal.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at [email protected]//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
	at [email protected]//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92)
	at [email protected]//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124)
	at [email protected]//org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105)
	at [email protected]//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82)
	at [email protected]//org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)
	at [email protected]//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77)
	at [email protected]//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)
	at [email protected]//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)
	at [email protected]//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13)
	at jdk.internal.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at [email protected]//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at [email protected]//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232)
	at [email protected]//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446)
	at [email protected]//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at [email protected]//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
	at [email protected]//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at [email protected]//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633)
	at [email protected]//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at [email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at [email protected]//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at [email protected]//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at [email protected]//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191)
	at [email protected]//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:251)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:279)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:171)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:160)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:187)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:180)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:82)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForDisk(BackendImageTransfersResource.java:49)
	at jdk.internal.reflect.GeneratedMethodAccessor2260.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.ovirt.engine.api.restapi-definition//org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:91)
	at jdk.internal.reflect.GeneratedMethodAccessor2254.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at [email protected]//org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
	at [email protected]//org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
	at [email protected]//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
	at [email protected]//org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
	at [email protected]//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
	at [email protected]//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
	at [email protected]//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
	at [email protected]//org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
	at [email protected]//org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
	at [email protected]//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
	at [email protected]//org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
	at [email protected]//org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
	at [email protected]//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
	at [email protected]//org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
	at [email protected]//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at [email protected]//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at [email protected]//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)
	at [email protected]//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at [email protected]//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at [email protected]//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:251)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:186)
	at [email protected]//io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:227)
	at [email protected]//io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149)
	at [email protected]//io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:167)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:87)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:106)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:61)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:85)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:46)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:100)
	at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:91)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:272)
	at [email protected]//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at [email protected]//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at [email protected]//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at [email protected]//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at [email protected]//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at [email protected]//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at [email protected]//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
	at [email protected]//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
	at [email protected]//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at [email protected]//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at [email protected]//io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
	at [email protected]//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at [email protected]//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at [email protected]//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at [email protected]//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
	at [email protected]//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
	at [email protected]//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at [email protected]//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at [email protected]//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
	at [email protected]//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
	at [email protected]//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
	at [email protected]//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
	at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at [email protected]//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StartNbdServerVDS, error = Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}", code = 947
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:224)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:194)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand.executeVdsBrokerCommand(StartNbdServerVDSCommand.java:21)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
	at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:410)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at [email protected]//org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51)
	at [email protected]//org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78)
	at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
	at jdk.internal.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at [email protected]//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
	at [email protected]//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
	at [email protected]//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
	at [email protected]//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
	at [email protected]//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
	at [email protected]//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source)
	... 216 more

2023-11-29 17:25:13,039+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_CREATE_TICKET(1,068), Transfer was stopped by system. Reason: failed to create a signed image ticket.
2023-11-29 17:25:13,040+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Updating image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' phase from 'Initializing' to 'Cancelled'
2023-11-29 17:25:13,098+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Download with disk CentOS-8-GenericCloud-8.1.1911 was initiated by admin@ovirt@internalkeycloak-authz.
2023-11-29 17:25:13,100+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-11-29 17:25:13,101+08 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-145) [] Operation Failed: []
2023-11-29 17:25:13,158+08 INFO  [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-146) [63e5ef3b] User admin@ovirt@internalkeycloak-authz with profile [internalsso] successfully logged out
2023-11-29 17:25:13,171+08 INFO  [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-154) [22ab80fd] Running command: TerminateSessionsForTokenCommand internal: true.
2023-11-29 17:25:13,291+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' cancelled by system for Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26')
2023-11-29 17:25:13,292+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Updating image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' phase from 'Cancelled' to 'Finalizing Failure'
2023-11-29 17:25:15,336+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Finalizing failed image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' for Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26')
2023-11-29 17:25:15,336+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RemoveImageTicketVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, RemoveImageTicketVDSCommand(HostName = node151, RemoveImageTicketVDSCommandParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', ticketId='317f9abf-212d-4714-923a-e6d8f1ab8854', timeout='null'}), log id: 600ce72f
2023-11-29 17:25:15,342+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RemoveImageTicketVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, RemoveImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 600ce72f
2023-11-29 17:25:15,342+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Successfully stopped image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' for ticket '317f9abf-212d-4714-923a-e6d8f1ab8854'
2023-11-29 17:25:15,342+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Removing proxy image ticket '317f9abf-212d-4714-923a-e6d8f1ab8854' for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d'
2023-11-29 17:25:15,344+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StopNbdServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, StopNbdServerVDSCommand(HostName = node151, NbdServerVDSParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', serverId='b06937e2-da39-4685-86e5-634b0cdbdb3d', storageDomainId='null', imageId='null', volumeId='null', readonly='false', discard='false', detectZeroes='false', backingChain='true', bitmap='null'}), log id: 509d00cf
2023-11-29 17:25:15,371+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StopNbdServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, StopNbdServerVDSCommand, return: , log id: 509d00cf
2023-11-29 17:25:15,376+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' failed for Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26')
2023-11-29 17:25:16,379+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand' with failure.
2023-11-29 17:25:16,379+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Failed to transfer disk '00000000-0000-0000-0000-000000000000' for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d'
2023-11-29 17:25:16,379+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-11-29 17:25:16,381+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Updating image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' phase from 'Finalizing Failure' to 'Finished Failure'
2023-11-29 17:25:16,414+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: TRANSFER_IMAGE_CANCELLED(1,033), Image Download with disk CentOS-8-GenericCloud-8.1.1911 was cancelled.
^C

full log: full.log

To reproduce

see op log

Expected behavior

incremental backup is ok
@meilihao meilihao added the bug label Nov 29, 2023
@meilihao
Copy link
Author

my disk is qcow2 compat 0.10, upgrade compat to 1.1, incremental backup works.

@meilihao
Copy link
Author

meilihao commented Dec 5, 2023

The document only mentions that qcow2 is required. Does incremental backup require qcow2 v3?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@meilihao