Node Tree Disappears when ImportAtStartup is Enabled

[kikodev714]

Describe the bug
We have upgraded to Umbraco 12.3.6, uSkinned 5.2.0, uSync 12.23. This upgrade had some document type changes that we need to deploy to our target sites. We usually accomplish that by this uSync config:

"ImportAtStartup": "Settings",

But after the upgrade this is not working plus, the backoffice node tree disappears for all sections:

Also when I checked the logs, the import failed so the site won't run due to outdated document types.

To Reproduce
Steps to reproduce the behavior:

1. We have upgraded from Umbraco 10.6.1 to 12.3.6 so to replicate this you must have an upgraded source pointed to a non-upgraded DB
2. Set ImportAtStartup to "Settings" or "Content" or "All" - any of these settings will break the node tree
3. Run the project

Expected behavior

1. Node tree should display even with ImportAtStartup = Settings
2. uSync import should run on startup and complete to update the doctypes in the DB

Screenshots
I'm trying to make sense of these error logs but I need some assistance:
{"@t":"2024-02-07T13:56:36.4752674Z","@mt":"Error occurred executing {WorkItem}.","@l":"Error","@x":"System.InvalidOperationException: The Scope 21097e61-a3d4-452f-893d-bb411685606a being disposed is not the Ambient Scope 517c2219-2859-4719-86f2-c289d24e95cd. This typically indicates that a child Scope was not disposed, or flowed to a child thread that was not awaited, or concurrent threads are accessing the same Scope (Ambient context) which is not supported. If using Task.Run (or similar) as a fire and forget tasks or to run threads in parallel you must suppress execution context flow with ExecutionContext.SuppressFlow() and ExecutionContext.RestoreFlow().\r\n at Umbraco.Cms.Infrastructure.Scoping.Scope.Dispose()\r\n at Umbraco.Cms.Infrastructure.Examine.ExamineUmbracoIndexingHandler.DeferredReIndexForContent.<>c__DisplayClass6_0.<Execute>b__0(CancellationToken cancellationToken)\r\n at Umbraco.Cms.Infrastructure.HostedServices.QueuedHostedService.BackgroundProcessing(CancellationToken stoppingToken)","WorkItem":"workItem","SourceContext":"Umbraco.Cms.Infrastructure.HostedServices.QueuedHostedService","ProcessId":21124,"ProcessName":"iisexpress","ThreadId":5,"ApplicationId":"98ebf3be2c8ffd5e65a4cf9a8388b2ef7687b9b9","MachineName":"GGG-99","Log4NetLevel":"ERROR"}

{"@t":"2024-02-07T14:09:57.9061459Z","@mt":"Exception ({InstanceId}).","@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\n ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.\r\n at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\r\n at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n at `

I'm actually seeing a bunch of SQL lock errors when this happens.

About your Site (please complete the following information):

• Umbraco Version: 12.3.6
• uSync Version: 12.2.3
• uSkinned 5.2.0
• Browser: Same issue with Chrome, Edge, etc.

Additional context
When I switch the site to use ImportAtStartup: None - The node trees for all section appear again. When I have access to the node tree I can see the errors from USync. Here's an example:

[kikodev714]

Some additional info that might help:

To see the node tree again I temporarily set ImportAtStartup:None . Then I went to uSync history and found this:

As you can see it was able to import some but failed on some.

I also tried to play around with the \ContentType\ folder a bit. If it's empty ImportAtStartup:Settings doesn't break the node tree.
If I add config files slowly (in batches of 5), it doesn't break the node tree. However, if I add 6 or more, then the node tree breaks and if I look at the uSync logs, that Ambient Scope error appears in the failed configs.

[KevinJump]

Hi,

I think we might need some input from the uSkinned team on this one - It looks like the failure is happening when the examine indexes are being built, which is triggered from a notification from within Umbraco after usync has saved the item.

There might be custom code in uSkinned that does something with examine, or indeed you might have some custom code that alters the examine indexes? something is attempting to acess the scope then?

[marcloveUSN]

Hi Kevin,

We dont have any Notification Handlers that alter the examine index. We do update FieldDefinitions to allow sorting on nodename with and without culture.

We follow this guide: https://docs.umbraco.com/umbraco-cms/reference/searching/examine/indexing

//Custom Examine configuration
builder.Services.ConfigureOptions();

And we also transform index values to include a couple of new fields to make searching easier:

//Update default Examine Index
builder.Components().Append();

Within this class we transform the index values via:

indexProvider.TransformingIndexValues += IndexProviderTransformingIndexValues;

Does this approach seem like it may be the cause of the issue?

Cheers,

Marc

[kikodev714]

Hello @KevinJump @marcloveUSN

Just like to follow-up if we have some updates regarding this one.

Thanks.

[jemayn]

Just wanted to pitch in and say that we have the samme issue on one of our sites.
Have had to disable importAtStartup to get the site running again.

Weird thing is - I tried running a database trace and can see lots of Umbraco schema things being done to the db before it hits this error, so it's not all content that fails. Anecdotaly it seems to be often wrt member group changes.

I was wondering if there was a way to fx schedule a hangfire job on startup that then ran a usync import 5 minutes after startup? As we've had no issues with manual imports after the site is running.

I've found the uSync command line but seems to be aimed at triggering it from outside the runtime of the site.