Assertion failed in System.Text.Json.JsonSerializer.LookupProperty

[jkotas]

 Discovering: System.Text.Json.SourceGeneration.Roslyn3.11.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.SourceGeneration.Roslyn3.11.Tests (found 1815 of 1942 test cases)
  Starting:    System.Text.Json.SourceGeneration.Roslyn3.11.Tests (parallel test collections = on, max threads = 4)
Process terminated. Assertion failed.
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.HandlePropertyName.cs:line 31
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectDefaultConverter.cs:line 63
   at System.Text.Json.Serialization.Converters.JsonMetadataServicesConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/JsonMetadataServicesConverter.cs:line 53
...

Failed in #67814

Log + dump: https://dev.azure.com/dnceng/public/_build/results?buildId=1709347&view=ms.vss-test-web.build-test-results-tab&runId=46503608&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=190696

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

 Discovering: System.Text.Json.SourceGeneration.Roslyn3.11.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.SourceGeneration.Roslyn3.11.Tests (found 1815 of 1942 test cases)
  Starting:    System.Text.Json.SourceGeneration.Roslyn3.11.Tests (parallel test collections = on, max threads = 4)
Process terminated. Assertion failed.
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.HandlePropertyName.cs:line 31
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectDefaultConverter.cs:line 63
   at System.Text.Json.Serialization.Converters.JsonMetadataServicesConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/JsonMetadataServicesConverter.cs:line 53
...

Failed in #67814

Log + dump: https://dev.azure.com/dnceng/public/_build/results?buildId=1709347&view=ms.vss-test-web.build-test-results-tab&runId=46503608&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=190696

Author:	jkotas
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[jkotas]

A new race condition in the System.Text.Json tests?

[jkotas]

Failed again in #67788

[krwq]

I think it's same one surfaced elsewhere #67761

[eiriktsarpalis]

@krwq I'm guessing by the stacktrace that this is not related to JsonTypeInfo initialization? Is it a new failure?

[krwq]

I'm pretty sure it is related to initialization. I suspect that with my changes same thing just shows in different place as I'm changing when initialization happens.

[danmoseley]

No hits since the new diagnostics were added.

Execute: Web | Desktop | Web (Lens) | Desktop (SAW)

https://engsrvprod.kusto.windows.net/engineeringdata

WorkItems
| where FriendlyName == "System.Text.Json.SourceGeneration.Roslyn3.11.Tests"
| where  Queued > ago(2d)
| where ExitCode  > 0
| join Jobs on JobId
| project
  Queued,
  FriendlyName, ExitCode, Status,
  ConsoleUri,
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  RuntimeFlavor = tostring(parse_json(Properties).runtimeFlavor),
  BuildId = tostring(parse_json(Properties).BuildId),
  QueueName, Source

then examined the console log.

[krwq]

@danmoseley actually there seem to be few hits (also I changed to 1D since I merged my second change yesterday):

WorkItems
| where FriendlyName == "System.Text.Json.SourceGeneration.Roslyn3.11.Tests"
| where  Queued > ago(1d)
| where ExitCode  > 0

seems that join is causing to show no results.

I'm super surprised by which assert failed - quickly looking at the code further suggests some kind of memory corruption but at the same it's even more interesting why that would be exactly the same place in the code

Here are few relevant stacks I found [I'll be updating this once I find new ones]:

Stack trace 1, at least 2 hits; error code 134

Process terminated. Assertion failed.
key does not match re-computed value for the same sequence (case-sensitive)
   at System.Diagnostics.DebugProvider.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/DebugProvider.cs:line 22
   at System.Diagnostics.Debug.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 133
   at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 97
   at System.Diagnostics.Debug.Assert(Boolean condition, String message) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 86
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.HandlePropertyName.cs:line 37
   at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].ReadConstructorArgumentsWithContinuation(ReadStack& state, Utf8JsonReader& reader, JsonSerializerOptions options) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectWithParameterizedConstructorConverter.cs:line 376
   at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, ObjWCtorMixedParams& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectWithParameterizedConstructorConverter.cs:line 155
   at System.Text.Json.Serialization.Converters.JsonMetadataServicesConverter`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, ObjWCtorMixedParams& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/JsonMetadataServicesConverter.cs:line 58
   at System.Text.Json.Serialization.JsonConverter`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, ObjWCtorMixedParams& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs:line 221
   at System.Text.Json.Serialization.JsonConverter`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs:line 62
   at System.Text.Json.JsonSerializer.ReadCore[ObjWCtorMixedParams](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Helpers.cs:line 17
   at System.Text.Json.JsonSerializer.ReadCore[ObjWCtorMixedParams](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Stream.cs:line 651
   at System.Text.Json.JsonSerializer.ContinueDeserialize[ObjWCtorMixedParams](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Stream.cs:line 579
   at System.Text.Json.JsonSerializer.<ReadAllAsync>d__68`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext() in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Stream.cs:line 435
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<ReadAllAsync>d__68`1](<ReadAllAsync>d__68`1& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Text.Json.JsonSerializer.ReadAllAsync[ObjWCtorMixedParams](Stream utf8Json, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
   at System.Text.Json.JsonSerializer.DeserializeAsync[ObjWCtorMixedParams](Stream utf8Json, JsonTypeInfo`1 jsonTypeInfo, CancellationToken cancellationToken) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Stream.cs:line 209
   at System.Text.Json.SourceGeneration.Tests.AsyncStreamSerializerWrapper.<DeserializeWrapper>d__5`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext() in /_/src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/Serialization/JsonSerializerWrapper.SourceGen.cs:line 99
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<DeserializeWrapper>d__5`1](<DeserializeWrapper>d__5`1& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Text.Json.SourceGeneration.Tests.AsyncStreamSerializerWrapper.DeserializeWrapper[ObjWCtorMixedParams](Stream utf8Json, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Tests.StreamingJsonSerializerWrapper.<DeserializeWrapper>d__14`1[[System.Text.Json.Serialization.Tests.ObjWCtorMixedParams, System.Text.Json.SourceGeneration.Roslyn3.11.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext() in /_/src/libraries/System.Text.Json/tests/Common/StreamingJsonSerializerWrapper.cs:line 60
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<DeserializeWrapper>d__14`1](<DeserializeWrapper>d__14`1& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Text.Json.Serialization.Tests.StreamingJsonSerializerWrapper.DeserializeWrapper[ObjWCtorMixedParams](String json, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Tests.ConstructorTests.NumerousSimpleAndComplexParameters() in /_/src/libraries/System.Text.Json/tests/Common/ConstructorTests/ConstructorTests.ParameterMatching.cs:line 366
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<NumerousSimpleAndComplexParameters>d__66](<NumerousSimpleAndComplexParameters>d__66& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Text.Json.Serialization.Tests.ConstructorTests.NumerousSimpleAndComplexParameters()

[jkotas]

Failed in #68473 with:

key 824274671824495940 [propertyName=Description] does not match re-computed value 0 for the same sequence (case-sensitive). {
  Name: Description,
  NameAsUtf8.Length: 11,
  IsConfigured: True,
  IsIgnored: False,
  ShouldSerialize: True,
  ShouldDeserialize: True,
}
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.HandlePropertyName.cs:line 37