Exception during the sucessful finish of scatter vertex element

[dgrechka]

Getting

Internal error occurred (please report it at https://github.com/itislab/alpheus/issues): Failed to compute the artefacts:
        Newtonsoft.Json.JsonWriterException: Token PropertyName in state Property would result in an invalid JSON object. Path 'Origin.Fields[0].Outputs[0].Hash'.
   at Newtonsoft.Json.JsonWriter.AutoComplete(JsonToken tokenBeingWritten)
   at Newtonsoft.Json.JsonTextWriter.WritePropertyName(String name)
   at [email protected](Tuple`2 tupledArg) in C:\repos\alpheus\AlpheusCore\CustomSerializers.fs:line 24
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source) in E:\A\_work\130\s\src\fsharp\FSharp.Core\seq.fs:line 495
   at [email protected](Tuple`2 tupledArg, MdMap`2 value) in C:\repos\alpheus\AlpheusCore\CustomSerializers.fs:line 20
   at ItisLab.Alpheus.CustomSerializers.ArtefactVersionConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer) in C:\repos\alpheus\AlpheusCore\CustomSerializers.fs:line 28
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.Converters.DiscriminatedUnionConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at ItisLab.Alpheus.AlphFiles.save(AlphFile alphfile, String filepath) in C:\repos\alpheus\AlpheusCore\AlphFiles.fs:line 57
   at [email protected](Unit unitVar0) in C:\repos\alpheus\AlpheusCore\DependencyGraph.fs:line 213
   at ItisLab.Alpheus.DependencyGraph.ArtefactVertex.SaveAlphFile() in C:\repos\alpheus\AlpheusCore\DependencyGraph.fs:line 163
   at [email protected](LinkToArtefact out) in C:\repos\alpheus\AlpheusCore\DependencyGraph.fs:line 413
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source) in E:\A\_work\130\s\src\fsharp\FSharp.Core\seq.fs:line 495
   at [email protected](Unit _arg5) in C:\repos\alpheus\AlpheusCore\DependencyGraph.fs:line 413
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation`1 ctxt, FSharpFunc`2 userCode, b result1) in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 417
   at [email protected](AsyncActivation`1 ctxt) in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 589
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 109
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.FSharp.Control.AsyncResult`1.Commit() in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 349
   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronouslyInCurrentThread[a](CancellationToken cancellationToken, FSharpAsync`1 computation) in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 870
   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout) in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 890
   at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken) in E:\A\_work\130\s\src\fsharp\FSharp.Core\async.fs:line 1151
   at ItisLab.Alpheus.ComputationGraph.CommandMethod.Execute(FSharpList`1 inputs, FSharpOption`1 _arg1) in C:\repos\alpheus\AlpheusCore\ComputationGraph.fs:line 74
   at Angara.Execution.Runtime`1.buildEvaluation(m v, FSharpList`1 index, UInt64 time, State`2 state, CancellationTokenSource cts, Boolean doContinue, Unit unitVar0)

Alph file:

{
  "FileFormatVersion": 1,
  "Origin": {
    "Case": "CommandOrigin",
    "Fields": [
      {
        "Inputs": [
          {
            "RelativePath": "..\\..\\code\\dgrechka\\train_NASNetMobile_full_do_0_5.py",
            "Hash": "6D1D1C524753B759525804A80AF7B9DE6ED847CBC94B295DF246298BC6AFEBB2712B054ACE761500DE7B6EC798F33A958DEB2F0A9886B363BFF03EA3D28CB044"
          },
          {
            "RelativePath": "..\\..\\data\\bengaliai-cv19\\",
            "Hash": "D09946D9348B9135927E3C69F22AFB441F5EA040995229E4119F6B3B6DC10537DE1BF1FA6850A5BCFA1C90849BDA14AEC20B69EC410422F558CDA8CBE0A5362F"
          },
          {
            "RelativePath": "..\\..\\data\\5foldCvSplits\\*.val_ids.csv",
            "Hash": {
              "0": "B09EA6865EFC699090D8983F4A2BC36F82249E1E3163A6D5EB8611524F76BF024102DDCBD4A401BEEEDEB71FE1D5DE1752F338840722624B17BF2B4A3FA3228D",
              "1": "541369BD74236A263A5A7525C712B57836D98C183F442D1C96A366C55D7443FFE7543B7F5A6D11EDCFD1202040D441A2C64861AD0C8CB74307D58ECB6138FD95",
              "2": "377CB26C5DA8AF87EEFBB60383573841774D9329FFA41A8620A877B0B426D03992553A18E41A1C6D841DB3BCBF270A7A9F478A36F20034F3B663909822FB3A6B",
              "3": "5A708C1F023A3B8EDB9E31ADC9D95D66DA1428532643F6EC05F895A55982B0EE30FD0EC741E7DB122ED94A559C10D4A55CC2C6F149FFBAA11E34E85E0EA8B6B0",
              "4": "126EC1D9569D45674A0EB7F9CB1A861F8CBFA854CEBA03061EFE070F004C151F8BFDD72980139891A828E1D03B99BFDE36DBAF0C46F149F7106C0DA04AD20F8F"
            }
          },
          {
            "RelativePath": "..\\dgrechka_2_NASNetMobile_bottleneck\\*\\",
            "Hash": {
              "0": "DD0AAF3FBB7E769D26B0B2954D48ADADE1850FBEB2368327F4713EA3E46FE34360F6A753147BA3FB51720A9174907E4EBB291F470FE3B3B2CD78EB05DD45D71F",
              "1": "D0D614DA94CC12D9415EA1B87CABE36FEB63A838D5161F4F1E5F477B097D039B3D7AF99E980DD70DF1556E09F2CF771DC129D638EADA84C7F80C04D0A3D1F45C",
              "2": "E3BCA383F3520C4A7B578DD1A05BB0207F82FD1B2FA95B55911051A8D6FE737E93833430C7B237F7EED783A973B8F67834C75EE09C3E20DDA2C02FC1C9D57C03",
              "3": "B605D3EA64AC6BC93E213DDB8043C1A97E91614FD97D84BDB9607806695E8A608D9A79025D9BABD87F7B18C8DBBC44D411404AEC7EF7A2EF676CBDE38821B616",
              "4": "E3EDC023F14F5196F1F3C63E54DB1025CA8D76CD81191EE19409484C897D199DCE896A4121296B223B19ACF2B3158061CEBDFA43C210C2E565C9156D72891270"
            }
          },
          {
            "RelativePath": "..\\..\\code\\tfDataIngest\\tfDataSetParquet.py",
            "Hash": "2A1123333AA05FE72A5ABAFD3D792A4793F6D97F7A2798FA4E85E854CD1A08F7B8D20CE9FCD9C1D1719AF77C9145DCEC3587C7CACD1AD497DF26FF7EE333ADC0"
          },
          {
            "RelativePath": "..\\..\\code\\tfDataIngest\\tfDataSetParquetAnnotateTrain.py",
            "Hash": "1E73A8FCF0B94A4A9F02883CA67C6343DEE46E3616AF3A3397B2CEF77AC54F3F565E3902590391A711956F9E074DEEF45FB80D286DFDAD21A59CFF738B327800"
          },
          {
            "RelativePath": "..\\..\\code\\models\\NASNetMobile.py",
            "Hash": "817396CA3E415CE9ACFFB2532DFBB1DF0C4A4FB68BA15A34E0DBA2FFCD821958795E5A97005C6FF86F72ED8A9F725331D8F829247B97B68233DEB2C6C93AF707"
          },
          {
            "RelativePath": "..\\..\\code\\tfMetrics\\macroAveragedRecallForLogits.py",
            "Hash": "390F7DAE9CD506008D2613E3683B8E9D065CE1A1A26FFCB5FA186457BEB13C08B54FAEE849618C91617119B816172E85A9D82B432243A3C17B6AE32C543605E0"
          }
        ],
        "Outputs": [
          {
            "RelativePath": "*\\",
            "Hash": {
              "0": "61F04E3150B112955061F4E9DF851DC1054806E76167961FF968FF7BA4F6E2AC2F39BC64CFADC7937C874004BC16E54B3AD162EE2DD489DD8383858DA769A0E7",
              "1": "9DD9AE7574B73D535B9E7E5A1581F8E99578E5697053A2B464D63A73073361F9E1C2FB4549085072F441CB8A0B0423AA46A1B71F024E4CAAF8E0DB83467122B7",
              "2": "64A50D588D40A706C0F40A9A836CCFBF428197288F81478D7B9063E4005C143E518E400D68574AF02D183D6635CC795DD4DACA1F3D463901DC9CD8C2208A79BE",
              "3": "83716EECE2C21C7A3F5D24B7EA2A97E1F8BF504EB9B7D6C28903747B259B7D8A151A61451631FF4553745D1AEBAB676DD31F54AE909143882F0B0938E94FEDC0",
              "4": "A8E6918DC84047CDF622C6B0E530D8D09B7D0F6151EA170FD7454D3A9A481605E61B993301C75E022A40D84020A4C5CAB04CB3523CA2C5B840ED93046F987144"
            }
          }
        ],
        "ResourceGroups": [
          "gpu"
        ],
        "SuccessfulExitCodes": [
          0
        ],
        "OutputIndex": 0,
        "WorkingDirectory": "..\\..",
        "Command": "python $in1 $in2 $in3 $in4 $out1",
        "Signature": "BF05913C5C3619384F19AB86DC010D8A7C4D85D7",
        "OutputsCleanDisabled": false
      }
    ]
  },
  "IsTracked": true
}

Note: input version has changed, so the outputs must be recalculated.

[dgrechka]

Possibly resolved by PR #93

[dgrechka]

Still present.
Possible way to reproduce.

1. Scatter command method produces output
2. Some initial file index is removed (causing index space to shrink)
3. Rerun scatter command method fails to update alph file