Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lambda crashing because of out of memory #1496

Open
sjsingh85 opened this issue Aug 23, 2024 · 1 comment
Open

Lambda crashing because of out of memory #1496

sjsingh85 opened this issue Aug 23, 2024 · 1 comment
Labels
bug Something isn't working .NET Pull requests that update .net code

Comments

@sjsingh85
Copy link

Describe the bug
We are using this lambda layer for a .Net 6 based lambda. When we have multiple executions of the lambda in a small duration, the lambda is getting killed sometimes even before executing any code - only logs from this otel layer gets logged which show the below mentioned errors

Steps to reproduce
Out lambda is fairly simple and doesn't do much. We do create metrics to be sent and logs. The problem comes when we have more amount of concurrent executions - say 40 concurrent executions of lambda

What did you expect to see?
No errors/crashing of lambda

What did you see instead?
Lambda crashing

What version of collector/language SDK version did you use?
Version: layer:opentelemetry-collector-amd64-0_8_0. I just saw there is a new release 0_10_0 and will test that out for now.

What language layer did you use?
Config: .Net 6

Additional context
Collector.yaml:
receivers:
otlp:
protocols:
grpc:
http:

exporters:
otlphttp:
endpoint: "https://otlp.nr-data.net"
headers:
api-key: ${NEW_RELIC_LICENSE_KEY}

processors:
memory_limiter:
check_interval: 1s
limit_percentage: 50 # hard memory limit, applies backpressure on receiver to retry after memory is freed
spike_limit_percentage: 30 # measurement should not increase by more than 30% in 1s
batch:
timeout: 10s
decouple:
max_queue_size: 20

service:
telemetry:
logs:
level: "debug"
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp]
processors: [memory_limiter, batch, decouple]
metrics:
receivers: [otlp]
exporters: [otlphttp]
processors: [memory_limiter, batch, decouple]
logs:
receivers: [otlp]
exporters: [otlphttp]
processors: [memory_limiter, batch, decouple]

Error Log:
timestamp,message
1724201556611,"2024-08-21T00:52:36.605Z e3dc1a02-15e2-58a5-b8c6-453d2962f745 fail System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, Char* pThreadName)
at System.Threading.Thread.StartCore()
at OpenTelemetry.BatchExportProcessor1..ctor(BaseExporter1 exporter, Int32 maxQueueSize, Int32 scheduledDelayMilliseconds, Int32 exporterTimeoutMilliseconds, Int32 maxExportBatchSize)
at OpenTelemetry.Logs.OtlpLogExporterHelperExtensions.<>c__DisplayClass2_0.b__0(IServiceProvider sp)
at OpenTelemetry.Logs.LoggerProviderBuilderExtensions.<>c__DisplayClass4_0.b__0(IServiceProvider sp, LoggerProviderBuilder builder)
at OpenTelemetry.Logs.LoggerProviderBuilderSdk.OpenTelemetry.Logs.IDeferredLoggerProviderBuilder.Configure(Action2 configure) at OpenTelemetry.Logs.OpenTelemetryDependencyInjectionLoggerProviderBuilderExtensions.ConfigureBuilder(LoggerProviderBuilder loggerProviderBuilder, Action2 configure)
at OpenTelemetry.Logs.LoggerProviderBuilderExtensions.AddProcessor(LoggerProviderBuilder loggerProviderBuilder, Func2 implementationFactory) at Microsoft.Extensions.Logging.OpenTelemetryLoggingExtensions.<>c.<AddOpenTelemetryInternal>b__5_0(IServiceProvider sp, LoggerProviderBuilder logging) at OpenTelemetry.Logs.LoggerProviderSdk..ctor(IServiceProvider serviceProvider, Boolean ownsServiceProvider) at OpenTelemetry.Logs.LoggerProviderBuilderBase.<>c.<.ctor>b__3_0(IServiceProvider sp) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) at Microsoft.Extensions.Logging.OpenTelemetryLoggingExtensions.<>c.<AddOpenTelemetryInternal>b__5_1(IServiceProvider sp) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) at Microsoft.Extensions.Logging.LoggerFactory.Create(Action1 configure)
at OpenTelemetry.Instrumentation.AWSLambda.AWSLambdaWrapper.TraceInternalAsync[TInput,TResult](TracerProvider tracerProvider, Func3 handlerAsync, TInput input, ILambdaContext context, ActivityContext parentContext) at lambda_method1(Closure , Stream , ILambdaContext , Stream ) at Amazon.Lambda.RuntimeSupport.HandlerWrapper.<>c__DisplayClass8_0.<GetHandlerWrapper>b__0(InvocationRequest invocation) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/HandlerWrapper.cs:line 54 at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.InvokeOnceAsync(CancellationToken cancellationToken) in /src/Repo/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/LambdaBootstrap.cs:line 217 " 1724201556654,"END RequestId: e3dc1a02-15e2-58a5-b8c6-453d2962f745 " 1724201556654,"REPORT RequestId: e3dc1a02-15e2-58a5-b8c6-453d2962f745 Duration: 157.60 ms Billed Duration: 158 ms Memory Size: 1024 MB Max Memory Used: 390 MB XRAY TraceId: 1-66c53a54-633781a86de01c896820574e SegmentId: 7bff311c2eaab23f Sampled: true " 1724201916512,"START RequestId: a7d9b706-544a-5937-857b-e83969a2ecd9 Version: $LATEST " 1724201916516,"thread 'sender' panicked at 'called Result::unwrap()on anErrvalue: Os { code: 11, kind: WouldBlock, message: ""Resource temporarily unavailable"" }', /local/p4clients/pkgbuild-YqrIz/workspace/build/AWSLogsLambdaInsights/AWSLogsLambdaInsights-1.0.229.0/AL2012/DEV.STD.PTHREAD/build/private/cargo-home/registry/src/-47f107018cd5df98/tokio-0.2.25/src/runtime/blocking/pool.rs:234:14 " 1724201916516,"note: run withRUST_BACKTRACE=1environment variable to display a backtrace " 1724201916516,"{""level"":""info"",""ts"":1724201916.514789,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002311e00] Closing: connection error: desc = \""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:36634: i/o timeout\"""",""grpc_log"":true} " 1724201916541,"{""level"":""info"",""ts"":1724201916.5410585,""caller"":""[email protected]/processor.go:82"",""msg"":""started forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""traces""} " 1724201916543,"{""level"":""info"",""ts"":1724201916.5434678,""caller"":""[email protected]/processor.go:82"",""msg"":""started forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""metrics""} " 1724201916544,"{""level"":""info"",""ts"":1724201916.5443344,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:36634: i/o timeout\"""",""grpc_log"":true} " 1724201916546,"{""level"":""info"",""ts"":1724201916.546361,""caller"":""[email protected]/processor.go:82"",""msg"":""started forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""logs""} " 1724201916561,"thread 'sender' panicked at 'calledResult::unwrap()on anErr` value: PoisonError { .. }', /local/p4clients/pkgbuild-YqrIz/workspace/build/AWSLogsLambdaInsights/AWSLogsLambdaInsights-1.0.229.0/AL2012/DEV.STD.PTHREAD/build/private/cargo-home/registry/src/-47f107018cd5df98/tokio-0.2.25/src/runtime/blocking/pool.rs:122:59
"
1724201916561,"stack backtrace:
"
1724201916562,"0: 0x55afde107c9d -
"
1724201916562,"1: 0x55afde12e20c -
"
1724201916562,"2: 0x55afde1019d1 -
"
1724201916562,"3: 0x55afde109445 -
"
1724201916562,"4: 0x55afde109166 -
"
1724201916562,"5: 0x55afde1099e6 -
"
1724201916562,"6: 0x55afde1098d7 -
"
1724201916562,"7: 0x55afde108154 -
"
1724201916562,"8: 0x55afde109609 -
"
1724201916562,"9: 0x55afddaffca3 -
"
1724201916562,"10: 0x55afddaffd93 -
"
1724201916562,"11: 0x55afddfb9a3a -
"
1724201916562,"12: 0x55afddfb9b14 -
"
1724201916562,"13: 0x55afddc0216e -
"
1724201916563,"14: 0x55afddc02c15 -
"
1724201916563,"15: 0x55afddc08d49 -
"
1724201916563,"16: 0x55afddb9cb31 -
"
1724201916563,"17: 0x55afddba3431 -
"
1724201916563,"18: 0x55afde10e503 -
"
1724201916565,"19: 0x7fc55461d44b - start_thread
"
1724201916581,"20: 0x7fc554d4152f - __clone
"
1724201916581,"21: 0x0 -
"
1724201916581,"thread panicked while panicking. aborting.
"
1724201916584,"{""level"":""info"",""ts"":1724201916.5839233,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033ec000] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5849848,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00586e300] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5850742,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027ca80] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.585144,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003451500] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.585212,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5800] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5852835,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e1980] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5853589,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450900] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5854304,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4f500] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.585496,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c9380] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5855658,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025c1500] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5856276,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4fe00] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5856953,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9cf00] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5858326,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a3ed80] Closing: EOF"",""grpc_log"":true}
"
1724201916585,"{""level"":""info"",""ts"":1724201916.5859032,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002311380] Closing: EOF"",""grpc_log"":true}
"
1724201916586,"{""level"":""info"",""ts"":1724201916.5859637,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002310900] Closing: EOF"",""grpc_log"":true}
"
1724201916586,"{""level"":""info"",""ts"":1724201916.586,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e4180] Closing: EOF"",""grpc_log"":true}
"
1724201916586,"{""level"":""info"",""ts"":1724201916.5860891,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d800] Closing: EOF"",""grpc_log"":true}
"
1724201916586,"{""level"":""info"",""ts"":1724201916.586157,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d980] Closing: EOF"",""grpc_log"":true}
"
1724201916586,"{""level"":""info"",""ts"":1724201916.5861888,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002563980] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6012404,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450000] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6013799,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e0900] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6014462,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033ec780] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6015067,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e1c80] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6015685,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273f500] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.601624,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003451980] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.601688,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002310d80] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6017432,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273e900] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6018028,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9de00] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.601858,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002562d80] Closing: EOF"",""grpc_log"":true}
"
1724201916601,"{""level"":""info"",""ts"":1724201916.6019177,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9c600] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6020176,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e180] Closing: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49772: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6020682,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1]grpc: Server.Serve failed to create ServerTransport: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49772: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.5863836,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004b85080] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6023812,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012a180] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.602539,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003451080] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.602619,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002562780] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6026938,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002562180] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.602759,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002310300] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6028152,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e0300] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6028826,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450300] Closing: EOF"",""grpc_log"":true}
"
1724201916602,"{""level"":""info"",""ts"":1724201916.6029375,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a3e780] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5840573,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9c000] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.6031222,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033ec000] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5841267,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002a49980] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.603224,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00586e300] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5841496,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025c0480] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.6033196,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027ca80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5841649,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002563e00] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.6034083,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003451500] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.584179,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00586f080] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.6034951,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5800] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5841951,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e000] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.603577,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e1980] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5842078,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c9200] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.6036584,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450900] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5842211,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012be00] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.603735,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4f500] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.584235,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e4900] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.6038146,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c9380] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.5842505,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004b84600] Closing: EOF"",""grpc_log"":true}
"
1724201916603,"{""level"":""info"",""ts"":1724201916.603895,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025c1500] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5842636,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c9c80] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6040409,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4fe00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5842783,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d380] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6041217,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9cf00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.584291,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033eca80] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6042047,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a3ed80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843072,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012ac00] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.604287,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002311380] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843198,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012a900] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.604367,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002310900] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843322,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012a780] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6044397,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e4180] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843458,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012ad80] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6045191,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d800] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843604,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012b200] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6046138,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d980] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843732,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012b380] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.604697,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002563980] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843866,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012aa80] Closing: EOF"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.6047804,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450000] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916604,"{""level"":""info"",""ts"":1724201916.5843997,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450780] Closing: EOF"",""grpc_log"":true}
"
1724201916605,"{""level"":""info"",""ts"":1724201916.6050308,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e0900] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916605,"{""level"":""info"",""ts"":1724201916.5844135,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c8480] Closing: EOF"",""grpc_log"":true}
"
1724201916605,"{""level"":""info"",""ts"":1724201916.6051855,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033ec780] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916605,"{""level"":""info"",""ts"":1724201916.5862398,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003b7c300] Closing: EOF"",""grpc_log"":true}
"
1724201916605,"{""level"":""info"",""ts"":1724201916.6053147,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e1c80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.586268,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003b7ca80] Closing: EOF"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.606511,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273f500] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.5862854,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ea80] Closing: EOF"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.6066124,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003451980] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.586302,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273f200] Closing: EOF"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.6067035,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002310d80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.586317,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012b800] Closing: EOF"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.6067939,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273e900] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.5863297,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273ea80] Closing: EOF"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.606879,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9de00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.5863428,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e4c00] Closing: EOF"",""grpc_log"":true}
"
1724201916606,"{""level"":""info"",""ts"":1724201916.6069632,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002562d80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916607,"{""level"":""info"",""ts"":1724201916.5863574,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e780] Closing: EOF"",""grpc_log"":true}
"
1724201916607,"{""level"":""info"",""ts"":1724201916.6070416,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9c600] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916607,"{""level"":""info"",""ts"":1724201916.5863712,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00018fe00] Closing: EOF"",""grpc_log"":true}
"
1724201916607,"{""level"":""info"",""ts"":1724201916.5874233,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9d800] Closing: EOF"",""grpc_log"":true}
"
1724201916607,"{""level"":""info"",""ts"":1724201916.6071618,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004b85080] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.6220894,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e4900] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.5874434,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5380] Closing: EOF"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.6225612,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012a180] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.5874596,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5680] Closing: EOF"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.622695,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003451080] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.5874724,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5500] Closing: EOF"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.6228166,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002562780] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.5874848,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a38300] Closing: EOF"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.6229253,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002562180] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916622,"{""level"":""info"",""ts"":1724201916.5883315,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a38480] Closing: EOF"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6230166,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002310300] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.588347,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027fc80] Closing: EOF"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6231296,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033e0300] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6232028,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450300] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6232305,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a3e780] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6233418,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9c000] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6234481,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002a49980] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.623513,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025c0480] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6235325,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc002563e00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6236343,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00586f080] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6236994,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e000] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6237187,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c9200] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6238198,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012be00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.6238844,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5380] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916623,"{""level"":""info"",""ts"":1724201916.623903,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5680] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.6239624,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e5500] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.6239872,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a38300] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.6240559,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004a38480] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.6241448,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027fc80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.624894,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003450780] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.6249328,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004b84600] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916624,"{""level"":""info"",""ts"":1724201916.6249473,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c9c80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6250498,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d380] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.625081,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0033eca80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6251924,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012ac00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6252596,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012a900] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6252794,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012a780] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6253543,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012ad80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.62545,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012b200] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.625469,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012b380] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.625549,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012aa80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6256442,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00012b800] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6256633,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0051c8480] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6257675,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003b7c300] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6258328,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003b7ca80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916625,"{""level"":""info"",""ts"":1724201916.6258512,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ea80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916626,"{""level"":""info"",""ts"":1724201916.6259117,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273f200] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916626,"{""level"":""info"",""ts"":1724201916.6259334,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e780] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916626,"{""level"":""info"",""ts"":1724201916.6259456,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00273ea80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916626,"{""level"":""info"",""ts"":1724201916.6259584,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc0025e4c00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916626,"{""level"":""info"",""ts"":1724201916.6259818,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00018fe00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916626,"{""level"":""info"",""ts"":1724201916.6261203,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc000f9d800] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916640,"APPLICATION LOG"
1724201916641,"APPLICATION LOG"
1724201916646,"{""level"":""info"",""ts"":1724201916.6464407,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004b84a80] Closing: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49818: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916646,"{""level"":""info"",""ts"":1724201916.646597,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1]grpc: Server.Serve failed to create ServerTransport: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49818: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916647,"{""level"":""info"",""ts"":1724201916.6471176,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc004b84f00] Closing: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49810: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916647,"{""level"":""info"",""ts"":1724201916.6472147,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1]grpc: Server.Serve failed to create ServerTransport: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49810: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916647,"{""level"":""info"",""ts"":1724201916.647308,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e900] Closing: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49788: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916647,"{""level"":""info"",""ts"":1724201916.6474068,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1]grpc: Server.Serve failed to create ServerTransport: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49788: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916660,"{""level"":""info"",""ts"":1724201916.6478443,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ec00] Closing: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49794: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916661,"{""level"":""info"",""ts"":1724201916.6610782,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1]grpc: Server.Serve failed to create ServerTransport: connection error: desc = ""transport: http2Server.HandleStreams failed to receive the preface from client: read tcp 127.0.0.1:4317->127.0.0.1:49794: read: connection reset by peer"""",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7274742,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ef00] Closing: EOF"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7275012,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d680] Closing: EOF"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7275503,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ef00] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7277231,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4f200] Closing: EOF"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.727796,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4f080] Closing: EOF"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7278566,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4f200] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7278147,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc00027d680] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916727,"{""level"":""info"",""ts"":1724201916.7277992,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ed80] Closing: EOF"",""grpc_log"":true}
"
1724201916728,"{""level"":""info"",""ts"":1724201916.7278678,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4f080] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916728,"{""level"":""info"",""ts"":1724201916.7280085,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4ed80] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201916728,"{""level"":""info"",""ts"":1724201916.7281575,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e600] Closing: EOF"",""grpc_log"":true}
"
1724201916728,"{""level"":""info"",""ts"":1724201916.7282357,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[transport] [server-transport 0xc003a4e600] loopyWriter exiting with error: transport closed by client"",""grpc_log"":true}
"
1724201917042,"{""level"":""info"",""ts"":1724201917.0428164,""caller"":""[email protected]/processor.go:92"",""msg"":""stopped forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""metrics""}
"
1724201917080,"{""level"":""info"",""ts"":1724201917.0803485,""caller"":""[email protected]/processor.go:92"",""msg"":""stopped forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""logs""}
"
1724201917080,"{""level"":""info"",""ts"":1724201917.0803916,""caller"":""[email protected]/processor.go:92"",""msg"":""stopped forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""traces""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0806994,""logger"":""lifecycle.manager"",""msg"":""Received SHUTDOWN event""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0810657,""caller"":""[email protected]/processor.go:82"",""msg"":""started forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""logs""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0810897,""caller"":""[email protected]/processor.go:82"",""msg"":""started forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""metrics""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0811079,""caller"":""[email protected]/processor.go:82"",""msg"":""started forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""traces""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0811546,""logger"":""telemetryAPI.Listener"",""msg"":""HTTP Server closed:"",""error"":""http: Server closed""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0812926,""caller"":""[email protected]/collector.go:331"",""msg"":""Received shutdown request""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0813682,""caller"":""[email protected]/service.go:243"",""msg"":""Starting shutdown...""}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0815387,""caller"":""zapgrpc/zapgrpc.go:176"",""msg"":""[core] [Server #1 ListenSocket #2]ListenSocket deleted"",""grpc_log"":true}
"
1724201917081,"{""level"":""info"",""ts"":1724201917.0815985,""caller"":""[email protected]/processor.go:92"",""msg"":""stopped forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""metrics""}
"
1724201917299,"{""level"":""info"",""ts"":1724201917.2996376,""caller"":""[email protected]/processor.go:92"",""msg"":""stopped forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""logs""}
"
1724201917509,"{""level"":""info"",""ts"":1724201917.508876,""caller"":""[email protected]/processor.go:92"",""msg"":""stopped forwarding data"",""kind"":""processor"",""name"":""decouple"",""pipeline"":""traces""}
"
1724201917509,"{""level"":""info"",""ts"":1724201917.508961,""caller"":""extensions/extensions.go:59"",""msg"":""Stopping extensions...""}
"
1724201917509,"{""level"":""info"",""ts"":1724201917.508972,""caller"":""[email protected]/service.go:257"",""msg"":""Shutdown complete.""}
"
1724201917509,"{""level"":""info"",""ts"":1724201917.5090528,""msg"":""done""}
"
1724201917514,"RequestId: a7d9b706-544a-5937-857b-e83969a2ecd9 Error: signal: aborted
Extension.Crash
"
1724201917514,"END RequestId: a7d9b706-544a-5937-857b-e83969a2ecd9
"
1724201917514,"REPORT RequestId: a7d9b706-544a-5937-857b-e83969a2ecd9 Duration: 1001.88 ms Billed Duration: 1002 ms Memory Size: 1024 MB Max Memory Used: 391 MB
XRAY TraceId: 1-66c53bbc-96d7bfacacbdd180b5c0fdfd SegmentId: 2f6206591a76f93e Sampled: true
"
@sjsingh85 sjsingh85 added the bug Something isn't working label Aug 23, 2024
@serkan-ozal
Copy link
Contributor

serkan-ozal commented Aug 28, 2024 •
edited
Loading

Hi @sjsingh85,

I think this might not be caused by a bug in the OTEL components (collector and .NET SDK here), but a conflict/bug between AWS Lambda environment and .NET runtime. So, high number of produced spans by OTEL might just trigger this issue.

The reason is that, actually even though there is still some amount of available memory (limit=1024MB, max used=391MB, the Lambda crashes with OOM.

REPORT RequestId: a7d9b706-544a-5937-857b-e83969a2ecd9 Duration: 1001.88 ms Billed Duration: 1002 ms Memory Size: 1024 MB Max Memory Used: 391 MB

Then, when checking whether is there any other folks faced with similar issue without OTEL and came across this one: aws/aws-lambda-dotnet#1699

So, if that is possible, you may try with .NET 8 and/or follow the suggestions here: aws/aws-lambda-dotnet#1699

Please let us know whether those helped or not.

@nslaughter nslaughter added the .NET Pull requests that update .net code label Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working .NET Pull requests that update .net code
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sjsingh85 @serkan-ozal @nslaughter