2024-12-21T0537Z

Guides/DiscriminatingRCCLogs/README.md

@@ -0,0 +1,85 @@
+So we need to extract every valid FLog setting.
+
+But we don't have the FFlags for 2018M available. But 16src has [this information](https://github.com/Jxys3rrV/roblox-2016-source-code/blob/4de2dc3a380e1babe4343c49a4341ceac749eddb/App/v8datamodel/FastLogSettings.cpp#L9) and more.
+
+## Where to get the flags from?
+
+So we're going to combine:
+
+### FLogs from 16src
+
+I've attached some of the flags to [a text file](./flogs-16src.txt). Flags are defined in code by a C++ macro called `LOGVARIABLE` or `DYNAMIC_LOGVARIABLE` and have a default value provided with them. So I've preserved the call syntax. The first 81 are copied from `FastLogSettings.cpp`, whereas [the rest are taken from other places in 16src](https://github.com/search?q=repo%3AJxys3rrV%2Froblox-2016-source-code+%28%2F%28%3F-i%29DYNAMIC_LOGVARIABLE%5C%28%2F+OR+%2F%28%3F-i%29LOGVARIABLE%5C%28%2F%29&type=code).
+
+### The 2021E FFlags
+
+Not very useful for RCC, as some important log flags such as `RCCServiceInit` are not included.
+
+Flags were collected from [MaximumADHD/Roblox-FFlag-Tracker](https://github.com/MaximumADHD/Roblox-FFlag-Tracker/blob/50e7d23fb6723871b6f960b2ff530099e2485be0/PCDesktopClient.json).
+
+Consult [this text file](./flogs-2021E.txt).
+
+### The 2019 Sitetest4 deployment
+
+Taken from [`Roblox.Settings.json`](https://github.com/FlarfGithub/RobloxLabs-Roblox.SiteTest4/blob/6286186bcf2adc898b7c9c04bf9d2fdc4ae69f81/Default/Roblox.Settings.json#L131). Processed to remove duplicates. But I also got rid of the various defaults.
+
+Consult [this text file](./flogs-sitetest4.txt).
+
+### Miscellaneous flags from my personal experience
+
+I've also added FLog names such as `Output` and `RCCServiceJobs` since they show up a lot in my logs for 2021E servers.
+
+Just to make sure, I kept track of which FFlags are used the most. This is done by counting how many times each FFlag appears in [a typical log file](./dev_20241211T185113Z_RCC_11372_last.log).
+
+```py
+import re, collections
+with open('./dev_20241211T185113Z_RCC_11372_last.log', 'r') as log_file: # Replace with your log file path.
+    log_entries = log_file.readlines()
+
+log_entry_counts = collections.defaultdict(int)
+for entry in log_entries:
+    function_names = re.findall(r'(?<=FLog::)([^\]]+)', line)
+    for name in function_names:
+        log_entry_counts[name] += 1
+```
+
+The results are shown below:
+
+| FLog Name      | Count |
+| -------------- | ----- |
+| RCCServiceInit | 1     |
+| LocalStorage   | 2     |
+| RCCServiceJobs | 5     |
+| RCCExecuteInfo | 8     |
+| Output         | 16    |
+| NetworkAudit   | 95    |
+| Error          | 8832  |
+
+Not much variety in what FLogs are shown by default.
+
+Notice how `[FLog::Output]` is always preceded by `6` and `[DFLog::NetworkAudit]` is preceded by `19`?
+
+These are their default 'log levels'. FLogs with a log level below `6` do not display in the RCC log. The values `6` and `19` correspond to the default values for the `FLogNetworkAudit` and `FLogOutput` flags, respectively.
+
+You can define these values in the following files:
+
+- For 2018M: `./ClientSettings/RCCService.json`
+- For 2021E: `./DevSettingsFile.json`
+
+## What I did next
+
+So I combined the files into [a single list](./flogs-all.txt) (inputs are ordered below) and removed duplicates from that.
+
+- FLogs taken as-is from the table above.
+- In [the 16src list](./flogs-16src.txt), I did regex replacement `.+\(([^,]+)(, .+)?\)` with `$1`.
+- No modifications done from the [sitetest](./flogs-sitetest4.txt) list.
+- In [the 2021E list](./flogs-2021E.txt), I did regex replacement `"D?FLog([^"]+)": .+` with `$1`.
+
+Then...
+
+```shell
+awk '{print "\"FLog" $0 "\": " NR+100-1 ","}' "flogs-all.txt" > flogs-json-snippet.txt
+```
+
+We start with a value of 100 to ensure that each custom log levels will be unique and won't clash with any standard or predefined log levels.
+
+I notice that for 2018M, the `RCCServiceInit` always sticks to 6. So we have to treat it specially. So 6 is reserved for `RCCServiceInit`. We also need to ensure that `RCCServiceInit` is not included in the list of custom log levels.

Guides/DiscriminatingRCCLogs/flogs-16src.txt

@@ -0,0 +1,169 @@
+LOGVARIABLE(VideoCapture, 0)
+LOGVARIABLE(Verbs, 0)
+LOGVARIABLE(Plugins, 0)
+LOGVARIABLE(COMCalls, 1)
+LOGVARIABLE(FullWindowMessages, 0)
+LOGVARIABLE(RobloxWndInit, 1)
+LOGVARIABLE(CrashReporterInit, 1)
+LOGVARIABLE(Explorer, 0)
+LOGVARIABLE(HardwareMouse, 2)
+LOGVARIABLE(UserInputProfile, 0)
+LOGVARIABLE(DragProfile, 0)
+LOGVARIABLE(GuiTargetLifetime, 0)
+LOGVARIABLE(MouseCommandLifetime, 0)
+LOGVARIABLE(GoldenHashes, 1)
+LOGVARIABLE(DeviceLost, 0)
+LOGVARIABLE(FRM, 0)
+LOGVARIABLE(AdornableLifetime, 0)
+LOGVARIABLE(AdornRenderStats, 0)
+LOGVARIABLE(ViewRbxBase, 2)
+LOGVARIABLE(ViewRbxInit, 1)
+LOGVARIABLE(ThumbnailRender, 0)
+LOGVARIABLE(AdornVB, 0)
+LOGVARIABLE(SSAO, 0)
+LOGVARIABLE(PushPixels, 0)
+LOGVARIABLE(RenderStatsOnLogs, 2)
+LOGVARIABLE(VisualEngineInit, 0)
+LOGVARIABLE(PartInstanceLifetime, 0)
+LOGVARIABLE(JointInstanceLifetime, 0)
+LOGVARIABLE(PrimitiveLifetime, 0)
+LOGVARIABLE(JointLifetime, 0)
+LOGVARIABLE(RenderBreakdown, 2)
+LOGVARIABLE(RenderBreakdownDetails, 2)
+LOGVARIABLE(DataModelJobs, 2)
+LOGVARIABLE(FullRenderObjects, 0)
+LOGVARIABLE(RbxMegaClustersUpdate, 2)
+LOGVARIABLE(RbxMegaClusterInit, 2)
+LOGVARIABLE(GfxClusters, 2)
+LOGVARIABLE(GfxClustersFull, 0)
+LOGVARIABLE(MegaClusters, 1)
+LOGVARIABLE(MegaClusterInit, 1)
+LOGVARIABLE(MegaClusterDirty, 0)
+LOGVARIABLE(MegaClusterDecodeStream, 0)
+LOGVARIABLE(MegaClusterNetwork, 2)
+LOGVARIABLE(MegaClusterNetworkInit, 1)
+LOGVARIABLE(DisableInvalidateOnUnbind, 0)
+LOGVARIABLE(CloseDataModel, 1)
+LOGVARIABLE(TouchedSignal, 0)
+LOGVARIABLE(TextureContentProvider, 0)
+LOGVARIABLE(LegacyLock, 0)
+LOGVARIABLE(Physics, 0)
+LOGVARIABLE(ContentProviderRequests, 0)
+LOGVARIABLE(MouseCommand, 0)
+LOGVARIABLE(ScriptContextAdd, 0)
+LOGVARIABLE(ScriptContextRemove, 0)
+LOGVARIABLE(ScriptContext, 0)
+LOGVARIABLE(ScriptContextClose, 1)
+LOGVARIABLE(WeakThreadRef, 0)
+LOGVARIABLE(CoreScripts, 1)
+LOGVARIABLE(UseLuaMemoryPool, 0)
+LOGVARIABLE(LuaMemoryPool, 0)
+LOGVARIABLE(LuaProfiler, 0)
+LOGVARIABLE(LuaScriptTimeoutSeconds, 0)
+LOGVARIABLE(Network, 0)
+LOGVARIABLE(HttpQueue, 0)
+LOGVARIABLE(Player, 1)
+LOGVARIABLE(NetworkCache, 1)
+LOGVARIABLE(ReplicationDataLifetime, 0)
+LOGVARIABLE(NetworkInstances, 0)
+LOGVARIABLE(NetworkReadItem, 0)
+LOGVARIABLE(JoinSendExtraItemCount, 1)
+LOGVARIABLE(MaxNetworkReadTimeInCS, 0)
+LOGVARIABLE(DistanceMetricP1, 0)
+LOGVARIABLE(ttMetricP1, 0)
+LOGVARIABLE(US14116, 0)
+LOGVARIABLE(MachineIdUploader, 1)
+LOGVARIABLE(RakNetDisconnect, 0)
+LOGVARIABLE(Http, 0)
+LOGVARIABLE(Legacy, 0)
+LOGVARIABLE(ScriptPrint, 3)
+LOGVARIABLE(ClientSettings, 1)
+LOGVARIABLE(PlayerShutdownLuaTimeoutSeconds, 0)
+LOGVARIABLE(Graphics, 0)
+LOGVARIABLE(VR, 0)
+LOGVARIABLE(NamedMutex, 0)
+LOGVARIABLE(ScopedConnection, 0)
+LOGVARIABLE(Crash, 1)
+LOGVARIABLE(HangDetection, 0)
+LOGVARIABLE(ContentProviderCleanup, 0)
+LOGVARIABLE(ISteppedLifetime, 0)
+LOGVARIABLE(MutexLifetime, 0)
+LOGVARIABLE(TaskScheduler, 0)
+LOGVARIABLE(TaskSchedulerInit, 0)
+LOGVARIABLE(TaskSchedulerRun, 0)
+LOGVARIABLE(TaskSchedulerFindJob, 0)
+LOGVARIABLE(TaskSchedulerSteps, 0)
+LOGVARIABLE(Asserts, 0)
+LOGVARIABLE(FWLifetime, 0)
+LOGVARIABLE(FWUpdate, 0)
+LOGVARIABLE(KernelStats, 0)
+LOGVARIABLE(TestLogGroup, 0)
+LOGVARIABLE(DXVideoMemory, 4)
+DYNAMIC_LOGVARIABLE(WebChatFiltering, 0)
+LOGVARIABLE(FMOD, 0)
+LOGVARIABLE(Sound, 0)
+DYNAMIC_LOGVARIABLE(SoundTiming, 0)
+DYNAMIC_LOGVARIABLE(SoundTrace, 0)
+LOGVARIABLE(StudioAnalytics, 1)
+LOGVARIABLE(StepAnimatedJoints, 0)
+LOGVARIABLE(CyclicExecutiveWorldSteps, 0)
+LOGVARIABLE(WorldStepsBehind, 0)
+LOGVARIABLE(WorldStepsBehindG, 0)
+LOGVARIABLE(AnalyticsLog, 0)
+LOGVARIABLE(HeartBeatFailure, 0)
+LOGVARIABLE(PhysicsStepsPerSecond, 0)
+LOGVARIABLE(InstanceTreeManipulation, 0)
+LOGVARIABLE(TaskSchedulerTiming, 0)
+DYNAMIC_LOGVARIABLE(HttpTrace, 0)
+LOGVARIABLE(DataStore, 0)
+DYNAMIC_LOGVARIABLE(PartStreamingRequests, 0)
+LOGVARIABLE(ReflectionMetadata, 1)
+LOGVARIABLE(Serializer, 0)
+LOGVARIABLE(TerrainCellListener, 0)
+DYNAMIC_LOGVARIABLE(PlaceLauncher, 0)
+LOGVARIABLE(RenderLightGrid, 0)
+LOGVARIABLE(RenderLightGridAgeProportion, 5)
+LOGVARIABLE(RenderLightGridBorderGlobalCutoff, 32)
+LOGVARIABLE(RenderLightGridBorderSkylightCutoff, 32)
+LOGVARIABLE(RenderFastCluster, 0)
+LOGVARIABLE(Voxelizer, 0)
+LOGVARIABLE(RCCServiceInit, 1)
+LOGVARIABLE(RCCServiceJobs, 1)
+LOGVARIABLE(RCCDataModelInit, 1)
+DYNAMIC_LOGVARIABLE(PlayerChatInfoExponentialBackoffLimitMultiplier, 9)
+LOGVARIABLE(SlowHttpRequest, 0)
+LOGVARIABLE(LuaBridge, 0)
+LOGVARIABLE(PathfindingDetail, 0)
+LOGVARIABLE(PathfindingPerf, 0)
+LOGVARIABLE(ThreadRefCounts, 1)
+LOGVARIABLE(HumanoidFloorProcess, 0)
+DYNAMIC_LOGVARIABLE(PreloadLinkedScriptsTiming, 0)
+LOGVARIABLE(RenderRequest, 0)
+DYNAMIC_LOGVARIABLE(DeferredVoxelUpdates, 0)
+DYNAMIC_LOGGROUP(PartStreamingRequests)
+LOGVARIABLE(UnitTestOn, 1)
+LOGVARIABLE(UnitTestOff, 0)
+LOGVARIABLE(CyclicExecutiveTiming, 0)
+LOGVARIABLE(CyclicExecutiveThrottling, 0)
+LOGVARIABLE(Zero, 1)
+LOGVARIABLE(DataStoreBudget, 0)
+LOGGROUP(MegaClusterNetwork)
+LOGGROUP(MegaClusterNetworkInit)
+LOGVARIABLE(NetworkStatsReport, 0)
+LOGVARIABLE(NetworkStepsMultipliers, 0)
+LOGGROUP(ReplicationDataLifetime)
+LOGGROUP(NetworkInstances)
+DYNAMIC_LOGVARIABLE(NetworkPacketsReceive, 0)
+LOGVARIABLE(RenderTextureCompositor, 0)
+LOGVARIABLE(RenderTextureCompositorBudget, 0)
+LOGVARIABLE(NetworkStreaming, 0)
+LOGVARIABLE(PhysicsSenderSleepingLog, 0)
+LOGVARIABLE(BrowserActivity, 0)
+LOGVARIABLE(Android, 6)
+DYNAMIC_LOGVARIABLE(GoogleAnalyticsTracking, 0)
+LOGVARIABLE(AutoSave, 0)
+DYNAMIC_LOGVARIABLE(NamedMutex, 0)
+DYNAMIC_LOGVARIABLE(MaxJoinDataSizeKB, 100)
+DYNAMIC_LOGVARIABLE(NetworkJoin, 0)
+DYNAMIC_LOGVARIABLE(AnalyticsLog, 0)
+DYNAMIC_LOGVARIABLE(R15Character, 0)

Guides/DiscriminatingRCCLogs/flogs-2021E.txt

@@ -0,0 +1,30 @@
+"DFLogAnalyticsDeferredEphemeralReportingDiagnostic": 0,
+"DFLogErrorDescription": 6,
+"DFLogMaxJoinDataSizeKB": 20,
+"DFLogMessagingService": 6,
+"DFLogPartStreamingRequests": 0,
+"DFLogPlaceLauncherTeleportLog": 6,
+"DFLogPlayerCharacterLoadingLog": 0,
+"DFLogSessionCrypto": 333,
+"DFLogSignalRConnection": 6,
+"DFLogSoundCheckResult": 4,
+"DFLogSoundTrace": 4,
+"DFLogTrackTeamCreateSaveMD5HashOutput": 6,
+"FLogBrowserActivity": 3,
+"FLogCloseDataModel": 3,
+"FLogDXVideoMemory": 4,
+"FLogError": 6,
+"FLogFastLogValueChanged": 0,
+"FLogFullRenderObjects": 0,
+"FLogGraphics": 6,
+"FLogHangDetection": 3,
+"FLogLegacyLock": 0,
+"FLogLuaBridge": 2,
+"FLogNetwork": 7,
+"FLogNetworkItemQueueDtor": 1,
+"FLogPlayerShutdownLuaTimeoutSeconds": 1,
+"FLogRccSoapMicroprofiling": 6,
+"FLogStudioLocalSaveMD5HashOutput": 6,
+"FLogUpdateUIManager": 6,
+"FLogVR": 6,
+"FLogWrapLogChannel": 0,