Add analytics logging to MosaicMLLogger
#3106
Conversation
…a-to-metadata-for-analytics
…nterval` to analytics logs
…a-to-metadata-for-analytics
…:mosaicml/composer into angel/add-data-to-metadata-for-analytics
…:mosaicml/composer into angel/add-data-to-metadata-for-analytics
|
Added some tests and adjusted the way we display |
…:mosaicml/composer into angel/add-data-to-metadata-for-analytics
| backend, _, _ = parse_uri(self.analytics_data.save_folder) | ||
| metrics['composer/cloud_provided_save_folder'] = backend if backend else 'local' | ||
|
|
||
| # Save interval can be passed in w/ multiple types. If the type is a function, then |
There was a problem hiding this comment.
Is there some idea for utility of analytics on save interval? Nothing is coming to mind for me
There was a problem hiding this comment.
this was requested in a Slack thread a while back. this included some less-helpful metrics (i.e. num_workers) so if it isn't useful i'll take it out. @dakinggg
There was a problem hiding this comment.
yeah i think just drop this one
…alytics logging fails
| @@ -96,10 +115,57 @@ def log_hyperparameters(self, hyperparameters: Dict[str, Any]): | |||
| def log_metrics(self, metrics: Dict[str, Any], step: Optional[int] = None) -> None: | |||
| self._log_metadata(metrics) | |||
|
|
|||
| def log_analytics(self, state: State, loggers: Tuple[LoggerDestination, ...]) -> None: | |||
There was a problem hiding this comment.
Should this be a callback instead? It seems like all loggers and experiment tracking tools would benefit from having these extra information for navigation or reproducibility purposes.
| def init(self, state: State, logger: Logger) -> None: | ||
| try: | ||
| self.log_analytics(state, logger.destinations) | ||
| except: |
There was a problem hiding this comment.
can we log the exception here? We should also probably only log once. if its failing consistently, we don't want to spam the run with warnings
|
@mvpatel2000 are you suggesting a new logger function |
|
@mvpatel2000 @angel-ruiz7 @dakinggg is this one basically done? or are there still concerns? |
Add
log_analyticsfunction toMosaicMLLoggerThis adds a
log_analyticsfunction to theMosaicMLLogger, which records these metrics about composer runs on the Mosaic platform:fsdp_configsharding_strategystate_dict_typeactivation_checkpointingforward_prefetchbackward_prefetchmixed_precisiondevice_meshautoresumealgorithmscloud_providerscloud_provider_datacloud_provider_checkpointssave_intervalloggersprecisionoptimizerBy logging these fields from the
MosaicMLLoggerinto a run's metadata, we can then parse the metadata in SQL and add these metrics into UC. This will enable anyone on the genai team to create dashboards / queries for analytical purposes and view the patterns in these metrics across runs on the platform. (see the genai_runs table for more info)Output of the
Quick StartExample