Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix tensorflow import in dataset wrapper to handle invalid tensorflow installation due to unsupported protobuf #176

Merged
merged 1 commit into from
Aug 21, 2024

Conversation

imatiach-msft
Copy link
Contributor

@imatiach-msft imatiach-msft commented Aug 21, 2024

This PR is mainly to fix one of our nightly notebook tests that is failing. In the notebook test we are using the EndpointWrapper from ml-wrappers but getting the error for the file https://github.com/Azure/azureml-examples/blob/main/sdk/python/responsible-ai/text/responsibleaidashboard-text-classification-blbooksgenre.ipynb:

2024-08-19T18:56:24.3991935Z     615 def __new__(cls, name, full_name, index, number, type, cpp_type, label,
2024-08-19T18:56:24.3992539Z     616             default_value, message_type, enum_type, containing_type,
2024-08-19T18:56:24.3993063Z     617             is_extension, extension_scope, options=None,
2024-08-19T18:56:24.3993504Z     618             serialized_options=None,
2024-08-19T18:56:24.3993993Z     619             has_default_value=True, containing_oneof=None, json_name=None,
2024-08-19T18:56:24.3994691Z     620             file=None, create_key=None):  # pylint: disable=redefined-builtin
2024-08-19T18:56:24.3995311Z --> 621   _message.Message._CheckCalledFromGeneratedFile()
2024-08-19T18:56:24.3995727Z     622   if is_extension:
2024-08-19T18:56:24.3996143Z     623     return _message.default_pool.FindExtensionByName(full_name)
2024-08-19T18:56:24.3996499Z 
2024-08-19T18:56:24.3996669Z TypeError: Descriptors cannot be created directly.
2024-08-19T18:56:24.3997428Z If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
2024-08-19T18:56:24.3998317Z If you cannot immediately regenerate your protos, some other possible workarounds are:
2024-08-19T18:56:24.3998940Z  1. Downgrade the protobuf package to 3.20.x or lower.
2024-08-19T18:56:24.3999803Z  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

After close investigation, it looks like one package, azure-ai-ml, is installing google-api-core which depends on a newer version of protobuf, after an older version of tensorflow==2.9.3 was installed with an older protobuf range. When ml-wrappers tries to import tensorflow for a helper utility, the import fails with TypeError. This PR prevents this error from occurring by ignoring the TypeError and printing a debug message:

>>> from ml_wrappers.model import EndpointWrapperModel
2024-08-21 10:53:19.392103: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-08-21 10:53:19.392270: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Could not import tensorflow in ml-wrappers due to TypeError, required if using a Tensorflow model. Inner exception: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
>>>

Also, as part of this PR the ml-wrappers version was bumped so a release can be done, the is_sequential utility method was updated/fixed to work with latest tensorflow version and some tests were re-enabled.

@imatiach-msft imatiach-msft force-pushed the ilmat/handle-protobuf branch 3 times, most recently from 00a4b28 to 91d9e38 Compare August 21, 2024 16:59
@imatiach-msft imatiach-msft merged commit 0b5a721 into microsoft:main Aug 21, 2024
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants