Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Split configuration for device and others in compile model #26977

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
e10f512
Split configuration for device and others
praasz Oct 9, 2024
32750d6
Fix call compile model with weights
praasz Oct 9, 2024
cdc5655
Remove modify core state in compile_model
praasz Oct 13, 2024
f9a2362
Remove not used variable
praasz Oct 13, 2024
b1387b2
Fix MSVC build error
praasz Oct 13, 2024
4f33336
Auto-Batch remove prop validation to meta device
praasz Oct 14, 2024
a6ad92e
Add missing supported properties in Template
praasz Oct 14, 2024
ce4a5a0
Add parse only properties in auto-batch
praasz Oct 14, 2024
d9e02b9
Merge remote-tracking branch 'origin/master' into split-config-in-com…
praasz Oct 14, 2024
945aab2
Template plugin sync supported properties
praasz Oct 14, 2024
a89eed8
CPU plugin sync supported properties
praasz Oct 14, 2024
0426bda
Hetero plugin sync supported properties
praasz Oct 14, 2024
8ae4d36
Make `parse_only_meta_device` member private
praasz Oct 14, 2024
1220a36
Merge branch 'master' into split-config-in-compile-model
praasz Oct 16, 2024
2a9b1e6
Merge branch 'master' into split-config-in-compile-model
praasz Oct 21, 2024
327c93a
Remove not used parameter in function
praasz Oct 22, 2024
82e337f
Merge branch 'master' into split-config-in-compile-model
praasz Oct 22, 2024
45e270d
Merge branch 'master' into split-config-in-compile-model
praasz Oct 23, 2024
ec5b0cf
Merge branch 'master' into split-config-in-compile-model
praasz Oct 25, 2024
837f633
Merge branch 'master' into split-config-in-compile-model
praasz Oct 28, 2024
cd5516b
Merge branch 'master' into split-config-in-compile-model
praasz Nov 12, 2024
803b5c3
Merge branch 'master' into split-config-in-compile-model
praasz Nov 14, 2024
b7e1ea7
Merge branch 'master' into split-config-in-compile-model
praasz Nov 15, 2024
d9dfae2
Merge branch 'master' into split-config-in-compile-model
praasz Nov 18, 2024
fabdba5
Merge branch 'master' into split-config-in-compile-model
praasz Nov 18, 2024
82b6c5c
Add kv_cache_precision as supported property by GPU
praasz Nov 20, 2024
982b9e8
Merge branch 'master' into split-config-in-compile-model
praasz Nov 21, 2024
c1ffd36
Merge branch 'master' into split-config-in-compile-model
praasz Nov 29, 2024
ce822f7
Merge branch 'master' into split-config-in-compile-model
praasz Dec 2, 2024
2772f3d
Filter core properties by using local core config
praasz Dec 10, 2024
46c05ba
Revert changes
praasz Dec 10, 2024
434b67f
Update `parseDeviceNameIntoConfig` to provide local core configuration
praasz Dec 11, 2024
78b5226
Merge remote-tracking branch 'origin/master' into split-config-in-com…
praasz Dec 11, 2024
c0b700b
Fix issues after merge
praasz Dec 11, 2024
7dc7b73
Clean core properties for HW devices in parseDeviceNameIntoConfig
praasz Dec 11, 2024
9c7f13a
Add comment about cache_dir removal from compile config
praasz Dec 11, 2024
511bb88
Re order code in core_impl.cpp
praasz Dec 11, 2024
ca0ba63
Merge branch 'master' into split-config-in-compile-model
praasz Dec 11, 2024
99508e5
Merge remote-tracking branch 'origin/master' into split-config-in-com…
praasz Dec 12, 2024
f182198
Explicitly remove copy assign operator for CoreConfig
praasz Dec 12, 2024
3370ef8
Merge branch 'master' into split-config-in-compile-model
praasz Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/inference/src/cpp/core.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ CompiledModel Core::compile_model(const std::shared_ptr<const ov::Model>& model,
const std::string& device_name,
const AnyMap& config) {
OV_CORE_CALL_STATEMENT({
const auto device_not_supported = _impl->get_device_unsupported_properties(device_name, config);
set_property(device_not_supported);
auto exec = _impl->compile_model(model, device_name, config);
return {exec._ptr, exec._so};
});
Expand All @@ -119,6 +121,8 @@ CompiledModel Core::compile_model(const std::wstring& model_path, const AnyMap&

CompiledModel Core::compile_model(const std::string& model_path, const std::string& device_name, const AnyMap& config) {
OV_CORE_CALL_STATEMENT({
const auto device_not_supported = _impl->get_device_unsupported_properties(device_name, config);
set_property(device_not_supported);
auto exec = _impl->compile_model(model_path, device_name, config);
return {exec._ptr, exec._so};
});
Expand All @@ -137,6 +141,8 @@ CompiledModel Core::compile_model(const std::string& model,
const std::string& device_name,
const AnyMap& config) {
OV_CORE_CALL_STATEMENT({
const auto device_not_supported = _impl->get_device_unsupported_properties(device_name, config);
set_property(device_not_supported);
praasz marked this conversation as resolved.
Show resolved Hide resolved
auto exec = _impl->compile_model(model, weights, device_name, config);
return {exec._ptr, exec._so};
});
Expand All @@ -146,6 +152,8 @@ CompiledModel Core::compile_model(const std::shared_ptr<const ov::Model>& model,
const RemoteContext& context,
const AnyMap& config) {
OV_CORE_CALL_STATEMENT({
const auto device_not_supported = _impl->get_device_unsupported_properties(context.get_device_name(), config);
set_property(device_not_supported);
auto exec = _impl->compile_model(model, ov::SoPtr<ov::IRemoteContext>{context._impl, context._so}, config);
return {exec._ptr, exec._so};
});
Expand Down
61 changes: 52 additions & 9 deletions src/inference/src/dev/core_impl.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,28 @@ void clean_batch_properties(const std::string& deviceName, ov::AnyMap& config, c
}
}
}

/**
* @brief Splits input configuration on two parts: supported and others
*
* @param config Configuration with properties to be splitted.
* @param supported List of supported configuration properties.
* @return Pair of configs where first is supported properties, second are others.
*/
std::pair<ov::AnyMap, ov::AnyMap> split_config(const ov::AnyMap& config, const ov::AnyMap& supported) {
auto split_config = std::make_pair(ov::AnyMap{}, config);
auto& plugin_config = split_config.first;
auto& others_config = split_config.second;

for (const auto& supported_property : supported) {
auto config_property = config.find(supported_property.first);
if (config_property != config.end()) {
plugin_config.insert(plugin_config.end(), *config_property);
others_config.erase(supported_property.first);
}
}
return split_config;
}
} // namespace

bool ov::is_config_applicable(const std::string& user_device_name, const std::string& subprop_device_name) {
Expand Down Expand Up @@ -748,12 +770,12 @@ ov::SoPtr<ov::ICompiledModel> ov::CoreImpl::compile_model(const std::shared_ptr<
res = load_model_from_cache(cacheContent, plugin, parsed._config, ov::SoPtr<ov::IRemoteContext>{}, [&]() {
return compile_model_and_cache(plugin,
model,
parsed._config,
get_device_properties(device_name, parsed._config),
ov::SoPtr<ov::IRemoteContext>{},
cacheContent);
});
} else {
res = plugin.compile_model(model, parsed._config);
res = plugin.compile_model(model, get_device_properties(device_name, parsed._config));
}
return res;
}
Expand All @@ -779,10 +801,14 @@ ov::SoPtr<ov::ICompiledModel> ov::CoreImpl::compile_model(const std::shared_ptr<
cacheContent.blobId = ov::ModelCache::compute_hash(model, create_compile_config(plugin, parsed._config));
std::unique_ptr<CacheGuardEntry> lock = cacheGuard.get_hash_lock(cacheContent.blobId);
res = load_model_from_cache(cacheContent, plugin, parsed._config, context, [&]() {
return compile_model_and_cache(plugin, model, parsed._config, context, cacheContent);
return compile_model_and_cache(plugin,
model,
get_device_properties(deviceName, parsed._config),
context,
cacheContent);
});
} else {
res = plugin.compile_model(model, context, parsed._config);
res = plugin.compile_model(model, context, get_device_properties(deviceName, parsed._config));
}
return res;
}
Expand All @@ -797,7 +823,6 @@ ov::SoPtr<ov::ICompiledModel> ov::CoreImpl::compile_model(const std::string& mod
ov::SoPtr<ov::ICompiledModel> compiled_model;

auto cacheManager = coreConfig.get_cache_config_for_device(plugin, parsed._config)._cacheManager;

if (cacheManager && device_supports_model_caching(plugin) && !is_proxy_device(plugin)) {
// Skip caching for proxy plugin. HW plugin will load network from the cache
CacheContent cacheContent{cacheManager, model_path};
Expand All @@ -806,10 +831,14 @@ ov::SoPtr<ov::ICompiledModel> ov::CoreImpl::compile_model(const std::string& mod
compiled_model =
load_model_from_cache(cacheContent, plugin, parsed._config, ov::SoPtr<ov::IRemoteContext>{}, [&]() {
auto model = read_model(model_path, std::string{});
return compile_model_and_cache(plugin, model, parsed._config, {}, cacheContent);
return compile_model_and_cache(plugin,
model,
get_device_properties(device_name, parsed._config),
{},
cacheContent);
});
} else {
compiled_model = plugin.compile_model(model_path, parsed._config);
compiled_model = plugin.compile_model(model_path, get_device_properties(device_name, parsed._config));
}
return compiled_model;
}
Expand All @@ -836,13 +865,13 @@ ov::SoPtr<ov::ICompiledModel> ov::CoreImpl::compile_model(const std::string& mod
auto model = read_model(model_str, weights);
return compile_model_and_cache(plugin,
model,
parsed._config,
get_device_properties(device_name, parsed._config),
ov::SoPtr<ov::IRemoteContext>{},
cacheContent);
});
} else {
auto model = read_model(model_str, weights);
compiled_model = plugin.compile_model(model, parsed._config);
compiled_model = plugin.compile_model(model, get_device_properties(device_name, parsed._config));
}
return compiled_model;
}
Expand Down Expand Up @@ -1655,3 +1684,17 @@ std::map<std::string, ov::Version> ov::CoreImpl::get_versions(const std::string&

return versions;
}

ov::AnyMap ov::CoreImpl::get_device_properties(const std::string& device_name, const AnyMap& config) const {
const auto parsed = parseDeviceNameIntoConfig(device_name, config);
return is_virtual_device(parsed._deviceName)
? parsed._config
: split_config(parsed._config, get_supported_property(parsed._deviceName, parsed._config, false)).first;
}

ov::AnyMap ov::CoreImpl::get_device_unsupported_properties(const std::string& device_name, const AnyMap& config) const {
const auto parsed = parseDeviceNameIntoConfig(device_name, config);
return is_virtual_device(parsed._deviceName)
? AnyMap{}
: split_config(parsed._config, get_supported_property(parsed._deviceName, parsed._config, false)).second;
}
18 changes: 18 additions & 0 deletions src/inference/src/dev/core_impl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,24 @@ class CoreImpl : public ov::ICore, public std::enable_shared_from_this<ov::ICore
}

Any get_property(const std::string& device_name, const std::string& name, const AnyMap& arguments) const override;

/**
* @brief Get the device supported properties from configuration.
*
* @param device_name Device name.
* @param config Input configuration.
* @return Properties map supported by device.
*/
AnyMap get_device_properties(const std::string& device_name, const AnyMap& config) const;

/**
* @brief Get not suported properties by device from configuration.
*
* @param device_name Device name.
* @param config Input configuration.
* @return Properties map not supported by device.
*/
AnyMap get_device_unsupported_properties(const std::string& device_name, const AnyMap& config) const;
};

} // namespace ov
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,19 @@ TEST_P(InferRequestPropertiesTest, ReusableCPUStreamsExecutor) {
}
}
}

TEST_P(InferRequestPropertiesTest, ConfigHasUnsupportedPluginProperty) {
configuration.insert({ov::enable_mmap(false)});
if (target_device.find(ov::test::utils::DEVICE_AUTO) == std::string::npos &&
target_device.find(ov::test::utils::DEVICE_MULTI) == std::string::npos &&
target_device.find(ov::test::utils::DEVICE_HETERO) == std::string::npos &&
target_device.find(ov::test::utils::DEVICE_BATCH) == std::string::npos) {
OV_ASSERT_NO_THROW(core->set_property(target_device, configuration));
}
// Compile model to target plugins
execNet = core->compile_model(function, target_device, configuration);
OV_ASSERT_NO_THROW(execNet.create_infer_request());
}
} // namespace behavior
} // namespace test
} // namespace ov
Loading