CUDA support #57
Comments
|
hmmm I was able to compile it by adding - // OrtSessionOptionsAppendExecutionProvider_CUDA(sessionOptions, 0);
+ unsafe {
+ OrtSessionOptionsAppendExecutionProvider_CUDA(session_options_ptr, 0);
+ } |
|
Oh yeah actually I was able to get CUDA based inferencing working with just diff --git a/onnxruntime-sys/wrapper.h b/onnxruntime-sys/wrapper.h
index e63d352..c7c0cde 100644
--- a/onnxruntime-sys/wrapper.h
+++ b/onnxruntime-sys/wrapper.h
@@ -1 +1,2 @@
-#include "onnxruntime_c_api.h"
+#include "onnxruntime/core/providers/cuda/cuda_provider_factory.h"
+#include "onnxruntime/core/session/onnxruntime_c_api.h"
diff --git a/onnxruntime/src/session.rs b/onnxruntime/src/session.rs
index c3b6e88..53d2b0b 100644
--- a/onnxruntime/src/session.rs
+++ b/onnxruntime/src/session.rs
@@ -125,6 +125,14 @@ impl<'a> SessionBuilder<'a> {
Ok(self)
}
+ /// Use CUDA
+ pub fn use_cuda(self) -> Result<SessionBuilder<'a>> {
+ unsafe {
+ sys::OrtSessionOptionsAppendExecutionProvider_CUDA(self.session_options_ptr, 0);
+ }
+ Ok(self)
+ }
+
/// Set the session's allocator
///
/// Defaults to [`AllocatorType::Arena`](../enum.AllocatorType.html#variant.Arena)and then regenerating the bindings and building with the ORT_USE_GPU environment variables and stuff. On my machine a Titan Xp with CUDA is about 8 to 10 times faster than using the CPU (AMD Ryzen 9 3900x). Not sure how to make these changes work with people who don't use CUDA though. Maybe need some kind of |
dllu
changed the title
|
Great! I'm glad you were able to make it work. As you found out, uncommenting the lines in the example will not work; it's a copy-paste from the original C example which I left when I ported the example. See https://github.com/microsoft/onnxruntime/blob/v1.4.0/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp#L41-L43 From your last patch it seems there is no need for a I'm not sure about how the function is being called though. If you look at how different functions are called, there is a difference. For example in I don't have access to an nvidia system for now so it's hard for me to test this... |
|
Unlike relevant documentation from The |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am interested in getting onnxruntime-rs running with CUDA based inference. (I'm also interested in getting AMDMIGraphX inference working but that's a whole nother can of worms)
Anyway in
onnxruntime-rs/onnxruntime-sys/examples/c_api_sample.rsthere is:But uncommenting the line doesn't work since symbols
OrtSessionOptionsAppendExecutionProvider_CUDAandsessionOptionsare not available.Also generally the CUDA doesn't seem to be working since it is still using CPU for inferencing even though I compiled with
with
onnxruntimecompiled with./build.sh --use_cuda --cudnn_home /usr/ --cuda_home /opt/cuda/ --config RelWithDebInfo --parallel --build_shared_liband installed in/usr/local.The text was updated successfully, but these errors were encountered: