Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for RDMA Atomics + Support for user-defined initiator and responder resources on connect/accept #51

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Support for RDMA Atomics + Support for user-defined initiator and responder resources on connect/accept #51

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
14ee342
Added support for user-defined responder and initiator resources on c…
TaranovK Mar 31, 2020
0e7f403
added support for atomics and made a simple test
TaranovK Mar 31, 2020
402d8a5
added support for atomics and made a simple test
TaranovK Apr 5, 2020
f96a00c
added fully functioning query device
TaranovK Apr 6, 2020
759e282
added setting max connparams for various endpoints
TaranovK Apr 21, 2020
09a75a0
bug in old disni connect/accept.
Jun 6, 2020
642ed59
bumped the version
TaranovK Feb 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 124 additions & 2 deletions libdisni/src/verbs/com_ibm_disni_verbs_impl_NativeDispatcher.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,7 @@ JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1connect(

if (cm_listen_id != NULL) {
memset(&conn_param, 0, sizeof(conn_param));
conn_param.initiator_depth = dev_attr.max_qp_rd_atom;
conn_param.initiator_depth = dev_attr.max_qp_init_rd_atom;
conn_param.responder_resources = dev_attr.max_qp_rd_atom;
conn_param.retry_count = (unsigned char)retry;
conn_param.rnr_retry_count = (unsigned char)rnr_retry;
Expand All @@ -430,6 +430,41 @@ JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1connect(
}
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _connectV2
* Signature: (JJ)V
*/
JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1connectV2(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the idea of multiple connect implementations. Why not only keep V2 and expose query_device to Java. This solves multiple problems at once: 1) setting initiator/responder resource without knowing what the max are 2) two implementations of the same function

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. However, since the libdisni is shipped separately, it may force some developers to recompile their java code which uses the library.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should be ok. Developers can always choose to use an old version of the library if needed.

JNIEnv *env, jobject obj, jlong id, jlong param) {
struct rdma_cm_id *cm_listen_id = NULL;
struct rdma_conn_param *conn_param = NULL;

cm_listen_id = (struct rdma_cm_id *)id;
conn_param = (struct rdma_conn_param *)param;

if (cm_listen_id != NULL && conn_param!=NULL) {
int ret = rdma_connect(cm_listen_id, conn_param);
if (ret == 0) {
log("j2c::connect: ret %i, guid %" PRIu64 "\n", ret,
ibv_get_device_guid(cm_listen_id->verbs->device));
} else {
log("j2c::connect: rdma_connect failed\n");
JNU_ThrowIOExceptionWithLastError(env,
"j2c::connect: rdma_connect failed");
}
} else {
if(cm_listen_id == NULL){
log("j2c:connect: cm_listen_id null\n");
JNU_ThrowIOException(env, "j2c:connect: cm_listen_id null\n");
} else {
log("j2c:connect: conn_param null\n");
JNU_ThrowIOException(env, "j2c:connect: conn_param null\n");
}
}
}


/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _accept
Expand All @@ -446,7 +481,7 @@ JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1accept(

if (cm_listen_id != NULL) {
memset(&conn_param, 0, sizeof(conn_param));
conn_param.initiator_depth = dev_attr.max_qp_rd_atom;
conn_param.initiator_depth = dev_attr.max_qp_init_rd_atom;
conn_param.responder_resources = dev_attr.max_qp_rd_atom;
conn_param.retry_count = (unsigned char)retry;
conn_param.rnr_retry_count = (unsigned char)rnr_retry;
Expand All @@ -463,6 +498,40 @@ JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1accept(
}
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _acceptV2
* Signature: (JJ)V
*/
JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1acceptV2(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment to connectV2

JNIEnv *env, jobject obj, jlong id, jlong param) {
struct rdma_cm_id *cm_listen_id = NULL;
struct rdma_conn_param *conn_param = NULL;

cm_listen_id = (struct rdma_cm_id *)id;
conn_param = (struct rdma_conn_param *)param;

if (cm_listen_id != NULL && conn_param!=NULL) {
int ret = rdma_accept(cm_listen_id, conn_param);
if (ret == 0) {
log("j2c::connect: ret %i, guid %" PRIu64 "\n", ret,
ibv_get_device_guid(cm_listen_id->verbs->device));
} else {
log("j2c::connect: rdma_connect failed\n");
JNU_ThrowIOExceptionWithLastError(env,
"j2c::connect: rdma_connect failed");
}
} else {
if(cm_listen_id == NULL){
log("j2c:connect: cm_listen_id null\n");
JNU_ThrowIOException(env, "j2c:connect: cm_listen_id null\n");
} else {
log("j2c:connect: conn_param null\n");
JNU_ThrowIOException(env, "j2c:connect: conn_param null\n");
}
}
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _ackCmEvent
Expand Down Expand Up @@ -823,6 +892,59 @@ Java_com_ibm_disni_verbs_impl_NativeDispatcher__1queryOdpSupport(JNIEnv *env,
return ret;
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _queryMaxResponderResources
* Signature: (J)I
*/
JNIEXPORT jint JNICALL
Java_com_ibm_disni_verbs_impl_NativeDispatcher__1queryMaxResponderResources(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is not what I had in mind. I would like to be able to get all device attributes in Java. Otherwise we will be adding new JNI functions for every member of the struct along the way.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. it was just more work.

JNIEnv *env,
jobject obj,
jlong id) {
jint ret = -1;
struct ibv_context *context = (struct ibv_context *)id;

struct ibv_device_attr dev_attr;
ret = ibv_query_device(context, &dev_attr);

if(ret == 0) {
ret = dev_attr.max_qp_rd_atom;
} else {
log("j2c::queryMaxResponderResources: ibv_query_device failed, error %s\n",
strerror(ret));
ret = -1;
JNU_ThrowIOExceptionWithLastError(env, "j2c::queryMaxResponderResources: ibv_query_device failed");
}
return ret;
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _queryMaxInitiatorDepth
* Signature: (J)I
*/
JNIEXPORT jint JNICALL
Java_com_ibm_disni_verbs_impl_NativeDispatcher__1queryMaxInitiatorDepth(JNIEnv *env,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment queryMaxResponderResources

jobject obj,
jlong id) {
jint ret = -1;
struct ibv_context *context = (struct ibv_context *)id;

struct ibv_device_attr dev_attr;
ret = ibv_query_device(context, &dev_attr);

if(ret == 0) {
ret = dev_attr.max_qp_init_rd_atom;
} else {
log("j2c::queryMaxInitiatorDepth: ibv_query_device failed, error %s\n",
strerror(ret));
ret = -1;
JNU_ThrowIOExceptionWithLastError(env, "j2c::queryMaxInitiatorDepth: ibv_query_device failed");
}
return ret;
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _expPrefetchMr
Expand Down
37 changes: 37 additions & 0 deletions libdisni/src/verbs/com_ibm_disni_verbs_impl_NativeDispatcher.h
100755 → 100644
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions src/main/java/com/ibm/disni/RdmaEndpointGroup.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,10 @@ public RdmaConnParam getConnParam() {
return connParam;
}

public void setConnParam(RdmaConnParam connParam) {
this.connParam = connParam;
}

public synchronized void close() throws IOException, InterruptedException {
logger.info("shutting down group");
if (closed.get()){
Expand Down
4 changes: 4 additions & 0 deletions src/main/java/com/ibm/disni/verbs/IbvContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,4 +101,8 @@ public IbvCQ createCQ(IbvCompChannel compChannel, int ncqe, int comp_vector) thr
}

public int queryOdpSupport() throws IOException { return verbs.queryOdpSupport(this); }

public int queryMaxResponderResources() throws IOException { return verbs.queryMaxResponderResources(this); }

public int queryMaxInitiatorDepth() throws IOException { return verbs.queryMaxInitiatorDepth(this); }
}
4 changes: 2 additions & 2 deletions src/main/java/com/ibm/disni/verbs/IbvSendWR.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ public Rdma getRdma() {
}

/**
* Unsupported.
*
*
* @return the atomic
*/
Expand Down Expand Up @@ -322,7 +322,7 @@ public String getClassName() {
}

/**
* Unsupported.
*
*/
public static class Atomic {
protected long remote_addr;
Expand Down
34 changes: 29 additions & 5 deletions src/main/java/com/ibm/disni/verbs/RdmaConnParam.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
package com.ibm.disni.verbs;

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

// TODO: Auto-generated Javadoc
//struct rdma_conn_param {
Expand Down Expand Up @@ -49,8 +51,11 @@ public class RdmaConnParam {
protected byte retry_count;
protected byte rnr_retry_count;
protected byte srq;
protected byte reserved;
protected int qp_num;

public static int CSIZE = 24;

public RdmaConnParam() {
this.private_data_addr = 0;
this.private_data_len = 0;
Expand All @@ -60,6 +65,7 @@ public RdmaConnParam() {
this.retry_count = 0;
this.rnr_retry_count = 0;
this.srq = 0;
this.reserved = 0;
this.qp_num = 0;
}

Expand Down Expand Up @@ -111,7 +117,7 @@ public byte getResponder_resources() {
* @param responder_resources the new responder resources.
*/
public void setResponder_resources(byte responder_resources) throws IOException {
throw new IOException("Operation currently not supported");
this.responder_resources = responder_resources;
}

/**
Expand All @@ -129,7 +135,7 @@ public byte getInitiator_depth() {
* @param initiator_depth the new initiater depth.
*/
public void setInitiator_depth(byte initiator_depth) throws IOException {
throw new IOException("Operation currently not supported");
this.initiator_depth = initiator_depth;
}

/**
Expand All @@ -147,7 +153,7 @@ public byte getFlow_control() {
* @param flow_control the new flow control.
*/
public void setFlow_control(byte flow_control) throws IOException {
throw new IOException("Operation currently not supported");
this.flow_control = flow_control;
}

/**
Expand Down Expand Up @@ -201,7 +207,7 @@ public byte getSrq() {
* @param srq the new shared receive queue.
*/
public void setSrq(byte srq) throws IOException {
throw new IOException("Operation currently not supported");
this.srq = srq;
}

/**
Expand All @@ -219,6 +225,24 @@ public int getQp_num() {
* @param qp_num the new qp_num
*/
public void setQp_num(int qp_num) throws IOException {
throw new IOException("Operation currently not supported");
this.qp_num = qp_num;
}


public void writeBack(ByteBuffer buffer) {
buffer.putLong(private_data_addr);
TaranovK marked this conversation as resolved.
Show resolved Hide resolved
buffer.put(private_data_len);
buffer.put(responder_resources);
buffer.put(initiator_depth);
buffer.put(flow_control);
buffer.put(retry_count);
buffer.put(rnr_retry_count);
buffer.put(srq);
buffer.put(reserved);
buffer.putInt(qp_num);
}

public int size() {
return CSIZE;
}
}
18 changes: 18 additions & 0 deletions src/main/java/com/ibm/disni/verbs/RdmaVerbs.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,24 @@ public abstract IbvCQ createCQ(IbvContext context,
*/
public abstract int queryOdpSupport(IbvContext context) throws IOException;

/**
* Query the maximum number of incoming RDMA read and atomic operations that the local side can accept.
*
* @param context the device context.
* @return dev_attr.max_qp_rd_atom
* @throws Exception on failure.
*/
public abstract int queryMaxResponderResources(IbvContext context) throws IOException;

/**
* Query The maximum number of outstanding RDMA read and atomic operations that the local side can have
*
* @param context the device context.
* @return dev_attr.max_qp_init_rd_atom
* @throws Exception on failure.
*/
public abstract int queryMaxInitiatorDepth(IbvContext context) throws IOException;

/**
* Prefetch part of a memory region.
* Can be used only with MRs registered with IBV_EXP_ACCESS_ON_DEMAND
Expand Down
Loading