Add a memory barrier before setting an inline data segment's byte count

We need a memory barrier before setting an inline segment byte count
to make sure that all the inline data for a cacheline has been written
before changing the cacheline's byte-count from 0xffffffff to
something valid.

Signed-off-by: Ishai Rabinovitz <ishai@mellanox.co.il>
Signed-off-by: Jack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: Roland Dreier <rolandd@cisco.com>

1 files changed, 13 insertions, 0 deletions

diff --git a/src/qp.c b/src/qp.c
index 8f4bc0f..4e3c358 100644
--- a/src/qp.c
+++ b/src/qp.c
@@ -274,6 +274,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr,
 					wqe += to_copy;
 					addr += to_copy;
 					seg_len += to_copy;
+					wmb(); /* see comment below */
 					seg->byte_count = htonl(MLX4_INLINE_SEG | seg_len);
 					seg_len = 0;
 					seg = wqe;
@@ -289,6 +290,18 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr,
 
 			if (seg_len) {
 				++num_seg;
+				/*
+				 * Need a barrier here to make sure
+				 * all the data is visible before the
+				 * byte_count field is set.  Otherwise
+				 * the HCA prefetcher could grab the
+				 * 64-byte chunk with this inline
+				 * segment and get a valid (!=
+				 * 0xffffffff) byte count but stale
+				 * data, and end up sending the wrong
+				 * data.
+				 */
+				wmb();
 				seg->byte_count = htonl(MLX4_INLINE_SEG | seg_len);
 			}