[openib-general] mthca and DDR not hidden

Sat Oct 2 12:26:12 PDT 2004

Hello!
Quoting r. Sean Hefty (mshefty at ichips.intel.com) "Re: [openib-general] mthca and DDR not hidden":
> On Thu, 30 Sep 2004 16:40:05 -0400
> Hal Rosenstock <halr at voltaire.com> wrote:
> 
> > On Thu, 2004-09-30 at 16:32, Roland Dreier wrote:
> > > Also, the
> > > consumer must either create its own MR using that PD, or have access
> > > to at least the L_Key for the MAD layer's MR.
> > 
> > That would be an acceptable solution too: to have the consumer create
> > the MR based on the MAD PD. Then the lkey doesn't need exposing but
> > exposing and copying the lkey seems simpler to me,
> 
> Okay - I think I got it now.  Some more thoughts on the topic:
> 
> It seems like having the consumer create the MR would be a more
> generic solution, but the least efficient.  The idea of exposing MR
> implies that the user know the access layer implementation in order to
> get the right virtual address when posting MADs.  (The user must know
> that the MR applies to all of system memory.)  Giving
> out only the lkey seems less desirable, since the client must also
> assume the starting virtual address.

I'd like to suggest that the mad layer could expose an allocator
function that the user will call to grab the memory for the mad.
This function would return the memory pointer and the appropriate
lkey value.

This would make it possible to change the implementation to have
even multiple MRs if needed (for example for tavor you need
two MRs to cover the whole 64 bit address space).

On systems like the Apple's G5 where enabling memory for DMA
consumes IOMMU resources, it may even be a good idea to
implement this by keeping a pool of MADs and allocate from there.

MST