NUMA-aware page allocation

Allocate page-granular memory bound to a NUMA node, or interleaved across a set of them. More...

Files
file	rmemnuma.h
	NUMA placement for whole-page allocations.

Functions
rboolean	r_mem_numa_available (void)
	Whether this build and this machine actually place allocations on the node they are asked for.
rsize	r_mem_numa_page_size (void)
	Granularity every allocation here is rounded up to.
rpointer	r_mem_numa_alloc_local (rsize size)
	Allocate size bytes placed by the system's default policy.
rpointer	r_mem_numa_alloc_onnode (rsize size, ruint node)
	Allocate size bytes bound to NUMA node node.
rpointer	r_mem_numa_alloc_interleaved (rsize size, const RBitset *nodeset)
	Allocate size bytes spread page-by-page across nodeset.
void	r_mem_numa_free (rpointer ptr, rsize size)
	Release an allocation from this API.
rboolean	r_mem_numa_bind (rpointer ptr, rsize size, ruint node)
	Bind the pages spanning ptr to node node.
rboolean	r_mem_numa_interleave (rpointer ptr, rsize size, const RBitset *nodeset)
	Interleave the pages spanning ptr across nodeset.
rboolean	r_mem_numa_node_of (rconstpointer ptr, ruint *node)
	Which node currently backs the page containing ptr.

Detailed Description

Allocate page-granular memory bound to a NUMA node, or interleaved across a set of them.

Function Documentation

r_mem_numa_alloc_interleaved()

rpointer r_mem_numa_alloc_interleaved	(	rsize	size,
		const RBitset *	nodeset
	)

Allocate size bytes spread page-by-page across nodeset.

Trades the latency of a local binding for the combined bandwidth of several nodes, which pays off for a buffer read by threads on all of them. Falls back to default placement where interleaving is not supported, which includes win32 - it can place an allocation on one chosen node but not spread it over several.

Parameters

size	Bytes required (rounded up to a page).
nodeset	Nodes to interleave over; NULL or empty means every node this process is allowed to allocate from.

Returns

Zero-filled allocation, or NULL on failure.

r_mem_numa_alloc_local()

rpointer r_mem_numa_alloc_local

(

rsize

size

)

Allocate size bytes placed by the system's default policy.

In practice that means first touch: each page lands on the node of whichever CPU first writes to it, which is exactly what you want when the thread that fills a buffer is also the one that reads it. Unlike the other two allocators this needs no NUMA support, so it is the portable way to get page-granular memory from this API.

Parameters

size	Bytes required (rounded up to a page).

Returns

Zero-filled allocation, or NULL on failure.

r_mem_numa_alloc_onnode()

rpointer r_mem_numa_alloc_onnode	(	rsize	size,
		ruint	node
	)

Allocate size bytes bound to NUMA node node.

Every page comes from node's memory, whichever CPU touches it. Where placement is not enforced (see r_mem_numa_available) the allocation still succeeds with default placement rather than failing.

Parameters

size	Bytes required (rounded up to a page).
node	Node index, as reported by r_sys_topology_node_id.

Returns

Zero-filled allocation, or NULL on failure.

r_mem_numa_available()

rboolean r_mem_numa_available

(

void

)

Whether this build and this machine actually place allocations on the node they are asked for.

FALSE means every call here still works, but the allocators fall back to the system's default placement and the re-placement calls fail outright.

Placement being enforced does not imply a range can be re-placed after the fact: win32 fixes it at allocation time, so r_mem_numa_bind and r_mem_numa_interleave fail there regardless.

Returns

TRUE if the platform honours node placement.

r_mem_numa_bind()

rboolean r_mem_numa_bind	(	rpointer	ptr,
		rsize	size,
		ruint	node
	)

Bind the pages spanning ptr to node node.

Works on a whole allocation or on a page-aligned slice of one, so a single mapping can be cut up per worker thread. Pages already faulted in are left where they are; only pages touched from here on follow the new binding.

Parameters

ptr	Start of the range (rounded down to a page boundary).
size	Range length in bytes.
node	Node index, as reported by r_sys_topology_node_id.

Returns

TRUE on success, FALSE if node does not exist or the platform cannot re-place a live range (win32 among them - allocate with r_mem_numa_alloc_onnode instead).

r_mem_numa_free()

void r_mem_numa_free	(	rpointer	ptr,
		rsize	size
	)

Release an allocation from this API.

Parameters

ptr	Allocation from r_mem_numa_alloc_local / r_mem_numa_alloc_onnode / r_mem_numa_alloc_interleaved. NULL is a no-op.
size	The same size that was allocated - the mapping has to be unmapped at the length it was created with.

r_mem_numa_interleave()

rboolean r_mem_numa_interleave	(	rpointer	ptr,
		rsize	size,
		const RBitset *	nodeset
	)

Interleave the pages spanning ptr across nodeset.

The r_mem_numa_bind notes on alignment and already-faulted pages apply here too.

Parameters

ptr	Start of the range (rounded down to a page boundary).
size	Range length in bytes.
nodeset	Nodes to interleave over; NULL or empty means every node this process is allowed to allocate from.

Returns

TRUE on success, FALSE if the platform cannot re-place a live range.

r_mem_numa_node_of()

rboolean r_mem_numa_node_of	(	rconstpointer	ptr,
		ruint *	node
	)

Which node currently backs the page containing ptr.

Reports where a page actually ended up, which is the only way to confirm a first-touch policy did what you expected. Faults the page in to answer, so the very call that asks is what decides the answer for an untouched page.

Parameters

ptr	Address inside a mapped page.
node	Receives the node index (must be non-NULL).

Returns

TRUE on success, FALSE if the platform cannot report it or ptr is not mapped.

r_mem_numa_page_size()

rsize r_mem_numa_page_size

(

void

)

Granularity every allocation here is rounded up to.

The system page size. The bytes between a requested size and the page boundary are usable.

Returns

Page size in bytes (never 0).