===== Byte and Word Access =====
The original Alpha architecture has no byte and no 16-bit word memory instructions: every integer load and store is a longword or a quadword, and byte and word access is synthesized from ''LDQ_U''/''STQ_U'' plus the extract, insert and mask instructions. The byte/word extension (BWX), introduced with the EV56 (21164A), added byte and word loads and stores.
Linux 6.10 removed support for processors older than the EV56, and Debian builds for EV56 or later, so every current Alpha/Linux system has BWX. [(>[[https://git.kernel.org/torvalds/c/a4184174be36|"alpha: drop pre-EV56 support"]], Linux 6.10)] For software built today the rules are short:
* **Build for EV56 or later.** With ''-mcpu=ev56'' (or ''pca56'', ''ev6'', ''ev67''), byte and word loads and stores are single instructions, and a byte or word store does not disturb its neighbors. The compiler's default may still be the pre-BWX baseline (see [[documentation:toolchains#distribution_baselines|Distribution baselines]]); ''%%__alpha_bwx__%%'' is defined when BWX code generation is enabled, so ''%%gcc -dM -E - < /dev/null | grep __alpha_bwx__%%'' shows whether it is.
* **Assemble for EV56 or later.** Without ''.arch ev56'' or ''-mcpu=ev56'', the assembler silently expands ''stb'' and ''stw'' into non-atomic pre-BWX sequences (see [[documentation:toolchains#binutils|Toolchains]]).
* **Byte and word atomics are still emulated.** There are no byte or word load-locked or store-conditional instructions, even with BWX; atomic operations on ''char'' and ''short'' objects operate on the containing quadword (see [[#load-locked_and_store-conditional_are_not_affected|below]]).
* **Keep independently updated atomic variables apart.** A store-conditional fails if another processor writes anywhere in the locked range, which is at least a quadword and often more (see [[#reservation_granularity|Reservation granularity]]).
The pre-BWX sequences and the sparse I/O space that the original design required are described in the later sections of this page, for reading old code and kernel sources.
See also [[documentation:porting:unaligned_access|Unaligned Access]], [[documentation:porting:memory_model|Memory Model]] and [[documentation:amask|Architecture Mask (amask)]].
==== BWX: the Byte/Word Extension ====
BWX comprises six instructions, ''LDBU'', ''LDWU'', ''SEXTB'', ''SEXTW'', ''STB'', and ''STW'', and is reported by bit 0 of ''AMASK''. [(>{{wiki:documentation:references:alpha_architecture_reference_manual_4th_edition.pdf?linkonly|Alpha Architecture Reference Manual, Fourth Edition}}, Appendix D)]
^ Instruction ^ Opcode ^ Operation ^
| ''LDBU Ra, disp(Rb)'' | 0x0A | load byte, zero-extended to 64 bits |
| ''LDWU Ra, disp(Rb)'' | 0x0C | load 16-bit word, zero-extended to 64 bits |
| ''STW Ra, disp(Rb)'' | 0x0D | store 16-bit word |
| ''STB Ra, disp(Rb)'' | 0x0E | store byte |
| ''SEXTB Rb, Rc'' | 0x1C.00 | sign-extend byte |
| ''SEXTW Rb, Rc'' | 0x1C.01 | sign-extend word |
There is no sign-extending byte or word //load//; the equivalent is ''LDBU'' followed by ''SEXTB''.
The Architecture Reference Manual states which form to prefer:
LDBU and STB are the recommended way to perform byte load and store operations on Alpha implementations that support them; use them rather than the extract, insert, and mask byte instructions described in this section.
Linux does not emulate these instructions: a process that executes one on a pre-EV56 processor receives ''SIGILL''.
=== Which processors have BWX ===
^ Processor ^ BWX ^
| 21064, 21064A (EV4, EV45) | no |
| 21066, 21068 (LCA) | no |
| 21164 (EV5) | no |
| 21164A (EV56) | yes |
| 21164PC (PCA56) | yes |
| 21264 and variants (EV6, EV67, EV68) | yes |
| 21364 (EV7) | yes |
=== Compiler support ===
alpha-unknown-linux-gnu-gcc -mcpu=ev56 … # or pca56, ev6, ev67
alpha-unknown-linux-gnu-gcc -mbwx … # explicit
alpha-unknown-linux-gnu-gcc -mno-bwx … # force the pre-BWX sequences
With BWX enabled, ordinary C produces single instructions:
lb: ldbu $0,0($16)
lsb: ldbu $0,0($16) ; sextb $0,$0
sb: stb $17,0($16)
lw: ldwu $0,0($16)
sw: stw $17,0($16)
Two further flags, added in GCC 15, address the non-atomicity described below: [(>[[https://gcc.gnu.org/onlinedocs/gcc/DEC-Alpha-Options.html|DEC Alpha Options]], GCC manual)]
^ Flag ^ Meaning ^
| ''-msafe-bwa'' | when BWX is not available, make byte and aligned word accesses thread and async-signal safe |
| ''-msafe-partial'' | make other partial memory accesses safe, such as piecemeal accesses to unaligned data and block accesses to the leading and trailing parts of objects that do not start and end on a quadword boundary; this applies with or without BWX |
The compile-time predicate is ''%%__alpha_bwx__%%'', defined when BWX code generation is enabled.
GCC has no builtins for the BWX instructions: ''%%__builtin_alpha_ldbu%%'', ''%%__builtin_alpha_ldwu%%'', ''%%__builtin_alpha_stb%%'' and ''%%__builtin_alpha_stw%%'' do not exist. The instructions are obtained by compiling with ''-mbwx'' and writing ordinary ''unsigned char'' and ''unsigned short'' accesses. GCC provides builtins for the pre-BWX byte-manipulation instructions: ''%%__builtin_alpha_extbl%%'', ''%%__builtin_alpha_insbl%%'', ''%%__builtin_alpha_mskbl%%'' and the rest of the extract, insert and mask family, along with ''%%__builtin_alpha_amask%%'', ''%%__builtin_alpha_implver%%'', ''%%__builtin_alpha_cmpbge%%'', ''%%__builtin_alpha_zap%%'' and ''%%__builtin_alpha_zapnot%%''.
=== Detecting BWX at runtime ===
''amask(1) == 0'' indicates that BWX is present. See [[documentation:amask|Architecture Mask (amask)]] for the instruction, its bit assignments and the kernel's ''AT_HWCAP'' export.
==== Atomicity of byte and word access ====
=== The problem ===
On a pre-BWX Alpha, storing a byte means loading the containing quadword, modifying one byte of it, and storing the whole quadword back. If another processor stores to a //different// byte of the same quadword in between, that store is silently lost.
C and C++ both require that distinct scalar objects can be updated concurrently without interfering. Pre-BWX Alpha cannot honor that for ''char'' and ''short'' without ''-msafe-bwa'', which GCC 15 added for this purpose. [(>[[https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=3d4d82211c8cbfde0b852bde1603b5d549426df7|"Alpha: Add option to avoid data races for sub-longword memory stores [PR117759]"]], Maciej W. Rozycki, GCC, 30 Mar 2025)] The Linux kernel likewise assumes atomic byte stores.
Michael Cree, the Debian Alpha port maintainer, estimated that possibly a third of the bugs in the port arose from non-atomic byte and word accesses, and noted that kernel developers wanted to assume atomic byte and word access on all architectures, which only pre-BWX Alpha prevented. [(>[[https://lists.debian.org/msgid-search/Y5gtYxqv78cTCdau@creeky|"Re: glibc regression on alpha with 2.34+"]], Michael Cree, debian-alpha, 13 Dec 2022)] [(>[[https://lists.debian.org/msgid-search/20140905084622.GC5281@omega|"RFC: compile with BWX thus only support EV56 and later CPUs"]], Michael Cree, debian-alpha, 5 Sep 2014)]
=== How it ended ===
Debian Ports raised its Alpha baseline to EV56 in 2023, and Linux followed in 6.10. Alpha EV5 and earlier was the only CPU family the kernel supported without native byte access, and parts of the RCU code already relied on byte access and did not work reliably on EV5. [(>[[https://lists.debian.org/msgid-search/b4adb884e4c1014ecce7d1857adb7feb48ba837f.camel@physik.fu-berlin.de|"Re: [GIT PULL] alpha: cleanups and build fixes for 6.10"]], John Paul Adrian Glaubitz quoting Arnd Bergmann, debian-alpha, 12 May 2024)] Linux 6.10 dropped pre-EV56 support, and removed Jensen, LCA and APECS support along with it.
Linux and Debian therefore require EV56 or later.
=== Load-locked and store-conditional are not affected ===
The locked primitives exist only at longword and quadword granularity:
* ''LDL_L'' / ''STL_C'', 32-bit
* ''LDQ_L'' / ''STQ_C'', 64-bit
There is no ''LDB_L'', ''STB_C'', ''LDW_L'' or ''STW_C'', and BWX adds none. Atomic byte and word operations must be synthesized, on every Alpha, from the quadword primitives plus masking.
GCC generates the following for ''%%__sync_fetch_and_add((unsigned char *)p, 1)%%'' at ''-mcpu=ev4'':
afa:
bic $16,7,$3 /* round down to a QUADWORD */
ldq_u $1,0($16)
extbl $1,$16,$1
$L8:
mb
addl $1,1,$2
and $1,0xff,$5
insbl $2,$16,$2
$L11:
ldq_l $6,0($3) /* load-locked the containing quadword */
extbl $6,$16,$1
cmpeq $1,$5,$4
beq $4,$L12
mskbl $6,$16,$4
bis $4,$2,$4
stq_c $4,0($3)
beq $4,$L11
mb
$L12:
beq $4,$L8
mov $5,$0
ret $31,($26),1
Three points follow:
* The alignment mask is ''bic $16,7'', rounding to a quadword rather than a longword, because ''extbl'', ''insbl'' and ''mskbl'' index by the low //three// bits of the address.
* The primitive is ''LDQ_L''/''STQ_C''. A hand-written version using ''LDL_L'' with quadword-relative extract and insert offsets silently corrupts memory for every address whose bit 2 is set.
* At ''-mcpu=ev56'' the loop body is identical; only the initial load becomes ''ldbu''. BWX changes nothing here.
=== Reservation granularity ===
The reservation is neither per-byte nor per-longword. From the Architecture Reference Manual:
When a LDx_L instruction is executed without faulting, the processor records the target physical address in a per-processor locked_physical_address register and sets the per-processor lock_flag.
If processor A's lock_flag is set and processor B successfully does a store within A's locked range of physical addresses, then A's lock_flag is cleared. A processor's locked range is the aligned block of %%2**N%% bytes that includes the locked_physical_address. The %%2**N%% value is implementation dependent. It is at least 8 (minimum lock range is an aligned quadword) and is at most the page size for that implementation (maximum lock range is one physical page).
and, on what to do about it:
* Hardware implementations are encouraged to lock no more than 128 bytes.
* Software implementations are encouraged to separate locked locations by at least 128 bytes from other locations that could potentially be written by another processor while the first location is locked.
The granularity relevant to contention is therefore an implementation-defined aligned block, which may be larger than a cache block. Two atomic variables in the same lock range cause each other's ''STQ_C'' to fail, even if they are in different cache blocks; separating them by 128 bytes, as the manual recommends, avoids this on every implementation.
/* BAD: two independently-updated atomic bytes inside one lock range.
Every update to one makes the other's STQ_C fail and retry. */
struct {
_Atomic unsigned char producer_flag;
_Atomic unsigned char consumer_flag;
} state;
/* BETTER: separate them by the architecture's recommended distance. */
struct {
_Atomic unsigned char producer_flag;
char pad[127];
_Atomic unsigned char consumer_flag;
} state_padded;
Folding the flags into a single word and using bit operations avoids the problem entirely, leaving one contended location:
_Atomic uint32_t flags;
atomic_fetch_or_explicit(&flags, 1U << 0, memory_order_relaxed);
atomic_fetch_and_explicit(&flags, ~(1U << 1), memory_order_relaxed);
=== Contention causes retries rather than deadlock ===
Sustained contention can starve an individual thread, but cannot stall the system. The architecture requires that:
If two processors attempt STx_C instructions to the same lock range and that lock range was accessed by both processors' preceding LDx_L instructions, exactly one of the stores succeeds.
The portability hazard lies elsewhere, and is covered under [[documentation:porting:memory_model#forward_progress_rules|Forward progress rules]]: an ''LDx_L''/''STx_C'' body containing any other memory access, a taken branch, a subsetted instruction, or too many instructions may fail every time on some implementations.
==== Pre-BWX byte and word access ====
This section describes the code generated for processors without BWX, which current kernels no longer support.
=== Loading a byte ===
ldq_u $0,0($16) /* aligned quadword containing the byte */
extbl $0,$16,$0 /* extract, selecting the byte by $16<2:0> */
''EXTBL'' uses the low three bits of its second operand, that is, the byte offset within a //quadword//. This is significant for the atomic sequences described below.
A signed byte costs two more instructions, because there is no sign-extending load:
ldq_u $0,0($16)
extbl $0,$16,$0
sll $0,56,$0
sra $0,56,$0
=== Storing a byte ===
A store is a read-modify-write of the surrounding quadword. The Architecture Reference Manual gives the canonical sequence:
LDA R6, X(R11) ; R6<2:0> = (X mod 8) = 5
LDQ_U R1, X(R11) ; Ignores va<2:0>, R1 = yyAx xxxx
INSBL R5, R6, R3 ; R3 = 00A0 0000
MSKBL R1, R6, R1 ; R1 = yy0x xxxx
OR R1, R3, R1 ; R1 = yyAx xxxx
STQ_U R1, X(R11)
The sequence is five instructions plus the address materialization, where x86 uses a single ''MOV''. GCC emits the same form, using ''insql'' rather than ''insbl'' (equivalent when the source register already holds a zero-extended byte):
insql $17,$16,$17
ldq_u $1,0($16)
mskbl $1,$16,$1
bis $17,$1,$17
stq_u $17,0($16)
This read-modify-write is not atomic, which is the source of a large class of defects on Alpha. Two processors storing to two different bytes of the same quadword can lose one of the two stores entirely. See [[#atomicity_of_byte_and_word_access|Atomicity of Byte and Word Access]].
=== Loading a 16-bit word ===
An //aligned// word is two instructions, exactly like a byte:
ldq_u $0,0($16)
extwl $0,$16,$0
An //unaligned// word may straddle a quadword boundary, so it needs both halves:
ldq_u $0,0($16)
ldq_u $1,1($16)
extwl $0,$16,$0
extwh $1,$16,$1
bis $0,$1,$0
zapnot $0,3,$0
=== Storing a 16-bit word ===
The unaligned case is the longest of these sequences: two read-modify-writes, ten instructions.
LDA R6, X(R11) ; R6<2:0> = (X mod 8) = 5
LDQ_U R2, X+1(R11)
LDQ_U R1, X(R11)
INSWH R5, R6, R4
INSWL R5, R6, R3
MSKWH R2, R6, R2
MSKWL R1, R6, R1
OR R2, R4, R2
OR R1, R3, R1
STQ_U R2, X+1(R11) ; Must store high then low for
STQ_U R1, X(R11) ; degenerate case of aligned
The Architecture Reference Manual's comment states the ordering constraint: the high quadword must be stored before the low one, so that the degenerate case where both ''LDQ_U'' hit the same quadword still produces the right answer.
An //aligned// word store is a single read-modify-write, as for a byte.
==== I/O space: sparse and dense ====
=== Why sparse windows exist ===
An EV4 cannot issue a byte or word write to the bus, while ISA and EISA devices require byte-granular access. The transfer length and the byte lane are therefore encoded in the physical address, and the chipset decodes them from a longword access.
For the mainstream chipsets (APECS, LCA, CIA/Pyxis, T2) the layout is:
Physical address = IO_BASE + (io_port << 5) + ((length - 1) << 3)
^ Field ^ Bits ^ Values ^
| length - 1 | addr<4:3> | 0 = byte, 1 = word, 2 = tribyte, 3 = longword or quadword |
| byte lane | addr<6:5> | from ''io_port & 3'', placed there by the ''<< 5'' |
In practice the offsets are ''0x00'' for a byte, ''0x08'' for a word and ''0x18'' for a longword. The encoded value for a longword is 3, not 2; value 2 denotes a tribyte transfer.
The chipset performs no byte shifting. From the DECchip 21171-CA Technical Reference Manual:
Software must use longword load or store instructions (LDL/STL) to perform a reference that is of longword length or less on the PCI bus.
The bytes to be transferred must be positioned within the longword in the correct byte lanes as indicated by the PCI byte enable.
The hardware will do no byte shifting within the longword.
Software must place the byte in the correct lane. Linux uses ''%%__kernel_insbl(b, addr & 3)%%'' on write and ''%%__kernel_extbl(result, addr & 3)%%'' on read:
__EXTERN_INLINE void cia_iowrite8(u8 b, void __iomem *xaddr)
{
...
w = __kernel_insbl(b, addr & 3);
*(vuip) ((addr << 5) + base_and_type) = w;
}
Only the Jensen replicated the byte to all four lanes.
Barriers are mandatory rather than advisory. From the same manual:
Programmers are required to insert memory barrier (MB) instructions between sparse space accesses to prevent collapsing in the 21164 write buffer.
Sparse access is markedly slower than dense: every read is a load plus an extract, every write a load plus an insert, each needs an ''MB'', and consecutive ports are 32 bytes apart so nothing coalesces.
=== Dense windows ===
A dense mapping is linear: physical byte N corresponds to bus address N, with no stride and no lane encoding.
Dense space is a chipset feature rather than a CPU feature. Pre-BWX chipsets such as APECS, LCA and T2 do provide a dense memory window, usable for longword and quadword access on an EV4. They cannot provide byte and word access through it, so Linux converted a dense address back into the sparse window to service ''readb''/''writeb''. From the APECS support removed in Linux 6.10:
if (addr >= APECS_DENSE_MEM) {
addr -= APECS_DENSE_MEM;
APECS_SET_HAE;
base_and_type = APECS_SPARSE_MEM + 0x00;
}
Byte/word-capable windows arrived with Pyxis, which added them as separate windows rather than making the existing dense window byte-capable:
#define CIA_DENSE_MEM (IDENT_ADDR + 0x8600000000UL)
#define CIA_BW_MEM (IDENT_ADDR + 0x8800000000UL)
#define CIA_BW_IO (IDENT_ADDR + 0x8900000000UL)
#define CIA_BW_CFG_0 (IDENT_ADDR + 0x8a00000000UL)
#define CIA_BW_CFG_1 (IDENT_ADDR + 0x8b00000000UL)
Tsunami, Polaris and Irongate likewise. The kernel expresses the capability per machine vector, not by a runtime CPU check:
#define t2_trivial_rw_bw 0 /* T2: must use sparse */
#define cia_bwx_trivial_rw_bw 1 /* Pyxis: plain dereference */
#define tsunami_trivial_rw_bw 1
#define irongate_trivial_rw_bw 1
''trivial_rw_bw = 1'' means ''readb''/''writeb'' are ordinary dereferences, which the compiler turns into ''LDBU''/''STB'' when built for BWX. Byte and word I/O therefore requires both a BWX CPU and a byte/word-capable chipset; neither alone is sufficient.
^ ^ Sparse ^ Dense ^
| Chipset requirement | any | a chipset that provides a dense window |
| Byte/word access | yes, via lane encoding | only through a byte/word window (Pyxis and later) plus a BWX CPU |
| Address stride | 32x | 1x |
| Physical space used | large | compact |
| Software complexity | high | low |
| Performance | slow | fast |
=== I/O memory is not accessed as memory ===
Linus Torvalds stated the rule that has been kernel policy since, on the Alpha list in 1996: [(>[[https://github.com/alphalinux/mailing-list-archives/blob/main/axp-list-mbox/1996-July.mbox#L2088|"Re: MMIO for new NCR810 driver?"]], Linus Torvalds, axp-list, 13 Jul 1996)]
You MUST NOT access IO memory any other way than read/write[bwl]().
or memcpy_fromio/toio(). EVER.
[...]
On other architectures you'll only see a _window_ into the IO space, so that the IO access functions have to set up the window correctly, and the same processor virtual address can actually be _different_ IO ports depending on where the window happens to be. Or look at the Jensen, for example: it has _no_ support for dense memory, and you _cannot_ make the IO memory look like a memory access.
That "window" is the HAE (host address extension) register, which supplies the high address bits that do not fit in the sparse encoding. Every sparse access has to set it, and the kernel caches its value in ''alpha_mv.hae_cache'' to avoid rewriting it unnecessarily.
=== BWX in the core logic ===
A processor with BWX can make byte-granular accesses to I/O space only if the core logic also implements them. Byte and word access to I/O space was reported never to have worked on the 21172 (CIA-2), with Pyxis the first core logic to implement it correctly, although FreeBSD later enabled it on a 21172-based AlphaServer 800 without reported problems. [(>[[https://github.com/alphalinux/mailing-list-archives/blob/main/freebsd-alpha-mbox/gmane-2005-September.mbox#L5417|"[patch] cia BWX oddity in RELENG4"]], Paul V. Bolotoff, freebsd-alpha, 30 Sep 2005)]
==== Jensen ====
The [[hardware:systems|Jensen]] (DEC 2000 AXP, DECpc AXP 150), an early EV4-based EISA machine whose Linux support was removed in 6.10, predates these conventions: it used a 128-byte sparse stride, replicated bytes to all four lanes, had no dense space, and reached its keyboard, RTC, parallel and serial ports through a separate local I/O window.
==== Summary ====
^ Topic ^ Key point ^
| Pre-BWX byte load | ''LDQ_U'' + ''EXTBL'', 2 instructions |
| Pre-BWX byte store | ''LDQ_U'' + ''INSBL'' + ''MSKBL'' + ''OR'' + ''STQ_U'', 5 instructions, not atomic |
| Pre-BWX unaligned word store | two read-modify-writes, 10 instructions, high quadword stored first |
| BWX | six instructions: ''LDBU'', ''LDWU'', ''STB'', ''STW'', ''SEXTB'', ''SEXTW'' |
| BWX availability | EV56, PCA56, EV6 and later; ''SIGILL'' on earlier processors under Linux |
| BWX detection | ''amask(1) == 0'' means present; ''AMASK'' clears bits for features that exist |
| BWX builtins | none exist; ''-mbwx'' with ordinary C is used instead |
| Byte/word atomicity | pre-BWX cannot provide it; this is why Linux 6.10 dropped pre-EV56 |
| Byte atomics | ''LDQ_L''/''STQ_C'' on the containing quadword, on every Alpha; BWX adds no locked byte access |
| Reservation granularity | implementation-defined aligned block: quadword minimum, page maximum, 128 bytes recommended |
| Sparse I/O | length-1 in addr<4:3>, lane in addr<6:5>; software positions the byte; ''MB'' required |
| Dense I/O | a chipset property; byte/word through it needs a Pyxis-or-later window plus a BWX CPU |
| Jensen | 128-byte stride, byte replication, no dense space, and a separate local I/O path |
==== References ====
* {{wiki:documentation:references:alpha_architecture_reference_manual_4th_edition.pdf?linkonly|Alpha Architecture Reference Manual, Fourth Edition}}. §4.6 covers the extract, insert and mask instructions and the canonical byte and word sequences; §4.11.1 covers ''AMASK''; Appendix D, Table D-3 gives the AMASK bit assignments. The 1992 and 1995 editions predate BWX and AMASK; the [[documentation:references|Alpha Architecture Handbook, Version 4]] also documents both.
* {{wiki:documentation:references:alpha_architecture_reference_manual_1st_edition.pdf?linkonly|Alpha Architecture Reference Manual, First Edition}}, §4.2, for the ''LDx_L''/''STx_C'' lock range definition.
* DECchip 21171-CA Technical Reference Manual, Chapter 6, for the PCI sparse space encodings and the memory barrier requirement.
* [[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/alpha/include/asm/special_insns.h|''arch/alpha/include/asm/special_insns.h'']] for ''amask()''. The ''core_apecs.h'', ''core_lca.h'' and ''jensen.h'' headers were removed in Linux 6.10 by ''430ad3f0ddd7'' ("alpha: remove LCA and APECS based machines") and ''f81f335a56a8'' ("alpha: remove DECpc AXP150 (Jensen) support"), and are available in earlier revisions.
* Arnd Bergmann, ''a4184174be36'' ("alpha: drop pre-EV56 support"), Linux 6.10.
* [[documentation:amask|Architecture Mask (amask)]].
See [[history:community#mailing_lists|Mailing Lists]] for archive locations.
{{tag>documentation porting}}