Introduction
Welcome to "Learn unsafe Rust", a compassionate and comprehensive resource for learning unsafe Rust.
The materials in this book are broken down into three major volumes:
Each volume has multiple chapters. Those chapters can generally be read in any order, whereas each chapter builds on the foundations laid by the previous chapter.
Undefined behavior
“People shouldn't call for demons unless they really mean what they say.”
— C.S. Lewis, The Last Battle
"Undefined behavior" is a bit of a strange notion. On one hand, the reference clearly defines some (but not all) causes of undefined behavior. This list includes some causes that are generally well-known: dereferencing a null pointer, causing a data race, executing incorrect inline assembly. These all have a direct translation for real, common machines and so it is common to misunderstand "undefined behavior" to be "platform-specific behavior". Maybe on x86 it will continue on, perhaps on ARM it will cause a fault. While this can be true, undefined behavior is usually more nuanced because of:
Abstract machines
High-level programming languages allow programming for a wide variety of targets by abstracting away the specific properties of each one, and targeting a single "abstract machine". C and C++ have their own "abstract machines", and so does Rust. This means that the semantics and rules of an abstract machine depend heavily on the language that it's for.
When we write Rust code, we're writing code for this abstract machine. We're not writing code that follows the rules for some set of targets; there is only one set of rules for the abstract machine. It's just that the consequences for breaking those rules depends on the target and the compiler itself. With this perspective, it's easier to see that undefined behavior is platform-independent.
Rust's abstract machine
Rust's abstract machine has not been rigorously defined, and it may never be. Efforts to rigorously define Rust's abstract machine are usually colloquially called "standardizing Rust". You may even have heard of some of these efforts, like the Ferrocene Language Specification. It's important to note that these standards are for some language arbitrarily close to Rust; they're not standards for the official Rust language.
Rather than describing the entirety of Rust's abstract machine, Rust's official reference has defined just some of the rules of the abstract machine. Breaking one of these rules definitely results in undefined behavior. These are the rules that we'll cover in the Core unsafety and Advanced unsafety volumes. There are also ideas about undefined behavior that are being explored right now, but they haven't been officially adopted as rules yet. Some of these are covered in the Expert unsafety volume.
Triggering undefined behavior
Undefined behavior in Rust is always triggered by some condition being met, and usually this condition is just "some code getting executed in a particular way" or "some code violating an invariant upheld by the compiler". Because of this, it's often tempting to think of undefined behavior as telling your program "if you get here, do whatever you want". However, undefined behavior is a purely compile-time concept. It's not telling your program "do whatever you want", it's telling the compiler "assume this can never happen". The compiler may not have a better response than saying "if you get here, panic". Or, it may be able to use that promise to better optimize your code.
As an example, consider this Rust code:
#![allow(unused)] fn main() { use std::hint::unreachable_unchecked; unsafe fn char_to_int(c: char) -> u8 { match c { '0' => 0, '1' => 1, '2' => 2, '3' => 3, '4' => 4, '5' => 5, '6' => 6, '7' => 7, '8' => 8, '9' => 9, _ => unsafe { unreachable_unchecked() }, } } }
This converts a char '0'..'9'
to its corresponding integer value. In the last
arm of the match, we call unreachable_unchecked()
, which is a compiler hint
that says "it would be undefined behavior to reach here". Because we promised
the compiler that c
won't be any value other than 0..9
, it could optimize
this function into something like:
#![allow(unused)] fn main() { unsafe fn char_to_int(c: char) -> u8 { c as u8 - b'0' } }
Consider what would happen if we called char_to_int('A')
. 'A'
has a value of
65
as a u8
, and '0'
has a value of 48
as a u8
. So the optimized
version of our function would return 17
. But what if the compiler chose to
optimize our function a different way instead:
#![allow(unused)] fn main() { unsafe fn char_to_int(c: char) -> u8 { c as u8 & 0b1111 } }
This does the same thing as our original version for all characters '0'..'9'
.
Now consider what this version of our function would do if we called
char_to_int('A')
. 'A'
has a value of 65
(0b0010_0001
in binary
notation), so this version would return 0010_0111 & 0000_1111 = 0111 = 15
.
This is a different result than we would have gotten with the previous
optimization!
Finally, consider this version:
#![allow(unused)] fn main() { unsafe fn char_to_int(c: char) -> u8 { let c = c as u8; if c < b'0' || c > b'9' { panic!() } else { c - b'0' } } }
This version doesn't return anything, it panics! In fact, the compiler would be
allowed to put anything in where the panic!()
is located; the resulting
function would be just as correct and optimal as this one.
This strikes at the core of what "undefined behavior" is. The Rust compiler transforms code based on a set of assumptions that always hold. If you break one of these assumptions, then the behavior of your program is undefined because it's impossible to know what transforms the compiler is doing based on them.
Unsoundness
Now that we know exactly what undefined behavior is, we can understand what it means for some Rust code to be unsound. Unsound code refers to either:
- An abstraction (e.g. a function or a trait) that can trigger undefined behavior even when used as prescribed.
- A particular invocation of unsafe code that causes undefined behavior under allowed circumstances.
This leads to two broad rules:
If you can trigger undefined behavior with purely safe code, it's unsound
In purely safe code (that is, code that contains no unsafe
blocks), the
compiler is in charge of enforcing all of the rules to avoid undefined behavior.
If we somehow manage to cause undefined behavior, then there must be some API
that we use which is unsound.
This is also why it can be difficult to build safe abstractions around unsafe code. It's your job as the abstraction designer to make sure that no possible arrangement of safe code can cause undefined behavior. That's a lot to consider!
Unsafe code must accurately document its safety conditions, or it's unsound
The "safety conditions" for unsafe traits and functions are just the conditions under which it does not trigger undefined behavior. These conditions aren't checked by the compiler, they're checked by the people who write the code itself. Therefore, unsafe impls and blocks must be manually checked to verify that the code written upholds all of the conditions required to avoid undefined behavior. Any unsafe code that can trigger undefined behavior when its safety conditions are upheld is unsound.
Common misconceptions
There are a couple misconceptions about UB that often muddy the water when talking about it.
"If it works, it's sound"
Undefined Behavior may be present even if the compiler does end up compiling the code according to the programmer's intent. A future version of the compiler may behave differently, or future changes to an innocuous portion of the code may cause it to fall to the other side of an invisible threshold. Technically it may even compile differently but only on Tuesdays, though that type of nondeterminism is generally rare.
"UB is about what the optimizer is allowed to do"
This is to some extent true but the actual situation is far more nuanced.
It's common for people to think about UB in terms of what an optimizer "is and isn't allowed to do", and in terms of optimizations they know can occur. For example, it's pretty straightforward to see that sneakily writing to memory that you're not supposed to can cause undefined behavior when the optimizer decides to elide a memory read that occurs after your illicit write.
Firstly, some forms of UB just have to do with rules the underlying processor enforces.
But more than that, there are plenty of miscompiles that are hard to explain by simply thinking in terms of why the optimizer would do such a thing.
This is because it's less about what the optimizer is allowed to do and more about what it is allowed to assume. When a program has UB, the optimizer may make an incorrect assumption that snowballs into bigger and bigger incorrect assumptions that cause very unexpected behavior.
It's often very useful to think of potential optimizations the optimizer may do around your code, but that is not sufficient for evaluating whether your code has UB.
Throughout this book there will be examples of how various optimizations may break code exhibiting undefined behavior, however it is crucial to learn the rule behind the breakage rather than just the nature of the optimization.
Core unsafety
Dangling and unaligned pointers
Data races
Intrinsics
ABI and FFI
Platform features
Inline assembly
Advanced unsafety
Uninitialized memory
"I'm Nobody! Who are you? Are you — Nobody — too?"
— Emily Dickinson
While we have covered invalid values, there's another thing that is a kind of invalid value, but has nothing to do with actual bit patterns: Uninitialized memory.
Safely working with uninitialized memory
The basic rule of thumb is: never refer to uninitialized memory with anything other than a raw pointer or something wrapped in MaybeUninit<T>
. Having a stack value or temporary that is uninitialized and has a type that is not MaybeUninit<T>
(or an array of MaybeUninit
s) is always undefined behavior.
A good model for uninitialized memory is that there's an additional value that does not map to any concrete bit pattern (think of it as "byte value #257"), but can be introduced in the abstract machine in various ways, and makes most values invalid.
Any attempt to read uninitialized bytes as a "type that cares about initializedness" will be UB, and the presence of this byte in non-padding locations is considered UB for most types. Most types care about initialized-ness; and the list of types that doesn't derives from treating initializedness as a property of the byte:
- Zero-sized types do not care about initializedness, since they do not have bytes
- Unions do not care about initializedness if they have a variant that does not care about initialized-ness
MaybeUninit<T>
does not care about initializedness since it is internally a union ofT
and a zero-sized type.[MaybeUninit<T>; N]
does not care about initializedness since it doesn't have any bytes that care about initializedness
Fundamentally, initializedness is a property of memory, but whether or not initializedness matters is a property of the access (in particular, of the type used by the access). For types that care about initializedness, typed operations working with uninitialized memory are typically UB, and having a value that contains uninitialized memory is immediately UB.
ptr::copy
is explicitly an untyped copy, and thus it will copy all bytes, including padding, and including initialized-ness, to the destination, regardless of the type T
.
Most other operations copying a type (for example, *ptr
and mem::transmute_copy
) will be typed, and will thus ignore padding and be UB if ever fed uninitialized memory in non-padding positions (assuming the type involved cares about initializedness). This also applies to let x = y
and mem::transmute
, however in those cases if the source data were uninitialized that would already have been UB.
If you explicitly wish to work with uninitialized and partially-initialized types, MaybeUninit<T>
is a useful abstraction since it can be constructed with no overhead and then written to in parts. It's also useful to e.g. refer to an uninitialized buffer with things like &mut [MaybeUninit<u8>]
.
Similarly with invalid values, there are open issues (UGC #77, UGC #346) about whether it is UB to have references to uninitialized memory. When writing unsafe code we recommend you avoid creating such references, choosing to always use MaybeUninit
. When auditing unsafe code, there may be cases where references to uninitialized values are actually safe as long as no uninitialized values are read out of it. In particular, UGC #346 indicates that it is extremely unlikely that having &mut
references to uninitialized values will be immediately UB.
Sources of uninitialized memory
mem::uninitialized()
and MaybeUninit::assume_init()
mem::uninitialized()
is a deprecated API that has a very tempting shape: it lets you do things like let x = mem::uninitialized()
when you want to construct an uninitialized value. It's almost always UB to use since it immediately sets x
to uninitialized memory, which is UB because uninitialized memory is an invalid value for almost all types and it's unsound to produce invalid values.
Use MaybeUninit<T>
instead.
It is still possible to create uninitialized values using MaybeUninit::uninit()
with MaybeUninit::assume_init()
if you have not, in fact, assured that things are initialized.
mem::uninitialized()
is exactly equivalent to MaybeUninit::uninit().assume_init()
, but it is deprecated since MaybeUninit
actually provides the flexibility needed to deal with uninitialized memory safely.
Padding
Padding bytes in structs and enums are usually but not always uninitialized. This means that treating a struct as a bag of bytes (by, say, treating &Struct
as &[u8; size_of::<Struct>()]
and reading from there) is UB even if you don't write invalid values to those bytes, since you are accessing uninitialized u8
s.
The "usually but not always" caveat can be usefully framed as "padding bytes are uninitialized unless proven otherwise". Padding is a property of the access (i.e., the type), not memory, and these bytes are set to being uninitialized whenever a type is created or copied/moved around, but they can be written to by getting a reference to the memory behind the type1, and will be preserved at that spot in memory as long as the type isn't overwritten as a whole.
For example, treating an initialized byte buffer as an &Struct
and then later reading the padding bytes will give initialized values. However, treating an initialized byte buffer as an &mut Struct
and then writing a new Struct
to it will lead to those bytes becoming uninitialized since the Struct
copy will "copy" the uninitialized padding bytes. Similarly, using mem::transmute()
(or mem::zeroed()
) to transmute a byte buffer to a Struct
will uninitialize the padding because it performs a typed copy of the Struct
.
Because ptr::copy
is an untyped copy, it can be used to copy over explicitly-initialized padding.
See the discussion in [UGC #395][ugc395] for more examples.
Unions
Reading a union type as the wrong variant can lead to reading uninitialized memory, for example if the union was initialized to a smaller variant, or if the padding of the two variants don't overlap perfectly.
Rust does not have strict aliasing like C and C++: type punning with a union is safe as long as the corresponding transmute is safe.
MaybeUninit<T>
is actually just a union between T
and ()
under the hood: the rules for correct usage of MaybeUninit
are the same as the rules for correct usage of a union.
Freshly allocated memory
Freshly allocated memory (e.g. the yet-unused bytes in Vec::with_capacity()
or just the result of Allocator::allocate()
) is usually uninitialized. You can use APIs like Allocator::allocate_zeroed()
if you wish to avoid this, though you can still end up making invalid values the same way you can with mem::zeroed()
.
Generally after allocating memory one should make sure that the only part of that memory being read from is known to have been written to. This can be tricky in situations around complex data structures like probing hashtables where you have a buffer which only has some segments initialized, determined by complex conditions.
Not exactly uninitialized: Moved-from values
The following code is UB:
#![allow(unused)] fn main() { use std::ptr; let x = String::new(); // String is not Copy let mut v = vec![]; let ptr = &x as *const String; v.push(x); // move x into the vector unsafe { // dangling pointer reads from moved-from memory let ghost = ptr::read(ptr); } }
Any type of move will do this, even when you "move" the value into a different variable with stuff like let y = x;
.
This isn't quite uninitialized: it's just that using after a move is straight up UB in Rust, much like reading from freed memory. In particular, unlike most pointers to uninitialized values, this dangling pointer is unsound to write to as well.
Working with dangling pointers can often lead to similar problems as working with uninitialized values.
Note that Rust does let you "partially move" out of fields of a struct, in such a case the whole struct is now no longer a valid value for its type, but you are still allowed to "use" the struct to look at other fields, and the value as a whole is no longer usable. When doing such things, make sure there are no pointers that still think the struct is whole and valid.
Caveat: ptr::drop_in_place()
, ManuallyDrop::drop()
, and ptr::read()
ptr::drop_in_place()
and ManuallyDrop::drop()
are interesting: they both call the destructor2 on a value (or a pointed-to value in the case of drop_in_place
). From the perspective of safety they are identical; they are just different APIs for dealing with manually calling destructors.
ManuallyDrop::drop()
makes the following claim:
Other than changes made by the destructor itself, the memory is left unchanged, and so as far as the compiler is concerned still holds a bit-pattern which is valid for the type T.
In other words, Rust does not consider these operations to do the same invalidation as a regular "move from" operation, even though they may have a similar feel. They do not create dangling pointers, and they do not themselves overwrite the memory with an uninitialized value.
There is an open issue about whether Drop::drop()
is itself allowed to produce uninitialized or invalid memory, so it may not be possible to rely on this in a generic context.
ptr::read()
similarly claims that it leaves the source memory untouched, which means that it is still a valid value. Of course, ptr::read()
on a pointer pointing to uninitialized memory will still create an uninitialized value.
For all of these APIs, actually using the dropped or read-from memory may still be fraught depending on the invariants of the value; it's quite easy to cause a double-free by materializing an owned value from the original data after it has already been read-from or dropped.
However, they do not produce uninitialized memory.
Still, it is convenient when writing unsafe code to operate as if these functions produce uninitialized memory on the original source location.
When you might end up making an uninitialized value
Some of the APIs and methods above create uninitialized memory in a pretty straightforward way — don't call MaybeUninit::assume_init()
if things are not actually initialized!
When writing tricky data structures you may end up mistakenly assuming uninitialized memory is initialized. For example imagine building a probing hashmap, backed with allocated memory: only inhabited buckets will be initialized, and if your logic for determining which buckets are inhabited is broken, your code may risk producing uninitialized values.
A subtle case is when you write to uninitialized memory the wrong way. The following code uses a write to a *mut String
that is pointing to uninitialized memory, and exhibits undefined behavior:
#![allow(unused)] fn main() { use std::mem::MaybeUninit; let mut val: MaybeUninit<String> = MaybeUninit::uninit(); let ptr: *mut String = val.as_mut_ptr(); unsafe { // UB! *ptr = String::from("hello world"); } }
This is UB because writing to raw pointers, under the hood, still calls destructors on the old value, the same way a write to an &mut T
does. This is usually quite convenient, but here the old value is uninitialized, and calling a destructor on it is undefined.
APIs like ptr::write()
and MaybeUninit::write()
exist to sidestep this problem. Logically, a write to a raw pointer is functionally the same as a ptr::read()
of the pointer (with the read-value being dropped) followed by a ptr::write()
with the new value.
Signs an uninitialized value was involved
This is largely similar to the situation for invalid values: The compiler is allowed to assume memory is never uninitialized, and since uninitialized memory is a kind of invalid value, all of the failure modes of invalid values are possible.
Often when reading from uninitialized memory you'll see reads to the same, unchanged, memory producing different values.
This is not an exhaustive list: ultimately, having an uninitialized value is UB and it remains illegal even if there are no optimizations that will break.
Be sure to use &[MaybeUninit<u8>]
if treating a type with uninitialized padding as manipulatable memory!
The "destructor" is different from the Drop
trait. Calling the destructor is the process of calling a type's Drop::drop
impl if it exists, and then calling the destructor for all of its fields (also known as "drop glue"). I.e. it's not just Drop
, but rather the entire destruction, of which the Drop
is one part. Types that do not implement Drop
may still have contentful destructors if their transitive fields do.
Invalid values
“If you tell the truth, you don't have to remember anything.”_ — Mark Twain
Values of a particular type in Rust may never have an "invalid" bit pattern for that type. This is true even if that value is never read from afterwards, or if that value simply exists behind an unread reference. From the reference:
"Producing" a value happens any time a value is assigned to or read from a place, passed to a function/primitive operation or returned from a function/primitive operation.
A lot of basic types don't have any rules about invalid values. For example, all bit patterns of the integer types (and arrays of the integer types) are valid. But most other types have some concept of validity.
Types of invalid values
Uninitialized memory
Values of any type can be "uninitialized", which is considered instantly UB even for types like integers. We discuss this further in the chapter on uninitialized memory. For now this chapter will largely cover cases where a type may have an invalid bit pattern, rather than other cases where it may be invalid due to e.g. not having an initialized bit representation at all.
Primitive types with invalid values
bool
s that have bit patterns other than those for true
and false
are invalid. The same goes for char
s representing byte patterns that are considered invalid in UTF-32 (anything that is either a surrogate character, or greater than char::MAX
).
Pointers with invalid values
&T
and &mut T
may not be null, nor may they be unaligned for values of type T
.
fn
pointers and the metadata part of dyn Trait
may not be null either.
Most smart pointer types like Box<T>
and Rc<T>
are invalid when null. Library types may achieve the same behavior using the NonNull<T>
pointer type.
It's also currently invalid for Vec<T>
to have a null pointer for its buffer! Vec<T>
uses NonNull<T>
internally, and empty vectors use a pointer value equal to the alignment of T
.
While the details have not been hammered out yet (UCG #412 and related issues), generally a reference type (&T
) should be considered valid if the underlying pointer is dereferenceable (points to valid memory).
A special case of this is zero-sized types, which are allowed to point to "dangling" memory as long as it is not memory that has been deallocated. Furthermore, pointers to zero sized types constructed from integer literals are always valid, see the std::ptr
docs.
Vec<T>
is a bit of a special case where is allowed to refer to invalid-but-aligned-and-non-null memory when it is empty.
There are a lot of other reasons that a pointer type may not be valid, but these are the ones having to do with the bit pattern and memory allocation. We'll be covering the others in more depth in other chapters (@@note: where?).
"shallow" vs "deep" validity
An open question in Rust's model is whether references and reference-like types have "shallow" validity (roughly, the rules above), or "deep" validity (where a reference is valid only when the pointed-to data is valid, and that applies transitively). This issue is tracked upstream as UGC #77. The current discussion seems to skew towards shallow validity as opposed to deep validity, but this may change.
For the purposes of writing unsafe code, it is convenient to imagine the boundary as being such that &T
/&mut T
references should never point to memory containing invalid values of type T
. However, when auditing existing unsafe code it may be okay to allow scenarios that assume only shallow validity is required, depending on your risk appetite.
Enums with invalid values
Any bit pattern not covered by a variant of an enum is also invalid. For example, with the following enum:
#![allow(unused)] fn main() { enum Colors { Red = 1, Orange = 2, Yellow = 3, Green = 4, Blue = 5, Indigo = 6, Violet = 7, } }
a bit pattern of 8
or 0
(assuming that it gets represented as the explicit discriminant integers) is undefined behavior.
Or in this enum:
#![allow(unused)] fn main() { enum Stuff { Char(char), Number(u32), } }
setting the discriminant bit to something that is not the discriminant of Char
or Number
is invalid. Similarly, setting the discriminant bit to that for Char
but having the value be invalid for a char
is also invalid.
str
The string slice type str
does not actually have any validity constraints: Despite being only for UTF-8 encoded strings, it is valid for str
s to be in any bit pattern, provided you do not call any methods on the string that are not about directly accessing the memory behind it.
Basically, the UTF-8 validity of str
is an implicit safety requirement for most of its methods, however it is fine to hold on to an &str
that points to random bytes. This is a difference between things being "insta-UB" and "UB on use": invalid value UB is typically "insta UB" (it's UB even if you don't do anything with the invalid value), but here you're allowed to do this as long as you don't use the data in certain ways.
This is something that can be relied on when doing things like manipulating or constructing str
s byte-by-byte, where there may be intermediate invalid states.
Of course, reference types like &str
must still satisfy all of the rules about reference validity (being non-null, etc).
Invalid values for general library types
In general, types may have various invalid values based on their internal representation (which may not be stable!).
In addition to NonNull<T>
, the Rust standard library provides NonZeroUsize
and a bunch of other similar NonZero
integer types that work as its integer counterparts, and libraries may use these internally.
Note that Rust's default representation for types is not stable! What might be a valid bit pattern one day may become invalid later, unless you're only relying on things that are known to be invariant. Converting a type to its bits, sending it over the network, and converting it back is extremely fragile, and will break if the two sides are on different platforms or even Rust versions.
As a library user you may not assume anything about the representation of a library type unless it is explicitly documented as such, or if it has a public representation that is known to be stable (for example a public #[repr(C)]
enum)
When you might end up making an invalid value
Invalid values have a chance to crop up when you're reinterpreting a chunk of memory as a value of a different type. This can happen when calling mem::transmute()
, mem::transmute_copy()
, or mem::zeroed()
, when casting a reference to a region of memory into one of a different type, or when accessing the wrong variant of a union
. The value need not be on the stack to be considered invalid: if you gin up an &bool
that points to a bit pattern that is not a valid bool
, that can instantly be UB (in a "deep validity" world) even if you don't read from the reference.
Note that since uninitialized memory is a type of invalid value, any way to produce uninitialized memory (including mem::uninitialized()
) is also a way of producing invalid values.
Invalid values can also be created when receiving values over FFI where either the signature of the function is incorrect (e.g. saying an FFI function accepts bool
when the other side thinks it accepts a u8
), or where there are differences in notions of validity across languages.
A subtle case of this comes up occasionally in FFI code due to differences in expectations between how enums are used in Rust and C.
In C, it is common to use enums to represent bitmasks, doing something like this:
typedef enum {
Active = 0x01;
Visible = 0x02;
Updating = 0x03;
Focused = 0x04;
} NodeStatus;
where the value make take states like Active | Focused | Visible
. These combined values, as well as the "no flags set" value 0
are invalid in Rust. If this type is represented as an enum in Rust (even if it is #[repr(C)]
!), it will be UB to accept values of this type over FFI from C. Generally in such cases it is recommended to use an integer type instead, and represent the mask values as constants.
Signs an invalid value was involved
The compiler is allowed to assume that values are never invalid; and it may use invalid states to signal other things, or pack types into smaller spaces.
For example, the type Option<Box<T>>
will use the fact that the reference cannot be null to fit the entire type into the the same space Box<T>
takes up, with the null pointer state representing None
.
This can go even further with stuff like Option<Option<Option<bool>>>
fitting into a single byte, up to and including the type with 254 Option
s surrounding one bool
. This general class of optimization is known as a "niche optimization", with bits representing invalid values being called "niches".
In such scenarios, invalid values may lead to values being interpreted as a different value, for example an Option<NodeStatus>
using the enum from above would be interpreted as None
if NodeStatus
were represented as a Rust enum and an "empty status" value was received over C.
Furthermore, invalid values will break match
statements, usually (but not necessarily) leading to an abort.
Debuggers also tend to behave strangely with invalid values, displaying incorrect values, or even having the value change from read to read.
This is not an exhaustive list: ultimately, having an invalid value is UB and it remains illegal even if there are no optimizations that will break.