1 PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()
3 Most of the time, you can use values from rcu_dereference() or one of
4 the similar primitives without worries. Dereferencing (prefix "*"),
5 field selection ("->"), assignment ("="), address-of ("&"), addition and
6 subtraction of constants, and casts all work quite naturally and safely.
8 It is nevertheless possible to get into trouble with other operations.
9 Follow these rules to keep your RCU code working properly:
11 o You must use one of the rcu_dereference() family of primitives
12 to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU
13 will complain. Worse yet, your code can see random memory-corruption
14 bugs due to games that compilers and DEC Alpha can play.
15 Without one of the rcu_dereference() primitives, compilers
16 can reload the value, and won't your code have fun with two
17 different values for a single pointer! Without rcu_dereference(),
18 DEC Alpha can load a pointer, dereference that pointer, and
19 return data preceding initialization that preceded the store of
22 In addition, the volatile cast in rcu_dereference() prevents the
23 compiler from deducing the resulting pointer value. Please see
24 the section entitled "EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH"
25 for an example where the compiler can in fact deduce the exact
26 value of the pointer, and thus cause misordering.
28 o Avoid cancellation when using the "+" and "-" infix arithmetic
29 operators. For example, for a given variable "x", avoid
30 "(x-x)". There are similar arithmetic pitfalls from other
31 arithmetic operators, such as "(x*0)", "(x/(x+1))" or "(x%1)".
32 The compiler is within its rights to substitute zero for all of
33 these expressions, so that subsequent accesses no longer depend
34 on the rcu_dereference(), again possibly resulting in bugs due
37 Of course, if "p" is a pointer from rcu_dereference(), and "a"
38 and "b" are integers that happen to be equal, the expression
39 "p+a-b" is safe because its value still necessarily depends on
40 the rcu_dereference(), thus maintaining proper ordering.
42 o Avoid all-zero operands to the bitwise "&" operator, and
43 similarly avoid all-ones operands to the bitwise "|" operator.
44 If the compiler is able to deduce the value of such operands,
45 it is within its rights to substitute the corresponding constant
46 for the bitwise operation. Once again, this causes subsequent
47 accesses to no longer depend on the rcu_dereference(), causing
48 bugs due to misordering.
50 Please note that single-bit operands to bitwise "&" can also
51 be dangerous. At this point, the compiler knows that the
52 resulting value can only take on one of two possible values.
53 Therefore, a very small amount of additional information will
54 allow the compiler to deduce the exact value, which again can
55 result in misordering.
57 o If you are using RCU to protect JITed functions, so that the
58 "()" function-invocation operator is applied to a value obtained
59 (directly or indirectly) from rcu_dereference(), you may need to
60 interact directly with the hardware to flush instruction caches.
61 This issue arises on some systems when a newly JITed function is
62 using the same memory that was used by an earlier JITed function.
64 o Do not use the results from the boolean "&&" and "||" when
65 dereferencing. For example, the following (rather improbable)
73 p = rcu_dereference(gp)
75 q += p != &oom_p1 && p != &oom_p2;
76 r1 = *q; /* BUGGY!!! */
78 The reason this is buggy is that "&&" and "||" are often compiled
79 using branches. While weak-memory machines such as ARM or PowerPC
80 do order stores after such branches, they can speculate loads,
81 which can result in misordering bugs.
83 o Do not use the results from relational operators ("==", "!=",
84 ">", ">=", "<", or "<=") when dereferencing. For example,
85 the following (quite strange) code is buggy:
92 p = rcu_dereference(gp)
95 r1 = *q; /* BUGGY!!! */
97 As before, the reason this is buggy is that relational operators
98 are often compiled using branches. And as before, although
99 weak-memory machines such as ARM or PowerPC do order stores
100 after such branches, but can speculate loads, which can again
101 result in misordering bugs.
103 o Be very careful about comparing pointers obtained from
104 rcu_dereference() against non-NULL values. As Linus Torvalds
105 explained, if the two pointers are equal, the compiler could
106 substitute the pointer you are comparing against for the pointer
107 obtained from rcu_dereference(). For example:
109 p = rcu_dereference(gp);
110 if (p == &default_struct)
113 Because the compiler now knows that the value of "p" is exactly
114 the address of the variable "default_struct", it is free to
115 transform this code into the following:
117 p = rcu_dereference(gp);
118 if (p == &default_struct)
119 do_default(default_struct.a);
121 On ARM and Power hardware, the load from "default_struct.a"
122 can now be speculated, such that it might happen before the
123 rcu_dereference(). This could result in bugs due to misordering.
125 However, comparisons are OK in the following cases:
127 o The comparison was against the NULL pointer. If the
128 compiler knows that the pointer is NULL, you had better
129 not be dereferencing it anyway. If the comparison is
130 non-equal, the compiler is none the wiser. Therefore,
131 it is safe to compare pointers from rcu_dereference()
132 against NULL pointers.
134 o The pointer is never dereferenced after being compared.
135 Since there are no subsequent dereferences, the compiler
136 cannot use anything it learned from the comparison
137 to reorder the non-existent subsequent dereferences.
138 This sort of comparison occurs frequently when scanning
139 RCU-protected circular linked lists.
141 Note that if checks for being within an RCU read-side
142 critical section are not required and the pointer is never
143 dereferenced, rcu_access_pointer() should be used in place
144 of rcu_dereference(). The rcu_access_pointer() primitive
145 does not require an enclosing read-side critical section,
146 and also omits the smp_read_barrier_depends() included in
147 rcu_dereference(), which in turn should provide a small
148 performance gain in some CPUs (e.g., the DEC Alpha).
150 o The comparison is against a pointer that references memory
151 that was initialized "a long time ago." The reason
152 this is safe is that even if misordering occurs, the
153 misordering will not affect the accesses that follow
154 the comparison. So exactly how long ago is "a long
155 time ago"? Here are some possibilities:
161 o Module-init time for module code.
163 o Prior to kthread creation for kthread code.
165 o During some prior acquisition of the lock that
168 o Before mod_timer() time for a timer handler.
170 There are many other possibilities involving the Linux
171 kernel's wide array of primitives that cause code to
172 be invoked at a later time.
174 o The pointer being compared against also came from
175 rcu_dereference(). In this case, both pointers depend
176 on one rcu_dereference() or another, so you get proper
179 That said, this situation can make certain RCU usage
180 bugs more likely to happen. Which can be a good thing,
181 at least if they happen during testing. An example
182 of such an RCU usage bug is shown in the section titled
183 "EXAMPLE OF AMPLIFIED RCU-USAGE BUG".
185 o All of the accesses following the comparison are stores,
186 so that a control dependency preserves the needed ordering.
187 That said, it is easy to get control dependencies wrong.
188 Please see the "CONTROL DEPENDENCIES" section of
189 Documentation/memory-barriers.txt for more details.
191 o The pointers are not equal -and- the compiler does
192 not have enough information to deduce the value of the
193 pointer. Note that the volatile cast in rcu_dereference()
194 will normally prevent the compiler from knowing too much.
196 However, please note that if the compiler knows that the
197 pointer takes on only one of two values, a not-equal
198 comparison will provide exactly the information that the
199 compiler needs to deduce the value of the pointer.
201 o Disable any value-speculation optimizations that your compiler
202 might provide, especially if you are making use of feedback-based
203 optimizations that take data collected from prior runs. Such
204 value-speculation optimizations reorder operations by design.
206 There is one exception to this rule: Value-speculation
207 optimizations that leverage the branch-prediction hardware are
208 safe on strongly ordered systems (such as x86), but not on weakly
209 ordered systems (such as ARM or Power). Choose your compiler
210 command-line options wisely!
213 EXAMPLE OF AMPLIFIED RCU-USAGE BUG
215 Because updaters can run concurrently with RCU readers, RCU readers can
216 see stale and/or inconsistent values. If RCU readers need fresh or
217 consistent values, which they sometimes do, they need to take proper
218 precautions. To see this, consider the following code fragment:
235 p->a = 42; /* Each field in its own cache line. */
238 rcu_assign_pointer(gp1, p);
241 rcu_assign_pointer(gp2, p);
250 p = rcu_dereference(gp2);
253 r1 = p->b; /* Guaranteed to get 143. */
254 q = rcu_dereference(gp1); /* Guaranteed non-NULL. */
256 /* The compiler decides that q->c is same as p->c. */
257 r2 = p->c; /* Could get 44 on weakly order system. */
259 do_something_with(r1, r2);
262 You might be surprised that the outcome (r1 == 143 && r2 == 44) is possible,
263 but you should not be. After all, the updater might have been invoked
264 a second time between the time reader() loaded into "r1" and the time
265 that it loaded into "r2". The fact that this same result can occur due
266 to some reordering from the compiler and CPUs is beside the point.
268 But suppose that the reader needs a consistent view?
270 Then one approach is to use locking, for example, as follows:
289 p->a = 42; /* Each field in its own cache line. */
292 spin_unlock(&p->lock);
293 rcu_assign_pointer(gp1, p);
297 spin_unlock(&p->lock);
298 rcu_assign_pointer(gp2, p);
307 p = rcu_dereference(gp2);
311 r1 = p->b; /* Guaranteed to get 143. */
312 q = rcu_dereference(gp1); /* Guaranteed non-NULL. */
314 /* The compiler decides that q->c is same as p->c. */
315 r2 = p->c; /* Locking guarantees r2 == 144. */
317 spin_unlock(&p->lock);
318 do_something_with(r1, r2);
321 As always, use the right tool for the job!
324 EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH
326 If a pointer obtained from rcu_dereference() compares not-equal to some
327 other pointer, the compiler normally has no clue what the value of the
328 first pointer might be. This lack of knowledge prevents the compiler
329 from carrying out optimizations that otherwise might destroy the ordering
330 guarantees that RCU depends on. And the volatile cast in rcu_dereference()
331 should prevent the compiler from guessing the value.
333 But without rcu_dereference(), the compiler knows more than you might
334 expect. Consider the following code fragment:
340 static struct foo variable1;
341 static struct foo variable2;
342 static struct foo *gp = &variable1;
346 initialize_foo(&variable2);
347 rcu_assign_pointer(gp, &variable2);
349 * The above is the only store to gp in this translation unit,
350 * and the address of gp is not exported in any way.
361 return p->a; /* Must be variable1.a. */
363 return p->b; /* Must be variable2.b. */
366 Because the compiler can see all stores to "gp", it knows that the only
367 possible values of "gp" are "variable1" on the one hand and "variable2"
368 on the other. The comparison in reader() therefore tells the compiler
369 the exact value of "p" even in the not-equals case. This allows the
370 compiler to make the return values independent of the load from "gp",
371 in turn destroying the ordering between this load and the loads of the
372 return values. This can result in "p->b" returning pre-initialization
375 In short, rcu_dereference() is -not- optional when you are going to
376 dereference the resulting pointer.