Session:Making RCU Safe For Battery-Powered Devices ELC2012

Jump to: navigation, search

Session Details

Embedded Linux Conference 2012
February 16, 2012
Paul E. McKenney
Free Electrons (450x800) or Free Electrons (Full HD) or Linux Foundation


Back in the 1990s, energy efficiency was not even on the list of RCU-related concerns. Systems using RCU at that time dealt with either large quantities of heavy equipment or large financial flows, so that the energy consumption of the entire server (let alone RCU's contribution) was way down in the noise. This situation changed dramatically with the 2002 introduction of RCU into the Linux kernel. Since then, RCU's energy-efficiency code has been rewritten more than five times. The most recent rewrite was motivated by workloads where RCU was leaving tens of percent of potential energy savings on the table. This talk will give a brief overview of how energy efficiency came to RCU, the current situation, future prospects, and generally applicable lessons learned.


Paul E. McKenney is a Distinguished Engineer with the IBM Linux Technology Center, where he maintains the RCU implementation within the Linux kernel. He has been coding for almost four decades, and more than half of that on parallel hardware. His prior lives include working on the DYNIX/ptx kernel at Sequent, work on packet radio, Internet protocols, and system administration at SRI International, and work on soft-realtime systems as a self-employed contract programmer. His hobbies include what passes for running at his age along with the usual house-wife-and-kids habit.



Transcribed by
Arnout Vandecappelle (0:00 - 6:59)
Verified by
1 -

0:00 - 1:00: [slide 1 - title] [some pre-presentation chatting] >> PAUL MCKENNEY: [slide 2 - Overview] I'm going to go through a little obligatory what-is-RCU. At my age, you have to have at least some slide about "The Good Old Days", as a log [?] or something, and something about that. We'll take a look at RCU's many energy-efficiency variants over the years, the current state of its energy-efficiency, and of course future directions.

[slide 3 - What is RCU?] Now how many people here have actually written code that uses RCU? Good, we got some, great! Now how many people have ever viewed code that has calls to RCU in it?

1:00 - 2:00: Okay, a few more, good. How many people have heard of RCU before today? All right, that's good. Not so long ago [?] I wouldn't have seen any hands for that last one either. [remark from the audience] Yes, there was a time when I could have said yes to the first one and no to the last one, but that was long ago. Anyway.

[slide 4 - A very brief introduction to RCU] OK, let's go through it quickly. This is just a taste of it. I'll just take a few minutes to go through it. I have a slide at the end of this section that has a list of places to go to get more information. These slides are posted somewhere, they will be on my website as well.

If you only want to hear the number one sentence about RCU, the thing to remember is that first bullet there: RCU is a synchronization technique that is sometimes used in place of reader-writer locking. The reason that it's used in place of reader-writer locking is that it has extremely low read-side overhead. When I say extremely low, I mean that if you have a real linux kernel that's built with CONFIG_PREEMPT=n - without preemption - it will be zero.

2:00 - 3:00: And I mean zero: #define rcu_read_lock() newline. So, the C preprocessor sees it, the compiler doesn't. Of course, "free" is a very good price, it gives you great performance scalability, real-time response.

The other nice thing about it, is something called hazard pointers. A guy named Maged Michael worked on this. You can have RCU readers making progress, and at the same time RCU writers making progress. So you can have a reader reading an item, while an updater is updating that same data item, and have something sensible happen. This makes for some very nice non-lockup and non-blockage techniques and properties.

The main place you want to use it is read-mostly data. I'll have a slide near the end that gives a "Go", "Wait", "Stop" kind of thing about where to use it. This is increasingly important.

3:00 - 4:00: There was a time, when I was the average age in this room, where, if you have a machine, and you wanted to plug a disk in it, you had to change lines in the source code of the kernel. Then you could bring down your system and plug your disk, and make sure you boot the kernel that has the change because otherwise it will panic coming up with the extra disk. And even worse if you take one out. Well, our machines are a bit more civilized these days, I can slide a memory stick in this thing any time I want, and it will pop it up and say "Hey, here's your disk, what do you want to do with it." What that means, is that a bunch of stuff that used to be stuck in the source code of the kernel is now data structures. Things like routing tables, security policies, storage configuration. These things almost never change. But they might change at any time. That's the sort of thing that RCU works really well for.

4:00 - 5:00: I will have a couple of slides showing specific operations for it. One thing is - I said up there that the readers can make progress even while you're changing data items, so, clearly you have to be able to take the thing and insert new data while the reader is going through the thing and have it all work out correctly. We'll look at that. Even more exciting is when you want to remove some data while a reader is charging [?] through your list or your data structure, and have that work out right. We'll take a look at how that works. Given that the readers aren't telling us anything - remember that you can have zero code for the reader - we need a way for the writers to figure out whether the readers are there or not, so they can do things without upsetting readers. We'll have a slide on that as well.

[slide 5 - Publication of And Subscription To New Data] This slide is talking about when you want to add new data to your data structure and present it to the readers. On this slide and the next slide, if you have something that's red, it means it's dangerous for updaters. Readers could be looking at it, new readers can show up at any time, we can't stop them, we don't even know they're there. So we have to be very careful for the things that are marked red.

Yellow (not on this slide but it will be on the next slide) is dangerous. There can't be any new readers getting to that piece of data, but there might be old ones still there.

5:00 - 6:00: And finally green is where the updater has full control over it, it can do what it wants.

We have time moving from left to right on this diagram. We're just showing the state of the system, with these arrows meaning a change. We have the global pointer red: any reader at any time can just go to this variable and grab it and does something. We don't know they're there, we don't know when they're showing up or when they're leaving, we can't stop them. What we want to do, is to get to this final state, where the global pointer is pointing to this data structure that's initialized in the way we want to. We want the readers to actually see the data structure in this state.

We take three steps. The first thing we do is we allocate the new data. We get this block of memory, we get a pointer to it, but readers can't get to it - there's no path to it. This is uninitialized, it's just garbage memory that kmalloc or whatever gave us. The next thing we do is, we initialize it.

6:00 - 7:00: Now it has some sensible data in it. Readers still can't reach to it, there's no way to get there. Once we're there, we use this primitive, rcu_assign_pointer(), which is essentially a glorified assignment statement. It's an assignment statement that does the change, including some architecture-specific instructions, needed to make sure that the readers see this [the new data structure] and not this [the old data]. If you don't use this special function but just use an equal sign - gptr = tmp, what can happen is that both the compiler and the CPU can change the order of things, which means these readers coming in might see garbage instead of the nice initialized stuff you want.

The readers have their own special sauce for their assignment statement. Instead of just saying "give me a copy", like localvar = gptr, they have to use rcu_dereference(), again, to prevent the compiler and in some cases the CPU from changing the code around and causing things to show up out of order.

7:00 - 8:00:

8:00 - 9:00:

9:00 - 10:00:

10:00 - 11:00:

11:00 - 12:00:

12:00 - 13:00:

13:00 - 14:00:

14:00 - 15:00:

15:00 - 16:00:

16:00 - 17:00:

17:00 - 18:00:

18:00 - 19:00:

19:00 - 20:00:

20:00 - 21:00:

21:00 - 22:00:

22:00 - 23:00:

23:00 - 24:00:

24:00 - 25:00:

25:00 - 26:00:

26:00 - 27:00:

27:00 - 28:00:

28:00 - 29:00:

29:00 - 30:00:

30:00 - 31:00:

31:00 - 32:00:

32:00 - 33:00:

33:00 - 34:00:

34:00 - 35:00:

35:00 - 36:00:

36:00 - 37:00:

37:00 - 38:00:

38:00 - 39:00:

39:00 - 40:00:

40:00 - 41:00:

41:00 - 42:00:

42:00 - 43:00:

43:00 - 44:00:

44:00 - 45:00:

45:00 - 46:00:

46:00 - 47:00:

47:00 - 48:00:

48:00 - 49:00:

49:00 - 50:00:

50:00 - 51:00:

51:00 - 52:00:

52:00 - 53:00: