| Date: | February 22, 2006 / year-entry #67 |
| Tags: | code |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20060222-11/?p=32193 |
| Comments: | 48 |
| Summary: | The registry is a convenient place to record persistent cross-process data in a uniform and multi-thread-safe manner. It roams with the user if you store it in HKEY_CURRENT_USER, and individual keys can be secured (even on systems that use FAT, which doesn't otherwise support security). But that doesn't mean that it's free. The cost of... |
The registry is a convenient place to record persistent cross-process data in a uniform and multi-thread-safe manner. It roams with the user if you store it in But that doesn't mean that it's free. The cost of opening a key, reading a value, and closing it is around 60,000 to 100,000 cycles (I'm told). And that's assuming the key you're looking for is in the cache. If you open the key and hold it open, then the act of reading a value costs around 15,000 to 20,000 cycles. (These numbers are estimates for Windows XP; actual mileage may vary.) Consequently, you shouldn't be reading a registry key in your inner loop. Not only does it cost you CPU time at query time, but the constant hammering of the registry means that the data structures used by the registry to locate and store your key (including the entry in the registry cache) are kept in the system working set. Don't read a registry key on every mouse move; read the value once and cache the result. If you need to worry about somebody changing the value while your program is running, you can establish a protocol for people to follow when they want to change a setting. Windows, for example, uses functions such as Whenever possible, optimize for the common case, not the rare case. The common case is that the registry value hasn't changed. By using a notification mechanism, you move the cost of "But what if the value changed?" out of your inner loop and into code that doesn't execute most of the time. (Remember, the fastest code is code that never runs.) Of course, you don't want to burn a thread waiting on the notification event. I use the thread pool. The One caveat to heed with the |
Comments (48)
Comments are closed. |
"Calling the update function" Presumably you mean calling SystemParametersInfo with the fWinIni set to an appropriate value?
Was there ever any publicly released document on why MS chose to move to using a registry anyway?
I remember back in the pre-registry days, us UNIX folks were mocked because we stored configuration info in ASCII text files.
But in the end it turns out that storing things in text files is probably faster, just as secure, and a lot easier to manage.
Are there actually programs that access the registry upon every mouse move?
Given the nature of the registry (a place to store configuration variables), the cost of reading values has a low impact, since a program would only read it’s values when it starts, and only write values when the user changes the configuration. I think it was a well-known fact that using the registry in an application is costly. Didn’t all win>=32 developers knew already? Hmm.
The Registry tends to be a lot more friendly to programmatic alteration than flat text files. For instance, if I want to alter the configuration of something, I can just set a string or dword value in the registry in the vast majority of cases instead of having to write a configuration file parser and writer for that particular program’s config file format.
Yes, more standard formats like .ini files can help alieviate this, but then you still don’t necessarily have a centralized location to say find where a program is installled or alter it’s settings regardless of where the actual binary happened to end up on the end user system.
I would suspect that a linear search through a text file is also probably going to be much slower than the much more optimized hive data structures in memory.
Yes, Skywing I was writing that but I accidentally closed Firefox. However, I still have something to add. When your program is a self-serving EXE file that can be dragged around by users, the registry is quite a useful tool. However, when there’s a lot to store then it’s better to use a file because registry performance degrades when it grows. Thus, all registry-enabled programs would slow. Wether the user actually notices it is another, ofcourse.
When using a file, make sure the performance is consistant, as Skywing pointed out. Make sure it’ll be compatible with future (and then older) versions, that it uses the right character set, and so on.
When searching, you don’t have to search linearly, and you don’t have to use the filesystem API in the process either. And you get to optimize for the actual data type(s) being stored.
"Are there actually programs that access the registry upon every mouse move?"
Error: Result set too large to display. :)
Get a copy of SysInternals RegMon and start it with nothing filtered out. It’s incredible how much registry banging happens on most systems, even when you’re not physically interacting with the system.
RC, you’ve also written recently about MsgWaitForMultipleObjects, which has become my new favorite API. And it happens that I’m using it in concert with RegNotifyChangeKeyValue. I suppose deciding which strategy is "better" is an application-level trade-off, and RegisterWaitForSingleObject might be useful in a program without a message pump, but it still strikes me as unnecessarily complex. In fact, this whole notion of spawning a thread just so it can block on something feels wrong to me, which is why I find MsgWaitForMultipleObjects so appealing. Can you elaborate?
Dave, how about a port of XCruise to the registry (and libSDL)?
http://xcruiser.sourceforge.net/
Grab regmon from http://www.sysinternals.com and be horrified at just how frequently the registry is polled by various programs. My favourite is how often Yahoo Messenger looks for new versions of Flash – about once a second. The little talking heads are all coded in Flash.
To me, it seems that Microsoft does not practice what Raymond is preaching. Go get regmon, filter for just "explorer.exe", and go browse your Start Menu.
On Windows XP, I browsed Start -> Programs -> Microsoft Visual Studio .NET 2003 and then hovered over the link to start up Visual Studio. All told, about 1000 registry accesses occurred during this process. By Raymond’s given values, that means that on the order of 60 million to 100 million cycles were just spent so that I could browse the start menu.
I’ll grant you, this isn’t exactly in a critical section. Still, it seems pretty overkill when you consider that this is data that should not be changing that often.
Andrew/Andrea,
Huh?
1) OK, maybe, but you can do this with regedit.
2) It’s easy enough to fix the format with notepad. Change to/from Unicode, edit the first line.
3) You mean simultaneously use the same config on multiple machines? Roaming profiles. Like Raymond said.
4) You mean share the same config for all users? Why would you want to do that? You can accomplish this by storing it in HKLM and changing the permissions, I guess – the same way you’d have to change permissions for a writable central config file on the machine.
5) And you’d have to mess about with permissions for a writable simple config file too.
6) It’s easy: HKCUSoftwareManufacturerProgramNamewhatever
7) How so? I can see more scope for security errors parsing a config file than using the registry API.
Skywing said:
> I would suspect that a linear search
> through a text file is also probably
> going to be much slower than the much
> more optimized hive data structures
> in memory.
Yes, but suspicions are not the proper way to design an OS.
The nice thing about UNIX/Linux is that there’s often open discussions, benchmarks, and even papers written about performance and security decisions.
With Windows, unless Raymond or someone similar posts it to their blogs, you have no idea why anything is done, and often you can just assume that it was done for fairly arbitrary reasons.
Re: thread affinity:
There are two ways around this.
If you’re going to do only a little work based on the handle being signalled, it’s no big deal since what happens is that the event is signalled when the queuing thread terminates (I think it’s an IRP that gets queued when you request the notification so when the queuing thread is destroyed, all the outstanding IRPs are cancelled and the completion routines will signal the event).
If you’re going to do a lot of work, you want to use a persistent thread in the thread pool to queue the original notification and then register the wait.
The latter is overkill if you’re just going to read one value from an already open key handle but if you’re going to tear down some big configuration data structure and rebuild it and you’re not in control of the thread lifetimes on which your code runs (e.g. a typical component) you should queue the work item to a persistent thread to start the wait.
"In fact, this whole notion of spawning a thread just so it can block on something feels wrong to me…"
Why so? It’s conceptually easy to have a thread with a single-minded goal blocking until it can achieve that goal. Contrast that with dispatching off a message loop or one thread waiting for a variety of events. Now of course it gets more complicated if the thread needs to synchronize with something once the goal is achieved, sometimes you get that for free with the other approaches.
stb:
I have a 3.4ghz processor – that’s 3,400,000,000 cycles per second (not counting hyperthreading). So at 100,000 cycles per registry read, that’s still 34,000 reads a second.
So Explorer reading 1000 values for something the user initiates (say by hovering the mouse over something in the start menu) isn’t really that big a deal, is it?
You said:
When the event fires on the thread pool, handle the change, then ask your persistent thread to start a new cycle of RegNotifyChangeKeyValue.
You should do:
When the event fires on the thread pool, ask your persistent thread to start a new cycle of RegNotifyChangeKeyValue, then handle the change. Otherwise you have a race condition where a registry change occurs after you handle the change but before you RegNotifyChangeKeyValue.
jon:
As I said, I agree that this is not a critical section. That doesn’t change the fact that this seems hugely wasteful.
On the one hand, it’s easy to discount it by saying that yes, many modern systems could do this sort of read many times over easily. On the other hand, I find it hard to believe that this information couldn’t be held onto longer by explorer.exe, instead of requiring that it re-read it constantly.
Also, remember that not everyone has as nice a computer as you do. This is something that developers (myself included) tend to lose sight of. If we were talking about a game, then I’d say the developers can assume whatever level of performance they want, but this is a fairly simple operation.
To demonstrate that point, a question: were you on a 800MHz machine, which would be capable of only 8000 reads per second, would you hold the same opinion? Personally, I would think simple menu navigation taking up 1/8 of the CPU resources seems more than just a little bit overkill.
It seems reasonable to think that this kind of thinking ("it’s only 1000 reads …") is exactly why each version of Windows needs more resources than the last. Little bits add up.
Dave, if a thread is going to do real work that the user cares about, that is a thread worth creating, and the blocking is merely incidental to getting its job done. But when the only motive for creating a thread is that the programmer encountered an API which blocks, and the only work for that thread to do once that API unblocks is to post a message to the main thread, something has gone wrong from a design perspective, because the programmer wanted the API, not the thread. It seems to me that non-abusive use of RegNotifyChangeKeyValue is likely to fit into the latter scenario more often than the former.
"Are there actually programs that access the registry upon every mouse move?"
Yes, Microsoft Mouse (point32.exe) does. Once again, take RegMon and you will see how bad it hits the registry (with every single mouse movement).
"7) How so? I can see more scope for security errors parsing a config file than using the registry API."
When opening a file without write sharing, no other process can "break your file" during parsing.
With registry, there is no global lock.
There is only a lock operation for each read/write operation.
http://www.oreilly.com/catalog/winreg/chapter/ch04ex.html
"Multiple operations, thus, are not atomic. For instance, if your application uses RegEnumValue to enumerate all the values belonging to a particular key, a mutex is used to insure the integrity of the individual read operation. But the integrity of the enumeration as a whole cannot be guaranteed; that is, there is no guarantee that the properties of the value entries and the values themselves will be the same when you finish your enumeration as they were when you began your enumeration. In fact, since the registry’s locking scheme is implemented at a system level by the Virtual Machine Manager, it is not possible to apply it to multiple operations, since the system has no way of knowing how many operations you intend to perform. Attempting to apply it to multiple operations would inevitably block other applications’ access to the registry and bring the system to a crawl. Consequently, the documentation for RegEnumKey, RegEnumKeyEx, and RegEnumValue in the Win32 SDK warns that, while enumerating keys or values, your application should not modify any of the objects being enumerated.
So it is quite possible, when you’re in the middle of a series of registry operations, that some other application can change either the data itself or information about the data (like the number of value entries belonging to a subkey or the number of bytes in the longest data value). For instance, before using RegEnumValue to enumerate a series of values, you call RegQueryInfoKey to determine both the number of value entries and the length of the longest data item. As you’re enumerating the values, another application deletes a value that you haven’t yet enumerated, and writes new data to another value entry that you haven’t enumerated that is significantly longer than RegQueryInfoKey reported."
"I would suspect that a linear search through a text file is also probably going to be much slower than the much more optimized hive data structures in memory."
Depends on the type of information you need.
Often, the program just needs to read a big list of settings, and often the whole list is read at once.
So, with a text file, it is quite easy and fast, all the data is read at a time.
If there is not a fixed number of settings, but an association between some keys and settings, the file works well, but the registry sucks… because with the registry, you must use a RegEnumKeyEx function call, which is very slow, because the registry was not designed to be used like that.
The registry can only be used efficiently with direct accesses (enumeration is slow).
A wondeful example of total misuse of the registry, is the "new" submenu of the context menu of the background of shell folders.
I didn’t read the Windows code, but under Windows 98, getting access to this submenu is very very slow.
It is very probably due to an enumeration of all keys under HKCR, and research of each ShellNew key under each file extension key.
It needs about 1.5 second on my computer (and that, for only 9 menu items)
There is no caching mechanism.
My computer is a K6-2 550Mhz, with a 100Mhz motherboard, and 192MB RAM.
Much better than common computers that were used in 1998.
1.5 second, it is 825 millions CPU cycles!
Clearly, the ShellNew information is not stored in a consistent way in the registry base.
And the absence of caching system is inacceptable.
Imagine my CPU ran 55Mhz instead of 550Mhz (Windows 95 ran on 486 DX2 66), it would need at least 15 seconds!
And, it needed 2.5 seconds, last month, before I upgraded my 333Mhz CPU to a 550Mhz one.
Even that 1.5 second delay I use alternative ways for creating files (such as using DOS prompt), because that’s just too slow.
PS: That is an incorrect way to use the registry base, but I don’t mean that every use of the registry base is bad.
And, there are some things that are better done with config files, as there are things that are well done with the registry base.
There’re very few annoying things like a program storing configs in the registry.
Possibly, there’s *nothing* worse than that, even getting a virus is maybe better – as long as it doesn’t write in the registry.
This is because registry-oriented persistence is a real pain in the ace under several points of view:
1) It makes a case for an import/export configuration feature(which isn’t mostly provided)
2) Makes program backup much harder, because if there’s no configuration import/export you have to copy the keys and the values, but mind what format you choose, because you may end up being incompatible with Windows 9x/ME/Godknowswhat
3) Sharing configurations among machines is plain impossible
4) Sharing configurations for users on the same machine(!!!!!) is much harder
5) You have to mess with permissions about the keys
6) Choosing the right key to store the right data takes time out of real work
7) Registry is more "vulnerable" than simple configuration files
Raymond?
Cheers,
Andrew
Andrew = to signal I’m male
Andrea = real name :-)
Regedit has to be done *manually*… do it for several machines ;-)
Same for fixing… it’s ok for one machine, how about tens or hundreds?
No, I mean that you can have some basic file on a shared network with some sensitive settings already filled being lots easier than having to do Godknowswhat to achieve the same.
Not really the same as file, if you keep your configuration file in the shared documents folder, for instance. About this aspect, it’s
interesting to notice that said config file could possibly be encrypted, loaded in memory, being decripted, settings loaded then unloaded from memory. Do this with registry settings.
About HKCU: not quite. You just said you’d need to load it from HKLM if you want to share among users… now what? ;-) Using a file, all that changes is a PATH. That’s it.
About vulnerability: try modifying a value in the registry and then do the same in a config file… which is easier to correct? Consider that the key tree might be complex and have several similar branches… using a simple configuration file, ini for example, you can comment sections once and forget about them.
Cheers,
Andrew
I noticed a while back that one version of Valve’s Steam client would do tens of registry reads per second – all the time.
I noticed because I had regmon open debugging one of our own apps, and I had to filter out Steam’s rampant registry abuse just to see what was going on.
I believe Valve have fixed the problem now. But this was done by the supposedly lightweight tray app, and happened *all* the time. Then again, Steam falls into the ‘pointlessly skinned interface’ class of apps too, so I shouldn’t be that surprised.
"Regedit has to be done *manually*… do it for several machines ;-)"
reg.exe is your friend
"Same for fixing… it’s ok for one machine, how about tens or hundreds? "
Group Policy is your other friend
Used properly, the registry is great. The problem is the large number of applications which don’t use it correctly. Personally, I’d have designed it differently, but that certainly wouldn’t be a bunch of text files.
A problem (I find) with linux text config files is the numerous formats that abound. Win16 had an advantage with its common INI file format.
When # of PCs==small, it *is* easier to hack a text file quickly than fire up RegEdit and locate the key and then change the value.
However as # of PCs approaches LotsAndLots the Registry and group policy etc starts looking very attractive.
"A problem (I find) with linux text config files is the numerous formats that abound. Win16 had an advantage with its common INI file format."
The best solution is IMHO the way MacOS X does it, with plist files, which are basically XML files (although they can be stored in a binary format too, but those can be converted to and from the XML file).
"A problem (I find) with linux text config files is the numerous formats that abound. Win16 had an advantage with its common INI file format."
steveg: Right, it would be good to normalize the format for new softwares.
In particular, I wonder why some files use ‘!’ for comments while others use ‘#’.
Also, I think that the registry should be mainly used for shared settings.
Settings used by more than one program.
It makes no sense to have two independent programs, with independent settings be blocked on the mutex of the registry base… It looks like a cooperative multitasking.
Also, something I like with Linux config files, is the very good documentation that comes with, and the comments!
The registry base never contains comments.
Programs that use registry base, generally are not designed to allow the user modifying settings manually (and settings change from one version to another).
Linux config files are designed to be modified by the user, and remains compatible between releases of the same program.
Of course, that is not inherent to the registry base, but the fact that programs use the registry base in an obscure way, make it a big bunch of undocumented settings which are not correctly import/exportable from one machine to another.
…I’d like to add that some programs use a scripting language for the config file (usually because they’ve got scripting support anyway), so you can "code" the config files, like creating macro’s that evaluate variables and act accordingly. Uber-geeky stuff :)
One prime example is ofcourse the bash config file (~/.bashrc and /etc/bash.bashrc), which is basically just another bash script :)
"Regedit has to be done *manually*…"
regedit /e file key
If you want to be able to configure a registry key on multiple machines, you can create an .ADM file for Group Policy. You can decide to do it on a per-user basis if you use the User Configuration part rather than the System Configuration part.
Your application should always check HKCU as well as HKLM, preferring the settings from HKCU if set. I personally wish that the URL protocol handling code in IE and the shell would check HKCU as well as HKLM so that RSS Bandit didn’t have to write the feed: protocol keys into HKLM, which obviously (unless you weaken the ACLs) must be done as an administrator or power user.
"Was there ever any publicly released document on why MS chose to move to using a registry anyway?"
I think Eric or Raymond has addressed this before, but the old .ini file structure was horrible, with clashes when two programs tried to update their own settings in win.ini at the same time, and the growing size of the file.
Maybe someone will remind us what the reasons for going to the registry were. Someone also commented that if you want to abolish the registry, you have to replace it with something.
PingBack from http://techies.teamlupus.hu/2006/02/23/registry-kezeles/
When you say ‘the thread pool’ do you mean some windows service/api or something I have to write myself?
"So is Raymond being a hypocrite, or was msmouse coded by an interim?"
A) What makes you think that Raymond coded msmouse?
B) Even if he did, why couldn’t he learn from a mistake and post what not to do? Jackass.
Legolas — Windows 2000 introduced a thread pool API for applications to use, instead of having to write their own thread pool. See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/thread_pooling.asp.
Polling is bad, but reading on demand may not.
I can not imagine when everybody decides to listen to registry change notification. Will the system be able to handle that?
J,
Remember that Microsoft is a single entity, like the Borg. Raymond is Microsoft. I am Microsoft. "We" all coded all applications release by Microsoft and are therefore all responsible for all misbehavior in all Microsoft application and any version of any Microsoft O/S. Any criticism of coding practices that are later revealed to exist somewhere in the BILLIONS of lines of Microsoft code qualifies as hypocrisy.
Geez, get with the program dude!
Why does ActiveSync hammer the registry?
PingBack from http://www.janus-net.de/2006/02/27/avoid-polling-registry-keys-in-net-c/
PingBack from http://blogs.msdn.com/oldnewthing/archive/2007/11/26/6523907.aspx
PingBack from http://www.nikvoronin.com/2007/11/27/pochemu-ini-fajly-ustareli-a-reestr-net/
As we worked towards the recent release of Internet Explorer 8 Beta 1, the IE team focused hard on performance.
PingBack from http://www.b-spot.se/designing-for-add-on-performance/
  최근 Internet Explorer 8 Beta 1 출시를 위해 개발중인 IE 팀은 성능에 심혈을 기울이고 있습니다. IE  향상을 위한 노력의 일환으로 실행한
PingBack from http://blogs.msdn.com/oldnewthing/archive/2009/03/26/9508968.aspx