Windows - hotfixing overview

There is a detailed explanation over here, but I wanted to give a quick run-down of the patching process we use for hotfixes, and the main difference between a collection of hotfixes and a service pack.

Most users will just use Windows Update to let the OS determine which critical updates it needs to download and install so never see inside the packages, and some that do have asked why there are different versions of the same file(s) inside.

If you take a hotfix executable and run it with the /x argument then you can extract the files manually to a temporary location.
Common to all hotfixes should be the "update" folder (though the contents will vary) and the uninstallation files at the root, but there are folders created for each variant of the OS in both "GDR" and "QFE" flavours - what are they, and which one gets applied?

GDR stands for "General Distribution Release", which "QFE" is "Quick Fix Engineering".
At any given time, your OS is on one of these branches - defaulting to GDR - and when the hotfix is installed the system is checked for OS, SP level and branch so it knows it has the correct version to apply.

When Windows versions are first released they are usually RTM ("Release To Manufacturing"), and as service packs are applied they become SP1, SP2, etc.

Immediately after installing Windows or a service pack, you are on the GDR branch.
If you only install security-related updates (e.g. via Windows Update) then you remain on the GDR branch.
The moment you install a non security-related update (e.g. a manual download of a hotfix mentioned in a KB) then you move onto the QFE branch and remain there until you apply a service pack.


Why the 2 branches?
GDR updates, being security-related, do not contain code changes for other issues such as adding functionality via a registry value or changing a default setting.
QFE updates contain both types of code change.

This is to ensure that companies have the option to apply security patches and be certain there are no functional differences to how the system behaves by default (unless that is the specific purpose of the update).
A lot of companies have strict change control procedures, and we always recommend that patches are tested in a development environment before putting into production.


So why include all versions of the hotfix?
You may have seen how the KBs "for IT professionals" have many links for the different versions of the hotfix based on the target OS - if we relied on people knowing which SP level their machines were at as well as the correct base OS version then this list would be larger still and prone to more user errors.

(I have seen people that complain their patch did not install because they managed to download the x64 version instead of the x86 version, for example.)

This process is applicable for every binary file included in Windows, which means that there are also multiple branches in the source code for a single OS version.

This means that when you see a report that "foo.dll" in SP2 of Windows XXX has a security vulnerability, the following source code branches need checking for the vulnerability and patched and tested independently:
- Windows XXX - SP2 - GDR
- Windows XXX - SP2 - QFE
- Windows XXX - SP1 - GDR
- Windows XXX - SP1 - QFE
- Windows XXX - RTM - GDR
- Windows XXX - RTM - QFE

If foo.dll is a component in Windows YYY and Windows ZZZ also, then the RTM and SPx versions of each of these need GDR and QFE update checks too.

Next time you see an update which applies to Windows 2000, XP and 2003, bear in mind that (as of today) this would be: 10 branches for 2000 (RTM & SP1-SP4), 6 for XP (RTM, SP1, SP2) and 4 for 2003 (RTM & SP1).
That means 20 separate code changes which need regression and stress testing in every language, if you only consider the 32-bit flavours - we also have the IA64 and x64 versions of XP and 2003.


Example
Alf and Bert both have Windows XXX RTM installed with no updates applied.

The following updates have been made public for foo.exe in Windows XXX:
- KB123456: security patch, widget overflow leads to system instability
- KB124678: security patch, malformed widgets lead to remote code execution potential
- KB125983: non-security patch, registry key added to control default widget size
- KB128898: security patch, widget flood leads to denial of service

Alf goes to Windows Update, or enables Automatic Updates, and the 3 KBs 123456, 124678 and 128898 are listed for download as critical updates.
Alf installs these and has the following versions applied:
- Windows XXX - RTM - GDR 123456
- Windows XXX - RTM - GDR 124678
- Windows XXX - RTM - GDR 128898
Alf's Windows XXX remains on the GDR branch and has no ability to control the default widget size via the registry even though he applied a hotfix for foo.exe which was released after KB125983.

Bert has several servers running Windows XXX already so has manually downloaded the administrative downloads packages to apply updates before his new servers can contact the Internet.
As he has an application which prefers to use a larger default widget size, he has also obtained KB125983 and has a .reg file to import his standard setting.
Bert installs the 4 KBs listed in the order they were released, and has the following versions applied:
- Windows XXX - RTM - GDR 123456
- Windows XXX - RTM - GDR 124678
- Windows XXX - RTM - QFE 125983
- Windows XXX - RTM - QFE 128898

The difference here is that both Alf and Bert have the same in terms of security patches, but the GDR version of 128898 does not have the code change for altering the default widget size while the QFE version does.

Later, SP1 is released for Windows XXX and Alf and Bert apply it - they are now both back on the GDR branch and both have the ability to control the default widget size as all 4 code changes are rolled up into the service pack.


Uninstalling hotfixes
Care should be taken when uninstalling hotfixes, as the installation process backs up the current version of the file into the hotfix's uninstall folder and if it is removed then this is the version which is restored.

Taking Alf as an example, if before applying SP1 he had uninstalled KB124678 he would actually regress to the KB123456 QFE version and would have effectively removed the security fix from KB128898 which would still however be listed in Add/Remove Programs.

It is not often that multiple hotfixes are installed and then removed out of sequence, but sometimes people play "catch-up" with hotfixes and install dozens at a time then try removing 1 or 2 when they have issues.
(Another reason to always test patches in a development environment 1 at a time before applying to production servers.)

The files which are replaced by hotfixes are listed on the KB article on support.microsoft.com, along with the version, size & timestamp - so if in doubt you check check the version you have against these details.
(Right-click on the file, click Properties, click Version tab, select File Version)

Bear in mind that a hotfix could replace more than 1 file, where dependencies exist, and so long as you have the version listed in the KB or higher then you should be okay.
If you apply a hotfix which replaces a.dll and b.exe with version 1234 of each, then uninstall a hotfix which updated a.dll by itself and regressed the one file to version 1200 then you could run into trouble as b.exe may require a.dll version 1234 or higher.


Why the reboot after applying some hotfixes?
Current versions of Windows cannot have the kernel processes patched on the fly - the files on the disk are replaced and then loaded when the system starts up, but the version in memory is still the original one from the last restart.

Due to dependencies of services on other services, any many dependencies on key core components it is not so simple to say "unload the module from memory, update the version on disk and then reload the module".

Imagine an update for TCPIP.SYS which provides the interface for ALL TCP/IP connectivity and services - how would your Server service, IIS, DNS Server service, DHCP Server service, etc. cope with that? They would need to be restarted also - then there is the problem of not knowing how (or when it is safe) to stop & restart 3rd party applications or services relying on this module.


This is something which is changing in Windows Vista/Longhorn - the patching of some modules will be possible on a running system without a restart being required.
This is achieved by deliberately inserting some instructions which do "nothing" at key points in the compiled code, and can be replaced by instructions to jump to a different point in memory to execute the fixed version of a function.

Pseudo-code example:
original module foo.dll in memory
function bar():
0000 entry point
0001 create and initialise new widget
0002 do nothing
0003 copy widget pointed to by register into created widget
0004 do nothing
0005 modify created widget
0006 do nothing
0007 put pointer to widget in register and return

A problem is found with the "modify created widget" code, this is fixed and a new version of foo.dll is created in a hotfix.
Applying the hotfix would replace the files in the DLL cache and the main location in the Windows folder, then load the fixed portion of code into a memory area accessible by the running foo.dll module, e.g.
FF00 modify created widget fixed version
FF01 jump to 0006
Next the instruction at 0004 in the version of foo.dll in memory is changed to:
0004 jump to FF00

Now the running version of foo.dll is patched in realtime, and when the system restarts it has the "proper" version loaded from disk as normal, which contains:
0000 entry point
0001 create and initialise new widget
0002 do nothing
0003 copy widget pointed to by register into created widget
0004 do nothing
0005 modify created widget fixed version
0006 do nothing
0007 put pointer to widget in register and return

Very interesting write up Paul, i had no idea there were so many branches of code for each patch.

Quote:

---

Originally Posted by Funkstar

Very interesting write up Paul, i had no idea there were so many branches of code for each patch.

---

Agreed, TVM for posting was very interesting to read!

Watercooled falls victim to the spam thread revival... :p

3 year old thread resurrection. :P

D'oh!! Didn't realise it was that old. Oh well it was still interesting to read ;)

Crivens, that was an unexpected email notification :)

I've blogged some on how we deal with the branches and GDR vs LDR packages more recently, complete with an astonishing MS Paint job ;)

http://blogs.technet.com/mrsnrub/arc...ching-out.aspx
http://blogs.technet.com/mrsnrub/arc...e-ldr-wth.aspx
http://blogs.technet.com/mrsnrub/arc...nstallers.aspx

I was going to add a more detailed dive into the hotpatching methodology too, so I'll get round to that and stick up an entry soon.

Reported. Again.

Quote:

---

Originally Posted by kalniel

Reported. Again.

---

Thanks - he's (the spam poster) has gone! (But did unwittingly do a service in reviving an interesting thread!)

Hi Paul - LTNS , hope all is going well in the land of ridiculously fast web connectivity ;)

I lurk, therefore I am ;)

"Ridiculously" fast?
Naah, not compared to some.
http://www.speedtest.net/result/588902501.png
Does me, though :)

Currently going through the 5th Edition of Windows Internals to see if there's anything I missed with the internal training on Vista & W2K8, and seeing how Windows 7 is less than 2 weeks from GA, I'm going to take a look at updating the sticky again.

Have you already become so accustomed to Swedish superwebs that you've forgot poor little us on the British Isles crawling along at 24/1mbps tops? :lol:

Quote:

---

Originally Posted by Paul Adams

I lurk, therefore I am ;)

"Ridiculously" fast?
Naah, not compared to some.
http://www.speedtest.net/result/588902501.png
Does me, though :)

Currently going through the 5th Edition of Windows Internals to see if there's anything I missed with the internal training on Vista & W2K8, and seeing how Windows 7 is less than 2 weeks from GA, I'm going to take a look at updating the sticky again.

---

And that will be well worth waiting for!

Here's the more detailed look at hot patching, for those who don't go blind looking at assembler ;)
http://blogs.technet.com/mrsnrub/arc...-and-cold.aspx