Subscribe to receive all latest blog updates

The article describes the general approach to shell extensions creation by the example of shortcut menu and icon overlay handlers. A number of non-obvious problems, which may be encountered during the development of these types of extensions, are also described together with their solutions.

Contents:

What is Shell extensions?

Creating shell extension handler

  Implementation of the required functions and interfaces

  Definition of interfaces, specific to shell extensions

  Shell extension registration

  Resources preparation

  Installation and Startup

  Removing

Common problems and their solutions

Conclusion

Downloads

Written by:
Dasha Tkachenko,
Software developer of Driver Team

What is Shell extensions?

Shell extension handlers allow to supplement usual set of actions while working with Windows Explorer. Using shell extensions, you can, for example, add a new tab into the Properties window, change file preview, and more. Shell extensions can be represented as individual plug-ins to Windows Explorer. Usually shell extensions are used for applications with special file types. Before performing any act, shell calls registered extension handlers to change this action. Here are the most common examples of programs that establish its own set of shell extensions: Dropbox, TortoiseSVN, WinRAR, or SkyDrive.

Depending on the type, extension handlers can be added either to all types of objects within Windows Explorer or only to certain types.

Some extensions used with specific file types

  • Shortcut menu handler – allows to add new items to the context menu
  • Icon handler – allows to change default icon of the file
  • Property sheet handler – allows to add or change tabs in the Properties window
  • Infotip handler – changes the pop-up tip, which appears when you hover over the file
  • Metadata handler – provides access to the file metadata (properties)

Extensionsthat do not dependon the file type

  • Column handler – adds a column in the Details view mode
  • Icon overlay handler – allows to add overlay for the file icon
  • Drag-and-drop handler – is called when the object is being dragged, and changes the appearing menu
  • Search handler – allows to change the default search algorithm, which is called in the Windows Explorer window or in the Start menu.

Don’t forget that shell extensions development can affect the speed of Windows Explorer.

Creating shell extension handler

We’ll discuss shell extensions creation by the example of overlay and the context menu extension. Other types of extensions can be implemented by analogy.

Each extension handler is a COM-object. Handler must have its own GUID and be registered (the path is determined by the type of extension handler). Now let’s discuss the stages of existence and creation of extension handler.

Implementation of the required functions and interfaces

Since extension handlers are COM-objects, they are implemented as DLL. At the same time, as well as each COM-object, DLL must export the following functions:

Class factory is used to create the component object and must implement the IClassFactory interface. Here's how the component class factory looks like:

class ClassFactory : public IClassFactory
    {
    public:
        // IUnknown
        IFACEMETHODIMP QueryInterface(REFIID riid, void **ppv);
        IFACEMETHODIMP_(ULONG) AddRef();
        IFACEMETHODIMP_(ULONG) Release();

        // IClassFactory
        IFACEMETHODIMP CreateInstance(IUnknown *pUnkOuter, REFIID riid, void **ppv);
        IFACEMETHODIMP LockServer(BOOL fLock);

        explicit ClassFactory();
        ~ClassFactory();

    private:
        long m_cRef;
    };

In addition, the class itself with the extension handler logic must implement the IUnknown interface, like all COM-objects. Interfaces that are specific to different types of shell extensions inherit from this interface. IUnknown interface is as follows:

class IUnknown
{
public:
   virtual HRESULT QueryInterface(REFID riid, void** ppv)=0;
   virtual ULONG AddRef () = 0;
   virtual ULONG Release() = 0;
};

It is worth noting that several extension handlers can be implemented in a single DLL. But at the same time, you should specify GUID, perform registering in the registry, and set up a class factory for each of them (as for class factory, a common one can be used for all of them).

Definition of interfaces, specific to shell extensions

In addition to the required IUnknown COM interface, extension class must implement interface that is specific to shell extensions. For different types of extensions, there is a different set of required interfaces.

For shortcut menu handler, it is the IShellExtInit interface and IContextMenu, and for overlay icon handler - IShellIconOverlayIdentifier. Let’s consider each of them in detail.

IShellExtInit

Provides only one function - an extension handler initialization function

class IShellExtInit : public IUnknown
{
public:
   virtual HRESULT Initialize(LPCITEMIDLIST pidlFolder, LPDATAOBJECT pDataObj, HKEY hKeyPro-gID) = 0;
};

The Initialize method is called by Windows Explorer before you perform an action, which this handler extends. For example, in case of the context menu extension, this function will be called after clicking on selected files and before the menu opening.

In the Initialize method, you can implement logic that filters extension work depending on the type of the selected files or any other conditions. Furthermore, the list of objects, which user chooses, can be obtained in this function.

IContextMenu

It contains functions for working with context menu:

class IContextMenu : public IUnknown
{
public:
   virtual HRESULT QueryContextMenu(HMENU hMenu, UINT indexMenu, UINT idCmdFirst, UINT idCmdLast, UINT uFlags) = 0;
   virtual HRESULT InvokeCommand(LPCMINVOKECOMMANDINFO pici) = 0;
   virtual HRESULT GetCommandString(UINT_PTR idCommand, UINT uFlags, UINT *pwReserved, LPSTR pszName, UINT cchMax) = 0;

};

The QueryContextMenu method is responsible for adding new menu items to the context menu. This is where the logic for new menu item creation is added. The menu item can be created by the WinApi InsertMenuItem function.

Each menu item is assigned to its id. The idCmdFirst parameter is a minimum value of id, which can be assigned to a menu item. Accordingly, idCmdLast is a maximum one. It is important that if a menu item contains a submenu, the submenu items are assigned to ids, which are seriated after the id of the menu item.

The QueryContextMenu method returns the difference between the first and the last assigned id +1.

QueryContextMenu is called by Windows Explorer only if the extension is registered for the selected file types.

In the InvokeCommand method, clicks on the added menu items are processed. Each menu button is identified by the verb value, which is specified while creating the item in QueryContextMenu. Verb is a string with the command name that is executed by the menu item. Verb is given in two versions: with and without Unicode using. Therefore, the verb format should be checked before command proceeding.

The GetCommandString method is used to obtain a canonical verb name, assigned to the menu item. This method is optional and set for Windows XP and earlier versions of Windows. It is used to display the hint text for the selected menu item in the status bar.

IShellIconOverlayIdentifier

This interface must be implemented by the overlay icon handler:

class IShellIconOverlayIdentifier: public IUnknown
{
public:
   STDMETHOD(GetOverlayInfo)(LPWSTR pwszIconFile, int cchMax, int *pIndex, DWORD* pdw-Flags) = 0;
   STDMETHOD(GetPriority)(int* pPriority) = 0;
   STDMETHOD(IsMemberOf)(LPCWSTR pwszPath, DWORD dwAttrib) = 0;

};

The GetOverlayInfo method is used for the first call to overlay handler in order to add an icon to the system list of pictures. After that, the icon cannot be changed and subsequent calls of GetOverlayInfo must return the path and the icon index. If method returns an error status for the first run, the icon will not be loaded, even if subsequent calls to GetOverlayInfo will be successful. This can lead to a situation when handler is working and the icon is not displayed.

Icon index is a serial number of icon (starting with 0) in the DLL resource list.

The GetPriority method is used to specify the overlay icon priority over other icons. Depending on it, handler can either be a top priority or not. The highest priority is 0.

The IsMemberOf method must contain logic that indicates whether to apply the icon to the given object in Windows Explorer or not. This method is called in turn for all objects in the current Windows Explorer window on window opening and after its update. If you do not specify any filters here and just return S_OK, the icon will be displayed on all objects in Windows Explorer.

Shell extension registration

The logic for extension registration must be implemented in the DllRegisterServer function. This function is called during the shell extension installation and is responsible for the COM-object registration. Accordingly, the DllUnregisterServer function must contain code for deleting records from the registry.

As noted above, the registry hive, which contains information about the extension, differs depending on the type of extension handler. For example, for the shortcut menu handler, the path is as follows:

HKEY_CLASSES_ROOT\<file_type>\shellex\ContextMenuHandlers\ExtensionName

The <file_type> parameter may be defined as follows:

  • for a specific file type: .bmp, .docx, .png
  • for all file types: *
  • for a group of objects: Drive, Directory, Folder, LibraryLocation.

Shortcut menu handler must be registered for each object type separately.

Icon overlay handler is registered once by the path:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers\ExtensionName

In addition, you need to register the COM-component itself:

HKEY_CLASSES_ROOT\CLSID\<COM_object_GUID>\InprocServer32\ThreadingModel

For example, the DllRegisterServer function, which registers overlay icon handler and shortcut menu handler for all types of files ("*"), might look like this:

// GUIDs for COM-objects
const CLSID kMenuExtCLSID = { 0xa1239638, 0xb2ba, 0xa2c8, { 0xa4, 0x3b, 0xf6, 0xe2, 0x10, 0x81, 0xf, 0x77 } };
const CLSID kOverlayExtCLSID = { 0xa4569638, 0xc2ba, 0x42c8, { 0xb4, 0xfb, 0xc6, 0xe6, 0x10, 0x91, 0xf, 0x23 } };

STDAPI DllRegisterServer(void)
{
    HRESULT hr;
    wchar_t szModule[MAX_PATH];

    if (GetModuleFileName(g_Dll->DllInst(), szModule, ARRAYSIZE(szModule)) == 0)
    {
        hr = HRESULT_FROM_WIN32(GetLastError());
        return hr;
    }

    // register COM-object for shortcut menu handler
    hr = RegisterInprocServer(szModule, kMenuExtCLSID, L"AppCustomExtensions", L"Apartment");

    if (SUCCEEDED(hr))
    {
        // register shortcut menu handler
        hr = RegisterShellExtContextMenuHandler(L"*", kMenuExtCLSID, L"ContextMenuExt");
    }

    // register COM-object for overlay icon handler
    hr = RegisterInprocServer(szModule, kOverlayExtCLSID, L"AppCustomExtensions", L"Apartment");

    if (SUCCEEDED(hr))
    {
        // register overlay icon handler
        hr = RegisterShellExtOverlayIconHandler(kOverlayExtCLSID, L"OverlayIconExt");
    }

    return hr;
}

The RegisterInprocServer, RegisterShellExtContextMenuHandler, and RegisterShellExtOver-layIconHandler functions contain the code for adding new hives and keys to the registry.

Resources preparation

The icons must be prepared to be correctly displayed by extension handlers.

The .bmp picture is needed for a context menu. Below we will describe how to maintain a transparent background images.

The case of overlay handler is a bit more complicated. The icon must be in .ico format and support the following dimensions:

  • 10x10 (for 16x16)
  • 16x16 (for 32x32)
  • 24x24 (for 48x48)
  • 128x128 (for 256x256)

Icon will be displayed in the lower left corner and should take no more than 25% of the file icon (except for 10x10 for 16x16).

When scaling objects in Windows Explorer, the right size will be chosen.

Installation and Startup

Shell extensions installation is performed using the standard regsvr32 utility. For example:

> regsvr32 C:\Example\ShellExtensions.dll

When you run this command, the DllRegisterServer function is called, and this function provides extension registration in the registry. All the logic for adding and removing entries from the registry must be implemented by developer.

After this, the shortcut menu handler immediately begins its work and it can be verified by opening a menu. As for the overlay handler, the icons are not going to appear immediately after the registration. To do this, you must restart Windows Explorer, in order to download the needed resources.

Removing

Shell extension removing, as well as installation, is performed using regsvr32:

> regsvr32 \u C:\Exampe\ShellExtensions.dll

Meanwhile, the DllUnregisterServer function is called, and records are being deleted from the registry.

After removal of the overlay handler, Windows Explorer also needs rebooting.

Common problems and their solutions

When developing extension handlers, several not quite obvious problems can appear. Some of them are listed below together with the ways to overcome them.

  1. Overlay icons are not displayed at all

As it was described above, the problem may be in logic of the GetOverlayInfo function, but it can also be different. Windows uses only 15 registered overlay icon handlers, all others are simply ignored. List of the lucky ones is determined by the name in the registry branch: HKEY_LOCAL_MACHINE \ Software \ Microsoft \ Windows \ CurrentVersion \ Explorer \ ShellIconOverlayIdentifiers - the first 15 are taken in alphabetical order. Therefore, if Dropbox or TortoiseSVN, which records a number of extensions, is installed in the system, then it can become a reason for not displaying icons.

Solution: To add any particular character before the overlay handler name during registration. It will provide the extension location to be at the beginning of the list of extensions. For example, we can add spaces before the handler name.

  1. Displaying icon with a transparent background in the context menu

The problem is that the .bmp picture, which does not support transparency, is used when you add a menu item. If you use .ico or .png file, then the icon will be displayed on a black background after its conversion.

  • Solution: To redraw .ico to .bmp format in the code during extension functioning.

For example, the following implementation can be used (UX-THEME.DLL is used):

m_beginPainFunc = (BeginPaintFunction)::GetProcAddress(m_hUxTheme, 
"BeginBufferedPaint");
m_endPaintFunc = (EndPaintFunction)::GetProcAddress(m_hUxTheme, 
"EndBufferedPaint");

	...

 SIZE iconSize;
        RECT iconRect;
        HBITMAP hBmp = NULL;
        HDC hdcDest = CreateCompatibleDC(NULL);

        // set the icon size
        iconSize.cx = GetSystemMetrics(SM_CXSMICON);
        iconSize.cy = GetSystemMetrics(SM_CYSMICON);
        SetRect(&iconRect, 0, 0, iconSize.cx, iconSize.cy);

        if (hdcDest)
        {
            if (SUCCEEDED(CreateHbitmap(hdcDest, &iconSize, NULL, &hBmp)))
            {
                HBITMAP oldBitmap = (HBITMAP)SelectObject(hdcDest, hBmp);
                if (oldBitmap)
                {

// set the necessary parameters
                    BLENDFUNCTION bfAlpha = { AC_SRC_OVER, 0, 255, AC_SRC_ALPHA };
                    BP_PAINTPARAMS paintParams = {0};

                    paintParams.cbSize = sizeof(paintParams);
                    paintParams.dwFlags = BPPF_ERASE;
                    paintParams.pBlendFunction = &bfAlpha;

// repaint the icon
                    HDC hdcBuffer;
                    HPAINTBUFFER hPaintBuffer = m_beginPainFunc(hdcDest, &iconRect, BPBF_DIB, &paintParams, &hdcBuffer);

                    if (hPaintBuffer)
                    {
                        if (DrawIconEx(hdcBuffer, 0, 0, hIcon, iconSize.cx, iconSize.cy, 0, NULL, DI_NORMAL))
                        {
                            ConvertBufferToPARGB32(hPaintBuffer, hdcDest, hIcon, iconSize);
                        }

                        m_endPaintFunc(hPaintBuffer, TRUE);
                    }

                    SelectObject(hdcDest, oldBitmap);
                }
            }

            DeleteDC(hdcDest);
        }
  1. Problems with context menu displaying

There can be several problems, associated with context menu:

  • After registering shell extension for LibraryLocation, context menu extension is not displayed.

The problem may be in the implementation of the IShellExtInit interface Initialize method, which is called to initialize the shell extension. IdataObject, which is used for receiving objects, for which the extension will be displayed, is passed to the method as a parameter. Attempts to receive the CF_HDROP format objects from it, when you click on the library folders (Documents, Music ...), do not lead to the expected result.

  • Solution: you must use the CFSTR_SHELLIDLIST format, which is similar to CF_HDROP, but contains PIDL (a pointer to the object id in the ITEMIDLIST shell-structure) instead of the file path. It allows CFSTR_SHELLIDLIST to process system and virtual file system objects.  
Int m_shellIdList = RegisterClipboardFormat(CFSTR_SHELLIDLIST);
 
IFACEMETHODIMP ContextMenuExt::Initialize(
LPCITEMIDLIST pidlFolder, LPDATAOBJECT pDataObj, HKEY hKeyProgID)
{
    if (pDataObj == NULL)
    {
      return E_INVALIDARG;
    }
    
    HRESULT hr = E_FAIL;
  
    // instead of 
    // FORMATETC etc = { CF_HDROP, NULL, DVASPECT_CONTENT, -1, TYMED_HGLOBAL };
    FORMATETC fe = { (CLIPFORMAT) m_shellIdList, NULL, DVASPECT_CONTENT, -1, TYMED_HGLOBAL };
    . . . 
}
  • In case of shell extension registration for AllFileSystemObjects, clicking the .lnk files adds extension to the context menu twice.

The problem is that the extension works both for the object and for the link, which points on it.
 

Solution: In the QueryContextMenu method of the IContextMenu interface, not only CMF_DEFAULTONLY flag must be checked, but also the context menu extensions should ignore objects with the CMF_NOVERBS and CMF_VERBSONLY flags.

  • The context menu extension is not displayed for some types of files, although it can be registered for all types.

The problem may be in the implementation of the QueryContextMenu method of the IcontextMenu interface. In case of success, it should return HRESULT with code value that is equal to the maximum offset of the command identifier that was assigned plus one.

For example, if idCmdFirst equals 5 and 3 more menu items with 5, 7, 8 command identifiers were added, then the return value should be: MAKE_HRESULT(SEVERITY_SUCCESS, 0, 8 - 5 + 1).

If you forget to plus one, the context menu extension can still work but not in all cases.

  1. The problem with starting a thread from DllMain

If a thread is started in DllMain (for example, to initialize something), there may be problems with DLL unloading using FreeLibrary. For example, DLL with shell extensions loads dynamically in order to unregister extensions manually.

Meanwhile, a thread to initialize logging is starting in DllMain. After that, DllUnregisterServer is called, and it quickly works out, then, as you might expect, FreeLibrary is called. In this case, it may happen that a thread, called from DllMain, either has not completed to execute or has not started yet. And DLL has already been unloaded from the process context. As a result, memory areas are damaged and access violations may occur at any time when there is a reference to these areas.

Solution: Wait for the end of the initialization thread in the exported DLL functions.

Conclusion

In this article, we discussed all types of shell extensions, as well as a general approach to their implementation. In addition, basic steps for shortcut menu and overlay icon handlers creation were described. There were also examined several pitfalls in implementing these extensions.

Downloads

Get the sample project sources (ZIP)