How to retrieve text under the cursor (mouse pointer)

Date:April 23, 2004 / year-entry #156
Orig Link:
Comments:    23
Summary:Microsoft Active Accessibilty is the technology that exposes information about objects on the screen to accessibility aids such as screen readers. But that doesn't mean that only screen readers can use it. Here's a program that illustrates the use of Active Accessibility at the most rudimentary level: Reading text. There's much more to Active Accessibility...

Microsoft Active Accessibilty is the technology that exposes information about objects on the screen to accessibility aids such as screen readers. But that doesn't mean that only screen readers can use it.

Here's a program that illustrates the use of Active Accessibility at the most rudimentary level: Reading text. There's much more to Active Accessibility than this. You can navigate the objects on the screen, read various properties, even invoke commands on them, all programmatically.

Start with our scratch program and change these two functions:

OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)
  SetTimer(hwnd, 1, 1000, RecalcText);
  return TRUE;

PaintContent(HWND hwnd, PAINTSTRUCT *pps)
  if (g_pszText) {
      RECT rc;
      GetClientRect(hwnd, &rc);
      DrawText(pps->hdc, g_pszText, lstrlen(g_pszText),
               &rc, DT_NOPREFIX | DT_WORDBREAK);

Of course, the fun part is the function RecalcText, which retrieves the text from beneath the cursor:

#include <oleacc.h>

POINT g_pt;
LPTSTR g_pszText;

  POINT pt;
  if (GetCursorPos(&pt) &&
    (pt.x != g_pt.x || pt.y != g_pt.y)) {
    g_pt = pt;
    IAccessible *pacc;
    VARIANT vtChild;
    if (SUCCEEDED(AccessibleObjectFromPoint(pt, &pacc, &vtChild))) {
      BSTR bsName = NULL;
      BSTR bsValue = NULL;
      pacc->get_accName(vtChild, &bsName);
      pacc->get_accValue(vtChild, &bsValue);
      LPTSTR pszResult;
      DWORD_PTR args[2] = { (DWORD_PTR)(bsName ? bsName : L""),
                            (DWORD_PTR)(bsValue ? bsValue : L"") };
                        FORMAT_MESSAGE_FROM_STRING |
                        TEXT("Name: %1!ws!\r\n\r\nValue: %2!ws!"),
                        0, 0, (LPTSTR)&pszResult, 0, (va_list*)args)) {
        g_pszText = pszResult;
        InvalidateRect(hwnd, NULL, TRUE);


Let's take a look at this function. We start by grabbing the cursor position and seeing if it changed since the last time we checked. If so, then we ask AccessibleObjectFromPoint to identify the object at those coordinates and give us an IAccessible pointer plus a child identifier. These two pieces of information together represent the object under the cursor.

Now it's a simple matter of asking for the name (get_accName) and value (get_accValue) of the object and format it nicely.

Note that we handled the NULL case of the BSTR in accordance with Eric's Complete Guide to BSTR Semantics.

For more information about accessibility, check out Sara Ford's WebLog, in particular the bit titled What is Assistive Technology Compatibility.

Comments (23)
  1. Cooney says:

    That’s pretty cool. Two questions:

    1: How heavy is that parsing and traversal?

    2: could you defer the calculation until needed and just cache the current mouse position?

  2. Raymond Chen says:
    1. It’s up to the implementation of IAccessible.

      2. Certainly. This was just a Q&D program.
  3. Peter Torr says:

    (About to look clueless — hey, I’m a PM!)

    Aren’t you freeing an invalid g_pszText the first time round?

    And wouldn’t you want to VariantInit(&vtChild) to begin with?

    (OK, back to making coffee for the devs…)

  4. Raymond Chen says:
    1. LocalFree is documented to do nothing when passed NULL.

      2. The pvarChild parameter to AccessibleObjectFromPoint is an [out] parameter, so you don’t need to initialize the value on the way in.

  5. Michael says:

    But, g_pszText is never initialized to NULL though, therefore the first call to LocalFree is called on a random pointer.

  6. Raymond Chen says:

    Global pointer variables are initialized to NULL by default.

  7. TomTom says:

    I did as the instruction says and I got the following error.

    (‘accessibility’ is the name of the project.)

    accessibility error LNK2019: unresolved external symbol _AccessibleObjectFromPoint@16 referenced in function "void stdcall RecalcText(struct HWND *,unsigned int,unsigned int,unsigned long)" (?RecalcText@@YGXPAUHWND__@@IIK@Z)

    I use VC7.1. Did anyone have this error?

  8. Raymond Chen says:

    The documentation for AccessibleObjectFromPoint says

    Library: Use Oleacc.lib.

  9. TomTom says:

    Thanks! I was able to compile and link.

  10. TomTom says:

    Hi, this is me again.

    I followed the example by creating a Win32 project. Because Win32 applications do not look cool these days, I am wondering if this can be done from a managed code. If there is a tip to do it, can you please let me know?

    Your help is very much appreciated. Thank you.

  11. Raymond Chen says:

    "not actually a .NET blog"

  12. TomTom says:

    I managed to find a way to call the function from a C# program using pinvoke. It took me so long, but I am glad that it can be done.

    So, here is my problem. :-). It looks like your code cannot display non-ASCII characters. Do you have a plan to make the code globalized using Unicode? If not, can you suggest me how I can do it?


  13. Raymond Chen says:

    Um, you need to remember to #define UNICODE and then select an appropriate font into the DC. I had hoped it wasn’t necessary to mention that. I left it out because that wasn’t the point of the article. My articles aren’t complete solutions; they are highlights of specific technology.

  14. Ray says:

    This program grabs the whole sentence under the cursor. Is there any way that it could just get the individual word under the cursor?

  15. Raymond Chen says:

    I don’t know. The point here was to get people to notice IAccessible. I leave you to explore it more on your own.

  16. Mayu says:

    How about in visual basic 6.0?

    I’m currently developing code to press a button on a particular program window.

    If you have any advice on this, that would be great.

    pls email:


  17. Raymond Chen says:

    I’m assuming my readers are smart enough to translate programs from one language to another. In fact, IAccessible itself is dispatch-based so it is even easier to use from a scripting language than from C++.

  18. Raymond Chen says:

    Commenting on this entry has been closed.

  19. You’ll probably use it too.

  20. Because not everybody uses the same class library.

Comments are closed.

*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index