| Date: | January 3, 2006 / year-entry #7 |
| Tags: | code |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20060103-12/?p=32793 |
| Comments: | 40 |
| Summary: | An increasingly important developer tax is supporting Remote Desktop Connection properly. When the user is connected via a Remote Desktop Connection, video operations are transferred over the network connection to the client for display. Since networks have high latency and nowhere near the bandwidth of a local PCI or AGP bus, you need to adapt... |
An increasingly important developer tax is supporting Remote Desktop Connection properly. When the user is connected via a Remote Desktop Connection, video operations are transferred over the network connection to the client for display. Since networks have high latency and nowhere near the bandwidth of a local PCI or AGP bus, you need to adapt to the changing cost of drawing to the screen. If you draw a line on the screen, the "draw line" command is sent over the network to the client. If you draw text, a "draw text" command is sent (along with the text to draw). So far so good. But if you copy a bitmap to the screen, the entire bitmap needs to be transferred over the network. Let's write a sample program that illustrates this point. Start with our new scratch program and make the following changes:
void Window::Register()
{
WNDCLASS wc;
wc.style = CS_VREDRAW | CS_HREDRAW;
wc.lpfnWndProc = Window::s_WndProc;
...
}
class RootWindow : public Window
{
public:
virtual LPCTSTR ClassName() { return TEXT("Scratch"); }
static RootWindow *Create();
protected:
LRESULT HandleMessage(UINT uMsg, WPARAM wParam, LPARAM lParam);
LRESULT OnCreate();
void PaintContent(PAINTSTRUCT *pps);
void Draw(HDC hdc, PAINTSTRUCT *pps);
private:
HWND m_hwndChild;
};
void RootWindow::Draw(HDC hdc, PAINTSTRUCT *pps)
{
FillRect(hdc, &pps->rcPaint, (HBRUSH)(COLOR_WINDOW + 1));
RECT rc;
GetClientRect(m_hwnd, &rc);
for (int i = -10; i < 10; i++) {
TextOut(hdc, 0, i * 15 + rc.bottom / 2, TEXT("Blah blah"), 9);
}
}
void RootWindow::PaintContent(PAINTSTRUCT *pps)
{
Draw(pps->hdc, pps);
}
LRESULT RootWindow::HandleMessage(
UINT uMsg, WPARAM wParam, LPARAM lParam)
{
switch (uMsg) {
...
case WM_ERASEBKGND: return 1;
...
}
There is an odd division of labor here;
the
void RootWindow::PaintContent(PAINTSTRUCT *pps)
{
if (!IsRectEmpty(&pps->rcPaint)) {
HDC hdc = CreateCompatibleDC(pps->hdc);
if (hdc) {
int x = pps->rcPaint.left;
int y = pps->rcPaint.top;
int cx = pps->rcPaint.right - pps->rcPaint.left;
int cy = pps->rcPaint.bottom - pps->rcPaint.top;
HBITMAP hbm = CreateCompatibleBitmap(pps->hdc, cx, cy);
if (hbm) {
HBITMAP hbmPrev = SelectBitmap(hdc, hbm);
SetWindowOrgEx(hdc, x, y, NULL);
Draw(hdc, pps);
BitBlt(pps->hdc, x, y, cx, cy, hdc, x, y, SRCCOPY);
SelectObject(hdc, hbmPrev);
DeleteObject(hbm);
}
DeleteDC(hdc);
}
}
}
Our new
Now connect to the computer via a Remote Desktop Connection
and run it again.
Since Remote Desktop Connection disables "Show window contents
while dragging", you can't use resizing to trigger redraws,
so instead maximize the program and restore it a few times.
Notice the long delay before the window is resized when you
maximize it.
That's because we are pumping a huge bitmap across the
Remote Desktop Connection as part of that
Go back to the old version of the We have one method that is faster over a Remote Desktop Connection, and another method that is faster when run locally. Which should we use? We use both, choosing our drawing method based on whether the program is running over a Remote Desktop Connection.
void RootWindow::PaintContent(PAINTSTRUCT *pps)
{
if (GetSystemMetrics(SM_REMOTESESSION)) {
Draw(pps->hdc, pps);
} else if (!IsRectEmpty(&pps->rcPaint)) {
... as before ...
}
}
Now we get the best of both worlds.
When run locally, we use the double-buffered drawing which
draws without flickering,
but when run over a Remote Desktop Connection, we use the simple
This is a rather simple example of adapting to Remote Desktop Connection. In a more complex world, you may have more complicated data structures associated with the two styles of drawing, or you may have background activities related to drawing that you may want to turn on and off based on whether the program is running over a Remote Desktop Connection. Since the user can dynamically connect and disconnect, you can't just assume that the state of the Remote Desktop Connection when your program starts will be the state for the lifetime of the program. We'll see next time how we can adapt to a changing world. |
Comments (40)
Comments are closed. |
Applications that don’t support Remote Desktop well: Media Player.
If it’s playing and I connect over RDC, then Media Player pauses.
Am I alone in thinking that this is a bit hokey? The whole point of a UI system is to abstract you from the hardware. I no more should care about running over a RDC link as I should care if the video card is Nvidia, AIT, etc. When I saw the start of the article, I assumed that there would be a description of how to "properly" manage repainting so as to work correctly under all circumstances. But "if (display_is_rdc) DrawRDCWay() else DrawNormal()" is just ludicrous. Does this mean that the local display is flickery, while the remote one is clean?
The ideal is to move the double-buffering to the OS, and then have it done on the machine with the physical display, rather than the machine with the application running.
Sounds like RDC is a nasty hack to me.
I’m going to be ruthless, Remote Desktop (or to be precise, XP user switching) does invoke taxes, but this one is pretty minimal. If its just painting performance, then skip it. Apps that try and do fancy scrolling graphics where you have to select moving things, that is where the effort is justified, because they are unusable. The example above offers a lower experience but is still workable.
Making your app work properly when there is >1 user simultaneously online can be a lot harder if there is any shared resource that you all need. And its hard to test. worry about that, not painting games.
Abstracting it away only works if the things you’re abstracting are sufficiently equal. In this case there is a massive difference in the data rate of the local bus vs. the remote link, so abstracting them to be the same doesn’t quite work so well. It’s like considering a poor guy and a rich guy to be the same and expecting both to be able to pony up a million bucks in a day.
Why not just repaint the regions that really need to be updated instead of repainting the entire window? Of course this would mean you only call ie. DrawText for text that is in or overlaps the update region. This not only fixes the flickering but also works fine over RDP.
In the days of CP/M I handled display by having a 24×80 array of characters that ought to be on the screen and a 24×80 array of characters that actually were on the terminal screen. Then a subroutine compared the two and sent a minimal set of commands to transmit to the terminal so that its screen corresponded with what I wanted.
This meant that my program could "draw" as little or as much as it liked, because useless redraws were optimised away.
There must be a good reason why Remote Desktop doesn’t do this. What is it?
Universalis: Because pixels are less efficient than characters. Suppose you type "a". Which is faster, transmitting "draw an ‘a’ at coordinates x,y" or "draw the following 23 pixels at these 23 coordinates"?
As little or as much? You want VNC then :-)
Here’s a question which I really should just look up and answer for myself, but someone might know anyway: How does RDP deal with drawing with fonts that aren’t on the client machine?
We’ve established that RDP just forwards something resembling the "draw this text at x,y" GDI call over the network, but in order to carry that out it needs the font definition. Does RDP transparently send the font over just as it does for bitmaps? Does it detect which fonts are already present on the client and not transmit those? Is any allowance made for font licencing forbidding the transmission of the font?
For some reason I find RDP an order of magnitude faster than the ATI Rage 128 (I think) in my server.
This example gets the point across, but it’s a bit contrived. To avoid the nasty flicker, it’s really better to dispense with CS_xREDRAW and be more intelligent about *what* to redraw on resize, and where to paint the background. Then the paint will boil down to ExtTextOut and FillRect, and we will have killed two birds with one stone :)
atyk: The CS_VREDRAW is necessary since a resize forces a complete repaint due to the centering. Yes, in this simple example, one could get away with being more clever in the PaintContent but you can imagine how in more complicated scenarios (e.g. metafile playback) this is not practical.
Another application that doesn’t support Remote Desktop well: Firefox.
Images on websites appear all garbled and ugly if you’re running Firefox over Remote Desktop.
Hayden: Re: UI system abstraction. Keep in mind that GDI is 20 years old and was designed way before remote desktop. WPF (aka Avalon) should be much better in this regard. Or, "ought to be", since I haven’t played around with it to know for sure.
James Schend: Is that maybe because your RDP connection is running at a low color depth? I’m not sure how else images would "appear all garbled and ugly".
As far as abstracting away the network: It sort of works, in some cases. X does it fairly well (the programmer doesn’t have to care whether the display is local or remote), but you still have programmers assuming that everything runs locally. And then remotely displaying a complicated X program (for instance, my installation of Thunderbird, which uses Gtk2) over a low-bandwidth connection (128-256kbit/sec) is really, really slow. (Not ugly, mind you, just really slow.)
Dang, forgot about this:
There’s also lbxproxy (low-bandwidth X proxy), which compresses the X stream and merges some requests, so it to works better over slow connections. I’m not sure how well Thunderbird (or Gtk2 in general) works through it, because I’m tunneling the X traffic through ssh2 already. (Which does some compression on its own, so maybe it would still be just as slow, I’m not sure. I don’t know if ssh’s X tunnel merges anything.)
RDC abstracts awy the network – notice that progams written to local video still run over RDC. They just run poorly due to the huge difference in display bandwidth. That was my point here: Don’t run poorly. Your users deserve better.
As someone who uses Remote Desktop to work from home with some regularlity, I can tell you that this can get very frustrating.
Having said that though, I’ve never really had much bother with double buffering. What *really* gets up my nose is apps which needlessly "fade in" dialog boxes, menus or tooltips despite the relevant effects being disabled. Also, apps that try to implement "scrolling" by continually repainting their entire window over and over don’t work too well. Fortunately I’ve only ever seen one of the latter. I see the former all the time.
This is useful to know. Does anyone here know how to make similar optimizations using Avalon/WPF under Remote Desktop? Or a web site that discusses it? I guess my main question is, how does WPF handle rendering under the hood in a terminal environment? Does it render everything as bitmaps or does it still render lines, text, etc. that can be optimized across the wire?
Tell the Money team to work with remote desktop. None of their charts draw with RDC (they just say "Loading…") and they use lots of really huge bitmaps as backgrounds even on Remote Desktop connections.
Having reverse engineered RDP in the past, I can answer how it works under the hood. The original version (‘Hydra Alpha’) followed the T.128 series of standards to the letter (they covered video conferencing type stuff).
Performance sucked. Not suprising as there were around 70 companies involved in the standards effort, and it was one of those standards that 70 ways to say hello, 70 ways to draw a line etc.
From that point on Microsoft diverged (and rightly so) in the interest of performance. The largest bitmap that gets transferred is 64×64 pixels. Generally that is what is transferred most of the time for most screen updates, using multiples where necessary.
Glyphs are also sent down and can be placed on the screen. This is how some text is drawn. There are also commands for drawing lines etc. However the vast majority of applications do stunts as mentioned in this article and comments, and so all that gets transferred are huge numbers of 64×64 pixel bitmaps to be drawn onscreen.
The protocol also has encryption and licensing. The former proved to be similar to SSL but it was like someone had ignored the intro to each chapter in crypto book that said "do not do things this way". The crypto was improved in RDP 5. The licensing has the client telling the server what OS version it is, providing an optional license token and the server chooses to allow it or supply a new one.
One thing that took ages was a field that seemed to be a redundant length field. Ultimately it turned out to be garbage that wasn’t filled in with anything by the client and ignored by the server. However it seemed meaningful at the time …
Jack, I would hope that the Money team works on a lot of other stuff before Remote Desktop performance … Money is the most insanely frustrating piece of software I’ve ever used; it is a classic case of coding and testing to a spec and ignoring the user experience.
I disagree with whoever says this kind of things should be abstracted away. No way. You have never used RDP if you say that. Never mind the bandwidth, think about the latency, neither is remotely comparable with an AGP bus and monopolizing the "bus" with large bitmaps totally kills the user experience. Don’t double-buffer, disable animations, disable backgrounds, forget the eye-candy
And please don’t abuse SM_REMOTESESSION like too many Microsoft applications do… Windows Movie Maker, you liar, you can work just fine on a remote connection
mirobin, that’s exactly how I feel about LRM-519 DVR LG and MS recently put out.
Hayden Clark, the current state of D3D requires you to know a lot more than simply the vendor of your graphics card. Beside the vendor you’ll need to know the card model, and most probable the driver revision. And if that weren’t enough you’ll have to juggle the various CAPS (capabilities) exposed by each of the cards. Or you can always design for the minimum spec system – how boring!
Back on topic. I’d recommend ensuring your application works over RDC – that’s it.
As with all optimization efforts make sure the resulting optimization is true. I would not implement the purposed solution without performance testing, that is certain. Take for instance running publisher, or some other layout based application. Most likely there will be several images within the layout of varying size. Occasionally there will be overlap, images spanning the view, and other cases. When working with these types of applications I can imagine scenarios where the sum of the components making up the composition is greater than the composition. Don’t be blind, measure.
There are two vectors of speed to consider when optimizing. The first is execution speed, everyone knows this, you use profiling tools to measure it. The second is user perception of speed. Surprisingly the two are not equivalent. Back in 98 I did some testing of a layout application I was working on. I profiled drawing several different types of layouts directly to a client DC verse drawing them to a memory DC and then blitting the result. Drawing directly to the client DC was faster in all cases, but users perceived double buffering to be faster. Why was this the case? Because users don’t know when drawing begins in the double buffered case, they only see the result which appears ‘instantaneously’. This compared to seeing each element composed onto the DC when drawing to the client directly.
There are two competing strategies for optimization; do what is fast and do what appears fast. Both scenarios require testing.
James
Ok, to improve on this example, what’s the way to detect if the HDC is doing some form of buffering for you (i.e. when running on longhorn or drawing to the printer)?
Does RDP transfer primitives drawn by GDI+ as well? I’m a bit worried that increasing usage of .NET’s GDI+-based System.Drawing means more software rendering and thus correspondingly poor performance over slow links. The increased visual fluff in the Visual Studio .NET UI makes it quite a bit slower than the old Win32-based VC6 UI over remote connections.
RDP does have support for some graphics primitives, but they don’t get too complicated. Go to http://www.rdesktop.org/#docs and get the T.128 spec. Section 18 lists what can happen which mostly is lines, rectangles, glyphs and bitmaps. It does do raster ops the Windows way which is what makes this quite fun to implement in an X windows environment. I guess it is a bit late to report it now, but Microsoft also misimplemented the bitmap encoding :-)
http://blogs.msdn.com/oldnewthing/archive/2005/05/24/421440.aspx
Oh, and RDP 6.0 will allegedly support WinFX primitives, which can presumably be accelerated by the client. So there’s another thing to plan for…
James: Yes, of course 3D and graphic programs which use the hardware accelleration need to do more device-specific things (but, hang on, what’s DirectX and Direct3D for….). But it seems to me that we are playing games with even a basic simple program that just draws stuff.
I think my main issue is that of asdf – are there cases where double-buffering is supported in the GDI, rather than having to do it by hand in the app? Then RDP could announce double-buffer support, and we are back to transferring GDI instructions over the link instead of vast bitmaps.
I think the reality of this kind of code branching is that you’ll have bugs that break the UI over RDC, rather than just being slow.
Ben Cooke: In addition to Anonymous Coward’s excellent comment, I should add that the glyphs are cached, as well as bitmaps, text, cursors, etc.
And there’s a way for the client to list known fonts, so that even the glyphs don’t need to be sent over the wire. However, this isn’t done even by the latest client. And I don’t know whether the server could cope with it.
As for Raymond’s argument here, I wholly agree that the tax should be paid in low-bandwidth situations. However, I think there should be a decent measure of display bandwidth available for programs to use, for faster as well as slower.
The Movie Maker designers presumably reckoned it would be too slow over RDP, but it works well enough – you don’t *watch* videos in Movie Maker! I use RDP over a fast network at home, often for hours, and I hate not having ClearType so much that I’ve patched the kernel to allow it. What happens when we all have gigabit connections?
Perhaps this is already included in Vista’s new benchmark API, but I can’t find any information on that at the moment…
Mark
Er. Somethingawful.com, not Fark… but the point still applies.
Yikes, that is broken.
It almost looks like a palette issue — like the palette that they think they’re using to display the image is different than the one that’s active on the system. Or something.
If you set the target machine’s desktop (the real one, not the RDP one) to 8-bit, does it fix the issue?
Yeah, that’s probably what it is. I saw the same thing with X11 on an 8bit display
Same thing as always… The "solution" included within Windows is useless; and third party, usually opensource apps come to the rescue.
"Windows File Sharing"? Bleh, using a decent FTP server is much more easier (both to set up as you want it, and for the client). Not to mention no 10 user limit, and no choosing between "simple file sharing" (and you can’t set permissions on your own computer) or "having to create user accounts just for that" (and people wanting to connect having to enter username and password).
"Remore Desktop"? Bleh, almost any VNC implementation (specially UltraVNC with the mirror video driver) defeats it in responsiveness and efficiency – by an order of magnitude. (Though it doesn’t serve the same purpose, I’m talking about using remotely single-user machine). Also, no "applications workarounding stuff and crippling themselves" b.s. Framebuffer with decent compression algorithms > anything that transfers draw calls.
"NTFS"? Bleh, an experimental Linux EXT2 filesystem driver runs EXTREMELY faster (specially with large numbers of small files), and doesn’t have any "fragmentation bullshit" (*) (Microsoft lies to you: fragmentation is the result of the filesystem code sucking, not a fact of life – I have batch files that copy stuff around that will destroy a newly created NTFS partition fragmentation-wise, but an EXT2 partition of the same size survives with no trouble at all). The problem is that said driver crashes too often :)
"Notepad"? Heck, even MS-DOS "EDIT" was better: it handled UNIX-style files (LF only; not CR LF) and binary files (hex editor). Yet another app to replace if you want to use it past the very basic stuff. By the way, my localized version changed some keyboard shortcuts, so Find is no longer CTRL+F. Retarded.
"Sound recorder"? Nice 60 second limit per record button click you have there.
"Paint"? Well, at least now it can save to non-retarded file formats. Kinda OK for screenshots, I guess.
"CMD.EXE"? Ha. Hahaha. Time to download cygwin and use bash.
"Worpad"? Regressions from Windows 98. It has a Word 6.0/95 plugin ("mswrd6.wpc") but it refuses to read such files. Also, I can get it to write files that no Microsoft app will open, but which OpenOffice will. A bit ironic.
"Windows Media Player"? The fact that "mplayer2.exe" is still included should tell you something…
"Zip file support"? Ha. Hahaha. Oh, and thanks for treating them like folders even if I use another program. I really enjoy the 10 second pause when I open a folder with thousands of ZIP files on it, and I enjoy the status bar not showing total size when multiple ZIP files are selected. I have to create a 0 byte file of another type and select it too for the sum to be displayed.
Want to upgrade any of them (to path holes, that is)? Prepare to reboot. Compare with "just download the latest version and unzip it".
I could go on and on… The situation sucks, because if any of this apps became useful, MS would get sued :'(
(*) Also, NTFS has one decision that is high on my list of "incredibly retarded stuff", up there with the disk cache having the same or more priority than apps: the 1/8 disk space reservation for the MFT. At 1K per entry, that assumes the average file size will be about 8K, and that an 80 gig drive will have 10 million files. Yeah right. And sometimes, when you fill more than 7/8 of the disk, it "gives up" and your MFT ends up in 50+ little pieces. To workaround this, the first thing I have to do is creating 300K files to make the MFT grow to 300 megs and survive fragmentation later on.
Wouldn’t it be a great feature if somethingawful.com couldn’t render well? harharhar
I don’t believe Scott’s comment, but I am similarly cynical. Raymond points out the developer tax here, but thats a one time cost per write or fix. There is a more significant tax here in terms of test.
For this reason, optimizations like this might make a lot of sense for objects in the shell (ie the Start button) that are renderred a bazillion times on any particular day. But for a typical UI element, especially given the infrequency of automated UI testing, this sort of optimization is probably not a net gain.
Kinda Folish: Go wank elsewhere. And fix your grammar on the way, nothing in the world is ‘EXTREMELY faster’.
PingBack from http://blogsitos.com/linux/2006/01/06/network-graphics-can-be-expensive/