I'm trying to implement a simple Unicode console in a window, in
as-low-level-as-possible C (it's a compatibility layer for another C
program).

Right now I'm storing the text in an array of WCHARs and am using
brain-dead code like this:

WCHAR* p = buffer;
for (int y = 0; y < screenheight; y++)
{
  int ys = y * textheight;
  for (int x = 0; x < screenwidth; x++)
  {
    int xs = x * textwidth;
    /* code here will eventually set the foreground and background
     * colour depending on the contents of the buffer, which is why
     * we're drawing each character individually */
    TextOutW(dc, xs, xy, &p, 1);
    p++;
  }
}

However, I'm finding this has two major problems:

- firstly, it's not doing any kind of font substitution. If I try to
draw a character that's not in the font (which right now is
SYSTEM_FIXED_FONT) I just get a black square. That's not what I want.
Does Windows GDI do any kind of automatic font substitution, and if so,
how do I turn it on?

- secondly, and very surprisingly, it's dog slow. It's taking several
hundred milliseconds to redraw a 150x40 character window. I thought
TextOutW() is the fastest way to draw text? Is there anything I might be
doing to slow things down?

The redraw speed is the biggest problem; right now it flickers horribly
whenever I try to update! I could always double-buffer it, but for
heavens' sake it's just *text*, it shouldn't be necessary.

Can anyone suggest anything?

-- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ life←{ ↑1 ⍵∨.^3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵ }
│ --- Conway's Game Of Life, in one line of APL

David Given wrote:
[..]
> Right now I'm storing the text in an array of WCHARs and am using
> brain-dead code like this:
[..] (brain dead code) [..]

> - firstly, it's not doing any kind of font substitution. If I try to
> draw a character that's not in the font (which right now is
> SYSTEM_FIXED_FONT) I just get a black square. That's not what I want.
> Does Windows GDI do any kind of automatic font substitution, and if so,
> how do I turn it on?

Font substitution is an application thing, not an OS feature.
You can use various strategies for font substitution -- I'd go for 
'simplest is best' if I were you. You would typically need to get all 
supported characters for all installed font, and sort through them for 
each 'right' character to display.

You'd need to ensure the width is the same, though, or else your 
calculations go out of whack.

> - secondly, and very surprisingly, it's dog slow. It's taking several
> hundred milliseconds to redraw a 150x40 character window. I thought
> TextOutW() is the fastest way to draw text? Is there anything I might be
> doing to slow things down?

You are already slowing things down as much as you can (apart from a 
sleep() call in your inner loop) :-D

 >      [..] which is why
 >      * we're drawing each character individually */
 >     TextOutW(dc, xs, xy, &p, 1);

The overhead of drawing one character at a time is *horrible*. Draw as 
much as you can in one go. You also don't have to measure every 
character's width

> The redraw speed is the biggest problem; right now it flickers horribly
> whenever I try to update! I could always double-buffer it, but for
> heavens' sake it's just *text*, it shouldn't be necessary.

150 * 40 = 6000 characters. Each character is at least one filled 
complex polygon (some characters have more than one poly). You do the 
math. Even single line-buffering may yield a faster and less-flickery 
result.

http://www.catch22.net/ discusses the basics of drawing complex 
characters and strings.
However, for a simple console you might want to implement your own 
character glyph cache: one bitmap per character. Blit to screen as fast 
as possible -- no transparency needed. Store a black-and-white bitmap if 
you *really* are going for speed-without-the-memory, grayscale if you 
don't mind some postprocessing (to draw in different colors), or full 
color, optimized for your screen DC (and you have the choice of cacheing 
just 'plain' colored text, or all possible color combo's you might need).

[Jw]

On Jan 21, 12:06=A0am, David Given <d...@cowlark.com> wrote:
> I'm trying to implement a simple Unicode console in a window
> [snip]
> Does Windows GDI do any kind of automatic font substitution

Not in the sense you mean, as far as I know.  If it's a Unicode
console, why are you using SYSTEM_FIXED_FONT anyway?  Surely you need
to be using a Unicode font such as Lucida Console?

> - secondly, and very surprisingly, it's dog slow. It's taking several
> hundred milliseconds to redraw a 150x40 character window.

Are you sure?  That seems an improbably long time.  You could speed it
up by getting TextOutW to output complete *lines* rather than
individual *characters*; that would also have other advantages such as
allowing you to use a proportional-faced font and handling kerning
better.  Have you considered embedding newlines and using DrawTextW to
output the whole thing?

> The redraw speed is the biggest problem; right now it flickers horribly

You don't describe the context in which your code is called.  Is it in
WM_PAINT?  Do you handle WM_ERASEBKGND (or change the class
hbrBackground to NULL)?

Richard.
http://www.rtrussell.co.uk/

On Jan 21, 9:35=A0am, Richard Russell <n...@rtrussell.co.uk> wrote:
>=A0You could speed it up by getting TextOutW to output complete
> *lines* rather than individual *characters*;

Or, noting your requirements to change the colour, split the line into
blocks having the same foreground and background colour, and output
each of those using TextOutW.

Richard.
http://www.rtrussell.co.uk/

On 21/01/10 09:30, [Jongware] wrote:
[...]
> Font substitution is an application thing, not an OS feature.

Well, I come from a Unix background, and there font substitution is done
at the GUI level as a matter of course: if you ask for it to render a
glyph, and it can't find the glyph in the current font, it'll use Panose
to try and find a glyph in the closest matching font to the one you
asked for.

The end result is that you can draw any text in any script in any font
and it will Just Work. It may look a bit funny if you've got an odd set
of fonts installed, but it'll work. Applications don't need to worry
about it.

[...]
> The overhead of drawing one character at a time is *horrible*. Draw as
> much as you can in one go.

This really surprises me. Why is it so much faster to draw multiple
glyphs with a single call to TextOutW() than to draw multiple glyphs
with multiple calls to TextOutW()? Is this system call overhead?
TextOutW() *itself* should just be a simple loop that reads a glyph from
the string and draws each one. What does it use to draw the glyphs? Is
this function available for application use?

[...]
> 150 * 40 = 6000 characters. Each character is at least one filled
> complex polygon (some characters have more than one poly). You do the
> math. Even single line-buffering may yield a faster and less-flickery
> result.

Surely Windows caches font glyphs in bitmaps! It can't possibly
rasterise each glyph onto the screen when the application asks to draw it!

> http://www.catch22.net/ discusses the basics of drawing complex
> characters and strings.

Thanks, lots of good stuff there.

> However, for a simple console you might want to implement your own
> character glyph cache: one bitmap per character. Blit to screen as fast
> as possible -- no transparency needed.

This does sound like my best bet. I can do all the complicated work to
try and determine which font to use for a glyph lazily, the first time
the app tries to draw the glyph (it has to be lazy because I don't know
what Unicode subset the app will be using).

Does Windows have any standard data structure libraries (hash map,
associative array, etc) that will help with this, or will I have to roll
my own?

In addition, are you *certain* that there isn't a Windows utility
somewhere that will do what I want? Unfortunately Windows seems to use
the phrase 'font substitution' in a slightly different meaning to the
one I'm used to, so I don't know what keywords to search for. Rolling my
own is going to be a horrible amount of code, and it's such a universal
thing that surely there must be code already on the system that will do
it for me...

-- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ life←{ ↑1 ⍵∨.^3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵ }
│ --- Conway's Game Of Life, in one line of APL

On 21 Jan, 00:06, David Given <d...@cowlark.com> wrote:
> - firstly, it's not doing any kind of font substitution. If I try to
> draw a character that's not in the font (which right now is
> SYSTEM_FIXED_FONT) I just get a black square. That's not what I want.
> Does Windows GDI do any kind of automatic font substitution, and if so,
> how do I turn it on?

I'm not an expert on this, but I had a similar problem. There seems to
be a bug in Windows. What is supposed to happen is that, when it is
going to display some text, it checks through it to see if there are
any characters it can't handle with its current font ("font" may not
be the correct technical term here...) and it switches in a different
one which can handle the characters. However, for a number of
characters (ones I've found so far include club, heart, spade, diamond
and Euro) it thinks it can handle them but it can't and displays them
as short vertical lines.

A solution suggested by Alf Steinbach is to do:

SaveDC(hdc);
SelectObject(hdc, GetStockObject(DEFAULT_GUI_FONT));

before you display the characters and

RestoreDC(hdc, -1);

after. This changes to a font which is very ugly but does display
properly. An alternatively technique which seems to work is to include
a nul character in the characters to be printed - this seems to
trigger the "get all the characters" font without actually displaying
anything. Eg

buff[0] = *p;
buff[1] = 0;
TextOutW(dc, xs, xy, p, 2);

I take no responsibility for the latter!!

Hope this helps.
Paul.

On 22/01/10 00:07, David Given wrote:
[...]
> This does sound like my best bet. I can do all the complicated work to
> try and determine which font to use for a glyph lazily, the first time
> the app tries to draw the glyph (it has to be lazy because I don't know
> what Unicode subset the app will be using).

Well, I've done that; it wasn't as much code as I thought, actually.

What *was* a hideous amount of code was the Panose matching for the font
substitution engine --- getting Panose information from Windows is like
pulling hen's teeth! Not to mention that some fonts just don't have any
Panose information, so I just have to make it up.

So I now have code where it'll build a cache of individual bitmaps, each
containing a single glyph in a particular style. The first time the app
wants to render a particular glyph it'll enumerate down a list of fonts
sorted by Panose difference from the user's selected font until it find
one which supports the glyph. Once the cache is populated, the screen
can be redrawn very quickly (and flicker-free, too) by blitting bitmaps
from the cache onto the screen. I believe I'm very close to achieving
redraw nirvana, which is to touch every pixel in the window exactly once.

Unfortunately, what I'm discovering is that GetGlyphIndicesW() is
outright *lying* to me about whether a font contains a particular glyph
or not. Specifically, it's telling me that some fonts --- FixedSys [*],
for example --- contain all glyphs. As I result, I'm still getting
glyph-not-found blocks instead of the Unicode characters I'm looking for.

Does anyone know of a more accurate way of determining whether a font
contains a genuine representation of a particular glyph?



[*] Yes, I know you're going to tell me not to use fonts like FixedSys
if I want to do Unicode. Unfortunately this is fundamentally a console
application; I'm having to write my own console because the standard
Windows console has lousy Unicode support. As such, I know damned well
that my users are going to want to use all the traditional bitmap
console fonts like FixedSys. So I still have to make it work.

-- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ life←{ ↑1 ⍵∨.^3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵ }
│ --- Conway's Game Of Life, in one line of APL

David Given wrote:
> On 22/01/10 00:07, David Given wrote:
> [...]
>> This does sound like my best bet. I can do all the complicated work to
>> try and determine which font to use for a glyph lazily, the first time
>> the app tries to draw the glyph (it has to be lazy because I don't know
>> what Unicode subset the app will be using).
> 
> Well, I've done that; it wasn't as much code as I thought, actually.
> 
>[..] Once the cache is populated, the screen
> can be redrawn very quickly (and flicker-free, too) by blitting bitmaps
> from the cache onto the screen. I believe I'm very close to achieving
> redraw nirvana, which is to touch every pixel in the window exactly once.

"Nirvana ... mmmmmhhhh ..." (Homer Simpson)
Yes, that's what I was hoping for. Using your own caching system will 
defeat Windows' built-in routines with their massive overhead. The only 
case where Windows *could* have been faster is if it would use back-door 
techniques, such as caching & blitting from video memory. (And 
apparently it does not. Should I copyright that idea right now?)

Be sure to test your caching on a few different video cards (or with 
different settings on the same card). You never know -- you might have 
to include an "unchached" fallback routine.

> Unfortunately, what I'm discovering is that GetGlyphIndicesW() is
> outright *lying* to me about whether a font contains a particular glyph
> or not. Specifically, it's telling me that some fonts --- FixedSys [*],
> for example --- contain all glyphs. As I result, I'm still getting
> glyph-not-found blocks instead of the Unicode characters I'm looking for.
> 
> Does anyone know of a more accurate way of determining whether a font
> contains a genuine representation of a particular glyph?

GetFontUnicodeRanges perhaps? The MSDN suggests it returns Unicode 
/ranges/ rather than individual /characters/, but that could actually be 
an advantage, as (for example) all Hebrew characters would come from one 
and the same font, all Arabics from another, etc.

> [*] Yes, I know you're going to tell me not to use fonts like FixedSys
> if I want to do Unicode. Unfortunately this is fundamentally a console
> application; I'm having to write my own console because the standard
> Windows console has lousy Unicode support. As such, I know damned well
> that my users are going to want to use all the traditional bitmap
> console fonts like FixedSys. So I still have to make it work.

Heh :-)

[Jw]

On 25/01/10 11:09, [Jongware] wrote:
[...]
> GetFontUnicodeRanges perhaps? The MSDN suggests it returns Unicode
> /ranges/ rather than individual /characters/, but that could actually be
> an advantage, as (for example) all Hebrew characters would come from one
> and the same font, all Arabics from another, etc.

Yeah, yeah, I know I'm replying to an ancient post, but I've finally
fixed my problem and wish to comment for the record. Thusly:

GetGlyphIndicesW() *does not work*. Apparently Microsoft have said so.
It returns inaccurate results.

GetFontUnicodeRanges() *does* work, but it doesn't do what it says on
the tin --- it doesn't return Unicode Ranges, as specified by Unicode,
but instead it returns the full coverage of the Unicode font as a list
of ranges consisting of offset and length. So it does give you complete
code point granularity. This is not obvious until you look at the
documentation in some detail. (How can MSDN be both so detailed and so
useless at the same time?)

So now it's all working. Well, mostly --- I still have problems with
Unicode code points that don't fit in a 16-bit value, and there are a
few oddities like being unable to create instances of the Terminal font
despite the font chooser letting me select it, but it's all vastly improved.

If anyone knows the Panose descriptor for FixedSys, please let me know...

-- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│
│ life←{ ↑1 ⍵∨.^3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵ }
│ --- Conway's Game Of Life, in one line of APL