Tuesday, November 17, 2009

NewCompiler

I want to use the NewCompiler in SecureSqueak.

The main reason for this is that I want to use its IR ("intermediate representation") objects for moving code between images. The IR is effectively the bytecodes, but in object form, one object for one instruction, such that a final compilation step can be run on them to make CompiledMethods.

This is important in SecureSqueak as it makes implementing a "bytecode verifier" much easier. Instead of verifying the bytecodes for incorrect variable accesses, invalid jump destinations and so forth, we just get the kernel to run the last step of compilation itself. This makes it impossible (*ahem* less possible) for malicious bytecodes to be injected into SecureSqueak.

The rest of the compiler is completely optional. You merely need *something* that can generate the IR — it doesn't even need to be Smalltalk code.

The current NewCompiler maintainers use Pharo, so I'll be moving development over to that for now.

Tuesday, September 22, 2009

UGP / SecureSqueak 0.1 released

Download from here: http://sourceforge.net/projects/securesqueak/
Or watch the video: http://vimeo.com/6698843.

This is the first release of my "Unnamed Grand Project". It includes the SecureSqueak kernel (kind of), SiteBrowser, Subcanvas and the DPON object replication framework. Currently the "SecureSqueak kernel" is just the Squeak 3.10 kernel converted to use namespaces. Version 0.2 of UGP will start introducing security into it.

This release is for show and tell only; it barely works and isn't at all secure.

Sunday, September 6, 2009

Subcanvas, with windows!


...and they actually work, too! The yellow parts are the borders; you can drag them around, and clicking on them brings them to the front.

The Subcanvas API is starting to prove that it really can work. The yellow bits are a subcanvas. Each of those contains a subcanvas with the window contents. The background is a subcanvas. The buttons (grey things with text) are subcanvases and will pop up a brand new window when you click on them.

Next up: choose better colours and make the test sites actually do something.

Tuesday, September 1, 2009

Subcanvas: Introducing Easels

Last night I started writing a real test for my Subcanvas graphics/events architecture: a trivial windowing system as the humble beginnings of a SiteBrowser [A SiteBrowser lets you browse "Sites" across the Internet, similar to navigating between Projects in Squeak]. The first problem I encountered was: from ordinary application code, how do I add a subcanvas? You aren't meant to hold references to canvases in your code, and the method to add a canvas is Canvas>>addCanvas. That would mean that you can only add a canvas from within an event handler. Oops. I needed to refactor the API to make it less brain-dead.

As a result, I introduce the Easel, an object which specifies the structure of Canvases and their children. An Easel is an object that your code should hold a reference to, and its API contains all the methods used to add, remove, move, resize, raise and lower subcanvases. I've also moved the canvas handler accessing methods to it, and added a method to cause a redraw event on a canvas.

I'll be updating http://gulik.pbwiki.com/Canvas soon.

Saturday, August 1, 2009

Subcanvas with events


What is the picture of? It's a simple test for the Subcanvas graphics / events API. It's Squeak, but with Morphic temporarily disabled and a Subcanvas test running full-screen and reacting to keyboard and mouse events. The blue squares are from mouse move events (offset by a few mm). The big green rectangle is a subcanvas that draws little red squares rather than blue squares on mouse move events. All the other sized squares are from mouse click events. The text is from keyboard events; the little white lines above text are key press events and the white lines below are key release events.

Font rendering here is done very simply. It will be improved.

The performance of this implementation is currently usable with a single, small subcanvas, but the performance will suffer incredibly as subcanvases get bigger. This is because subcanvases are BitBltted in their entirety after each event; the implementation is currently naïve and there is a lot of room for optimisation.

I'm disappointed by Squeak's graphics performance. BitBlts on Forms are very slow. As a result, I'm not going to try to optimise this implementation of Subcanvas (based on Forms and BitBlt) and instead re-implement it in the future using either a raw X11 socket with XRender, XDamage etc, or use OpenGL in 2-D. Subcanvas at heart is an API, so any users of it should be unaffected. Currently I'm favouring reimplementing it to use X11 as most POSIX systems (Linux, Solaris, BSD, Apple with OS/X) have an accelerated X server, and I could bundle something like XMing with MS Windows. Forms, BitBLT and friends could then be completely purged from the VM leaving it headless.

Sunday, June 28, 2009

Keyboard events

Mouse events in Subcanvas now mostly work. A canvas's target now receives mouse movement events, mouse button press and release events and canvas enter and exit events. Now, I've moved my attention to keyboard events.

There are three types of keyboard events in Squeak: Key Press, Character and Key Release. Key Press and Key Release should give only the description of which physical key on the keyboard has been pressed or released, including modifier keys (shift, CTRL, etc). Currently the Squeak VM uses the "Mac-Roman" encoding to encode which key was pressed or released, which is less than ideal and still differs per platform. The Character event contains a Unicode character. A single Unicode character may have required multiple key presses and even some UI interaction to be composed.

There are three types of hardware keyboard in the world; the US one (104 keys), the European one (105 keys) and the East Asian one (109 keys). These are very similar to each other and based on the PS/2 keyboard. Almost every keyboard in the world is based on one of these hardware prototypes with different printed letters on it.

I've also found that the USB specification (made by Microsoft) has reasonably elegant "scan-codes" that specify the actual keys pressed with no regard for which character is printed on that key. I believe these codes would provide for a much more elegant way of specifying which keys were pressed in the "press" and "release" events than the Mac-Roman keys currently provided by the Squeak VM. Unfortunately, any notion of the original USB scan codes have long been forgotten by the OS by the time the character arrives at the Squeak VM, so the VM or the image will need to map OS-specific event IDs to the equivalent USB scan codes. It is a bit of a roundabout way to do it, but the end result is an OS-independent manner of doing raw keyboard events. Perhaps SqueakNOS can strip away the layers of abstraction and pass USB scan-codes through directly? :-)

The end result is that an application using Subcanvas can either use the character events, or it can use the key press and key release events to roll its own "Input Method Editor". This gives a good level of initial functionality for most languages in the world by using the Unicode character input directly from the VM, as well as a future growth path for implementing new "Input Method Editors" (or games) using the raw key press and release events.

On another chain of thought, I'm considering making Pango and Freetype "libraries" available in SecureSqueak / UGP. If they're provided as a resource such that Subcanvas does not have any architectural dependencies on them, they can be implemented as a "deprecated on birth" resource that will provide necessary text rendering functionality until pure Smalltalk-based solutions can be written. At this stage, I'm a bit wary of just how complex the rendering of multilingual text can be.

Saturday, April 11, 2009

Squeak on ARM

I'm trying to get Squeak running on my HP Jornada 820. I've used an ancient 3.3 VM until now under Windows CE, but I'd like to get it running under Linux to see if I can improve performance a bit, especially with BitBLT which I know can perform a lot faster on this hardware.

I've learned one lesson: embedded programming requires patience. Everything is slow.

So far I have:

  • Gotten Linux (2.4) running on the Jornada from a CF card. This was the easy part.
  • Installed Debian Sarge (3.3) from the dusty archives of yore. You see, it needs to support the aforementioned 2.4 kernel.
  • Gotten GCC working, which needs a swap space on the CF card. I can hear it screaming in pain! "Hello, world!\n" works.
  • Nearly managed to get Squeak to compile. It failed on the Rome plug-in, which was internal.
  • Stripped Squeak of all plugins and discovered an unnecessary dependency on a Balloon 3D plugin. It's easy to fix, but I have better things to do.
  • Decided that I couldn't cause any more suffering on my CF card and now I'm setting up a Scratchbox environment for cross-compilation.

Currently, I'm stuck on the executables from the cross-compilation environment. GCC on the Jornada makes ELF binaries for "GNU/Linux 2.2" that run fine, but the cross-compiler makes ELF executables with no such versioning... which don't run. Google will inform me.

Monday, March 23, 2009

Concurrency

I'm depressed.

Over the last couple of weeks, I've been playing with various Smalltalk implementations. I always had this idea that some of them could use multiple CPU cores. As it turns out, none of them do it, and the ones that do concurrency don't do it well.

VisualWorks
is single-threaded. Squeak is single-threaded. Smalltalk/X didn't fully use two CPU cores. Smalltalk/MT should, but didn't. Huemul 0.7 tried... but crashed.

[update: Smalltalk/MT does use multiple cores as Peter Lount points out in the comments, but is severely limitied by the garbage collector.]

Igor's Hydra VM does concurrency the hard way: by making a completely separate VM in a separate thread but in the same process (using OS terminology here): VMs can communicate with each other, but each is single-threaded. Gemstone, I hear, can make multiple Gems which share a transacted memory, but each Gem is single threaded.

Surely it can't be that hard to make a VM that does fine-grained concurrency?

Smalltalk, the language, is ideally suited to concurrency. The language lends itself to creating concurrent abstractions and already has basic support, but there's no implementation which can take advantage of the CPU power.

Smalltalk VMs are not going to be running any faster if we don't start exploring concurrency. The free ride in MHz increases is about to end soon (in theory, anyway). Moreso, the best Smalltalk VM can only use quarter of the power of a quad-core CPU, and this unused potential will increase exponentially as the number of cores increases exponentially.

What am I going to do about it? Well, I'm going to be modifying the Process scheduler in Squeak to simulate multiple CPU cores. Individual Processes will have their speed throttled down so that using multiple forked processes will be required to get a speed increase. Then I will, eventually, start writing concurrent frameworks to take advantage of the "multiple cores". This way, if some smart alec makes me a nice concurrent VM, they have concurrent code to take advantage of it.

Thursday, March 12, 2009

Playing with Blocks

Today I learned about blocks. We Smalltalkers all know and love blocks: those bits of code in square brackets that we can do all sorts of dandy tricks with. So what evils are there?


Well, Evil Trick number one:

b := [ ^ Transcript show: 'In the block'; cr ].

... in another method:

b value.
Transcript show: 'After the block evaluation'; cr.

The latter message will not be printed on the transcript. When you evaluate a block that has a return statement in it (a "non-local" return), the current context is abruptly terminated and the method context where the block was defined is returned from.


Evil trick number two:

b := [ ^ 1 ].
[ b value ] fork.

This will cause >>cannotReturn: to be evaluated on the BlockContext. The block has a non-local return in it but it cannot return. The containing MethodContext of that block is not on the call stack, because the block is evaluated in another process which has its another stack. When a non-local return happens, the VM searches down the call stack until it finds the method that the block is defined in.

This will also happen if you get a block with a non-local return by calling a method that defines it and then evaluate that block after the method returns:

someMethod
^ [ ^ 1 ].

self someMethod value. "Fails"


Evil trick number three:

s := Semaphore new.
b := [ s wait ].
[ b value ] fork.
[ b value ] fork.

This will fail in Squeak, although it shouldn't. Squeak has an optimisation that prevents two concurrent evaluations of the same block. I'm not entirely sure why, but it certainly hinders concurrent programming in Squeak (that, and the fact that Squeak is single threaded).

It also happens if you get a block to evaluate itself:

b := [:block | block value: block].
b value: b


Evil trick number four:

[ |b| b := 'bar'] value.
[ |b| ^ b] value.

You would expect to get a nil, but in Squeak, you'll get a 'bar' instead. This is because block variables are defined in their method contexts rather than in their block contexts. This is what the community is complaining about when they complain about the lack of closure support. So, how do you fix it?

[ |b| b := 'bar'] value.
[ |b| ^ b] fixTemps value.

This version does not work: fixTemps disallows a block from doing non-local returns. How about:

| result |
[ |b| b := 'bar'] value.
[ |b| result := b] fixTemps value.
^ result.

Lame, but it actually works like it should. nil is returned.

Sunday, February 22, 2009

Low-level graphics in Squeak

It wasn't until I dug into how low-level graphics worked in Squeak that I realised how trivial it was.

To draw on the screen, use the global variable "Display". This is an instance of a subclass of Form. A Form is a bitmap. To draw on Forms, you use "BitBlt" instances, which represent drawing commands.

For handling events from the mouse and keyboard, you use the global variable Sensor, which is an instance of EventSensor. In a loop, you call EventSensor>>nextEvent to get the next event, which is either "nil" or the next keyboard or mouse event. Yes, you need to poll it, and if you don't want to use 100% CPU, then you need to suspend your thread for, say, 20ms on every iteration.

Except for the details, that's all you need to do to draw on the screen and handle events. There's plenty of example code in a stock Squeak image.

Now comes the tricky part: how do you wrestle control of these objects from Morphic? Well, as it turns out, Morphic wasn't designed in the resiliant, secure and reactive way I would have designed it. Morphic is single threaded! When you run your code, you're running it in the same Process that handles event handling and screen drawing! So if you never return from your event polling loop, Morphic never runs (unless a Transcript is open and you write something to it).

I find the single-threadedness of Morphic rather quaint.

Alternatively, you can use:
Project uiProcess suspend. "This suspends the Morphic process; make sure your code runs in another process!"
Project spawnNewProcess " If you terminated the Morphic process, this makes a fresh one. "

Sunday, February 8, 2009

SiteBrowser

Last night I started work on a new SiteBrowser. This time, it will only use the Subcanvas API to do drawing and event management.

Hopefully it won't take much more time before I can release SecureSqueak version 0.1. I think I'll revise the schedule to make lots more releases each with fewer new features.

In the meanwhile, Matthew Fulmer has sent a progress report about Squeak 4.0. I'm not sure if that should affect me. My current state of thinking is that I'll be making such vast changes to the kernel that it doesn't really matter which version I begin with.

Monday, January 19, 2009

Subcanvas progress

I'm always impressed by how quick you can get stuff working in Smalltalk and Squeak. I spent a couple of hours over the last weekend getting events working in Subcanvas. I haven't touched it for a month now as I've only just come back from a 3-week holiday with my family.

So now Subcanvas can process left mouse button clicks. I coded up a small demo where the mouse click handler drew small 5mm x 5mm squares on the screen. You can add a child canvas, and the events and redrawing works in the simple case.