Yajaf is retired

I'm posting this to say that YAJAF has been officially closed. I'm sure it's probably obvious since I haven't posted in several months. I had intended to keep writing javascript articles, but my new project has eaten 100% of my time. To those of you who have been following the articles, thanks for reading! I may come back when I have time and write additional informational pieces but the way this new thing is taking off I can't make any promises.

So what happened with YAJAF? Aside from some of the really nasty toned comments I got, indignant that I would even *dare* to try and make money off of a javascript library, there just wasn't that much interest. While I still think YAJAF is better than 90% of the libraries I've seen (boy are they proliferating), some of them are starting to incorporate some of the neat features in YAJAF (although not all) and some of the newer libraries have features YAJAF is missing. I think given enough time, the current popular libraries will evolve (and combine) to become better than YAJAF so it doesn't make much sense for me to keep developing it. I thoght about open sourcing it but there didn't even seem to be enough interest for that. Additionally, while I love open source as much as the next guy, I gotta eat. :) So I've come to the decision that I'm going to keep YAJAF closed and use it for my own internal projects. Thanks to those of you that gave me constructive comments!

Slower Ninja Foo

I'm still furiously working on my YAJAF enabled project. The 'ninja foo' articles will be slower in coming but I will try not to be overly negligent. I'm hoping to have some demos ready in the next couple weeks. Stay tuned...

jmnf 9: Practical Garbage Collection

javascript magic-ninja-foo episode 9: Practical Garbage Collection part 1

In this episode I'm going to briefly touch on the theory of garbage collection and then continue in the next episode with some strategies for overcoming common problems that arise due to garbage collection in javascript.

Theory

For those not familiar with the idea of garbage collection, it's an automatic memory management system for use in writing and running programs. Many newer (and some older) programming languages use some idea of garbage collection, a common example being Java. There are pluses and minuses to using a garbage collector, but that discussion is beyond the scope of this article. Suffice to say that javascript interpreters, by definition, implement one and therefore merit some brief reflection.

In essence, garbage collection alleviates the programmer from having to be disciplined in the management of system memory. In languages not using garbage collection (or some other automatic memory management system), the onus of allocating and freeing memory is on the programmer. Frequently in 'programmer controlled' memory management systems, functional bugs, syntax errors, inexperience, or even just sloppiness can lead to a condition known as a memory leak.

Memory Leaks

A memory leak is a condition where a program has allocated some amount of memory for program use, but fails to release it when no longer needed. For small, short-lived programs it might not be a problem. Exiting the 'runtime' state of the program might automatically free the memory or the operating system may 'scrub' the memory depending on the language/OS/implementation.

The worst case scenario, however, is either when the program has a long run-time and/or there are no 'safety nets' to release allocated memory. Over time the continuous allocation of memory without freeing gradually consumes more and more of the system until finally, the machine has no more available memory for program use, usually resulting in a program crash or worse, a system crash.

Implementation Theory

The basic theory of object oriented garbage collection is that there is a chunk of memory controlled by the garbage collector (the heap) where objects are stored when instantiated. When references exist pointing to an object in memory from the application context, some way even if indirect of accessing that object, its memory persists. If the garbage collector realizes that there is no way for the application to 'reach' objects in memory, no references exist, then the garbage collector reclaims the memory used by the object.

There are a couple of common ways of implementing a garbage collector. The first and most common is called tracing garbage collection. Tracing garbage collectors usually use explicit data 'sets' of objects in various states of reach-ability (usually three, sometimes called 'three color'). During garbage collection cycles, these states are updated with one state being reclaimable. An example of this type is the 'mark and sweep' algorithm. These garbage collectors are usually slower and more complicated than the the other common implementation.

The other common way to build a garbage collecting system involves an algorithm called reference counting. Reference counting uses a small data structure in every object instantiated that increments for every reference pointing to it, and decrements whenever those references are destroyed. When the reference count reaches zero, the object is free to be garbage collected (with there being no way to reach the object). Reference counting garbage collectors are much easier to implement but can be limited.

Practical implications of GC using javascript

Garbage collectors implemented using reference counting algorithms have some direct impact for javascript programmers. Part of the appeal of garbage collection is the abstraction of memory management away from the programmer so they can remain ignorant of the inner workings of its operation. Ironically certain implementations of javascript interpreters, namely the js interpreter included with Internet Explorer (as of version 6) by Microsoft, require us to go into the inner workings of its garbage collector so that we can write effective code. IE's javascript interpreter uses a 'reference counting' garbage collector that, unfortunately, has several shortcomings.

The biggest issue with reference counting garbage collectors, if not architected carefully, is that they may miss some circular references. Imagine an application creates two objects, each with a reference to the other. Each object has a count of 2 (one from the application, and one from the other object). When the application discards its reference to the two objects, each of their counts are decremented by 1, but they're still not zero (each still has one reference to the other) even though they are no longer reachable from the application. This means that if the reference counting garbage collector isn't careful, it will not reclaim the memory used resulting in a memory leak.


Most reference counting garbage collectors implement strategies to avoid this scenario. In the above simple example, IE (version 6) does reclaim the offending memory. There are, however, other more complex cases where it doesn't. This seemingly arbitrary nature can make debugging IE memory leaks very frustrating. Ironically even Microsoft's own description of the leak pattern doesn't actually leak in IE 6.


While the scenario described in the Microsoft article doesn't leak, it's not difficult to come across the memory leak problem. It took a little time but I came up with the following example javascript that actually does leak in IE 6.

function TestMessage(message,target) {
   /* Function with state, outer closure arguments*/
   function MyInnerFunction() {
      function display() {
         var msg = document.createTextNode(message+" ");
         target.appendChild(msg);
      }
      return display;
   }
   /*assign inner function to object property*/
   this.display = MyInnerFunction();
}

/*create DOM element*/
var testdiv = document.createElement('div');

/*create new object*/
myMessage = new TestMessage("Event Handled!",testdiv);

/*make the DOM element visible and 
  assign event handler */
testdiv.style.position = "absolute";
testdiv.style.height = "100%";
testdiv.style.width = "100%";
testdiv.onmousedown = myMessage.display;

I have some DOM methods in the above code. Don't worry if you're not familiar with these functions, I'll discuss them in more detail in a later article. Essentially the snippet is creating an element that covers the 'body' content area that when clicked shows the message "Event Handled!".

While this code is lexically simple, the actual closures, execution objects, and what's referencing what is actually fairly complicated. I leave it as an exercise for the reader to diagram out the closures, scopes, and persistent execution objects represented by the above code. You can see why the reference counting garbage collector in IE6 could become tripped up by complicated circular reference patterns.

Putting this code into a web page you can witness the memory leak for yourself. Try hitting 'reload' a few times on the page with the 'Task Manager' open showing processes. Each reload should show 'iexplore' memory usage increasing from 2k - 20k per reload. Even if you navigate away from the page, that memory is now locked up by the IE garbage collector. The only way to reclaim the memory is to shut down IE and restart it. A 20k leak per reload may seem trivial but I've seen example leaks that consume 100's of K per reload depending on the complexity of the circular references. If multiple pages contained memory leaks of this magnitude, problems could arise very quickly.

Generally you want to look for the following two types of javascript that can cause memory leaks in the internet explorer javascript interpreter.

• Using complex nested closures. Remember each closure creates it's own scope / execution object that can persist if there are references to its member properties.
  
• Crossing the javascript / DOM boundary. Assigning properties to DOM elements that point to javascript objects, especially nested closures.

The second case becomes very common when using javascript object methods as event handlers for DOM objects.

In the next article I will discuss some strategies for avoiding these scenarios.

Two Week Neglect

I must apologize to my regular javascript-ninja-foo readers since I'm in the middle of a project where time is of the essence. Hopefully within the next two weeks I will post the next in the series "Episode 9: Practical Garbage Collection".

To those seeking YAJAF, the library and it's release status are in a bit of flux. Hopefully the new project will fully showcase the neat aspects of the library. Stay tuned!

jmnf 8: stricter encapsulation

javascript magic-ninja-foo episode 8: Objects part 5 stricter encapsulation.

This is the second part in a discussion on encapsulation in javascript. The previous article should have been mostly obvious to the regular reader, or those intimately familiar with object oriented concepts but I felt it was necessary to provide a simple foundation for encapsulation.

There is an alternate way to enforce stricter encapsulation, although this approach incurs tradeoffs. Here again the articles converge. Remembering functions with state, and understanding scope, we can begin to get some ideas on how this might be accomplished.

To illustrate this I'm going to redefine the constructor function from the previous article in a slightly different way. Again, if you're a regular reader, you might already see the implications of javascript written in this way.


(Paste the examples into the Yajaf shell window to try them out)

/* Constructor function */
function Orange() {
   var color = "orange";
   var type = "citrus";

   /*color setter - lexical closure*/
   function setColor(incolor) {
      color = incolor;
   }

   /*color getter - lexical closure*/
   function getColor() {
      return color;
   }

   /*assign the inner functions as
     methods to this the new
     object*/
   this.setColor = setColor;
   this.getColor = getColor;
}
/*make an orange object*/
var myOrange = new Orange();
/*set then get, blue oranges?*/
myOrange.setColor("blue");
alert(myOrange.getColor());

The first thing to note is that we've defined a few variables using the var keyword inside the constructor function. I've said before that variables and properties of objects are fundamentally the same thing. These 'inner' variables become properties of the 'execution object' that is created during the execution of the constructor function.

Normally, once the constructor function has completed, the execution object is garbage collected. However, since we've assigned the inner lexically scoped functions (episode 3) as methods on the new object, we've essentially tied the newly created object to the execution object keeping the execution object from being garbage collected. This technique is similar to 'functions with state' (episode 3) except that we can 'return' as many methods as we wish pointing into the execution object rather than a single return function. This is accomplished using the temporary binding of the new and 'execution object' that occurs during object creation through the 'this' keyword (episode 4).

There is no direct way to access the 'execution object' therefore there is no easy way to circumvent the getters and setters to access the two "var" defined variables. This effectively makes them strictly private. While the new object exists we continue to hold onto the 'execution object' to act as a private container to hold state variables. This effectively provides the illusion of a single object with potentially private instance variables and methods.

It might be easier to define what's going on graphically. We'll start by talking about what happens in the example from the previous article and then how this approach differs. Hopefully this will help in understanding what's going on but also begin to show what the tradeoffs are for using this approach..

The first case:

The key here is that the constructor function 'execution object' does it's job, sets up the properties of the new object, and then is GC'd after it's work is done.




The second case:

In this case, however, by having external references to the 'execution object' it remains in existence. Since the 'getter and setters' were defined inside the constructor function, they have access to the 'outer' lexical scope of that function (the var variables).




There are two prices to pay for this approach the first is flexibility. In the previous article, when I decided to add boundary checks to the setter method I intentionally dynamically altered the setter method during runtime. Taking advantage of the fluid nature of objects, I was able to redefine the setter method without stopping and restarting the application.

With this 'strict' approach, once the constructor function completes there is no longer any reference to its 'execution object'. This makes it impossible to alter any method operating directly on the 'execution object's' properties, effectively keeping them private. You could still create a 'wrapper' method for boundary checks, but the actual method defined inside the 'execution object' that has access to the 'private' variables can no longer be altered. This is either a positive or a negative depending on how you write javascript code.

The second, more important, price to pay for this is memory. The case described in the previous article could be argued to be very efficient. Function objects exist only while executing and then are garbage collected. In the latter case, for every object you create that has strict 'private' variables, you're creating two objects, one to hold your private state variables and methods (by hanging onto the 'execution object'), and another to hold your public state variables and methods. The public methods act as the 'gateway' to access those methods and variables within the 'private' object.

It's also possible to mix and match these approaches. Objects that need 'private' methods and variables could be designed with the latter approach, and objects with less stringent requirements could be made with the former approach.

Javascript is extremely flexible, so there are likely other ways to accomplish this same end effect. As always, your mileage may vary.


Next Episode

What's special about YAJAF

Everyone seemingly likes to bash javascript, but on a fundamental level, it's actually a fairly powerful language. When I say "powerful" I mean in the sense that Paul Graham uses in his essay "Beating The Averages", where powerful denotes a language that has more "abstractness" and ability to do what you need it to do. I'm not a lisp programmer although I have the goal to one day learn. This means that, according to Paul, I still look 'up' at lisp not exactly understanding what's going on with all those parenthesis. Over my time hacking with javascript, though, I've developed this strange feeling that javascript is actually 'higher' on the power continuum than C++ and Java.

Now I'm sure I'm inviting angry comments by saying that, but what I mean is that by giving you more access to the structure of the programming language itself, javascript lets you do some interesting things. You can actually write programs in a 'functional programming' paradigm, creating functions that take functions as arguments and return functions as results. You can also write object oriented code, or just procedural code if you wish. The fact that javascript is so flexible is something that I think is commonly overlooked in the language.

Almost all the libraries I've seen have taken the common procedural programming meme that was pervasive in javascript from the early browser days and added some more 'magic functions', sometimes to create objects and sometimes to create complicated widgets or effects. This does work, but for me, is a strange approach to designing UI elements and feels unnatural.

What was my objective with writing YAJAF? Well quoting Paul Graham:

Ordinarily technology changes fast. But programming languages are different: programming languages are not just technology, but what programmers think in. They're half technology and half religion.

Everyone (me included) who develops code can be described by this statement. When I started writing YAJAF, I had a few goals:

• Develop code that runs in a browser
• Not require an end user to download anything special (e.g. Flash) to execute the code, a seamless experience
• Work across as many modern browsers as possible.

All of these goals pointed to one thing: using javascript, something I was *loathe* to do. I thought well if I *have* to use javascript I'm going to see if I can create some way to make it feel more natural to me coming from Java, C# and other 'strict' OO programming languages. I thought I was going to have to create some really ugly hacks to make javascript work in an object oriented way. The more I investigated, though, the more I realized the language lends itself to being 'twisted' to do what you want it to.

As I began this project, I kept coming back to the idea in that quote. I started creating a way to write javascript that looks and feels like Java or C# (what I was used to). A thought kept popping up in the back of my mind, surely someone else would be interested in this, no? There are a lot of people with Java and C++/C# religion out there.

I have a design philosophy that great design taps into the tacit knowledge of those you are designing for (why newfangled keyboards, with strange key layouts, will never catch on). Programmers are used to writing code a certain way, with a certain religion. How much UI design experience is out there for Java, C++, and C#? Wouldn't it be great if that could be tapped for this next stage of software development, web applications?

This leads me to one of the big questions people have asked me, is YAJAF a framework or a library? Well in all honesty, the framework piece is very small. The framework involves a few tricks, manipulating javascript itself, and a couple of very small 'autogen' pieces of code to make javascript feel friendlier coming from other OO languages (specifically Java, the last language I did any significant programming in). The windowing toolkit, data io manager, event manager, error handler and other constructs are all part of the library. Unlike other 'frameworks' the goal is not to create ready-bake code skeletons that are then populated, but to create a solid foundation for writing javascript.

All this boils down to the following piece of YAJAF code that could be executed anywhere Yajaf is loaded.

var myPanel = new FX.FXWT.FXpanel();
var myFrame = new FX.FXWT.FXframe();
myFrame.setHW(100,200);
myFrame.setXY(50,50);
myFrame.add(myPanel);

var myButton = new FX.FXWT.FXbutton("press me!")

myPanel.add(myButton);

desktop.add(myFrame);

Voila, we've crated a frame widget with a panel, and a button on the panel.

All of that though is the YAJAF library in action, again mimicking Java. The two things that are special, first I can write code that uses a library in a way thats familiar, and second, the actual library components are written in such a way that I can extend the behavior of them in any way I wish. I could create new 'classes' extending the base 'classes' and adding custom code.

I've come to the decision that I'm going to start opening up the YAJAF framework, but I need to polish off a couple of the rough edges of the framework before I do. Hopefully this quick essay will help those visiting understand what I think makes this framework different.

jmnf 7: encapsulation

javascript magic-ninja-foo episode 7: Objects part 4 encapsulation.

We're now going to start to delve a little more deeply into object orientation in javascript. Continuing down the list of of characteristics of object oriented programming languages (described in episode 5), we're going to look at encapsulation and how it can be accomplished in javascript.

In a nutshell, encapsulation means avoiding directly altering the value of instance variables in an object. Isolating the variables has two primary benefits: one it allows you to do boundary checks on values and two, by creating a "gateway" to your object's data, you create an interface you can alter at a later point without disrupting other OO code that uses your object. There are some other implications of encapsulation but they're beyond the scope of this article.

When we wish to access instance variables, we use special methods frequently called 'getters and setters' (or more formally accessors and mutators) to manipulate and inspect the state of the object.

Nothing in the following code snippet should be surprising.
(Paste the examples into the Yajaf shell window to try them out)

/*color setter*/
function setColor(color) {
   this.color = color;
}
/*color getter*/
function getColor() {
   return this.color;
}
/* Constructor function */
function Orange() {
   this.color = "orange";
   this.type = "citrus";

   /*assign the functions as
     methods */
   this.setColor = setColor;
   this.getColor = getColor;
}
/*make an orange object*/
var myOrange = new Orange();
/*set then get, blue oranges?*/
myOrange.setColor("blue");
alert(myOrange.getColor());

We've created a 'color' getter and setter, so in a sense this is a form of encapsulation. You'll notice that we set 'color' to blue in the example. Maybe we have code that operates on the color of oranges (to what end I have no idea) but let's assume a blue orange has the potential to cause serious hiccups in our code (not to mention uncomfortable stares at the blue orange).

We might do a boundary check in the setter method to check the input value. Lets say we only allow green, brown, orange, and mottled colors. We're going to use the dynamic nature of javascript to change the setter method of just the object.

/*reset the color of the object*/
myOrange.setColor("orange");
/* Define a new setter */
function newSetter(color) {
   if( color != "orange" 
     || color != "brown" 
     || color != "green" 
     || color != "mottled" ) {
        alert("no "+color+
              " oranges allowed!");
        return;
        }
   this.color = color;
}
myOrange.setColor = newSetter;

/*try the blue orange again*/
myOrange.setColor("blue");
alert(myOrange.getColor());

This new boundary checking setter keeps us from setting the orange to blue (firing a browser alert to tell us so) and when we check the object we still have an orange orange.

To the veteran OO programmer though, it's obvious that there is a flaw in this object. The color instance variable is still accessible without a method. If we're not disciplined in our use of the getters and setters, we could just as easily modify the instance variable directly.

continuing from above example

myOrange.color = "sparkly";

alert(myOrange.getColor());

We've circumvented the setter by setting the object property directly. Depending on the complexity of your javascript code, this could cause serious headaches.

If we we're writing code in Java, or C++ we would use the 'public' and 'private' keywords to enforce strict encapsulation. They essentially define the external visibility of the members of the object (instance variables and methods). Setting instance variables to private and defining public "getter and setter" methods to manipulate them keeps us from inadvertently creating sparkly oranges.

There's no direct means of specifying access control to data members for objects in javascript, the keywords 'private' and 'public' don't exist. How then can we enforce strict encapsulation? I've heard javascript ninja-masters say that the above code and 'discipline' was the best way to accomplish encapsulation. There are however, a couple of other ways to accomplish stricter encapsulation, which I'll talk about in the next installment.

I've decided to split this article into two. Next, encapsulation part 2, a stricter approach

Yajaf shell

I spent some time throwing together a real time javascript shell (read eval loop type), so people who were learning could try out the examples immediately. I always try and make my javascript code work across all modern browsers, but in this case I was hitting a wall. Firefox worked great, but everything else acted strangely. Just like I mentioned in my previous posts, sometimes you just can't get away from ugly, hairy scope issues. After much wrangling with 'window.eval' (I'll talk about eval in a later post) I found a much more robust and feature rich shell out there. After looking at the js code in that, I found a neat trick (using location.href) to use the global window object for evaluation. I'm indebted to those guys for a new neat idea to get my shell up and working. I'm going to attach a link to my shell for the examples so that people can try it out.

Try it out now.

Arrow keys go up and down in history. No multi-line input. The only built in function is lp(OBJECT) which lists the properties of an object.