Cloudshift.cl

haXe, Node.js, Remote Data Sneak Peak

Sat, 29 Oct 2011 05:59:00 -0300

I recently posted a haxe data store I’m working on here

it has persistent hashes and indexable/relatable buckets.

example here https://github.com/cloudshift/Data/blob/master/usage/Test.hx

I have first cut of a remote client working, flavor ..

import cloudshift.Core;
using cloudshift.Mixin;
import cloudshift.Data;

class TestRemote {
public static function
main() {
Data.store(REMOTESQLITE(“http://localhost:8083/”)).then(function(store) {
store.bucket(“users”).then(function(users) {
users.where(‘blah=”elennena”’).then(function(recs) {
trace(recs.stringify());
});
users.insert({email:”lorena@ritchie.com”,name:”lore”,passwd:”and why not”}).then(function(u) {
trace(u);
});
});
});
}
}

This has the same bucket interface as the in-process version (couple of methods throw as not applicable on client).

Works over haXe remoting, and with a tweak to the neko code generator that I’ve already made to my js generator, would be client compatible with neko.

Any serverside bucket can be made available over http - via http async remoting - and security can be added with my session project.

Because proxy async works via haxe.Http I’ve had to modify haxe.Http and add a #if nodejs for requests, but I can ship this in the nodejs haxelib until Nicolas sees fit to recognize Node.js as a first class platform - but don’t hold your breath.

Node complements sqlite very well as being a great single threaded server on a very fast database that prefers non concurrent access. So now you don’t have to do your data access in-process and can spread the load over multiple buckets on multiple servers.

I’ve posted to github, I won’t complete til end of next week as there’s a long weekend here and I haven’t finalised the api.

Waking up to haXe Macros

Wed, 26 Oct 2011 16:21:00 -0300

I’ve been a bit slow to get into haXe macros but i found a use case that I wasn’t able to solve to my satisfaction any other way.

I blogged previously about a Part - a self contained component. A Part knows how to start/stop itself, can signal errors and exceptions and has it’s own set of events.

I wanted to combine Parts into Assemblies, where an Assembly knows what it’s parts are and contains the logic that specifies how these Parts interact, via the events the parts emit. An assembly is also a Part. So an Assembly is a typical collection class, which you can iterate over.

To retain typing when dealing with an Assembly and it’s constituent Parts one needs to use an Enum as a container, e.g. to specify an Assembly and it’s Parts I do this.

enum ViewParts {

LOGINVIEW(loginView:LoginView);
REGISTERVIEW(registerView:RegisterView);
LAUNCHVIEW(launchView:LaunchView);
BASEVIEW(baseView:BaseView);
CONTROLLER(controller:Controller);
}

_viewAsm = new ViewAssembly();

_viewAsm.add(REGISTERVIEW(new RegisterView(“#pageView”))); _viewAsm.add(LOGINVIEW(newLoginView(“#pageView”)));
_viewAsm.add(LAUNCHVIEW(new LaunchView(“#pageView”)));
_viewAsm.add(BASEVIEW(new BaseView()));
_viewAsm.add(CONTROLLER(new Controller()));

The whole point of an assembly is to provide controlling logic for the constituent parts, but the problem _was_ that adding the view to the collection as an enum greatly inhibits the ease at which you can get at the objects you want to manipulate, i.e. you need to use a switch to get at them.

So with a simple macro I’ve been able to remove this limitation. Below is the controller code from within the _viewAsm assembly which switches between the views, I’ve only included two cases for brevity. The main point to note is that I can access my views by name; loginView, registerView, controller etc. These are fully typed, no manual var declarations, no untyped string switches in sight, or unnecessary switch dereference to get an object from it’s container!

watch(function(vp,state) {
switch(vp) {
case LOGINVIEW(lv):
switch(state) {
case Started:
case Stopped:

case Error(s):
case Except(ex):
ex.toString().info();
case Event(s):
switch(s) {
case onLogin(email,passwd):
controller.doLogin(email,passwd);
case onRegisterView:
lv.stop();
registerView.start();
}
}

case REGISTERVIEW(rv):
switch(state) {
case Started:
rv.observe(function(op) {
switch(op) {
case onRegister(email,password,name):
controller.doRegister(email,password,name);
case onLoginView:
registerView.stop();
loginView.start();
}
});
default:
}
});

The Assembly class has a macro that adds for each constructor of the enum a new field of the correct type into the Assembly class, thus all the parts added may be accessed internally or externally by name. The Assembly, in a way, implements a Dynamic<ViewParts>.

The new assembly field name is created by the macro from the enum by extracting the var name, e.g. loginView from the constructor

LOGINVIEW(loginView:LoginView);

and adding it as haXe meta data to the enum. At runtime the meta data is extracted and applied as each new enum constructor is added.

What’s great with this is I get single instance dependency injection to boot!

Here’s how I attach the macro to the Assembly class (which ViewAssembly derives from)

@:autoBuild(cloudshift.core.PartAssemblyMacro.build())
class AssemblyImpl<T,Q> implements PartAssembly<T,Q>, implements Part<Q>{}

And here’s the macro

class PartAssemblyMacro {
@:macro public static function build() : Array<Field> {
var pos = haxe.macro.Context.currentPos();
var fields = haxe.macro.Context.getBuildFields();
var local = haxe.macro.Context.getLocalClass().get();

switch(local.superClass.params[0]) {
case TEnum(t,params):
var en = t.get();
for (n in en.names) {
var enumFld = en.constructs.get(n);
switch(enumFld.type) {
case TFun(args,ret):
for (a in args) {
var thispos = haxe.macro.Context.currentPos();
enumFld.meta.add(“woot”,[{ expr : EConst(CString(a.name)), pos : thispos }],thispos);
switch(a.t) {
case TInst(ref,prms):
var classDef = ref.get();
var myt = TPath({ pack : [], name : classDef.name, params : [], sub : null });
fields.push({ name : a.name, doc : null, meta : [], access : [APublic], kind : FVar(myt,null), pos : pos });
default:
}
}
default:
}
}
default:
}
return fields;
}
}

I haven’t included the runtime where I decode the metadata and add the field into the assembly instance but it’s trivial.

To sum up, I’m blown away by the possibilies now that I’ve woken up to haXe macros.

My Part/Assembly system will be posted on Github after a bit more testing as part of the https://github.com/cloudshift/Core

Unobtrusive Class Extensions

Tue, 25 Oct 2011 12:21:00 -0300

As a framework developer, one of my concerns is how to introduce functionality with as little disruption to the user of the framework as possible.

For example, inheritance as the primary mechanism of extension is used as the first resort for most users, hence if the framework forces the use of it’s base classes it becomes a larger question if it will be used.

Let me explain my problem. I want to integrate a component model into an application, I want to be able to start/stop components, to query their state, to observe them using an Observer pattern. But I don’t want to introduce a special base class to do this, this enables my component model to be more readily adopted.

One can implement an interface, but if your extension requires state, and has many methods it’s fairly intrusive.

One can use “using” to mixin static methods, but as a framework developer your mixin methods will not be tied to a user’s class but something more generic.

So the solution is to use a combination of the two methods above. In the code my component is called a Part, so I want to be able to mixin Part functionality with minimal intrusion to a user class, so I defined a Part interface with a single var.

interface Part {
var part_:Part_;
}

To use my component model the user must implement Part, to do so they have to use the real Part_ implementation class.

class MyClass extends MyBase , implements Part {

public var part_:Part_;
public function new() {
part_ = new Part_();
}

public function start() {
part_.notify(“started”);
}

}

The user requires to create a part_ var and initialize it - the part_ var provides any plumbing the extension requires within the class, e.g. in my Part I initialize an observable and have other functionality.

When one uses the class, you want the Part functionality without accessing part_, and this is where we can use “using” to make the extension as seamless as possible.

class PartExtensions {

public static var observe(Part:p,cb:Dynamic->Void){
p.part_.events.observe(cb);
}

}

Finally

import MyClass;
using PartExtensions;

var inst = new MyClass() ;

….

inst.observe(function(msg) { });

For two lines of code to get started the user can get full integration with Part extensions without having to rethink their class hierarchy. That’s pretty good without macros.

haXe, Node.js Object Store

Fri, 21 Oct 2011 06:24:00 -0300

I’ve posted here a Sqlite backed object store for haXe/Node.js, fully asynchronous and Promise based. Classes or typedefs are stored as serialized objects and there’s no class annotation required.

No spod as it’s not async. The design is based on initial work i did with pstore, an experimental Redis object store.

There is a persistent hash. Usage.

There is a concept of “buckets”. A bucket contains an object of a given type. Usage.

The bucket provides indexing by supporting “indexers”, functions that are run on your object to produce an index key before the object is serialized to the data store on an insert or update. Indexer values are indexed ultimately by sqlite indexes. Indexers are the query mechanism too. If your indexer returns null, the object will be ignored in searches on that index.

You may link objects between buckets, thereby producing relations.

Future directions will be more drivers as required probably Redis to start, command line backup/dump to json.

haXe, Node.js and Workers

Thu, 13 Oct 2011 19:39:00 -0300

Node.js is single threaded, so holding up the event loop with heavy processing is a no-no.

The way of solving this problem is to start a new sub-process that does the heavy lifting, connect to it’s stdin/stdout and then manage some kind of protocol between the main and the sub-process.

This is a great way of doing things, the main event loop fires off a sub-process and continues on it’s merry way until there is some data returned. Further, the OS can schedule the sub-processes over multiple cores and it removes an entire class of bugs associated with threaded programming.

Node makes it quite easy to perform this task with it’s childprocess function, however at each stage you’re continually thinking of a new protocol to talk to the sub-process over the wire.

haXe already has the remoting protocol, so all we need is an async stdio remoting protocol to take advantage of typed calls to sub-process.

I’ve just posted on github the means of doing this, here’s the worker that runs in a sub-process

import cloudshift.Worker;

class OtherProcess extends Worker {

public function foo(x,y,cb:Int->Void):Void {
cb(x+y);
}
}

A Worker extends haxe.remoting.Context, and addObject is provided with “this” as the target, thus all public functions can be workers.

and here’s the main process defined in the same source file as the Worker

class Test {
public static function main() {
Worker.run(Test);
}

public function new() {
var worker = Worker.create(OtherProcess);
worker.OtherProcess.foo.call([1,4],function(result) {
trace(“woot:”+result);
});
}

You can also create an AyncProxy for typed calls …

class OtherProxy extends haxe.remoting.AsyncProxy<OtherProcess> { }

…..

var _proxy = new OtherProxy(worker.OtherProcess);
_proxy.foo(2,5,function(res) {
trace(“proxy res:”+res);
});

So how does it work. Let’s say you compiled your app into test.js, thus to run you do

node test.js

on the first invocation, Worker.run(Test) runs your main class as the default.

However Worker.create(OtherProcess) calls a new instance of node as a childprocess like this

node test.js OtherProcess-someID

Worker.run, is called on the new instantiation of the subprocess and knows it must use OtherProcess, not the default Test, as the main entry point. Thus a new sub-process is started from the very same compilation unit.

One then adds the ability to do haXe remoting over the stdin/stdout channels of the sub-process and you have a very clean way of doing heavy lifting in Node.js in an non threaded async manner.

My code is first cut and nasty, but I was eager to share ;)

haXe, Node.js and Remotes

Sun, 09 Oct 2011 09:59:00 -0300

haxe remoting is a useful technique for transferring serialized objects between haxe platforms. Remoting can be used asynchronously over http.

Node.js although not an officially supported platform of haXe is available via these type signatures.

http://github.com/cloudshift/hx-node

That is, haxe can generate javascript from statically typed haxe code.

Node.js can be a great haxe remoting target with some tweaks. It needs to process http posts, decode the serialized object pass it to a function, gather the output and pass a serialized object back.

The main issue is haxe remoting on the backend assumes a blocking server, that is, the processing remote function must return a value, not callback a value. This doesn’t work well in a single threaded non-blocking server such as node. Thus the standard haxe.remote.HttpConnection.processRequest function needs altered as does haxe.remote.Context.

I’ve made the required tweaks here in this project which also provides an http server.

http://github.com/cloudshift/Http

You can create a haxe remote like this

var http = Http.server(“localhost”,8082,{ server: “MyServer 0.5”,root:”www” });
var remote = Http.remote(http,”/my/remote”);
remote.addObject(“MyRemote”,new MyRemotes());

where MyRemotes looks like this

class MyRemotes {
public function new() {}

public function foo(x:Int,y:Int,cb:Int->Void) {
cb(x+y);
}}

Notice we’re calling back rather than returning from foo. And on the client

var URL = “http://localhost:8082/my/remote”;
var cnx = haxe.remoting.HttpAsyncConnection.urlConnect(URL);
cnx.setErrorHandler( function(err) trace(“Error : “+Std.string(err)) );
var proxy = new MyRem(cnx.MyRemote);
cnx.MyRemote.foo.call([5,2],function(v) { trace(“value = “+v); });

This is precisely the normal haXe client code for an async remoting call. How about remoting proxies?

By creating a proxy class you may have typed calls on the client

import MyRemotes;
class MyRem extends haxe.remoting.AsyncProxy<MyRemotes> { }

The haXe code generator creates a typed proxy class automagically for you, allowing the following

var proxy = new MyRem(cnx.MyRemote);
proxy.foo(5,4,function(v) { trace(“value=”+v); });

However the default haxe js generator wants to try and serialize the final function which is incorrect, so use the custom generator provided like this

-js www/client.js-cp src
-cp ../http
—macro exclude(“MyRemotes”)
—macro cloudshift.http.JSGenerator.use()
-main TestClient

Clojurescript

Thu, 21 Jul 2011 16:32:00 -0400

Compiles to Javascript, has support for Nodejs.

I learned lisp and functional programming on Clojure a couple of years ago, but I’m not keen on the JVM, particularly the memory consumption, so I dropped it, and reverted to haXe, targetting Js and nodejs with my type sigs; coding in a more functional style, which has been very beneficial.

This exciting development allows me to use my platform of choice, nodejs, in, I’ll bet, the best supported lisp compiling to javascript that’s available.

I’ll be posting more on this as I get into it, but already the clojurescript use of the google closure compiler has tweaked me to add that to my haXe pipeline.

Javascript as an assembler is the only way to go.

"‘Simplicity is the ultimate sophistication.’ — Leonardo da Vinci"

Mon, 18 Jul 2011 08:27:42 -0400

“‘Simplicity is the ultimate sophistication.’ — Leonardo da Vinci”

from emacs

Mon, 18 Jul 2011 07:46:09 -0400

;; This buffer is for notes you don’t want to save, and for Lisp evaluation. ;; If you want to create a file, visit that file with C-x C-f, ;; then enter the text in that file’s own buffer.