Managing instances of collaboration

Written in Concepts, Tech - on Sunday, March 21, 2010 16:32 - 3 Comments

In this post, we’ll explain how to manage collaboration instances (we call the workspaces) on the XTWIP infrastructure. The workspace is an XML description of the data in your application, which several instances and several users can access, modify and interact with at the same time.

Think of the workspace as some sort of real time interactive, collaborative document.

workspace = real time XML document

While running a collaborative instance of your application, all changes are tracked, sent to all online participants and saved on the XTWIP. However, in this post we’re going to show you how to create, delete and access a workspace.

Creating a workspace

As a third party application developer, all you need to do is post a request to our front end server, authenticating yourself by sending the devKey provided to you. The server will respond with the workspaceID assigned to that workspace. You will need to store this ID, since you will later need it to refer the workspace. For more information on how to post a create workspace request, go through the XTWIP JSON-RPC API Reference [pdf].

Here’s what happens when you make the request

The user hits a “New” button and the server-side for your application receives a request.
Your server then forwards the request to the XTWIP server, providing the devKey as an authentication mechanism. Optionally, it can also provide a name for the workspace or an initial content of the workspace XML (by default, the name is set to “Workspace” and the XML content is set to be empty).
The XTWIP front end performs a number of tasks before stepping further:
- - Checks the availability of your devKey
  - Checks if you have the right to create additional workspaces (if the creation limits per developer has not been exceeded)
  - Are you a paying customer ? Should any additional quality of service conditions be met for this request ?
  - It balances the request on the least occupied backend server, depending on your developer subscription type.
  - Finally, it forwards the request to the backend server that best fits several criteria (load, quality of service).
The backend server actually creates the workspace and it confirms the operations as successful. In case the operation fails, the request will be sent again to another available backend server.
The front end server marks the operation as successful, stores the workspace ID in the database together with the server onto which it is hosted. It returns the workspace ID to the developer server.
The developer server should also store the workspace ID, for any further reference (access request, deletion etc). It also notifies the end-user of the successful operation.

As a general fact, the business logic inside the XTWIP infrastructure is transparent to you. In order to manage collaboration instances, you only need to post the JSON-RPC requests and interpret the results.

Accesing a workspace

After you’ve created the workspace, your main concern will be how to access it: how to make your application aware of the collaboration instance and make sure all changes are delivered on that instance.

The XTWIP SDK for Flash/Flex automatically looks for the authentication information it needs to connect to the workspace in the flashvars of the embed code of your application. In order to get that authentication information however, you need to follow the steps described below:

The user decides to open an existing workspace, so the browser sends a request to the developer server.
Your developer server makes any additional verification (“does this user have rights to the workspace requested?” sort of thing). If everything checks out, the request for access is forwarded to the XTWIP front-end, together with the devKey associated to the application. Optionally, the email and the name of the user may be provided, so each user can have a unique identity in the collaboration environment.
The XTWIP front-end authenticates that the developer identified by devKey indeed has the right to access the workspace. It also finds the back-end server that currently stores the workspace. After that, a request for a temporary token is sent to the back-end server.
The back-end server randomly generates a temporary username & temporary password and then sends them to the front-end.
The front-end forwards them to the developer server, together with the IP and the port of the back-end server that is hosting the collaboration session.
The developer server pastes the temporary username and temporary password into the embed code for the collaborative application.
The application runs, the XTWIP Flash/Flex SDK loads the authentication data from the embed code (flashvars) and it connects to the back-end server, providing the credentials for authentication. The application instance has now joined collaboration on the workspace and may send/receive data, changes, chat messages, presence, ownership and so on.

This combination of devKey and temporary credentials bares several guarantees of security and flexibility:

Each developer can only access those workspaces he/she created.
Developers can define their own access control policies for their users (since this logic is to be implemented in the developer server, not in the XTWIP infrastructure)
Users cannot access workspaces without a confirmation of access rights from the application developer.
Users do not have access to devKey and do not have access to directly creating, deleting or accessing workspaces (without going through the application developer)
Temporary credentials expire minutes after the user has left the collaboration session. In order to reconnect, the user must obtain a new set of temporary credentials.

Deleting a workspace

The workflow for deleting a workspace is almost the same as for creating one, the only exception being you need to provide the ID of the workspace you wish to delete.

We hope this post provided an overview on how you, as a developer, can manage the collaboration instances hosted on the XTWIP infrastructure. For a more detailed understanding of creating, deleting, accessing and retrieving workspaces, go through the XTWIP JSON-RPC API Reference [pdf].

Less conversation(s), more action(s)

Written in Concepts, Stories - on Saturday, March 13, 2010 14:46 - 1 Comment

With the exponential growth of social networks and the explosion of real time data streams, everyone is about the conversation in the real time web. Of course, this is great value for a lot users, since the information travels faster and faster from producers to consumers. However, the value of sharing links, (re)tweeting the latest developments and adding friends on Facebook is bounded. There is only as much relevant information at any given time and following more and more people won’t change that. To the contrary, social media is believed to have some significant negative effects, such as eroding attention spans and disrupting individual focus.

When developing XTWIP, we had in mind harnessing the power of the real time web in order to make people focus, work together and get things done while keeping up to date with relevant developments from co-workers. Our service helps developers bring their application users together in producing information, rapidly reviewing it and delivering it on time. XTWIP is about shifting real time from quantity to quality.

Most real time services on the web have an outward perspective, inviting people to share and re-share information that has already been produced (blog/magazine articles, pictures, artwork, witty quotes and so on). XTWIP challenges developers to rethink the possibilities of their applications so they can engage teams of users to:

be more creative & inspired
brainstorm while actually working
distribute tasks naturally and efficiently
deliver faster, with less effort
turn negative stress (deadlines) into positive stress (reciprocal engagement)

And have you ever noticed how annoying is to have work and communication as two separate processes ? You sit at your desk trying to focus on your tasks. All of a sudden, 5 emails rush in a completely mess up your thought patterns, even if those emails directly concern what you are working on. What if you could receive the feedback and the help you need directly in the design you’re creating ? Or in the document you’re writing ? Or in the workflow you’re so meticulously laying out ?

Wouldn’t it be better if the people who nag you about “how you should change the color scheme on that web page concept” could actually change the color scheme themselves, side-by-side with you ?

How about not spending 10 minutes on the phone with a customer who’s trying to explain how to drag 2 buttons around around to the upper-left side corner ? How about if he could do it himself while you watch and fine-tune? That would be a very empowering feeling to the customer and it would be a great time (and stress) saver for you.

So what if you’re an architect or interior designer ? Maybe your clients are as busy and you are and they can’t afford dropping by your office in the middle of the day. They could just drop a link in the browser and they could drag the virtual furniture around, while you discretely assist them in making the right choices, step by step. This would save a lot of frustration on both sides, don’t you think ?

Or how about shopping together with your online friends ? Maybe it’s for a group gift you want to buy or even just for the fun of it.

Wait wait… I think I have a better one. How about you design your own Nikes together with your friends ?

Our vision is about merging work and communication into one single process. About actually producing information in a social, fun and interactive way.

XTWIP is the backend technology that makes this work. We have taken the first step to making the real time web more than a context(less?) conversation. Now we need you – developers, designers and visionaries – to meet us half way.

Or – as Elvis so eloquently put it – “A little less conversation, a little more action please“

Do you have a great idea about a real time collaboration application ? Drop in a comment, send us your feedback give or twitter us about it.

Types of Collaboration: Object based

Written in Concepts, Tech - on Saturday, March 13, 2010 11:54 - 1 Comment

This implementation of real time collaboration seems to offer the best balance between performance, usability and developer needs. This is why we chose it for the current XTWIP implementation of collaboration.

The object based approach follows these 4 simple rules:

The data model of the collaborative application can be represented as an XML. Of course, the application can pull data from Twitter, Flickr or Facebook, but by data model we mean the data the application stores.

Whenever the application changes something, it just notifies the XTWIP client-side SDK that “something has changed”. The SDK automatically detects what has changed and send the difference over to the XTWIP server and to the other application instances.

In order to better avoid conflicts, the XML may have atomic tags or non-atomic tags. An atomic tag has an unique ID attribute and it may contain either atomic or non-atomic tags. A non-atomic tag does not have a unique ID and it may only contain other non-atomic tags. The XTWIP SDK provides the unique IDs and it guarantees they are unique across all application instances. You can think of atomic tag as components which the developer considers important enough to separate and non-atomic tags as properties of those components. See the example below to get a better idea.

<!– The document is an important tag, so it gets a unique ID –>

<document id=”7″>

<!– The same goes for the button –>

<button id=”9″>

<!– The attributes of the button however, are not that important from a collaborative stand-point –>

<label text=”Collaborative Button” color=”6600FF”>

<background src=”xtwip_logo.jpg>

<background color=”222222″>

<link url=”http://www.xtwip.com”>

</button>
[...]

</document>

When two application instances (actually, two users) change the same atomic tag at the same time, the last change has priority over the first. Of course, a merging of the content will be attempted – failing that, the last change will remain. The distinction between atomic and non-atomic tags comes in very handy here, because it tells the merging algorithm how it should try to match tags.

This (apparently) simple protocol provides application developers with a simple gateway to collaboration. No longer do they have to write server-side code or write their own custom code for diffing and merging differences. The XTWIP SDK together with the XTWIP Collaboration Service provide that for just a few lines of code. All developers need to do is read the tutorial, drop a few lines of code into their application and they can redeploy their application as a collaboration experience.

Let’s look at the advantages:

Very easy to deploy. There is not specific server-side logic and developers can just embed their application and the XTWIP SDK takes care of the rest (connecting to server, handshaking, getting data, posting differences and so on)
Very easy to implement. All you need to add to your application is: listening for when outside changes appear (so you know when to redraw), notifying the SDK of when something has changed, serializing your data model (XML serialization is already built in the SDK) and adding a connect statement on application load. That’s it !
Fast execution, low CPU cost. Unlike Operational Transforms methods, object based diffing is significantly faster.
Guaranteed consistency. This means that all application instances will rapidly converge to the same instance of the data model, no matter how many changes are being performed.

There are only two disadvantages:

Number one. If two users happen to modify the exact same atomic object at the same time (within 300 ms of each other, on average), the XTWIP SDK will automatically merge the two changes. When merging fails, the last change performed will implicitly have priority. Merging means a further analysis of what has changed (attributes, child tags) and putting it all together. Merging usually fails when the two users change exactly the same attributes and no automatic conclusion can be drawn as to who’s right.

In our experience, however, this type of fine-grained micro-conflict between versions is very rare. When it does happen, the users notice it right away. They tend to immediately react and choose a final state for the change they wanted.

Number two. Whole pieces of the data model are serialized and send across the network, so if your application deals a lot with raster image editing, object-based collaboration is not the best idea (at least, not by itself).

If you want to try this type of real time collaboration with your own application, create a Developer Account on XTWIP and there you’ll find everything you need:

A short tutorial on how to get started
The client side SDK
A full documentation on the protocol
A full description with examples on how to manage (create, delete, request access) to multiple collaborative instances (workspaces)

Types of Collaboration: Event-based

Written in Concepts, Tech - on Thursday, March 11, 2010 13:06 - 3 Comments

For this example, we’ll consider a simple application in you can draw boxes, and you have only 4 options:

makeNewBox(centerX,centerY)
deleteOldBox(boxID)
translateBox(boxID, X, Y)
rotateBox(boxID, alpha)

It means that instances of the application transmit across the network only the actions taken in order to get from the old state to the new state. This is extremely useful if you have data models that take up a lot of space (like 3D polygon models), but changes made to them can be described using very little space.

So let’s say you have application A and application B, working on the same instance containing square1 and square2. Let’s say square1 is already rotated by 30 degrees, and instance A rotates it with another 15 degrees (to a total of 45 degrees). Instead of sending out rotateBox(square1, 45), it sends out rotateBox(square1,15). The great advantage of this approach if two rotations occur in (near) real time, they can get composed over the network. So if, at the same time, application B, applies and additional rotation of 10 degrees, the end result (in both instance A and B).

The order on the generated events will be the order they reach the XTWIP server. It this case, it may seem that the order of the events is irrelevant, but most actions (operations) are not commutative – that is changing the order of two actions changes the end result they yield.

The advantages of this approach:

Events end up having one single order in all application instances (everyone sees the same thing)
Concurrent changes (same time, same object) are composed
Small traffic across the network

The disadvantages are however more subtle and only through implementing experience did we learn about them:

Composing concurrent changes isn’t always the best thing to do. Coming back to the previous example, if user A rotates the square by 15 degree and user B rotates it by 10 degrees, the end result will be a total rotation of 25 degrees. This is a mix of what both users wanted. But in fact it isn’t what either of them wanted.

Requires server-side logic – in this model, the XTWIP server has to be aware of what rotateBox means. So the logic for actually performing the rotation has to be on the server. The downside is that with every application you deploy, the logic for all its operations has to be deployed on the XTWIP server as well. So this means more work for developers who use our server for deploying their apps and more work for us (because we would have to enforce advanced security restrictions on application logic). More work = bad news

These significant disadvantages of the event-driven model lead us to the decision that we needed an alternative way of offering real time collaboration, with less debugging, less risks and less work. This is how we came up with the object-driven collaboration model …

* Keep following the Types of Collaboration series to get a true insight on how XTWIP provides real time collaboration for your Web & Rich Internet Applications.

Types of Collaboration: Introduction

Written in Concepts, Tech - on Thursday, March 11, 2010 13:03 - 0 Comments

When we started XTWIP, we had a simple vision: allowing people to change the same content at the same time. We strongly believe that this can help getting more work done together, without losing time with emails, phone calls and (worst of all) meetings.

However, we soon realized that from a technical stand-point there are three fundamental ways of implementing real time collaboration. To the end-user this probably has no impact whatsoever. But for the developer who makes his/her application collaborative, it is of great importance to choose the best mix of these models:

Event based collaboration – actions are sent over the network. The XTWIP server has the logic to execute application specific operations on its own data model.

Object based – XML-formatted object structures are sent over the network, depending on the part that changes. The XTWIP server requires no logic to execute application specific operations.

Data based collaboration – actions are sent over the network. The XTWIP server has no logic to execute application specific operations and it periodically polls client-sided applications for the latest data model. The server stores the data model, serves it to new clients, but it doesn’t understand the data, nor does it have the logic to change it.

Operational Transforms (OT) based collaboration (this is the stuff Google Wave uses). The collaboration server has logic to execute general purpose editing operations (insert, delete, skip) and also the logic to “merge” these operations together. For instance, if two users send concurrent actions (almost) simultaneously, those specific actions are modified by OT when they reach their destination so that they make sense in the new context. The server stores and periodically updates the data model. However, OT is very CPU intensive and the results it produces are usually more than what most application developers need.

In the following posts, we’ll explain the differences between these types of collaboration and how you should pick the one the suits best your use case/business case.

XTWIP demo is here !

Written in Concepts, Tech - on Saturday, January 30, 2010 8:18 - 1 Comment

We’re inviting you to a new demo of XTWIP collaborative solution . Click on the link below to see the real time capabilities of XTWIP in a simple sample application developed by one of our collaborators.

www.xtwip.com/demo/

This application is cooler if several people use it at the same time. So don’t forget to invite your friends.

Failing that, open the link in two separate browser windows and see how the changes from one reflect on the other.

Don’t forget to give us feedback - just post a comment below or Twitter us.

Google Wave & TWIP

Written in Concepts, Tech - on Monday, November 9, 2009 20:21 - 3 Comments

Recently, we have been asked whether TWIP is a direct competitor of Wave, the latest and hottest technology since Maps.

How is TWIP different ? Do we believe in the success of Wave? Do we have the same market?

“Are you out of your mind competing with Google?” some of you could ask. Yes, we are. But we don’t think of it as competition.

First of all, we thank Google for introducing and explaining the concept of real time collaboration to the online community and to the web industry. Wave is a powerful conversational tool, extremely extensible and easy to integrate with other web-applications. We hope that, along with other tools on the market, developers will become more and more attracted to building collaborative software.

So what are the differences between Wave and TWIP?

From a technical standpoint, the two solutions converge to the same idea : servers that manage the communication and synchronize the content between users. TWIP uses XMPP for message management just like Wave. To some extent, both use Operational Transforms to merge changes into the data model.

In terms of target, we are not addressing the same users.

Wave is the newest conversational tool. It has a lot of cool features (have you seen the spell checker?…wow), all perfect for it’s raison d’etre: a conversational tool. Even if extended, it is still a tool for real-time ubercool chat. The External API developed for integrating Wave into other websites is purposed to offer conversations as widgets.

As far as we are concerned…

We are building a platform to host & support applications, each of which individually has it’s own scope. They can exists just fine without TWIP. We are only offering an easy solution for those who want this cool feature named real-time collaboration. They may use it however they want. Even build online games if that’s what they like.

While Wave is more about facilitating the conversation, TWIP is about merging conversation & workflow. Switching between actual work (coding, designing) and feedback (revisions, brainstorming, user reviews) can be time-consuming and exhausting. That’s why we want to rid people of multi-tasking, allowing them to communicate as well as to create at the same time.

Our platform is ready, just a few more finishing touches and it will be online. We’re eager to receive your feedback.

Here are our plans for the near future:

an alpha version will be ready for you to test, crash and enjoy
organizing sessions where all participants will receive TWIP accounts
to interact with some of the coolest web-application developers – so we can better understand their needs and they can better understand the true value of real-time collaboration.
a new application will hatch: collaborative research (for beta version)

Please let us know if you’re interested in any of these. Your feedback is our driving lesson.

See ya

The battle for the Client-side

Written in Concepts, Tech - on Tuesday, March 10, 2009 11:47 - 6 Comments

[ A bit of TWIP's technical side, behind the scene ]

TWIP is going to be more than just a bunch of collaborative apps. It will also be the protocol behind those apps, that we plan to make available to other developers as well. So we started thinking about how we could make this protocol compatible with as many technologies as possible.

Java is what we’re working on right now. Dot Net is not too far. But then we thought: let’s make the TWIP protocol available for JavaScript and GWT. The GWT/JS trend is taking over the client-sided Internet one day at the time, so why not make a TWIP API for it? And it can be done…

… but there is a catch.

JavaScript can’t work with sockets. It can only work with (XML) requests. JS can’t keep a connection open and wait for the server to send incoming messages from other users. Rather, the client must periodically send requests for updates and wait for the server to provide them. Although this isn’t a major drawback, it is a inconvenience, since any protocol working exclusively with request isn’t exactly real-time. And this is one of the reasons Google Docs and other collaborative platforms out there have trouble with guaranteeing true synchronization.

So here’s the deal… First, the TWIP API will be Java – that’s for sure. Next we’ll consider writing a Dot Net API. And last, but must definitely not least – GWT and JavaScript – with the XMLRequest-oriented protocol. What we aim for is making these APIs interoperable with the TWIP server, so third-party developers can easily develop their collaborative apps in whatever fashion they see fit. We’ll leave it up to them to decide if on the client-side they go for the robust connection-oriented approach (in Java, JavaFX or Dot Net) or for the light-weight, flexible XML request based approach (in JavaScript and GWT).

So what do you think ?

Green or Blue ?

Written in Brain DUMP, Concepts - on Thursday, January 22, 2009 10:07 - 9 Comments

We need your opinion for the color of our logo.

Should it be green ?

9th Prime [Green

…or blue ?

9th Prime [Blue

We’d really appreciate your suggestions.

The TWIP Concept

Written in Concepts, Tech - on Friday, December 26, 2008 23:35 - 0 Comments

The most essential part of TWIP is the real time collaboration. Everybody should see what everybody else is doing in a a split second. In order for that ho happen, you need a server. A central authority. A point with which all users synchronise.

Most of all, you need that to preserve coherence. Another big word, right ? Coherence means that all users do the operations in exactly the same order. Let’s say user A draws a red line and sends a message to user B (so B can have it drawn as well). User B, less than a second after, draws a green line on top of the red line. Now, without preserving coherence, user A will see the green line underneath the red line and user B will see the green line above the red line. This would happen because each user would get the message from the other user with a certain delay. And we wouldn’t fix that, the whole collaboration thing would suck, as we’d have no way of knowing that the same data is available on all computers. Fortunately, we have a way to fix that and this is done by a server, a central authority. Think of it as a commander who keeps the privates in attack formation. The thing for you to remember right know is coherence, as being essential.

Next there is the question of extensibility, scalability. Let’s say we fix this whole coherent method of synchronisation. A text editor. Or a picture editor. Or for a piece of software for editing programming code or 3D models. We’ll put a lot of work into the sync engine and into the application itself, but what about all the other apps ? So we thought it’d be waaaay cooler to have this synchronisation engine avaialable onto which we could build. Or anyone else for that matter. There would be no point in our trying to develop all these collaborative apps, when there are a of really intricate, exciting pieces of software available. All they need is the collaboration scheme.

So this is how we thought about the Framework. Basically, this would be the synchronisation engine onto which to build. Think of it like a programming interface or an API. Regardless (almost regardless) of application a few simple rules should be respected and the collaboration would come right in !

Then, we thought we’d do everything in Java, so it would be portable on any OS, both on the client side and on the server side. This even opens the perspective of making a mobile version of TWIP, but let’s no get ahead of ourselves.

Java opens more doors for our project. Like the transparent application distribution. This means that all the apps are stored on the server-side, and when a users wants a new app, it automatically downloads and plugs in together with all the TWIP resources on the client system.

As far as the server is concerned, it has four major roles:

Handling the real-time synchronisation (keeping the client apps in line, as previously discussed)
Storing an online version of the data (so users can have access to only one resource, instead of keeping individual copies on their computer)
Managing users and users rights.
Managing the apps stored online and making them available to users.

Look at the diagram below to get a better picture of the whole relationship between the clients and the server:

TWIP Client-Server Overview

On the server-side, synchronisation also means the server must keep an updated version of all the data. So the server must constantly read the messages received from clients and change the stored data accordingly.

Right now, I and my team-mate, Cristi, are working on the Framework and on the synchronsation schemes.

What would be really useful to us would be having your feedback. Tell us how collaboration would work for you. Tell us what you need and what you’d think is cool as far as team work is concerned.

Give us the top three applications you would use in a collaborative fashion, with your friends. Ask your friends. Leave a comment or two