The Virtual Quill

Data partitioning for the cloud

I've been posting about what an Identity Service (and, by extension, any cloud service) needs in order to be considered both pervasive and ubiquitous. That is, to be “available anywhere and every time we want to use it” as well as “available everywhere and any time we want to use it.” In my previous post I began to explore three abilities necessary to insure the ubiquity and pervasiveness of the services – the ability to be distributed, replicated and partitioned.

The identity data needs to be replicated for fault tolerance and also to move the data through the network so that its close to where it will be used. It should be distributed so that there are multiple read/write replicas (not just read-only or static catalog images) of the data. The third quality is partitioning.

If we're going to maintain multiple replicas of the data, and if we're going to allow changes from multiple replicas which must then be synchronized to all other replicas then we're going to create quite a bit of network traffic. Add to that the sheer quantity of data which today's (but especially tomorrow's) identity-enabled applications and devices will be placing in the directory and other identity services and you can see that all this replication and distribution could take up a huge amount of bandwidth.

One solution is partitioning - breaking up the datastore into manageable parts. Then, by a well-designed placement of replicas of the partitions you can insure that data is both physically and logically stored near to the point it will be used while still minimizing the amount of traffic on the network necessary for synchronizing the data.

Also, because the datastore is distributed as well as partitioned, you can view the entire tree as if it were stored in one place - even if there is no physical copy of the entire tree. And, of course, you can see this (and so can the identity-enabled apps and devices) from anywhere in the network because now your identity service is pervasive and ubiquitous.

Which leads us back to the original point – application designers, programmers and coders will only use your identity service if it is both pervasive and ubiquitous. q.e.d.