Monday, May 12, 2008

Getting more violent all the time

The distinguished Mr. Cameron has restated what he thinks is our major disagreement over synchronization and replication of identity data on the so-called "identity bus." He says:

"Sometimes an application needs to do complex searches involving information 'mastered' in multiple locations. I’ll make up a very simple 'two location' example to demonstrate the issue:

'What purchases of computers were made by employees who have been at the company for less than two years?'

Here we have to query 'all the purchases of computers' from the purchasing system, and 'all employees hired within the last two years' from the HR system, and find the intersection.

Although the intersection might only represent a few records, performing this query remotely and bringing down each result set is very expensive. No doubt many computers have been purchased in a large company, and a lot of people are likely to have been hired in the last two years. If an application has to perform this type of query with great efficiency and within a controlled response time, the remote query approach of retrieving all the information from many systems and working out the intersection may be totally impractical.

Compare this to what happens if all the information necessary to respond to a query is present locally in a single database. I just do a 'join' across the tables, and the SQL engine understands exactly how to optimize the query so the result involves little computing power and 'even less time'. Indexes are used and distributions of values well understood: many thousands of really smart people have been working on these optimizations in many companies for the last 40 years."

What Kim fails to note, however, is that a well designed virtual directory (see Radiant Logic's offering, for example) will allow you to do a SQL query to the virtual tables! You get the best of both: up to date data (today's new hires and purchases included) with the speed of an SQL join. And all without having to replicate or synchronize the data. I'm happy, the application is happy - and Kim should be happy too. We are in violent agreement about what the process should look like at the 40,000 foot level and only disagree about the size and shape of the paths - or, more likely, whether they should be concrete or asphalt.

Labels: , ,

But the issue is not with the language you use to perform the query: it's where the data is located. If you have data in separate physical databases then it's necessary to pull the data from the separate sources and join them locally. So, in Kim's example, if you have 5000 employees and have sold 10000 computers then you need to pull down the 15000 records over the network and perform the join locally (unless you have an incredibly smart distributed query optimiser which works across heterogeneous data stores). This is going to be more expensive than if the computer order and employee data are colocated.
Post a Comment

© 2003-2006 The Virtual Quill, All Rights Reserved


[Powered by Blogger]