Comments on: North by northwest http://www.buzzmachine.com/2006/07/08/north-by-northwest/ by Jeff Jarvis Fri, 10 Feb 2012 12:21:30 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Robert Feinman http://www.buzzmachine.com/2006/07/08/north-by-northwest/#comment-87512 Robert Feinman Sat, 08 Jul 2006 19:35:14 +0000 http://www.buzzmachine.com/index.php/2006/07/08/north-by-northwest/#comment-87512 Early in my professional career I, and some colleagues, did work on connections between scientific documents. We called it clustering at the time. Citations were a very good predictor of relevance, as were certain patterns of key words. The computer power and storage capacity weren't enough in those days to do more than organize specialized scientific subspecialties. Now that the horsepower is available it is sad to see that the algorithms used for analysis are rather unsophisticated. One of the reasons sites like Google work is because most people are satisfied with a close match to what they are looking for, rather than requiring high relevance. This weakness extends to our spy agencies as well, as their need to use wholesale harvesting of electronic data illustrates. If they really had algorithms for discovering patterns of illegal activity they would be doing targeted data capture. Early in my professional career I, and some colleagues, did work on connections between scientific documents. We called it clustering at the time. Citations were a very good predictor of relevance, as were certain patterns of key words. The computer power and storage capacity weren’t enough in those days to do more than organize specialized scientific subspecialties.

Now that the horsepower is available it is sad to see that the algorithms used for analysis are rather unsophisticated. One of the reasons sites like Google work is because most people are satisfied with a close match to what they are looking for, rather than requiring high relevance. This weakness extends to our spy agencies as well, as their need to use wholesale harvesting of electronic data illustrates. If they really had algorithms for discovering patterns of illegal activity they would be doing targeted data capture.

]]>