搜索引擎技术文章.

搜索引擎技术文章
谢煜波 http://blog.xieyubo.com/

SF：开源的ftp搜索引擎
http://gf.cs.hit.edu.cn
相关文档
超音速版
注意一些细节，让程序运行得更快(1/4)
注意一些细节，让程序运行得更快(2/4)
注意一些细节，让程序运行得更快(3/4)
注意一些细节，让程序运行得更快(4/4)
SF超音速版的数据结构(1/3)
SF超音速版的数据结构(2/3)
SF超音速版的数据结构(3/3)
亚音速版
SF 亚音速版系统架构 (1 / 3)
SF 亚音速版系统架构 (2 / 3)
SF 亚音速版系统架构 (3 / 3)
SF 搜索引擎 - IP来源统计开发文档

百度算法-查询处理以及分词技术 http://hi.baidu.com/jiewangzi/blog/item/0e7bc23593e81d1390ef3936.html
分两个部分来讲述：查询处理/中文分词。
现在分词算法已经算是比较成熟了，有简单的有复杂的，比如正向最大匹配，反向最大匹配，双向最大匹配，语言模型方法，最短路径算法等等，有兴趣的可以用GOOGLE去搜索一下以增加理解。

使用正向最大匹配算法实现中文分词简单模型-用trie树实现
http://blog.csdn.net/lyflower/archive/2006/12/21/1452091.aspx

搜索引擎CACHE策略研究
http://software.hit.edu.cn/eestudio/bbs/ShowPost.asp?ThreadID=271

MapReduce: Simplified Data Processing on Large Clusters
Jeffrey Dean and Sanjay Ghemawat
http://labs.google.com/papers/mapreduce.html

Abstract

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day.

Appeared in:
OSDI'04: Sixth Symposium on Operating System Design and Implementation,
San Francisco, CA, December, 2004.

Download: PDF Version

Slides: HTML Slides

Published on Sep 16, 2007 in categories