1銆佽В鍘婲utch鍖咃紝鍦∟utch鏍圭洰褰曚笅寤虹洰褰晆rls,閲岄潰寤轟竴浜涘寘鍚玌RL鐨勬枃鏈urlt.txt錛屼竴琛屼竴涓猆RL,鍐呭濡傦細(xì)http://www.aygfsteel.com
http://www.javaeye.com/
2銆佷慨鏀筩onf鐩綍涓嬬殑crawl-urlfilter.txt,鐗囨柇濡備笅錛?br># accept hosts in MY.DOMAIN.NAME
# +^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/
+^http://www.aygfsteel.com/
+^http://www.javaeye.com/
+^http://lucene.apache.org/
3銆佷慨鏀筩onf鐩綍涓嬬殑nutch-site.xml錛屽唴瀹瑰涓嬶細(xì)
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>http.agent.name</name> <value>Nutch</value> <description>HTTP 'User-Agent' request header. MUST NOT be empty - please set this to a single word uniquely related to your organization. NOTE: You should also check other related properties: http.robots.agents http.agent.description http.agent.url http.agent.email http.agent.version and set their values appropriately. </description> </property> <property> <name>http.robots.agents</name> <value>Nutch,*</value> <description>The agent strings we'll look for in robots.txt files, comma-separated, in decreasing order of precedence. You should put the value of http.agent.name as the first agent name, and keep the default * at the end of the list. E.g.: BlurflDev,Blurfl,* </description> </property> <property> <name>http.agent.description</name> <value>Nutch Search Engineer</value> <description>Further description of our bot- this text is used in the User-Agent header. It appears in parenthesis after the agent name. </description> </property> <property> <name>http.agent.url</name> <value>http://lucene.apache.org/nutch/bot.html</value> <description>A URL to advertise in the User-Agent header. This will appear in parenthesis after the agent name. Custom dictates that this should be a URL of a page explaining the purpose and behavior of this crawler. </description> </property> <property> <name>http.agent.email</name> <value>nutch-agent@lucene.apache.org</value> <description>An email address to advertise in the HTTP 'From' request header and User-Agent header. A good practice is to mangle this address (e.g. 'info at example dot com') to avoid spamming. </description> </property></configuration>娉ㄦ剰錛氬湪nutch-0.9.jar閲岄潰宸插寘鍚玭utch-site.xml錛?nbsp; conf鐩綍涓嬬殑鏂囦歡閮藉鍒惰繃鍒癱lasspath鏍逛笅錛屽鏋滄槸鍦╓EB鐜涓嬭繍琛宑lasspath涓嬬殑nutch-site.xml浼?xì)浼樺厛鍔犺浇锛屽鏋滃湪鍦ˋpplication鐜榪愯錛屽簲鎶婂涓妌utch-site.xml鎵撳叆鍒皀utch-0.9.jar鍖呴噷錛屽惁鍒欙紝涓婇潰鐨勪竴浜涘睘鎬т負(fù)絀轟笉鑳借繍琛屻?br>
4銆佸湪Windows涓嬭繍琛孨utch錛屽緢綆鍗曪紝鍙浣犺兘鎵цCrawl榪欎釜綾誨氨琛岋紝鍐欎竴涓狝nt鑴氭湰鏀懼湪Nuthc鐨勬牴鐩綍涓嬫墽琛屽畠?yōu)O(jiān)K錛屽唴瀹瑰涓嬶細(xì)
<project name="nutch-crawl" default="crawl" basedir="."> <property name="lib.dir" location="lib"/> <property name="conf.dir" location="conf"/> <path id="project.classpath"> <fileset dir="." includes="nutch-*.jar"/> <fileset dir="lib" /> <pathelement path="."/> <pathelement path="${conf.dir}"/> </path> <target name="crawl" > <echo>crwaling starting
</echo> <property name="JVM.extra.args" value="-Xmx512m" /> <java classname="org.apache.nutch.crawl.Crawl" classpathref="project.classpath" fork="true"> <jvmarg line="${JVM.extra.args}"/> <arg value="C:/dev-tools/nutch-0.9/urls"/> <arg value="-dir"/> <arg value="C:/dev-tools/nutch-0.9/crawl"/> <arg value="-depth"/> <arg value="3"/> <arg value="-threads"/> <arg value="15"/> </java> <echo>crwaling finished</echo> </target> </project>鑷蟲錛屽鏃犳剰澶栵紝Nutch宸茬粡嬈㈠揩鍦拌繍琛岃搗鏉ワ紝鏈鍚庡湪crawl鐩綍涓嬩綘浼?xì)鍙戠庮C綘鎯寵鐨勪笢瑗匡紝Enjoy it!
]]>