ItGo.me - 专注IT技术分享

首页 > Apache项目 > Nutch > nutch https配置

nutch https配置

时间:2016-07-06来源:网友分享 点击:
nutch 抓取 https页面。只需要在文件nutch-site.xml的plugin.includes元素中加入protocol-httpclient

如下:

<property>
  <name>plugin.includes</name>
  <value>protocol-http|protocol-httpclient|urlfilter-regex|parse-(html|tika|metatags)|filter-xpath|index-(basic|metadata|anchor|more)|scoring-opic|urlnormalizer-(pass|regex|basic)|language-identifier</value> 
  <description>Regular expression naming plugin directory names to
  include.  Any plugin not matching this expression is excluded.
  In any case you need at least include the nutch-extensionpoints plugin. By
  default Nutch includes crawling just HTML and plain text via HTTP,
  and basic indexing and search plugins. In order to use HTTPS please enable
  protocol-httpclient, but be aware of possible intermittent problems with the
  underlying commons-httpclient library.
  </description>
</property>

Nutch调试常用命令

Nutch常用管理命令 ./bin/nutch readdb myCrawl/crawldb -stats 显示crawldb的统计信息 ./bin/nutch readdb myCrawl/crawldb -dump crawlUrlInfo 导出每个url的信息到crawlUrlInfo文件夹中...

nutch https配置。nutch 抓取 https页面。只需要在文件nutch-site.xml的plugin.includes元素中加入protocol-httpclient
------分隔线----------------------------