com.digitalpebble.stormcrawler.parse.filter

Class XPathFilter

java.lang.Object

com.digitalpebble.stormcrawler.parse.ParseFilter

com.digitalpebble.stormcrawler.parse.filter.XPathFilter

Direct Known Subclasses:

LinkParseFilter

public class XPathFilter
extends ParseFilter

Simple ParseFilter to illustrate and test the interface. Reads a XPATH pattern from the config file and stores the value as metadata

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,List<com.digitalpebble.stormcrawler.parse.filter.XPathFilter.LabelledExpression>>	expressions

Constructor Summary

Constructors

Constructor and Description
XPathFilter()

Method Summary

Modifier and Type	Method and Description
void	configure(Map stormConf, com.fasterxml.jackson.databind.JsonNode filterParams) Called when this filter is being initialized
void	filter(String URL, byte[] content, DocumentFragment doc, ParseResult parse) Called when parsing a specific page
boolean	needsDOM() Specifies whether this filter requires a DOM representation of the document

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

expressions

protected final Map<String,List<com.digitalpebble.stormcrawler.parse.filter.XPathFilter.LabelledExpression>> expressions

Constructor Detail

XPathFilter

public XPathFilter()

Method Detail

filter

public void filter(String URL,
                   byte[] content,
                   DocumentFragment doc,
                   ParseResult parse)

Description copied from class: ParseFilter

Called when parsing a specific page

Specified by:

filter in class ParseFilter

Parameters:

URL - the URL of the page being parsed

content - the content being parsed

doc - the DOM tree resulting of the parsing of the content or null if ParseFilter.needsDOM() returns false

parse - the metadata to be updated with the resulting of the parsing

configure

public void configure(Map stormConf,
                      com.fasterxml.jackson.databind.JsonNode filterParams)

Description copied from class: ParseFilter

Called when this filter is being initialized

Overrides:

configure in class ParseFilter

Parameters:

stormConf - The Storm configuration used for the parsing bolt

filterParams - the filter specific configuration. Never null

needsDOM

public boolean needsDOM()

Description copied from class: ParseFilter

Specifies whether this filter requires a DOM representation of the document

Overrides:

needsDOM in class ParseFilter

Returns:

trueif this needs a DOM representation of the document, false otherwise.