com.digitalpebble.stormcrawler.filtering.metadata

Class MetadataFilter

java.lang.Object

com.digitalpebble.stormcrawler.filtering.metadata.MetadataFilter

All Implemented Interfaces:

URLFilter, Configurable

public class MetadataFilter
extends Object
implements URLFilter

Filter out URLs based on metadata in the source document

Constructor Summary

Constructors

Constructor and Description
MetadataFilter()

Method Summary

Modifier and Type	Method and Description
void	configure(Map stormConf, com.fasterxml.jackson.databind.JsonNode paramNode) Called when this filter is being initialized
String	filter(URL pageUrl, Metadata sourceMetadata, String urlToFilter) Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.digitalpebble.stormcrawler.util.Configurable

configure

Constructor Detail

MetadataFilter

public MetadataFilter()

Method Detail

configure

public void configure(Map stormConf,
                      com.fasterxml.jackson.databind.JsonNode paramNode)

Description copied from interface: Configurable

Called when this filter is being initialized

Specified by:

configure in interface Configurable

Parameters:

stormConf - The Storm configuration used for the ParserBolt

paramNode - the filter specific configuration. Never null

filter

public String filter(URL pageUrl,
                     Metadata sourceMetadata,
                     String urlToFilter)

Description copied from interface: URLFilter

Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Specified by:

filter in interface URLFilter

Parameters:

pageUrl - the URL of the page where the URL was found. Can be null.

sourceMetadata - the metadata collected for the page

urlToFilter - the URL to be filtered