com.digitalpebble.stormcrawler.filtering

Interface URLFilter

All Known Implementing Classes:

BasicURLFilter, BasicURLNormalizer, HostURLFilter, MaxDepthFilter, MetadataFilter, RegexURLFilter, RegexURLFilterBase, RegexURLNormalizer, RobotsFilter, SelfURLFilter, URLFilters

public interface URLFilter

Unlike Nutch, URLFilters can normalise the URLs as well as filtering them. URLFilter instances should be used via URLFilters

Method Summary

Modifier and Type	Method and Description
void	configure(Map stormConf, com.fasterxml.jackson.databind.JsonNode filterParams) Called when this filter is being initialized
String	filter(URL sourceUrl, Metadata sourceMetadata, String urlToFilter) Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Method Detail

configure

void configure(Map stormConf,
               com.fasterxml.jackson.databind.JsonNode filterParams)

Called when this filter is being initialized

Parameters:

stormConf - The Storm configuration used for the ParserBolt

filterParams - the filter specific configuration. Never null

filter

String filter(URL sourceUrl,
              Metadata sourceMetadata,
              String urlToFilter)

Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Parameters:

sourceUrl - the URL of the page where the URL was found. Can be null.

sourceMetadata - the metadata collected for the page

urlToFilter - the URL to be filtered