com.digitalpebble.stormcrawler.filtering

Interface URLFilter

All Superinterfaces:

Configurable

All Known Implementing Classes:

BasicURLFilter, BasicURLNormalizer, FastURLFilter, HostURLFilter, MaxDepthFilter, MetadataFilter, RegexURLFilter, RegexURLFilterBase, RegexURLNormalizer, RobotsFilter, SelfURLFilter, SitemapFilter, URLFilters

public interface URLFilter
extends Configurable

Unlike Nutch, URLFilters can normalise the URLs as well as filtering them. URLFilter instances should be used via URLFilters

Field Summary

Fields inherited from interface com.digitalpebble.stormcrawler.util.Configurable

LOG

Method Summary

Modifier and Type	Method and Description
String	filter(URL sourceUrl, Metadata sourceMetadata, String urlToFilter) Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Methods inherited from interface com.digitalpebble.stormcrawler.util.Configurable

configure, configure

Method Detail

filter

String filter(URL sourceUrl,
              Metadata sourceMetadata,
              String urlToFilter)

Returns null if the URL is to be removed or a normalised representation which can correspond to the input URL

Parameters:

sourceUrl - the URL of the page where the URL was found. Can be null.

sourceMetadata - the metadata collected for the page

urlToFilter - the URL to be filtered