com.digitalpebble.stormcrawler.util

Class URLPartitioner

java.lang.Object

com.digitalpebble.stormcrawler.util.URLPartitioner

public class URLPartitioner
extends Object

Generates a partition key for a given URL based on the hostname, domain or IP address. This can be called by the URLPartitionerBolt or any other component.

Constructor Summary

Constructors

Constructor and Description
URLPartitioner()

Method Summary

Modifier and Type	Method and Description
void	configure(Map stormConf)
String	getPartition(String url, Metadata metadata) Returns the host, domain, IP of a URL so that it can be partitioned for politeness, depending on the value of the config partition.url.mode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

URLPartitioner

public URLPartitioner()

Method Detail

getPartition

public String getPartition(String url,
                           Metadata metadata)

Returns the host, domain, IP of a URL so that it can be partitioned for politeness, depending on the value of the config partition.url.mode.

configure

public void configure(Map stormConf)