com.digitalpebble.stormcrawler.persistence

Class AbstractStatusUpdaterBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

com.digitalpebble.stormcrawler.persistence.AbstractStatusUpdaterBolt

All Implemented Interfaces:

Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt

Direct Known Subclasses:

MemoryStatusUpdater, StdOutStatusUpdater

public abstract class AbstractStatusUpdaterBolt
extends org.apache.storm.topology.base.BaseRichBolt

Abstract bolt used to store the status of URLs. Uses the DefaultScheduler and MetadataTransfer.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.storm.task.OutputCollector	_collector
static String	cacheConfigParamName Parameter name to configure the cache @see http://docs.guava-libraries.googlecode .com/git/javadoc/com/google/common/cache/CacheBuilderSpec.html Default value is "maximumSize=10000,expireAfterAccess=1h"
static String	maxFetchErrorsParamName Number of successive FETCH_ERROR before status changes to ERROR
static String	roundDateParamName Used for rounding nextFetchDates.
static String	useCacheParamName Parameter name to indicate whether the internal cache should be used for discovered URLs.

Constructor Summary

Constructors

Constructor and Description
AbstractStatusUpdaterBolt()

Method Summary

Modifier and Type	Method and Description
protected void	ack(org.apache.storm.tuple.Tuple t, String url) Must be overridden for implementations where the actual writing can be delayed e.g.
void	declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
void	execute(org.apache.storm.tuple.Tuple tuple)
void	prepare(Map stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.task.OutputCollector collector)
protected abstract void	store(String url, Status status, Metadata metadata, Date nextFetch)

Methods inherited from class org.apache.storm.topology.base.BaseRichBolt

cleanup

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Detail

useCacheParamName

public static String useCacheParamName

Parameter name to indicate whether the internal cache should be used for discovered URLs. The value of the parameter is a boolean - true by default.

maxFetchErrorsParamName

public static String maxFetchErrorsParamName

Number of successive FETCH_ERROR before status changes to ERROR

cacheConfigParamName

public static String cacheConfigParamName

Parameter name to configure the cache @see http://docs.guava-libraries.googlecode .com/git/javadoc/com/google/common/cache/CacheBuilderSpec.html Default value is "maximumSize=10000,expireAfterAccess=1h"

roundDateParamName

public static String roundDateParamName

Used for rounding nextFetchDates. Values are hour, minute or second, the latter is the default value.

_collector

protected org.apache.storm.task.OutputCollector _collector

Constructor Detail

AbstractStatusUpdaterBolt

public AbstractStatusUpdaterBolt()

Method Detail

prepare

public void prepare(Map stormConf,
                    org.apache.storm.task.TopologyContext context,
                    org.apache.storm.task.OutputCollector collector)

execute

public void execute(org.apache.storm.tuple.Tuple tuple)

ack

protected void ack(org.apache.storm.tuple.Tuple t,
                   String url)

Must be overridden for implementations where the actual writing can be delayed e.g. put in a buffer

store

protected abstract void store(String url,
                              Status status,
                              Metadata metadata,
                              Date nextFetch)
                       throws Exception

Throws:

Exception

declareOutputFields

public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)