com.digitalpebble.stormcrawler.bolt

Class StatusEmitterBolt

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichBolt

com.digitalpebble.stormcrawler.bolt.StatusEmitterBolt

All Implemented Interfaces:

Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt

Direct Known Subclasses:

FeedParserBolt, FetcherBolt, JSoupParserBolt, SimpleFetcherBolt, SiteMapParserBolt

public abstract class StatusEmitterBolt
extends org.apache.storm.topology.base.BaseRichBolt

Provides common functionalities for Bolts which emit tuples to the status stream, e.g. Fetchers, Parsers. Encapsulates the logic of URL filtering and metadata transfer to outlinks.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.storm.task.OutputCollector	collector

Constructor Summary

Constructors

Constructor and Description
StatusEmitterBolt()

Method Summary

Modifier and Type	Method and Description
protected boolean	allowRedirs()
void	declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
protected void	emitOutlink(org.apache.storm.tuple.Tuple t, URL sURL, String newUrl, Metadata sourceMetadata, String... customKeyVals) Used for redirections or when discovering sitemap URLs.
protected Outlink	filterOutlink(URL sURL, String newUrl, Metadata sourceMetadata, String... customKeyVals)
void	prepare(Map stormConf, org.apache.storm.task.TopologyContext context, org.apache.storm.task.OutputCollector collector)

Methods inherited from class org.apache.storm.topology.base.BaseRichBolt

cleanup

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.storm.task.IBolt

execute

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Detail

collector

protected org.apache.storm.task.OutputCollector collector

Constructor Detail

StatusEmitterBolt

public StatusEmitterBolt()

Method Detail

prepare

public void prepare(Map stormConf,
                    org.apache.storm.task.TopologyContext context,
                    org.apache.storm.task.OutputCollector collector)

declareOutputFields

public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)

emitOutlink

protected void emitOutlink(org.apache.storm.tuple.Tuple t,
                           URL sURL,
                           String newUrl,
                           Metadata sourceMetadata,
                           String... customKeyVals)

Used for redirections or when discovering sitemap URLs. The custom key / values are added to the target metadata post-filtering.

filterOutlink

protected Outlink filterOutlink(URL sURL,
                                String newUrl,
                                Metadata sourceMetadata,
                                String... customKeyVals)

allowRedirs

protected boolean allowRedirs()