new SeedLoader(message_obj)
Represent crawler seed loader.
Parameters:
| Name | Type | Description |
|---|---|---|
message_obj |
Message |
- Source:
Methods
editSeedFile(fn)
Opens a nano like editor for editing seed file.
Parameters:
| Name | Type | Description |
|---|---|---|
fn |
function |
- Source:
getSeed(Arrayopt)
Return seed info.
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
Array |
<optional> |
var_args - accepts var_args for domain names |
- Source:
pull(fn)
Pulls updated seed copy and if not exists stops the crawler.
Parameters:
| Name | Type | Description |
|---|---|---|
fn |
function | callback |
- Source:
pullSeedFromDB(fn)
Pulls updated copy from db.
Parameters:
| Name | Type | Description |
|---|---|---|
fn |
function | callback |
- Source:
removeSeed(host)
Removes a seed by domain name
Parameters:
| Name | Type | Description |
|---|---|---|
host |
String |
- Source:
seed(fn)
Seeds the crawler
Parameters:
| Name | Type | Description |
|---|---|---|
fn |
function | fn |
- Source:
seedFile(path, seed_data, fn)
Reads, parses and inserts seed file into db.
Parameters:
| Name | Type | Description |
|---|---|---|
path |
String | path to seed.json |
seed_data |
Object | if you do not have path, and want to insert already parsed data |
fn |
function | callback |
- Source:
setDB(pool_obj)
Injecting MongoDB obj
Parameters:
| Name | Type | Description |
|---|---|---|
pool_obj |
MongoDB |
- Source:
setLogger(l)
Injecting Logger obj
Parameters:
| Name | Type | Description |
|---|---|---|
l |
Logger |
- Source:
updateLocalSeed(dic)
Updates local copy of the seed file
Parameters:
| Name | Type | Description |
|---|---|---|
dic |
Object | JSON representation of the seed file |
- Source:
(private, inner) generateMD5(str)
Generates MD5 hash of any string
Parameters:
| Name | Type | Description |
|---|---|---|
str |
String |
- Source: