new Robots(message_obj)
Represents Robots.txt parser
Parameters:
| Name | Type | Description |
|---|---|---|
message_obj |
Message |
- Source:
Members
bots
Parsed data is stored in this JSON object.
- Source:
Methods
parse(fn)
Parses robots.txt for loaded urls.
Parameters:
| Name | Type | Description |
|---|---|---|
fn |
function | callback |
- Source:
(private, inner) addToCache(key, value)
Adds parsed data for a url into mongodb.
Parameters:
| Name | Type | Description |
|---|---|---|
key |
String | url |
value |
String | parsed data from robots.txt |
- Source:
(private, inner) loadCache(fn)
Loades already parsed robots.txt from mongodb store.
Parameters:
| Name | Type | Description |
|---|---|---|
fn |
function | callback |
- Source: