Options
All
  • Public
  • Public/Protected
  • All
Menu

Module functions/crawl

Index

Enumerations

Interfaces

Type aliases

Variables

Functions

Type aliases

CrawlConfig

CrawlConfig: Static<typeof RSS> | Static<typeof Twitter> | Static<typeof Webhose> | Static<typeof Sitemap>

Crawler Configuration

CrawlType

CrawlType: "rss" | "twitter" | "webhose" | "sitemap"

Crawler Base Types

Variables

Const crawlLgr

crawlLgr: Function = ...

Crawler Logger

Const crawlTypes

crawlTypes: CrawlType[] = ...

Crawler Types List

Functions

assembleCrawlConfigs

  • Assemble all Crawler Configurations for those specified. If no feeds are specified, defaults will be used, unless the noDefaults param is set to True.

    Parameters

    • feeds: FeedList = {}

      list of feed dids separated by type

    • noDefaults: boolean = false

      don't use defaults if list of dids is empty

    Returns Promise<CrawlConfig[]>

    array containing configs, empty if none

crawlFeeds

  • Crawl specified feeds and return new posts.

    Parameters

    • configs: CrawlConfig[]

      array of crawl configs to use

    • store: boolean = false

      should the posts be stored

    • proxyConfig: ProxyConfig = false

      config object for proxy service to use

    • testLevel: number = 0

      0-4 indicating what test data should be returned, 0 is normal

    Returns Promise<ListResponse>

    Success or Failure Message Response including list of new posts

crawlPath

  • crawlPath(did?: string): string

  • Get the path to a crawler directory.

    Parameters

    • did: string = ''

      optional crawl did

    Returns string

    path to crawler directory

getCrawlConfig

  • Get the crawler configurations for the specified crawlers. If dids parameter is skipped or false, return current default configurations. If dids is true, return all available configs.

    Parameters

    • type: CrawlType

      lowercase Crawl type config

    • dids: string | boolean = false

      comma separated list of crawl dids

    Returns Promise<DataResponse>

    Success or Failure Message Response including an array of the configs

getDefaultCrawl

  • Get an array of the default DIDs for a specified crawler.

    Parameters

    Returns DataResponse

    Success or Failure Message Response including an array of the default DIDs

removeCrawl

  • Remove specified crawlers completely. If a specified crawl is a default, it will be removed from the list of defaults.

    Parameters

    • type: CrawlType

      lowercase Crawl type config

    • dids: string

      comma separated list of crawl dids

    Returns Promise<DataResponse>

    Success or Failure Message Response including an array of the files deleted

removeCrawledPosts

  • removeCrawledPosts(did: string, postHashes: string[]): Promise<FileResponse>

  • Remove all posts (signed or otherwise) from specified crawlers completely.

    Parameters

    • did: string

      did of crawl to delete posts from

    • postHashes: string[]

      list of hashes of posts to remove

    Returns Promise<FileResponse>

    Success or Failure Message Response including an array of the files deleted

setDefaultCrawl

  • Set the defaults for a specified crawler. Unless indicated otherwise, specified dids will be added to current default list.

    Parameters

    • type: CrawlType

      lowercase Crawl type config

    • dids: string

      comma separated list of crawl dids

    • overwrite: boolean = false

      should the list of dids completely overwrite current defaults

    Returns DataResponse

    Success or Failure Message Response including the current defaults

signCrawledPosts

  • signCrawledPosts(config: CrawlConfig, store?: boolean, posts?: { announce: string[]; created: number; createdBy: string; files: { name: string; offset: number; length: string; }[]; info: { publicSquare: { package: { description: string; image: string; geo: string; lang: string; body: string; publishDate: number; title: string; canonicalUrl: string; politicalSubdivison: string; references: { ...; }; }; }; }; infoHash: string; name: string; provenance: { signature: string; jwk: { x: string; y: string; crv: "P-384"; alg: "ES384"; kty: "EC"; kid: string; }; publisher: { name: string; description?: string | undefined; image?: string | undefined; url?: string | undefined; tagline?: string | undefined; bio?: string | undefined; }; }; urlList: string[] }[]): Promise<DataResponse>

  • Retrieve all posts stored by the crawler and sign them. Default behavior is to store the newly signed posts with the originals, but that can be overridden with the store param.

    Parameters

    • config: CrawlConfig

      config of Crawl

    • store: boolean = true

      should the signed posts be stored

    • posts: { announce: string[]; created: number; createdBy: string; files: { name: string; offset: number; length: string; }[]; info: { publicSquare: { package: { description: string; image: string; geo: string; lang: string; body: string; publishDate: number; title: string; canonicalUrl: string; politicalSubdivison: string; references: { ...; }; }; }; }; infoHash: string; name: string; provenance: { signature: string; jwk: { x: string; y: string; crv: "P-384"; alg: "ES384"; kty: "EC"; kid: string; }; publisher: { name: string; description?: string | undefined; image?: string | undefined; url?: string | undefined; tagline?: string | undefined; bio?: string | undefined; }; }; urlList: string[] }[] = []

      array of posts

    Returns Promise<DataResponse>

    Success or Failure Message Response including an array of the signed posts and their hashes

writeRSSCrawl

  • writeRSSCrawl(config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { etag: string; type: "rss"; url: string }): DataResponse

  • Create a RSS configuration. This configuration can be used to overwrite current config.

    Parameters

    • config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { etag: string; type: "rss"; url: string }

      RSS config

    Returns DataResponse

    Success or Failure Message Response

writeSitemapCrawl

  • writeSitemapCrawl(config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { since: string; type: "sitemap"; url: string }): DataResponse

  • Create a Sitemap configuration. This configuration can be used to overwrite current config.

    Parameters

    • config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { since: string; type: "sitemap"; url: string }

      Sitemap config

    Returns DataResponse

    Success or Failure Message Response

writeTwitterCrawl

  • writeTwitterCrawl(config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { lastTweet: string; type: "twitter"; userId: number; username: string }): Promise<DataResponse>

  • Create a Twitter configuration. This configuration can be used to overwrite current config.

    Parameters

    • config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { lastTweet: string; type: "twitter"; userId: number; username: string }

      Twitter config

    Returns Promise<DataResponse>

    Success or Failure Message Response

writeWebhoseCrawl

  • writeWebhoseCrawl(config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { type: "webhose"; url: string }): DataResponse

  • Create a Webhose configuration. This configuration can be used to overwrite current config.

    Parameters

    • config: { defaults: { image?: string | undefined; geo?: string | undefined; lang?: string | undefined; politicalSubdivision?: string | undefined; }; filters: undefined | { crawl: { path?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; markup?: { includes?: string[] | undefined; excludes?: string[] | undefined; } | undefined; }; value: { ...; }; }; kid: string; lastPost: string; type: string } & { type: "webhose"; url: string }

      Webhose config

    Returns DataResponse

    Success or Failure Message Response