Botmation Documentation

Concurrency

Concurrency

There is a great open-source npm module for running concurrent Puppeteer Chromium instances called puppeteer-cluster.

Functional bots ran concurrently

Since BotAction's operate solely on Puppeteer pages, you can combine Botmation and Puppeteer-Cluster to run functional web bots concurrently.

Here is a simplified version of the puppeteer-cluster example:

import { Cluster } from 'puppeteer-cluster'
(async () => {
try {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_BROWSER,
maxConcurrency: 3
})
const screenshotBotForClustering = async ({ page, data: url }: {page: Page, data: any}) =>
await chain(
goTo(url),
screenshot(url.replace(/[^a-zA-Z]/g, '_')),
log('screenshot of ' + url + ' saved')
)(page)
cluster.queue('https://nodejs.org/', screenshotBotForClustering)
cluster.queue('https://github.com/', screenshotBotForClustering)
cluster.queue('https://www.typescriptlang.org/', screenshotBotForClustering)
await cluster.idle()
await cluster.close()
} catch(error) {
console.error(error)
}
})()

Dynamic Clustering Bots

To create a Bot for clustering (bots in a cluster are ran concurrently), you create an async function that takes an object {page: Page, data: any} as a param, for dynamically queueing the Puppeteer code, multiple times with varying options such as starting URL.

Check out puppeteer-cluster for documentation.

Edit this page on GitHub
Baby Bot