Tutorial

Introduction

Automating tasks in the web through Puppeteer is about programatically controlling the browser as if you were a real User. It's done by capturing what the User does in steps, then writing each in code.

The Chrome Puppeteer Recorder extension is a great example to see what we'll be doing in this tutorial, except the final code will be declaratively composed with BotAction's.

In this tutorial, we're going to make a bot that logs into LinkedIn.

tldr: The login() method and more are included in the @botmation/linkedin package. It's documented here.

Research

When it comes to logging in, it's best to find a service's login page, instead of relying on what you find on the home page, a dropdown, or modal. The home page and general site UX are subject to change frequently, while a dedicated login page remains mostly unchanged, and therefore will experience less breaking changes, as time moves forward.

For LinkedIn, they have a dedicated Login page. Perfect.

Some websites do not link their dedicated login page, for various reasons, but it's worth trying a few URL patterns such as "example.com/login", "example.com/auth/login.html", if you don't find it initially.

Next, we need to do build a login() BotAction for LinkedIn. In order for the bot to login, it will have to go to the login page, click the form's input fields, enter the correct information, hit the "Sign in" button, then wait for the new page to load.

Most of these steps requires the bot to interact with specific HTML elements such as input and button. Therefore, we need to figure out how to refer to them, by writing HTML selectors that identify them.

Let's look at the rendered HTML source code for the Login Form.

Right-click around the center of the form, then click "Inspect Element"

Here's a cleaned up version of the page's login form:

Copy
<form method="post" class="login__form" action="/checkpoint/lg/login-submit" novalidate="">
  <div class="form__input--floating">
    <input id="username" name="session_key" type="text" aria-describedby="error-for-username" required="" validation="email|tel" autofocus="" aria-label="Email or Phone">
    <label class="form__label--floating" for="username" aria-hidden="true">
      Email or Phone
    </label>

    <input id="password" type="password" aria-describedby="error-for-password" name="session_password" required="" validation="password" aria-label="Password">
    <label for="password" class="form__label--floating" aria-hidden="true">
      Password
    </label>

    <button class="btn__primary--large from__button--floating" data-litms-control-urn="login-submit" type="submit" aria-label="Sign in">
      Sign in
    </button>
  </div>
</form>

There were many hidden inputs and other HTML elements that don't matter, so for simplicity sake, I removed them

We care about three things from this form:

1. Username input field
2. Password input field
3. Submit form button

For each one of these, we're going to write a HTML selector to grab it.

When it comes to writing HTML selectors, it's best to avoid uniquely generated identifiers, and mostly rely on HTML structure, CSS classes/id's and HTML attributes. For this example, the following selectors are generic enough to survive possible page changes, but possibly not specific enough to avoid future form collisions:

1. Username input field

Copy
form input[id="username"]

2. Password input field

Copy
form input[id="password"]

3. Submit form button

Copy
form button[type="submit"]

Fortunately, most dedicated login pages only have one visible form, so this should suffice.

Building the BotAction

This BotAction is going to be an assembly of other BotAction's, specifically goTo(), click(), type(), waitForNavigation and log().

The flow is simple. Go to the login page, click the input fields, type the information, click the submit button, wait for navigation to complete, and log a friendly success message.

Let's see the code:

Copy
const login = (emailOrPhone: string, password: string): BotAction =>
  chain(
    goTo('https://www.linkedin.com/login'),
    click('form input[id="username"]'),
    type(emailOrPhone),
    click('form input[id="password"]'),
    type(password),
    click('form button[type="submit"]'),
    waitForNavigation,
    log('LinkedIn Login Complete')
  )

This function is customizable through its higher-order function that provides the emailOrPhone and password values as information to enter into the form.

If you're new to composing BotAction's, read more here in this overview documentation.

Assembling the Bot

Now let's assemble the Bot that will log in and take a screenshot of our LinkedIn feed:

Copy
// Separate Helper Function
const generateTimeStamp = (): string => {
  let x = new Date();

  // Title the screenshot file with a timestamp
  // "year-month-date-hours-minutes" ie "2020-8-21-13-20"
  return x.getFullYear() + '-' +
       ( x.getMonth() + 1 ) + '-' +
         x.getDate() + '-' +
         x.getHours() + '-' +
         x.getMinutes();
}

// Main script
(async() => {
  const browser = await puppeteer.launch({headless: false});
  const pages = await browser.pages();
  const page = pages.length === 0 ? await browser.newPage() : pages[0];

  // Bot
  const linkedin_bot = chain(
    login('linkedin-account@email.com', 'linkedin-account-password'),
    goTo('https://www.linkedin.com/feed/'),
    screenshot(generateTimeStamp()) // filename ie "2020-8-21-13-20.png"
  );

  await linkedin_bot(page);
})()

Simple, declarative, and composable. We can reuse our linkedin_bot multiple times, and the parts that compose it.

Challenge

How would you go about making linkedin_bot dynamic as in a function that accepts the auth information so the bot could be used in multiple pages with varying credentials?

Hint: Use a higher-order function

Going Further

How about scraping the news feed to like posts published by your a select group of people? Check out this working example for how.