Trying Lambda Functions with Puppeteer and getting Browser error

Support
1

Hey, I wasn’t sure if this is related to Netlify or Puppeteer, but does someone know what’s causing the following error: Browser is not downloaded. Run "npm install" or "yarn install".

It’s from a catch block in the the following code

const chromium = require('chrome-aws-lambda');
const puppeteer = require('puppeteer-core');

exports.handler = async function (event, context, callback) {
  const molUrl = 'https://paikat.te-palvelut.fi/tpt/?searchPhrase=web%20developer&locations=Oulu&announced=0&leasing=0&english=false&sort=1';
  let browser = null;
  let urls = null;
  
  try {
    // setup
    browser = await chromium.puppeteer.launch({
      executablePath: await chromium.executablePath,
      args: chromium.args,
      headless: chromium.headless,
      //ignoreDefaultArgs: ['--disable-extensions'],
      defaultViewport: chromium.defaultViewport,
    });
  
    // Do stuff with headless chrome
    const page = await browser.newPage();
    await page.goto(molUrl); 
    await page.waitForSelector('.resultsList a'); 

    urls = await page.evaluate(() => {
      let results = [];
      let items = document.querySelectorAll('.resultsList a');
      items.forEach((item) => {
        const itemShortUrlArray = /^.+[\?]/.exec(item.getAttribute('href'));
        const  itemShortUrl = itemShortUrlArray[0].slice(0, -1);

        results.push({
          url:  itemShortUrl,
          text: item.innerText,
        });
      });
      return results;
    });
    
    console.log('urls', {urls});
  } catch (err) {
    console.error('Catch error:', err.message);
  } finally {
    if (browser !== null) {
      await browser.close()
    }
  }
  
  return {
    statusCode: 200,
    body: JSON.stringify({ 
      urls: urls
    })
  };
}

Here’s the full local server log

$ npm run lambda-serve

> mol-job-search-automation@1.0.0 lambda-serve C:\xampp\htdocs\mol-job-search-automation
> netlify-lambda serve functions

netlify-lambda: Starting server
Hash: e27c2e9de00b1c3503e6
Version: webpack 4.41.6
Time: 2550ms
Built at: 2020-02-24 23:56:25
      Asset     Size  Chunks             Chunk Names
scrapeMol.js  679 KiB       0  [emitted]  scrapeMol
Entrypoint scrapeMol = scrapeMol.js
  [0] external "fs" 42 bytes {0} [built]
  [1] external "path" 42 bytes {0} [built]
  [4] ../node_modules/puppeteer/lib/helper.js 8.62 KiB {0} [built]
  [5] ../node_modules/puppeteer-core/lib/helper.js 8.62 KiB {0} [built]
  [7] external "url" 42 bytes {0} [built]
[13] external "https" 42 bytes {0} [built]
[109] ../node_modules/puppeteer-core/index.js 1.5 KiB {0} [built]
[111] ../node_modules/puppeteer-core/lib/Puppeteer.js 2.7 KiB {0} [built]
[112] ./scrapeMol.js 1.57 KiB {0} [built]
[113] ../node_modules/chrome-aws-lambda/source/index.js 5.91 KiB {0} [built]
[114] ../node_modules/lambdafs/build/index.js 3.72 KiB {0} [built]
[136] ../node_modules/chrome-aws-lambda/source/puppeteer/lib sync ^\.\/.*$
226 bytes {0} [built]
[155] ../node_modules/puppeteer/index.js 1.5 KiB {0} [optional] [built]
[191] ../node_modules/puppeteer-core/lib/api.js 1.75 KiB {0} [built]
[197] ../node_modules/puppeteer-core/package.json 3.34 KiB {0} [built]
    + 183 hidden modules

WARNING in ../node_modules/lambdafs/build/index.js 23:31-46
Critical dependency: the request of a dependency is an expression
@ ../node_modules/chrome-aws-lambda/source/index.js
@ ./scrapeMol.js

WARNING in ../node_modules/ws/lib/buffer-util.js
Module not found: Error: Can't resolve 'bufferutil' in 'C:\xampp\htdocs\mol-job-search-automation\node_modules\ws\lib'
@ ../node_modules/ws/lib/buffer-util.js
@ ../node_modules/ws/lib/receiver.js
@ ../node_modules/ws/index.js
@ ../node_modules/puppeteer-core/lib/WebSocketTransport.js
@ ../node_modules/puppeteer-core/lib/Launcher.js
@ ../node_modules/puppeteer-core/lib/Puppeteer.js
@ ../node_modules/puppeteer-core/index.js
@ ./scrapeMol.js

WARNING in ../node_modules/ws/lib/validation.js
Module not found: Error: Can't resolve 'utf-8-validate' in 'C:\xampp\htdocs\mol-job-search-automation\node_modules\ws\lib'
@ ../node_modules/ws/lib/validation.js
@ ../node_modules/ws/lib/receiver.js
@ ../node_modules/ws/index.js
@ ../node_modules/puppeteer-core/lib/WebSocketTransport.js
@ ../node_modules/puppeteer-core/lib/Launcher.js
@ ../node_modules/puppeteer-core/lib/Puppeteer.js
@ ../node_modules/puppeteer-core/index.js
@ ./scrapeMol.js
Lambda server is listening on 9000
Request from ::1: GET /scrapeMol.js
Catch error: Browser is not downloaded. Run "npm install" or "yarn install"
Response with status 200 in 246 ms.

I’m not sure if the warning about the Critical dependency is anything to do with this. I’ve googled it and haven’t been able to figure out a fix for that either. Other things to note, I’m working locally (netlify-lambda serve functions) on Windows.

If anyone has any advice, I would really appreciate it.

Thanks.

2

Hello @n2fole00, welcome to the community. It looks like you are trying to deploy a function using the netlify-lambda. Note that this method won’t work for any dependencies that have a binary component (which chrome-aws-lambda does, I assume).

Could you try using zip-it-and-ship-it or manually zipping your function? I have an example repo here that demonstrates all the different methods.

1 Like
3

Hello @Dennis
I’m having the exact same issue trying to use puppeteer on Windows with netlify-lamba build and serve. I wasn’t aware about the binary thing.

I’ve tried the following :
– manually zipping : tried to replicate and deploy your repo, I have a “function not found” message
– zip it and ship it : can’t use zip in the command line
– bundled : works with your repo, fail with chrome-aws-lambda
– unbundled with netlify dev : got error ‘Failed to launch the browser process! spawn D:\TEMP\chromium ENOENT\n’
– webpack-node-externals as described in the docs : same error as OP
– similar repo (GitHub - ireade/netlify-puppeteer-screenshot-demo) with netlify dev : ‘Failed to launch chrome! spawn D:\TEMP\chromium ENOENT\n’

Any help would be appreciate.
Have a good day

4

Hi, thanks for the suggestion. Bare with me…I’m pretty new to Node, but I tried the following.

Added

"build": "npm run build:bundled; npm run build:zipped",
"build:bundled": "netlify-lambda build lambda/bundled-function",
"build:zipped": "cd lambda/zipped-function; npm i; cd ..; zip -r ../functions/zipped-function.zip zipped-function"

to package.json (“scripts”)

Changed netlify.toml to

[build]
functions = "functions/"

Then ran

$ npm run build

> mol-job-search-automation@1.0.0 build C:\xampp\htdocs\mol-job-search-automation
> npm run build:bundled; npm run build:zipped

npm ERR! missing script: build:bundled;
npm ERR!
npm ERR! Did you mean one of these?
npm ERR!     build:bundled
npm ERR!     build:zipped

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\leke\AppData\Roaming\npm-cache\_logs\2020-02-27T21_55_48_084Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! mol-job-search-automation@1.0.0 build: `npm run build:bundled; npm run build:zipped`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the mol-job-search-automation@1.0.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\leke\AppData\Roaming\npm-cache\_logs\2020-02-27T21_55_48_187Z-debug.log
5

Hmm, not sure, but that does indeed look like a problem with package.json. Perhaps it doesn’t parse the semicolon next to the command well, and npm run build:bundled ; npm run build:zipped will work better?

6

Hey, thanks for the reply. Yeah, seems to be a Windows issue. I’ll try this on Linux instead.

1 Like
7

let us know what you figure out!