I’m doing some side-project with scraping dynamic contents page with chrome-aws-lambda.
After i success to deploy to netlify. Somehow, i get error ERROR Task timed out after 10.01 seconds randomly. . I tried with local development and still working normal ( you can see the log below).
I was follow-up this topic to make my function run-once ( to reduce the load time too) but still get the error
> netlify dev
◈ Netlify Dev ◈
◈ Ignored general context env var: LANG (defined in process)
◈ Ignored general context env var: LANGUAGE (defined in process)
◈ No app server detected and no "command" specified
◈ Using current working directory
◈ Unable to determine public folder to serve files from
◈ Setup a netlify.toml file with a [dev] section to specify your dev server settings.
◈ See docs at: https://cli.netlify.com/netlify-dev#project-detection
◈ Running static server from "scrap-nodeJS"
◈ Loaded function cases.
◈ Loaded function hello.
◈ Functions server is listening on 37945
◈ Server listening to 3999
┌─────────────────────────────────────────────────┐
│ │
│ ◈ Server now ready on http://localhost:8888 │
│ │
└─────────────────────────────────────────────────┘
◈ Rewrote URL to /.netlify/functions/hello
Request from ::1: GET /.netlify/functions/hello
Response with status 200 in 21 ms.
◈ Rewrote URL to /.netlify/functions/cases
Request from ::1: GET /.netlify/functions/cases
Done
Response with status 200 in 2997 ms.
This is my current code :
const chromium = require('chrome-aws-lambda');
exports.handler = async function (event, context) {
let browser;
const launchBrowser = async () => {
if (browser) return;
browser = await chromium.puppeteer.launch({
executablePath: await chromium.executablePath,
args: chromium.args,
defaultViewport: chromium.defaultViewport,
headless: true,
});
};
try {
await launchBrowser()
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (req) => {
// prevent render css,img,only render text
if (req.resourceType() === 'stylesheet' || req.resourceType() == 'font' || req.resourceType() == 'image') {
req.abort();
} else {
req.continue();
}
});
await page.goto('https://vnexpress.net/covid-19/covid-19-viet-nam');
const element = await page.waitForSelector('.total_case_today'); // select the element
const cases = await element.evaluate(el => el.textContent); // grab the textContent from the
return {
statusCode: 200,
body: JSON.stringify({
cases: cases
}),
};
} catch (error) {
console.log(error);
} finally {
console.log("Done");
}
};
so sad. Any chance to increased the max time?.