Functions: node-cron fetch from external api and write results to file

Support
,
1

Hi I’m new to Netlify and lambda functions.

I have a small app that connects to an external api. The api has rate limits, so I am attempting to periodically connect the the api and write the response to the file system and then serve up that response as a new endpoint.

Is this even possible?

My idea was to use node-cron to trigger a fetch and use fs.writeFileSync to save the response to the file system. Nothing I have tried is working. I’m still stuck at the fetch and write to file. I haven’t written the cron in yet.

something along these lines. Any thoughts?

const fetch = require('node-fetch')
const fs = require('fs')

const checkStatus = (res) => {
  if (res.status >= 200 && res.status < 300) {
    return res.json()
  } else {
    let err = new Error(res.statusText)
    err.response = res
    throw err
  }
}

exports.handler = async function (event, context, callback) {
  const apiUrl = 'https://some-api-requuest/'

  try {
    const response = await fetch(apiUrl)
    let data = await checkStatus(response)
    data = JSON.stringify(data)

    fs.writeFileSync('./response.json', data)

    const bodyObj = {
      statusCode: 200,
      body: 'success'
    }

    callback(null, bodyObj);
  } catch (err) {
    callback(err);
  }
}
2

Hey @gregcarv,
I think iz… not possible. Netlify Functions are supposed to be ephemeral- no permanent storage, no IP address you can reliably access again. When you call one, it spins up on a random server somewhere and runs the code it’s supposed to run and then goes away. The next time you call it, it may spin up on a different server, maybe in a different region. So while you probably could write data to a file, it wouldn’t be accessible to you in the future because you likely wouldn’t be on the same filesystem. That said, if the response is small enough, maybe you wouldn’t have to write it out- you could just store it as a variable in memory?

Furthermore, Netlify abstracts away a lot of the complicatedness of AWS Lambdas (that’s what Netlify Functions are under the hood) but this also means you don’t have access to things you might if you were using AWS directly. This post may be interesting to you:

I also don’t think the cron job would work because, with the Jamstack, there’s no server at runtime. There are serverless workarounds (Hey, let's create a functional calendar app with the JAMstack | CSS-Tricks - CSS-Tricks) but I don’t know of a way to implement a cron job with Netlify Functions.

So probably not a good fit for what you’re trying to do. But Netlify Functions can do lots of other cool stuff if you wanna poke around in here:

Let us know if we can answer any other questions on this!

3

Thanks, that’s pretty much what I thought. I am trying to find some alternate way to do it. At this moment I am leaning more towards running the job locally and triggering a commit and push. Thank you for the feedback!

4

I came up with a different solution. I set up a second netlify site that connects to a smaller repo which only includes a lambda functions file with data.

I used windows task scheduler to run a cmd file with my node and git commands in it:

  1. fetch data
  2. update a file with the data and the lambda functions
  3. commit
  4. push

I have 2 folders in my commit: public & functions
Public just has a dummy index.html page
Functions has my lambda function with the body as the data.

My app now uses the endpoint created with the lambda functions in the second site, and my schedule only runs 3 times a day.

3 Likes