How to write a background function - I'm new to this, sorry

Support
1

I have the following scripts and would like to run the “build-it-all” script nightly as a background function since grabbing our shopify products can take some time.

"scripts": {
    "test": "jest --watch",
    "get-products": "node shopify_editing/get-shopify-products",
    "get-shopify-pages": "node shopify_editing/get-shopify-pages",
    "get-sanity-pages": "node sanity_editing/get-sanity-pages",
    "get-blog-articles": "node blog_editing/get-shopify-blogs",
    "get-zendesk-articles": "node zendesk_editing/GetZDeskArticles",
    "build-it-all": "npm run get-products; npm run get-shopify-pages; npm run get-sanity-pages; npm run get-blog-articles; npm run get-zendesk-articles; npm run build-index;",
    "build-index": "node build_index"
  }

How do I accomplish something like this? Am I SOL when it comes to something as simple like “Hey, Netlify, run ‘build-it-all’ daily at midnight”? The documentation I have found seems overly complicated for my newbie mind. Any help on this would be greatly appreciated. So far, all I have completed is creating a functions folder: functions/search-background.mjs

2

You don’t have to write any script. You don’t even have to build the function. Just write the function and place it in the netlify/functions directory. Netlify will build the function itself along with the site.

3

I already have the scripts written - that is how I am accomplishing this specific task that I want to automate. I am unsure what you mean by “You don’t even have to build the function. Just write the function…”.

For more background: I currently run npm run build-it-all which then runs the five “get-*” scripts, then grabs all the resulting files those scripts created and runs the build-index script. If I want to run this nightly, I assume this will have to be six different functions (the five “get-*” and the build-index).

From your response, I took that to mean that I would grab the functions that those scripts are running and put those in their own respective function files. The function that get-blog-articles runs is:

const path = require('path');
const fs = require('fs');
require('dotenv').config();

const PromiseArraySerialize = require('../PromiseArraySerialize');
const {
  GetCredentials,
  GetBlogArticles,
} = require('../shopify_editing/ShopifyApi');

const stores = ['main', 'test', 'charter-school'];
const { Textify } = require('../common/Textify');
let storeToQuery = stores[0];

let credentials = GetCredentials('main');

async function getShopifyBlog() {
  let articles = await GetBlogArticles({ creds: credentials });
  articles.forEach((art) => {
    art.body_html = Textify(art.body_html);
  });

  return articles;
}

getShopifyBlog().then((articles) => {
  let finalRestingPlace = path.resolve(
    __dirname,
    '../_data',
    'shopifyBlog.json'
  );
  fs.writeFileSync(finalRestingPlace, JSON.stringify(articles), 'utf8');
});

How would I turn the above into a function?

4

I thought you’ve written scripts to build the function (as it says “build-it-all”).

As for convering this to a background function:

export default async function() {
  await runScript1() // pseudo-code
  await runScript2()
  // and so on...
}

I can’t answer the specific of how you can conver your scripts into a function as I don’t know your scripts to start with. As long as you have written Node.js code that a server can execute, you should simply be able to import those files and call the relevant functions. With your provided code:

import {GetBlogArticles, GetCredentials} from '../shopify_editing/ShopifyApi'
import PromiseArraySerialize from '../PromiseArraySerialize'
import {resolve} from 'node:path'
import {Textify} from '../common/Textify'
import {writeFileSync} from 'node:fs'
export default async function() {
  const articles = await await GetBlogArticles({
    creds: GetCredentials('main')
  })
  articles.forEach(art => {
    art.body_html = Textify(art.body_html)
  })
  writeFileSync(resolve(__dirname, '../data', 'shopifyBlog.json'), JSON.stringify(articles), 'utf8')
}

The specifics for this would change depending on your project structure. We cannot provide code-level assistance, so this is something you’d have to comtinue working on by referring to the documentation.

1 Like
5

Yes, the scripts are listed in my first post of this thread. I have been doing npm run build-it-all whenever I wanted to complete this overall task. The individual tasks are to get all of our blog posts from Shopify, get all the products from Shopify, get all our pages from Shopify, get all our pages from Sanity, and get all our articles from Zendesk, then build an Alogolia index from all that information. I can run each script separately if I just need to grab new pages from Zendesk, for example, by running npm run get-zendesk-articles then npm run build-index. But if I just want to get everything, I run npm run build-it-all.

If you look at my first post from this thread, you will see that each individual script points to their respective file. I didn’t want to copy each file in here as I thought it would get pretty cluttered, but I could do it if that would be helpful. In my second response, I provided the contents of just one of those files so that you could see an example of what I was working with.

I am fairly new to this side of things, so please excuse my lack of experience and knowledge.

Looking at your two replies I think I got a little more understanding, but I am not 100% I have it figured out. Considering the actual function of my scripts works, all I need to do is write ONE function file (/functions/buildAlgoliaIndex-background.mjs) that imports and awaits those functions, correct? Then I can just add a config after that to tell it when to run??

I created the following function file below and have pushed my changes to git, but am not seeing anything in my Functions area of Netlify, nor am I seeing Algolia update with the new information. I have a feeling I am missing something obvious here or I am just not understanding. I have read the documentation many times, but I am not connecting the dots.

import main from "apps/search-index/sanity_editing/get-sanity-pages"
import StoreZDeskArticles from "apps/search-index/zendesk_editing/GetZDeskArticles"
import { allShopifyPages } from "apps/search-index/shopify_editing/get-shopify-pages"
import { allBlogPages } from "apps/search-index/blog_editing/get-shopify-blogs"
import { allShopifyProducts } from "apps/search-index/shopify_editing/get-shopify-products"
import Main from "apps/search-index/build_index"

export default async function() {
    await main() // get Sanity pages
    await StoreZDeskArticles() // get Zendesk articles
    await allShopifyPages() // get Shopify pages
    await allBlogPages() // get blog posts from Shopify
    await allShopifyProducts() // get Shopify products
    await Main() // grab the results from the functions above and create an index

    console.log("Done");
  }
  export const config = {
    schedule: "10 * * * *" // timing set to a few minutes after I push to git
}
6

Oh my gosh. I deployed my site again and got this information. I am going to wait and see if it works… Holding my breath!! (okay, not actually holding my breath, but crossing my fingers!)

Screenshot 2024-03-25 at 1.56.19 PM
Screenshot 2024-03-25 at 1.56.19 PM2802×588 34.6 KB