Trigger Netlify to add new Airtable data

Support
,
1

I am using airtable as my backend, but the data on my site only updates when netlify rebuilds, which is triggered by a commit push or by a build hook. I’m assuming this is because I’m using gatsby to access airtable and not the airtable API directly.

At the moment, I can run an airtable script with a fetch version of the build hook:

fetch("https://api.netlify.com/build_hooks/xxxx", {
  body: "{}",
  headers: {
    "Content-Type": "application/x-www-form-urlencoded"
  },
  method: "POST"
})

But I can only do this with the pro plan on airtable, which is $240/year, which isn’t great for a student budget. I thought about doing a crontab script, but I don’t want the trigger to be local, in case I eventually leave the project for which the site is built, and in case my computer is off and can’t actually trigger.

Is there any other (cheap) way I can trigger netlify to rebuild?

Thanks,
Z

2

Hey @zwolf

You could use a Scheduled Function which can run at set intervals such as @daily, or @weekly or more fine-grained. While they are not necessarily production-ready (they are still part of Netlify Labs) the function is deployed alongside the rest of the site.

3

Hi @coelmay,

Thank you for the suggestion! I tried setting it up, following this tutorial by @raymondcamden (hi!), and running last night, but today - no dice. The last build was yesterday evening when I installed the function.

image

const { schedule } = require('@netlify/functions')
const fetch = require('node-fetch');

const REBUILD_URL = process.env.GATSBY_NETLIFY_BUILD_HOOK;

const handler = async function(event, context) {
    console.log("Received event:", event)

    await fetch(REBUILD_URL, { method: 'POST'});

    return {
        statusCode: 200,
    };
};

module.exports.handler = schedule("@daily", handler);

Netlify sees the file though.

image
image1262×804 25.5 KB

Any idea what I messed up?

Thanks,
Z

4

I’m not sure you’ve messed anything up @zwolf

Try changing the logging period from “Real-time” to a greater time frame to see if anything was logged e.g.

scheduled-function-log-note
scheduled-function-log-note2394×1342 238 KB

You might think about using a try...catch to handle any exceptions.

5

Nada. I also added fake data to my airtable last night to see if the site would show it in the morning as a quick and easy check, so I had that to help support the conclusion that it didn’t work, haha.

Will this work?

const { schedule } = require('@netlify/functions')
const fetch = require('node-fetch');

const REBUILD_URL = process.env.GATSBY_NETLIFY_BUILD_HOOK;

const handler = async function(event, context) {
    console.log("Received event:", event)

    try{
        let response = await fetch(REBUILD_URL, { method: 'POST'});
        if (!response.ok){
            throw new Error(response.statusText);
        }
    } catch (err){
        console.log('automated_build_function error: ', err)
    }

    return {
        statusCode: 200,
    };
};

module.exports.handler = schedule("@daily", handler);
6

At a glance, yes.

If you change @daily to @hourly while testing, you have more debugging chances (and switch to @daily again once you know things are working.)

7

Already done. I’ll post in a little bit if there’s an update. :slight_smile:

image

8

Got a response this time!

May 3, 09:00:01 PM: ab9cd85c INFO   Received event: {
May 3, 09:00:01 PM: ab9cd85c   rawUrl: 'https://main--oceanaffinity-starter.netlify.app/.netlify/functions/automate_build_function',
May 3, 09:00:01 PM: ab9cd85c   rawQuery: '',
May 3, 09:00:01 PM: ab9cd85c   path: '/.netlify/functions/automate_build_function',
May 3, 09:00:01 PM: ab9cd85c   httpMethod: 'POST',
May 3, 09:00:01 PM: ab9cd85c   headers: {
May 3, 09:00:01 PM: ab9cd85c     'accept-encoding': 'gzip',
May 3, 09:00:01 PM: ab9cd85c     'content-length': '35',
May 3, 09:00:01 PM: ab9cd85c     'content-type': 'application/json',
May 3, 09:00:01 PM: ab9cd85c     host: 'main--oceanaffinity-starter.netlify.app',
May 3, 09:00:01 PM: ab9cd85c     'user-agent': 'Netlify Clockwork',
May 3, 09:00:01 PM: ab9cd85c     'x-country': 'US',
May 3, 09:00:01 PM: ab9cd85c     'x-forwarded-for': '18.190.67.238, 100.64.0.22',
May 3, 09:00:01 PM: ab9cd85c     'x-forwarded-proto': 'https',
May 3, 09:00:01 PM: ab9cd85c     'x-netlify-event': 'schedule',
May 3, 09:00:01 PM: ab9cd85c     'x-nf-client-connection-ip': '18.190.67.238',
May 3, 09:00:01 PM: ab9cd85c     'x-nf-request-id': '01G2699GP7E91VNAYNYYADXY7Y',
May 3, 09:00:01 PM: ab9cd85c     'x-webhook-signature': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJuZXRsaWZ5Iiwic2hhMjU2IjoiNThmYTNhM2Q2ZTg5NmJmNmFmMmE0NjM0YjY3ZDkzOWJlNTU5YjY0NGJjODMxOGJlNzAyMGE3ZGU5ZDM2MGExYSJ9.7be3eFYqAq45qKOPDy9YuS8VIWxWkh4oxq6lQ4aBSj8'
May 3, 09:00:01 PM: ab9cd85c   },
May 3, 09:00:01 PM: ab9cd85c   multiValueHeaders: {
May 3, 09:00:01 PM: ab9cd85c     'Accept-Encoding': [ 'gzip' ],
May 3, 09:00:01 PM: ab9cd85c     'Content-Length': [ '35' ],
May 3, 09:00:01 PM: ab9cd85c     'Content-Type': [ 'application/json' ],
May 3, 09:00:01 PM: ab9cd85c     'User-Agent': [ 'Netlify Clockwork' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Country': [ 'US' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Forwarded-For': [ '18.190.67.238, 100.64.0.22' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Forwarded-Proto': [ 'https' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Netlify-Event': [ 'schedule' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Nf-Client-Connection-Ip': [ '18.190.67.238' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Nf-Request-Id': [ '01G2699GP7E91VNAYNYYADXY7Y' ],
May 3, 09:00:01 PM: ab9cd85c     'X-Webhook-Signature': [
May 3, 09:00:01 PM: ab9cd85c       'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJuZXRsaWZ5Iiwic2hhMjU2IjoiNThmYTNhM2Q2ZTg5NmJmNmFmMmE0NjM0YjY3ZDkzOWJlNTU5YjY0NGJjODMxOGJlNzAyMGE3ZGU5ZDM2MGExYSJ9.7be3eFYqAq45qKOPDy9YuS8VIWxWkh4oxq6lQ4aBSj8'
May 3, 09:00:01 PM: ab9cd85c     ],
May 3, 09:00:01 PM: ab9cd85c     host: [ 'main--oceanaffinity-starter.netlify.app' ]
May 3, 09:00:01 PM: ab9cd85c   },
May 3, 09:00:01 PM: ab9cd85c   queryStringParameters: {},
May 3, 09:00:01 PM: ab9cd85c   multiValueQueryStringParameters: {},
May 3, 09:00:01 PM: ab9cd85c   body: '{"next_run":"2022-05-04T02:00:00Z"}',
May 3, 09:00:01 PM: ab9cd85c   isBase64Encoded: false
May 3, 09:00:01 PM: ab9cd85c }
May 3, 09:00:01 PM: ab9cd85c INFO   automated_build_function error:  TypeError: Only absolute URLs are supported
May 3, 09:00:01 PM: ab9cd85c     at getNodeRequestOptions (/var/task/node_modules/node-fetch/lib/index.js:1327:9)
May 3, 09:00:01 PM: ab9cd85c     at /var/task/node_modules/node-fetch/lib/index.js:1440:19
May 3, 09:00:01 PM: ab9cd85c     at new Promise (<anonymous>)
May 3, 09:00:01 PM: ab9cd85c     at fetch (/var/task/node_modules/node-fetch/lib/index.js:1437:9)
May 3, 09:00:01 PM: ab9cd85c     at Runtime.handler (/var/task/netlify/functions/automate_build_function.js:10:30)
May 3, 09:00:01 PM: ab9cd85c     at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)
May 3, 09:00:01 PM: ab9cd85c  Duration: 16.64 ms	Memory Usage: 66 MB	Init Duration: 392.10 ms

So, I’ve spent the last little bit trying to understand relative vs absolute urls. My REBUILD_URL is simply the build hook, https://api.netlify.com/build_hooks/xxxx. For some reason, the host is listed as main–oceanaffinity.netlify.app, which is the netlify subdomain for the site. I thought maybe it was because the build hook was originally made for the subdomain (the actual domain was recently purchased), so I deleted the build hook, created a new one linked (linked to ‘main’, the only option), and tried it again. Same thing.

May 3, 10:00:02 PM: 6799a315 INFO   Received event: {
  rawUrl: 'https://main--oceanaffinity.netlify.app/.netlify/functions/automate_build_function',
  rawQuery: '',
  path: '/.netlify/functions/automate_build_function',
  httpMethod: 'POST',
  headers: {
    'accept-encoding': 'gzip',
    'content-length': '35',
    'content-type': 'application/json',
    host: 'main--oceanaffinity.netlify.app',
    'user-agent': 'Netlify Clockwork',
    'x-country': 'US',
    'x-forwarded-for': '18.190.67.238, 100.64.0.62',
    'x-forwarded-proto': 'https',
    'x-netlify-event': 'schedule',
    'x-nf-client-connection-ip': '18.190.67.238',
    'x-nf-request-id': '01G26CQCPSJ49R4Q6STXW7SREN',
    'x-webhook-signature': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJuZXRsaWZ5Iiwic2hhMjU2IjoiNjAzZjllOTlmODQyZmJkOTZkOTVmNTNhZDExODUyY2JhMzNlYWRjNjRjZjFiYjk0ODJjNTQ4MTI0YzY3ZmQwNyJ9.6AclyvXiD0nZ5btjvNIhcdpgDOA4GDtIZHksmBUYb_0'
  },
  multiValueHeaders: {
    'Accept-Encoding': [ 'gzip' ],
    'Content-Length': [ '35' ],
    'Content-Type': [ 'application/json' ],
    'User-Agent': [ 'Netlify Clockwork' ],
    'X-Country': [ 'US' ],
    'X-Forwarded-For': [ '18.190.67.238, 100.64.0.62' ],
    'X-Forwarded-Proto': [ 'https' ],
    'X-Netlify-Event': [ 'schedule' ],
    'X-Nf-Client-Connection-Ip': [ '18.190.67.238' ],
    'X-Nf-Request-Id': [ '01G26CQCPSJ49R4Q6STXW7SREN' ],
    'X-Webhook-Signature': [
      'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJuZXRsaWZ5Iiwic2hhMjU2IjoiNjAzZjllOTlmODQyZmJkOTZkOTVmNTNhZDExODUyY2JhMzNlYWRjNjRjZjFiYjk0ODJjNTQ4MTI0YzY3ZmQwNyJ9.6AclyvXiD0nZ5btjvNIhcdpgDOA4GDtIZHksmBUYb_0'
    ],
    host: [ 'main--oceanaffinity.netlify.app' ]
  },
  queryStringParameters: {},
  multiValueQueryStringParameters: {},
  body: '{"next_run":"2022-05-04T03:00:00Z"}',
  isBase64Encoded: false
}
May 3, 10:00:02 PM: 6799a315 INFO   automated_build_function error:  TypeError: Only absolute URLs are supported
    at getNodeRequestOptions (/var/task/node_modules/node-fetch/lib/index.js:1327:9)
    at /var/task/node_modules/node-fetch/lib/index.js:1440:19
    at new Promise (<anonymous>)
    at fetch (/var/task/node_modules/node-fetch/lib/index.js:1437:9)
    at Runtime.handler (/var/task/netlify/functions/automate_build_function.js:13:30)
    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)
May 3, 10:00:02 PM: 6799a315 Duration: 17.89 ms	Memory Usage: 66 MB	Init Duration: 336.81 ms

Thoughts?

9

This is expected. Everything like this is done with the netlify.app domain, not the custom domain.

Absolute: https://example.com/some/path/to/somewhere.html
Relative: /some/path/to/somewhere.html
You can find more in this post from @perry Linking between two sites on the same team - #4 by perry

Here is the scheduled function I was using to auto rebuild a site. Instead of using the build hook, it used the API directly.

10

I guess I’m confused about how the build hook fails to qualify as an absolute url. It starts with https://, it is a complete* url and not a subset of one, and it is not a path to a local file. I don’t get it.

Thank you for creating an example using the API. I have a really stupid question and I can’t seem to find it in the documentation. If process.env.SITE_ID is the site id listed under Site Settings > Site Details > Site Information, and process.env.API_UA_STRING is the Personal Access Token under my profile > Applications, is process.env.NETLIFY_API_ID my OAuth Client ID?

11

SITE_ID is one of the Read Only Deploy Metadata available in Netlify. I used this rather than create a separate variable.

Not OAuth, it is a Personal Access Token see here.

This is generally something like your app name and email e.g MyApp (me@example.com)

12 

May 4, 11:00:03 AM: 0d56ded6 ERROR  Invoke Error 	
{"errorType":"FetchError",
"errorMessage": "invalid json response body at https://api.netlify.com/api/v1/sites/undefined/builds reason: Unexpected end of JSON input",
"message":"invalid json response body at https://api.netlify.com/api/v1/sites/undefined/builds reason: Unexpected end of JSON input",
"type":"invalid-json",
"stack":
["FetchError: invalid json response body at https://api.netlify.com/api/v1/sites/undefined/builds reason: Unexpected end of JSON input",
"    at /var/task/netlify/functions/automate_rebuild_api.js:5625:40",
"    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
"    at async Runtime.handler (/var/task/netlify/functions/automate_rebuild_api.js:6510:18)"
]}

May 4, 11:00:03 AM: 0d56ded6 Duration: 87.00 ms	Memory Usage: 66 MB

So, it is saying that the site ID is undefined. I’ve triple checked that the site ID in my .env file matches the Site ID for my website given under Site Information. I do not use commas, semicolons, or quotations when defining my environmental variables. I printed the environmental variable out to the console from a component to make sure that it is at least accessed correctly and that the variable name written in the URL is correct. And I double checked that node-fetch is indeed installed.

The UA Token is the personal access token and the UA String is “OAHU (me@gmail.com)”, but without quotations in the .env.

What am I missing?

import fetch from "node-fetch";

const handler = async function(event, context) {


    // Trigger a rebuild
    const postUrl = `https://api.netlify.com/api/v1/sites/${process.env.GATSBY_NETLIFY_SITE_ID}/builds`;

    const response = await fetch(postUrl, {
        method: "POST",
        headers: {
            Authorization: `Bearer ${process.env.GATSBY_NETLIFY_UA_TOKEN}`,
            "User-Agent": `${process.env.GATSBY_API_UA_STRING}`
        }
    });

    const result = await response.json();
    console.log("Result: ", result);

    return {
        statusCode: 200,
    };
};

module.exports.handler = handler;

Edit: I’m possibly an idiot. My site ID was copied exactly, including the dashes/hyphens/whatever. I was going through the documentation some more and found an example sans hyphens. Gonna run it again and post an update at 6ish.

Edit 2 Electric Boogaloo: Same error. Jury’s still out on whether I’m an idiot though.

13

Found this thread and this one talking about how/why gatsby environmental variables might be undefined in deployment. Going to try removing ‘GATSBY_’ prefix from my netlify variables, since they aren’t used in components, and see how that goes.

1 Like
14

Same errors as before. Absolute url error for the build hook scheduled function and /site/undefined/builds for the API post scheduled function.

Any thoughts?

15

Hey @zwolf

I deployed a test with the same function and it worked perfectly, so I’m not sure why it is not working for you.

But then, looking again at the log, the SITE_ID is undefined. Can you try again using ${process.env.SITE_ID} as in the demo function I linked to (this is what I used.) This is not a value you set, this is a value provided by Netlify. The only values you need to set are the API token and UA string.

16

Hi, @zwolf. The webhook URL that triggers builds at Netlify doesn’t require authentication to use it. Also, it can be a GET. You don’t need to POST to that URL.

From what I can see mistake here is this URL:

    const postUrl = `https://api.netlify.com/api/v1/sites/${process.env.GATSBY_NETLIFY_SITE_ID}/builds`;

The build hook URL looks like this:

https://api.netlify.com/build_hooks/<id code here>

If you make the GET (or POST) to that URL you won’t need to login. You were using a URL like that in your original post. If you do the same in the scheduled function you should be able to get working much more easily.

17

Good afternoon @coelmay and @luke!

@coelmay…weird. If it’s a value that netlify provides, why is it coded as an environmental variable?

@luke, welcome to the rollercoaster of all the things I’m apparently capable of breaking. I currently have two methods of trying to automate the netlify build. The first is using the build hook, ala this tutorial:

const { schedule } = require('@netlify/functions')
const fetch = require('node-fetch');

const handler = async function(event, context) {
    console.log("Received event:", event)

    const REBUILD_URL = process.env.GATSBY_NETLIFY_BUILD_HOOK+'?trigger_branch=main&trigger_title=triggered+by+This+Awesome+Service';

    try{
        let response = await fetch(REBUILD_URL, { method: 'POST'});
        if (!response.ok){
            throw new Error(response.statusText);
        }
    } catch (err){
        console.log('automated_build_function error: ', err)
    }

    return {
        statusCode: 200,
    };
};

module.exports.handler = schedule("@hourly", handler);

This code returns a ‘Only absolute URLs are supported’ error.

And the second was provided by @coelmay:

import fetch from "node-fetch";

const handler = async function(event, context) {


    // Trigger a rebuild
    const postUrl = `https://api.netlify.com/api/v1/sites/${process.env.SITE_ID}/builds`;


    const response = await fetch(postUrl, {
        method: "POST",
        headers: {
            Authorization: `Bearer ${process.env.NETLIFY_API_ID}`,
            "User-Agent": `${process.env.API_UA_STRING}`
        }
    })

    const result = await response.json();
    console.log("Result: ", result);

    return {
        statusCode: 200,
    };
};

module.exports.handler = handler;

The good news is that I am no longer receiving an {undefined} error where the site id should be. WOO!

The bad news is that I’m getting a new error. (Site id is accurate but redacted.)

May 6, 01:00:04 PM: 808ea3ea ERROR  Invoke Error 	{"errorType":"FetchError","errorMessage":"invalid json response body at https://api.netlify.com/api/v1/sites/xxxx/builds reason: Unexpected end of JSON input","message":"invalid json response body at https://api.netlify.com/api/v1/sites/xxxx/builds reason: Unexpected end of JSON input","type":"invalid-json","stack":["FetchError: invalid json response body at https://api.netlify.com/api/v1/sites/xxxx/builds reason: Unexpected end of JSON input","    at /var/task/netlify/functions/automate_rebuild_api.js:5625:40","    at processTicksAndRejections (internal/process/task_queues.js:95:5)","    at async Runtime.handler (/var/task/netlify/functions/automate_rebuild_api.js:6510:18)"]}
May 6, 01:00:04 PM: 808ea3ea Duration: 93.80 ms	Memory Usage: 66 MB

Are you impressed with my ability to break literally every step of code yet? Cause I’m rolling.

18

Because it is an environment variable, a built-in read-only variable as shown in the deploy metadata link I previously shared.

19

Ohhhh. Cue lightbulb moment. They weren’t presented as process.env variables and with the lack of sleep and without that context I guess it just didn’t click. Thank you for being patient and explaining it again. I appreciate it, truly.

20

As a little more insight, this scheduled function was part of my FYDLI project which initially used a scheduled function to rebuild the _redirects file (no longer does though.) To make it easier for others to deploy the project, (and because there was an issue with the NetlifyAPI from the netlify package—see here) I opted to use the SITE_ID environment variable to remove an additional set-up step in creating a build hook (in hindsight, easier to create a build hook and configure a single variable than it is to create an API token and configure two variables.)

So by no means is what I have done the only way. Using a build hook should work just the same with the added benefit as @luke pointed out of not requiring authentication.