Builds - Specifying a data source for builds

On Gatsby Cloud, we previously used the x-gatsby-cloud-data-source header to specify which data source should be used when triggering a webhook build. See original Gatsby Cloud docs here. This feature was incredibly useful for our use-case, as it allowed us to trigger builds for high-priority data while maintaining a cached state for non-critical nodes.

During testing for our migration to Netlify, we’ve picked up that this header no longer works. Has anyone figured out an alternative way to target specific source plugins when triggering builds/deploys with the Netlify API (or build hooks)?

@SeanMcleod It may not be a direct swap in replacement, but I suspect you could achieve this by leveraging the Payload of a Netlify Build Hook.

See:
https://docs.netlify.com/configure-builds/build-hooks/#payload

You can send a custom payload in your build hook POST request. The contents must be passed as a string, which will be URL-encoded and made available in the triggered build as an environment variable. You can access it in the build by using the variable INCOMING_HOOK_BODY.

1 Like

Thanks @nathanmartin - I did run across this option and managed to get a proof-of-concept working.

There were a few pitfalls I found with using the webhook build payload (although I may be missing something), mainly that internal source plugin changes were necessary to get it up and running. This is fine for us as we maintain multiple private source plugins, but others might run into trouble if they’re using third-party plugins.

Realistically what I was aiming to achieve was something close to Gatsby Cloud’s Cloud/Data Update Builds, but this post makes it clear that this probably isn’t achievable.

Although incremental builds are working in some fashion, equivalent builds have gone from 10-20 seconds in Gatsby Cloud to ~2 minutes on Netlify which is more than a little disheartening.

@SeanMcleod Great work!

I’ve only ever used Gatsby under duress, (inherited some projects), as I always found it inherently slow and have never been big into react.

The purchase of Gatsby by Netlify definitely hasn’t done it any favors thus far in regards to build speed.

1 Like