Firebase: include GOOGLE_APPLICATION_CREDENTIALS json file in bundle

JeffML · October 9, 2019, 7:51pm

Hello,

I was getting a cryptic error when trying to bundle up the firebase code with netlify-lambda. That issue was fixed by following these instructions.

With that problem solved, my next is to get the credentials loaded via the GOOGLE_APPLICATION_CREDENTIALS environment variable. I can add this to my environment in the Build & deploy settings in the Netflify project console:

GOOGLE_APPLICATION_CREDENTIALS = "./mycreds-xxx.json"

However, the json file is not included in functions-build. It looks like Webpack 4 is being used, and I’ve tried adding a require("./mycred-xxx.json") in the function javascript, but that didn’t work.

I wonder if someone with Webpack experience could help me out? I’m sure the solution is simple, but I’ve not found it yet.

JeffML · October 10, 2019, 4:03am

As per the instructions linked in my OP, I created a webpack.functions.js, changed the build instructions to use that for the functions folder contents. Now I need to get my google credentials JSON file into the functions-build folder to bundle (I think that’s how it works). To do that I added a plugin to copy the credentials file to the output folder:

const nodeExternals = require('webpack-node-externals');
const CopyPlugin = require('copy-webpack-plugin');

module.exports = {
  externals: [nodeExternals()],
  plugins: [
    new CopyPlugin([
      { from: 'pgnfen2-xxx.json', to: '../functions-build' },
    ]),
  ],
};

and verified as working when I run yarn run build:functions. Next I added GOOGLE_APPLICATION_CREDENTIALS to the Build & deploy environment variables, using pgnfen2-xxx.json the path value.

Still getting the following error when calling the function via HTTPS GET:

{"errorType":"Error","errorMessage":"The file at pgnfen2-xxx.json does not exist, or it is not a file. ENOENT: no such file or directory, lstat '/var/task/pgnfen2-xxx.json'"

Here’s the relevant stuff I see in the deploy log:

8:36:46 PM: Different functions path detected, going to use the one specified in the toml file: 'functions-build' versus '/functions' in the site
8:36:46 PM: Creating functions prep folder
...
8:37:26 PM: > chessmap@0.1.0 build:functions /opt/build/repo
8:37:26 PM: > netlify-lambda build functions -c ./webpack.functions.js
8:37:26 PM: netlify-lambda: Building functions
8:37:28 PM: Creating an optimized production build...
8:37:28 PM: Hash: 62d374a2be1915e4b5d1
8:37:28 PM: Version: webpack 4.40.2
8:37:28 PM: Time: 1392ms
8:37:28 PM: Built at: 10/10/2019 3:37:28 AM
8:37:28 PM:                                        Asset      Size  Chunks             Chunk Names
8:37:28 PM: ../functions-build/pgnfen2-xxx.json  2.25 KiB          [emitted]
8:37:28 PM:                              firebaseTest.js  3.55 KiB       0  [emitted]  firebaseTest
8:37:28 PM:                            quickstartTest.js  3.61 KiB    1, 0  [emitted]  quickstartTest
8:37:28 PM:                                      test.js  1.22 KiB       2  [emitted]  test
8:37:28 PM: Entrypoint firebaseTest = firebaseTest.js
8:37:28 PM: Entrypoint quickstartTest = quickstartTest.js
8:37:28 PM: Entrypoint test = test.js
8:37:28 PM: [0] ./firebaseTest.js 675 bytes {0} {1} [built]
8:37:28 PM: [1] external "@google-cloud/firestore" 42 bytes {0} {1} [built]
8:37:28 PM: [2] ./pgnfen2-0dcc7c57af09.json 2.25 KiB {0} {1} [built]
8:37:28 PM: [3] ./quickstartTest.js 93 bytes {1} [built]
8:37:28 PM: [4] ./test.js 514 bytes {2} [built]
8:37:28 PM: [5] external "apollo-server-lambda" 42 bytes {2} [built]
8:37:37 PM: Compiled with warnings.

I tried changing the path to ../functions-build/pgnfen2-xxx.json as shown in the deploy log, but get a new error complaining that /var/functions-build does not exist.

Just need to get that JSON credentials file in a path where it can be reached via the environment variable. Help?

fool · October 14, 2019, 3:59pm

Hi Jeff,

I think you’d want to make sure that file is next to your function file, in your functions directory, after we build your site (and before we deploy it), and then refer to it as ./pngfen2-xxx.json in your code.

JeffML · October 14, 2019, 7:12pm

I’ve done that without any luck.

Sadly, I’m back to this error now when I run start:server and go to my function via local URL:

Function invocation failed: TypeError [ERR_INVALID_ARG_TYPE]: The “path” argument must be of type string. Received type number

I had fixed this problem by adding a separate webpack config file to functions folder, but I accidently typed ‘yarn’ instead of ‘yarn build’ recently and now it’s back and I am not able to get rid of it. Deploying to netlify using the CLI gives me a different issue:

{“errorType”:“Error”,“errorMessage”:"The file at ./pgnfen2-xxx.json does not exist, or it is not a file.

Even though, if I build locally, this .json file is placed into the functions-build target folder for webpack. This is why I think the json file gets dropped.

According to the docs, I shouldn’t have to bundle the functions anymore, I can deploy them as is. If that’s the case, I can try getting rid of netlify-lambda in my builds and avoid webpack altogether and just use the Netlify Dev process.

JeffML · October 14, 2019, 10:55pm

I decided to simplify by removing netlify-lambda and webpack from my build process. This means I have a single functions folder with my .js and .json files. I changed my netlify.toml file to direct it to /functions, not /function-build. I run netlify deploy and it deploys okay. Still getting this error though:

{"errorType":"Error","errorMessage":"The file at ./pgnfen2-0dcc7c57af09.json does not exist, or it is not a file.

Okay, so @futuregerald suggested I try the zip-it-and-ship-it utility. I added the module, created a utility script as shown in the docs:

 const { zipFunctions } = require('@netlify/zip-it-and-ship-it')

zipFunctions('functions', 'functions-dist').catch(e => { throw e })

(Note, this script will throw an exception if the function-dist folder does not already exist)

What I seeis this:

The destination folder does not have a zip version of the JSON file from the original function folder. Now if I deploy (after changing the .toml file to point to the functions-dist folder), I get the exact same error as before:

{“errorType”:“Error”,“errorMessage”:"The file at ./pgnfen2-0dcc7c57af09.json does not exist, or it is not a file.

Since netlify-lambda uses the zip-it package, I wonder if it is related?

I know there is a programatic way to validate the Google credentials using an API, so maybe that is what I’ll be forced to do.

JeffML · October 15, 2019, 8:17pm

Stubborn as I am, I gave it one last shot after looking at zip-it-and-ship-it more closely. Here’s what I finally did:

added a local npm project creds to the functions folder
used yarn to add it to function/node_modules (yarn add file:./creds)
included a require('creds') in my lambda function
added a build:functions script that executes zip-it-and-ship-it on the functions folder (output to functions-dist)
change netlify.toml to point functions to functions-dist
set export GOOGLE_APPLICATION_CREDENTIALS="./node_modules/creds/pgnfen2-0dcc7c57af09.json"

When I run build:functions, I can check the node-modules/creds folder and see that there’s a dummy index.js (for export) and my .json credentials file. So far, so good. Now I run netlify deploy, which is successful.

Now I go to my browser and enter /,netlify/functions/firebaseTest in the address bar and hit ENTER. I get an error back:

{"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'creds'",...

To reiterate: when I build locally I have a functions/node_modules folder with “creds” in it. Apparently it’s not there during deployment.

Giving up now. I am defeated.

Perhaps relevant: Colocated function deployments · Issue #11 · netlify/zip-it-and-ship-it · GitHub

jared.krinke · December 29, 2019, 5:27am

Edit: Removed a less elegant solution (really, a hack) I was using.

Jeff’s working example code is here: GitHub - JeffML/firebase-lambda2: Demonstration of hooking up React-Lambda-GraphQL-Firebase on a Netlify server

Mostly unrelated side note: “netlify deploy” seems to want to upload a static site along with a function (i.e. I couldn’t find a way to have a “functions/API only” site). Definitely do not just let it upload your current directory as the static site (because if you do, your credentials and source code will be visible to all!).

JeffML · December 29, 2019, 5:50am

I also struggled with this (especially since someone else made it seem like placing the .json file next to the function code would work).

But it does work. Full article explaining how it works here.

jared.krinke · December 29, 2019, 4:04pm

Ha, I didn’t notice that you were the author of that article. You are right, after (correctly) following your example, I was able to get a JSON file to upload along with my JavaScript code.

Putting the JSON file next to my JavaScript source file wasn’t sufficient; I had to also load the file from my function’s JavaScript source using “require”. My mistake yesterday was that I was using “fs.readFileSync” instead of “require” to read the JSON file. The Netlify packaging tool seems to only pull in files that are loaded via “require”.

I’ll delete most of my original post to avoid confusion.