Fetching mongoDB data with lambda functions

Support
,
1

I’m having an issue fetching mongoDB data using lambda functions when not working locally.

I set up a sample mongodb filled with their sample data and simply want to pull it to be used on my website. When I work locally via the netlify-cli I am able to properly query the database and return the value I’m looking for without error. It’s when I push the function up to my site that I begin to run into issues with it timing out.

My code can be seen below (username and pw changed)

/* eslint-disable */
const fetch = require('node-fetch');
const { MongoClient } = require('mongodb');

async function getData() {
  const uri =
    'mongodb+srv://<user>:<password>@cluster0-2pfxk.mongodb.net/test?retryWrites=true&w=majority';
  const client = new MongoClient(uri, {
    useNewUrlParser: true,
    useUnifiedTopology: true
  });

  try {
    await client.connect();
    const test = await client
      .db('sample_airbnb')
      .collection('listingsAndReviews')
      .findOne({ name: 'Apt Linda Vista Lagoa - Rio' });
    return test;
  } catch (err) {
    console.log(err); // output to netlify function log
  } finally {
    await client.close();
  }
}

exports.handler = async function(event, context) {
  try {
    const data = await getData();
    return {
      statusCode: 200,
      body: JSON.stringify({ id: data._id })
    };
  } catch (err) {
    console.log(err); // output to netlify function log
    return {
      statusCode: 500,
      body: JSON.stringify({ msg: err.message }) 
    };
  }
};

As mentioned before, when run locally, this works as expected outputting the desired _id. It’s not until I push it to my project that I run into timeout issues. I made sure that I am closing the client, so I don’t think it’s a matter of me leaving that open, but I don’t know.

The function has the following endpoint: https://mongodb--virtualbackgrounds.netlify.com/.netlify/functions/mongo

Any advice or suggestions would be greatly appreciated!

Thanks.

1 Like
2

It took me awhile, but I finally realized that the issue was due to not whitelisting Netlify on MongoDB. This is why locally testing it was running fine, but then once it was pushed up it would timeout.

I temporarily set mongoDB to allow connections from everywhere, and sure enough it got the data.
Leading to my next issue that perhaps someone else knows something about – Setting up VPC Peering so that the lambda function will always be whitelisted for the DB.

I’m very much a beginner in terms of back-end so this is all pretty new to me. If anyone has any ideas or advice, I’m all ears!

1 Like
3

@braveux Funnily, I was in the same exact situation, struggled all day, and finally figured it out (whitelisting all) literally 10 minutes ago. I, too, would appreciate any community support/advice on this as I don’t want to leave the DB open to all connections.

Is VPC Peering the next thing to learn?

5

Hey @braveux and @lilbigwheel! Linking this conversation which I think may be useful:

TL;DR There’s no way to whitelist an IP range for our functions because there’s no way to know which range they will be spun up on when they’re invoked. Our suggestions in the past have been to whitelist everyone but make sure you have a very strong MongoDB user password (and put your DB_URL in a Netlify environment variable)… or switch to FaunaDB.

Maybe not the answer you wanted to hear, but hopefully it’s helpful- let us know if you find workarounds or if we can answer any other questions on this!

6

For anyone who might come to this post after having problems with the timeout from the Netlify Functions while using MongoDB integration (I’m also using a Free server from MongoDB Atlas):

Create indexes. MongoDB might take some time (more than 10 seconds) to look for a simple string with a collection with 5 documents and an index will fix that.

If you’re using MongoDB Atlas, check this link to learn how to create index using the data explorer: https://docs.atlas.mongodb.com/data-explorer/indexes/

If you’re not using Atlas, you can check this link to create the index manually: https://docs.mongodb.com/manual/tutorial/create-indexes-to-support-queries/

1 Like
7

It also takes time because it needs to establish a connection.

Reference https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda/

Are there any good practices to follow with Netlify?

8

Nothing specific, but the same as the best practices for any function. I guess we can summarize them as:

  1. make sure you are monitoring the runtime of your function and appropriately cleaning up and bailing by the 9s mark (that’ll give your cleanup code a second to run before we cut you off at 10 seconds, ready or not!)
  2. whatever calls the function should be checking its return code, and you should be setting a good return code particularly for cases like the above where the data save does not work, so that your visitors get a good experience (e.g. “looks like we had trouble saving, try again by clicking this button or contact tech support if this error message persists”)
  3. while your function can connect in any fashion, it will do so over the public internet, so you’ll want to make sure the connection is encrypted to prevent attacks.