External API call from Next.js API function failing intermittently, callback never running

Support
,
1

I am using a Next.js API route (i.e. server-side function) to upload a file to Cloudinary. This works fine locally, but always fails the first time on Netlify (and sometimes fails a few times before ultimately succeeding).

Here is the page where you can see the behavior. Feel free to create some albums by uploading images, selecting a cover image, and clicking the “Create Info.json” button. Unfortunately, there’s no way for you to know if the info.plist has successfully been created; I can check on Cloudinary myself, but to see the new album reflected in the site, I would need to rebuild the site since it uses SSG. What you would presumably be able to do is see the function log (see below), which is not behaving as expected.

Here is the code which is running in the API route:

import { v2 as cloudinary } from 'cloudinary'
import { Readable } from 'stream';

export default async function uploadInfoJSON(req, res) {
    console.log("enter upload info.json")
    if (req.method !== 'POST') {
        res.status(405).send({ message: 'Only POST requests allowed' })
        return
    }

    cloudinary.config({
        cloud_name: process.env.NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME,
        api_key: process.env.CLOUDINARY_KEY,
        api_secret: process.env.CLOUDINARY_SECRET,
        secure: true
    })

    const { slug } = req.query;
    var upload_stream = cloudinary.uploader.upload_stream(
        {
            public_id: 'info.json',
            folder: `albums/${slug}`, // Passed into the API route as [slug].js
            resource_type: 'raw'
        },
        (error, result) => {
            if (error) {
                console.log("ERROR:", error, result);
            } else {
                console.log("SUCCESS!")
            }
        }
    )
    
    var jsonString = JSON.stringify(req.body, null, 2)
    console.log("JSON=", jsonString)
    Readable.from(jsonString).pipe(upload_stream)

    res.status(200).end()
};

I have a button which the user clicks on a web page, and then a POST request is sent to this route with the JSON as the body. This always works first time locally. I click the button, the file is uploaded, and I see “SUCCESS” in the server-side console logging in my terminal.

On Netlify, however, I see this in the “Function Next.js SSR handler” logs:

Sep 5, 09:51:29 PM: INIT_START Runtime Version: nodejs:18.v11 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:2d38a43033930b7e89d977b60a6556a2e84bcc276ff59eab90d1327b252528ef
Sep 5, 09:51:30 PM: 64f72c4c INFO enter upload info.json
Sep 5, 09:51:30 PM: 64f72c4c INFO JSON= {
"display_name": "Test 3",
"description": "",
"cover_photo": "photo3",
"captions": {
"photo1": "caption 1",
"photo2": "caption 2",
"photo3": "caption 3"
},
"is_public_album": true
}
Sep 5, 09:51:30 PM: 64f72c4c INFO [POST] /api/9v70CS/upload/test3 (SSR)
Sep 5, 09:51:30 PM: 64f72c4c Duration: 403.09 ms Memory Usage: 111 MB Init Duration: 532.42 ms

So the beginning and end of the function are reached. However, I see neither “ERROR” nor “SUCCESS” in the log, so the callback seems not to be running. More importantly, the file is almost never created the first time I click the button. (Meaning there seems to be some trouble with the outbound call as well.) When I click the upload button a second time (or sometimes it requires a third or fourth click), the file IS ultimately created. (So the outbound call succeeds, but the callback is still never being run.)

This behavior sounds similar to what folks described in this Netlify Forum post from a couple of years back, but I don’t see a solution there. And the intermittent nature of my issue is perplexing.

I found this thread talking about Next.js functions failing on Netlify due to timeouts from excessive imports. So I made sure I was importing sparsely. Is 403+532ms OK for a cold start? It’s way less than the 10s Netlify Function timeout. Anyway, subsequent requests are much faster (e.g. 75ms), but seem to also never run the upload callback. So while it’s always great for the function to run faster, that solution doesn’t seem to have been the fix for my issue.

Is there something special I need to do to ensure that my Functions can send external requests reliably and run callbacks? (I’d expect the Function to wait until the callback was run before returning.)

2

I’ve worked out the problem. I didn’t realize that all aspects of streams are asynchronous, even when they’re chained together. So my code was behaving like this:

  1. upload_stream() returned the writeable stream immediately.
  2. The pipe was invoked asynchronously.
  3. If it took a little too long, the Readable stream didn’t get time to fully pass its contents to the upload_stream before the function ended, so the file wasn’t uploaded.
  4. In either case, the function was over by the time the cloudinary result callback was due to be called.

I worked this out after reading this GitHub issue thread. The solution was to wrap the upload in a Promise and use await to guarantee that the upload is complete before the API function ends.

import { v2 as cloudinary } from 'cloudinary'

// Based on https://github.com/cloudinary/cloudinary_npm/issues/130#issuecomment-865314280
function uploadStringAsync(str, album_name) {
    return new Promise((resolve, reject) => {
        cloudinary.uploader.upload_stream(
            {
                public_id: 'info.json',
                resource_type: 'raw',
                folder: `albums/${album_name}`
            },

            function onEnd(error, result) {
                if (error) {
                    console.log("ERROR:", error, result);
                    return reject(error)
                }

                console.log("Uploaded info.json:", str)
                resolve(result)
            }
        )
            .end(str)
    })
}

export default async function uploadInfoJSON(req, res) {
    if (req.method !== 'POST') {
        res.status(405).send({ message: 'Only POST requests allowed' })
        return
    }

    cloudinary.config({
        cloud_name: process.env.NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME,
        api_key: process.env.CLOUDINARY_KEY,
        api_secret: process.env.CLOUDINARY_SECRET,
        secure: true
    })

    const { slug } = req.query;
    const jsonString = JSON.stringify(req.body, null, 2)

    await uploadStringAsync(jsonString, slug)

    res.status(200).end()
};
3

thanks for sharing your solution!