Setting Netlify Blobs results in empty storage

Support
1

I’m utilising Netlify’s Blob storage for a project and developing via netlify dev locally.

I’m finding that when setting data against a key, that the data results in being blank. No errors are thrown, its a simple text change, and the text isn’t long at all - no special characters or anything. Since the set function itself doesn’t return anything, I cant tell if there’s something off in the function itself or not.

I find that it happens more when a key already exists and I’m attempting to make an update, however, I have experienced it happen on creation too, where it just creates an empty blob.

I’ve checked the data being passed in prior to running the set function, which is present and as expected.

The get and delete functions seem to work everytime, just seems to be the set.

Heres the relative code blocks used:

const reqBody = await req.text()

const callback = (status, response) => {
    return new Response(JSON.stringify(response), { status });
}

const updateClient = async (client, body) => {
        const store = getStore("clients");
        const exists = await store.get(client);
        console.log(exists);
        if (!exists) {
            throw "Client doesn't exist";
        }
        console.log(body);
        await store.set(client, body);
        return {success: true};
}

if (subject === 'clients' && action === 'update' && client !== 'null' && getMethod() === 'PUT') {
        report();
        try {
            console.log(reqBody);
            const result = await updateClient(client, reqBody);
            console.log(result);
            return callback(200, result);
        } catch (e) {
            return callback(500, e);
        }
}

An example key would be clientname and body is typically a JSON string like {"key":"clientname", "name":"Client Name"} and I’m just changing the client name and it results in the blob being blank.

Is this a known issue at all?

2

I haven’t experienced this myself. But, for JSON, I use the .setJSON method. Would be happy to check if you can provide a repo to reproduce the issue.

3

Unfortunately, providing a repo of this will be quite difficult as it requires API connections to a number of internal systems to function correctly. The app without these will just not run sadly.

I yield the same experience whether I use .set() or .setJSON() as well.

In light of the fact that its “random” (or so it seems), I have created a “success confirmation” flow that if found to have failed, it instead retires the set function rather than just blindly returning that it was successful.

Heres what I’ve implemented on both my create and update functions:

const reqBody = req.body ? await req.json() : await req.text();

const callback = (status, response) => {
   return new Response(JSON.stringify(response), { status });
}

const storeClient = async (client, body, attempt = 0) => {
        const store = getStore("clients");
        console.log('body:');
        console.log(body);
        await store.setJSON(client, body);
        const confirmSuccess = await store.get(client);
        console.log('confirmSuccess:');
        console.log(confirmSuccess);
        if (confirmSuccess === JSON.stringify(body)) {
            return {success: true};
        } else if (attempt >= 5) {
            return {fail: true};
        } else {
            console.log('attempt:');
            console.log(attempt);
            return await storeClient(client, body, attempt + 1);
        }
}

const createClient = async (client, body) => {
        const store = getStore("clients");
        const exists = await store.get(client);
        console.log('exists:');
        console.log(exists);
        if (exists) {
            throw "Client already exists";
        }

        return await storeClient(client, body);
}
if (subject === 'clients' && action === 'create' && client !== 'null' && getMethod() === 'POST') {
        report();
        try {
            console.log('reqBody:');
            console.log(reqBody);
            const result = await createClient(client, reqBody);
            console.log('result:');
            console.log(result);
            return callback(200, result);
        } catch (e) {
            return callback(500, e);
        }
}

const updateClient = async (client, body) => {
        const store = getStore("clients");
        const exists = await store.get(client);
        console.log('exists:');
        console.log(exists);
        if (!exists) {
            throw "Client doesn't exist";
        }
        
        return await storeClient(client, body);
}
if (subject === 'clients' && action === 'update' && client !== 'null' && getMethod() === 'PUT') {
        report();
        try {
            console.log('reqBody:');
            console.log(reqBody);
            const result = await updateClient(client, reqBody);
            console.log('result:');
            console.log(result);
            return callback(200, result);
        } catch (e) {
            return callback(500, e);
        }
}

And the result of running the create flow:

Request from ::1: POST /datastore/clients/clientname/create/

-----------------------
----- New Request -----
-----------------------
--------
- POST -
- /datastore/clients/clientname/create -
--------
reqBody:
{ key: 'clientname', name: 'Client Name' }
exists:
null
body:
{ key: 'clientname', name: 'Client Name' }
confirmSuccess:

attempt:
0
body:
{ key: 'clientname', name: 'Client Name' }
confirmSuccess:

attempt:
1
body:
{ key: 'clientname', name: 'Client Name' }
confirmSuccess:

attempt:
2
body:
{ key: 'clientname', name: 'Client Name' }
confirmSuccess:

attempt:
3
body:
{ key: 'clientname', name: 'Client Name' }
confirmSuccess:

attempt:
4
body:
{ key: 'clientname', name: 'Client Name' }
confirmSuccess:
{"key":"clientname","name":"Client Name"}
result:
{ success: true }
Response with status 200 in 395 ms.

As you can see by the code, this is simply looping the same functionality and yielding different results with the same data and code - luckily in this case, on the 5th and final attempt, it was successful.

I think the set/setJSON function needs to have some sort of feedback to confirm the storage of the data was successful. This looping function does that job but feels very inefficient.

I’ll keep an eye on the amount of attempts required and report back on any changes, but something feels fundamentally wrong here.

4

Hey Ash,

I wanted to circle back here and see if you’re still seeing this behavior. Is the above workaround you’ve implemented still feasible, or are you still seeing roughly the same amount of attempts before successfully setting the data? I can confirm that this is not a known issue and not one that we’ve been able to replicate in further testing.

I definitely understand how it would be difficult to stand up a test repo, but any additional context you can provide to help replicate this on our end will help determine what might be going on here. Thanks again!

5

Hey @marcus.little

Yes, I can confirm i’m still experiencing this issue. I’ve actually had to bump the reattempts up to 10 as I did have a case where the 5 reattempts still failed randomly. Since implementing the workaround, and having the reattempts increased to 10, I have not had the issue since. It also been helpful to get the feedback in the response that the call was successful (by having my process recheck whats stored matches what was attempted and returning a success result).

If time permits, I’ll try and create a demo repo with aspects of the build that pertain to the blobs to help replicate the experience. I wish I could give you patterns I’ve found or reasons that result in this occurrence, however as mentioned (and proven with the reattempt workaround), there doesn’t seem to be any rhyme or reason to it - the data is the same, the connection is the same, yet I’ve seen first handle numerous times where the file is simply “blank” after a set call. Again - this is locally, so I’m able to see the values in the files as they happen. I have not yet attempted this being hosted on netlify as yet - it would be a good test to drop the reattempts to zero and see if the same happens when running on Netlify servers or not, therefore it being a local issue only.

6

That’s interesting. In production, I’d be a bit surprised if you’d be able to call .get() immediately after .set() since the data is being stored in an eventually consistent mode. If anything, I’d have imagined local to be able to support these fast read/writes as it’s writing to your local filesystem.

Depends how far you wish to experiment with this, but maybe you can use something like npmjs.com/package/chokidar and setup a watcher on the local blobs directory to see what happens (if anything), in terms of changes to the files in there.