File system issue for api / trying to fs.readFileSync dynamically

[JDNFT]

I’m having an issue in a function with fs (file system) in my node project. In this support thread (List files in another directory from lambda function - #2 by coelmay), it seems that maybe the person had a similar issue, but I’m struggling to land on a workable solution. Here’s what’s going on…

So, I have an API function that takes a param at the end like this:

https://[myapp].netlify.app/.netlify/functions/api/[id]

It then builds a file path like this …
const filePath = ‘…/mydirectory/’ + id;

… and then I need it to go grab some static JSON from a file at that location …
const data = fs.readFileSync(filePath, ‘utf8’);

… and then it would return the data …
return { statusCode: 200, body: data };

If I hard-code the file path, it’ll go find that file. But if I need to dynamically do it as shown here, it will always give me a “file not found” error.

The thread linked to above suggests compiling all of the data into one file and then requiring that. However, I have 10,000 different JSON files in that directory. So, I don’t think it’s practical to include 50mb of data just so the program can access one small chunk.

I know tha these files exist, as I can see them uploaded in the deploy logs. I guess, within a function, though, they seem to need to be required. Thus, I’m stuck. Any thoughts on how to proceed? Ironicaly, I was thinking, “well, I could put that JSON data on an API and fetch it easily” but … THIS is my API, lol, so it s/b doable right from here somehow. Anyway, I’m kinda new to node, so maybe I’m just missing something). But, what should I do if I need my API to pull from 10,000 JSON files like this? Whole different approach? Or, is it possible for this function to work?

Many thanks,
-Jim

[nathanmartin]

@JDNFT I presume that the ... is a typo in this instance, since you used it frequently in your post:

@JDNFT

const filePath = ‘…/mydirectory/’ + id;

A community member is more likely to be able to propose a solution if you show the real code, not placeholders.

[JDNFT]

Correct. Without the typo, it errors.

[nathanmartin]

Cool story I suppose.

Not much more that I can add if you’re uninterested in sharing more.

Best of luck.

[dig]

@JDNFT

Whole different approach?

Yes. It’s called a database. Or NoSQL if you prefer.

[JDNFT]

I’m sorry, I was responding from my phone and did not see your request for the full code. Of course I’m interested in sharing more! I’ll respond again soon when I’m back on my regular machine.

[JDNFT]

Yes that’s the direction I’ve moved in (firebase). However I’m still curious if it can be done via the file system.

[dig]

@JDNFT

However I’m still curious if it can be done via the file system.

Curiosity killed the cat. Nothing brought it back.

Just because you can (potentially) do something doesn’t mean you should.

[JDNFT]

Ha! True, of course. But this script has existed for years as a PHP-based API and handles all requests like that just fine. Messing with a server file-system is of course probably more suited for PHP. Anyway, I’m far enough along with Firebase now that I guess it doesn’t matter terribly if this is even possible.

[JDNFT]

I wasn’t aware that people preferred the entire code. While I’ve moved on to exploring a database solution to this, the original code was more or less like this:

// This code is a serverless function, hosted at Netlify.
// It lives as a file called api.mjs inside the "functions" directory of my site.
// Such a function would show at a URL like: https://[MYAPP].netlify.app/.netlify/functions/[functionName]
// so, https://[MYAPP].netlify.app/.netlify/functions/api would be the main access point, and...
// https://[MYAPP].netlify.app/.netlify/functions/api/5  would be a sample request.

exports.handler = async (event, context) => {

    const fs = require('fs');
    const path = require('path');
    const https = require('https');
  
    const CONTRACT_ADDRESS = '[ETHEREUM SMART CONTRACT ADDRESS]';
    const ETHERSCAN_API_KEY = '[MY ETHERSCAN API KEY]';
  
    // This just grabs some data from Etherscan -- works fine.
    async function fetchEtherscanAPI() {
        return new Promise((resolve, reject) => {
            const url = `https://api.etherscan.io/api?module=stats&action=tokensupply&contractaddress=${CONTRACT_ADDRESS}&apikey=${ETHERSCAN_API_KEY}`;
            https.get(url, (res) => {
                let data = '';
                res.on('data', (chunk) => {
                    data += chunk;
                });
                res.on('end', () => {
                    try {
                        const jsonData = JSON.parse(data);
                        resolve(jsonData);
                    } catch (error) {
                        reject(error);
                    }
                });
            }).on('error', (error) => {
                reject(error);
            });
        });
    }
  
    try {
        // Extract the token ID from the request URL
        const id = event.path.split('/').pop();
  
        // Check if the id is a number
        if (!/^\d+$/.test(id)) {
            return {
                statusCode: 400,
                body: JSON.stringify({ error: 'Invalid token ID' })
            };
        }
  
        // Query the Etherscan API to get the token supply
        const response = await fetchEtherscanAPI();
        const tokenSupply = parseInt(response.result);
  
        // Check if the token ID exists
        if (tokenSupply >= id) {
            
            // THIS is the part that does not work...
            // Construct the file path:
            const filePath = '../tokendata/' + id;
            // Potentially, the issue is this:
            // issue: https://answers.netlify.com/t/list-files-in-another-directory-from-lambda-function/39183/2
  
            // Check if the file exists
            // This will always error, as coded here.
            // It seems to want to point to I believe something like "/var/task/[filepath]"
            // ... which seems like I could fix the path to fix this, however that does not work
            // I think it does not work because the file is not specifically *required*.
            if (!fs.existsSync(filePath)) {

                return {
                    statusCode: 404,
                    body: JSON.stringify({ error: 'File not found', filePath })
                };

            } else {

              // Send JSON data 
              const data = fs.readFileSync(filePath, 'utf8');
              return {
                statusCode: 200,
                body: data
              };

            }
  
        } else {
  
            return {
                statusCode: 404,
                body: JSON.stringify({ error: 'Token ID does not exist.' })
            };
        }
  
    } catch (error) {
  
        console.error('Error querying Etherscan API:', error);
  
        return {
            statusCode: 500,
            body: JSON.stringify({ error: 'Internal server error' })
        };
  
    }
  
  };

One other potential solution (which I did not try, as I didn’t learn about it until after I refactored to use a database instead) was that maybe this could be addressed in the netlify.toml file, in a “[funtions]” section by my specifying to include all files in a certain directory, as in:

[build]
  functions = "functions"
  
[functions]
  included_files = ["tokendata/*"]

Might that work? But of course, here I would have 10,000 files, so I wasn’t sure if that would cause issues. Again, I’ve pretty much given up on using the file system, but I still wanted to post all of this just in case others run into this same issue and perhaps need a bit of help.

[hrishikesh]

The included_files configuration would work as long as the directory structure of the Netlify Functions as the folder you’re including is correct. For example in your code, you’ve set the functions directory as functions so I’m assuming you’re directory structure is:

./
|-- functions/
  |-- someFile.js // here you're accessing  `../tokendata/${id}`
|-- tokendata/
  |-- 12345.json
|-- netlify.toml

For a file structure like this, it should work as you’ve described. But, if you say your files were 50 MB in size, there’s a slight chance you’d run into AWS Lambda limit of 50 MB size for function bundle. Having so many files could also potentially cause Lambda file descriptor issue.

[JDNFT]

That works when I test locally using the netlify cli (via “netlify dev”)… localhost finds the file and returns it as expected.

However, if I deploy that code to Netlify for real, I will get a 404 file not found error, per my code. So, for example, it’ll say “{“error”:“File not found”,“filePath”:”…/tokendata/8"}"

So, the same filepath works locally, but not on Netlify. (My directory structure is exactly as you’ve assumed.)

[hrishikesh]

Won’t be possible to guess much without seeing a repo.

[JDNFT]

I’d be happy to make a public one, if you’d like to have a look… it’s private for now, as I have API keys & stuff, but I can convert to environment vars & share, if you’re willing to have a peek?