database - What is the best way to handle data structure for firebase if you need to be able to search by multiple fields

Question

I am trying to create a funko pop database with over 12k funko pops that i have scraped from websites.

How I scraped them from the websites is based on series. So one series would look like this

{
funkoData :
(array)
  0 image : "https://cconnect.s3.amazonaws.com/wp-content/uploads/2014/09/2014- 
           Funko-Pop-Marvel-Thor-Series-2-36-Loki-Helmet.jpg"
    name : "Loki - with helmet"
    number : "36"

  1 image : "https://cconnect.s3.amazonaws.com/wp-content/uploads/2014/10/2014- 
             Funko-Pop-Marvel-Thor-Series-2-36-Loki-Helmet-Black-and-White-Hot- 
             Topic.jpg"
    name : "Black and White Loki with Helmet - Hot Topic Exclusive"
    number : "36"

   2 image : "https://cconnect.s3.amazonaws.com/wp-content/uploads/2014/09/2014- 
             Funko-Pop-Marvel-Thor-Series-2-54-Odin.jpg"
      name : "Odin"
      number : "54"

   3 image : "https://cconnect.s3.amazonaws.com/wp-content/uploads/2014/09/2014- 
             Funko-Pop-Marvel-Thor-Series-2-55-Heimdall.jpg"
      name : "Heimdall"
      number : "55"

   4 image : "https://cconnect.s3.amazonaws.com/wp-content/uploads/2014/09/2014- 
             Funko-Pop-Marvel-Thor-Series-2-56-Lady-Sif.jpg"
      name : "Lady Sif"
      number : "56"
(end array)

genre : "2014 Funko Pop Marvel Thor Series 2 Vinyl Figures"
}

right now i have a collection of about 750 'genres' that all look like this. So a document has fields : funkoData that is an array of objects of the actual funko pops, and field : genre that matches the series of funko pops for that document.

How i am currently searching the firestore db is with this function below, as firestore can only do exact matches and in this case where funko pop names can include thor, thor with stormbreak, thor with hammer, glow in the dark thor, doing a query search if someone searches for thor, would yield no results unless there was an exact match which is obviously not what I want. Also the search needs to search through genre as well as name as there could be a genre that someone is searching for to get back a whole list of funko pops related that as well as having names included for the funko pops as well. example would be like searching thor in the data above would give back the document matching genre: "2014 Funko Pop Marvel Thor Series 2 Vinyl Figures" as thor is in the string, and any funko pops that have thor in the field name as well.

const getFunkoPopQuery = async (req, res, next) => {
  try {
    console.log(req.params);
    const query = req.params.query.trim().toLowerCase();
    const funkoPops = await firestore.collection("funkoPops");
    const data = await funkoPops.get();
    const funkoArr = [];
    if (data.empty) {
      res.status(200).data([]);
    } else {
      data.forEach((doc) => {
        const funkoObj = new FunkoPop(doc.data().genre, doc.data().funkoData);
        funkoArr.push(funkoObj);
      });

      // genre matching if query is not a number
      let genreMatches = [];
      if (isNaN(query)) {
        genreMatches = funkoArr.filter((funko) =>
          funko.genre.toLowerCase().includes(query)
        );
      }

      // name & number matching
      const objToSearch = {
        notNullNameArr: [],
        notNullNumbArr: [],
        nameMatches: [],
        numbMatches: [],
      };

      funkoArr.forEach((funko) => {
        const genre = funko.genre;
        if (funko.funkoData) {
          const funkoDataArr = funko.funkoData;
          funkoDataArr.forEach((data) => {
            if (data.name) {
              objToSearch.notNullNameArr.push({
                funkoData: [data],
                genre: genre,
              });
            }
            if (data.number) {
              objToSearch.notNullNumbArr.push({
                funkoData: [data],
                genre: genre,
              });
            }
          });
        }
      });
      // find name that includes query
      objToSearch.notNullNameArr.forEach((funko) => {
        const genre = funko.genre;
        const name = funko.funkoData.filter((data) =>
          data.name.toLowerCase().includes(query)
        );
        if (Object.keys(name).length > 0) {
          objToSearch.nameMatches.push({
            genre,
            funkoData: name,
          });
        }
      });
      // find number that matches query
      objToSearch.notNullNumbArr.forEach((funko) => {
        const genre = funko.genre;
        const number = funko.funkoData.filter((data) => data.number === query);
        if (Object.keys(number).length > 0) {
          objToSearch.numbMatches.push({
            genre,
            funkoData: number,
          });
        }
      });

      const searchFinds = {
        genre: genreMatches,
        name: objToSearch.nameMatches,
        number: objToSearch.numbMatches,
      };

      res.status(200).send(searchFinds);
    }
  } catch (error) {
    res.status(400).send(error.message);
  }
};

this functions allows for a text search that matches the fields: genre, name & number (which are both nested in the array funkoData), except every time you search we have to get all the documents in firestore which we obviously blow through the 50k read quota since each time you search for a funko pop, it would be 750 reads.

Anyone with more knowledge on how to set up a data structure, or who knows how to work with firestore in this type of limitation have any advice, or changes to either the data or the query that could help would be greatly appreciated!

Thank you!

question from:https://stackoverflow.com/questions/65848409/what-is-the-best-way-to-handle-data-structure-for-firebase-if-you-need-to-be-abl

Answer 1

Waitting for answers