python - Where to store a large pandas dataframe (consumed by a webapp)?

Question

This question already has an answer here: question from:https://stackoverflow.com/questions/65647898/where-to-store-a-large-pandas-dataframe-consumed-by-a-webapp

Answer 1

Where did the data come from to populate the dataframe?

If it was a database for example, you could just grab the data the user needed at that time rather than putting all your data into memory.

On the other hand, if your data didn't come from a database then that would be my first suggestion. Put the data into a database, these things were built for that purpose. You can create a dataframe from an sql query (pandas.read_sql).

If you really must have 5g of data in memory, then perhaps an in memory database would be suitable. For this purpose Redis comes to mind.

You could either pickle the dataframe and store the whole thing as an object, or you could break it down to records and store the individually and pull down only the data the user requests. This would remove the requirement for your application to store 5g of data in memory all the time.