python - Resetting Pandas dataframe in iterable loop resulting in ValueError

Question

I need help with resetting pandas dataframe within a for loop. Here is the psuedo code of my current logic -

import pandas as pd
df_out = pd.DataFrame(columns = ['col1','col2','col3','col4'])

for filename in os.listdir(directory):

    #some logic that results in dataset stored in a list called - output

    #output stored in dataframe
    df = pd.DataFrame(output, columns = ['col1','col2'])

    #some other logic that is used to get col3 using list called - col3_output
    df.loc[:,'col3'] = col3_output

    #some other logic that is used to get col4 using list called - col4_output
    df.loc[:,'col4'] = col4_output

    #note - col3 and col4 output cannot be derived from existing columns i.e. col1, col2

    #reset the lists to empty for next iteration of file
    col3_output = []
    col4_output = []

    #assign output to df_out
    df_out = df_out.append(df)

    #######################################
    ## ERORR OCCURING HERE
    #######################################
    #resetting dataframe or deleting data
    del df

#write final df_out to file
#some logic

I have tried resetting data frame using df.iloc[0:0] and dropping the new columns that I am creating but to no avail. I get the error -

ValueError: Must have equal len keys and value when setting with an iterable

question from:https://stackoverflow.com/questions/65893322/resetting-pandas-dataframe-in-iterable-loop-resulting-in-valueerror

Answer 1

If i understood correct you just want an empty dataframe? if so df = pd.DataFrame() would just overwrite the old one and make an empty df.