Saving data

Assuming the data has been succesfully returned from the API server endpoint, and is now loaded in memory, it can be saved. This functionality is handled within the save_data.py script.

Prerequisites

Given that the main data storage associated with this project is a MariaDB database, the connection to this database must be established first. This is done through the sqlalchemy package’s create_engine function, which is further described in the database section.

Save the data using a single function call

Once the database engine is available, it can be used in the pandas DataFrame object’s method to_sql under the con argument to directly inject the data into the MariaDB database. For the actual implementation, see save_data.py::save_df_to_db method.

What to do when the database connection fails

If the database connection fails, there is a backup mechanism that automatically saves the data to a local .csv file instead. For this, a BACKUP_FOLDER_PATH static variable is used to specify the path under which the file should be created.

Note

Although the .csv files are a reliable way to store data locally, their contents will not get displayed in the application. Periodically check the backup path for these backup files, and call the save_df_to_db function on these files to save them into the database. Given the simple database structure, this process should be quite reliable and error-proof.