How to Create Nested Dictionaries from a Data Frame in Python Using Pandas

Discover how to easily convert a pandas DataFrame into a list of nested dictionaries, suitable for processing data with APIs. --- This video is based on the question https://stackoverflow.com/q/76763351/ asked by the user 'Empusas' ( https://stackoverflow.com/u/7357166/ ) and on the answer https://stackoverflow.com/a/76763440/ provided by the user 'Quang Hoang' ( https://stackoverflow.com/u/4238408/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions. Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: How to create nested dictionaries from a data frame in Python? Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l... The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license. If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com. --- Transforming a DataFrame into Nested Dictionaries in Python If you're working with data in Python, especially in data science or backend development, you are likely familiar with pandas, a powerful data manipulation library. You might encounter situations where you need to transform a pandas DataFrame into a format more suitable for processing, such as nested dictionaries. In this post, we will explore how to accomplish this gracefully and efficiently. The Problem at Hand Let's consider a scenario where we have the following pandas DataFrame, which contains personal details of individuals along with their addresses: NameLast NameStreetCityPostcodeJohnDoeMilky Way 1Star City31415JimBeamMoonshine Rd 8Sin City12345JoeBiden1600 Pennsylvania Avenue NWWashington, D.C.20500We want to convert this DataFrame into a list of dictionaries in the following nested format to process it with a given API: [[See Video to Reveal this Text or Code Snippet]] The Solution To achieve this transformation elegantly using pandas, follow these steps: Step 1: Set the Index First, you need to set the index of the DataFrame to include 'Name' and 'Last Name'. This makes it easier to aggregate the subsequent columns into a dictionary: [[See Video to Reveal this Text or Code Snippet]] Step 2: Aggregate Address Details Next, use the agg method to aggregate the address details into a dictionary format. Below is the key line of code that performs this operation: [[See Video to Reveal this Text or Code Snippet]] This line takes each row of the DataFrame and converts the specified columns into a dictionary format. Step 3: Convert to DataFrame and Reset Index Now, we need to convert the resulting structure back into a DataFrame and then reset the index: [[See Video to Reveal this Text or Code Snippet]] Step 4: Finally, Convert to List of Dictionaries The last step is to convert the DataFrame to a list of dictionaries. You can do this using the to_dict method with the records parameter: [[See Video to Reveal this Text or Code Snippet]] Putting It All Together Here’s the complete code snippet that transforms your DataFrame into the desired format: [[See Video to Reveal this Text or Code Snippet]] Result After executing the above code, you will get the desired list of nested dictionaries, similar to the initial format we set out to achieve. The result will look like this: [[See Video to Reveal this Text or Code Snippet]] Conclusion Changing a pandas DataFrame into a list of nested dictionaries may seem complex initially, but with the right approach and pandas functions, it can be done in a clean and efficient way. This method avoids the need to iterate over DataFrame rows manually, leveraging pandas capabilities to handle data more elegantly. Feel free to apply this technique to your own DataFrames as needed, and happy coding!