pandas-data-analysis

name Pandas Data Analysis description Master data manipulation, analysis, and visualization with Pandas, NumPy, and Matplotlib version 2.1.0 sasmp_version 1.3.0 bonded_agent 03-data-science bond_type PRIMARY_BOND retry_strategy exponential_backoff observability {"logging":true,"metrics":"data_processing_time"} Pandas Data Analysis Overview Master data analysis with Pandas, the powerful Python library for data manipulation and analysis. Learn to clean, transform, analyze, and visualize data effectively. Learning Objectives Load and manipulate data from various sources (CSV, Excel, SQL, APIs) Clean and transform messy datasets Perform exploratory data analysis (EDA) Aggregate and group data for insights Create compelling visualizations Optimize performance for large datasets Core Topics 1. Pandas DataFrames & Series Creating DataFrames from various sources Indexing and selecting data (loc, iloc, at, iat) Filtering and boolean indexing Adding/removing columns and rows Data types and conversions Code Example: import pandas as pd import numpy as np # Create DataFrame data = { 'name' : [ 'Alice' , 'Bob' , 'Charlie' , 'David' ], 'age' : [ 25 , 30 , 35 , 28 ], 'salary' : [ 50000 , 60000 , 75000 , 55000 ], 'department' : [ 'IT' , 'HR' , 'IT' , 'Sales' ] } df = pd.DataFrame(data) # Indexing and filtering it_employees = df[df[ 'department' ] == 'IT' ] high_earners = df.loc[df[ 'salary' ] > 55000 , [ 'name' , 'salary' ]] # Adding calculated columns df[ 'annual_bonus' ] = df[ 'salary' ] * 0.10 df[ 'age_group' ] = pd.cut(df[ 'age' ], bins=[ 0 , 30 , 40 , 100 ], labels=[ 'Young' , 'Mid' , 'Senior' ]) print (df) 2. Data Cleaning & Transformation Handling missing data (dropna, fillna, interpolate) Removing duplicates String operations and text cleaning Date/time parsing and manipulation Type conversions and casting Applying custom functions (apply, map, applymap) Code Example: import pandas as pd # Load data with missing values df = pd.read_csv( 'sales_data.csv' ) # Handle missing values df[ 'price' ].fillna(df[ 'price' ].median(), inplace= True ) df[ 'category' ].fillna( 'Unknown' , inplace= True ) df.dropna(subset=[ 'customer_id' ], inplace= True ) # Clean text data df[ 'product_name' ] = df[ 'product_name' ]. str .strip(). str .lower() df[ 'product_name' ] = df[ 'product_name' ]. str .replace( '[^a-zA-Z0-9 ]' , '' , regex= True ) # Convert dates df[ 'order_date' ] = pd.to_datetime(df[ 'order_date' ]) df[ 'year' ] = df[ 'order_date' ].dt.year df[ 'month' ] = df[ 'order_date' ].dt.month # Remove duplicates df.drop_duplicates(subset=[ 'order_id' ], keep= 'first' , inplace= True ) # Apply custom function def categorize_price ( price ): if price < 50 : return 'Low' elif price < 100 : return 'Medium' else : return 'High' df[ 'price_category' ] = df[ 'price' ].apply(categorize_price) 3. Aggregation & Grouping GroupBy operations Aggregation functions (sum, mean, count, etc.) Pivot tables and cross-tabulation Multi-level indexing Window functions (rolling, expanding) Code Example: import pandas as pd # Sample sales data df = pd.read_csv( 'sales.csv' ) # GroupBy aggregation dept_stats = df.groupby( 'department' ).agg({ 'salary' : [ 'mean' , 'min' , 'max' ], 'employee_id' : 'count' }) # Multiple groupby sales_by_region_product = df.groupby([ 'region' , 'product_category' ])[ 'sales' ]. sum () # Pivot table pivot = df.pivot_table( values= 'sales' , index= 'product_category' , columns= 'quarter' , aggfunc= 'sum' , fill_value= 0 ) # Rolling window (moving average) df[ 'sales_ma_7d' ] = df.groupby( 'product_id' )[ 'sales' ].transform( lambda x: x.rolling(window= 7 , min_periods= 1 ).mean() ) # Cumulative sum df[ 'cumulative_sales' ] = df.groupby( 'product_id' )[ 'sales' ].cumsum() 4. Data Visualization Matplotlib basics Seaborn for statistical plots Pandas built-in plotting Customizing plots Creating dashboards Code Example: import pandas as pd import matplotlib.pyplot as plt import seaborn as sns # Set style sns.set_style( 'whitegrid' ) # Load data df = pd.read_csv( 'sales_data.csv' ) # 1. Line plot - Sales trend over time df.groupby( 'month' )[ 'sales' ]. sum ().plot(kind= 'line' , figsize=( 10 , 6 )) plt.title( 'Monthly Sales Trend' ) plt.xlabel( 'Month' ) plt.ylabel( 'Total Sales ($)' ) plt.show() # 2. Bar plot - Sales by category category_sales = df.groupby( 'category' )[ 'sales' ]. sum ().sort_values(ascending= False ) category_sales.plot(kind= 'bar' , figsize=( 10 , 6 )) plt.title( 'Sales by Category' ) plt.xlabel( 'Category' ) plt.ylabel( 'Total Sales ($)' ) plt.xticks(rotation= 45 ) plt.show() # 3. Histogram - Price distribution df[ 'price' ].hist(bins= 30 , figsize=( 10 , 6 )) plt.title( 'Price Distribution' ) plt.xlabel( 'Price ($)' ) plt.ylabel( 'Frequency' ) plt.show() # 4. Box plot - Salary by department df.boxplot(column= 'salary' , by= 'department' , figsize=( 10 , 6 )) plt.title( 'Salary Distribution by Department' ) plt.suptitle( '' ) plt.show() # 5. Heatmap - Correlation matrix corr = df[[ 'age' , 'salary' , 'years_experience' ]].corr() sns.heatmap(corr, annot= True , cmap= 'coolwarm' , center= 0 ) plt.title( 'Correlation Matrix' ) plt.show() Hands-On Practice Project 1: Customer Analytics Analyze customer purchase behavior and segmentation. Requirements: Load customer transaction data Clean and prepare dataset Calculate RFM (Recency, Frequency, Monetary) metrics Customer segmentation Visualize insights Generate executive summary Key Skills: Data cleaning, aggregation, visualization Project 2: Time Series Analysis Analyze sales trends and forecast future performance. Requirements: Load time series data Handle missing dates Calculate moving averages Identify trends and seasonality Detect anomalies Create interactive visualizations Key Skills: Time series operations, rolling windows, plotting Project 3: Data Quality Report Build automated data quality assessment tool. Requirements: Check for missing values Identify duplicates Detect outliers Validate data types Generate quality metrics Export HTML report Key Skills: Data validation, statistical analysis, reporting Assessment Criteria Load and clean real-world datasets efficiently Perform complex data transformations Use GroupBy for aggregations Create insightful visualizations Handle missing and inconsistent data Optimize performance for large datasets Document analysis with clear explanations Resources Official Documentation Pandas Docs - Official documentation NumPy Docs - NumPy documentation Matplotlib Docs - Plotting library Learning Platforms Kaggle - Free Pandas course DataCamp - Interactive courses Python for Data Analysis - Wes McKinney's book Tools Jupyter Notebook - Interactive development Google Colab - Cloud notebooks Anaconda - Data science distribution Next Steps After mastering Pandas, explore: Scikit-learn - Machine learning SQL - Database querying Apache Spark - Big data processing Tableau/Power BI - Business intelligence tools