{"id":657,"date":"2022-08-14T09:25:16","date_gmt":"2022-08-14T09:25:16","guid":{"rendered":"https:\/\/tbekk.com\/devstream\/?p=657"},"modified":"2022-09-11T11:14:30","modified_gmt":"2022-09-11T11:14:30","slug":"eda-is-more-of-an-art-than-science","status":"publish","type":"post","link":"https:\/\/tbekk.com\/devstream\/2022\/08\/14\/eda-is-more-of-an-art-than-science\/","title":{"rendered":"EDA Is More Of An Art Than Science"},"content":{"rendered":"\n

Know your data first \u2014 Its super important to build models<\/p>\n\n\n\n

Exploratory Data Analysis (EDA) is the one the significant process (if not the most significant) in building a machine learning model for a specific problem because without knowing your data one can hardly make a useful model.<\/p>\n\n\n\n

\u201cAll models are wrong but some models<\/strong> are<\/strong> useful<\/strong>.\u201d \u2014 George E.P Box<\/p><\/blockquote>\n\n\n\n

EDA enable us to understand the distributions, anomalies and relations between features in the data unfortunately there is no standard procedure or process in place for EDA as it is currently more of an art than science which is why many beginners struggle with EDA and often clueless about how to go about it. Due to absence of any structure some people make a hasty move by applying machine learning algorithms which often produce a sub-optimal model which might work well on specific sample but struggle in production.<\/p>\n\n\n\n

EDA is still more of an art and once you in a middle of the analysis things often get tangled if you are going in without a plan so in this blog I will try jot down the heuristic way of how many data scientist approach EDA in my data science projects.<\/p>\n\n\n\n

The Bird\u2019s Eye View:<\/h1>\n\n\n\n

Before jumping right into complex multivariate analysis and finding correlations within data it is a common practice to have a basic understanding of the data such as fields, data types, possible data issues and distributions which will enable us to ask the right questions in in-depth analysis.<\/p>\n\n\n\n

Basic Info:<\/strong><\/p><\/blockquote>\n\n\n\n

Knowing very basic details such as number of rows, columns, how many fields are having null or missing values can come in handy it is seems pretty basic right? but it will be building block to do more advanced exploration.<\/p>\n\n\n\n

# load data into dataframe #
df = pd.read_csv('titles.csv')

# check few rows #
df.head(5)

# shape of data # 
df.shape

# basic info # 
df.info()<\/em><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Five Number Summary:<\/strong><\/p><\/blockquote>\n\n\n\n
Going beyond basic details five number summary can help you pin-point the key statistics about your data such as mean, median, mode, quartiles, standard deviation, minimum & maximum. These statistics can help figure out if there is anything you need to look into more detail such as a feature having a very high standard deviation or its mean and median are depicting two different center point of distribution (in case of skewness in the data)<\/p>\n\n\n\n
# 01. Five Number Summary for Numeric Features # \ndf.describe()\n\n\n# 02. Five Number Summary for Categorical Features #\ndf.describe(include='O')<\/pre>\n\n\n\n
Pairwise Analysis:<\/strong><\/p><\/blockquote>\n\n\n\n
In order to get a quick glimpse of relation between all variables pair plot is a fairly quick and easy way to plot all possible combinations for us. These charts allow us to know unusual pattern (if exist) and more importantly it is where we start to build many hypothesis.<\/p>\n\n\n\n
# 01. Relevant Imports #\nimport matplotlib.pyplot \nimport seaborn as sns\n\n# 02. Pairplot # \nsns.pairplot(df)<\/pre>\n\n\n\n
\"\"\/
Seaborn Pairplot<\/a><\/figcaption><\/figure>\n\n\n\n
Distributions:<\/strong><\/p><\/blockquote>\n\n\n\n
Knowing your data distributions can come in handy while understanding the nature of the data & more importantly when applying statistical methods which have their own set of assumptions and limitation it is a good idea to validate the specific distribution to build a more reliable model.<\/p>\n\n\n\n
Though five number summary also provide key statistics about data which provide understanding about distribution but in general it is good idea to plot histogram and density plot to understand the shape, outliers and skewness in the data.<\/p>\n\n\n\n
# 01. Imports # \nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# 02. Load Data # \npenguins = sns.load_dataset(\"penguins\")\n\n# 02. Histogram # \nsns.displot(penguins, x=\"flipper_length_mm\")\nplt.savefig('Histogram.png')\n\n# 03. Histogram with hue and step # \nsns.displot(penguins, x=\"flipper_length_mm\", hue=\"species\", element=\"step\")\nplt.savefig('Histogram_with_hue_step.png')\n\n# 04. Histogram with hue and stack # \nsns.displot(penguins, x=\"flipper_length_mm\", hue=\"species\", multiple=\"stack\")\nplt.savefig('Histogram_with_hue_stack.png')\n\n# 05. Histogram with hue and stack # \nsns.displot(penguins, x=\"flipper_length_mm\", col=\"sex\")\nplt.savefig('Histogram_with_col.png')\n\n\n# 06. Density Plot # \nsns.displot(penguins, x=\"flipper_length_mm\", kind=\"kde\")\nplt.savefig('density.png')<\/pre>\n\n\n\n
<\/td><\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
In general EDA along the way will also provide you with insights about where data cleaning and processing is required because real-world dataset are often not in most desired form.<\/p>\n\n\n\n
Deep Dive & build hypothesis:<\/h1>\n\n\n\n
Once we have basic understanding of the data it is time to dive into more complex analysis by asking questions and building hypothesis along the way.<\/p>\n\n\n\n
Univariate, Bivariate & Multivariate Analysis:<\/strong><\/p><\/blockquote>\n\n\n\n
It is always a good idea to have little information about the data to begin with with questioning the data such as can gender in titanic tragedy impact survival chances? Are discounts driving people to buy coffee at starbucks more than thrice in a week?<\/p>\n\n\n\n
# 01. Imports #\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# 02. Load Data # \ntips = sns.load_dataset(\"tips\")\n\n# 03. Facets - Example 01 # \n\ng = sns.FacetGrid(tips, col=\"time\", hue=\"sex\")\ng.map_dataframe(sns.scatterplot, x=\"total_bill\", y=\"tip\")\ng.add_legend()\ng.savefig(\"facet_plot_example_1.png\")\n\n# 03. Facets - Example 02 # \ng = sns.FacetGrid(tips, col=\"sex\", row=\"time\", margin_titles=True)\ng.map_dataframe(sns.scatterplot, x=\"total_bill\", y=\"tip\")\ng.set_axis_labels(\"Total bill ($)\", \"Tip ($)\")\ng.set_titles(col_template=\"{col_name} patrons\", row_template=\"{row_name}\")\ng.set(xlim=(0, 60), ylim=(0, 12), xticks=[10, 30, 50], yticks=[2, 6, 10])\ng.tight_layout()\ng.savefig(\"facet_plot_example_2.png\")\n\n# 04. Joint Plot # \nsns.jointplot(data=penguins, x=\"bill_length_mm\", y=\"bill_depth_mm\", hue=\"species\")\nplt.savefig('Jointplot.png')\n\n# 05. Linear regression plot # \ng = sns.lmplot(x=\"total_bill\", y=\"tip\", hue=\"smoker\", data=tips, markers=[\"o\", \"x\"])\nplt.savefig('lmplot.png')\n\n# 06. Scatterplot # \nsns.scatterplot(data=tips, x=\"total_bill\", y=\"tip\", hue=\"size\")\nplt.savefig('scatterplot.png')\n\n# 07. Barplot or category plot # \ng = sns.catplot(x=\"sex\", y=\"total_bill\",\n                hue=\"smoker\", col=\"time\",\n                data=tips, kind=\"bar\",\n                height=4, aspect=.7)\n\nplt.savefig('barplot.png')\n\n# 08 Heatmap # \n# Load the example flights dataset and convert to long-form\nflights_long = sns.load_dataset(\"flights\")\nflights = flights_long.pivot(\"month\", \"year\", \"passengers\")\n\n# Draw a heatmap with the numeric values in each cell\nf, ax = plt.subplots(figsize=(9, 6))\nsns.heatmap(flights, annot=True, fmt=\"d\", linewidths=.5, ax=ax)\nplt.savefig('heatmap.png')<\/pre>\n\n\n\n
<\/td><\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Let data visualizations answer the questions about the data and while doing so there will be more questions that will be emerge in the process and often getting more complex hence towards involving more than two variables (multivariate analysis)<\/p>\n\n\n\n
For more details about multivariate analysis you can read my blog specifically on this topic here<\/a> and similarly for univariate analysis you can refer this<\/a>.<\/p>\n\n\n\n
Hypothesis building:<\/strong><\/p><\/blockquote>\n\n\n\n
Hypothesis in simple terms \u201cassumptions about the data<\/strong>\u201d. These assumptions will be develop during prior steps. Jot down all the hypothesis and try to validate it by data visualizations in the first place and in the second phase there are more accurate statistical methods for hypothesis testing.<\/p>\n\n\n\n
\"\"
Survival Count wrt Gender in Titanic Dataset<\/figcaption><\/figure>\n\n\n\n