Applying Descriptive Statistics using Pandas

Statistical analysis is used to analyze the results and deduce or infer meaning about the underlying dataset or the reality that it attempts to describe.

Statistical analysis may be used to:

• Present key findings revealed by a dataset.
• Summarize information.
• Calculate measures of cohesiveness, relevance, or diversity in data.
• Make future predictions based on previously recorded data.
• Test experimental predictions.

Any data scientist will spend much of their day performing these functions.

Aggregation functions

An essential piece of analysis of large data is efficient summarization: computing aggregations like sum(), mean(), median(), min(),and max(), in which a single number gives insight into the nature of a potentially large dataset. In this section, we’ll explore aggregations in Pandas, from simple operations akin to what we’ve seen on NumPy arrays.

.sum() function. It returns a number, the sum of all items in an iterable.

Consider the following DataFrame

df.sum()

Output:

dtype: int64

We can also find the sum of  individual columns in the DataFrame 

df[‘A'].sum()

Output:

229

Now we will learn how to find mean(), median(), min(),and max() values. We already know what mean median is

.mean()function returns the mean of the values for the requested axis. If the method is applied on a pandas series object, then the method returns a scalar value which is the mean value of all the observations in the data frame. If the method is applied on a pandas dataframe object, then the method returns a pandas series object which contains the mean of the values over the specified axis.

df.mean()

Output:

We can also find the mean of  individual columns in the DataFrame 

df['A'].mean()

Output:

76.33333333333333

.median()function return the median of the underlying data in the given DataFrame.

df.median()

Output:

.max() function returns the maximum of the values in the given object. If the input is a series, the method will return a scalar which will be the maximum of the values in the series. If the input is a dataframe, then the method will return a series with maximum of values over the specified axis in the dataframe. By default the axis is the index axis.

df.max()

Output:

Similarly .min()function returns the minimum of the values in the given object. If the input is a series, the method will return a scalar which will be the minimum of the values in the series. If the input is a dataframe, then the method will return a series with minimum of values over the specified axis in the dataframe

df.min()

Output:

.count() is used to count the no. of non-NA/null observations across the given axis. It works with non-floating type data as well.

df.count()

Output:

.std() function return sample standard deviation over requested axis.

df.std()

Output:

If we have some missing values in our DataFrame, we skip those missing values by passing the argument  df.std(skipna=True)

.describe()is used to view some basic statistical details like percentile, mean, std etc. of a data frame or a series of numeric values. When this method is applied to a series of string, it returns a different output which is shown in the examples below.

df.describe()

Output:

.corr() is used to find the pairwise correlation of all columns in the dataframe. Any NaN values are automatically excluded. For any non numeric data type columns in the dataframe it is ignored.

df.corr()

Output:

The output dataframe can be interpreted as for any cell, row variable correlation with the column variable is the value of the cell. As mentioned earlier, that the correlation of a variable with itself is 1. For that reason all the diagonal values are 1.00.

Importance of GroupBy in Statistical Analysis

Pandas’ GroupBy is a powerful and versatile function in Python. It allows you to split your data into separate groups to perform computations for better analysis.GroupBy is used to quickly summarize data and aggregate it in a way that’s easy to interpret.

.groupby() method allows you to group rows of data together and call aggregate functions

Consider the following Data Frame

Now you can use the .groupby() method to group rows together based on Statistical analysis column name. For instance let’s group based on Team. This will create a DataFrameGroupBy object

df.groupby(‘Team’)

The grouped data is saved in an object 

Output:

<pandas.core.groupby.DataFrameGroupBy object at 0x113014128>

Now we can save this object as a new variable with the name team

team = df.groupby(‘Team’)

Now we can perform all the aggregation functions on that variable

team.mean().astype(‘int32’)

Output:

We can perform all aggregation functions like mean median mode on this grouped data.

We can also group two columns at the same. This is how you can do it

teams=df.groupby(['Team','Rank'])
teams.mean().astype(‘int32’).   

Output:

Filtering and Sorting 

In order to perform Statistical analysis, We should also perform a lot of filtering operations. Pandas provide many methods to Statistical analysis filter a Data frame

We can filter the data using Python Comparison Operators and Python Boolean Operators

Comparison Operators 

These operators compare the values on either sides of them and decide the relation among them. They are also called Relational operators.

==

If the values of two operands are equal, then the condition becomes true.

!=

If values of two operands are not equal, then condition becomes true.

>

If the value of left operand is greater than the value of right operand, then condition becomes true.

<

If the value of left operand is less than the value of right operand, then condition becomes true.

>=

If the value of left operand is greater than or equal to the value of right operand, then condition becomes true.

<=

If the value of left operand is less than or equal to the value of right operand, then condition becomes true.

Boolean Operators

Python has three Boolean operators that are typed out as plain English words:

or

Returns True if one of the statements is true

And

Returns True if both statements are true

not

Reverse the result, returns False if the result is true

Now we will learn how to filter the values of the DataFrame based on certain condition 

Consider the following DataFrame

If I want to find the names of the team with ranking 1. This is how we can do it

df[df[‘Rank’]==1]

Output:

If I want to find the names of the team with ranking greater than 3

df[df[‘Rank’]<3]

Output:

We can also check multiple conditions using Boolean operators

Now we will find the teams with ranking 1 and points greater than 800

df[(df[‘Rank’]==1)&(df['Points']>800)]

Output: