Introduction While working with pandas DataFrames, we may get the truncated text data especially if the data size is large. The truncation of the text data while displaying can create difficulties when attempting to thoroughly analyze the complete content. This is frustrating, especially when the text contains important details that are crucial for the analysis. […]
In the domain of data analysis and manipulation, pandas is a powerhouse library in Python. However, when working with larger datasets or complex dataframes, displaying all columns can be a challenging task. When we display the content of a pandas dataframe, pandas try to fit all the dataframe columns on the screen. As a result,
The Easiest Way to Display All Columns of a Pandas DataFrame Read More »
Categorical variables are very common data types in machine learning datasets. These variables represent non-numeric values such as days of the week, gender, colors, etc. However, typically, we need to convert these categorical variables to a numerical format before using them in machine learning algorithms. One-hot encoding is a powerful technique that accomplishes this transformation
Pandas is one of the most popular Python libraries that is used for data manipulation and for data analysis. It provides very convenient and useful methods to analyze tabular data. One of Pandas dataframe’s essential functions is its to_sql method that allows seamless integration with various databases. However, it’s crucial to understand how to handle
Handling exceptions: Rollback pandas dataframe’s to_sql operation Read More »
In today’s digital age, organizations deal with large amounts of sensitive data that includes PII data such as names, addresses, phone numbers, and email addresses. Protecting this data is critical to prevent identity theft and other types of fraud, and PII detection is a key step in the process. In this post, we will discuss
PII Data Identification using Presidio Open Source ML Library Read More »
This post briefs how we can use the k-fold cross-validation to evaluate a Machine Learning model performance using the Scikit-learn library in Python. We know that the performance of a Machine Learning model depends on the training dataset. Also, if the training dataset has a peculiarity, the model created with that dataset will not work
Introduction to k-fold Cross-Validation in Python Read More »
The exploratory data analysis is a very important step in a Data Science project. It helps us to visualize the data and identify any hidden trends that might not be visible with summary statistics alone. So, we can use matplotlib and seaborn libraries to create stunning visuals in Python. However, the pandas.plotting module of the
Create pair plots using scatter_matrix method in pandas Read More »
We know that EDA (Exploratory Data Analysis), is the process of organizing, plotting, and summarizing the data to find trends, patterns, and outliers using statistical and visual methods. Here, we have already discussed various methods of performing EDA with their pros and cons on an underlying dataset. ECDF plot is another visual method of performing
In the previous post, “Tidy Data in Python – First Step in Data Science and Machine Learning”, we discussed the importance of the tidy data and its principles. In a Machine Learning project, once we have a tidy dataset in place, it is always recommended to perform EDA (Exploratory Data Analysis) on the underlying data
