2023

Dynamically Create Spark DataFrame Schema from Pandas DataFrame

Leave a Comment / Python, Spark / / / , , , , ,

Apache Spark has become a powerful tool for processing large-scale data in a distributed environment. One of its key components is the Spark DataFrame, which offers a higher-level abstraction over distributed data and enables efficient data manipulation. Spark DataFrame is typically used to manipulate large amounts of data in a distributed environment. When working within […]

Dynamically Create Spark DataFrame Schema from Pandas DataFrame Read More »

Python Regex – re match vs re search vs re findall

Leave a Comment / Python / / / ,

Python Regular expressions, known as regex, are a powerful tool for pattern matching and string manipulation. Python provides a built-in module called re that allows us to use regular expressions. This module offers several functions for performing various regex operations, including matching, searching, and finding all occurrences of a pattern. In this blog post, we

Python Regex – re match vs re search vs re findall Read More »

Git: Step-by-Step Guide to Rebasing the Develop Branch onto Main

Leave a Comment / Big Data/Cloud / / / , ,

Rebasing the develop branch onto the main branch is a popular workflow in Git that allows you to incorporate the latest changes from the main branch into the develop branch while maintaining a linear history. This is very useful especially when working on a project working together with multiple teams and developers. This post provides

Git: Step-by-Step Guide to Rebasing the Develop Branch onto Main Read More »

SQL Server Docker Installation: Step-by-Step Guide for Windows

Leave a Comment / Big Data/Cloud, SQL Server / / / , , , , ,

SQL Server is a very popular, powerful, and versatile option in the ever-evolving landscape of database management. It is a robust and widely used relational database management system (RDBMS) developed and managed by Microsoft. SQL Server natively supports SQL (Structured Query Language) for querying and manipulating data stored in the tables. This makes SQL Server

SQL Server Docker Installation: Step-by-Step Guide for Windows Read More »

Displaying Long Strings in Pandas: How to Print Complete Text in DataFrame Without Truncation

Leave a Comment / Data Analysis, Python / / / , , ,

Introduction While working with pandas DataFrames, we may get the truncated text data especially if the data size is large. The truncation of the text data while displaying can create difficulties when attempting to thoroughly analyze the complete content. This is frustrating, especially when the text contains important details that are crucial for the analysis.

Displaying Long Strings in Pandas: How to Print Complete Text in DataFrame Without Truncation Read More »

The Easiest Way to Display All Columns of a Pandas DataFrame

Leave a Comment / Data Analysis, Python / / / , , ,

In the domain of data analysis and manipulation, pandas is a powerhouse library in Python. However, when working with larger datasets or complex dataframes, displaying all columns can be a challenging task. When we display the content of a pandas dataframe, pandas try to fit all the dataframe columns on the screen. As a result,

The Easiest Way to Display All Columns of a Pandas DataFrame Read More »

Simplify Data Analysis: One-Hot Encoding for Multi-Valued Categorical Variables in Pandas DataFrame

Leave a Comment / Data Analysis, Python / / / , , , ,

Categorical variables are very common data types in machine learning datasets. These variables represent non-numeric values such as days of the week, gender, colors, etc. However, typically, we need to convert these categorical variables to a numerical format before using them in machine learning algorithms. One-hot encoding is a powerful technique that accomplishes this transformation

Simplify Data Analysis: One-Hot Encoding for Multi-Valued Categorical Variables in Pandas DataFrame Read More »

Handling exceptions: Rollback pandas dataframe’s to_sql operation

Leave a Comment / Data Analysis, Python / / / , , , ,

Pandas is one of the most popular Python libraries that is used for data manipulation and for data analysis. It provides very convenient and useful methods to analyze tabular data. One of Pandas dataframe’s essential functions is its to_sql method that allows seamless integration with various databases. However, it’s crucial to understand how to handle

Handling exceptions: Rollback pandas dataframe’s to_sql operation Read More »

Read and write data from Cosmos DB to Spark

Leave a Comment / Spark / / / , , ,

In the vast and ever-expanding landscape of big data technologies, Apache Spark is an open-source, lightning-fast, and versatile framework that ignites the power of large-scale data analytics. It is a powerful distributed data processing framework that helps us to analyze and derive insights from massive datasets. On the other hand, Cosmos DB is a globally

Read and write data from Cosmos DB to Spark Read More »

Create pandas dataframe from MongoDB collection

Leave a Comment / Data Analysis, Python / / / , ,

In this post, we will learn how we can create pandas dataframe from MongoDB collection. MongoDB is a popular NoSQL database that stores data in a JSON-like format and offers a flexible and scalable solution for managing large volumes of data. When working with data stored in MongoDB, it is often necessary to analyze and

Create pandas dataframe from MongoDB collection Read More »