Previously, we have created an Azure DevOps Continuous Integration (CI) and Continuous deployment (CD) pipelines to independently deploy a SQL Server database project. However, in an enterprise data warehouse environment, the databases are mostly dependent on other databases. Because, in DWH, we access objects from multiple databases (like accessing the staging layer objects into the […]
In the previous post, we have created a CI (Continous Integration) pipeline for a SQL Server database project. For this demo, as like our previous demo, we will be using a SQL Server instance running on an on-prem machine along with a locally installed Azure DevOps Server. Please note that we are not using the
In this post, we are going to discuss how we can enable continuous integration and continuous deployment for a SQL Server Database project using Azure DevOps Server. For this demo, we will be using a SQL Server instance running on-prem along with a locally installed Azure DevOps Server. Continuous integration and continuous delivery (in short
Data compression is a technique that encodes the original data in such a way so that it can be represented with fewer bits on the disk. The data compression process is used to reduce the size of the data files on the disk. We know that the Hadoop framework is meant for large scale data
Data compression in Hive – An Introduction to Hadoop Data Compression Read More »
Apache Spark is a very powerful general-purpose distributed computing framework. It provides a different kind of data abstractions like RDDs, DataFrames, and DataSets on top of the distributed collection of the data. Spark is highly scalable Big data processing engine which can run on a single cluster to thousands of clusters. To follow this exercise,
Read and write data to SQL Server from Spark using pyspark Read More »
Apache Spark is a general-purpose big data processing engine. It is a very powerful cluster computing framework which can run from a single cluster to thousands of clusters. It can run on clusters managed by Hadoop YARN, Apache Mesos, or by Spark’s standalone cluster manager itself. To read more on Spark Big data processing framework,
Install Spark on Windows (Local machine) with PySpark – Step by Step Read More »
Apache Spark is a general purpose distributed computing engine used for Big Data processing – Batch and stream processing. It provides high level APIs like Spark SQL, Spark Streaming, MLib, and GraphX to allow interaction with core functionalities of Apache Spark. Spark also facilitates several core data abstractions on top of the distributed collection of
RDD, DataFrame, and DataSet – Introduction to Spark Data Abstraction Read More »
What is Spark Apache spark is an open source general purpose distributed cluster computing framework. It is an unified computing engine for big data processing. Spark is designed for lightning fast cluster computing especially for fast computation. An application can run up to 100 times faster than Hadoop MapReduce using Spark in-memory cluster computing. Also,
Big Data processing using Apache Spark – Introduction Read More »
Apache Hive is a big data query language which is used to read, transform and write large datasets in a distributed environment. It has a SQL like syntax which gets translated into a MapReduce job in order to execute on Hadoop clusters. In Hadoop ecosystem, we use Hive for batch processing to extract, transform and
In this article, we will discuss two important concepts “Partitioning and Bucketing” in Hive. These are used to improve query performance and it is important to understand them so that you can apply them efficiently. So let’s start with Partitioning. Partitioning in Hive Partitioning is a technique which is used to enhance query performance in
