Author : Gopal Krishna Ranjan

Gopal is a passionate Data Engineer and Data Analyst. He has implemented many end to end solutions using Big Data, Machine Learning, OLAP, OLTP, and cloud technologies. He loves to share his experience at https://www.sqlrelease.com/. Connect with Gopal on LinkedIn at https://www.linkedin.com/in/ergkranjan/.

Relationship between Binomial and Poisson distributions

In this post, we are going to discuss the Relationship between Binomial and Poisson distributions. We know that Poisson distribution is a limit of Binomial distribution for a large n (number of trials) and small p (independent probability for each trial) values. A large number of trials n with very small probability p indicates a rare event in a binomial distribution. Considering this, we will simulate these distributions and then we will create a  CDF (cumulative distributed function) plot of Binomial and Poisson distributions. It will help us to understand the similarity between a Poisson experiment and a rare event Binomial experiment.

In this post, we will not be going into the mathematical details of Binomial and Poisson distributions. However, we will be using NumPy’s random module available in Python to simulate these distributions using a technique called bootstrapping.

Relationship between Binomial and Poisson distributions

Let’s start by understanding the … More

Delete statement

In the previous chapter, we discussed the T-SQL’s Update statement. Now, in this chapter, we are going to discuss how we can use T-SQL’s DELETE command to delete records from a table in SQL Server. We will also discuss some best practices we need to follow while using the DELETE command in MS SQL Server.

What is DELETE command

DELETE command is used to delete/remove existing records from a SQL table. Depending on the need, we can delete all or a few selected records from a table. If required, we can delete the records in a table based on the records which exist in other tables. However, in this case, we need to join all the table(s) in the FROM clause of the DELETE statement. Let’s have a look at the syntax of the DELETE statement:

Syntax of DELETE command in SQL Server

To delete all the records in … More

Convert Jupyter notebooks to PDF

Jupyter lab is the next-generation web-based UI experience for Jupyter notebook users. It facilitates a tab-based programming interface that is highly extensible and interactive. It supports 40+ programming languages. We have already discussed how we can use Jupyter notebooks for interactive data analysis with SQL Server. With the help of Jupyter notebooks, we can keep headings, comments, code, output, and advanced charts and visuals in a single document in an orderly fashion. It helps Data Scientists and Data Analysts to have highly interactive presentations. In case you have already installed Jupyter notebooks and want to know how we can change the home directory for Jupyter notebooks, visit the blog “Change Jupyter Notebook startup folder on Windows and Mac OS “. Let’s discuss how we can Convert Jupyter notebooks to PDF documents directly from the web-browser or using nbconvert command from command prompt.

During … More

Interactive Data Analysis with SQL Server using Jupyter Notebooks

In this post “Interactive Data Analysis with SQL Server using Jupyter Notebooks“, we will demonstrate how we can use Jupyter Notebooks for interactive data analysis with SQL Server. Jupyter notebooks are one of the most useful tools for any Data Scientist/Data Analyst. It supports 40+ programming languages and facilitates web-based interactive programming IDE. We can put comments, headings, codes, and output in one single document. This document maintains the context to the original data source which means we can re-execute the code whenever we need it. This feature facilitates Data scientists/Data analysts to play with the code during the presentations. Also, these notebooks are very handy in sharing and can be shared easily across the teams.

What is Jupyter Lab

Jupyter Lab is the next-generation web-based tool for Jupyter notebooks. It enables tab based programming model which is highly extensible. We can arrange multiple windows … More

Update statement

In the previous chapter, we discussed the T-SQL’s Insert statement. Now, in this chapter, we are going to discuss the T-SQL’s UPDATE command. We will discuss how we can use the T-SQL’s Update command with the help of examples. We will also discuss some best practices we need to follow while using the UPDATE command in SQL Server.

What is UPDATE command

UPDATE command is used to modify existing records in a SQL table. Depending on the need, we can update all or a few selected records in a table. Also, we can update the records in a table based on the records of other tables. However, in this case, we need to join all the tables in the FROM clause of the UPDATE statement. Let’s have a look at the syntax of the UPDATE statement which updates all rows of the table:

More

Data compression in Hive – An Introduction to Hadoop Data Compression

Data compression is a technique that encodes the original data in such a way so that it can be represented with fewer bits on the disk. The data compression process is used to reduce the size of the data files on the disk. We know that the Hadoop framework is meant for large scale data processing (Big Data processing) which includes lots of data files stored on HDFS or supported file systems. So data compression can be very helpful in reducing storage requirements, and in reducing the amount of data to be transferred between mappers and reducers which usually occurs over the network. In Hadoop, data compression can be implemented using Hive or any other MapReduce component. In this post, we will discuss the widely used HiveQL data compression formats or codec (compressor/decompressor) schemes.

Data compression in Hive

HiveQL supports different codec schemes that are used to compress … More

Python use case – Export SQL table data to excel and CSV files – SQL Server 2017

In this post, we are going to discuss how we can export SQL Server table data to an Excel file or to a CSV file using Python’s pandas library. Prior to SQL Server 2017, we could use one of the below methods to export data from SQL Server to Excel or CSV file:

1. Create an SSIS package to export the data from SQL Server – This option can be a good choice if we want to reuse the export process again and again. Also, if we want to put moderate/complex transformations during data export, this option can be a better choice.
2. Use SQL Server Import/export wizard – SQL Server provides in-built data export/import wizard which can be used in case we want to export data with no/minimal transformations.
3. OS-based copy-paste functionality – We can simply copy the query output from SQL Server using Ctrl + C option and then open
More

SQL Server – Error 1061: The service cannot accept control messages at this time

Sometimes when we try to restart “SQL Server service” we might get an error “Windows could not stop the SQL Server (MSSQLSERVER) service on Local Computer” with error code and description “Error 1061: The service cannot accept control messages at this time“. In this post, “SQL Server – Error 1061: The service cannot accept control messages at this time”, we will discuss the workaround which can help us to fix this issue.

This error occurs when we try to restart the SQL Server service using SSMS object explorer or/and using the services console. Let’s try to restart the SQL Server service using object explorer and services console.

Restarting SQL Server services using Object explorer:

Restarting SQL Server using services console:

Below is the error screenshot:

Fix – Error 1061: The service cannot accept control messages at this time

To fix this issue, we can … More

Read and write data to SQL Server from Spark using pyspark1

Apache Spark is a very powerful general-purpose distributed computing framework. It provides a different kind of data abstractions like RDDs, DataFrames, and DataSets on top of the distributed collection of the data. Spark is highly scalable Big data processing engine which can run on a single cluster to thousands of clusters. To follow this exercise, we can install Spark on our local machine and can use Jupyter notebooks to write code in an interactive mode. In this post “Read and write data to SQL Server from Spark using pyspark“, we are going to demonstrate how we can use Apache Spark to read and write data to a SQL Server table.

Read SQL Server table to DataFrame using Spark SQL JDBC connector – pyspark

Spark SQL APIs can read data from any relational data source which supports JDBC driver. We can read the data of a SQL Server table … More

Install Spark on Windows (Local machine) with PySpark – Step by Step

Apache Spark is a general-purpose big data processing engine. It is a very powerful cluster computing framework which can run from a single cluster to thousands of clusters. It can run on clusters managed by Hadoop YARN, Apache Mesos, or by Spark’s standalone cluster manager itself. To read more on Spark Big data processing framework, visit this post “Big Data processing using Apache Spark – Introduction“. Here, in this post, we will learn how we can install Apache Spark on a local Windows Machine in a pseudo-distributed mode (managed by Spark’s standalone cluster manager) and run it using PySpark (Spark’s Python API).

Install Spark on Local Windows Machine

To install Apache Spark on a local Windows machine, we need to follow below steps: