If you’ve been searching for a job or working in big data, you will know that companies are increasingly expecting you to have experience in Power Bi, Tableau, SQL, MongoDB, and Python, as well as Microsoft Excel. But which one do you learn first to become more valuable in the marketplace?

Learning Python is the logical first step. It can enhance what you can do with all of the above tools and will go a long way in improving your skills and career opportunities in analyzing big data. If you know Python scripting well enough to produce any kind of result or visualization, it may even negate your need to know software packages such as Tableau and Power Bi, which are limited in scope by user interfaces. Python libraries like 

 can also query data to get the same results as SQL can, which is just another reason to consider learning how to program in Python first.

Another benefit of learning Python is that most data-analytics applications have APIs that can be accessed by Python. Beginning with Python gives you potential to be a more advanced user of other data-analytics software. Here are a few specific examples:

Tableau has an API that can be queried using Python.

Power Bi supports running Python scripts.

Python libraries can work with Excel data and automate many Excel functions.

Python libraries can be used to do SQL queries on structured data sets.

Python libraries can be used to do NoSQL queries on unstructured data sets.

In this roundup, you’ll explore a few basic code snippets from some core Python analytics libraries that will demonstrate how easy Python is to learn. The time you spend learning Python libraries for data analysis will be a good investment, and its capabilities are being expanded all the time.

The Python libraries that this roundup will review include 

 These particular libraries have been chosen because they make up the core Python libraries used by data analysts and data scientists. They’re all useful tools that are worth exploring, and you’ll get to see a summary of ten interesting sample projects that leverage Python for big data analysis.

pandas Library for Importing and Finding Data

Reading data is the first step in any data science project. 

 is arguably the best Python library to learn about for importing data into any Python analytics project. It’s included in parts of Higher National Diploma analytics courses covering Python as well as MSc in Data Analytics courses.

Data is imported in a pandas DataFrame (df), which can be considered a bit like a database table. The DataFrame can be further queried and manipulated for big data analysis projects. The 

 library supports commands that are very similar to SQL for filtering information and can do join analysis the same way SQL does.

 library to import data into a DataFrame from just about any file type, as demonstrated by the code snippet below, which has been adapted from 

This simple piece of code can tell you a lot about how easy it is to learn Python:

In the first line of code, Python is importing the pandas library and referencing it as 

 just so you don’t have to write “pandas” out in full every time you want to use one of its functions.

The second line of code defines the DataFrame variable as 

 and imports the contents of the csv file into it by using the pandas function 

 from the pandas library and applying it to call in the csv file.

The final line of code prints the contents of the DataFrame to the screen.

Knowing all the functions available for reading different file types is the cornerstone to working with big data. You can also import data in JSON format from online APIs into a DataFrame.

If you are interested in doing so, the code below, adapted from 

You can also use pandas to do SQL-type queries on large data sets. A typical SQL query might look like this:

If you are unsure of what analytics technology to learn first and have limited time, you don’t necessarily need to learn SQL before learning Python. All core SQL queries can be done with pandas commands. In time, if you do learn SQL, there are Python libraries such as 

 that provide an interface between Python and SQL syntax.

NumPy Library for Math Functions and Setting Data Dimensions

 library involve many math functions. There is some overlap between what the 

 library offers and what is in Python’s own math library as the data science and analytics community have widely adopted importing NumPy into Python.

 is continuously optimized to work with the latest CPU architectures. Its arrays can work fifty times faster than using Python lists to store data. For that reason, NumPy is frequently used in Python data-science projects, where speed and resources are very important.

Example 3: Using NumPy Math Functions to Create a Haversine Distance Calculation

 supports radians, sin, cos, square root, delta, lambda, arctan, and many more mathematical functions, which can then be combined to make even greater mathematical functions to fit your purpose.

In the example below, a programmer has created a function that calculates 

. Here, the variables for the longitude and latitude of London and Paris are passed through the function, and the distance is printed on the screen:

Example 4: Resetting the Dimensions of Data Using NumPy

 to reset the dimensions of data sets extracted from existing DataFrames/lists (e.g., taking a list of eight numbers and converting it into an array of two rows and four columns).

Resetting the dimensions of data is particularly useful for machine learning exercises, where the dimensions of the training data need to match the dimensions of the test data.

For image-recognition scripts, you might have to use 

 to convert the dimensions of an imported image file or files.

Matplotlib Library for Graph Visualizations and Editing Image Colors

If you’re used to using Excel, Tableau, or PowerBi to create your visualizations, you understand that you can only be as creative with the visualization of your data as the scope of the software allows you to.

The Matplotlib library has a wide range of graphs that cover practically every kind of business use. Check out the template visualizations available on the Matplotlib website 

, and adapt online tutorials to meet your data set needs.

Python is a very cohesive programming language, and these visualization libraries often have many tutorials available that can take you step-by-step through how to efficiently work with data imported into either 

 data arrays. Let’s take a look at an example of each.

Example 5: Using Matplotlib to Create a Scatter and Bar Graph

This example shows how to use the Matplotlib library to interpret data from pandas DataFrames into visualizations and graphs. In the code block below, adapted from 

, the data in the DataFrame is manually typed. However, the data could just as easily be imported from a spreadsheet or from an online API.

Example 6: Using Matplotlib to Import Image Data as a NumPy Array

More and more training sets in big data use and compare images.

Matplotlib is typically considered to be a library that does graphs, but it can be equally useful for importing images into presentations that you want to create using Python.

Matplotlib can also be used to load images as number arrays that can be used with NumPy:

Example 7: Using Matplotlib to Distort an Image Using NumPy Array Slicing

The values of the numbers that make up an image can be changed or sliced to distort the images, creating more training images from a single image. This is useful as it allows you to create larger training data sets for machine learning image-recognition projects from fewer images.

, the image imported using the Matplotlib library is distorted to a different hue:

Plotly Library for Creating Maps and Graphs

Plotly is another impressive library for making high-quality graphs. Whether you decide to use Matplotlib or Plotly for creating graphs is a matter of personal preference as the syntax is very similar. One important distinction that may give Plotly a leg up, though, is its very impressive gallery of 

 for working with longitude and latitude data.

For a full list of Plotly’s interactive graphs, check out their 

Example 8: Using Plotly to Plot a Line on a Map from London to Paris

Plotly is excellently suited for big data projects involving 

. Below is some starter code, adapted from 

, to create a map image and draw a line between two longitude and latitude coordinates:

SciPy Library for Regression and Hypothesis Testing

If you haven’t found the mathematical function that you need in Python’s own math library or in NumPy, SciPy is the next library to check out. It can be used for a number of advanced and scientific math calculations. The calculations most frequently used by big data scientists would include linear regression and hypothesis-testing functions for z-tests and t-tests.

Example 9: Using SciPy to Make Linear Predictions

 demonstrates how to use linear regression to predict the speed of a ten-year-old car. You can adapt this code to your own purpose—say, predicting the speed an asteroid will hit Earth based on its weight by changing the 

 measurements for previously known speeds:

scikit-learn Library for Machine Learning 

Finally, the scikit-learn library is a great resource to help you learn machine learning. Courses that teach students how to do predictions with machine learning often start out with the 

The first part of using the K-means algorithm is knowing how many clusters exist in a data set. As you progress with Python, you will find the code that can help you calculate the optimum amount of clusters that probably exist in a data set.

In some cases, you may already know how many different clusters exist in a data set. For instance, in the code example below, the author knows that there are two clusters in the manually typed data set: cluster 0 is cake, and cluster 1 is bread.

Example 10: Using scikit-learn to Make Predictions

, explores how to predict whether a recipe is for cake or bread based on various measurements in the recipe. To better help you understand what the code is doing, the words “cake” and “bread” have been manually typed out to indicate clusters 1 and 2 rather than using the numeric values described above.

The code below includes the measurements of sugar, flour, and eggs, respectively, which are used to train the data:

[335,265,4],[200,200,4],[225,225,4],[300,300,4]

Then, these numbers are followed by a list of some bread-recipe measurements found online:

Together, they make a long list in a NumPy array that looks like this:

np.array([[335,265,4],[200,200,4],[225,225,4],[300,300,4],[4,300,0],[6,500,0],[0,750,0],[0,500,0]])

Each of the measurements is categorized in a subsequent list. The first four classifications are cake and relate to the measurements provided for the first four recipes; the next four classifications are bread and relate to the measurements provided for the last four recipes:

['cake', 'cake', 'cake', 'cake', 'bread', 'bread', 'bread', 'bread']

Again, the full coding below has been manually typed for illustrative training purposes. This example demonstrates a very simple script program where a user is prompted to provide measurements of three ingredients. When the K-Means machine learning algorithm is presented with these numbers, the algorithm has to decide, based on what it knows from the training data, whether those combinations of numbers more closely match a cake recipe or a bread recipe.

The following code shows you that making a machine learning script can be quite simple with even a small understanding of Python and knowledge for which library and function to use.

This roundup covered a variety of libraries and examples that have demonstrated the benefits of using Python for big data analysis. First, you learned about using 

 for importing and filtering data in your big data projects.

Once you have imported and filtered the relevant data using pandas, you will want to start applying functions from libraries like 

, which have a range of statistical and scientific mathematical functions to summarize the data.

Once you’ve accomplished that, you’ll want to explore how to create nice data visualizations. There are some great Python libraries to do so: 

 is a great option, or you could consider 

, which is especially useful if you’re working with geographical data.

The basics of the above libraries are easy to learn from online tutorials that cover the core topics of what every data professional should know in order to use Python to extract precise data and present it nicely. Often, these libraries are used in conjunction with free, open source editors like 

Posts from Christopher Collins

10 examples of using Python for big data analysis

Exploring some of the most powerful Python modules for data analysis

Christopher Collins

Writing Openlayer posts since

Number of posts