Planet Python

Last update: November 19, 2019 04:47 AM UTC

November 19, 2019

The No Title® Tech Blog

Book review – Supercharged Python, by Brian Overland and John Bennet

If you have been following beginner or even intermediate guides on Python and are starting to feel the need for more advanced learning, this book may be the one you have been looking for.

November 19, 2019 01:27 AM UTC

November 18, 2019

Podcast.__init__

From Simple Script To Beautiful Web Application With Streamlit

Building well designed and easy to use web applications requires a significant amount of knowledge and experience across a range of domains. This can act as an impediment to engineers who primarily work in so-called back-end technologies such as machine learning and systems administration. In this episode Adrien Treuille describes how the Streamlit framework empowers anyone who is comfortable writing Python scripts to create beautiful applications to share their work and make it accessible to their colleagues and customers. If you have ever struggled with hacking together a simple web application to make a useful script self-service then give this episode a listen and then go experiment with how Streamlit can level up your work.

November 18, 2019 09:57 PM UTC

Ian Ozsvald

“Higher Performance Python” at PyDataCambridge 2019

I’ve had the pleasure of speaking at the first PyDataCambridge conference (2019), this is the second PyData conference in the UK after PyDataLondon (which colleagues and I co-founded 6 years back). I’m super proud to see PyData spread to 6 regional meetups and now 2 UK conferences.

We had over 200 attendees and the conference (and a swanky black-tie conference dinner) and the single-track event had a rich set of topics (schedule). For me scikit-multiflow (extending sklearn to streaming data) was a hit along with model stability checking (by FarFetch) and an overview of GA2M (an extended Generalised Adaptive Model with explainability). Thanks to the speakers for fine talks and the audience for fine questions and Cambridge Spark and the PyDataCambridge meetup for helping make it all happen!

I spoke on Higher Performance Python with a focus towards making Pandas operations go faster and an eye on the upcoming Second Edition of our High Performance Python (O’Reilly) book. The talk covers:

• Using line_profiler to evaluate sklearn’s LinearRegression vs NumPy’s lstsq (spoiler – lstsq is much faster but that’s due to sklearn being much safer, the slow-down is all due to safety code in sklearn that helps keep your productivity higher overall)
• Using Pandas for line-by-line iteration (slow) vs apply (faster) and apply with raw=True to expose NumPy arrays (fastest)
• Using Numba to JIT compile lstsq using apply with raw=True for a huge speed-up
• Using Dask to parallelise the Numba solution for further speed-ups
• Advice on being a “highly performant data scientist”

The last point is important – going “compiler happy” and writing highly efficient code may well slow down your team and your overall velocity. Amongst other items I recommended profiling first, maybe introducing Dask & Numba only with a team’s consent and looking at tools like Bulwark to add tests to DataFrames to avoid being derailed by strange data bugs.

Right now Micha and I are busily working to complete the second edition of our book, all going well it’ll be in for Christmas with a publication date around April 2020.

The post “Higher Performance Python” at PyDataCambridge 2019 appeared first on Entrepreneurial Geekiness.

November 18, 2019 08:03 PM UTC

Python Engineering at Microsoft

Python in Visual Studio Code – November 2019 Release

We are pleased to announce that the November 2019 release of the Python Extension for Visual Studio Code is now available. You can  download the Python extension from the Marketplace, or install it directly from the extension gallery in Visual Studio Code. If you already have the Python extension installed, you can also get the latest update by restarting Visual Studio Code. You can learn more about  Python support in Visual Studio Code  in the documentation.  

In this release we focused mostly on product quality. We closed a total of 60 issues, 39 of them being bug fixes. However, we’re also pleased to deliver delightful features such as: 

• Add imports “quick fix” when using the Python Language Server 
• Altair plot support  
• Line numbers in the Notebook Editor.  

If you’re interested, you can check the full list of improvements in our changelog. 

Add Imports “Quick Fix” when using the Python Language Server 

We’re excited to announce that we have brought the magic of automatic imports to Python developers in VS Code by way of an add imports quick fix. Automatic imports functionality was one of the most requested features on our GitHub repo (GH21), and when you enable the Microsoft Language Server, you will get this new functionality. To enable the Language Server, add the setting “python.jediEnabled”: false to your settings.json file. 

The add imports quick fix within VS Code is triggered via a code action lightbulb. To use the quick fix, begin typing a package name within the editor for which you do not have an import statement at the header of the file. You will notice that if a code action is available for this package (i.e. you have a module installed within your environment with the name you’ve supplied), a yellow squiggle will appear. If you hover over that text, a code action lightbulb will appear indicating that an ’import’ code action is available for the package. You’ll see a list of potential imports (again, based on what’s installed within your environment), allowing you to choose the package that you wish to import. 

The add imports code action will also recognize some of the most popular abbreviations for the following Python packages: numpy as np, tensorflow as tf, pandas as pd, matplotlib.pyplot as plt, matplotlib as mpl, math as m, scipy.io as spio, and scipy as sp. 

The import suggestion list is ordered such that all import statements that appear at the top of the list are package (or module) imports; those that appear lower in the list are import statements for additional modules and/or members (e.g. classes, objects, etc.) from specified packages. 

Make sure you have linting enabled since this functionality is tied to the Language Server linting capability. You can enable linting by opening the Command Palette (View > Command Palette…), running the “Python: Enable Linting” command and selecting “On” in the drop-down menu. 

Altair plots support

The Notebook Editor and the Python Interactive window now both support rendering plots built with Altair,  a declarative statistical visualization library for Python.

Line Numbers in the Notebook Editor

Line numbers are now supported in the notebook editor. On selected code cells, you can toggle the line numbers by pressing the “L” key.

Other Changes and Enhancements

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python in Visual Studio Code. Some notable changes include:

• Fix running a unittest file to not execute only the first test. (thanks Nikolay Kondratyev). (#4567)
• Added commands translation for Farsi and Turkish (thanks Nikronic). (#8092)
• Added command translations for Turkish (thanks alioguzhan). (#8320)
• Place all plots on a white background regardless of theme. (#8000)

We are continuing to A/B test new features, so if you see something different that was not announced by the team, you may be part of the experiment! To see if you are part of an experiment, you can check the first lines in the Python extension output channel. If you wish to opt-out of A/B testing, you can open the user settings.json file (View > Command Palette… and run Preferences: Open Settings (JSON)) and set the “python.experiments.enabled” setting to false.

Be sure to download the Python extension for Visual Studio Code now to try out the above improvements. If you run into any problems, please file an issue on the Python VS Code GitHub page.

The post Python in Visual Studio Code – November 2019 Release appeared first on Python.

November 18, 2019 06:52 PM UTC

Test and Code

94: The real 11 reasons I don't hire you - Charity Majors

You've applied for a job, maybe lots of jobs.
Depending on the company, you've gotta get through:

• a resume review
• a coding challange
• a phone screen
• maybe another code example
• an in person interview

If you get the job, and you enjoy the work, awesome, congratulations.

If you don't get the job, it'd be really great to know why.

Sometimes it isn't because you aren't a skilled engineer.

What other reasons are there?

Well, that's what we're talking about today.

Charity Majors is the cofounder and CTO of Honeycomb.io, and we're going to talk about reasons for not hiring someone.

This is a very informative episode both for people who job hunt in the future and for hiring managers and people on the interview team.

Special Guest: Charity Majors.

Sponsored By:

• PyCharm Professional: Try PyCharm Pro before Nov 28 with a 4 month free trial. Promo Code: TESTNCODE2019

Support Test & Code: Python Software Testing & Engineering

Links:

• The (Real) 11 Reasons I Don’t Hire You — The article

November 18, 2019 05:15 PM UTC

Stack Abuse

Quicksort in Python

Introduction

Quicksort is a popular sorting algorithm and is often used, right alongside Merge Sort. It's a good example of an efficient sorting algorithm, with an average complexity of O(n log_n). Part of its popularity also derives from the ease of implementation.

We will use simple integers in the first part of this article, but we'll give an example of how to change this algorithm to sort objects of a custom class.

Quicksort is a representative of three types of sorting algorithms: divide and conquer, in-place, and unstable.

• Divide and conquer: Quicksort splits the array into smaller arrays until it ends up with an empty array, or one that has only one element, before recursively sorting the larger arrays.
• In place: Quicksort doesn't create any copies of the array or any of its subarrays. It does however require stack memory for all the recursive calls it makes.
• Unstable: A stable sorting algorithm is one in which elements with the same value appear in the same relative order in the sorted array as they do before the array is sorted. An unstable sorting algorithm doesn't guarantee this, it can of course happen, but it isn't guaranteed.

This is something that becomes important when you sort objects instead of primitive types. For example, imagine you have several Person objects that have the same age, i.e. Dave aged 21 and Mike aged 21. If you were to use Quicksort on a collection that contains both Dave and Mike, sorted by age, there is no guarantee that Dave will come before Mike every time you run the algorithm, and vice versa.

Quicksort

The basic version of the algorithm does the following:

Divide the collection in two (roughly) equal parts by taking a pseudo-random element and using it as a pivot.

Elements smaller than the pivot get moved to the left of the pivot, and elements larger than the pivot to the right of it.

This process is repeated for the collection to the left of the pivot, as well as for the array of elements to the right of the pivot until the whole array is sorted.

When we describe elements as "larger" or "smaller" than another element - it doesn't necessarily mean larger or smaller integers, we can sort by any property we choose.

If we have a custom class Person, and each person has a name and age, we can sort by name (lexicographically) or by age (ascending or descending).

How Quicksort Works

Quicksort will, more often than not, fail to divide the array into equal parts. This is because the whole process depends on how we choose the pivot. We need to choose a pivot so that it's roughly larger than half of the elements, and therefore roughly smaller than the other half of the elements. As intuitive as this process may seem, it's very hard to do.

Think about it for a moment - how would you choose an adequate pivot for your array? A lot of ideas about how to choose a pivot have been presented in Quicksort's history - randomly choosing an element, which doesn't work because of how "expensive" choosing a random element is while not guaranteeing a good pivot choice; picking an element from the middle; picking a median of the first, middle and last element; and even more complicated recursive formulas.

The most straight-forward approach is to simply choose the first (or last) element. This leads to Quicksort, ironically, performing very badly on already sorted (or almost sorted) arrays.

This is how most people choose to implement Quicksort and, since it's simple and this way of choosing the pivot is a very efficient operation (and we'll need to do it repeatedly), this is exactly what we will do.

Now that we have chosen a pivot - what do we do with it? Again, there are several ways of going about the partitioning itself. We will have a "pointer" to our pivot, and a pointer to the "smaller" elements and a pointer to the "larger" elements.

The goal is to move the elements around so that all elements smaller than the pivot are to its left, and all larger elements are to its right. The smaller and larger elements don't necessarily end up sorted, we just want them on the proper side of the pivot. We then recursively go through the left and right side of the pivot.

A step by step look at what we're planning to do will help illustrate the process. Using the array shown below, we've chosen the first element as the pivot (29), and the pointer to the smaller elements (called "low") starts right after, and the pointer to the larger elements (called "high") starts at the end.

• 29 is the first pivot, low points to 99 and high points to 44

29 | 99 (low),27,41,66,28,44,78,87,19,31,76,58,88,83,97,12,21,44 (high)

• We move high to the left until we find a value that's lower than our pivot.

29 | 99 (low),27,41,66,28,44,78,87,19,31,76,58,88,83,97,12,21 (high),44

• Now that our high variable is pointing to 21, an element smaller than the pivot, we want to find a value near the beginning of the array that we can swap it with. It doesn't make any sense to swap with a value that's also smaller than the pivot, so if low is pointing to a smaller element we try and find one that's larger.
• We move our low variable to the right until we find an element larger than the pivot. Luckily, low was already positioned on 99.
• We swap places of low and high:

29 | 21 (low),27,41,66,28,44,78,87,19,31,76,58,88,83,97,12,99 (high),44

• Right after we do this, we move high to the left and low to the right (since 21 and 99 are now in their correct places)
• Again, we move high to the left until we reach a value lower than the pivot, which we find right away - 12
• Now we search for a value larger than the pivot by moving low to the right, and we find the first such value at 41

This process is continued until the low and high pointers finally meet in a single element:

29 | 21,27,12,19,28 (low/high),44,78,87,66,31,76,58,88,83,97,12,99,44

• We've got no more use of this pivot so the only thing left to do is to swap pivot and high and we're done with this recursive step:

28,21,27,12,19,29,44,78,87,66,31,76,58,88,83,97,12,99,44

As you can see, we have achieved that all values smaller than 29 are now to the left of 29, and all values larger than 29 are to the right.

The algorithm then does the same thing for the 28,21,27,12,19 (left side) collection and the 44,78,87,66,31,76,58,88,83,97,12,99,44 (right side) collection.

Implementation

Sorting Arrays

Quicksort is a naturally recursive algorithm - divide the input array into smaller arrays, move the elements to the proper side of the pivot, and repeat.

Let's go through how a few recursive calls would look:

• When we first call the algorithm, we consider all of the elements - from indexes 0 to n-1 where n is the number of elements in our array.
• If our pivot ended up in position k, we'd then repeat the process for elements from 0 to k-1 and from k+1 to n-1.
• While sorting the elements from k+1 to n-1, the current pivot would end up in some position p. We'd then sort the elements from k+1 to p-1 and p+1 to n-1, and so on.

That being said, we'll utilize two functions - partition() and quick_sort(). The quick_sort() function will first partition() the collection and then recursively call itself on the divided parts.

Let's start off with the partition() function:

def partition(array, start, end):
    pivot = array[start]
    low = start + 1
    high = end

    while True:
        # If the current value we're looking at is larger than the pivot
        # it's in the right place (right side of pivot) and we can move left,
        # to the next element.
        # We also need to make sure we haven't surpassed the low pointer, since that
        # indicates we have already moved all the elements to their correct side of the pivot
        while low <= high and array[high] >= pivot:
            high = high - 1

        # Opposite process of the one above
        while low <= high and array[low] <= pivot:
            low = low + 1

        # We either found a value for both high and low that is out of order
        # or low is higher than high, in which case we exit the loop
        if low <= high:
            array[low], array[high] = array[high], array[low]
            # The loop continues
        else:
            # We exit out of the loop
            break

    array[start], array[high] = array[high], array[start]

    return high

And finally, let's implement the quick_sort() function:

def quick_sort(array, start, end):
    if start >= end:
        return

    p = partition(array, start, end)
    quick_sort(array, start, p-1)
    quick_sort(array, p+1, end)

With both of them implemented, we can run quick_sort() on a simple array:

array = [29,99,27,41,66,28,44,78,87,19,31,76,58,88,83,97,12,21,44]

quick_sort(array, 0, len(array) - 1)
print(array)

Output:

[12, 19, 21, 27, 28, 29, 31, 41, 44, 44, 58, 66, 76, 78, 83, 87, 88, 97, 99]

Since the algorithm is unstable, there's no guarantee that these two 44's were in this order to each other. Maybe there were originally switched - though this doesn't mean much in an integer array.

Sorting Custom Objects

There are a few ways you can rewrite this algorithm to sort custom objects in Python. A very Pythonic way would be to implement the comparison operators for a given class, which means that we wouldn't actually need to change the algorithm implementation since >, ==, <=, etc. would also work on our class object.

Another option would be to allow the caller to supply a method to our algorithm which would then be used to perform the actual comparison of the objects. Rewriting the algorithm in this way for use with custom objects is fairly straight-forward. Keep in mind however that the algorithm isn't stable.

Let's start off with a Person class:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return self.name

This is a pretty basic class with only two properties, name and age. We want to use age as our sorting key, which we'll do by providing a custom lambda function to the sorting algorithm.

But first, let's see how this provided function is used within the algorithm. Instead of doing a direct comparison with the <= or >= operators, we instead call the function to tell is which Person is higher in age:

def partition(array, start, end, compare_func):
    pivot = array[start]
    low = start + 1
    high = end

    while True:
        while low <= high and compare_fun(array[high], pivot):
            high = high - 1

        while low <= high and not compare_fun(array[low], pivot):
            low = low + 1

        if low <= high:
            array[low], array[high] = array[high], array[low]
        else:
            break

    array[start], array[high] = array[high], array[start]

    return high

def quick_sort(array, start, end, compare_func):
    if start >= end:
        return

    p = partition(array, start, end, compare_func)
    quick_sort(array, start, p-1, compare_func)
    quick_sort(array, p+1, end, compare_func)

And now, let's sort a collection of these objects. You can see that the object comparison is provided to the quick_sort call via a lambda, which does the actual comparison of the age property:

p1 = Person("Dave", 21)
p2 = Person("Jane", 58)
p3 = Person("Matthew", 43)
p4 = Person("Mike", 21)
p5 = Person("Tim", 10)

array = [p1,p2,p3,p4,p5]

quick_sort(array, 0, len(array) - 1, lambda x, y: x.age < y.age)
for person in array:
    print(person)

The output is:

Tim
Dave
Mike
Matthew
Jane

By implementing the algorithm in this way, it can be used with any custom object we choose, just as long as we provide an appropriate comparison function.

Optimizations of Quicksort

Given that Quicksort sorts "halves" of a given array independently, it's very convenient for parallelization. We can have a separate thread that sorts each "half" of the array, and we could ideally halve the time needed to sort it.

However, Quicksort can have a very deep recursive call stack if we are particularly unlucky in our choice of a pivot, and parallelization isn't as efficient as it is with Merge Sort.

It's recommended to use a simple, non-recursive algorithm for sorting small arrays. Even something simple like insertion sort is more efficient on small arrays than Quicksort. So ideally we could check whether our subarray has only a small number of elements (most recommendations say about 10 or less), and if so, we'd sort it with Insertion Sort instead.

A popular variation of Quicksort is the Multi-pivot Quicksort, which breaks up the original array into n smaller arrays, using n-1 pivots. However, most of the time only two pivots are used, not more.

Fun fact: Dual-pivot Quicksort, along with Insertion Sort for smaller arrays was used in Java 7's sorting implementation.

Conclusion

As we have previously mentioned, the efficiency of Quicksort depends highly on the choice of pivot - it can "make or break" the algorithm's time (and stack space) complexity. The instability of the algorithm is also something that can be a deal breaker when using custom objects.

However, despite all this, Quicksort's average time complexity of O(n*log_n) and its relatively low space-usage and simple implementation, make it a very efficient and popular algorithm.

If you want to learn more, check out our other article, Sorting Algorithms in Python, which covers more sorting algorithms in Python, but not as in-depth.

November 18, 2019 05:13 PM UTC

Codementor

Solving CAPTCHA with Web automation

All you need to know about CAPTCHA bypassing.

November 18, 2019 04:24 PM UTC

Python Software Foundation

Python Software Foundation Fellow Members for Q3 2019

We are happy to announce our newest PSF Fellow Members for Q3!

Q3 2019

November 18, 2019 02:35 PM UTC

Real Python

Pandas GroupBy: Your Guide to Grouping Data in Python

Whether you’ve just started working with Pandas and want to master one of its core facilities, or you’re looking to fill in some gaps in your understanding about .groupby(), this tutorial will help you to break down and visualize a Pandas GroupBy operation from start to finish.

This tutorial is meant to complement the official documentation, where you’ll see self-contained, bite-sized examples. Here, however, you’ll focus on three more involved walk-throughs that use real-world datasets.

In this tutorial, you’ll cover:

• How to use Pandas GroupBy operations on real-world data
• How the split-apply-combine chain of operations works
• How to decompose the split-apply-combine chain into steps
• How methods of a Pandas GroupBy object can be placed into different categories based on their intent and result

This tutorial assumes you have some experience with Pandas itself, including how to read CSV files into memory as Pandas objects with read_csv(). If you need a refresher, then check out Reading CSVs With Pandas.

You can download the source code for all the examples in this tutorial by clicking on the link below:

Housekeeping

All code in this tutorial was generated in a CPython 3.7.2 shell using Pandas 0.25.0. Before you proceed, make sure that you have the latest version of Pandas available within a new virtual environment:

$ python -m venv pandas-gb-tut
$ source ./pandas-gb-tut/bin/activate
$ python -m pip install pandas

The examples here also use a few tweaked Pandas options for friendlier output:

import pandas as pd

# Use 3 decimal places in output display
pd.set_option("display.precision", 3)

# Don't wrap repr(DataFrame) across additional lines
pd.set_option("display.expand_frame_repr", False)

# Set max rows displayed in output to 25
pd.set_option("display.max_rows", 25)

You can add these to a startup file to set them automatically each time you start up your interpreter.

In this tutorial, you’ll focus on three datasets:

1. The U.S. Congress dataset contains public information on historical members of Congress and illustrates several fundamental capabilities of .groupby().
2. The air quality dataset contains periodic gas sensor readings. This will allow you to work with floats and time series data.
3. The news aggregator dataset which holds metadata on several hundred thousand news articles. You’ll be working with strings and doing groupby-based text munging.

You can download the source code for all the examples in this tutorial by clicking on the link below:

Once you’ve downloaded the .zip, you can unzip it to your current directory:

$ unzip -q -d groupby-data groupby-data.zip

The -d option lets you extract the contents to a new folder:

./
│
└── groupby-data/
    │
    ├── legislators-historical.csv
    ├── airqual.csv
    └── news.csv

With that set up, you’re ready to jump in!

Example 1: U.S. Congress Dataset

You’ll jump right into things by dissecting a dataset of historical members of Congress. You can read the CSV file into a Pandas DataFrame with read_csv():

import pandas as pd

dtypes = {
    "first_name": "category",
    "gender": "category",
    "type": "category",
    "state": "category",
    "party": "category",
}
df = pd.read_csv(
    "groupby-data/legislators-historical.csv",
    dtype=dtypes,
    usecols=list(dtypes) + ["birthday", "last_name"],
    parse_dates=["birthday"]
)

The dataset contains members’ first and last names, birth date, gender, type ("rep" for House of Representatives or "sen" for Senate), U.S. state, and political party. You can use df.tail() to vie the last few rows of the dataset:

>>> df.tail()
      last_name first_name   birthday gender type state       party
11970   Garrett     Thomas 1972-03-27      M  rep    VA  Republican
11971    Handel      Karen 1962-04-18      F  rep    GA  Republican
11972     Jones     Brenda 1959-10-24      F  rep    MI    Democrat
11973    Marino        Tom 1952-08-15      M  rep    PA  Republican
11974     Jones     Walter 1943-02-10      M  rep    NC  Republican

The DataFrame uses categorical dtypes for space efficiency:

>>> df.dtypes
last_name             object
first_name          category
birthday      datetime64[ns]
gender              category
type                category
state               category
party               category
dtype: object

You can see that most columns of the dataset have the type category, which reduces the memory load on your machine.

The “Hello, World!” of Pandas GroupBy

Now that you’re familiar with the dataset, you’ll start with a “Hello, World!” for the Pandas GroupBy operation. What is the count of Congressional members, on a state-by-state basis, over the entire history of the dataset? In SQL, you could find this answer with a SELECT statement:

SELECT state, count(name)
FROM df
GROUP BY state
ORDER BY state;

Here’s the near-equivalent in Pandas:

>>> n_by_state = df.groupby("state")["last_name"].count()
>>> n_by_state.head(10)
state
AK     16
AL    206
AR    117
AS      2
AZ     48
CA    361
CO     90
CT    240
DC      2
DE     97
Name: last_name, dtype: int64

You call .groupby() and pass the name of the column you want to group on, which is "state". Then, you use ["last_name"] to specify the columns on which you want to perform the actual aggregation.

You can pass a lot more than just a single column name to .groupby() as the first argument. You can also specify any of the following:

• A list of multiple column names
• A dict or Pandas Series
• A NumPy array or Pandas Index, or an array-like iterable of these

Here’s an example of grouping jointly on two columns, which finds the count of Congressional members broken out by state and then by gender:

>>> df.groupby(["state", "gender"])["last_name"].count()
state  gender
AK     M          16
AL     F           3
       M         203
AR     F           5
       M         112
                ...
WI     M         196
WV     F           1
       M         119
WY     F           2
       M          38
Name: last_name, Length: 104, dtype: int64

The analogous SQL query would look like this:

SELECT state, gender, count(name)
FROM df
GROUP BY state, gender
ORDER BY state, gender;

As you’ll see next, .groupby() and the comparable SQL statements are close cousins, but they’re often not functionally identical.

Pandas GroupBy vs SQL

This is a good time to introduce one prominent difference between the Pandas GroupBy operation and the SQL query above. The result set of the SQL query contains three columns:

1. state
2. gender
3. count

In the Pandas version, the grouped-on columns are pushed into the MultiIndex of the resulting Series by default:

>>> n_by_state_gender = df.groupby(["state", "gender"])["last_name"].count()
>>> type(n_by_state_gender)
<class 'pandas.core.series.Series'>
>>> n_by_state_gender.index[:5]
MultiIndex([('AK', 'M'),
            ('AL', 'F'),
            ('AL', 'M'),
            ('AR', 'F'),
            ('AR', 'M')],
           names=['state', 'gender'])

To more closely emulate the SQL result and push the grouped-on columns back into columns in the result, you an use as_index=False:

>>> df.groupby(["state", "gender"], as_index=False)["last_name"].count()
    state gender  last_name
0      AK      F        NaN
1      AK      M       16.0
2      AL      F        3.0
3      AL      M      203.0
4      AR      F        5.0
..    ...    ...        ...
111    WI      M      196.0
112    WV      F        1.0
113    WV      M      119.0
114    WY      F        2.0
115    WY      M       38.0

[116 rows x 3 columns]

This produces a DataFrame with three columns and a RangeIndex, rather than a Series with a MultiIndex. In short, using as_index=False will make your result more closely mimic the default SQL output for a similar operation.

Also note that the SQL queries above explicitly use ORDER BY, whereas .groupby() does not. That’s because .groupby() does this by default through its parameter sort, which is True unless you tell it otherwise:

>>> # Don't sort results by the sort keys
>>> df.groupby("state", sort=False)["last_name"].count()
state
DE      97
VA     432
SC     251
MD     305
PA    1053
      ...
AK      16
PI      13
VI       4
GU       4
AS       2
Name: last_name, Length: 58, dtype: int64

Next, you’ll dive into the object that .groupby() actually produces.

How Pandas GroupBy Works

Before you get any further into the details, take a step back to look at .groupby() itself:

>>> by_state = df.groupby("state")
>>> print(by_state)
<pandas.core.groupby.generic.DataFrameGroupBy object at 0x107293278>

What is that DataFrameGroupBy thing? Its .__str__() doesn’t give you much information into what it actually is or how it works. The reason that a DataFrameGroupBy object can be difficult to wrap your head around is that it’s lazy in nature. It doesn’t really do any operations to produce a useful result until you say so.

One term that’s frequently used alongside .groupby() is split-apply-combine. This refers to a chain of three steps:

1. Split a table into groups
2. Apply some operations to each of those smaller tables
3. Combine the results

It can be difficult to inspect df.groupby("state") because it does virtually none of these things until you do something with the resulting object. Again, a Pandas GroupBy object is lazy. It delays virtually every part of the split-apply-combine process until you invoke a method on it.

So, how can you mentally separate the split, apply, and combine stages if you can’t see any of them happening in isolation? One useful way to inspect a Pandas GroupBy object and see the splitting in action is to iterate over it. This is implemented in DataFrameGroupBy.__iter__() and produces an iterator of (group, DataFrame) pairs for DataFrames:

>>> for state, frame in by_state:
...     print(f"First 2 entries for {state!r}")
...     print("------------------------")
...     print(frame.head(2), end="\n\n")
...
First 2 entries for 'AK'
------------------------
     last_name first_name   birthday gender type state        party
6619    Waskey      Frank 1875-04-20      M  rep    AK     Democrat
6647      Cale     Thomas 1848-09-17      M  rep    AK  Independent

First 2 entries for 'AL'
------------------------
    last_name first_name   birthday gender type state       party
912   Crowell       John 1780-09-18      M  rep    AL  Republican
991    Walker       John 1783-08-12      M  sen    AL  Republican

If you’re working on a challenging aggregation problem, then iterating over the Pandas GroupBy object can be a great way to visualize the split part of split-apply-combine.

There are a few other methods and properties that let you look into the individual groups and their splits. The .groups attribute will give you a dictionary of {group name: group label} pairs. For example, by_state is a dict with states as keys. Here’s the value for the "PA" key:

>>> by_state.groups["PA"]
Int64Index([    4,    19,    21,    27,    38,    57,    69,    76,    84,
               88,
            ...
            11842, 11866, 11875, 11877, 11887, 11891, 11932, 11945, 11959,
            11973],
           dtype='int64', length=1053)

Each value is a sequence of the index locations for the rows belonging to that particular group. In the output above, 4, 19, and 21 are the first indices in df at which the state equals “PA.”

You can also use .get_group() as a way to drill down to the sub-table from a single group:

>>> by_state.get_group("PA")
      last_name first_name   birthday gender type state                party
4        Clymer     George 1739-03-16      M  rep    PA                  NaN
19       Maclay    William 1737-07-20      M  sen    PA  Anti-Administration
21       Morris     Robert 1734-01-20      M  sen    PA   Pro-Administration
27      Wynkoop      Henry 1737-03-02      M  rep    PA                  NaN
38       Jacobs     Israel 1726-06-09      M  rep    PA                  NaN
...         ...        ...        ...    ...  ...   ...                  ...
11891     Brady     Robert 1945-04-07      M  rep    PA             Democrat
11932   Shuster       Bill 1961-01-10      M  rep    PA           Republican
11945   Rothfus      Keith 1962-04-25      M  rep    PA           Republican
11959  Costello       Ryan 1976-09-07      M  rep    PA           Republican
11973    Marino        Tom 1952-08-15      M  rep    PA           Republican

This is virtually equivalent to using .loc[]. You could get the same output with something like df.loc[df["state"] == "PA"].

Next, what about the apply part? You can think of this step of the process as applying the same operation (or callable) to every “sub-table” that is produced by the splitting stage. (I don’t know if “sub-table” is the technical term, but I haven’t found a better one 🤷‍♂️)

From the Pandas GroupBy object by_state, you can grab the initial U.S. state and DataFrame with next(). When you iterate over a Pandas GroupBy object, you’ll get pairs that you can unpack into two variables:

>>> state, frame = next(iter(by_state))  # First tuple from iterator
>>> state
'AK'
>>> frame.head(3)
     last_name first_name   birthday gender type state        party
6619    Waskey      Frank 1875-04-20      M  rep    AK     Democrat
6647      Cale     Thomas 1848-09-17      M  rep    AK  Independent
7442   Grigsby     George 1874-12-02      M  rep    AK          NaN

Now, think back to your original, full operation:

>>> df.groupby("state")["last_name"].count()
state
AK      16
AL     206
AR     117
AS       2
AZ      48
...

The apply stage, when applied to your single, subsetted DataFrame, would look like this:

>>> frame["last_name"].count()  # Count for state == 'AK'
16

You can see that the result, 16, matches the value for AK in the combined result.

The last step, combine, is the most self-explanatory. It simply takes the results of all of the applied operations on all of the sub-tables and combines them back together in an intuitive way.

Example 2: Air Quality Dataset

The air quality dataset contains hourly readings from a gas sensor device in Italy. Missing values are denoted with -200 in the CSV file. You can use read_csv() to combine two columns into a timestamp while using a subset of the other columns:

import pandas as pd

df = pd.read_csv(
    "groupby-data/airqual.csv",
    parse_dates=[["Date", "Time"]],
    na_values=[-200],
    usecols=["Date", "Time", "CO(GT)", "T", "RH", "AH"]
).rename(
    columns={
        "CO(GT)": "co",
        "Date_Time": "tstamp",
        "T": "temp_c",
        "RH": "rel_hum",
        "AH": "abs_hum",
    }
).set_index("tstamp")

This produces a DataFrame with a DatetimeIndex and four float columns:

>>> df.head()
                      co  temp_c  rel_hum  abs_hum
tstamp
2004-03-10 18:00:00  2.6    13.6     48.9    0.758
2004-03-10 19:00:00  2.0    13.3     47.7    0.726
2004-03-10 20:00:00  2.2    11.9     54.0    0.750
2004-03-10 21:00:00  2.2    11.0     60.0    0.787
2004-03-10 22:00:00  1.6    11.2     59.6    0.789

Here, co is that hour’s average carbon monoxide reading, while temp_c, rel_hum, and abs_hum are the average temperature in Celsius, relative humidity, and absolute humidity over that hour, respectively. The observations run from March 2004 through April 2005:

>>> df.index.min()
Timestamp('2004-03-10 18:00:00')
>>> df.index.max()
Timestamp('2005-04-04 14:00:00')

So far, you’ve grouped on columns by specifying their names as str, such as df.groupby("state"). But .groupby() is a whole lot more flexible than this! You’ll see how next.

Grouping on Derived Arrays

Earlier you saw that the first parameter to .groupby() can accept several different arguments:

• A column or list of columns
• A dict or Pandas Series
• A NumPy array or Pandas Index, or an array-like iterable of these

You can take advantage of the last option in order to group by the day of the week. You can use the index’s .day_name() to produce a Pandas Index of strings. Here are the first ten observations:

>>> day_names = df.index.day_name()
>>> type(day_names)
<class 'pandas.core.indexes.base.Index'>
>>> day_names[:10]
Index(['Wednesday', 'Wednesday', 'Wednesday', 'Wednesday', 'Wednesday',
       'Wednesday', 'Thursday', 'Thursday', 'Thursday', 'Thursday'],
      dtype='object', name='tstamp')

You can then take this object and use it as the .groupby() key. In Pandas-speak, day_names is array-like. It’s a one-dimensional sequence of labels.

Now, pass that object to .groupby() to find the average carbon monoxide ()co) reading by day of the week:

>>> df.groupby(day_names)["co"].mean()
tstamp
Friday       2.543
Monday       2.017
Saturday     1.861
Sunday       1.438
Thursday     2.456
Tuesday      2.382
Wednesday    2.401
Name: co, dtype: float64

The split-apply-combine process behaves largely the same as before, except that the splitting this time is done on an artificially-created column. This column doesn’t exist in the DataFrame itself, but rather is derived from it.

What if you wanted to group not just by day of the week, but by hour of the day? That result should have 7 * 24 = 168 observations. To accomplish that, you can pass a list of array-like objects. In this case, you’ll pass Pandas Int64Index objects:

>>> hr = df.index.hour
>>> df.groupby([day_names, hr])["co"].mean().rename_axis(["dow", "hr"])
dow        hr
Friday     0     1.936
           1     1.609
           2     1.172
           3     0.887
           4     0.823
                 ...
Wednesday  19    4.147
           20    3.845
           21    2.898
           22    2.102
           23    1.938
Name: co, Length: 168, dtype: float64

Here’s one more similar case that uses .cut() to bin the temperature values into discrete intervals:

>>> bins = pd.cut(df["temp_c"], bins=3, labels=("cool", "warm", "hot"))
>>> df[["rel_hum", "abs_hum"]].groupby(bins).agg(["mean", "median"])
       rel_hum        abs_hum
          mean median    mean median
temp_c
cool    57.651   59.2   0.666  0.658
warm    49.383   49.3   1.183  1.145
hot     24.994   24.1   1.293  1.274

In this case, bins is actually a Series:

>>> type(bins)
<class 'pandas.core.series.Series'>
>>> bins.head()
tstamp
2004-03-10 18:00:00    cool
2004-03-10 19:00:00    cool
2004-03-10 20:00:00    cool
2004-03-10 21:00:00    cool
2004-03-10 22:00:00    cool
Name: temp_c, dtype: category
Categories (3, object): [cool < warm < hot]

Whether it’s a Series, NumPy array, or list doesn’t matter. What’s important is that bins still serves as a sequence of labels, one of cool, warm, or hot. If you really wanted to, then you could also use a Categorical array or even a plain-old list:

• Native Python list: df.groupby(bins.tolist())
• Pandas Categorical array: df.groupby(bins.values)

As you can see, .groupby() is smart and can handle a lot of different input types. Any of these would produce the same result because all of them function as a sequence of labels on which to perform the grouping and splitting.

Resampling

You’ve grouped df by the day of the week with df.groupby(day_names)["co"].mean(). Now consider something different. What if you wanted to group by an observation’s year and quarter? Here’s one way to accomplish that:

>>> # See an easier alternative below
>>> df.groupby([df.index.year, df.index.quarter])["co"].agg(
...     ["max", "min"]
... ).rename_axis(["year", "quarter"])
               max  min
year quarter
2004 1         8.1  0.3
     2         7.3  0.1
     3         7.5  0.1
     4        11.9  0.1
2005 1         8.7  0.1
     2         5.0  0.3

This whole operation can, alternatively, be expressed through resampling. One of the uses of resampling is as a time-based groupby. All that you need to do is pass a frequency string, such as "Q" for "quarterly", and Pandas will do the rest:

>>> df.resample("Q")["co"].agg(["max", "min"])
             max  min
tstamp
2004-03-31   8.1  0.3
2004-06-30   7.3  0.1
2004-09-30   7.5  0.1
2004-12-31  11.9  0.1
2005-03-31   8.7  0.1
2005-06-30   5.0  0.3

Often, when you use .resample() you can express time-based grouping operations in a much more succinct manner. The result may be a tiny bit different than the more verbose .groupby() equivalent, but you’ll often find that .resample() gives you exactly what you’re looking for.

Example 3: News Aggregator Dataset

Now you’ll work with the third and final dataset, which holds metadata on several hundred thousand news articles and groups them into topic clusters:

import datetime as dt
import pandas as pd

def parse_millisecond_timestamp(ts: int) -> dt.datetime:
    """Convert ms since Unix epoch to UTC datetime instance."""
    return dt.datetime.fromtimestamp(ts / 1000, tz=dt.timezone.utc)

df = pd.read_csv(
    "groupby-data/news.csv",
    sep="\t",
    header=None,
    index_col=0,
    names=["title", "url", "outlet", "category", "cluster", "host", "tstamp"],
    parse_dates=["tstamp"],
    date_parser=parse_millisecond_timestamp,
    dtype={
        "outlet": "category",
        "category": "category",
        "cluster": "category",
        "host": "category",
    },
)

To read it into memory with the proper dyptes, you need a helper function to parse the timestamp column. This is because it’s expressed as the number of milliseconds since the Unix epoch, rather than fractional seconds, which is the convention. Similar to what you did before, you can use the Categorical dtype to efficiently encode columns that have a relatively small number of unique values relative to the column length.

Each row of the dataset contains the title, URL, publishing outlet’s name, and domain, as well as the publish timestamp. cluster is a random ID for the topic cluster to which an article belongs. category is the news category and contains the following options:

• b for business
• t for science and technology
• e for entertainment
• m for health

Here’s the first row:

>>> df.iloc[0]
title       Fed official says wea...
url         http://www.latimes.co...
outlet             Los Angeles Times
category                           b
cluster     ddUyU0VZz0BRneMioxUPQ...
host                 www.latimes.com
tstamp      2014-03-10 16:52:50.6...
Name: 1, dtype: object

Now that you’ve had a glimpse of the data, you can begin to ask more complex questions about it.

Using Lambda Functions in .groupby()

This dataset invites a lot more potentially involved questions. I’ll throw a random but meaningful one out there: which outlets talk most about the Federal Reserve? Let’s assume for simplicity that this entails searching for case-sensitive mentions of "Fed". Bear in mind that this may generate some false positives with terms like “Federal Government.”

To count mentions by outlet, you can call .groupby() on the outlet, and then quite literally .apply() a function on each group:

>>> df.groupby("outlet", sort=False)["title"].apply(
...     lambda ser: ser.str.contains("Fed").sum()
... ).nlargest(10)
outlet
Reuters                         161
NASDAQ                          103
Businessweek                     93
Investing.com                    66
Wall Street Journal \(blog\)     61
MarketWatch                      56
Moneynews                        55
Bloomberg                        53
GlobalPost                       51
Economic Times                   44
Name: title, dtype: int64

Let’s break this down since there are several method calls made in succession. Like before, you can pull out the first group and its corresponding Pandas object by taking the first tuple from the Pandas GroupBy iterator:

>>> title, ser = next(iter(df.groupby("outlet", sort=False)["title"]))
>>> title
'Los Angeles Times'
>>> ser.head()
1       Fed official says weak data caused by weather,...
486            Stocks fall on discouraging news from Asia
1124    Clues to Genghis Khan's rise, written in the r...
1146    Elephants distinguish human voices by sex, age...
1237    Honda splits Acura into its own division to re...
Name: title, dtype: object

In this case, ser is a Pandas Series rather than a DataFrame. That’s because you followed up the .groupby() call with ["title"]. This effectively selects that single column from each sub-table.

Next comes .str.contains("Fed"). This returns a Boolean Series that is True when an article title registers a match on the search. Sure enough, the first row starts with "Fed official says weak data caused by weather,..." and lights up as True:

>>> ser.str.contains("Fed")
1          True
486       False
1124      False
1146      False
1237      False
          ...
421547    False
421584    False
421972    False
422226    False
422905    False
Name: title, Length: 1976, dtype: bool

The next step is to .sum() this Series. Since bool is technically just a specialized type of int, you can sum a Series of True and False just as you would sum a sequence of 1 and 0:

>>> ser.str.contains("Fed").sum()
17

The result is the number of mentions of "Fed" by the Los Angeles Times in the dataset. The same routine gets applied for Reuters, NASDAQ, Businessweek, and the rest of the lot.

Improving the Performance of .groupby()

Let’s backtrack again to .groupby(...).apply() to see why this pattern can be suboptimal. To get some background information, check out How to Speed Up Your Pandas Projects. What may happen with .apply() is that it will effectively perform a Python loop over each group. While the .groupby(...).apply() pattern can provide some flexibility, it can also inhibit Pandas from otherwise using its Cython-based optimizations.

All that is to say that whenever you find yourself thinking about using .apply(), ask yourself if there’s a way to express the operation in a vectorized way. In that case, you can take advantage of the fact that .groupby() accepts not just one or more column names, but also many array-like structures:

• A 1-dimensional NumPy array
• A list
• A Pandas Series or Index

Also note that .groupby() is a valid instance method for a Series, not just a DataFrame, so you can essentially inverse the splitting logic. With that in mind, you can first construct a Series of Booleans that indicate whether or not the title contains "Fed":

>>> mentions_fed = df["title"].str.contains("Fed")
>>> type(mentions_fed)
<class 'pandas.core.series.Series'>

Now, .groupby() is also a method of Series, so you can group one Series on another:

>>> import numpy as np
>>> mentions_fed.groupby(
...     df["outlet"], sort=False
... ).sum().nlargest(10).astype(np.uintc)
outlet
Reuters                         161
NASDAQ                          103
Businessweek                     93
Investing.com                    66
Wall Street Journal \(blog\)     61
MarketWatch                      56
Moneynews                        55
Bloomberg                        53
GlobalPost                       51
Economic Times                   44
Name: title, dtype: uint32

The two Series don’t need to be columns of the same DataFrame object. They just need to be of the same shape:

>>> mentions_fed.shape
(422419,)
>>> df["outlet"].shape
(422419,)

Finally, you can cast the result back to an unsigned integer with np.uintc if you’re determined to get the most compact result possible. Here’s a head-to-head comparison of the two versions that will produce the same result:

# Version 1: using `.apply()`
df.groupby("outlet", sort=False)["title"].apply(
    lambda ser: ser.str.contains("Fed").sum()
).nlargest(10)

# Version 2: using vectorization
mentions_fed.groupby(
    df["outlet"], sort=False
).sum().nlargest(10).astype(np.uintc)

On my laptop, Version 1 takes 4.01 seconds, while Version 2 takes just 292 milliseconds. This is an impressive 14x difference in CPU time for a few hundred thousand rows. Consider how dramatic the difference becomes when your dataset grows to a few million rows!

Pandas GroupBy: Putting It All Together

If you call dir() on a Pandas GroupBy object, then you’ll see enough methods there to make your head spin! It can be hard to keep track of all of the functionality of a Pandas GroupBy object. One way to clear the fog is to compartmentalize the different methods into what they do and how they behave.

Broadly, methods of a Pandas GroupBy object fall into a handful of categories:

1. Aggregation methods (also called reduction methods) “smush” many data points into an aggregated statistic about those data points. An example is to take the sum, mean, or median of 10 numbers, where the result is just a single number.

2. Filter methods come back to you with a subset of the original DataFrame. This most commonly means using .filter() to drop entire groups based on some comparative statistic about that group and its sub-table. It also makes sense to include under this definition a number of methods that exclude particular rows from each group.

3. Transformation methods return a DataFrame with the same shape and indices as the original, but with different values. With both aggregation and filter methods, the resulting DataFrame will commonly be smaller in size than the input DataFrame. This is not true of a transformation, which transforms individual values themselves but retains the shape of the original DataFrame.

4. Meta methods are less concerned with the original object on which you called .groupby(), and more focused on giving you high-level information such as the number of groups and indices of those groups.

5. Plotting methods mimic the API of plotting for a Pandas Series or DataFrame, but typically break the output into multiple subplots.

The official documentation has its own explanation of these categories. They are, to some degree, open to interpretation, and this tutorial might diverge in slight ways in classifying which method falls where.

You can take a look at a more detailed breakdown of each category and the various methods of .groupby() that fall under them:

Conclusion

In this tutorial, you’ve covered a ton of ground on .groupby(), including its design, its API, and how to chain methods together to get data in an output that suits your purpose.

You’ve learned:

• How to use Pandas GroupBy operations on real-world data
• How the split-apply-combine chain of operations works and how you can decompose it into steps
• How methods of a Pandas GroupBy can be placed into different categories based on their intent and result

There is much more to .groupby() than you can cover in one tutorial. Check out the resources below and use the example datasets here as a starting point for further exploration!

You can download the source code for all the examples in this tutorial by clicking on the link below:

More Resources on Pandas GroupBy

Pandas documentation guides are user-friendly walk-throughs to different aspects of Pandas. Here are some portions of the documentation that you can check out to learn more about Pandas GroupBy:

• The Pandas GroupBy user guide
• The Grouping cookbook

The API documentation is a fuller technical reference to methods and objects:

• DataFrame.groupby()
• DataFrame.resample()
• pandas.Grouper

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 18, 2019 02:00 PM UTC

PyCon

Attention! Attention! Tutorial Proposal Deadline Approaching

Tutorial Proposal Deadline is this Friday, November 22, 2019

November 18, 2019 01:50 PM UTC

Python Software Foundation

Why Sponsor PyCon 2020?

• Playback and Recorded PyCon Coverage: Over 17,000 people have subscribed to PyCon’s YouTube channel with over 723,000 views of either the keynotes or the recorded videos of PyCon 2018 sessions. Those videos continue to attract viewers today.
• Conference Page and Announcement E-Lists: The PyCon home page has thousands of unique visitors every year and this year we’ll have an opt-in at signup for a newsletter to receive PyCon announcements.
• Social Media: The PSF and PyCon Twitter accounts have more than 332,000 followers combined and thousands of followers of the PyCon speakers.

November 18, 2019 12:58 PM UTC

Django Weblog

Introducing DjangoCon Africa

Following the huge success of PyCon Africa, the Django community in Africa is ready to bring a new major software event to the continent - the very first DjangoCon Africa! The Django Software Foundation is excited to endorse and support this initiative.

Plans are already in motion for a DjangoCon Africa to be held in Addis Ababa, Ethiopia in November 2020. Actual dates to be announced as soon as key details are in place.

DjangoCon Africa will include 3 days of single-track talks, 1 day of workshops and sprints, and another day for touring for international visitors.

The event will also include a Django Girls workshop to be held the weekend before DjangoCon Africa. To make the conference as inclusive as possible, the event will offer financial aid to members of under-represented communities in software to ensure they can also attend.

The CFP, which is open to all, will also be announced as soon as key details are in place.

About Ethiopia

Ethiopia is a country in North East of Africa, commonly known as the Horn of Africa. It is a country with a rich history and many historical places to visit. The country is highly accessible to all, with African Union members having the option of applying for visa on arrival at Bole International Airport or applying for an e-visa like the rest of the world before traveling to Ethiopia.

The country also boasts of the largest airline in the whole of Africa, with the country’s airline, Ethiopian Airlines having 53 routes in Africa, 17 in Europe, 7 in the Americas, 14 in Asia and 10 in the Middle East. This makes this country very accessible to all of Africa and the rest of the world and hence an ideal location for the first DjangoCon Africa.

See you in Addis Ababa in November 2020 for the first ever DjangoCon Africa!

November 18, 2019 12:25 PM UTC

PyBites

You can now hone your testing / pytest skills on our platform

Writing test code is an essential skill. As PyBites we believe writing code is the only solution to becoming a master (Ninja) at programming. The same applies to test code. For that reason we extended our regular exercises with Test Bites.

In this article you will read about the feature showcasing it on our first ever Test Bite. We also share some details around implementation and a challenge we hit getting it to work. Enjoy and start honing your testing skills today!

Why did we add this?

It was one of the most requested features. Period.

It is also a logical progression. So far, coding our 200+ Bites of Py you have looked at test code and analyzed its output when Bites did not pass. This is of course very useful, but it does not teach you how to write tests for your code. Many users expressed how cool it would be if they could learn how to code in pytest on our platform. Well, now you can! We just added 3 Test Bites - stay tuned for more soon ...

How to test test code?

This was a challenge, but somebody on Slack suggested mutation testing and a quick google search yielded MutPy:

MutPy is a mutation testing tool for Python 3.3+ source code. MutPy supports standard unittest module, generates YAML/HTML reports and has colorful output. It applies mutation on AST level. You could boost your mutation testing process with high order mutations (HOM) and code coverage analysis.

Here we run it manually on Bite 05:

    $ mut.py -h
    usage: mut.py [-h] [--version] [--target TARGET [TARGET ...]]
                [--unit-test UNIT_TEST [UNIT_TEST ...]] [--runner RUNNER]
                [--report REPORT_FILE] [--report-html DIR_NAME]
                [--timeout-factor TIMEOUT_FACTOR] [--show-mutants] [--quiet]
                [--debug] [--colored-output] [--disable-stdout]
                [--experimental-operators] [--operator OPERATOR [OPERATOR ...]]
                [--disable-operator OPERATOR [OPERATOR ...]] [--list-operators]
                [--path DIR] [--percentage PERCENTAGE] [--coverage]
                [--order ORDER] [--hom-strategy HOM_STRATEGY]
                [--list-hom-strategies] [--mutation-number MUTATION_NUMBER]

    $ mut.py --target names.py --unit-test test_names.py --runner pytest --coverage
    [*] Start mutation process:
    - targets: names.py
    - tests: test_names.py
    [*] 6 tests passed:
    - test_names [0.37057 s]
    [*] Start mutants generation and execution:
    - [#   1] CRP names: [0.13190 s] killed by test_names.py::test_sort_by_surname_desc
    - [#   2] CRP names: [0.09615 s] killed by test_names.py::test_sort_by_surname_desc
    - [#   3] CRP names: [0.09483 s] killed by test_names.py::test_sort_by_surname_desc
    - [#   4] CRP names: [0.09546 s] killed by test_names.py::test_sort_by_surname_desc
    - [#   5] CRP names: [0.09294 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#   6] CRP names: [0.09068 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#   7] CRP names: [0.08975 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#   8] CRP names: [0.09013 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#   9] CRP names: [0.12359 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  10] CRP names: [0.09420 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  11] CRP names: [0.09482 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  12] CRP names: [0.09506 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  13] CRP names: [0.09875 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  14] CRP names: [0.09385 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  15] CRP names: [0.09451 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  16] CRP names: [0.09528 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  17] CRP names: [0.09253 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  18] CRP names: [0.09329 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  19] CRP names: [0.09638 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  20] CRP names: [0.09449 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  21] CRP names: [0.09480 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  22] CRP names: [0.09092 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  23] CRP names: [0.09388 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  24] CRP names: [0.09356 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  25] CRP names: [0.09271 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  26] CRP names: [0.09341 s] killed by test_names.py::test_dedup_and_title_case_names
    - [#  27] CRP names: [0.09397 s] killed by test_names.py::test_sort_by_surname_desc
    - [#  28] CRP names: [0.10249 s] killed by test_names.py::test_shortest_first_name
    [*] Mutation score [8.05135 s]: 100.0%
    - all: 28
    - killed: 28 (100.0%)
    - survived: 0 (0.0%)
    - incompetent: 0 (0.0%)
    - timeout: 0 (0.0%)
    [*] Coverage: 113 of 113 AST nodes (100.0%)

To see the mutations it tried, run it with the -m switch:

    $ mut.py --target names.py --unit-test test_names.py --runner pytest --coverage -m
    [*] Start mutation process:
    - targets: names.py
    - tests: test_names.py
    [*] 6 tests passed:
    - test_names [0.32475 s]
    [*] Start mutants generation and execution:
    - [#   1] CRP names:
    --------------------------------------------------------------------------------
    -  1: NAMES = ['arnold schwarzenegger', 'alec baldwin', 'bob belderbos', \
    +  1: NAMES = ['mutpy', 'alec baldwin', 'bob belderbos', \
    2:     'julian sequeira', 'sandra bullock', 'keanu reeves', \
    3:     'julbob pybites', 'bob belderbos', 'julian sequeira', \
    4:     'al pacino', 'brad pitt', 'matt damon', 'brad pitt']
    5:
    --------------------------------------------------------------------------------
    [0.10319 s] killed by test_names.py::test_sort_by_surname_desc
    - [#   2] CRP names:
    --------------------------------------------------------------------------------
    -  1: NAMES = ['arnold schwarzenegger', 'alec baldwin', 'bob belderbos', \
    +  1: NAMES = ['', 'alec baldwin', 'bob belderbos', \
    2:     'julian sequeira', 'sandra bullock', 'keanu reeves', \
    3:     'julbob pybites', 'bob belderbos', 'julian sequeira', \
    4:     'al pacino', 'brad pitt', 'matt damon', 'brad pitt']
    5:
    --------------------------------------------------------------------------------
    [0.09149 s] killed by test_names.py::test_sort_by_surname_desc
    ...
    ... many more ...
    ...

Pretty cool eh? There is also a good explanation and simple example on MutPy's project page.

How it works on our platform

As previously mentioned, a Test Bite looks like a regular Bite but instead of a given set of tests, you are provided with a code module that gets imported. You are asked to write one or more test functions:

Let's write some code for this Bite and see the different checks (warning: contains spoilers, if you want to try it out now, go write code here!)

Bad syntax

It first checks if your test code runs. Here I forgot a colon after my function declaration:

If you click Show pytest output you can see where you went wrong:

Tests pass

Secondly it checks if the tests pass. Here I added a statement that does not pass:

And it doesn't like that:

Coverage

Next step is code coverage. Our initial thought was to use pytest-cov but that would mean two commands to run on the code. If possible it's preferable to use one tool. Luckily MutPy had us covered (pun intended). As long as we run it with the --coverage flag (see above).

The code so far has a coverage of 50%:

Mutation testing

At this point we also see MutPy's mutation test output that we included (using the -m switch). Again use Show pytest output to reveal the full output.

As you can see MutPy started the mutants generation and execution. Sounds like a game no? It tries to kill the mutants using your test code.

More in a bit. Scrolling down you see the coverage for which it uses Python's ast or Abstract Syntax Trees.

Deleting one of the lines we have written so far we see that not all mutants get killed. Scrolling down you see 2 survivors:

Here is a mutant our tests addressed:

And here are the two survivors we still need to address:

Adding assert fib(1) == 1 earlier kills those survivors.

Let's work on the 50% coverage percentage next. Here is another test function to test a higher number:

This brings the coverage up to 93%! There is still one scenario we did not write tests for, negative numbers:

I'll leave that as an exercise to the reader :)

Finally when you pass syntax + pytest run + 100% coverage + 100% mutation score you pass the Bite:

As we're still testing the waters of this feature / the MutPy module, minimum coverage and mutation required scores might go down a bit if the Test Bites become too challenging ...

Remember MutPy == multiprocessing

As elaborated in PyBites Code Challenges behind the scenes we run our platform on Heroku and use AWS Lambda for code execution.

One tough nut to crack was AWS's lack of parallel processing, more specifically multiprocessing.Queue that MutPy uses (if you try it, you'll get OSError: [Errno 38] Function not implemented).

Luckily thanks to this article replacing multiprocessing.Queue with multiprocessing.Pipe in the MutPy module got it to work 🎉

Just leaving this note here if somebody runs into this.

Start today!

We hope this will not only improve your pytest skills, but also how you think about writing test code.

We are putting all Test Bites into a new / dedicated learning path soon so stay tuned ...

As always have fun and let us know if you have any feedback on this new feature, preferably on Slack:

• For platform questions use the #codechallenges channel.

• For testing use the #testing and #pytest channels.

• To share your Pythonic wins (be it with this feature or any other thing you use Python for), hop into our #checkins-and-wins channel.

Special thanks to AJ, David, Martin and Harrison for testing out this feature!

Keep Calm and Code in Python!

-- Bob

November 18, 2019 11:21 AM UTC

Django Weblog

Django 3.0 release candidate 1 released

Django 3.0 release candidate 1 is the final opportunity for you to try out the raft of new features before Django 3.0 is released.

The release candidate stage marks the string freeze and the call for translators to submit translations. Provided no major bugs are discovered that can't be solved in the next two weeks, Django 3.0 will be released on or around December 2. Any delays will be communicated on the django-developers mailing list thread.

Please use this opportunity to help find and fix bugs (which should be reported to the issue tracker). You can grab a copy of the package from our downloads page or on PyPI.

The PGP key ID used for this release is Mariusz Felisiak: 2EF56372BA48CD1B.

November 18, 2019 08:44 AM UTC

Codementor

Cyber Discovery - What it is all about

November 18, 2019 07:33 AM UTC

Mike Driscoll

PyDev of the Week: Martin Uribe

The post PyDev of the Week: Martin Uribe appeared first on The Mouse Vs. The Python.

November 18, 2019 06:05 AM UTC

November 17, 2019

Codementor

How and why I built Sudoku Solver

Sudoku Solving algorithm

November 17, 2019 02:50 PM UTC

Ian Ozsvald

Training Courses for 2020 Q1 – Successful Data Science Projects & Software Engineering for Data Scientists

Early next year I run new iterations of two of my existing training courses for Pythonic Data Scientists:

• Successful Data Science Projects (Jan, 1 day)
• Software Engineering for Data Scientists (Feb, 2 day)

Successful Data Science Projects focuses on reducing uncertainty in a new data science project. We’ll look at the reasons why these projects can fail (and heck – this is research – they can and occasionally should fail), review ways to derisk a project with tools you’re probably not yet using, plan out a Project Specification for agreement with stakeholders and review techniques to make your team more highly performant overall.

“After attending the course I can identify and communicate to the project team and client the uncertainties of the project efficiently. I am using the techniques covered on the course to write project initiation documents and put in place the necessary processes to reduce uncertainty. The course was very engaging and I was very happy to learn from Ian’s experience to ensure a successful delivery on all future projects.” – Dani Papamaximou, Data Scientist at Arcadia

Software Engineering for Data Scientists is a 2 day course aimed at data scientists (perhaps from an academic background) who lack strong software engineering skills. We cover reviewing “bad Notebook code”, refactoring this code, using a standardised folder structure (with cookiecutter), adding unit tests and defensive Pandas tests along with checking how to introduce these techniques back into your team.

“Ian’s Software Engineering for Data Scientists course provides an excellent overview of best practices with focus on testing, debugging and general code maintenance. Ian has a wealth of experience and also makes sure to keep on top of the latest tools and libraries in the Data Science world. I would especially recommend the course to Data Science practitioners coming from an academic rather than software engineering background.” – LibertyGlobal Mirka

I am also thinking of introducing a High Performance Python course based on the updates coming to the 2nd edition of my High Performance Python book (for release April 2020). You’ll get details about this on my low-frequency email training list and if you have strong thoughts about this, please get in contact!

The post Training Courses for 2020 Q1 – Successful Data Science Projects & Software Engineering for Data Scientists appeared first on Entrepreneurial Geekiness.

November 17, 2019 11:36 AM UTC

Python Circle

The simplest explanation of Decorators in Python

The simplest explanation of Decorators in Python, decorator example in python, what is a decorator, simple example decorator, decorator for a newbie, python decorators, changing function behavior without changing content, decorators

November 17, 2019 10:45 AM UTC

ABlog for Sphinx

ABlog v0.10 released

November 17, 2019 12:00 AM UTC

November 16, 2019

Codementor

How to Extract Data from PDF to Excel

The Portable Document Format (PDF) is a file format developed by Adobe to present documents, including text formatting and images, in a manner independent of application software, hardware, and…

November 16, 2019 08:31 AM UTC

Weekly Python StackOverflow Report

(cciii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2019-11-16 07:05:12 GMT

1. Joining elements of a list if those elements are in between two whitespaces - [14/4]
2. Fast alternative for numpy.median.reduceat - [9/4]
3. Python dictionary with multiple keys pointing to same list in memory efficient way - [7/4]
4. Strange performance results -- loop vs list comprehension and zip() - [7/2]
5. merge two dataframes and add column level with names - [7/1]
6. How to address 'OSError: libc not found' raised on Gunicorn exec of Flask app inside Alpine docker container - [6/5]
7. Transform multiple categorical columns - [6/3]
8. Handling very very small numbers in Python - [6/2]
9. python list + empty numpy array = empty numpy array? - [6/1]
10. How can I fix this pytorch error on Windows? (ModuleNotFoundError: No module named 'torch') - [6/1]

November 16, 2019 07:05 AM UTC

November 15, 2019

NumFOCUS

2019 NumFOCUS Awards and New Contributor Recognition

The post 2019 NumFOCUS Awards and New Contributor Recognition appeared first on NumFOCUS.

November 15, 2019 07:30 PM UTC

Paolo Amoroso

Two Books About the Kivy GUI Framework

The Kivy Python GUI framework is intriguing.

Not only it’s cross-platform but also supports Android. Java is too verbose and low level for me and Kivy is an opportunity for developing native Android apps without leaving Python.

Outside of the Kivy project documentation, there are few third-party advanced tutorials that go in more depth than the official tutorials. So, before diving into the code of the Kivy demos, I wanted some books to explore more features and get a broader picture of the framework and what it can do.

I found two potentially interesting books: Building Android Apps in Python Using Kivy with Android Studio: With Pyjnius, Plyer, and Buildozer by Ahmed Fawzy Mohamed Gad (Apress, 2019), and Kivy - Interactive Applications and Games in Python - Second Edition by Roberto Ulloa (Packt, 2015).

I read both and here are my impressions.

Building Android Apps in Python Using Kivy with Android Studio

The cover of Building Android Apps in Python Using Kivy with Android Studio in Google Play Books on my Pixel 2 XL.

Kivy - Interactive Applications and Games in Python

The cover of Kivy - Interactive Applications and Games in Python in Google Play Books on my Pixel 2 XL.

November 15, 2019 02:48 PM UTC

Kushal Das

PoC to auto attach USB devices in Qubes

Here is PoC based on qubesadmin API which can auto attach USB devices to any VM as required. By default Qubes auto attaches any device to the sys-usb VM, that helps with bad/malware full USB devices. But, in special cases, we may want to select special devices to be auto attached to certain VMs. In this PoC example, we are attaching any USB storage device, but, we can add some checks to mark only selected devices (by adding more checks), or we can mark few vms where no device can be attached.

I would love to see what all magical ideas you all come up with. Have fun with the code.

Btw, you can execute it in dom0 by

python3 autoattach.py

November 15, 2019 09:48 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
  • Planet Python Brasil
  • Planet Python Indonesia
  • Planet Python Poland
• Python Libraries
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Control F'd"
  • "Fredrik Håård's Blaag"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • 4teamwork
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AWeber Engineering
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Al-Ahmadgaid Asaad
  • Albert Hopkins
  • Alec Munro
  • Alex Clark
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amit Saha
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Ankit Chandawala
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • AppNeta Blog
  • Armin Ronacher
  • Arnav Khare
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Atul Varma -- Toolness
  • Audrey Roy Greenfeld
  • Automating OSINT
  • Baiju Muthukadan
  • Bajusz Tamás
  • Balthazar Rouberol
  • BangPypers
  • BeDjango
  • Ben Bass
  • Ben Rousch
  • Ben Tappin
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • Bill Ward / AdminTome
  • BioPython News
  • Bit of Cheese
  • Blue Yonder Tech
  • BlueBream
  • Bojan Mihelac
  • Brad Lucas
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Harring
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catalin George Festila
  • Catherine Devlin
  • Checking and Sharing
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Barra
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • Christopher Lenz
  • Chuck Thier
  • ClusterHQ
  • Cobe.io
  • CodeGrades
  • CodeSnipers
  • Codementor
  • Coder's Cat
  • Coding Diet
  • Collin Winter
  • Continuum Analytics Blog
  • Continuum Analytics News
  • Continuum Blog
  • Core Software
  • Corey Goldberg
  • Corey Oordt
  • Cormoran Project
  • Craig Kerstiens
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Damián Avila
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Dariusz Suchojad
  • Data Community DC
  • Data School
  • DataCamp
  • Dataquest
  • Dave Beazley
  • David Goodger
  • David Grant
  • David J C Beach
  • David MacIver
  • David Malcolm
  • David Marte
  • David Stanek
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diego Garcia
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • DreamHost
  • Duncan McGreggor
  • Dusty Phillips
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eko S. Wibowo
  • Eli Bendersky
  • Eniram Ltd.
  • Enthought
  • Eray Özkural (examachine)
  • Eric Holscher
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Even Wiik Thomassen
  • Evennia
  • Experienced Django
  • Fabio Zadrozny
  • Fazal Majid
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • Fredrik Lundh
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Gabriel Pettier
  • Galvanize
  • Gary Bernhardt
  • Gary Wilson
  • Geek Scrap
  • Georg Brandl
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • GoDjango
  • Gocept Weblog
  • Godson Gera
  • Gonçalo Margalho
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Gridarrow
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Guilherme Toti
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hany Fahim
  • Hector Garcia
  • Heikki Toivonen
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • IO Digital Sec
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Intellimath blog
  • Ionel Cristian Maries
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Krstic
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • James Mills
  • Jan-Carel Brand
  • Janusworx
  • Jarrod Millman
  • Jason Meyers
  • Jason Webb
  • Jaysinh Shukla
  • Jean-Louis Fuchs
  • Jean-Paul Calderone
  • Jeethu Rao
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Knupp
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Jacobsen
  • John Paulett
  • Jon Parise
  • Jonathan Dobson
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan Street
  • Jorge Niedbalski
  • Jorge Puente Sarrín
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Juan Rodríguez Monti
  • Juho Vepsäläinen
  • Julien Danjou
  • Julien Palard
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kate Editor
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kracekumar Ramaraju
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • Laurence Rowe
  • Laurent Luce
  • Laurent Szyster
  • Learn PyQt
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lightning Fast Shop
  • Lintel Technologies
  • Lion Kimbro
  • Lionel Tan
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Macken
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marc Kerins
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Mark Mruss
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mathieu Virbel
  • Mats Kindahl
  • Matt Harrison
  • Matt Layman
  • Matt Reid
  • Matt Wilkes
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Mattias Brändström
  • Mauveweb
  • Max Ischenko
  • Michael Bayer
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael Nelson
  • Michael Sparks
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikeal Rogers
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Morten W Petersen
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Nadia Alramli
  • Naomi Ceder
  • Natan Zabkar
  • Nathan Lemoine
  • Nathan Piccini Data Science Dojo Blog
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Amorim
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • NumFOCUS
  • Obey the Testing Goat
  • Ofosos
  • Oliver Andrich
  • Omaha Python Users Group
  • Ondřej Čertík
  • OpenHatch Python posts
  • PSF GSoC students blogs
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patricio Paez
  • Patrick Kennedy
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • PayPal Engineering Blog
  • Pedro Lima
  • Petecode
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philip Semanchuk
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Pierre Carbonnelle
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programando Ciência
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyBites
  • PyCarolinas
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyCon Australia
  • PyCon Pune
  • PyCon.DE & PyData Karlsruhe
  • PyPy Development
  • PyTennessee
  • PyTexas
  • Pypix
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python Open Mike
  • Python Piedmont Triad User Group
  • Python Software Foundation
  • Python Sweetness
  • Python Testing Cookbook
  • Python User Groups
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • PythonThreads
  • Pythonicity
  • Pythonology
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Reinout van Rees
  • Rene Dudfield
  • Reuven Lerner
  • Richard Gomes
  • Richard Jones
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Zaremba
  • Roberto Alsina
  • Robin Parmar
  • Robin Wilson
  • Roche Compaan
  • Rodrigo Araúj
  • Roman Imankulov
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • Salim Fadhley
  • Salman Haq
  • Sandipan Dey
  • Sandro Tosi
  • Sayan Chowdhury
  • Scripting the web with Python
  • Sean McGrath
  • Selena Deckelmann
  • Semaphore Community
  • Senthil Kumaran
  • Server Density
  • Shannon -jj Behrens
  • ShiningPanda
  • Shopkick Tech Blog
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stacks
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Petrea
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Dower
  • Steve Holden
  • Steven Klass
  • Stijn Debrouwere
  • Stories in My Pocket
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Sumith - Blog about SymPy/Python
  • Swaroop C H
  • Swisscom ICT
  • Sylvain Hellegouarch
  • Talk Python to Me
  • Tarek Ziade
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Ted Leung
  • Tennessee Leeuwenburg
  • Terri Oda
  • Terry Jones
  • Terry Peppers
  • Test and Code
  • The Artificial Intelligence Cookbook
  • The Code Bits
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • The Three of Wands
  • TheAutomatic.net
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Thomi Richards
  • Tiago Montes
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim van der Linden
  • Tobias Ivarsson
  • Toes to the Edge
  • Tom Christie
  • Tomasz Ducin
  • Tomasz Früboes
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Two Bit Arcade
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinta Software
  • Virgil Dupras
  • Vladimir Iakolev
  • Vladimir Perić
  • Wallaroo Labs
  • Wallix
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Wichert Akkerman
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • William Thompson
  • Wingware Blog
  • Wingware News
  • Wraithan
  • Yaco
  • Yaniv Aknin
  • Yann Larrivée
  • Yasoob Khalid
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • codingdirectional
  • eGenix.com
  • eRambler
  • gamingdirectional
  • hypothesis.works articles
  • kdev-python
  • leftmouseclickin
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github
    or e-mail planet at python.org
    (note, responses can take up to a few days)