Planet Python

Last update: September 24, 2021 07:41 PM UTC

September 24, 2021

Zero to Mastery

Python Monthly 💻🐍 September 2021

22nd issue of Python Monthly! Read by 20,000+ Python developers every month. This monthly Python newsletter is focused on keeping you up to date with the industry and keeping your skills sharp, without wasting your valuable time.

September 24, 2021 07:40 PM UTC

PyCharm

PyCharm 2021.3 EAP Is Open!

Introducing the Early Access Program for PyCharm 2021.3, where you can try the new PyCharm features before they are officially released!

This program is designed to give you a chance to try all the improvements to PyCharm and provide actionable feedback that will help ensure that PyCharm users enjoy a smooth experience with the stable release. Don’t hesitate to share your thoughts on topics such as the overall product performance, the version’s look and feel, bugs, new features, and more.

Feel free to share your feedback on Twitter (by mentioning @pycharm)) or by reporting issues to our tracker!

Important! PyCharm EAP builds are not fully tested and might be unstable.

The Toolbox App is the simplest way to get the EAP builds and keep both your stable and EAP versions up to date. You can also manually download the EAP builds from our website.

DOWNLOAD PYCHARM 2021.3 EAP

Brand new Jupyter notebook experience

In case you missed it, our team has been working hard to improve the entire Jupyter notebook experience in PyCharm. This effort even led us to create a new IDE, DataSpell, which is designed specifically for data scientists. The great news is that we have started bundling some of DataSpell’s functionality in PyCharm Professional, starting with the redesigned UI for local Jupyter notebooks. So, if you are not a data scientist yourself but often work with data in your workflow, make sure to try it out in this EAP.

In addition to the improved UI, we’ve also made manipulating Jupyter notebook cells smoother, added support for all traditional shortcuts, made the output display crisp, and more!

Working with FastAPI: new project type

We’ve added support for FastAPI, a popular high-performance web framework for building APIs. You can now specifically create a FastAPI project. When creating a new project, just choose FastAPI from the list of suggested project types.

When you use the new project type, PyCharm will run Uvicorn, the recommended web server for FastAPI projects.

PyCharm will also create a dedicated run configuration so that you can run and debug your project.

In the run configuration pop-up, you can choose additional options for the Uvicorn server command line.

Alternatively, you can run a FastAPI project that has already been created by opening it with PyCharm and creating a FastAPI run configuration. PyCharm will detect your application in the file and run Uvicorn for you.

To test your HTTP endpoints, there is a “test.http” file that lets you send GET, POST, and other requests to your application.

Python 3.10 support: parameter specification variables (PEP 612)

Python 3.10 solves the problem with having annotation parameters of inner functions in generic classes and generic functions. You can now forward the parameter types of one function to another using typing.ParamSpec and typing.Concatenate.

If you are ready to try out this new functionality, PyCharm can help – we’ve already added support for parameter specification variables. Type checking and type inference are supported for annotation parameters of inner functions in generic classes and generic functions. You can also make use of quick documentation for those operators.

Reworked UI for the Debug tool window

In our pursuit of a clear and straightforward user interface, we have reworked the Debug tool window. Our goal was to make the layout feel more lightweight by placing the actions on one line. We have also added a permanent line for evaluating expressions or adding a watcher.

Shared indexes for the most popular Python packages

In PyCharm 2021.1 we introduced prebuilt indexes for the most popular Python interpreters, which made standard library indexing much faster. Now we are adding prebuilt indexes for the most popular packages on PyPI. This should further improve the indexing time for projects that use these packages when you open them in PyCharm for the first time.

Note that these prebuilt indexes are not included in the PyCharm installation build. During the indexing of each new project, PyCharm will run a check to identify whether prebuilt indexes are available for the packages used in the project. If prebuilt indexes are available for any of the packages, PyCharm will notify you and ask whether you’d like to download them. If you want the prebuilt indexes to be downloaded without receiving notifications each time PyCharm detects a relevant package, you can choose ”Always download” in the dialog.

Ready to join the EAP?

DOWNLOAD PYCHARM 2021.3 EAP

Some ground rules

• EAP builds are free to use and expire 30 days after the build date.
• You can install an EAP build side by side with your stable PyCharm version.
• These builds are not fully tested and can be unstable.
• our feedback is always welcome. Please use our issue tracker and make sure to mention your build version.

How to download

You can download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP. If you’re on Ubuntu 16.04 or later, you can use snaps to get PyCharm EAP and stay up to date.

The PyCharm team

September 24, 2021 05:01 PM UTC

Mike Driscoll

Python 101 – Assignment Expressions

Assignment expressions were added to Python in version 3.8. The general idea is that an assignment expression allows you to assign to variables within an expression.

The syntax for doing this is:

NAME := expr

This operator has been called the “walrus operator”, although their real name is “assignment expression”. Interestingly, the CPython internals also refer to them as “named expressions”.

You can read all about assignment expressions in PEP 572. Let’s find out how to use assignment expressions!

Using Assignment Expressions

Assignment expressions are still relatively rare. However, you need to know about assignment expressions because you will probably come across them from time to time. PEP 572 has some good examples of assignment expressions.

# Handle a matched regex
if (match := pattern.search(data)) is not None:
    ...

# A more explicit alternative to the 2-arg form of iter() invocation
while (value := read_next_item()) is not None:
    ...

# Share a subexpression between a comprehension filter clause and its output
filtered_data = [y for x in data if (y := f(x)) is not None]

In these 3 examples, you are creating a variable in the expression statement itself. The first example creates the variable match by assigning it the result of the regex pattern search. The second example assigns the variable value to the result of calling a function in the while loop’s expression. Finally, you assign the result of calling f(x) to the variable y inside of a list comprehension.

It would probably help to see the difference between code that doesn’t use an assignment expression and one that does. Here’s an example of reading a file in chunks:

with open(some_file) as file_obj:
    while True:
        chunk_size = 1024
        data = file_obj.read(chunk_size)
        if not data:
            break
        if 'something' in data:
            # process the data somehow here

This code will open up a file of indeterminate size and process it 1024 bytes at a time. You will find this useful when working with very large files as it prevents you from loading the entire file into memory. If you do, you can run out of memory and cause your application or even the computer to crash.

You can shorten this code up a bit by using an assignment expression:

with open(some_file) as file_obj:
    chunk_size = 1024
    while data := file_obj.read(chunk_size):
        if 'something' in data:
            # process the data somehow here

Here you assign the result of the read() to data within the while loop’s expression. This allows you to then use that variable inside of the while loop’s code block. It also checks that some data was returned so you don’t have to have the if not data: break stanza.

Another good example that is mentioned in PEP 572 is taken from Python’s own site.py. Here’s how the code was originally:

env_base = os.environ.get("PYTHONUSERBASE", None)
if env_base:
    return env_base

And this is how it could be simplified by using an assignment expression:

if env_base := os.environ.get("PYTHONUSERBASE", None):
    return env_base

You move the assignment into the conditional statement’s expression, which shortens the code up nicely.

Now let’s discover some of the situations where assignment expressions can’t be used.

What You Cannot Do With Assignment Expressions

There are several cases where assignment expressions cannot be used.

One of the most interesting features of assignment expressions is that they can be used in contexts that an assignment statement cannot, such as in a lambda or the previously mentioned comprehension. However, they do NOT support some things that assignment statements can do. For example, you cannot do multiple target assignment:

x = y = z = 0  # Equivalent, but non-working: (x := (y := (z := 0)))

Another prohibited use case is using an assignment expression at the top level of an expression statement. Here is an example from PEP 572:

y := f(x)  # INVALID
(y := f(x))  # Valid, though not recommended

There is a detailed list of other cases where assignment expressions are prohibited or discouraged in the PEP. You should check that document out if you plan to use assignment expressions often.

Wrapping Up

Assignment expressions are an elegant way to clean up certain parts of your code. The feature’s syntax is kind of similar to type hinting a variable. Once you have the hang of one, the other should become easier to do as well.

In this article, you saw some real-world examples of using the “walrus operator”. You also learned when assignment expressions shouldn’t be used. This syntax is only available in Python 3.8 or newer, so if you happen to be forced to use an older version of Python, this feature won’t be of much use to you.

Related Reading

This article is based on a chapter from Python 101, 2nd Edition, which you can purchase on Leanpub or Amazon.

If you’d like to learn more Python, then check out these tutorials:

• Python 101 – How to Work with Images

• Python 101 – Documenting Your Code

• Python 101: An Intro to Working with JSON

• Python 101 – Creating Multiple Processes

The post Python 101 – Assignment Expressions appeared first on Mouse Vs Python.

September 24, 2021 12:41 PM UTC

Real Python

The Real Python Podcast – Episode #79: Measuring Your Python Learning Progress

Where are you along the path of learning Python? Do you feel like you're making progress? What are ways you can put the learning path into a more precise focus? This week on the show, we talk with previous guest Martin Breuss about his recent article "How Long Does It Take to Learn Python?"

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

September 24, 2021 12:00 PM UTC

Codementor

Debug Flask App in Docker with PyCharm

In PyCharm we can also debug flask application inside docker as well. So we don’t need to run project locally with different configuration. Which makes development process speedy.

September 24, 2021 04:19 AM UTC

Kushal Das

Rust, reproducibility and shadow-rs

Generally all of our Rust code are reproducible. If you build it in a fixed path, and also use SOURCE_DATE_EPOCH environment variable, the final library or executables will be producible. This is really helpful, for example while building cryptography python wheel, I can keep building it in a reproducible way even with the Rust dependencies.

A few days ago I saw shadow-rs, which can provide a lot of build time information. For example, in khata now I have a way to tell if I am using any custom build and also identify which one. I was a bit worried as shadow allows to store the build time too, but later found that the community already put in the patches so that it follows SOURCE_DATE_EPOCH.

So, I can still have reproducibility if I use the same build toolchain and environment while depending on shadow. Here is some example code for you to play around.

September 24, 2021 03:11 AM UTC

September 23, 2021

Python for Beginners

How to create an iterator in Python

Iterators are used to access the elements of an iterable object in a sequential manner. We can create an iterator for any container object such as a python dictionary, list, tuple, or a set. In this article, we will discuss how to create an iterator in python. We will also learn how to create custom iterators by overriding inbuilt methods.

Create iterator using inbuilt methods

We can create an iterator using the iter() function and the __next__() method. The iter() function takes a container object such as list, tuple, or set and returns an iterator with which we can access the elements of the container object.

To create an iterator for any container object using the iter() function, we just have to pass the object to the iter() function. The function creates an iterator and returns a reference to it. We can create an iterator using the iter() function as follows.

myList = [1, 2, 3, 4, 5, 6, 7]
myIter = iter(myList)
print("The list is:", myList)
print("The iterator is:", myIter)

Output:

The list is: [1, 2, 3, 4, 5, 6, 7]
The iterator is: <list_iterator object at 0x7f73fed18070>

In the output, you can observe that a list_iterator object has been created upon the execution of the iter() function.

How to access elements from an iterator?

To access the elements of the container object using iterators in a sequence, we can use a for loop as follows.

myList = [1, 2, 3, 4, 5, 6, 7]
myIter = iter(myList)
print("The list is:", myList)
print("The elements in the iterator are:")
for i in myIter:
    print(i)

Output:

The list is: [1, 2, 3, 4, 5, 6, 7]
The elements in the iterator are:
1
2
3
4
5
6
7

If you need to access the elements one by one, you can use the next() function or the __next() method. 

To traverse the iterator using next() function, we pass the iterator as an input argument to the function. It returns the next element in the iterator. The next() function also remembers the index at which the iterator had been traversed last time. When it is called again, it returns the next element which has not been traversed yet. This can be observed in the following example.

myList = [1, 2, 3, 4, 5, 6, 7]
myIter = iter(myList)
print("The list is:", myList)
print("The elements in the iterator are:")
element = next(myIter)
print(element)
element = next(myIter)
print(element)
element = next(myIter)
print(element)
element = next(myIter)
print(element)
element = next(myIter)
print(element)
element = next(myIter)
print(element)
element = next(myIter)
print(element)

Output:

The list is: [1, 2, 3, 4, 5, 6, 7]
The elements in the iterator are:
1
2
3
4
5
6
7

Process finished with exit code 0

The __next__() method works in a similar way as the next() function. Whenever the __next__() method is invoked on the iterator, it returns the next element that has not been traversed yet. 

myList = [1, 2, 3, 4, 5, 6, 7]
myIter = iter(myList)
print("The list is:", myList)
print("The elements in the iterator are:")
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)

Output:

The list is: [1, 2, 3, 4, 5, 6, 7]
The elements in the iterator are:
1
2
3
4
5
6
7

Process finished with exit code 0

When there is no element left for traversing and we use the next() function or the __next__() method, it raises the StopIteration exception.

myList = [1, 2, 3, 4, 5, 6, 7]
myIter = iter(myList)
print("The list is:", myList)
print("The elements in the iterator are:")
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)

Output:

Traceback (most recent call last):
  File "/home/aditya1117/PycharmProjects/pythonProject/webscraping.py", line 19, in <module>
    element = myIter.__next__()
StopIteration

It is advised to use these functions inside a python try except block to avoid the exception.

How to create a custom iterator in  Python

To create a custom iterator, we can override the __iter__() and __next__() methods that are set to default. We will understand how to create a custom iterator using the following example.

Suppose that you want to create an iterator that returns the square of each element of  a list or tuple when we iterate through the object using  the iterator. For this, we will override the __iter__() method and __next__() method. 

The constructor of the iterator will accept the list or tuple and the total number of elements in it. Then it will initialize the iterator class. As we need to keep track of the last element which was traversed, we will initialize an index field and set it to 0.

class SquareIterator:
    def __init__(self, data, noOfElements):
        self.data = data
        self.noOfElements = noOfElements
        self.count = 0

   

The __iter__() method is used to initialise an iterator. The __iter__() method is implemented as follows.

class SquareIterator:
    def __init__(self, data, noOfElements):
        self.data = data
        self.noOfElements = noOfElements
        self.count = 0

    def __iter__(self):
        return self

   

After overriding the __iter__() method, we will override the __next__() method. The __next__() method is used to traverse the next element that has not been traversed yet. Here, we need to return the square of the next element. 

As we have initialized the index to -1 in the __iter__() method, we will first increment the index. After that, we will return the square of the element at the specified index. Each time the __next__() method is invoked, the index will be incremented and the square of the element at the specified index will be given as output. When the number of traversed elements will equal the total number of elements in the list or tuple, we will raise the StopIteration exception. It will stop the iteration. The __next__() method can be defined as follows.

class SquareIterator:
    def __init__(self, data, noOfElements):
        self.data = data
        self.noOfElements = noOfElements
        self.count = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.count < self.noOfElements:
            square = self.data[self.count] ** 2
            self.count = self.count + 1
            return square
        else:
            raise StopIteration

The entire program to create the iterator is as follows.

class SquareIterator:
    def __init__(self, data, noOfElements):
        self.data = data
        self.noOfElements = noOfElements
        self.count = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.count < self.noOfElements:
            square = self.data[self.count] ** 2
            self.count = self.count + 1
            return square
        else:
            raise StopIteration


myList = [1, 2, 3, 4, 5, 6, 7]
myIter = SquareIterator(myList, 7)
print("The list is:", myList)
print("The elements in the iterator are:")
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)
element = myIter.__next__()
print(element)

Output:

The list is: [1, 2, 3, 4, 5, 6, 7]
The elements in the iterator are:
1
4
9
16
25
36
49

Process finished with exit code 0

Conclusion

In this article, we have studied two ways to create an iterator in Python. You can use custom iterators to perform different operations on the elements of a container object while traversing it. It is advised to implement some programs to create an iterator so that you can acquire a better understanding of the topic.

The post How to create an iterator in Python appeared first on PythonForBeginners.com.

September 23, 2021 01:13 PM UTC

Mike Driscoll

Python 101 – How to Create a Python Package

When you create a Python file, you are creating a Python module. Any Python file that you create can be imported by another Python script. Thus, by definition, it is also a Python module. If you have two or more related Python files, then you may have a Python package.

Some organizations keep all their code to themselves. This is known as closed-source. Python is an open-source language and most of the Python modules and packages that you can get from the Python Package Index (PyPI) are all free and open-source as well. One of the quickest ways to share your package or module is to upload it to the Python Package Index or Github or both.

In this article, you will learn about the following topics:

• Creating a Module
• Creating a Package
• Packaging a Project for PyPI
• Creating Project Files
• Creating setup.py
• Uploading to PyPI

The first step in the process is to understand what creating a reusable module looks like. Let’s get started!

Creating a Module

Any Python file you create is a module that you can import. You can try it out with some of the examples from this book by adding a new file to any of the articles’ code folders and attempt to import one of the modules in there. For example, if you have a Python file named a.py and then create a new file named b.py, you can import a into b through the use of import a.

Of course, that’s a silly example. Instead, you will create a simple module that has some basic arithmetic functions in it. You can name the file arithmetic.py and add this code to it:

# arithmetic.py

def add(x, y):
    return x + y

def divide(x, y):
    return x / y

def multiply(x, y):
    return x * y

def subtract(x, y):
    return x - y

This code is very naive. You have no error handling at all, for example. What that means is that you could divide by zero and cause an exception to be thrown. You could also pass incompatible types to these functions, like a string and an integer — that would cause a different kind of exception to be raised.

However, for learning purposes, this code is adequate. You can prove that it is importable by creating a test file. Create a new file named test_arithmetic.py and add this code to it:

# test_arithmetic.py

import arithmetic
import unittest

class TestArithmetic(unittest.TestCase):

    def test_addition(self):
        self.assertEqual(arithmetic.add(1, 2), 3)

    def test_subtraction(self):
        self.assertEqual(arithmetic.subtract(2, 1), 1)

    def test_multiplication(self):
        self.assertEqual(arithmetic.multiply(5, 5), 25)

    def test_division(self):
        self.assertEqual(arithmetic.divide(8, 2), 4)

if __name__ == '__main__':
    unittest.main()

Save your test in the same folder as your module. Now you can run this code using the following command:

$ python3 test_arithmetic.py 
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

This demonstrates that you can import arithmetic.py as a module. These tests also show the basic functionality of the code works. You can enhance these tests by testing division by zero and mixing strings and integers. Those kinds of tests will currently fail. Once you have a failing test, you can follow the Test Driven Development methodology to fix the issues.

Now let’s find out how to make a Python package!

Creating a Package

A Python package is one or more files that you plan on sharing with others, usually by uploading it to the Python Package Index (PyPI). Packages are generally made by naming a directory of files rather than a file itself. Then inside of that directory you will have a special __init__.py file. When Python sees the __init__.py file, it knows that the folder is importable as a package.

There are a couple ways to transform arithmetic.py into a package. The simplest is to move the code from arithmetic.py into arithmetic/__init__.py:

• create the folder arithmetic
• move/copy arithmetic.py to arithmetic/__init__.py
• if you used “copy” in the previous step then delete arithmetic.py
• run test_arithmetic.py

That last step is extremely important! If your tests still pass then you know your conversion from a module to a package worked. To test out your package, open up a Command Prompt if you’re on Windows or a terminal if you’re on Mac or Linux. Then navigate to the folder that contains the arithmetic folder, but not inside of it. You should now be in the same folder as your test_arithmetic.py file. At this point you can run python test_arithmetic.py and see if your efforts were successful.

It might seem silly to simply put all your code in a single __init__.py file, but that actually works fine for files up to a few thousand lines.

The second way to transform arithmetic.py into a package is similar to the first, but involves using more files than just __init__.py. In real code the functions/classes/etc. in each file would be grouped somehow — perhaps one file for all your package’s custom exceptions, one file for common utilities, and one file for the main functionality.

For our example, you’ll just split the four functions in arithmetic.py into their own files. Go ahead and move each function from __init__.py into its own file. Your folder structure should look like this:

arithmetic/
    __init__.py
    add.py
    subtract.py
    multiply.py
    divide.py

For the __init__.py file, you can add the following code:

# __init__.py
from .add import add
from .subtract import subtract
from .multiply import multiply
from .divide import divide

Now that you’ve made these changes what should be your next step? Hopefully, you said, “Run my tests!” If your tests still pass then you haven’t broken your API.

Right now your arithmetic package is only available to your other Python code if you happen to be in the same folder as your test_arithmetic.py file. To make it available in your Python session or in other Python code you can use Python’s sys module to add your package to the Python search path. The search path is what Python uses to find modules when you use the import keyword. You can see what paths Python searches by printing out sys.path.

Let’s pretend that your arithmetic folder is in this location: /Users/michael/packages/arithmetic. To add that to Python’s search path, you can do this:

import sys

sys.path.append("/Users/michael/packages/arithmetic")
import arithmetic

print(arithmetic.add(1, 2))

This will add arithmetic to Python’s path so you can import it and then use the package in your code. However, that’s really awkward. It would be nice if you could install your package using pip so you don’t have to mess around with the path all the time.

Let’s find out how to do that next!

Packaging a Project for PyPI

When it comes to creating a package for the Python Package Index (PyPI), you will need some additional files. There is a good tutorial on the process for creating and uploading a package to PyPI here:

• https://packaging.python.org/tutorials/packaging-projects/

The official packaging instructions recommend that you set up a directory structure like this:

my_package/
    LICENSE
    README.md
    arithmetic/
        __init__.py
        add.py
        subtract.py
        multiply.py
        divide.py
    setup.py
    tests/

The tests folder can be empty. This is the folder where you would include tests for your package. Most developers use Python’s unittest or the pytest framework for their tests. For this example, you can leave the folder empty.

Let’s move on and learn about the other files you need to create in the next section!

Creating Project Files

The LICENSE file is where you mention what license your package has. This tells the users of the package what they can and cannot do with your package. There are a lot of different licenses you can use. The GPL and MIT licenses are just a couple of popular examples.

The README.md file is a description of your project, written in Markdown. You will want to write about your project in this file and include any information about dependencies that it might need. You can give instructions for installation as well as example usage of your package. Markdown is quite versatile and even let’s you do syntax highlighting!

The other file you need to supply is setup.py. That file is more complex, so you’ll learn about that in the next section.

Creating setup.py

There is a special file named setup.py that is used as a build script for Python distributions. It is used by setuptools, which does the actual building for you. If you’d like to know more about setuptools, then you should check out the following:

• https://setuptools.readthedocs.io/en/latest/

You can use the setup.py to create a Python wheel. The wheel is a ZIP-format archive with a specially formatted name and a .whl extension. It contains everything necessary to install your package. You can think of it as a zipped version of your code that pip can unzip and install for you. The wheel follows PEP 376, which you can read about here:

• https://www.python.org/dev/peps/pep-0376/

Once you’re done reading all that documentation (if you wanted to), you can create your setup.py and add this code to it:

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="arithmetic-YOUR-USERNAME-HERE", # Replace with your own username
    version="0.0.1",
    author="Mike Driscoll",
    author_email="driscoll@example.com",
    description="A simple arithmetic package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/driscollis/arithmetic",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)

The first step in this code is to import setuptools. Then you read in your README.md file into a variable that you will use soon. The last bit is the bulk of the code. Here you call setuptools.setup(), which can take in quite a few different arguments. The example above is only a sampling of what you can pass to this function. To see the full listing, you’ll need to go here:

• https://packaging.python.org/guides/distributing-packages-using-setuptools/

Most of the arguments are self-explanatory. Let’s focus on the more obtuse ones. The packages arguments is a list of packages needed for your package. In this case, you use find_packages() to find the necessary packages for you automatically. The classifiers argument is used for passing additional metadata to pip. For example, this code tells pip that the package is Python 3 compatible.

Now that you have a setup.py, you are ready to create a Python wheel!

Generating a Python Wheel

The setup.py is used to create Python wheels. It’s always a good idea to make sure you have the latest version of setuptools and wheel installed, so before you create your own wheel, you should run the following command:

python3 -m pip install --user --upgrade setuptools wheel

This will update the packages if there is a newer version than the one you currently have installed. Now you are ready to create a wheel yourself. Open up a Command Prompt or terminal application and navigate to the folder that contains your setup.py file. Then run the following command:

python3 setup.py sdist bdist_wheel

This command will output a lot of text, but once it has finished you will find a new folder named dist that contains the following two files:

• arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
• arithmetic-YOUR-USERNAME-HERE-0.0.1.tar.gz

The tar.gz is a source archive, which means it has the Python source code for your package inside of it. Your users can use the source archive to build the package on their own machines, if they need to. The whl format is an archive that is used by pip to install your package on your user’s machine.

You can install the wheel using pip directly, if you want to:

python3 -m pip install arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl

But the normal method would be to upload your package to the Python Package Index (PyPI) and then install it. Let’s discover how to get your amazing package on PyPI next!

Uploading to PyPI

The first step to upload a package to PyPI is to create an account on Test PyPI. This allows you to test that your package can be uploaded on a test server and installed from that test server. To create an account, go to the following URL and follow the steps on that page:

• https://test.pypi.org/account/register/

Now you need to create a PyPI API token. This will allow you to upload the package securely. To get the API token, you’ll need to go here:

• https://test.pypi.org/manage/account/#api-tokens

You can limit a token’s scope. However, you don’t need to do that for this token as you are creating it for a new project. Make sure you copy the token and save it off somewhere BEFORE you close the page. Once the page is closed, you cannot retrieve the token again. You will be required to create a new token instead.

Now that you are registered and have an API token, you will need to get the twine package. You will use twine to upload your package to PyPI. To install twine, you can use pip like this:

python3 -m pip install --user --upgrade twine

Once installed, you can upload your package to Test PyPI using the following command:

python3 -m twine upload --repository testpypi dist/*

Note that you will need to run this command from within the folder that contains the setup.py file as it is copying all the files in the dist folder to Test PyPI. When you run this command, it will prompt you for a username and password. For the username, you need to use __token__. The password is the token value that is prefixed with pypi-.

When this command runs, you should see output similar to the following:

Uploading distributions to https://test.pypi.org/legacy/
Enter your username: [your username]
Enter your password:
Uploading arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
100%|?????????????????????| 4.65k/4.65k [00:01<00:00, 2.88kB/s]
Uploading arithmetic_YOUR_USERNAME_HERE-0.0.1.tar.gz
100%|?????????????????????| 4.25k/4.25k [00:01<00:00, 3.05kB/s]

At this point, you should now be able to view your package on Test PyPI at the following URL:

• https://test.pypi.org/project/arithmetic_YOUR_USERNAME_HERE

Now you can test installing your package from Test PyPI by using the following command:

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps arithmetic-YOUR-USERNAME-HERE

If everything worked correctly, you should now have the arithmetic package installed on your system. Of course, this tutorial showed you how to package things up for Test PyPI. Once you have verified that it works, then you will need to do the following to install to the real PyPI:

• Choose a memorable and unique name for the package
• Register an account at https://pypi.org
• Use twine upload dist/* to upload your package and enter your credentials for the account you registered on the real PyPI. You won’t need to use the --repository flag when uploading to the real PyPI as that server is the default
• Install your package from the real PyPI by using pip install your_unique_package_name

Now you know how to distribute a package of your own creation on the Python Package Index!

Wrapping Up

Python modules and packages are what you import in your programs. They are, in many ways, the building blocks of your programs. In this article, you learned about the following:

• Creating a Module
• Creating a Package
• Packaging a Project for PyPI
• Creating Project Files
• Creating setup.py
• Uploading to PyPI

At this point, you not only know what modules and packages are, but also how to distribute them via the Python Package Index. Now you and any other Python developer can download and install your packages. Congratulations! You are now a package maintainer!

Related Reading

This article is based on a chapter from Python 101, 2nd Edition, which you can purchase on Leanpub or Amazon.

If you’d like to learn more Python, then check out these tutorials:

• Python 101 – How to Work with Images

• Python 101 – Documenting Your Code

• Python 101: An Intro to Working with JSON

• Python 101 – Creating Multiple Processes

The post Python 101 – How to Create a Python Package appeared first on Mouse Vs Python.

September 23, 2021 12:30 PM UTC

EuroPython Society

List of EPS Board Candidates for 2021/2022

At this year’s EuroPython Society General Assembly (GA), planned for October, we will vote in a new board of the EuroPython Society for the term 2021/2022.

List of Board Candidates

The EPS bylaws require one chair, one vice chair and 2 - 7 board members. The following candidates have stated their willingness to work on the EPS board. We are presenting them here (in alphabetical order by surname).

We will be updating this list in the days before the GA. Please send in any nominations or self-nominations to board@europython.eu.

Please note that our bylaws do not restrict nominations to people on this list. It is even possible to self-nominate or nominate other candidates at the GA itself. However, in the interest of giving members a better chance to review the candidate list, we’d like to encourage all nominations to be made before the GA.

The following fine folks have expressed their desire to run as a group for the next EPS board elections: Patrick Arminio, Prof. Martin Christen, Nicolás Demarchi, Raquel Dou, Anders Hammarquist, Cheukting Ho, Francesco Pierfederici and Silvia Uberti. They have a good track record of working well together and share a common vision for the betterment of the EPS via strengthening ties to the larger Python community. They will present their plans and vision during the GA.

Marc-André Lemburg, who has been serving on the EPS board since 2012 and as its chair since 2017 is not running for the board again, as already announced in last year’s GA. He’s looking forward to enjoying the conference from the attendee perspective again.

Patrick Arminio

Software Developer / Python Italia Chair

Patrick started his journey with the Python community at PyCon Italia 2 in 2008. After many years of helping run PyCon Italia (and other conferences) as a volunteer he became the Chair of Python Italia in 2017.

He has also been nominated as a PSF Fellow for his contribution to conferences and also open source software.

He currently works as a Software Engineer in London.

Prof. Martin Christen

Teaching Python / using Python for research projects

Martin Christen is a professor of Geoinformatics and Computer Graphics at the Institute of Geomatics at the University of Applied Sciences Northwestern Switzerland (FHNW). His main research interests are geospatial Virtual- and Augmented Reality, 3D geoinformation, and interactive 3D maps.

Martin is very active in the Python community. He teaches various Python-related courses and uses Python in many research projects. He organizes the PyBasel meetup - the local Python User Group Northwestern Switzerland. He also organizes the yearly GeoPython conference. He is also a board member of the Python Software Verband e.V.

I would be glad to help with EuroPython, to be part of a great team that makes the next edition of EuroPython even better. I’m looking forward to a great physical/hybrid conference in Dublin.

Artur Czepiel

Software developer

Artur started writing in Python around 2008, originally using it mostly to implement backends for websites and later expanding to other areas.

He joined the EuroPython team in 2017 after watching a talk about the state of the conference software at the time. Then contributed patches, joined various work groups and helped with EuroPython 2018 and 2019 editions.

For the 2019 conference, he also joined the board where he helped with due diligence in the RFP process, but kept the focus on web/infra including major updates to the website software, and supporting other workgroups. Other than Europython, he's co-organising two local Python meetups in Kraków, Poland where he's based. He was also part of the team behind Remote Python Pizza, and provided minor software updates to other conferences.

I would like to join the EPS board again, with my main focus being (again) infrastructure. I've learned a lot, both during the 2019 term and since then, and I believe that for the 2022 conference we can improve the conference setup even more. My main focus will be the ease of organisation and ultimately making the conference experience better for everyone.

Nicolás Demarchi

Pythonista / Software Engineer

Nicolás is a self-taught software engineer working professionally for more than 15 years. After participating on some Linux User Groups and the Mozilla community, Nicolás joined the Python community around 2012, fell in love with it and never left. He is a founder and has been a board member of the Python Argentina NGO since 2016. In the PyAr community, as an organizer, he participated several events and conferences as organizer and/or speaker, ranging from Python Days in various cities, PyCamp and the Python Argentina national conference, being a core organizer in the 2018 one in Buenos Aires (an open and free conference with ~1500 attendees). Since 2014 Nicolás has been maintaining the Python Argentina infrastructure that supports the mailing list, webpages, etc.  He was (still helping a bit) the host of the Buenos Aires Python Meetup. In June 2019, Nicolás moved to Amsterdam and he is currently living and working there.  A few months after the move, he joined the organization of the Python Amsterdam meetup and he is working with a small team to build the local community: py.amsterdam. He also joined the https://pycamp.es/ team trying to replicate the Pycamp Argentina experience in Europe. In 2020 he volunteered in the Media Workgroup of Europython 2020 online as a core organizer.
He joined the EPS board in 2021 and helped to organize Europython2021-online.

I would like to continue in the EPS board because I think Europython is the event connecting all European communities and therefore the right place to invest my time. In addition, I believe I can learn a lot as a volunteer. For 2022 I want to work for the whole European Python community to have a better relation with EPS and to work on other smaller/local events apart from EuroPython.

Raquel Dou

Linguist / Python enthusiast

Raquel first met Python in 2013, during her MSc studies in Evolution of Language and Cognition, where she used Python to model the evolution of a simple communication system over time. She operates a small business providing language and technical services, in which Python is one of her primary tools.

She first attended EuroPython when it took place at her doorstep (Edinburgh) in 2018, and was an onsite volunteer. Since then she remained closely engaged in the EPS, as well as the organisation and execution of the conferences. She has been serving on the EPS board since 2019, working closely with the brilliant teams which delivered the two recent EuroPython Online editions. In 2021, besides leading the two amazing Sponsor and Support teams, she was also heavily involved in the conference lineup and speaker management.

During these 3 years, she has experienced warmth, openness, creativity and desire to do good in every aspect of her engagement with the Python community which proudly serves. For the next edition, she hopes to finally meet in Dublin every one of the volunteers she has worked so closely with for years. She shares a community building vision with the amazing EuroPython team and would love to continue in this exhilarating journey with them to take the Society to new heights.

Anders Hammarquist

Pythonista / Consultant / Software architect

Anders is running his own Python consultancy business, AB Struse, since 2019 and is currently mostly involved with using Python in industrial automation. He has been using Python since 1995, and fosters its use in at least four companies.

He helped organize EuroPython 2004 and 2005, and has attended and given talks at several EuroPythons since then. He has handled the Swedish financials of the EuroPython Society since 2016 and has served as board member since 2017.

Cheuk Ting Ho

Pythonista / Developer Advocate / Data Scientist

After spending 5 years researching theoretical physics at Hong Kong University of Science and Technology, Cheuk has transferred her analytical and logical skills in natural science and built a career in data science. Cheuk has been a Data Scientist before working in a team of developers building a revolutionary graph database.

Cheuk constantly contributes to the community by giving AI and deep learning workshops and organizing sprints for open source projects, at the same time contributing to open source projects including Pandas, Keras, Scikit-learn, Dateutil and maintaining open-source libraries. On top of speaking at conferences, Cheuk has joined the organizing team of EuroPython as a member of the programming workgroup since 2019 and was hosting the lightning talk in the same year.

Last year, Cheuk joined the EuroPython Society Board and was nominated to be the Python Software Foundation fellow. Cheuk has been leading the Financial Aid team to provide accessible tickets for people around the world to join EuroPython. Cheuk has also started a speaker mentorship program and organize workshops for new speakers. Believing in the benefit the society has with diversity and inclusion, Cheuk would like to continue bringing new faces to the society and keep connecting people in it. In 2021, hopefully we will meet again in person and Cheuk would like to make sure that EuroPython is accessible to everyone both online and in-person.

Francesco Pierfederici

Pythonic Beer Brewer and Drinker

The year 2000 was the year that Python saved Francesco from Perl and Java. He worked on a large variety of projects over the last 20 odd years all involving Python, mostly in scientific environments.

He is currently driving a 30m telescope with Python at IRAM (https://www.iram-institute.org). In his free time, he is trying to optimise his beer brewing with a host of sensors and micro controllers running MicroPython and still failing at that.

He has been volunteering with and in the EuroPython community since the conference in Rimini in 2017 and has helped with the website since 2019.

Why serve on the EPS board for a second year? Francesco loves the EuroPython conference and its volunteers, organisers and attendees. He would love the opportunity to give back to this community and to the Python community in general. A community that has given him so much.

Silvia Uberti

Sysadmin / IT Consultant

She is a Sysadmin with a degree in Network Security, really passionate about technology, traveling and her piano.

She’s an advocate for women in STEM disciplines and supports inclusiveness of underrepresented people in tech communities.

She fell in love with Python and its warm community during PyCon Italia in 2014 and became a member of EuroPython Sponsor Workgroup in 2017.

She enjoys working in it a lot and wants to help more!

Chair / Vice-Chair Nominations

Raquel Dou is running for the chair position and Artur Czepiel for the vice chair position.

What does the EPS Board do ?

The EPS board runs the day-to-day business of the EuroPython Society, including running the EuroPython conference events. It is allowed to enter contracts for the society and handle any issues that have not been otherwise regulated in the bylaws or by the General Assembly. Most business is handled on the board’s Telegram group or by email on the board mailing list. Board meetings are usually run as conference calls.

It is important to note that the EPS board is an active board, i.e. the board members are expected to put in a significant amount of time and effort towards the goals of the EPS and for running the EuroPython conference. This usually means 200+ hours work over a period of one year, with most of this being needed in the last six months before the conference. Many board members put in even more work to make sure that the EuroPython conferences become a success.

Board members are generally expected to take on leadership roles within the EuroPython Workgroups in order to facilitate good communication and quick decision making. They should be passionate about EuroPython, the Python community and working in a team of volunteers.

Enjoy,
EuroPython Society
https://www.europython-society.org/

September 23, 2021 11:32 AM UTC

Python Software Foundation

Katia Lira Awarded the PSF Community Service Award for Q2 2020

Katia Lira, Software Engineer from Mexico city, has been awarded the Python Software Foundation 2020 Q2 Community Service Award.

RESOLVED, that the Python Software Foundation award the Q2 2020 Community Service Award to Katia Lira for her contributions to PyCon LatAm as conference chair, which held its inaugural conference in 2019. Additionally, Katia is the DEFNA (Django Events Foundation North America) President and has collaborated in crafting the vision of PyLadies Global. She hosts and produces multiple Python/tech/community podcasts like El Dev Show in Spanish. She's a PyCon speaker and is well respected in the community.

We interviewed Katia to learn more about her inspiration and work with the Python community. We also asked two of Katia's associates - Valery Calderon and Cristian Maureira-Fredes to share more light about Katia's impact on the community.

What was your earliest memory of how you got into tech?

Growing up, I wanted to be an Architect. I loved doing the blueprints in AutoCAD. Then I switched to study web design but quickly found it easier to code than to do the UI. It felt effortless to use code to make a button send a form or trigger an animation, so I never looked back.

What was your earlier involvement with the Python community?

One of my best friends invited me to PyCon US in 2016. At the time, I was still learning Python and Django and was unaware of communities and conferences like that.

What inspires you to volunteer your time and resources in the Python Community?

It's always the conversations with people that make me want to continue volunteering and organizing spaces for sharing knowledge and building community, prioritizing Spanish as the language to share and engage.

How has your involvement in the Python community supported your career?

The most important thing is inspiration. Being involved in the community has widened my views on opportunities available to me and also the possibilities for growth. I discovered open source projects that push you to explore new tools and grow skills outside of day-to-day work.

Another thing is just being aware of the conversations around hiring and work. Especially when people are open about how to prepare for a technical interview and how they struggle with growing into a more senior role.

How has Covid affected your work with the Python community and what steps are you taking to push the community forward during these trying times?

I think we all tried to compensate with remote by joining as many virtual spaces as possible and it has been taxing on many of us. 

For the two conferences, I volunteer at - PyCon LatAm and DjangoCon US, we took 2020 off from having any events. That helped with not burning out our volunteers and organizers. And it made us ready for 2021 which has been easier because we had fewer uncertainties, we jumped in, knowing it was going to be fully remote and that the community members missed each other.

Katia Lira's Impact on the Python community

Katia has been a fundamental person in the whole PyCon LatAm initiative, being able to unify the many Latin American Python communities under the same umbrella. Hosting a LatAm conference seemed quite impossible if you ask me, but together with a wonderful team, they proved me wrong.

The enthusiasm and motivation I felt as a South American made me push forward the Python groups in my own country, from where we decided to organize our first small conference to a PyDay event. I asked Katia to be a keynote speaker and she agreed without any hesitation. 

Thanks to her talk, many people felt that being from LatAm was not a barrier to push for global communities and that people were responsible to make initiatives like PyLadies and PyCon LatAm as successful as they are.

Katia's keynote also motivated a lot of people from Chile. And this has led to an increase in community activities, beginning with new initiatives like the first PyLadies chapter - PyLadies Santiago.

And Katia's impact span beyond the LatAm region, to the global Spanish-speaking community.

Katia spear-headed the PyCon LatAm initiative. She is always open to helping people in the community by mentoring, giving talks, volunteering, and helping to organize events. She has also been of tremendous support to the past and present initiatives of creating room for diverse people within the Latin American community in the PSF.

Katia specifically helped me to propose my candidacy for the PyLadies Global Council.

In Latin America, there is a big gap in the culture of communities. And to make it better and inclusive, there's a lot of work that has been done and still needs to be done. Katia is helping to bridge this gap, which is a huge impact on the LatAm Python community.

September 23, 2021 04:42 AM UTC

Michał Bultrowicz

Setting up and syncing config on two laptops

I’ve created a script that should 1 set up a fresh Manjaro with all the software and configuration that I want in a workstation. It can also update the setup on being rerun. Now I have two laptops that behave and look the same 2, and it’s easy for me to maintain that state. Oh I wanted that for a long time :)

September 23, 2021 12:00 AM UTC

September 22, 2021

Python for Beginners

Difference between yield and return in Python

You might have used yield and return statements while programming in python. In this article, we will discuss the theoretical concepts of return and yield keywords. We will also look at the difference between working of yield and return statements in python.

What is yield and return in Python?

Yield and return are keywords in python. They are used in a function to pass values from one function to another in a program.

The return keyword

The return statements are used in a function to return objects to the caller function. We can return a single value like a number or string or a container object such as a python dictionary, a tuple, or a list. 

For example, the sumOfNums()  function returns a number to the caller in the following source code.

def sumOfNums(num1, num2):
    result = num2 + num1
    return result


output = sumOfNums(10, 20)
print("Sum of 10 and 20 is:", output)

Output:

Sum of 10 and 20 is: 30

Similarly, we can use return statements to return container objects as shown in the following example. Here the function “square” takes a list of numbers as input and returns a list of squares of the elements of the input list.

def square(list1):
    newList = list()
    for i in list1:
        newList.append(i * i)
    return newList


input_list = [1, 2, 3, 4, 5, 6]
print("input list is:", input_list)
output = square(input_list)
print("Output list is:", output)

Output:

input list is: [1, 2, 3, 4, 5, 6]
Output list is: [1, 4, 9, 16, 25, 36]

We can have more than one return statement in a function. However, once a return statement is executed in the program, the statements written after the return statement will never be executed. 

The yield keyword 

The yield statements are also used in a function to return values to the caller function. But the yield statement works in a different way. When the yield statement is executed in a function, it returns a generator object to the caller. The value in the generator object can be accessed using the next() function or a for loop as follows.

def square(list1):
    newList = list()
    for i in list1:
        newList.append(i * i)
    yield newList


input_list = [1, 2, 3, 4, 5, 6]
print("input list is:", input_list)
output = square(input_list)
print("Output from the generator is:", output)
print("Elements in the generator are:",next(output))

Output:

input list is: [1, 2, 3, 4, 5, 6]
Output from the generator is: <generator object square at 0x7fa59b674a50>
Elements in the generator are: [1, 4, 9, 16, 25, 36]

A function can have more than one yield statement. When the first yield statement is executed, it pauses the execution of the function and returns a generator to the caller function. When we perform the next operation on a generator using the next() function, the function again resumes and executes till the next yield statement. This process can be continued till the last statement of the function. You can understand this using the following example.

def square(list1):
    yield list1[0]**2
    yield list1[1] ** 2
    yield list1[2] ** 2
    yield list1[3] ** 2
    yield list1[4] ** 2
    yield list1[5] ** 2


input_list = [1, 2, 3, 4, 5, 6]
print("input list is:", input_list)
output = square(input_list)
print("Output from the generator is:", output)
print("Elements in the generator are:")
for i in output:
    print(i)

Output:

input list is: [1, 2, 3, 4, 5, 6]
Output from the generator is: <generator object square at 0x7fa421848a50>
Elements in the generator are:
1
4
9
16
25
36

You should keep in mind that when the generator is passed to the next() function after execution of the last yield statement, it causes StopIteration error. It can be avoided by using the next() function in a python try except block.

Difference between yield and return 

There are two major differences between the working of yield and return statements in python.

•  Return statement stops the execution of the function. Whereas, yield statement only pauses the execution of the function.
• The statements written in a program after the return statement are unreachable and are never executed. On the other hand, statements written after the yield statement are executed when the function resumes its execution.

Conclusion

In this article, we have studied yield and return statements in Python. We also looked at the difference between yield and return statement. To learn more about python programming, you can read this article on list comprehension. You may also like this article on the linked list in Python.

The post Difference between yield and return in Python appeared first on PythonForBeginners.com.

September 22, 2021 02:31 PM UTC

Real Python

The Django Template Language: Tags and Filters

 1$ python -m pip install django==3.2.5
 2$ django-admin startproject MoviePalace
 3$ cd MoviePalace
 4$ python manage.py startapp core

33INSTALLED_APPS = [
34    "django.contrib.admin",
35    "django.contrib.auth",
36    "django.contrib.contenttypes",
37    "django.contrib.sessions",
38    "django.contrib.messages",
39    "django.contrib.staticfiles",
40    "core",
41]

 1# core/views.py
 2from django.http import HttpResponse
 3from django.template import Context, Template
 4
 5def citizen_kane(request):
 6    content = """{{movie}} was released in {{year}}"""
 7    template = Template(content)
 8    context = Context({"movie": "Citizen Kane", "year": 1941})
 9
10    result = template.render(context)
11    return HttpResponse(result)

Read the full article at https://realpython.com/django-templates-tags-filters/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

September 22, 2021 02:00 PM UTC

Mike Driscoll

Creating an MP3 Tagger GUI with wxPython

I don’t know about you, but I enjoy listening to music. As an avid music fan, I also like to rip my CDs to MP3 so I can listen to my music on the go a bit easier. There is still a lot of music that is unavailable to buy digitally. Unfortunately, when you rip a lot of music, you will sometimes end up with errors in the MP3 tags. Usually, there is a mis-spelling in a title or a track isn’t tagged with the right artist. While you can use many open source and paid programs to tag MP3 files, it’s also fun to write your own.

That is the topic of this article. In this article, you will write a simple MP3 tagging application. This application will allow you to view an MP3 file’s current tags as well as edit the following tags:

• Artist
• Album
• Track Name
• Track Number

The first step in your adventure is finding the right Python package for the job!

Finding an MP3 Package

There are several Python packages that you can use for editing MP3 tags. Here are a few that I found when I did a Google search:

• eyeD3
• mutagen
• mp3-tagger
• pytaglib

You will be using eyeD3 for this chapter. It has a nice API that is fairly straight-forward. Frankly, I found most of the APIs for these packages to be brief and not all that helpful. However, eyeD3 seemed a bit more natural in the way it worked than the others that I tried, which is why it was chosen.

By the way, the package name, eyeD3, refers to the ID3 specification for metadata related to MP3 files.

However, the mutagen package is definitely a good fallback option because it supports many other types of audio metadata. If you happen to be working with other audio file types besides MP3, then you should definitely give mutagen a try.

Installing eyeD3

The eyeD3 package can be installed with pip. If you have been using a virtual environment (venv or virtualenv) for this book, make sure you have it activated before you install eyeD3:

python3 -m pip install eyeD3

Once you have eyeD3 installed, you might want to check out its documentation:

• https://pypi.org/project/eyeD3/

Now let’s get started and make a neat application!

Designing the MP3 Tagger

Your first step is to figure out what you want the user interface to look like. You will need the following features to make a useful application:

• Some way to import MP3s
• A way to display some of the metadata of the files
• A method of editing the metadata

Here is a simple mockup of what the main interface might look like:

This user interface doesn’t show how to actually edit the MP3, but it implies that the user would need to press the button at the bottom to start editing. This seems like a reasonable way to start.

Let’s code the main interface first!

Creating the Main Application

Now comes the fun part, which is writing the actual application. You will be using ObjectListView again for this example for displaying the MP3’s metadata. Technically you could use one of wxPython’s list control widgets. If you’d like a challenge, you should try changing the code in this chapter to using one of those.

Note: The code for this article can be downloaded on GitHub

Anyway, you can start by creating a file named main.py and entering the following:

# main.py

import eyed3
import editor
import glob
import wx

from ObjectListView import ObjectListView, ColumnDefn

class Mp3:

    def __init__(self, id3):
        self.artist = ''
        self.album = ''
        self.title = ''
        self.year = ''

        # Attempt to extract MP3 tags
        if not isinstance(id3.tag, type(None)):
            id3.tag.artist = self.normalize_mp3(
                id3.tag.artist)
            self.artist = id3.tag.artist
            id3.tag.album = self.normalize_mp3(
                id3.tag.album)
            self.album = id3.tag.album
            id3.tag.title = self.normalize_mp3(
                id3.tag.title)
            self.title = id3.tag.title
            if hasattr(id3.tag, 'best_release_date'):
                if not isinstance(
                    id3.tag.best_release_date, type(None)):
                    self.year = self.normalize_mp3(
                        id3.tag.best_release_date.year)
                else:
                    id3.tag.release_date = 2019
                    self.year = self.normalize_mp3(
                        id3.tag.best_release_date.year)
        else:
            tag = id3.initTag()
            tag.release_date = 2019
            tag.artist = 'Unknown'
            tag.album = 'Unknown'
            tag.title = 'Unknown'
        self.id3 = id3
        self.update()

Here you have the imports you need. You also created a class called Mp3 which will be used by the ObjectListView widget. The first four instance attributes in this class are the metadata that will be displayed in your application and are defaulted to strings. The last instance attribute, id3, will be the object returned from eyed3 when you load an MP3 file into it.

Not all MP3s are created equal. Some have no tags whatsoever and others may have only partial tags. Because of those issues, you will check to see if id3.tag exists. If it does not, then the MP3 has no tags and you will need to call id3.initTag() to add blank tags to it. If id3.tag does exist, then you will want to make sure that the tags you are interested in also exist. That is what the first part of the if statement attempts to do when it calls the normalize_mp3() function.

The other item here is that if there are no dates set, then the best_release_date attribute will return None. So you need to check that and set it to some default if it happens to be None.

Let’s go ahead and create the normalize_mp3() method now:

def normalize_mp3(self, tag):
    try:
        if tag:
            return tag
        else:
            return 'Unknown'
    except:
        return 'Unknown'

This will check to see if the specified tag exists. If it does, it simply returns the tag’s value. If it does not, then it returns the string: ‘Unknown’

The last method you need to implement in the Mp3 class is update():

def update(self):
    self.artist = self.id3.tag.artist
    self.album = self.id3.tag.album
    self.title = self.id3.tag.title
    self.year = self.id3.tag.best_release_date.year

This method is called at the end of the outer else in the class’s __init__() method. It is used to update the instance attributes after you have initialized the tags for the MP3 file.

There may be some edge cases that this method and the __init__() method will not catch. You are encouraged to enhance this code yourself to see if you can figure out how to fix those kinds of issues.

Now let’s go ahead and create a subclass of wx.Panel called TaggerPanel:

class TaggerPanel(wx.Panel):

    def __init__(self, parent):
        super().__init__(parent)
        self.mp3s = []
        main_sizer = wx.BoxSizer(wx.VERTICAL)

        self.mp3_olv = ObjectListView(
            self, style=wx.LC_REPORT | wx.SUNKEN_BORDER)
        self.mp3_olv.SetEmptyListMsg("No Mp3s Found")
        self.update_mp3_info()
        main_sizer.Add(self.mp3_olv, 1, wx.ALL | wx.EXPAND, 5)

        edit_btn = wx.Button(self, label='Edit Mp3')
        edit_btn.Bind(wx.EVT_BUTTON, self.edit_mp3)
        main_sizer.Add(edit_btn, 0, wx.ALL | wx.CENTER, 5)

        self.SetSizer(main_sizer)

The TaggerPanel is nice and short. Here you set up an instance attribute called mp3s that is initialized as an empty list. This list will eventually hold a list of instances of your Mp3 class. You also create your ObjectListView instance here and add a button for editing MP3 files.

Speaking of editing, let’s create the event handler for editing MP3s:

def edit_mp3(self, event):
    selection = self.mp3_olv.GetSelectedObject()
    if selection:
        with editor.Mp3TagEditorDialog(selection) as dlg:
            dlg.ShowModal()
            self.update_mp3_info()

Here you will use the GetSelectedObject() method from the ObjectListView widget to get the selected MP3 that you want to edit. Then you make sure that you got a valid selection and open up an editor dialog which is contained in your editor module that you will write soon. The dialog accepts a single argument, the eyed3 object, which you are calling selection here.

Note that you will need to call update_mp3_info() to apply any updates you made to the MP3’s tags in the editor dialog.

Now let’s learn how to load a folder that contains MP3 files:

def load_mp3s(self, path):
    if self.mp3s:
        # clear the current contents
        self.mp3s = []
    mp3_paths = glob.glob(path + '/*.mp3')
    for mp3_path in mp3_paths:
        id3 = eyed3.load(mp3_path)
        mp3_obj = Mp3(id3)
        self.mp3s.append(mp3_obj)
    self.update_mp3_info()

In this example, you take in a folder path and use Python’s glob module to search it for MP3 files. Assuming that you find the files, you then loop over the results and load them into eyed3. Then you create an instance of your Mp3 class so that you can show the user the MP3’s metadata. To do that, you call the update_mp3_info() method. The if statement at the beginning of the method is there to clear out the mp3s list so that you do not keep appending to it indefinitely.

Let’s go ahead and create the update_mp3_info() method now:

def update_mp3_info(self):
    self.mp3_olv.SetColumns([
        ColumnDefn("Artist", "left", 100, "artist"),
        ColumnDefn("Album", "left", 100, "album"),
        ColumnDefn("Title", "left", 150, "title"),
        ColumnDefn("Year", "left", 100, "year")
    ])
    self.mp3_olv.SetObjects(self.mp3s)

The update_mp3_info() method is used for displaying MP3 metadata to the user. In this case, you will be showing the user the Artist, Album title, Track name (title) and the Year the song was released. To actually update the widget, you call the SetObjects() method at the end.

Now let’s move on and create the TaggerFrame class:

class TaggerFrame(wx.Frame):

    def __init__(self):
        super().__init__(
            None, title="Serpent - MP3 Editor")
        self.panel = TaggerPanel(self)
        self.create_menu()
        self.Show()

Here you create an instance of the aforementioned TaggerPanel class, create a menu and show the frame to the user. This is also where you would set the initial size of the application and the title of the application. Just for fun, I am calling it Serpent, but you can name the application whatever you want to.

Let’s learn how to create the menu next:

def create_menu(self):
    menu_bar = wx.MenuBar()
    file_menu = wx.Menu()
    open_folder_menu_item = file_menu.Append(
        wx.ID_ANY, 'Open Mp3 Folder', 'Open a folder with MP3s'
    )
    menu_bar.Append(file_menu, '&File')
    self.Bind(wx.EVT_MENU, self.on_open_folder,
              open_folder_menu_item)
    self.SetMenuBar(menu_bar)

In this small piece of code, you create a menubar object. Then you create the file menu with a single menu item that you will use to open a folder on your computer. This menu item is bound to an event handler called on_open_folder(). To show the menu to the user, you will need to call the frame’s SetMenuBar() method.

The last piece of the puzzle is to create the on_open_folder() event handler:

def on_open_folder(self, event):
    with wx.DirDialog(self, "Choose a directory:",
                      style=wx.DD_DEFAULT_STYLE,
                      ) as dlg:
        if dlg.ShowModal() == wx.ID_OK:
            self.panel.load_mp3s(dlg.GetPath())

You will want to open a wx.DirDialog here using Python’s with statement and show it modally to the user. This prevents the user from interacting with your application while they choose a folder. If the user presses the OK button, you will call the panel instance’s load_mp3s() method with the path that they have chosen.

For completeness, here is how you will run the application:

if __name__ == '__main__':
    app = wx.App(False)
    frame = TaggerFrame()
    app.MainLoop()

You are always required to create a wx.App instance so that your application can respond to events.

Your application won’t run yet as you haven’t created the editor module yet.

Let’s learn how to do that next!

Editing MP3s

Editing MP3s is the point of this application, so you definitely need to have a way to accomplish that. You could modify the ObjectListView widget so that you can edit the data there or you can open up a dialog with editable fields. Both are valid approaches. For this version of the application, you will be doing the latter.

Let’s get started by creating the Mp3TagEditorDialog class:

# editor.py

import wx

class Mp3TagEditorDialog(wx.Dialog):

    def __init__(self, mp3):
        title = f'Editing "{mp3.id3.tag.title}"'
        super().__init__(parent=None, title=title)

        self.mp3 = mp3
        self.create_ui()

Here you instantiate your class and grab the MP3’s title from its tag to make the title of the dialog refer to which MP3 you are editing. Then you set an instance attribute and call the create_ui() method to create the dialog’s user interface.

Let’s create the dialog’s UI now:

def create_ui(self):
    self.main_sizer = wx.BoxSizer(wx.VERTICAL)

    size = (200, -1)
    track_num = str(self.mp3.id3.tag.track_num[0])
    year = str(self.mp3.id3.tag.best_release_date.year)

    self.track_number = wx.TextCtrl(
        self, value=track_num, size=size)
    self.create_row('Track Number', self.track_number)

    self.artist = wx.TextCtrl(self, value=self.mp3.id3.tag.artist,
                              size=size)
    self.create_row('Artist', self.artist)

    self.album = wx.TextCtrl(self, value=self.mp3.id3.tag.album,
                             size=size)
    self.create_row('Album', self.album)

    self.title = wx.TextCtrl(self, value=self.mp3.id3.tag.title,
                             size=size)
    self.create_row('Title', self.title)

    btn_sizer = wx.BoxSizer()
    save_btn = wx.Button(self, label="Save")
    save_btn.Bind(wx.EVT_BUTTON, self.save)

    btn_sizer.Add(save_btn, 0, wx.ALL, 5)
    btn_sizer.Add(wx.Button(self, id=wx.ID_CANCEL), 0, wx.ALL, 5)
    self.main_sizer.Add(btn_sizer, 0, wx.CENTER)

    self.SetSizerAndFit(self.main_sizer)

Here you create a series of wx.TextCtrl widgets that you pass to a function called create_row(). You also add the “Save” button at the end and bind it to the save() event handler. Finally you add a “Cancel” button. The way you create the Cancel button is kind of unique. All you need to do is pass wx.Button a special id: wx.ID_CANCEL. This will add the right label to the button and automatically make it close the dialog for you without actually binding it to a function.

This is one of the convenience functions built-in to the wxPython toolkit. As long as you don’t need to do anything special, this functionality is great.

Now let’s learn what to put into the create_row() method:

def create_row(self, label, text):
    sizer = wx.BoxSizer(wx.HORIZONTAL)
    row_label = wx.StaticText(self, label=label, size=(50, -1))
    widgets = [(row_label, 0, wx.ALL, 5),
               (text, 0, wx.ALL, 5)]
    sizer.AddMany(widgets)
    self.main_sizer.Add(sizer)

In this example, you create a horizontal sizer and an instance of wx.StaticText with the label that you passed in. Then you add both of these widgets to a list of tuples where each tuple contains the arguments you need to pass to the main sizer. This allows you to add multiple widgets to a sizer at once via the AddMany() method.

The last piece of code you need to create is the save() event handler:

def save(self, event):
    current_track_num = self.mp3.id3.tag.track_num
    if current_track_num:
        new_track_num = (int(self.track_number.GetValue()),
                         current_track_num[1])
    else:
        new_track_num = (int(self.track_number.GetValue()), 0)

    artist = self.artist.GetValue()
    album = self.album.GetValue()
    title = self.title.GetValue()

    self.mp3.id3.tag.artist = artist if artist else 'Unknown'
    self.mp3.id3.tag.album = album if album else 'Unknown'
    self.mp3.id3.tag.title = title if title else 'Unknown'
    self.mp3.id3.tag.track_num = new_track_num
    self.mp3.id3.tag.save()
    self.mp3.update()
    self.Close()

Here you check if the track number was set in the MP3’s tag. If it was, then you update it to the new value you set it to. On the other hand, if the track number is not set, then you need to create the tuple yourself. The first number in the tuple is the track number and the second number is the total number of tracks on the album. If the track number is not set, then you can’t know the total number of track reliably programmatically, so you just set it to zero by default.

The rest of the function is setting the various MP3 object’s tag attributes to what is in the dialog’s text controls. Once all the attributes are set, you can call the save() method on the eyed3 MP3 object, tell the Mp3 class instance to update itself and close the dialog. Note that if you try to pass in an empty value for artist, album or title, it will be replaced with the string Unknown.

Now you have all the pieces that you need and you should be able to run the program.

Here is what the main application looked like on my machine:

And here is what the editor dialog looked like:

Now let’s learn how to add a few enhancements to your program!

Adding New Features

Most applications of this type will allow the user to drag-and-drop files or folders onto them. They also usually have a toolbar for opening folders in addition to the menu. You learned how to do both of these in the previous chapter. You will now add these features to this program as well.

Let’s start by creating our DropTarget class to main.py:

import os

class DropTarget(wx.FileDropTarget):

    def __init__(self, window):
        super().__init__()
        self.window = window

    def OnDropFiles(self, x, y, filenames):
        self.window.update_on_drop(filenames)
        return True

Adding the drag-and-drop feature requires you to sub-class wx.FileDropTarget. You need to pass in the widget that will be the drop target as well. In this case, you want the wx.Panel to be the drop target. Then you override OnDropFiles so that it calls the update_on_drop() method. This is a new method that you will be adding shortly.

But before you do that, you need to update the beginning of your TaggerPanel class:

class TaggerPanel(wx.Panel):

    def __init__(self, parent):
        super().__init__(parent)
        self.mp3s = []
        drop_target = DropTarget(self)
        self.SetDropTarget(drop_target)
        main_sizer = wx.BoxSizer(wx.VERTICAL)      

Here you create an instance of DropTarget and then set the panel as the drop target via the SetDropTarget() method. The benefit of doing this is that now you can drag and drop files or folder pretty much anywhere on your application and it will work.

Note that the above code is not the full code for the __init__() method, but only shows the changes in context. See the source code on Github for the full version.

The first new method to look at is add_mp3():

def add_mp3(self, path):
    id3 = eyed3.load(path)
    mp3_obj = Mp3(id3)
    self.mp3s.append(mp3_obj)

Here you pass in the path of the MP3 file that you want to add to the user interface. It will take that path and load it with eyed3 and add that to your mp3s list.

The edit_mp3() method is unchanged for this version of the application, so it is not reproduced here.

Now let’s move on and create another new method called find_mp3s():

def find_mp3s(self, folder):
    mp3_paths = glob.glob(folder + '/*.mp3')
    for mp3_path in mp3_paths:
        self.add_mp3(mp3_path)

This code and the code in the add_mp3s() method might look a bit familiar to you. It is originally from the load_mp3() method that you created earlier. You are moving this bit of code into its own function. This is known as refactoring your code. There are many reasons to refactor your code. In this case, you are doing so because you will need to call this function from multiple places. Rather than copying this code into multiple functions, it is almost always better to separate it into its own function that you can call.

Now let’s update the load_mp3s() method so that it calls the new one above:

def load_mp3s(self, path):
    if self.mp3s:
        # clear the current contents
        self.mp3s = []
    self.find_mp3s(path)
    self.update_mp3_info()

This method has been reduced to two lines of code. The first calls the find_mp3s() method that you just wrote while the second calls the update_mp3_info(), which will update the user interface (i.e. the ObjectListView widget).

The DropTarget class is calling the update_on_drop() method, so let’s write that now:

def update_on_drop(self, paths):
    for path in paths:
        if os.path.isdir(path):
            self.load_mp3s(path)
        elif os.path.isfile(path):
            self.add_mp3(path)
            self.update_mp3_info()

The update_on_drop() method is the reason you did the refactoring earlier. It also needs to call the load_mp3s(), but only when the path that is passed in is determined to be a directory. Otherwise you check to see if the path is a file and load it up.

But wait! There’s an issue with the code above. Can you tell what it is?

The problem is that when the path is a file, you aren’t checking to see if it is an MP3. If you run this code as is, you will cause an exception to be raised as the eyed3 package will not be able to turn all file types into Mp3 objects.

Let’s fix that issue:

def update_on_drop(self, paths):
    for path in paths:
        _, ext = os.path.splitext(path)
        if os.path.isdir(path):
            self.load_mp3s(path)
        elif os.path.isfile(path) and ext.lower() == '.mp3':
            self.add_mp3(path)
            self.update_mp3_info()

You can use Python’s os module to get the extension of files using the splitext() function. It will return a tuple that contains two items: The path to the file and the extension.

Now that you have the extension, you can check to see if it is .mp3 and only update the UI if it is. By the way, the splitext() function returns an empty string when you pass it a directory path.

The next bit of code that you need to update is the TaggerFrame class so that you can add a toolbar:

class TaggerFrame(wx.Frame):

    def __init__(self):
        super().__init__(
            None, title="Serpent - MP3 Editor")
        self.panel = TaggerPanel(self)
        self.create_menu()
        self.create_tool_bar()
        self.Show()

The only change to the code above is to add a call to the create_tool_bar() method. You will almost always want to create the toolbar in a separate method as there is typically several lines of code per toolbar button. For applications with many buttons in the toolbar, you should probably separate that code out even more and put it into a class or module of its own.

Let’s go ahead and write that method:

def create_tool_bar(self):
    self.toolbar = self.CreateToolBar()
    
    add_folder_ico = wx.ArtProvider.GetBitmap(
        wx.ART_FOLDER_OPEN, wx.ART_TOOLBAR, (16, 16))
    add_folder_tool = self.toolbar.AddTool(
        wx.ID_ANY, 'Add Folder', add_folder_ico,
        'Add a folder to be archived')
    self.Bind(wx.EVT_MENU, self.on_open_folder,
              add_folder_tool)
    self.toolbar.Realize()

To keep things simple, you add a single toolbar button that will open a directory dialog via the on_open_folder() method.

When you run this code, your updated application should now look like this:

Feel free to add more toolbar buttons, menu items, a status bar or other fun enhancements to this application.

Wrapping Up

This article taught you a little about some of Python’s MP3 related packages that you can use to edit MP3 tags as well as other tags for other music file formats. You learned how to create a nice main application that opens an editing dialog. The main application can be used to display relevant MP3 metadata to the user. It also serves to show the user their updates should they decide to edit one or more tags.

The wxPython tookit has support for playing back certain types of audio file formats including MP3. You could create an MP3 player using these capabilities and make this application a part of that.

Download the Source

You can download the source code for the examples in this article on GitHub

Related Articles

Want to learn more about what you can create with wxPython? Check out the following articles:

• Creating a Simple Wizard with wxPython (Video)

• Creating a File Search GUI with wxPython

• Creating a GUI Application for NASA’s API with wxPython

The post Creating an MP3 Tagger GUI with wxPython appeared first on Mouse Vs Python.

September 22, 2021 12:30 PM UTC

Python Bytes

#251 A 95% complete episode (wait for it)

<p><strong>Watch the live stream:</strong></p> <a href='https://www.youtube.com/watch?v=ssB7SouW1tk' style='font-weight: bold;'>Watch on YouTube</a><br> <br> <p><strong>About the show</strong></p> <p>Sponsored by <strong>us:</strong></p> <ul> <li>Check out the <a href="https://training.talkpython.fm/courses/all"><strong>courses over at Talk Python</strong></a></li> <li>And <a href="https://pythontest.com/pytest-book/"><strong>Brian’s book too</strong></a>!</li> </ul> <p>Special guest: <strong><a href="https://twitter.com/brettsky">Brett Cannon</a></strong></p> <p><strong>Michael #1:</strong> <a href="https://auto-optional.daanluttik.nl/"><strong>auto-optional</strong></a></p> <ul> <li>by Daan Luttik</li> <li>Did you know that concrete types cannot be None in Python typing?</li> <li>This is wrong:</li> </ul> <pre><code> def do_a_thing(extra_info: str = None): ... </code></pre> <ul> <li>auto-optional will fix it:</li> </ul> <pre><code> def do_a_thing(extra_info: Optional[str] = None): ... </code></pre> <ul> <li>Why would you want this?</li> <li>Easily modify external libraries that didn't pay attention to proper use of optional to improve mypy linting.</li> <li>Force consistency in your own code-base: Enforcing that None parameter implies an Optional type.</li> <li>Run via the CLI: <code>auto-optional [path]</code></li> </ul> <p><strong>Brian #2:</strong> <a href="https://daniel.haxx.se/blog/2021/09/04/making-world-class-docs-takes-effort"><strong>Making World-Class Docs Takes Effort</strong></a></p> <ul> <li>Daniel Stenberg</li> <li>Six requirements for a project to get a gold star <ul> <li>docs in the code repo</li> <li>NOT extracted from the code</li> <li>examples, lots of examples, more than you think you need</li> <li>document every API call you provide</li> <li>easily accessible and browsable</li> <li>and hopefully offline readable as well</li> <li>easy to contribute to</li> </ul></li> <li>Non-stop iterating is key to having good docs.</li> <li>extra goodness <ul> <li>consistency for section titles</li> <li>cross-references</li> </ul></li> <li>I’d add <ul> <li>Check for grammar and spelling mistakes</li> <li>Consistency in all things, formatting, style, tone, depth of info of diff topics</li> <li>Don’t be afraid to have a personality. docs that include easter eggs, fun examples, tasteful jokes, etc are nice, as long as that fun stuff doesn’t complicate the docs.</li> <li>Don’t slam projects for having bad docs. Not all open source projects exist for your benefit. <ul> <li>You can make them better by contributing. :)</li> </ul></li> </ul></li> </ul> <p><strong>Brett #3:</strong> <a href="https://starship.rs/"><strong>Starship</strong></a></p> <ul> <li>Continuing the trend of stuff to help make your coding better, Python or not. 😉</li> <li>Also to make Michael’s new love of nerd fonts more useful. 😁</li> <li>And more Rust on this show as Paul Everitt says I must do. 😉</li> <li>Gives you a common shell prompt no matter which shell you use; I also find it easy to set up compared to most shells for their prompts</li> <li>Lots of integrated support for various developer things such as printing what Python version you have when the directory has a <code>pyproject.toml</code> file.</li> <li>Works nicely with the Python Launcher (as I mentioned the last time I was on).</li> <li>Has some pyenv support that I don’t use. 😁</li> </ul> <p><strong>Michael #4:</strong> <a href="https://pypi.org/project/jmespath/"><strong>JMESPath</strong></a></p> <ul> <li>via Josh Thurston</li> <li>Spent tons of time figuring out how to parse the pretty print results that had layers of nested dictionaries and lists. This module saved me time in a big way.</li> <li>JMESPath (pronounced “james path”) allows you to declaratively specify how to extract elements from a JSON document.</li> <li>For example, given this document:</li> </ul> <pre><code> {"foo": {"bar": "baz"}} </code></pre> <ul> <li>The jmespath expression <code>foo.bar</code> will return “baz”.</li> <li>Even works with a projection-like result:</li> </ul> <pre><code> {"foo": {"bar": [{"name": "one"}, {"name": "two"}]}} </code></pre> <ul> <li>The expression: <code>foo.bar[*].name</code> will return <code>["one", "two"]</code>. </li> <li>Negative indexing is also supported (-1 refers to the last element in the list). </li> <li>Given the data above, the expression <code>foo.bar[-1].name</code> will return <code>"two"</code>.</li> </ul> <p><strong>Brian #5:</strong> <a href="https://github.com/spotify/pedalboard"><strong>pedalboard</strong></a> - audio effects library</p> <ul> <li>from Spotify</li> <li>The “power, speed, and sound quality of a DAW”, but in Python.</li> <li><a href="https://engineering.atspotify.com/2021/09/07/introducing-pedalboard-spotifys-audio-effects-library-for-python">Introduction Article</a> (warning: weird color changing header image that is painful to look at, so scroll past that quickly) <ul> <li>Built-in support for a number of basic audio transformations:</li> <li><code>Convolution</code>, <code>Compressor</code>, <code>Chorus</code>, <code>Distortion</code></li> <li><code>Gain</code>, <code>HighpassFilter</code>, <code>LadderFilter</code>, <code>Limiter</code>, <code>LowpassFilter</code></li> <li><code>Phaser</code>, <code>Reverb</code></li> </ul></li> </ul> <p><strong>Brett #6:</strong> <a href="https://www.python.org/dev/peps/pep-0665/"><strong>PEP 665</strong></a> <strong>(and the</strong> <a href="https://discuss.python.org/t/pep-665-specifying-installation-requirements-for-python-projects/9911/152"><strong>journey so far</strong></a><strong>)</strong></p> <ul> <li>Attempt to standardize lock files for Python.</li> <li>Spent six months talking w/ folks privately to come up with the first public draft.</li> <li>Initially a strict lock file, but Poetry and PDM feedback was platform-agnostic was important.</li> <li>Proposal morphed to cover that.</li> <li>Took it public and led to over 150 comments on Discourse.</li> <li>People disliked it: from the title to the explanation to the proposed problem space to the actual solution.</li> <li>Gone back to the drawing board privately w/ one of the original objectors participating; looking like we are reaching a good consensus on how to frame things and how it should ultimately look.</li> <li>(Packaging) PEPs are hard.</li> </ul> <p><strong>Extras</strong></p> <p>Brian</p> <ul> <li>Python is popular, apparently, and <a href="https://www.zdnet.com/article/programming-languages-python-is-on-the-verge-of-another-big-step-forward/">“on the verge of another big step forward”</a> (another good place for dun, dun, duuunnn, ?) <ul> <li>"It only needs to bridge 0.16% to surpass C. This might happen any time now. If Python becomes number 1, a new milestone has been reached in the TIOBE index. Only 2 other languages have ever been leading the pack so far, i.e. C and Java."</li> </ul></li> </ul> <p>Michael</p> <ul> <li><a href="https://www.nerdfonts.com/"><strong>Nerd Fonts</strong></a></li> <li><a href="https://evrone.com/michael-kennedy-interview"><strong>Evrone interview with me</strong></a></li> <li><a href="https://twitter.com/HenrySchreiner3/status/1438227200466202625"><strong>Henry Schreiner’s Fish setup</strong></a></li> <li><a href="https://linuxize.com/post/how-to-create-bash-aliases/"><strong>Aliases rather than CLI/venvs</strong></a></li> </ul> <p>Brett</p> <ul> <li><a href="https://www.youtube.com/watch?v=1kTWxamIJ_k">Will McGugan did a webinar w/ Paul Everitt</a> about Textual (because it’s not a Python Bytes episode if Will’s name is not brought up).</li> <li>Python Launcher officially launched! (Last covered <a href="https://pythonbytes.fm/episodes/show/221/pattern-matching-and-accepting-change-in-python-with-brett-cannon">30 episodes ago</a>.) <ul> <li>Available in AUR, Fedora, and Homeberw (both macOS and Linux).</li> <li>No reported bugs since launch!</li> </ul></li> <li>Still doing my <a href="https://snarky.ca/tag/syntactic-sugar/">syntactic sugar blog posts</a>.</li> <li>The Python extension for VS Code has a <a href="https://code.visualstudio.com/docs/python/testing">refreshed testing UX</a>; we’re coming for you, Brian. 😉</li> </ul> <p><strong>Joke:</strong> <a href="https://geek-and-poke.com/geekandpoke/2021/2/2/the-last-5"><strong>Last 5%</strong></a></p>

September 22, 2021 08:00 AM UTC

Brett Cannon

Unravelling comprehensions

After failing to unravel generator expressions, in this post as part as my Python syntactic sugar post series I want to tackle comprehensions. Thanks to a change made in Python 3.0, recreating comprehensions using generator expressions is straightforward since comprehensions do not leak their loop variant.

List comprehensions

Unravelling [c for b in a] is list(c for b in a): take the body of the list comprehension and pass it to list() as a generator expression.

Set comprehensions

Unravelling {c for b in a} is set(c for b in a). I suspect you&aposre starting to see a pattern...

Dict comprehensions

There&aposs a slight trick to dict comprehensions in that the generator expression needs to return key/value pairs, otherwise it&aposs the same "pass a generator to the type&aposs constructor" trick. That means {c: d for b in a} becomes dict((c, d) for b in a).

Aside: why isn&apost there a tuple comprehension?

There are two reasons why there is no such thing as a "tuple comprehension".

One is how would you represent that syntactically? The obvious syntax of parentheses around a comprehension-like syntax, but that&aposs taken by generator expressions.

The second reason is a "tuple comprehension" is a misuse of tuples. The data structure is meant to provide an index-based struct instead of an immutable sequence. Consider a pair of key and value like we did for  dict comprehensions: (key, value). What&aposs represented by index 0 means something different than what is in index 1.

September 22, 2021 12:10 AM UTC

Python⇒Speed

Scanning your Conda environment for security vulnerabilities

You don’t want to deploy an application that has security vulnerabilities. That means your own code, but also third-party dependencies: it doesn’t matter how secure your code is if it’s exposing a TLS socket with a version of OpenSSL that has a remote code execution vulnerability.

For pip-based Python applications, you’d usually run vulnerability scanners on Python dependencies like Django, and on system packages like OpenSSL. With Conda, however, the situation is a little different: Conda combines both types of packages into one place. In addition, most vulnerability scanners don’t support Conda.

Let’s see what makes Conda different, and how you can scan packages for known vulnerabilities.

September 22, 2021 12:00 AM UTC

September 21, 2021

PyCoder’s Weekly

Issue #491 (Sept. 21, 2021)

#491 – SEPTEMBER 21, 2021
View in Browser »

Get Deeper Insights Into Your Python Code-Level Performance and Reduce End-User Latency

Datadog APM enables you to detect the methods that consume the most CPU, memory, and time under real work loads allowing you to optimize code in real-time to reduce end-user latency and cloud provider costs. Optimize your Python app performance at any scale. See for yourself with a free trial →
DATADOG sponsor

Discussions

Python Jobs

Senior Software Engineer (Washington D.C.)

Quorum

Senior Backend Software Engineer (Remote)

Clay

More Python Jobs >>>

Articles & Tutorials

Rev APIs Solve All of Your Speech-to-Text Needs

Rev.ai is the most sophisticated automatic speech recognition in the world. Our speech-to-text APIs are more accurate, easier to use, and have less bias than competitors like Google, Amazon, and Microsoft. Try Rev.ai free for five hours right now →
REV.AI sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #491.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

September 21, 2021 07:30 PM UTC

Django Weblog

Django 4.0 alpha 1 released

Django 4.0 alpha 1 is now available. It represents the first stage in the 4.0 release cycle and is an opportunity for you to try out the changes coming in Django 4.0.

Django 4.0 has a abundance of new features which you can read about in the in-development 4.0 release notes.

This alpha milestone marks the feature freeze. The current release schedule calls for a beta release in about a month and a release candidate about a month from then. We'll only be able to keep this schedule if we get early and often testing from the community. Updates on the release schedule are available on the django-developers mailing list.

As with all alpha and beta packages, this is not for production use. But if you'd like to take some of the new features for a spin, or to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the alpha package from our downloads page or on PyPI.

The PGP key ID used for this release is Mariusz Felisiak: 2EF56372BA48CD1B.

September 21, 2021 07:10 PM UTC

Real Python

Pass by Reference in Python: Best Practices

After gaining some familiarity with Python, you may notice cases in which your functions don’t modify arguments in place as you might expect, especially if you’re familiar with other programming languages. Some languages handle function arguments as references to existing variables, which is known as pass by reference. Other languages handle them as independent values, an approach known as pass by value.

If you’re an intermediate Python programmer who wishes to understand Python’s peculiar way of handling function arguments, then this course is for you. You’ll implement real use cases of pass-by-reference constructs in Python and learn several best practices to avoid pitfalls with your function arguments.

In this course, you’ll learn:

• What it means to pass by reference and why you’d want to do so
• How passing by reference differs from both passing by value and Python’s unique approach
• How function arguments behave in Python
• How you can use certain mutable types to pass by reference in Python
• What the best practices are for replicating pass by reference in Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

September 21, 2021 02:00 PM UTC

Mike Driscoll

How to Send Emails with Python

Python provides a couple of really nice modules that you can use to craft emails with. They are the email and smtplib modules. Instead of going over various methods in these two modules, you’ll spend some time learning how to actually use these modules.

Specifically, you’ll be covering the following:

• The basics of emailing
• How to send to multiple addresses at once
• How to send email using the TO, CC and BCC lines
• How to add an attachment and a body using the email module

Let’s get started!

Email Basics – How to Send an Email with smtplib

The smtplib module is very intuitive to use. You will write a quick example that shows how to send an email.

Open up your favorite Python IDE or text editor and create a new Python file. Add the following code to a that file and save it:

import smtplib

HOST = "mySMTP.server.com"
SUBJECT = "Test email from Python"
TO = "mike@someAddress.org"
FROM = "python@mydomain.com"
text = "Python 3.4 rules them all!"

BODY = "\r\n".join((
"From: %s" % FROM,
"To: %s" % TO,
"Subject: %s" % SUBJECT ,
"",
text
))

server = smtplib.SMTP(HOST)
server.sendmail(FROM, [TO], BODY)
server.quit()

Here you import only the smtplib module. Two-thirds of this code is used for setting up the email. Most of the variables are pretty self-explanatory, so you’ll focus on the odd one only, which is BODY.

Here you use the string’s join() method to combine all the previous variables into a single string where each line ends with a carriage return (“/r”) plus new line (“/n”). If you print BODY out, it would look like this:

'From: python@mydomain.com\r\nTo: mike@mydomain.com\r\nSubject: Test email from Python\r\n\r\nblah blah blah'

After that, you set up a server connection to your host and then you call the smtplib module’s sendmail method to send the email. Then you disconnect from the server. You will note that this code doesn’t have a username or password in it. If your server requires authentication, then you’ll need to add the following code:

server.login(username, password)

This should be added right after you create the server object. Normally, you would want to put this code into a function and call it with some of these parameters. You might even want to put some of this information into a config file.

Let’s put this code into a function.

import smtplib

def send_email(host, subject, to_addr, from_addr, body_text):
    """
    Send an email
    """
    BODY = "\r\n".join((
            "From: %s" % from_addr,
            "To: %s" % to_addr,
            "Subject: %s" % subject ,
            "",
            body_text
            ))
    server = smtplib.SMTP(host)
    server.sendmail(from_addr, [to_addr], BODY)
    server.quit()

if __name__ == "__main__":
    host = "mySMTP.server.com"
    subject = "Test email from Python"
    to_addr = "mike@someAddress.org"
    from_addr = "python@mydomain.com"
    body_text = "Python rules them all!"
    send_email(host, subject, to_addr, from_addr, body_text)

Now you can see how small the actual code is by just looking at the function itself. That’s 13 lines! And you could make it shorter if you didn’t put every item in the BODY on its own line, but it wouldn’t be as readable. Now you’ll add a config file to hold the server information and the from address.

Why would you do that? Many organizations use different email servers to send email or if the email server gets upgraded and the name changes, then you only need to change the config file rather than the code. The same thing could apply to the from address if your company was bought and merged into another.

Let’s take a look at the config file (save it as email.ini):

[smtp]
server = some.server.com
from_addr = python@mydomain.com

That is a very simple config file. In it, you have a section labeled smtp in which you have two items: server and from_addr. you’ll use ConfigParser to read this file and turn it into a Python dictionary. Here’s the updated version of the code (save it as smtp_config.py)

import os
import smtplib
import sys

from configparser import ConfigParser

def send_email(subject, to_addr, body_text):
    """
    Send an email
    """
    base_path = os.path.dirname(os.path.abspath(__file__))
    config_path = os.path.join(base_path, "email.ini")

    if os.path.exists(config_path):
        cfg = ConfigParser()
        cfg.read(config_path)
    else:
        print("Config not found! Exiting!")
        sys.exit(1)

    host = cfg.get("smtp", "server")
    from_addr = cfg.get("smtp", "from_addr")

    BODY = "\r\n".join((
        "From: %s" % from_addr,
        "To: %s" % to_addr,
        "Subject: %s" % subject ,
        "",
        body_text
    ))
    server = smtplib.SMTP(host)
    server.sendmail(from_addr, [to_addr], BODY)
    server.quit()

if __name__ == "__main__":
    subject = "Test email from Python"
    to_addr = "mike@someAddress.org"
    body_text = "Python rules them all!"
    send_email(subject, to_addr, body_text)

You have added a little check to this code. You want to first grab the path that the script itself is in, which is what base_path represents. Next, you combine that path with the file name to get a fully qualified path to the config file. You then check for the existence of that file.

If it’s there, you create a ConfigParser and if it’s not, you print a message and exit the script. you should add an exception handler around the ConfigParser.read() call just to be on the safe side though as the file could exist, but be corrupt or you might not have permission to open it and that will throw an exception.

That will be a little project that you can attempt on your own. Anyway, let’s say that everything goes you’ll and the ConfigParser object is created successfully. Now you can extract the host and from_addr information using the usual ConfigParser syntax.

Now you’re ready to learn how to send multiple emails at the same time!

Sending Multiple Emails at Once

Being able to send multiple emails at once is a nice feature to have.

Go ahead and modify your last example a little so you can send multiple emails!

import os
import smtplib
import sys

from configparser import ConfigParser

def send_email(subject, body_text, emails):
    """
    Send an email
    """
    base_path = os.path.dirname(os.path.abspath(__file__))
    config_path = os.path.join(base_path, "email.ini")

    if os.path.exists(config_path):
        cfg = ConfigParser()
        cfg.read(config_path)
    else:
        print("Config not found! Exiting!")
        sys.exit(1)

    host = cfg.get("smtp", "server")
    from_addr = cfg.get("smtp", "from_addr")

    BODY = "\r\n".join((
            "From: %s" % from_addr,
            "To: %s" % ', '.join(emails),
            "Subject: %s" % subject ,
            "",
            body_text
            ))
    server = smtplib.SMTP(host)
    server.sendmail(from_addr, emails, BODY)
    server.quit()

if __name__ == "__main__":
    emails = ["mike@someAddress.org", "someone@gmail.com"]
    subject = "Test email from Python"
    body_text = "Python rules them all!"
    send_email(subject, body_text, emails)

You’ll notice that in this example, you removed the to_addr parameter and added an emails parameter, which is a list of email addresses. To make this work, you need to create a comma-separated string in the To: portion of the BODY and also pass the email list to the sendmail method. Thus you do the following to create a simple comma-separated string: ‘, ‘.join(emails). Simple, huh?

Send email using the TO, CC and BCC lines

Now you just need to figure out how to send using the CC and BCC fields.

Let’s create a new version of this code that supports that functionality!

import os
import smtplib
import sys

from configparser import ConfigParser

def send_email(subject, body_text, to_emails, cc_emails, bcc_emails):
    """
    Send an email
    """
    base_path = os.path.dirname(os.path.abspath(__file__))
    config_path = os.path.join(base_path, "email.ini")

    if os.path.exists(config_path):
        cfg = ConfigParser()
        cfg.read(config_path)
    else:
        print("Config not found! Exiting!")
        sys.exit(1)

    host = cfg.get("smtp", "server")
    from_addr = cfg.get("smtp", "from_addr")

    BODY = "\r\n".join((
            "From: %s" % from_addr,
            "To: %s" % ', '.join(to_emails),
            "CC: %s" % ', '.join(cc_emails),
            "BCC: %s" % ', '.join(bcc_emails),
            "Subject: %s" % subject ,
            "",
            body_text
            ))
    emails = to_emails + cc_emails + bcc_emails

    server = smtplib.SMTP(host)
    server.sendmail(from_addr, emails, BODY)
    server.quit()

if __name__ == "__main__":
    emails = ["mike@somewhere.org"]
    cc_emails = ["someone@gmail.com"]
    bcc_emails = ["schmuck@newtel.net"]

    subject = "Test email from Python"
    body_text = "Python rules them all!"
    send_email(subject, body_text, emails, cc_emails, bcc_emails)

In this code, you pass in 3 lists, each with one email address a piece. you create the CC and BCC fields exactly the same as before, but you also need to combine the 3 lists into one so you can pass the combined list to the sendmail() method.

There is some talk on forums like StackOverflow that some email clients may handle the BCC field in odd ways that allow the recipient to see the BCC list via the email headers. I am unable to confirm this behavior, but I do know that Gmail successfully strips the BCC information from the email header.

Now you’re ready to move on to using Python’s email module!

Add an attachment / body using the email module

Now you’ll take what you learned from the previous section and mix it together with the Python email module so that you can send attachments.

The email module makes adding attachments extremely easy. Here’s the code:

import os
import smtplib
import sys

from configparser import ConfigParser
from email import encoders
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.utils import formatdate

def send_email_with_attachment(subject, body_text, to_emails,
                               cc_emails, bcc_emails, file_to_attach):
    """
    Send an email with an attachment
    """
    base_path = os.path.dirname(os.path.abspath(__file__))
    config_path = os.path.join(base_path, "email.ini")
    header = 'Content-Disposition', 'attachment; filename="%s"' % file_to_attach

    # get the config
    if os.path.exists(config_path):
        cfg = ConfigParser()
        cfg.read(config_path)
    else:
        print("Config not found! Exiting!")
        sys.exit(1)

    # extract server and from_addr from config
    host = cfg.get("smtp", "server")
    from_addr = cfg.get("smtp", "from_addr")

    # create the message
    msg = MIMEMultipart()
    msg["From"] = from_addr
    msg["Subject"] = subject
    msg["Date"] = formatdate(localtime=True)
    if body_text:
        msg.attach( MIMEText(body_text) )

    msg["To"] = ', '.join(to_emails)
    msg["cc"] = ', '.join(cc_emails)

    attachment = MIMEBase('application', "octet-stream")
    try:
        with open(file_to_attach, "rb") as fh:
            data = fh.read()
        attachment.set_payload( data )
        encoders.encode_base64(attachment)
        attachment.add_header(*header)
        msg.attach(attachment)
    except IOError:
        msg = "Error opening attachment file %s" % file_to_attach
        print(msg)
        sys.exit(1)

    emails = to_emails + cc_emails

    server = smtplib.SMTP(host)
    server.sendmail(from_addr, emails, msg.as_string())
    server.quit()

if __name__ == "__main__":
    emails = ["mike@someAddress.org", "nedry@jp.net"]
    cc_emails = ["someone@gmail.com"]
    bcc_emails = ["anonymous@circe.org"]

    subject = "Test email with attachment from Python"
    body_text = "This email contains an attachment!"
    path = "/path/to/some/file"
    send_email_with_attachment(subject, body_text, emails, 
                               cc_emails, bcc_emails, path)

Here you have renamed your function and added a new argument, file_to_attach. You also need to add a header and create a MIMEMultipart object. The header could be created any time before you add the attachment.

You add elements to the MIMEMultipart object (msg) like you would keys to a dictionary. You’ll note that you have to use the email module’s formatdate method to insert the properly formatted date.

To add the body of the message, you need to create an instance of MIMEText. If you’re paying attention, you’ll see that you didn’t add the BCC information, but you could easily do so by following the conventions in the code above.

Next, you add the attachment. You wrap it in an exception handler and use the with statement to extract the file and place it in your MIMEBase object. Finally, you add it to the msg variable and you send it out. Notice that you have to convert the msg to a string in the sendmail() method.

Wrapping Up

Now you know how to send out emails with Python. For those of you that like mini-projects, you should go back and add additional error handling around the server.sendmail portion of the code in case something odd happens during the process.

One example would be a SMTPAuthenticationError or SMTPConnectError. You could also beef up the error handling during the attachment of the file to catch other errors. Finally, you may want to take those various lists of emails and create one normalized list that has removed duplicates. This is especially important if you are reading a list of email addresses from a file.

Also, note that your from address is fake. You can spoof emails using Python and other programming languages, but that is very bad etiquette and possibly illegal depending on where you live. You have been warned!

Use your knowledge wisely and enjoy Python for fun and profit!

Related Reading

Want to learn more Python basics? Then check out the following tutorials:

• Python 101: An Intro to Working with JSON

• Python 101 – Creating Multiple Processes

• Python 101 – Debugging Your Code with pdb

The post How to Send Emails with Python appeared first on Mouse Vs Python.

September 21, 2021 12:30 PM UTC

Python for Beginners

Find the mirror image of a binary tree

Unlike a Python dictionary, a list, or a set, elements of a binary tree are represented in a hierarchical manner. Having hierarchy in a binary tree allows one to find its mirror image as each element in a binary tree has a fixed position . In this article, we will study  the algorithm to find the mirror image of a binary tree. We will also implement the algorithm in Python and will execute it on an example binary tree.

What is the mirror image of a binary tree?

Mirror image of a binary tree is another binary tree which can be created by swapping left child and right child at each node of a tree. So, to find the mirror image of a binary tree, we just have to swap the left child and right child of each node in the binary tree. Let us try to find the mirror image of the following tree.

To find the mirror image of the above tree, we will start from the root and swap children of each node.

At root , we will swap the left and right children of the binary tree. In this way, 20,11, and 22 will come into the right subtree of the binary tree and 53,52, and 78 will come to the left of the binary tree as follows.

Then we will move to the next level and swap the children of 53. Here, 78 will become the left child of 53 while 52 will become the right child of 53. Similarly we will swap the left child and right child of 20. In this way, 22 will become the left child of 20 while 11 will become the right child of 20. The output binary tree after swapping nodes at this level will be as follows.

Now we will move to the next level. At this level, all the nodes are leaf nodes and have no children. Due to this there will be no swapping at this level and the above image is the final output.

Algorithm to find mirror image of a binary tree

As we have seen above, We can find the mirror image of a binary tree by swapping the left child and right child of each node. Let us try to formulate the algorithm in a systematic way. 

In the last example, At the second level, each node has only leaf nodes as their children. To find the mirror image at a node at this level, we have just swapped the left and right child at each node at this level. At the root node, we have swapped its both subtrees. As we are swapping each subtree (leaf nodes are also subtree), we can implement this algorithm using recursion.

The algorithm for finding a mirror image of a binary tree can be formulated as follows.

1. Start from the root node.
2. Recursively find the mirror image of the left subtree.
3. Recursively find the mirror image of the right subtree.
4. Swap the left and right subtree.

Implementation of Algorithm in Python

Now we will implement the algorithm to find the mirror image of a binary tree in Python.

from queue import Queue


class BinaryTreeNode:
    def __init__(self, data):
        self.data = data
        self.leftChild = None
        self.rightChild = None


def mirror(node):
    if node is None:
        return None
    mirror(node.leftChild)
    mirror(node.rightChild)
    temp = node.leftChild
    node.leftChild = node.rightChild
    node.rightChild = temp


def insert(root, newValue):
    # if binary search tree is empty, create a new node and declare it as root
    if root is None:
        root = BinaryTreeNode(newValue)
        return root
    # if newValue is less than value of data in root, add it to left subtree and proceed recursively
    if newValue < root.data:
        root.leftChild = insert(root.leftChild, newValue)
    else:
        # if newValue is greater than value of data in root, add it to right subtree and proceed recursively
        root.rightChild = insert(root.rightChild, newValue)
    return root


def inorder(root):
    if root:
        inorder(root.leftChild)
        print(root.data, end=" ")
        inorder(root.rightChild)


root = insert(None, 50)
insert(root, 20)
insert(root, 53)
insert(root, 11)
insert(root, 22)
insert(root, 52)
insert(root, 78)
print("Inorder Traversal of tree before mirroring:")
inorder(root)
mirror(root)
print("\nInorder Traversal of tree after mirroring:")
inorder(root)

Output:

Inorder Traversal of tree before mirroring:
11 20 22 50 52 53 78 
Inorder Traversal of tree after mirroring:
78 53 52 50 22 20 11 

Here, we have created a binary tree node. After that, we defined functions to insert elements to the binary tree. We have also used the in-order tree traversal algorithm to print the elements of the tree before and after finding the mirror image.

Conclusion

In this article, we have implemented an algorithm to find the mirror image of a binary tree. To learn more about other data structures, you can read this article on Linked List in Python. Stay tuned for more articles on implementation of different algorithms in Python.

The post Find the mirror image of a binary tree appeared first on PythonForBeginners.com.

September 21, 2021 12:15 PM UTC

Stack Abuse

Machine Learning: Overfitting Is Your Friend, Not Your Foe

Note: These are the musings of a man - flawed and prone to misjudgement. The point of writing this is to promote a discussion on the topic, not to be right or contrarian. If any glaring mistakes are present in the writing, please let me know.

Let me preface the potentially provocative title with:

It's true, nobody wants overfitting end models, just like nobody wants underfitting end models.

Overfit models perform great on training data, but can't generalize well to new instances. What you end up with is a model that's approaching a fully hard-coded model tailored to a specific dataset.

Underfit models can't generalize to new data, but they can't model the original training set either.

The right model is one that fits the data in such a way that it performs well predicting values in the training, validation and test set, as well as new instances.

Overfitting vs. Data Scientists

Battling overfitting is given a spotlight because it's more illusory, and more tempting for a rookie to create overfit models when they start with their Machine Learning journey. Throughout books, blog posts and courses, a common scenario is given:

"This model has a 100% accuracy rate! It's perfect! Or not. Actually, it just badly overfits the dataset, and when testing it on new instances, it performs with just X%, which is equal to random guessing."

After these sections, entire book and course chapters are dedicated to battling overfitting and how to avoid it. The word itself became stigmatized as a generally bad thing. And this is where the general conception arises:

"I must avoid overfitting at all costs."

It's given much more spotlight than underfitting, which is equally as "bad". It's worth noting that "bad" is an arbitrary term, and none of these conditions are inherently "good" or "bad". Some may claim that overfit models are technically more useful, because they at least perform well on some data while underfit models perform well on no data, but the illusion of success is a good candidate for outweighing this benefit.

For reference, let's consult Google Trends and the Google Ngram Viewer. Google Trends display trends of search data, while the Google Ngram Viewer counts number of occurences of n-grams (sequences of n items, such as words) in literature, parsing through a vast number of books through the ages:

Everybody talks about overfitting and mostly in the context of avoiding it - which oftentimes leads people to a general notion that it's inherently a bad thing.

This is true, to a degree. Yes - you don't want the end model to overfit badly, otherwise, it's practically useless. But you don't arrive at the end model right away - you tweak it numerous times, with various hyperparameters. During this process is where you shouldn't mind seeing overfitting happening - it's a good sign, though, not a good result.

How Overfitting Isn’t as Bad as It’s Made Out to Be

A model and architecture that has the ability to overfit, is more likely to have the ability to generalize well to new instances, if you simplify it (and/or tweak the data).

• Sometimes, it isn't just about the model, as we'll see a bit later.

If a model can overfit, it has enough entropic capacity to extract features (in a meaningful and non-meaningful way) from data. From there, it's either that the model has more than required entropic capacity (complexity/power) or that the data itself isn't enough (very common case).

The reverse statement can also be true, but more rarely. If a given model or architecture underfits, you can try tweaking the model to see if it picks up certain features, but the type of model might just be plain wrong for the task and you won't be able to fit the data with it no matter what you do. Some models just get stuck at some level of accuracy, as they simply can't extract enough features to distinguish between certain classes, or predict values.

In cooking - a reverse analogy can be created. It's better to undersalt the stew early on, as you can always add salt later to taste, but it's hard to take it away once already put in.

In Machine Learning - it's the opposite. It's better to have a model overfit, then simplify it, change hyperparameters, augment the data, etc. to make it generalize well, but it's harder (in practical settings) to do the opposite. Avoiding overfitting before it happens might very well keep you away from finding the right model and/or architecture for a longer period of time.

In practice, and in some of the most fascinating use cases of Machine Learning, and Deep Learning, you'll be working on datasets that you'll be having trouble overfitting. These will be datasets that you'll routinely be underfitting, without the ability of finding models and architectures that can generalize well and extract features.

It's also worth noting the difference between what I call true overfitting and partial overfitting. A model that overfits a dataset, and achieves 60% accuracy on the training set, with only 40% on the validation and test sets is overfitting a part of the data. However, it's not truly overfitting in the sense of eclipsing the entire dataset, and achieving a near 100% (false) accuracy rate, while its validation and test sets sit low at, say, ~40%.

A model that partially overfits isn't one that'll be able to generalize well with simplification, as it doesn't have enough entropic capacity to truly (over)fit. Once it does, my argument applies, though it doesn't guarantee success, as clarified in the proceeding sections.

Case Study - Friendly Overfitting Argument

The MNIST handwritten digits dataset, compiled by Yann LeCun is one of the classical benchmark datasets used for training classification models.

It's also the most overused dataset, potentially ever.

Nothing wrong with the dataset itself - it's actually pretty good, but finding example upon example on the same dataset is boring. At one point - we overfit ourselves looking at it. How much? Here's my attempt at listing the first ten MNIST digits from the top of my head:

5, 0, 4, 1, 9, 2, 2, 4, 3

How did I do?

from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt

# Import and normalize the images, splitting out a validation set
(X_train_full, Y_train_full), (X_test, Y_test) = keras.datasets.mnist.load_data()

X_valid, X_train = X_train_full[:5000]/255.0, X_train_full[5000:]/255.0
Y_valid, Y_train = Y_train_full[:5000], Y_train_full[5000:]

X_test = X_test/255.0

# Print out the first ten digits
fig, ax = plt.subplots(1, 10, figsize=(10,2))
for i in range(10):
    ax[i].imshow(X_train_full[i])
    ax[i].axis('off')
    plt.subplots_adjust(wspace=1) 

plt.show()

Almost there.

I'll use this chance to make a public plea to all content creators to not overuse this dataset beyond the introductory parts, where the simplicity of the dataset can be used to lower the barrier to entry. Please.

Additionally, this dataset makes it hard to build a model that underfits. It's just too simple - and even a fairly small Multilayer Perceptron (MLP) classifier built with an intuitive number of layers and neurons per layer can easily reach upwards of 98% accuracy on the training, testing and validation set. Here's a Jupyter Notebook of a simple MPL achieving ~98% accuracy on both the training, validation and testing sets, which I spun up with sensible defaults.

I haven't even bothered to try tuning it to perform better than the initial setup.

The CIFAR10 and CIFAR100 Datasets

Let's use a dataset that's more complicated than MNIST handwritten digits, and which makes a simple MLP underfit but which is simple enough to let a decently-sized CNN to truly overfit on it. A good candidate is the CIFAR dataset.

There's a 10 classes of images in CIFAR10, and 100 in CIFAR100. Additionally, the CIFAR100 dataset has 20 families of similar classes, which means the network additionally has to learn the minute differences between similar, but different classes. These are known as "fine labels" (100) and "coarse labels" (20) and predicting these is equal to predicting the specific class, or just the family it belongs to.

For instance, here's a superclass (coarse label) and it's subclasses (fine labels):

A cup is a cylinder, similar to a soda can, and some bottles may be too. Since these low-level features are relatively similar, it's easy to chuck them all into the "food container" category, but higher-level abstraction is required to properly guess whether something is a "cup" or a "can".

What makes this job even harder is that CIFAR10 has 6000 images per class, while CIFAR100 has 600 images per class, giving the network less images to learn the ever so subtle differences from. Cups without handles exist, and cans without ridges do too. From a profile - it might not be too easy to tell them apart.

This is where, say, a Multilayer Perceptron simply doesn't have the abstraction power to learn, and it's doomed to fail, horribly underfitting. Convolutional Neural Networks are built based on the Neocognitron, which took hints from neuroscience and the hierarchical pattern recognition that the brain performs. These networks are able to extract features like this, and excel at the task. So much so that they oftentimes overfit badly and can't be used as is in the end - where we typically sacrifice some accuracy for the sake of generalization ability.

Let's train two different network architectures on the CIFAR10 and CIFAR100 dataset as an illustration of my point.

This is also where we'll be able to see how even when a network overfits, it's no guarantee that the network itself will definitely generalize well if simplified - it might not be able to generalize if simplified, though there is a tendency. The network might be right, but the data might not be enough.

In the case of CIFAR100 - just 500 images for training (and 100 for testing) per class is not enough for a simple CNN to really generalize well on the entire 100 classes, and we'll have to perform data augmentation to help it along. Even with data augmentation, we might not get a highly accurate network as there's just so much you can do to the data. If the same architecture performs well on CIFAR10, but not CIFAR100 - it means it simply can't distinguish from some of the more fine-grained details that make the difference between cylindrical objects that we call a "cup", "can" and "bottle", for instance.

The vast majority of advanced network architectures that achieve a high accuracy on the CIFAR100 dataset perform data augmentation or otherwise expand the training set.

Most of them have to, and that's not a sign of bad engineering. In fact - the fact that we can expand these datasets and help networks generalize better is a sign of engineering ingenuity.

Additionally, I'd invite any human to try and guess what these are, if they're convinced that image classification isn't too hard with images as small as 32x32:

Is Image 4 a few oranges? Ping pong balls? Egg yolks? Well, probably not egg yolks, but that requires prior knowledge on what "eggs" are and whether you're likely to find yolks sitting on the table, which a network won't have. Consider the amount of prior knowledge you may have regarding the world and how much it affects what you see.

Importing the Data

We'll be using Keras as the deep learning library of choice, but you can follow along with other libraries or even your custom models if you're up for it.

But first off, let's load it in, separate the data into a training, testing and validation set, normalizing the image values to 0..1:

from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt

# Starting with CIFAR10
(X_train_full, Y_train_full), (X_test, Y_test) = keras.datasets.cifar10.load_data()

X_valid, X_train = X_train_full[:5000]/255.0, X_train_full[5000:]/255.0
Y_valid, Y_train = Y_train_full[:5000], Y_train_full[5000:]

X_test = X_test/255.0

Then, let's visualize some of the images in the dataset to get an idea of what we're up against:

fig, ax = plt.subplots(5, 5, figsize=(10, 10))
ax = ax.ravel()

# Labels come as numbers of [0..9], so here are the class names for humans
class_names = ['Airplane', 'Automobile', 'Bird', 'Cat', 'Deer', 'Dog', 'Frog', 'Horse', 'Ship', 'Truck']

for i in range(25):
    ax[i].imshow(X_train_full[i])
    ax[i].set_title(class_names[Y_train_full[i][0]])
    ax[i].axis('off')
    plt.subplots_adjust(wspace=1) 

plt.show()

Underfitting Multilayer Perceptron

Pretty much no matter what we do, the MLP won't perform that well. It'll definitely reach some level of accuracy based on the raw sequences of information coming in - but this number is capped and probably won't be too high.

The network will start overfitting at one point, learning the concrete sequences of data denoting images, but will still have low accuracy on the training set even when overfitting, which is the prime time to stop training it, since it simply can't fit the data well. Training networks has a carbon footprint, you know.

Let's add in an EarlyStopping callback to avoid running the network beyond the point of common sense, and set the epochs to a number beyond what we'll run it for (so EarlyStopping can kick in).

We'll use the Sequential API to add a couple of layers with BatchNormalization and a bit of Dropout. They help with generalization and we want to at least try to get this model to learn something.

The main hyperparameters we can tweak here are the number of layers, their sizes, activation functions, kernel initializers and dropout rates, and here's a "decently" performing setup:

checkpoint = keras.callbacks.ModelCheckpoint("simple_dense.h5", save_best_only=True)
early_stopping = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)

model = keras.Sequential([
  keras.layers.Flatten(input_shape=[32, 32, 3]),
  keras.layers.BatchNormalization(),
  keras.layers.Dense(75),
    
  keras.layers.Dense((50), activation='elu'),
  keras.layers.BatchNormalization(),
  keras.layers.Dropout(0.1),
    
  keras.layers.Dense((50), activation='elu'),
  keras.layers.BatchNormalization(),
  keras.layers.Dropout(0.1),
    
  keras.layers.Dense(10, activation='softmax')
])

model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Nadam(learning_rate=1e-4),
              metrics=["accuracy"])

history = model.fit(X_train, 
                    Y_train, 
                    epochs=150, 
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint, early_stopping])

Let's see if the starting hypothesis is true - it'll start out learning and generalizing to some extent but will end up having low accuracy on both the training set as well as the testing and validation set, resulting in an overall low accuracy.

For CIFAR10, the network performs "okay"-ish:

Epoch 1/150
1407/1407 [==============================] - 5s 3ms/step - loss: 1.9706 - accuracy: 0.3108 - val_loss: 1.6841 - val_accuracy: 0.4100
...
Epoch 50/150
1407/1407 [==============================] - 4s 3ms/step - loss: 1.2927 - accuracy: 0.5403 - val_loss: 1.3893 - val_accuracy: 0.5122

Let's take a look at the history of its learning:

pd.DataFrame(history.history).plot()
plt.show()

model.evaluate(X_test, Y_test)

313/313 [==============================] - 0s 926us/step - loss: 1.3836 - accuracy: 0.5058
[1.383605718612671, 0.5058000087738037]

The overall accuracy gets up to ~50% and the network gets here pretty quickly and starts plateauing. 5/10 images being correctly classified sounds like tossing a coin, but remember that there are 10 classes here, so if it were randomly guessing, it'd on average guess a single image out of ten. Let's switch to the CIFAR100 dataset, which also necessitates a network with at least a tiny bit more power, as there are less training instances per class, as well as a vastly higher number of classes:

checkpoint = keras.callbacks.ModelCheckpoint("bigger_dense.h5", save_best_only=True)
early_stopping = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)

# Changing the loaded data
(X_train_full, Y_train_full), (X_test, Y_test) = keras.datasets.cifar100.load_data()

# Modify the model
model1 = keras.Sequential([
  keras.layers.Flatten(input_shape=[32, 32, 3]),
  keras.layers.BatchNormalization(),
  keras.layers.Dense(256, activation='relu', kernel_initializer="he_normal"),
    
  keras.layers.Dense(128, activation='relu'),
  keras.layers.BatchNormalization(),
  keras.layers.Dropout(0.1),

  keras.layers.Dense(100, activation='softmax')
])


model1.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Nadam(learning_rate=1e-4),
              metrics=["accuracy"])

history = model1.fit(X_train, 
                    Y_train, 
                    epochs=150, 
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint, early_stopping])

The network performs fairly badly:

Epoch 1/150
1407/1407 [==============================] - 13s 9ms/step - loss: 4.2260 - accuracy: 0.0836 - val_loss: 3.8682 - val_accuracy: 0.1238
...
Epoch 24/150
1407/1407 [==============================] - 12s 8ms/step - loss: 2.3598 - accuracy: 0.4006 - val_loss: 3.3577 - val_accuracy: 0.2434

And let's plot the history of its progress, as well as evaluate it on the testing set (which will likely perform as well as the validation set):

pd.DataFrame(history.history).plot()
plt.show()

model.evaluate(X_test, Y_test)

313/313 [==============================] - 0s 2ms/step - loss: 3.2681 - accuracy: 0.2408
[3.2681326866149902, 0.24079999327659607]

As expected, the network wasn't able to grasp the data well. It ended up having an overfit accuracy of 40%, and an actual accuracy of ~24%.

The accuracy capped at 40% - it wasn't really able to overfit the dataset, even if it overfit some parts of it that it was able to discern given the limited architecture. This model doesn't have the necessary entropic capacity required for it to truly overfit for the sake of my argument.

This model and its architecture simply isn't well suited for this task - and while we could technically get it to (over)fit more, it'll still have issues on the long-run. For instance, let's turn it into a bigger network, which would theoretically let it recognize more complex patterns:

model2 = keras.Sequential([
  keras.layers.Flatten(input_shape=[32, 32, 3]),
  keras.layers.BatchNormalization(),
  keras.layers.Dense(512, activation='relu', kernel_initializer="he_normal"),
    
  keras.layers.Dense(256, activation='relu'),
  keras.layers.BatchNormalization(),
  keras.layers.Dropout(0.1),
    
  keras.layers.Dense(128, activation='relu'),
  keras.layers.BatchNormalization(),
  keras.layers.Dropout(0.1),

  keras.layers.Dense(100, activation='softmax')
])

Though, this doesn't do much better at all:

Epoch 24/150
1407/1407 [==============================] - 28s 20ms/step - loss: 2.1202 - accuracy: 0.4507 - val_loss: 3.2796 - val_accuracy: 0.2528

It's much more complex (density explodes), yet it simply cannot extract much more:

model1.summary()
model2.summary()

Model: "sequential_17"
...
Total params: 845,284
Trainable params: 838,884
Non-trainable params: 6,400
_________________________________________________________________
Model: "sequential_18"
...
Total params: 1,764,324
Trainable params: 1,757,412
Non-trainable params: 6,912

Overfitting Convolutional Neural Network on CIFAR10

Now, let's try doing something different. Switching to a CNN will significantly help with extracting features from the dataset, thereby allowing the model to truly overfit, reaching much higher (illusory) accuracy.

We'll kick out the EarlyStopping callback to let it do its thing. Additionally, we won't be using Dropout layers, and instead try to force the network to learn the features through more layers.

Note: Outside of the context of trying to prove the argument, this would be horrible advice. This is opposite of what you'd want to do by the end. Dropout helps networks generalize better, by forcing the non-dropped neurons to pick up the slack. Forcing the network to learn through more layers it more likely to lead to an overfit model.

The reason I'm purposefully doing this is to allow the network to horribly overfit as a sign of it's ability to actually discern features, before simplifying it and adding Dropout to really allow it to generalize. If it reaches high (illusory) accuracy, it can extract much more than the MLP model, which means we can start simplyfing it.

Let's once again use the Sequential API to build a CNN, firstly on the CIFAR10 dataset:

checkpoint = keras.callbacks.ModelCheckpoint("overcomplicated_cnn_cifar10.h5", save_best_only=True)

model = keras.models.Sequential([
    keras.layers.Conv2D(64, 3, activation='relu', 
                        kernel_initializer="he_normal", 
                        kernel_regularizer=keras.regularizers.l2(l=0.01), 
                        padding='same', 
                        input_shape=[32, 32, 3]),
    keras.layers.Conv2D(64, 3, activation='relu', padding='same'),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(128, 2, activation='relu', padding='same'),
    keras.layers.Conv2D(128, 2, activation='relu', padding='same'),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(256, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(256, 3, activation='relu', padding='same'),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(64, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(64, 3, activation='relu', padding='same'),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Flatten(),    
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Adam(learning_rate=1e-3),
              metrics=["accuracy"])

model.summary()

history = model.fit(X_train, 
                    Y_train, 
                    epochs=150,
                    batch_size=64,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint])

Awesome, it overfit pretty quickly! Within just a few epochs, it started overfitting the data, and by epoch 31, it got up to 98%, with a lower validation accuracy:

Epoch 1/150
704/704 [==============================] - 149s 210ms/step - loss: 1.9561 - accuracy: 0.4683 - val_loss: 2.5060 - val_accuracy: 0.3760
...
Epoch 31/150
704/704 [==============================] - 149s 211ms/step - loss: 0.0610 - accuracy: 0.9841 - val_loss: 1.0433 - val_accuracy: 0.6958

Since there are only 10 output classes, even though we tried overfitting it a lot by creating an unnecessarily big CNN, the validation accuracy is still fairly high.

Simplifying the Convolutional Neural Network on CIFAR10

Now, let's simplify it to see how it'll fare with a more reasonable architecture. We'll add in BatchNormalization and Dropout as both help with the generalization:

checkpoint = keras.callbacks.ModelCheckpoint("simplified_cnn_cifar10.h5", save_best_only=True)
early_stopping = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)

model = keras.models.Sequential([
    keras.layers.Conv2D(32, 3, activation='relu', kernel_initializer="he_normal", kernel_regularizer=keras.regularizers.l2(l=0.01), padding='same', input_shape=[32, 32, 3]),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(32, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Conv2D(64, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(64, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.5),
    
    keras.layers.Flatten(),    
    keras.layers.Dense(32, activation='relu'),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(10, activation='softmax')
])

model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Adam(learning_rate=1e-3),
              metrics=["accuracy"])

model.summary()

history = model.fit(X_train, 
                    Y_train, 
                    epochs=150,
                    batch_size=64,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint, early_stopping])

This model has a (modest) count of 323,146 trainable parameters, compared to 1,579,178 from the previous CNN. How does it perform?

Epoch 1/150
704/704 [==============================] - 91s 127ms/step - loss: 2.1327 - accuracy: 0.3910 - val_loss: 1.5495 - val_accuracy: 0.5406
...
Epoch 52/150
704/704 [==============================] - 89s 127ms/step - loss: 0.4091 - accuracy: 0.8648 - val_loss: 0.4694 - val_accuracy: 0.8500

It actually achieves a pretty decent ~85% accuracy! Occam's Razor strikes again. Let's take a look at some of the results:

y_preds = model.predict(X_test)
print(y_preds[1])
print(np.argmax(y_preds[1]))

fig, ax = plt.subplots(6, 6, figsize=(10, 10))
ax = ax.ravel()

for i in range(0, 36):
    ax[i].imshow(X_test[i])
    ax[i].set_title("Actual: %s\nPred: %s" % (class_names[Y_test[i][0]], class_names[np.argmax(y_preds[i])]))
    ax[i].axis('off')
    plt.subplots_adjust(wspace=1)
    
plt.show()

The main misclassifications are two images in this small set - a dog was misclassified as a deer (respectable enough), but a closeup of an emu bird was classified as a cat (funny enough so we'll let it slide).

Overfitting Convolutional Neural Network on CIFAR100

What happens when we go for the CIFAR100 dataset?

checkpoint = keras.callbacks.ModelCheckpoint("overcomplicated_cnn_model_cifar100.h5", save_best_only=True)
early_stopping = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)

model = keras.models.Sequential([
    keras.layers.Conv2D(32, 3, activation='relu', kernel_initializer="he_normal", kernel_regularizer=keras.regularizers.l2(l=0.01), padding='same', input_shape=[32, 32, 3]),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(32, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(64, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(64, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Conv2D(64, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(64, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    
    keras.layers.Flatten(),    
    keras.layers.Dense(256, activation='relu'),
    keras.layers.BatchNormalization(),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.BatchNormalization(),
    
    keras.layers.Dense(100, activation='softmax')
])

model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Adam(learning_rate=1e-3),
              metrics=["accuracy"])

model.summary()

history = model.fit(X_train, 
                    Y_train, 
                    epochs=150,
                    batch_size=64,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint])

Epoch 1/150
704/704 [==============================] - 97s 137ms/step - loss: 4.1752 - accuracy: 0.1336 - val_loss: 3.9696 - val_accuracy: 0.1392
...
Epoch 42/150
704/704 [==============================] - 95s 135ms/step - loss: 0.1543 - accuracy: 0.9572 - val_loss: 4.1394 - val_accuracy: 0.4458

Wonderful! ~96% accuracy on the training set! Don't mind the ~44% validation accuracy just yet. Let's simplify the model real quick to get it to generalize better.

Failure to Generalize After Simplification

And this is where it becomes clear that the ability to overfit doesn't guarantee that the model could generalize better when simplified. In the case of CIFAR100, there aren't many training instances per class, and this will likely prevent a simplified version of the previous model learn well. Let's try it out:

checkpoint = keras.callbacks.ModelCheckpoint("simplified_cnn_model_cifar100.h5", save_best_only=True)
early_stopping = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)

model = keras.models.Sequential([
    keras.layers.Conv2D(32, 3, activation='relu', kernel_initializer="he_normal", kernel_regularizer=keras.regularizers.l2(l=0.01), padding='same', input_shape=[32, 32, 3]),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(32, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Conv2D(64, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(64, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2D(128, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.5),
    
    keras.layers.Flatten(),    
    keras.layers.Dense(256, activation='relu'),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(100, activation='softmax')
])

model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Adam(learning_rate=1e-3),
              metrics=["accuracy"])

history = model.fit(X_train, 
                    Y_train, 
                    epochs=150,
                    batch_size=64,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint, early_stopping])

Epoch 1/150
704/704 [==============================] - 96s 135ms/step - loss: 4.4432 - accuracy: 0.1112 - val_loss: 3.7893 - val_accuracy: 0.1702
...
Epoch 48/150
704/704 [==============================] - 92s 131ms/step - loss: 1.2550 - accuracy: 0.6370 - val_loss: 1.7147 - val_accuracy: 0.5466

It's plateauing and can't really get to generalize the data. In this case, it might not be the model's fault - maybe it's just right for the task, especially given the high accuracy on the CIFAR10 dataset, which has the same input shape and similar images in the dataset. It appears that the model can be reasonably accurate with the general shapes, but not the distinction between fine shapes.

The simpler model actually performs better than the more complicated one in terms of validation accuracy - so the more complex CNN doesn't get these fine details much better at all. Here, the problem most likely lies in the fact that there are only 500 training images per class, which really isn't enough. In the more complex network, this leads to overfitting, because there's not enough diversity - when simplified to avoid overfitting, this causes underfitting as again, there's no diversity.

This is why the vast majority of the papers linked before, and the vast majority of networks augment the data of the CIFAR100 dataset.

It's genuinely not a dataset for which it's easy to get high accuracy on, unlike the MNIST handwritten digits dataset, and a simple CNN like we're building probably won't cut it for high accuracy. Just remember the number of quite specific classes, how uninformative some of the images are, and just how much prior knowledge humans have to discern between these.

Let's do our best by augmenting a few images and artificially expanding the training data, to at least try to get a higher accuracy. Keep in mind that the CIFAR100 is, again, a genuinely difficult dataset to get high accuracy on with simple models. The state of the art models use different and novel techniques to shave off errors, and many of these models aren't even CNNs - they're Transformers.

If you'd like to take a look at the landscape of these models, PapersWithCode has done a beautiful compilation of papers, source code and results.

Data Augmentation with Keras' ImageDataGenerator Class

Will data augmentation help? Usually, it does, but with a serious lack of training data like we're facing, there's just so much you can do with random rotations, flipping, cropping, etc. If an architecture can't generalize well on a dataset, you'll likely boost it via data augmentation, but it probably won't be a whole lot.

That being said, let's use Keras' ImageDataGenerator class to try and generate some new training data with random changes, in hopes of improving the model's accuracy. If it does improve, it shouldn't be by a huge amount, and it'll likely get back to partially overfitting the dataset without an ability to either generalize well or fully overfit the data.

Given the constant random variations in the data, the model is less likely to overfit on the same number of epochs, as the variations make it keep adjusting to "new" data. Let's run it for, say, 300 epochs, which is significantly more than the rest of the networks we've trained. This is possible without major overfitting, again, due to the random modifications made to the images while they're flowing in:

checkpoint = keras.callbacks.ModelCheckpoint("augmented_cnn.h5", save_best_only=True)

model = keras.models.Sequential([
    keras.layers.Conv2D(64, 3, activation='relu', kernel_initializer="he_normal", kernel_regularizer=keras.regularizers.l2(l=0.01), padding='same', input_shape=[32, 32, 3]),
    keras.layers.Conv2D(64, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Conv2D(128, 2, activation='relu', padding='same'),
    keras.layers.Conv2D(128, 2, activation='relu', padding='same'),
    keras.layers.Conv2D(128, 2, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Conv2D(256, 3, activation='relu', padding='same'),
    keras.layers.Conv2D(256, 3, activation='relu', padding='same'),
    keras.layers.Conv2D(256, 3, activation='relu', padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPooling2D(2),
    keras.layers.Dropout(0.4),
    
    keras.layers.Flatten(),    
    keras.layers.Dense(512, activation='relu'),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(100, activation='softmax')
])

    
train_datagen = ImageDataGenerator(rotation_range=30,
        height_shift_range=0.2,
        width_shift_range=0.2,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True,
        vertical_flip=True,
        fill_mode='nearest')

valid_datagen = ImageDataGenerator()

train_datagen.fit(X_train)
valid_datagen.fit(X_valid)

train_generator = train_datagen.flow(X_train, Y_train, batch_size=128)
valid_generator = valid_datagen.flow(X_valid, Y_valid, batch_size=128)

model.compile(loss="sparse_categorical_crossentropy",
              optimizer=keras.optimizers.Adam(learning_rate=1e-3, decay=1e-6),
              metrics=["accuracy"])

history = model.fit(train_generator, 
                    epochs=300,
                    batch_size=128,
                    steps_per_epoch=len(X_train)//128,
                    validation_data=valid_generator,
                    callbacks=[checkpoint])

Epoch 1/300
351/351 [==============================] - 16s 44ms/step - loss: 5.3788 - accuracy: 0.0487 - val_loss: 5.3474 - val_accuracy: 0.0440
...
Epoch 300/300
351/351 [==============================] - 15s 43ms/step - loss: 1.0571 - accuracy: 0.6895 - val_loss: 2.0005 - val_accuracy: 0.5532

The model is performing with ~55% on the validation set, and is still overfitting the data partially. The val_loss has stopped going down, and is quite rocky, even with a higher batch_size.

This network simply can't learn and fit the data with high accuracy, even though it variations on it do have the entropic capacity to overfit the data.

Conclusion?

Overfitting isn't inherently a bad thing - it's just a thing. No, you don't want overfit end-models, but it shouldn't be treated as the plague and can even be a good sign that a model could perform better given more data and a simplification step. This isn't guaranteed, by any means, and the CIFAR100 dataset has been used as an example of a dataset that's not easy to generalize well to.

The point of this rambling is, again, not to be contrarian - but to incite discussion on the topic, which doesn't appear to be taking much place.

Who am I to make this claim?

Just someone who sits home, practicing the craft, with a deep fascination towards tomorrow.

Do I have the ability to be wrong?

Very much so.

How should you take this piece?

Take it as you may - think for yourself if it makes sense or not. If you don't think I'm out of my place for noting this, let me know. If you think I'm wrong on this - by all means, please feel let me know and don't mince your words. :)

September 21, 2021 10:30 AM UTC

Wing Tips

Debug Python Code Run by Docker Compose with Wing Pro

This Wing Tip describes how to configure Docker Compose so that Python code running on selected container services can be debugged with Wing Pro. This makes it easy to develop and debug containerized applications written in Python.

Configuration

To configure Wing to use New Project in the Project menu. After selecting your source directory, choose Use Existing Python on the second dialog page, and then select Cluster:

Next create a new cluster configuration by pressing the New button. This displays the cluster configuration dialog. You will need to enter an identifier to use within Wing and point it at the docker-compose.yml file for the cluster. You will also need to select the main service to use as the default place to run your Python Shell and unit test processes:

Once you have created your cluster configuration, submit the New Project dialog to complete your project setup.

Working with the Cluster

You can now control your cluster from Wing's Containers tool, found in the Tools menu. This tool lists the services in your cluster and their status. You can right-click on items here or use the Options menu to build, start, debug, and stop your cluster:

Debug processes, unit tests, Wing's integrated Python Shell and OS Commands can all be run in context of the cluster, or optionally instead within isolated containers that match the cluster configuration but run without launching the whole cluster.

See Using Wing Pro with Docker Compose for more information working with clusters in Wing Pro.

That's it for now! We'll be back soon with more Wing Tips for Wing Python IDE.

As always, please don't hesitate to email support@wingware.com if you run into problems, have any questions, or have topic suggestions for future Wing Tips!

September 21, 2021 01:00 AM UTC

Brett Cannon

(Not) unravelling generator expressions

In this post on Python&aposs syntactic sugar, I want to try to tackle generator expressions. If you look at the language definition for generator expressions you will see that it says, "[a] generator expression yields a new generator object" for what is specified (which is essentially a compact for loop with an expression for the body). So what does that look like if you take away the Python "magic" and unravel it down to its core Python semantics?

The bytecode

Let&aposs take the following example:

def spam():
    return (b for b in a)

The bytecode for this is:

  1           0 LOAD_CONST               1 (<code object <genexpr> at 0x10076b500, file "<stdin>", line 1>)
              2 LOAD_CONST               2 (&aposspam.<locals>.<genexpr>&apos)
              4 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (a)
              8 GET_ITER
             10 CALL_FUNCTION            1
             12 RETURN_VALUE

Disassembly of <code object <genexpr> at 0x10076b500, file "<stdin>", line 1>:
  1           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                10 (to 14)
              4 STORE_FAST               1 (b)
              6 LOAD_FAST                1 (b)
              8 YIELD_VALUE
             10 POP_TOP
             12 JUMP_ABSOLUTE            2
        >>   14 LOAD_CONST               0 (None)
             16 RETURN_VALUE

You may notice a couple of things that are interesting about this:

1. The generator expression is very much just a for loop in a generator.
2. The generator expression is stored as a constant in the function.
3. a gets explicitly passed into the generator expression.

The semantics

The explicit passing of a is the surprising bit in how generator expressions work, but it actually makes sense when you read the explanation as to why this occurs:

... the iterable expression in the leftmost for clause is immediately evaluated, so that an error produced by it will be emitted at the point where the generator expression is defined, rather than at the point where the first value is retrieved. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable.

So by passing a in, the code for a is evaluated at the time of creation of the generator expression, not at the time of execution. That way if there&aposs an error with that part of the code the traceback will help you find where it was defined and not simply to where the generator expression happened to be run. Since subsequent for loops in the generator expression may rely on the loop variant in the first clause, you can&apost eagerly evaluate any other parts of the expression.

The unravelling

There&aposs a couple of details that required  to make unravelling a generator expression successful, so I&aposm going to build up a running example to cover all the various cases.

With only one for loop

Let&aposs start with (c for b in a) where c can be some expression. To unravel this we need to make a generator which takes in a as an argument to guarantee it is eagerly evaluated where the generator expression is defined.

def _gen_exp(_leftmost_iterable):
    for b in _leftmost_iterable:
        yield c
        
_gen_exp(a)

We end up with a generator function which takes a single argument for the leftmost iterable. Let&aposs see what this looks like in some code that would use the generator expression:

def spam(a, b):
    func(arg=(str(b) for b in a))

This would then unravel to:

def spam(a, b):
    def _gen_exp(_leftmost_iterable):
        for b in _leftmost_iterable:
            yield str(b)
    
    func(arg=_gen_exp(a))

With multiple for loops

Now let&aposs toss in another for loop: (e for b in a for d in c). This unravels to:

def _gen_expr(_leftmost_iterable):
    for b in _leftmost_iterable:
        d in c:
            yield e

Since only the leftmost iterable is evaluated eagerly we can rely on the scoping rules for closures to get all of the other variables from the call site implicitly (this is where Python&aposs simple namespace system comes in handy).

Putting this into an example like:

def spam():
    x = range(2)
    y = range(3)
    return ((a, b) for a in x for b in y)

lead to an unravelling of:

def spam():
    x = range(2)
    y = range(3)
    
    def _gen_exp(_leftmost_iterable):
        for a in _leftmost_iterable:
            for b in y:
                yield (a, b)
                
    return _gen_exp(x)

The generator expression needs x passed in because it&aposs the leftmost iterable, but everything else is captured by the closure.

Assignment expressions

Let&aposs make life complicated and throw in an assignment expression:

def spam():
    list(b := a for a in range(2))
    return b

The bytecode for this becomes:

  2           0 LOAD_GLOBAL              0 (list)
              2 LOAD_CLOSURE             0 (b)
              4 BUILD_TUPLE              1
              6 LOAD_CONST               1 (<code object <genexpr> at 0x1008393a0, file "<stdin>", line 2>)
              8 LOAD_CONST               2 (&aposspam.<locals>.<genexpr>&apos)
             10 MAKE_FUNCTION            8 (closure)
             12 LOAD_GLOBAL              1 (range)
             14 LOAD_CONST               3 (2)
             16 CALL_FUNCTION            1
             18 GET_ITER
             20 CALL_FUNCTION            1
             22 CALL_FUNCTION            1
             24 POP_TOP

  3          26 LOAD_DEREF               0 (b)
             28 RETURN_VALUE

Disassembly of <code object <genexpr> at 0x1008393a0, file "<stdin>", line 2>:
  2           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                14 (to 18)
              4 STORE_FAST               1 (a)
              6 LOAD_FAST                1 (a)
              8 DUP_TOP
             10 STORE_DEREF              0 (b)
             12 YIELD_VALUE
             14 POP_TOP
             16 JUMP_ABSOLUTE            2
        >>   18 LOAD_CONST               0 (None)
             20 RETURN_VALUE

The key thing to notice is the various *_DEREF opcodes which are what CPython uses to load/store nonlocal variables.

Now we could just add a nonlocal statement to our unravelled generator expression and assume we are done, but there is one issue to watch out for: has the variable previously been defined in the enclosing scope? If the variable doesn&apost exist when the scope with the nonlocal is defined (technically the compiler walking the AST has not seen the variable yet), Python will raise an exception: SyntaxError: no binding for nonlocal &aposb&apos found.

Python gets to take a shortcut when it comes to a generator expression with an assignment expression and simply consider the nonlocal as implicit without regards as to whether the variable was previously defined. But we don&apost get to cheat, and that means we may have to define the variable with a dummy value to make the CPython compiler happy.

But we also have to deal with whether the generator expression is ever run or runs but never sets b (i.e. the iterable has a length of 0). In the example that would raise UnboundLocalError: local variable &aposb&apos referenced before assignment. To replicate that we need to delete b if it never gets set appropriately.

What all of this means is our example unravels to:

def spam():
    b = _PLACEHOLDER
    
    def _gen_exp(_leftmost_iterable):
        nonlocal b
        for a in _leftmost_iterable:
            yield (b := a)
            
    list(_gen_expr(range(2)))
    if b is _PLACEHOLDER:
        del b
    return b

But remember, we only want to do any of this nonlocal work if there are assignment expressions to worry about.

The best laid plans ...

I actually wrote this entire post thinking I had solved the unravelling of generator expressions, and then I realized assignment expressions thwarted me in the end. Consider the following example:

def spam():
    return ((b := x for x in range(5)), b)

If you run that example you end up with UnboundLocalError: local variable &aposb&apos referenced before assignment. Now let&aposs unravel this:

def spam():
    b = _PLACEHOLDER
    def _gen_expr(_leftmost_iterable):
        nonlocal b
        for x in _leftmost_iterable:
            yield (b := x)
            
    return _gen_expr(range(5)), b

Unfortunately calling this function succeeds. And since del is a statement there&aposs no way to insert ourselves into that expression to prevent b from being resolved. This means we cannot unravel assignment expressions. 🙁

So while I thought I had unravelled assignment expressions, it seems that in the end I was unsuccessful. But I have decided to publish this anyway to show how I typically approach unravelling a bit of syntax and sometimes are unable to do it.

Aside: what came first, the expression or the comprehension?

If you have not been programming in Python for more than 15 years you may think generator expressions came first, then list comprehensions. But actually it&aposs the other way around: list comprehensions were introduced in Python 2.0 and generator expressions came in Python 2.4. This is because generators were introduced in Python 2.2 (thanks to the inspiration from Icon), and so the possibility of even having generator expressions didn&apost exist when list comprehensions came into existence (thanks to the inspiration from Haskell).

September 21, 2021 12:45 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AI Pool
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • Codementor
  • CodersLegacy
  • Coding Diet
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • Dave Beazley
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Denis Kurov
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Learn PyQt
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lintel Technologies
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Kennedy
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python Morsels
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Reuven Lerner
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Papers
  • The Three of Wands
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github