Planet Python

Last update: June 07, 2023 04:42 AM UTC

June 06, 2023

PyCoder’s Weekly

Issue #580 (June 6, 2023)

#580 – JUNE 6, 2023
View in Browser »

Companies like GitLab, Snowflake, and Slack Scan Their Code for Vulnerabilities Using Semgrep

Scan your code and dependencies for security vulnerabilities for free with Semgrep - the trusted OSS tool used by top companies like GitLab, Snowflake, and Slack. No security expertise needed, simply add your project and let Semgrep do the work in just minutes →
SEMGREP sponsor

Discussions

Articles & Tutorials

Write Code as if Failure Doesn’t Exist

Temporal is an open source programming model that can simplify your code, make your applications more reliable, and allow you to deliver more features faster. Check out the Temporal Python SDK Developer’s Guide to learn more and get started →
TEMPORAL sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #580.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

June 06, 2023 07:30 PM UTC

Real Python

Mazes in Python Part 1: Building and Visualizing

If you’re up for a little challenge and would like to take your programming skills to the next level, then you’ve come to the right place! In this hands-on video course, you’ll practice object-oriented programming, among several other good practices, while building a cool maze solver project in Python.

This is the first part in a two-part series. Throughout the series, you’ll go step by step through the guided process of building a complete and working project. This will include reading a maze from a binary file, visualizing it using scalable vector graphics (SVG), and finding the shortest path from the entrance to the exit.

In part one of the series, you’ll learn how to:

• Use an object-oriented approach to represent the maze in memory
• Visualize the maze and its solution using scalable vector graphics (SVG)

In the next part of the series, you’ll complete the project by creating a binary storage format for mazes and solving mazes using NetworkX.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

June 06, 2023 02:00 PM UTC

Mike Driscoll

Announcing The Python Show Podcast

The Python Show Podcast is something that I have thought about creating for several years. I have had several people tell me or ask me if I would do a podcast.

Of course, the problem with creating a podcast is choosing your niche. I don’t want to sound like everyone else. On the other hand, my blog, Mouse vs Python, has been popular and different enough that many people from around the world have found it helpful.

So I decided that I was overthinking things. My passion has been and will continue to be around Python programming. It makes sense for my podcast to also be about Python.

The post Announcing The Python Show Podcast appeared first on Mouse Vs Python.

June 06, 2023 01:49 PM UTC

Stack Abuse

Finding Numbers in Various Data Types in Python

Introduction

When working with Python, we often have to deal with data in the form of numbers or words. Sometimes, words and numbers are stored together, and our needs compel us to separate numbers from words.

In this article, we'll explain how to define words and numbers in Python. Then, we'll see how to separate numbers from words, in case they're stored together, using different methods and in various situations.

Strings, Integers, and Floats in Python

In Python, strings, integers, and floats are fundamental data types used to represent different kinds of values in a program.

Integers represent whole numbers without any decimal part and can be positive or negative. Here's how to define an integer in Python:

# Create a variable expressing an integer number
age = 15

We've created a variable called age and assigned it the value of 15. To verify the type of a variable in Python, we can use the built-in function type(). This is how it works:

# Show the type
type(age)

And we get:

int

So, we pass the variable age to the built-in function type() and it tells us that this is an integer, as we expected.

To express numbers, we can also use the float type. A float, which is short for floating-point numbers, represents real numbers with decimal points. They are generally suitable for scientific calculations, but not only. Here's how we define a float in Python:

# Create a variable expressing a float number
pi = 3.14

And again, to check the type of our variable:

# Show the type
type(pi)

And we get:

float

Strings are a sequence of characters enclosed in double quotes or single quotes. They are used to store text or, more generally, any kind of characters. This means that a string can even contain numbers, which is the focus of this article. So, for example, a string can be:

# Create a string with my name
my_name = "Federico"
# Show the type
type(my_name)

And we get:

str

But it can also be:

# Create a variable with text and number
federico_car = 'Federico has 1 car'

Finally, pay attention: we said that a string in Python is any character enclosed in quotes. This means that the type of the following:

# Create a string variable expressing a number
age = '15'

is str.

Now, with this overview in mind, let's see some methods to intercept numbers inside strings.

Methods to Find Numbers in Strings

Let's see an overview of some methods we can use to check if a string contains numbers.

The int() and float() Methods

The easiest ways to transform a string into a number are through the int() and float() methods. Let's see how we can use them.

Suppose we expressed our age as a string, but we want it as an integer. We can do it like so:

# Create a string variable expressing an integer number
age = '30'
# Transform string into integer type
age_int = int(age)
# Show type
type(age_int)

And we have:

int

So, we've defined the variable age as a string. Then, we passed it as the argument of the method int() and it transformed the string into an integer.

Now, suppose we have expressed a price as a string, but we want to convert it to a float. We can do it like so:

# Create a string variable expressing a decimal number
price = "34.99"

# Transform string to float type
price_float = float(price)

# Show type
type(price_float)

And we have:

float

Similarly, as before, we pass a string to the method float() and it transforms it into a float.

Now, these methods are easy to use but have a significant limitation: they can convert strings into numbers only if the value inside the quotes is a number. So, to get an understanding, consider the following example:

# Create a string with text and numbers
apples = "21 apples"

# Transform string to integer
apples_converted = int(apples)

# Show type
type(apples_converted)

And we get:

ValueError: invalid literal for int() with base 10: '21 apples'

This error means that we're trying to convert a string to an integer, but the string cannot be parsed as a valid integer. Of course, the same issue occurs if we parse a string containing only text or if we use the float() method.

Now, a question may arise: what if the text in the string expresses an integer and we want to convert it to a float? And how about vice-versa? Let's examine both scenarios:

# Create a string expressing a decimal
price = "30.5"

# Transform string into an integer
price_int = int(price)

# Show type
type(price_int)

And we get:

ValueError: invalid literal for int() with base 10

So, we have expressed the price of an object as a decimal number (although the type is a string!) and tried to convert it into an integer. This is not possible, as the error indicates.

Now, let's see the other case:

# Create a string expressing an integer
price = "30"

# Transform string into a float
price_float = float(price)

# Show type
type(price_float)

And we get:

float

So, we can convert a string that expresses a whole number into a float. In fact, if we want to see how Python expresses this number as a float, we can print it:

# Print transformed variable
print(price_float)

And, as we might expect, we get:

30.0

Now, these two methods are very basic and have some limitations, as we've seen. This is why we need to learn other methods to solve our problem in more general situations.

The isdigit() Method

The isdigit() method checks if all the characters in a string are digits (0-9). It returns True if the string contains only digits, and False otherwise. So, let's see a couple of examples:

# Create string with text and numbers
letters_and_numbers = "Hello123"

# Create string with only text
letters = "Hello World"

# Create string with only numbers
numbers = "123456"

Now, let's use the isdigit() method:

# Apply isdigit() method and print
print(letters_and_numbers.isdigit())
print(letters.isdigit())
print(numbers.isdigit())

And we get:

False
False
True

So, to use this method, we write the name of the variable we're verifying and we add .isdigit(). As we expected, the results show that only the variable numbers contains characters that are all digits.

This method is good, but we can do better in Python. Listing all the printed values, in fact, may become difficult to read. So, we can improve the above code like so:

# Apply isdigit() method and print, showing variables and results
print(f"Is {letters_and_numbers} an only digit string? {letters_and_numbers.isdigit()}")
print(f"Is {letters} an only digit string? {letters.isdigit()}")
print(f"Is {numbers} an only digit string? {numbers.isdigit()}")

And we get:

Is Hello123 an only digit string? False
Is Hello World an only digit string? False
Is 123456 an only digit string? True

So, inside print(), we can use f before the double quotes to insert the variables inside the curly brackets {}. This allows Python to return the actual value of the variable passed through the curly brackets. Then, knowing that the isdigit() method returns a boolean (True or False), the above code provides more readable results.

Now, it would be beneficial if this method could detect numbers in strings containing both digits and other characters. To achieve this, we can create a function like so:

# Create a function that detects digits in strings
def contains_number(string: str) -> bool:
    return any(char.isdigit() for char in string)

Next, we pass the letters_and_numbers variable to the contains_number() function and observe the result:

# Invoke the function, passing 'letters_and_numbers' as an argument
contains_number(letters_and_numbers)

And we get:

True

Exactly as we wanted: We were able to intercept digits in a variable that contains different kinds of characters. Now, let's explain this function step by step:

1. def contains_number(string: str) -> bool means that we are defining a function called contains_number() where we expect the argument, which we generically called string, to be a string (:str). Then, we know that the function will return a boolean (->bool). Note that this notation, called "Type Hints", is available from Python 3 onwards, and it is not mandatory. We could have written def contains_number(string): and the function would work correctly. Type Hints are just a useful way to inform the user on what types to expect when dealing with functions (and classes), so it's a kind of "a facilitator".

2. Now, let's explain any(char.isdigit() for char in string) which is what the function returns. First, we have created a generator expression (a generator is a special type of object in Python that allows us to generate a sequence of values dynamically, without needing to store all the values in memory at once) with char.isdigit() for char in string. This generates a sequence of Boolean values, indicating whether each character in the string is a digit. In particular, for char in string iterates over each character char in the argument string, passed to the function. Then, char.isdigit() checks if the character char is a digit. Finally, the any() function is a built-in Python function that takes an iterable as an argument and returns True if at least one element in the iterable is True. It returns False if all elements are False. So, in conclusion, any(char.isdigit() for char in string) evaluates to True if at least one character in the string is a digit, and False otherwise.

Now, let's see other methods.

Using Regular Expressions

Another method we can use to find if a string contains a number is through regular expressions (also called "regex"). Regular expressions are a sequence of characters that help us match or find patterns in text. Here's how we can use this Python module for our purposes:

import re

# Create strings
letters_and_numbers = "Hello123 0.3"
letters = "Hello World"
numbers = "1 2 3 4 5 6 0.5"

# Use regex to intercept numbers and print results
print(bool(re.search(r'\d', letters_and_numbers)))
print(bool(re.search(r'\d', letters)))
print(bool(re.search(r'\d', numbers)))

And we get:

True
False
True

So, first we need to import the re module to use regular expressions. Then, we can use the re.search() method for each variable to check for any digits. In regular expressions, \d represents any digit character. Next, we apply the bool() method, which returns a boolean value. If there are any digits in the string we are checking, we get True.

Note that, compared to the previous method, this one identifies numbers without using a function. This method also detects the decimal number in the string letters_and_numbers, unlike the previous method that only detects digits (numbers from 0 to 9). We'll explain more about this in the next paragraph.

Now, let's examine the last method.

The isnumeric() Method

The isnumeric() method works exactly like the isdigit() method: it returns True if all characters in the string are numbers. The difference between the two is that isnumeric() can identify a wider range of numbers, such as floats, fractions, superscripts, subscripts, and more. So, if we're aware we're searching for numbers that can be in different forms, then isnumeric() should be preferred.

On the coding side, we use it similarly to the isdigit() method. Let's see a simple example:

# Create strings
letters_and_numbers = "Hello123"
letters = "Hello World"
numbers = "1 2 3 4 5 6 0.5"

# Apply isnumeric() method and print results
print(letters_and_numbers.isnumeric())
print(letters.isnumeric())
print(numbers.isnumeric())

And we get:

False
False
True

Now, let's see some more advanced examples we may encounter while programming in Python.

Advanced Manipulation Examples

Now we want to show how we can use the methods we've seen above to intercept numbers in strings in more practical situations, for example when analyzing lists, dictionaries, and data frames.

Finding Numbers in Strings in Lists

Consider we have a list where we have stored some strings containing both numbers and text. We could extract the numbers with regular expressions like so:

import re

# Create a list with strings expressing text and numbers
data = ['I have 10 apples', 'There are 5 bananas', 'I will buy one apple']

# Create a function to retrieve numbers in strings
def extract_numbers(string):
    return re.findall(r'\d+', string)

# Iterate over the list, extract numbers, and print results
for string in data:
    numbers = extract_numbers(string)
    print(f"Numbers in string: {string} - {numbers}")

And we get:

Numbers in string: I have 10 apples - ['10']
Numbers in string: There are 5 bananas - ['5']
Numbers in string: I will buy one apple - []

So, here we report the number associated with the string, if present. The only differences between the previous example on regular expressions are:

1. We used re.findall(). This method takes two arguments: pattern and string. It searches for all occurrences of the pattern within the string and returns a list of all matched substrings.
2. In this case, the pattern is represented by \d+, which matches one or more consecutive digits in the string using regex.

So, we have stored numbers and text in some strings in a list called data. We have created a function called extract_numbers() that intercepts all the consecutive digits in the string through the method re.findall(), thanks to regex. We then iterate through the list with a for loop and invoke the function extract_numbers() so that all the strings in the list are checked. Then, the code prints the strings themselves and the numbers intercepted (if any).

Finding Numbers in Strings in Dictionaries

Now, suppose we have a shopping list stored in a dictionary where we report some fruit and their respective prices. We want to see if the price is expressed as a number. We can do it like so:

# Create function to intercept numbers in strings
def contains_number(string: str) -> bool:
    return any(char.isnumeric() for char in string)

# Create a dictionary
shopping_list = {
 'Banana': '1',
 'Apple': 'Five',
 'Strawberry': '3.5',
 'Pear': '3',
}

# Iterate over dictionary and print results
for key, value in shopping_list.items():
    if contains_number(value):
         print(f"Value for '{key}' contains a number: {value}")
    else:
         print(f"Value for '{key}' does not contain a number: {value}")

And we get:

Value for 'Banana' contains a number: 1
Value for 'Apple' does not contain a number: Five
Value for 'Strawberry' contains a number: 3.5
Value for 'Pear' contains a number: 3

So, we have created a function contains_number() as we have seen before, but here we've used the isnumeric() method (we have a decimal). We then store the prices of some fruits in a dictionary called shopping_list. With shopping_list.items(), we access the keys and values of the dictionary, and check if the values are numeric by invoking the function contains_number(). Finally, thanks to an if statement, we can separate the strings containing numbers from those containing only text, and print the results.

Finding Numbers in Strings in Pandas Data Frames

In Python, we can store data in data frames, which are collections of columns and rows (similar to Excel sheets, for simplicity). Data frames can be manipulated with a library called pandas in Python.

Suppose we want to create a column (we call a single column of a data frame a "Pandas series") where we have stored the price of an object from different suppliers on Amazon:

import pandas as pd

# Create dictionary
data = {'Amazon_prices': ['10', '8', '9.2', 'eleven', 'seven']}

# Transform dictionary into data frame
df = pd.DataFrame(data)

# Print data frame
print(df)

And we have:

        Amazon_prices
0           10
1            8
2           9.2
3          eleven
4           seven

So, we have stored some data in a dictionary called data. Then, with the pd.DataFrame() method, we've converted the dictionary into a Pandas data frame called df.

At this point, we can use the str.contains() method from the Pandas library, which is useful for checking patterns in strings. We can use regex to define the pattern like so:

# Intercept numbers in column
numeric_values = df['Amazon_prices'].str.contains(r'\d+', regex=True)

With the above code, we are checking if the column Amazon_prices contains numbers, thanks to regex. With df['Amazon_prices'], we are selecting the column of the data frame. Then, the .str.contains() method checks if we have at least one number in the strings, thanks to regex, as seen before. Finally, the regex=True activates the use of regex.

We can then create a new column to add to our data frame with the resulting booleans like so:

# Create a new column for data frame
df['IsNumeric'] = numeric_values

So, IsNumeric is the new column of our data frame, containing the booleans. We can now print the modified data frame with print(df) and we get:

   Amazon_prices  IsNumeric
0           10       True
1            8       True
2           9.2      True
3        eleven     False
4         seven     False

And then, we have a complete overview of the data frame.

Conclusions

In this article, we've seen various methods to determine if there are any numbers in strings, in different cases and situations. We've also demonstrated that there is no one-size-fits-all method; depending on the situation, we have to choose the one that best suits the specific problem we're facing.

June 06, 2023 12:36 PM UTC

Nicola Iarocci

Python `decimal.getcontext` does not work with bpython

I have been working on a side project for which I’m using bpython, a “fancy interface to the Python interpreter.” If you use the Python REPL often, you should check it out. It offers unique features like in-line syntax highlighting, readline-like autocomplete, a “rewind” function to pop the last line of code from memory, auto-indentation and more. Anyway, today I found a bug in bpython, and that’s that Python’s decimal.getcontext() does not work with it.

June 06, 2023 06:05 AM UTC

ListenData

Transformers Agent: AI Tool That Automates Everything

We have a new AI tool in the market called Transformers Agent which is so powerful that it can automate just about any task you can think of. It can generate and edit images, video, audio, answer questions about documents, convert speech to text and do a lot of other things.

Hugging Face, a well-known name in the open-source AI world, released Transformers Agent that provides a natural language API on top of transformers. The API is designed to be easy to use. With a single line code, it provides a variety of tools for performing natural language tasks, such as question answering, image generation, video generation, text to speech, text classification, and summarization.

June 06, 2023 05:05 AM UTC

June 05, 2023

Ned Batchelder

Multi-syntax configuration examples

Coverage.py reads settings from a number of different files, in either INI-file or TOML syntax, with different section headings depending on the file. This could be confusing: the docs showed the syntax needed for .coveragerc, but if you were using one of the other files, you had to mentally convert from one syntax to another.

To make it easier, I updated the documentation to show all the syntax possibilities when showing examples of configuration files. I used sphinx-code-tabs to create tabbed code sections.

The result looks like this: a three-tabbed code box. The tabs have the names of configuration files. Selecting a tab shows the syntax you need for that particular file:

Originally I wanted to write the settings just once, and use cog to run Python code to convert from INI syntax to TOML.

If you haven’t seen cog before, it lets you insert small chunks of Python into static files. The Python code is executed to generate more static content that appears alongside the Python code. It’s a simple tool that works well when you have a mostly static file, but you want part of it to be computed.

But I sensed that converting between syntaxes be difficult, especially since some of my examples (as the ones above) have comments in them.

So I went with a simpler approach: the Python snippets in the doc files have both INI and TOML chunks. The INI chunk is lightly massaged to make two of the tabs, since they only differ in the section names. To ensure the two input chunks are in sync, the code reads the configuration examples, and checks that the INI and TOML chunks create the same settings values.

This guarantees that the different syntaxes are all valid, both as generic INI or TOML, but also as coverage.py settings. And it guarantees that the two chunks show the same settings, differing only in syntax.

It’s a little bulky in the resulting .rst files, but it provides the reader with the information they need, and it keeps me from making mistakes.

If you want to see the code, the cog code is in cog_helpers.py, and a small example of how it’s used in the documentation is in context.rst.

June 05, 2023 11:16 PM UTC

Stack Abuse

How to Check if a String is Empty or None in Python

Introduction

In Python, it's often important to check whether a string is empty or None before performing operations on it. This can prevent unexpected errors and make your code more robust. But what is the most efficient and Pythonic way to do this? And what potential pitfalls should you watch out for?

In this article, we'll explore these questions, demonstrating how to correctly check if a string is empty or None in Python, as well as discussing some best practices to make your code more reliable and maintainable.

Boolean Evaluation in Python

In Python, values are considered "truthy" or "falsy" based on whether they evaluate to True or False in a boolean context. This concept plays a crucial role when checking conditions in code.

For strings, an empty string ("") is considered "falsy" — it evaluates to False in a boolean context. On the other hand, a non-empty string is "truthy" — it evaluates to True. The special value None is also considered "falsy", as shown in the following code snippet:

s1 = ""
s2 = "Hello"
s3 = None

print(bool(s1))  # False
print(bool(s2))  # True
print(bool(s3))  # False

This property of strings and None is extremely useful when you want to check if a string is empty or None. As we'll see in the next sections, you can use simple if statements to make these checks, leveraging the "falsiness" of an empty string and None.

Checking if a String is Empty

When checking if a string is empty in Python, we can take advantage of the fact that an empty string is "falsy". You can use either the == operator or the not operator to perform this check.

Method 1: Using the == Operator

s = ""
if s == "":
    print("String is empty")
else:
    print("String is not empty")

Method 2: Using the not Operator

s = ""
if not s:
    print("String is empty")
else:
    print("String is not empty")

In both of these methods, the if statement will print "String is empty" if the string s is empty. The not operator in the second example negates the "truthiness" or "falsiness" of the string, making the code slightly more concise. This is a common Pythonic way to check for empty strings.

Checking if a String is None

In Python, to check if a string is None, it's best practice to use the is operator rather than the == operator. This is because is checks if both operands are the same object, not just equivalent.

Advice: Read more about the differences between the is and the == operators in our article "'is' vs '==' in Python - Object Comparison".

Let's take a look at the practical use-case of the is operator for checking if a string is None":

s = None

if s is None:
    print("String is None")
else:
    print("String is not None")

As expected, the if statement will print "String is None" if the variable s is None.

Remember, None is a singleton in Python, which means there is only ever one instance of None. Thus, using is is more appropriate and can be more efficient when checking for None.

Checking if a String is Empty or None

To check if a string is either empty or None in Python, we'll combine the techniques we discussed in the previous sections. We'll use an or operator to check both conditions at once:

s = ""  # Try also with s = None

if s is None or s == "":
    print("String is empty or None")
else:
    print("String is not empty and not None")

Advice: Find out more about the or operator in Python by reading our "Guide to the Python or Operator".

In this example, the if statement checks two conditions: if s is None and if s is an empty string. If either condition is true, it will print "String is empty or None". If both conditions are false (meaning the string is not None and it's not empty), it will print "String is not empty and not None".

Tips and Advice

Don't use == Instead of is to Check for None

Python has two equality operators, == and is. While they sometimes work the same, they are not identical. The == operator checks for value equality, while is checks for identity, meaning that it checks whether the operands are the same object.

None is a singleton in Python - there's only one instance of None. Therefore, it's more appropriate and slightly more efficient to use is when checking for None.

s = None

# Recommended
if s is None:
    print("s is None")

# Not recommended
if s == None:
    print("s is None")

Take Advantage of Python's "Truthiness"

Python's treatment of different values as "truthy" or "falsy" in boolean contexts can simplify your code. Rather than explicitly comparing a string to an empty string (""), you can use the not keyword to check if a string is empty:

s = ""

# Recommended
if not s:
    print("String is empty")

# Not recommended
if s == "":
    print("String is empty")

Handle None When Performing Operations on Strings

If there's any chance a string could be None, always check before performing operations on it. Neglecting to do so can result in a TypeError:

s = None

# Results in TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
print(s + " more text")

To prevent such errors, you can check if the string is None before performing the operation:

if s is not None:
    print(s + " more text")
else:
    print("String is None, cannot perform operation")

Conclusion

We've seen that Python provides straightforward and efficient ways to perform these checks, leveraging the "truthiness" and "falsiness" of different values in boolean contexts. We've also discussed some potential pitfalls to avoid and best practices to follow when performing these checks, such as using the is operator to check for None and the not operator to check for an empty string.

June 05, 2023 04:51 PM UTC

Django Weblog

Django bugfix release: 4.2.2

Today we've issued the 4.2.2 bugfix release.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Natalia Bidart: 2EE82A8D9470983E.

June 05, 2023 02:25 PM UTC

Real Python

Using the NumPy Random Number Generator

>>> import numpy as np

>>> default_rng = np.random.default_rng()
>>> default_rng
'Generator(PCG64) at 0x1E9F2ABBF20'

>>> default_rng.random()
0.47418635476614734

Read the full article at https://realpython.com/numpy-random-number-generator/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

June 05, 2023 02:00 PM UTC

Mike Driscoll

PyDev of the Week: Daniel Alejandro Mesejo-León

This week we welcome Daniel Alejandro Mesejo-León (@searchsort) as our PyDev of the Week! Daniel is the creator of the trex package, which is used for efficient keyword extraction using RegEx.

You can see what else Daniel is up to by checking out his GitHub profile.

Let’s spend some time getting to know Daniel better!

Can you tell us a little about yourself (hobbies, education, etc):

My name is Daniel Mesejo. I’m a software engineer from Cuba currently living in Spain. I love sharing knowledge, have answered several questions related to Python, pandas, and numpy on StackOverflow, and have occasionally spoken at Python events. I enjoy reading and hiking; a curious fact about me is that I’m Erdos number 3.

Why did you start using Python?

I started using Python in college for some research projects. Once I started, I became obsessed with it. Python makes programming joyful; it gives me the same feeling I had as a kid solving logic puzzles.

What other programming languages do you know, and which is your favorite?

In my daily work, I use JVM-based languages Java, Kotlin, and Groovy. Of those, I prefer Kotlin; for me, it is a mix of Java and Python.

What projects are you working on now?

I want to re-start contributing to open-source projects such as dask, eland, and prefect. I also want to grow the usage of my project: trrex.

Which Python libraries are your favorite (core or 3rd party)?

My favorite libraries are collections, itertools, and operators; the number of things you can get done using only those three libraries is impressive.

How did the trrex package come about?

trrex transforms a list of strings into a regular expression so you can search and replace those words efficiently. I wrote it after reading https://stackoverflow.com/q/42742810/4001592. The question is commonly asked on StackOverflow, so I thought having a library for it might be helpful for others.

What are the top three things you love about trrex?

Its ease of use is just one function with three parameters with a string as the output

The performance of the obtained regular expression pattern matches that of even specialized libraries like flashtext.

It integrates well with other data science libraries: spacy, pandas, regex, scikit-learn.

Ease of use and combining well with others are traits of built-in Python functions, so I’m proud of that fact.

Is there anything else you’d like to say?

For those starting to learn Python and want to use StackOverflow, I suggest you look at https://sopython.com/canon/; it collects pointers to the most common questions. I also encourage everyone to try trrex when using regular expressions for keyword manipulation.

Thanks for doing the interview Daniel!

The post PyDev of the Week: Daniel Alejandro Mesejo-León appeared first on Mouse Vs Python.

June 05, 2023 12:34 PM UTC

eGenix.com

Python Meeting Düsseldorf - 2023-06-07

The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.

Ankündigung

Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:

07.06.2023, 18:00 Uhr
Raum 1, 2.OG im Bürgerhaus Stadtteilzentrum Bilk
Düsseldorfer Arcaden, Bachstr. 145, 40217 Düsseldorf

Programm

Bereits angemeldete Vorträge

• Marc-Andre Lemburg:
  Building your own Web Archive with Python

• Nico Kreiling:
  Raised by Pandas, striving for more. An opinionated introduction to Polars

• Charlie Clark: LT...
  Combining Descriptors and Slots
  

• Marc-Andre Lemburg: LT...
  What's new in Python 3.12
  

Startzeit und Ort

Wir treffen uns um 18:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden.

Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.

Über dem Eingang steht ein großes "Schwimm’ in Bilk" Logo. Hinter der Tür direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.

>>> Eingang in Google Street View

Corona

Die Corona Einschränkungen sind mittlerweile aufgehoben worden. Vorsicht ist zwar immer noch geboten, aber jetzt jedem selbst überlassen.

⚠️ Wichtig: Bitte nur dann anmelden, wenn ihr absolut sicher seid, dass ihr auch kommt. Angesichts der begrenzten Anzahl Plätze, haben wir kein Verständnis für kurzfristige Absagen oder No-Shows.

Einleitung

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf:

Format

Das Python Meeting Düsseldorf nutzt eine Mischung aus (Lightning) Talks und offener Diskussion.

(Lightning) Talk Anmeldung bitte formlos per EMail an info@pyddf.de

Kostenbeteiligung

Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.

Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.

Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.

Anmeldung

Da wir nur 25 Personen in dem angemieteten Raum empfangen können, möchten wir bitten, sich vorher anzumelden.

Meeting Anmeldung bitte per Meetup

Weitere Informationen

Weitere Informationen finden Sie auf der Webseite des Meetings:

              https://pyddf.de/

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

June 05, 2023 09:00 AM UTC

Python Software Foundation

The Python Language Summit 2023: Pattern Matching, __match__, and View Patterns

>>> import ast
>>> source = "x = 42"
>>> node = ast.parse(source)
>>> node
<ast.Module object at 0x000002A70F928D80>
>>> print(ast.dump(node, indent=2))
Module(
  body=[
    Assign(
      targets=[
        Name(id='x', ctx=Store())],
      value=Constant(value=42))],
  type_ignores=[])

How does working with ASTs relate to pattern-matching? Well, a function to determine whether (to a reasonable approximation) an arbitrary AST node represents the symbol collections.deque might have looked something like this, before pattern matching…

import ast

# This obviously won't work if the symbol is imported with an alias
# in the source code we're inspecting
# (e.g. "from collections import deque as d").
# But let's not worry about that here :-)

def node_represents_collections_dot_deque(node: ast.AST) -> bool:
    """Determine if *node* represents 'deque' or 'collections.deque'"""
    return (
        isinstance(node, ast.Name) and node.id == "deque"
    ) or (
        isinstance(node, ast.Attribute)
        and isinstance(node.value, ast.Name)
        and node.value.id == "collections"
        and node.value.attr == "deque"
    )

But in Python 3.10, pattern matching allows an elegant destructuring syntax:

import ast

def node_represents_collections_dot_deque(node: ast.AST) -> bool:
    """Determine if *node* represents 'deque' or 'collections.deque'"""
    match node:
        case ast.Name("deque"):
            return True
        case ast.Attribute(ast.Name("collections"), "deque"):
            return True
        case _:
            return False

I know which one I prefer.

For some, though, this still isn’t enough – and Michael “Sully” Sullivan is one of them. At the Python Language Summit 2023, Sullivan shared ideas for where pattern matching could go next.

Playing with matches (without getting burned)

Sullivan’s contention is that, while pattern matching provides elegant syntactic sugar in simple cases such as the one above, our ability to chain destructurings using pattern matching is currently fairly limited. For example, say we want to write a function inspecting Python AST that takes an ast.FunctionDef node and identifies whether the node represents a synchronous function with exactly two parameters, both of them annotated as accepting integers. The function would behave so that the following holds true:

>>> import ast
>>> source = "def add_2(number1: int, number2: int): pass"
>>> node = ast.parse(source).body[0]
>>> type(node)
<class 'ast.FunctionDef'>
>>> is_function_taking_two_ints(node)
True

With pre-pattern-matching syntax, we might have written such a function like this:

def is_int(node: ast.AST | None) -> bool:
    """Determine if *node* represents 'int' or 'builtins.int'"""
    return (
        isinstance(node, ast.Name) and node.id == "int"
    ) or (
        isinstance(node, ast.Attribute)
        and isinstance(node.value, ast.Name)
        and node.value.id == "builtins"
        and node.attr == "int"
    )

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    args = node.args.posonlyargs + node.args.args
    return len(args) == 2 and all(is_int(node.annotation) for node in args)

def is_int(node: ast.AST | None) -> bool:
    """Determine if *node* represents 'int' or 'builtins.int'"""
    match node:
        case ast.Name("int"):
            return True
        case ast.Attribute(ast.Name("builtins"), "int"):
            return True
        case _:
            return False

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [ast.arg(), ast.arg()] as arglist:
            return all(is_int(arg.annotation) for arg in arglist)
        case _:
            return False

That leaves a lot to be desired, however! The is_int() helper function can be rewritten in a much cleaner way. But integrating it into the is_function_taking_two_ints() is… somewhat icky! The code feels harder to understand than before, whereas the goal of pattern matching is to improve readability.

Something like this, (ab)using metaclasses, gets us a lot closer to what it feels pattern matching should be like. By using one of Python’s hooks for customising isinstance() logic, it’s possible to rewrite our is_int() helper function as a class, meaning we can seamlessly integrate it into our is_function_taking_two_ints() function in a very expressive way:

import abc
import ast

class PatternMeta(abc.ABCMeta):
    def __instancecheck__(cls, inst: object) -> bool:
        return cls.match(inst)

class Pattern(metaclass=PatternMeta):
    """Abstract base class for types representing 'abstract patterns'"""
    @staticmethod
    @abc.abstractmethod
    def match(node) -> bool:
        """Subclasses must override this method"""
        raise NotImplementedError

class int_node(Pattern):
    """Class representing AST patterns signifying `int` or `builtins.int`"""
    @staticmethod
    def match(node) -> bool:
        match node:
            case ast.Name("int"):
                return True
            case ast.Attribute(ast.Name("builtins"), "int"):
                return True
            case _:
                return False

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [
            ast.arg(annotation=int_node()), 
            ast.arg(annotation=int_node()),
        ]:
            return True
        case _:
            return False

This is still hardly ideal, however – that’s a lot of boilerplate we’ve had to introduce to our helper function for identifying int annotations! And who wants to muck about with metaclasses?

A slide from Sullivan's talk

A __match__ made in heaven?

Sullivan proposes that we make it easier to write helper functions for pattern matching, such as the example above, without having to resort to custom metaclasses. Two competing approaches were brought for discussion.

The first idea – a __match__ special method – is perhaps the easier of the two to immediately grasp, and appeared in early drafts of the pattern matching PEPs. (It was eventually removed from the PEPs in order to reduce the scope of the proposed changes to Python.) The proposal is that any class could define a __match__ method that could be used to customise how match statements apply to the class. Our is_function_taking_two_ints() case could be rewritten like so:

class int_node:
    """Class representing AST patterns signifying `int` or `builtins.int`"""
    # The __match__ method is understood by Python to be a static method,
    # even without the @staticmethod decorator,
    # similar to __new__ and __init_subclass__
    def __match__(node) -> ast.Name | ast.Attribute:
        match node:
            case ast.Name("int"):
                # Successful matches can return custom objects,
                # that can be bound to new variables by the caller
                return node
            case ast.Attribute(ast.Name("builtins"), "int"):
                return node
            case _:
                # Return `None` to indicate that there was no match
                return None

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [
            ast.arg(annotation=int_node()), 
            ast.arg(annotation=int_node()),
        ]:
            return True
        case _:
            return False

The second idea is more radical: the introduction of some kind of new syntax (perhaps reusing Python’s -> operator) that would allow Python coders to “apply” functions during pattern matching. With this proposal, we could rewrite is_function_taking_two_ints() like so:

def is_int(node: ast.AST | None) -> bool:
    """Determine if *node* represents 'int' or 'builtins.int'"""
    match node:
        case ast.Name("int"):
            return True
        case ast.Attribute(ast.Name("builtins"), "int"):
            return True
        case _:
            return False

def is_function_taking_two_ints(node: ast.FunctionDef) -> bool:
    """Determine if *node* represents a function that accepts two ints"""
    match node.args.posonlyargs + node.args.args:
        case [
            ast.arg(annotation=is_int -> True),
            ast.arg(annotation=is_int -> True),
        ]
        case _:
            return False

Match-maker, match-maker, make me a __match__…

A slide from Sullivan's talk

The reception in the room to Sullivan’s ideas was positive; the consensus seemed to be that there was clearly room for improvement in this area. Brandt Bucher, author of the original pattern matching implementation in Python 3.10, concurred that this kind of enhancement was needed. Łukasz Langa, meanwhile, said he’d received many queries from users of other programming languages such as C#, asking how to tackle this kind of problem.

The proposal for a __match__ special method follows a pattern common in Python’s data model, where double-underscore “dunder” methods are overridden to provide a class with special behaviour. As such, it will likely be less jarring, at first glance, to those new to the idea. Attendees of Sullivan’s talk seemed, broadly, to slightly prefer the __match__ proposal, and Sullivan himself said he thought it “looked prettier”.

Jelle Zijlstra argued that the __match__ dunder would provide an elegant symmetry between the construction and destruction of objects. Brandt Bucher, meanwhile, said he thought the usability improvements weren’t significant enough to merit new syntax.

Nonetheless, the alternative proposal for new syntax also has much to recommend it. Sullivan argued that having dedicated syntax to express the idea of “applying” a function during pattern matching was more explicit. Mark Shannon agreed, noting the similarity between this idea and features in the Haskell programming language. “This is functional programming,” Shannon argued. “It feels weird to apply OOP models to this.”

Addendum: pattern-matching resources and recipes

In the meantime, while we wait for a PEP, there are plenty of innovative uses of pattern matching springing up in the ecosystem. For further reading/watching/listening, I recommend:

• “A perfect match: The history, design and future of Python’s structural pattern matching” – A talk by Brandt Bucher at PyCon 2022
• “Structural Pattern Matching in the Real World” – A talk by Raymond Hettinger at Pycon Italia 2022
• RegexMatcher: a class integrating pattern matching with Python’s re module. A 2022 Advent of Code solution by Ned Batchelder.
• approximately: A way to compare float and complex numbers using pattern matching, while avoiding the perils of floating-point arithmetic. A StackOverflow Q&A by Raymond Hettinger.
• “A few related schemes for implementing view patterns in Python”: A gist by Michael Sullivan (from February 2023)

June 05, 2023 07:47 AM UTC

Read the Docs

Read the Docs newsletter - June 2023

News and updates

• ⚠️ A .readthedocs.yaml configuration file will be required for your future builds. Read more about this change in Migrate your project to .readthedocs.yaml configuration file v2.

• ✅️ Visiting a language slug of a project without specifying the version now redirects to the default version. For instance, /en/ redirects to /en/latest/.

• 🍿️ All the talks from Write the Docs Portland 2023, which we sponsored this year, are ready to watch: Open the new playlist on Youtube.

• 🐛️ Builds with multiline build.commands should now be working reliably. Thanks to everyone who helped out!

• 🐛️ sphinx-rtd-theme 1.2.1 has been released with an important bug fix that caused jQuery to not load in certain projects.

• 🔒️ We updated custom domain certificates to 1.2 as the minimum TLS version.

• 🔒️ A high-severity vulnerability has been fixed: Write access to projects via API V2 (/api/v2/project/* endpoints) for any logged-in user.

Possible issues

• ⚠️ Please make sure to read the blog post: Migrate your project to .readthedocs.yaml configuration file v2.

  If you didn’t have a .readthedocs.yaml configuration file, and you are introducing one for the first time, we’re excited for you to be able to use all our new features like specifying build tool versions (Node, Rust, etc.)!

  The blog post offers help with that 💡️

Awesome project of the month

The most recent addition to Awesome Read the Docs Projects 🕶️ is Ray’s documentation 🕶️.

Ray is an open source project, consisting of several components that all have their own section in a combined documentation set.

See the highlights in the following Twitter thread or Mastodon thread:

The most recent addition to our awesome projects list is Ray's documentation 🕶️.

Ray by @anyscalecompute is an open source project, consisting of several components that all have their own section in a combined documentation set.https://t.co/9YGpJvYpFD

Here is a 🤏 (small) 🧵 pic.twitter.com/QElmmJetPB

— Read the Docs (@readthedocs) May 31, 2023

It was always better to use a .readthedocs.yaml file. It gives so many more options and makes it possible to version your configuration in your Git repository ⚡️

Questions? Comments? Ideas for the next newsletter? Contact us!

June 05, 2023 12:00 AM UTC

June 02, 2023

Brett Cannon

Proposing a struct syntax for Python

Story time

When I go on vacation with a fellow Python developer, inevitably I will talk about Python. 😁 Back in September, Andrea and I drove the Cabot Trail with our friends Dusty and Jen, which led to a discussion about the match statement and how Dusty and I both wished more people would use the syntax. We created the match-variant package to try and help, but what Dusty and I really wanted was some syntax to make it easier to construct algebraic data types (ADTs) like Rust&aposs enum type to facilitate type checkers in performing variant exhaustion checking (e.g., if your ADT had the directions of the compass, how do you make sure you don&apost skip the "west" case in your match statement?). At the same time, I was trying to think of a way to potentially meld this with syntax to replace the enum module to provide a simpler, stable representation of the same concept (the enum module has had some tweaks made that had to be rolled back due to compatibility concerns in the last couple of releases).

While I was trying to think this through, I also realized people would inevitably abuse whatever I came up with to act like dataclasses. That then made me think about whether an ADT could come from dataclasses as a base? I then had a realization that you could either construct an ADT and then "fake" dataclasses by having a single variant (e.g., your ADT had a single variant called Point), or you could make it easy to construct a dataclass and then somehow tell type checkers via a union that all of these dataclasses should be considered a single ADT for the purposes of type checking (e.g., you had East, West, North, and South and then created a union like Compass = typing.Union[East, West, North, South]).

And then two things happened to cause my mind to snap to the latter, dataclass-like solution. One, Glyph blogged about the idea of syntax support for dataclasses which made me think I was on to something. And two, Hynek Schlawack and Bruce Eckel both had talks at PyCon US 2023 about composition and how object-oriented programming isn&apost the end-all-and-be-all to programming (which also aligns with me blogging about functions versus classes back in 2017), respectively. Those talks independently back up some ideas I had which I thought may be considered extreme, but I now think are sensible for this idea.

So, I want some syntax to make create data-only classes extremely easy.

Goals

I want syntax that will keep people from using collections.namedtuple() just for its attribute API and ignoring/regretting its index API. Or put another way, I want syntax to replace the simple, data-only uses of dataclasses. An analogy to other languages is C structs, Rust structs, and Go structs.

Because of the data-only orientation, I want it to be comprehensible by beginners after they learn functions but before they have to learn the complexity of classes. This should also facilitate using composition over inheritance in general.

Performance should also be a key focus as I could see this being used a lot for objects to store small amounts of data individually, but having a huge amount of instances in total.

Typing should be properly supported from the start, but not required. And structural/duck typing should be how runtime semantics operate.

The instances should be immutable. It fits into the replacing of collections.namedtuple() while also being easier to reason about and being usable as a key in a dictionary. Tied into the composition angle, this leans a bit more into the functional programming side of Python.

No inheritance. Other languages also seem to get by fine without inheritance for this sort of data structure.

No methods. That&aposs because methods are not that important if you don&apost have inheritance. At that point you might as well use a function and you can use a module or something to namespace things if you need. See Hynek and/or Bruce&aposs talk for more reasons why methods are not necessary.

The proposal

I want to introduce the new (soft) keyword struct (the name was chosen by my wife, Andrea, as more understandable than data once you explain struct is short for structure and how it&aposs used in other programming languages). You would be able to follow the keyword with the name of the struct class. The parentheses after the name would contain the positional-or-keyword and/or keyword-only parameters the constructor would accept. Each parameter name would directly map to an attribute name for which the parameter would be saved to. Because of this mapping of parameter name to attribute name, no positional-only parameters are allowed (the only deviation from function declearation syntax).

As an example, a struct called Point which had x and y parameters could be defined as:

struct Point(x: int, y: int)

This would essentially lead to the definition of a struct class which has:

• __slots__, for performance
• __match_args__, but only for positional-and-keyword arguments (you can still match on keyword-only arguments directly)
• __repr__(), and whose result is valid for eval() (if possible)
• __eq__(), based on __slots__ instead of isinstance() checks to support structural/duck typing
• __hash__()
• asdict(), to easily get the data out in another data structure (much like dataclasses.asdict())
• replace(), to make it easy to construct new instances with minor changes without having to call asdict() on the original instance, updating the dict however you need to, and then construcitng a new instance; this method is much like dataclasses.replace()

The equivalent Python code for our Point example would be:

from typing import Any, Self, TypedDict, Unpack


class _AsDictResult(TypedDict):
    """Return type of Point.asdict()."""

    x: int
    y: int


class _ReplaceParams(TypedDict, total=False):
    """For typing Point.replace()."""

    x: int
    y: int


class Point:
    __slots__ = ("x", "y")
    __match_args__ = ("x", "y")

    def __new__(cls, x: int, y: int) -> Self:
        """Create a new, immutable instance."""
        # Pretend this makes everything immutable in the end.
        self = mutable(cls.__slots__)
        self.x = x
        self.y = y
        return immutable(self)

    def __repr__(self) -> str:
        """Return the repr.

        Should be able to create a new instance via eval() if all attributes
        have a repr allowing for such usage.

        """
        params = []
        for attr in type(self).__slots__:
            params.append(f"{attr}={getattr(self, attr)!r}")
        return f"{type(self).__qualname__}({&apos, &apos.join(params)})"

    def __eq__(self, other: Any) -> bool | NotImplemented:
        """Check for equality.

        The comparison is done per-attribute to allow for duck typing (i.e.,
        nominal typing is not used as a shortcut for comparing).

        """
        attrs = frozenset(type(self).__slots__)
        other_attrs = frozenset(getattr(type(other), "__slots__", [object()]))
        if attrs != other_attrs:
            # Avoids the question of what to do if there are extra attributes
            # on `other`.
            return NotImplemented

        for attr in type(self).__slots__:
            if not hasattr(other, attr):
                return NotImplemented
            elif getattr(self, attr) != getattr(other, attr):
                return False
        else:
            return True

    def __hash__(self) -> int:
        """Hash the instance."""
        attr_tuple = tuple(getattr(self, attr) for attr in type(self).__slots__)
        return hash(attr_tuple)

    def asdict(self) -> _AsDictResult:
        """Return a dict representation of the object."""
        return {attr: getattr(self, attr) for attr in type(self).__slots__}

    def replace(self, **new_data: Unpack[_ReplaceParams]) -> Self:
        """Return a new instance, replacing value as appropriate.

        Raise a TypeError if any unexpected keys are provided.

        """
        current_data = self.asdict()
        if diff := set(new_data.keys()).difference(current_data.keys()):
            raise TypeError(
                f"{type(self).__qualname__}.replace() called with unexpected arguments:"
                f"{&apos, &apos.join(sorted(diff))}"
            )
        else:
            return type(self)(**(current_data | new_data))

I wish there was a way to do native docstring support while keeping this a single line, but e.g., struct Point(x: int, y: int), "a 2D point." just doesn&apost look right to me. Since it is a new keyword it might be possible to make a : optional and only usable to add a docstring, but that might be a little too weird when the : doesn&apost alow for other statements afterwards. Otherwise doing a bare string immediately after the definition could inherently be picked up as a docstring just like what PEP 257 calls an "attribute docstring".

If you squint a little bit, this kind of gets you immutable dicts, albeit with an attribute interface that can be typed without defining a separate TypedDict. Same goes for those people who want an attribute interface over dicts: define the equivalent struct and then creating instances using a dict; Point(**{&aposx&apos: 0, &aposy&apos: 0}) would convert a dict to the appropriate attribute interface much like types.SimpleNamespace, but with a more rigid API guarantee and typing built in.

As for the original ADT/enum dream, PEP 695 introduces the type keyword for type aliases. I can imagine you declare a union of structs to get that collection you expect a match statement to exhaustively check for (although that would probably require a PEP to define when type checkers can consider the union final/complete for pattern matching exhaustion). For instance, if you had struct Point1D(x), struct Point2D(x, y) and struct Point3D(x, y, z), you could do type Point = Point1D | Point2D | Point3D and have type checkers recognize that all three cases need to be handled by a match statement (maybe with some PatternExhaustion type or something to denote that fact).

But why?!?

To be very specific as to why I think this could be better than dataclasses:

• Typing is optional (there are still folks who don&apost want to lean into that and it simply isn&apost always necessary)
• Class construction should be faster (which intuitively you would think shouldn&apost matter, but I have talked to folks where this is an actual concern, especially when startup time is critical)
• Syntax typically leads to better tooling support since there&aposs no ambiguity
• Easier to teach than (data)classes, so can act as a stepping stone towards classes

And that&aposs it! I am sure this will be controversial, especially based on the restrictions I am proposing to keep the concept simple. I also have no illusions that this may not be popular and thus not go anywhere. But if folks show enough interest I&aposm willing to write this up as a PEP.

June 02, 2023 07:01 PM UTC

Real Python

The Real Python Podcast – Episode #158: Building Python CI With Docker & Applying for a Hacker Initiative Grant

Do you need a refresher on using Docker with Python? Would you like to learn how to configure a continuous integration pipeline with modern tools and Docker? This week on the show, Christopher Trudeau is here, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

June 02, 2023 12:00 PM UTC

Talk Python to Me

#418: How To Keep A Secret in Python Apps

Think about the different APIs and databases your application works with. Every one of them requires either an API key or a database connection string that itself contains a password. How do you let your application access this sensitive information without storing it in source code or putting in other compromising locations? We have Glyph Lefkowitz on the show to share his security fable as well as just good advice for keeping secrets out of Python code.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Glyph on Mastodon</b>: <a href="https://mastodon.social/@glyph" target="_blank" rel="noopener">@glyph@mastodon.social</a><br/> <b>ShhGit</b>: <a href="https://github.com/eth0izzle/shhgit" target="_blank" rel="noopener">github.com</a><br/> <b>Encrust</b>: <a href="https://github.com/glyph/Encrust" target="_blank" rel="noopener">github.com</a><br/> <b>GitHub Security Alerts</b>: <a href="https://docs.github.com/en/code-security/secret-scanning/managing-alerts-from-secret-scanning" target="_blank" rel="noopener">github.com</a><br/> <b>CIA Triad</b>: <a href="https://www.fortinet.com/resources/cyberglossary/cia-triad" target="_blank" rel="noopener">fortinet.com</a><br/> <b>pinpal</b>: <a href="https://github.com/glyph/pinpal/" target="_blank" rel="noopener">github.com</a><br/> <b>XKCD Authorization</b>: <a href="https://xkcd.com/1200/" target="_blank" rel="noopener">xkcd.com</a><br/> <b>Tokenring</b>: <a href="https://github.com/glyph/tokenring" target="_blank" rel="noopener">github.com</a><br/> <b>AWS Vault</b>: <a href="https://github.com/99designs/aws-vault" target="_blank" rel="noopener">github.com</a><br/> <b>Gimme-AWS-creds</b>: <a href="https://github.com/Nike-Inc/gimme-aws-creds" target="_blank" rel="noopener">github.com</a><br/> <b>Secrets in GitHub Actions</b>: <a href="https://docs.github.com/en/actions/security-guides/encrypted-secrets" target="_blank" rel="noopener">github.com</a><br/> <b>Python Client for HashiCorp Vault</b>: <a href="https://python-hvac.org/" target="_blank" rel="noopener">python-hvac.org</a><br/> <b>Pomodouroboros app</b>: <a href="https://github.com/glyph/Pomodouroboros" target="_blank" rel="noopener">github.com</a><br/> <b>DateType</b>: <a href="https://pypi.org/project/datetype/" target="_blank" rel="noopener">pypi.org</a><br/> <b>Haveibeenpwned</b>: <a href="https://haveibeenpwned.com" target="_blank" rel="noopener">haveibeenpwned.com</a><br/> <b>PEP 541</b>: <a href="https://peps.python.org/pep-0541/" target="_blank" rel="noopener">peps.python.org</a><br/> <b>Glyph's security talk at PyCon</b>: <a href="https://us.pycon.org/2023/schedule/presentation/149/" target="_blank" rel="noopener">us.pycon.org</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=Gey87cZXF3Q" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/418/how-to-keep-a-secret-in-python-apps" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div><br/> <strong>Sponsors</strong><br/> <a href='https://talkpython.fm/done-with-pycharm'>PyCharm</a><br> <a href='https://talkpython.fm/compiler'>RedHat</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>

June 02, 2023 08:00 AM UTC

June 01, 2023

ListenData

AutoGPT : Everything You Need To Know

In this post we have covered AutoGPT in detail. By end of this tutorial, you will not only understand how it works but also will be able to run it on your system. Auto-GPT has gained a significant amount of popularity in the media. It has become one of the most talked-about topics across various social media platforms after ChatGPT. It has not only captured the attention of people in Artifical Intelligence community but also people from other background. Media outlets across countries covered it and reported how it can automate everything ranging from simple to complex tasks.

June 01, 2023 10:53 PM UTC

Luke Plant

Django and Sass/SCSS without Node.js or a build step

Although they are less necessary than in the past, I like to use a CSS pre-processor when doing web development. I used to use LessCSS, but recently I’ve found that I can use Sass without needing either a separate build step, or a package that requires Node.js and npm to install it. The heart of the functionality is provided by libsass, an implementation of Sass as a C++ library.

On Linux systems, this can be installed as a package libsass or similar, but even better is that you can pip install it as a Python package, libsass.

When it comes to using it from a Django project, the first step is to install django-compressor.

Then, you need to add django-libsass as per its instructions.

That’s about it. As per the django-libsass instructions, somewhere in your base HTML templates you’ll have something like this:

{% compress css %}
  <link rel="stylesheet" type="text/x-scss" href="{% static "myapp/css/main.scss" %}" />
{% endcompress %}

You write your SCSS in that main.scss file (it doesn’t have to be called that), and it can @import other SCSS files of course.

Then, when you load a page, django-compressor will take care of running the SCSS files through libsass, saving the output CSS to a file and inserting the appropriate HTML that references that CSS file into your template output. It caches things very well so that you don’t incur any penalty if files haven’t changed — and libsass is a very fast implementation for when the processing does need to happen.

What this means is that you have eliminated both the need for Node.js/npm, and the need for a build step/process.

Of course, the SCSS → CSS compilation still has to happen, but it happens on demand in the same process that runs the web app, and it’s both fast enough and reliable enough that you simply never have to think about it again. So this is “build-less” in the same way that “server-less” means you don’t have to think about servers, and the same way that Python “doesn’t have a compilation step”.

Future proofing

On the Sass-lang page about libsass, they say it is “deprecated”, and on the project page page it says:

While it will continue to receive maintenance releases indefinitely, there are no plans to add additional features or compatibility with any new CSS or Sass features.

In other words, this is what I prefer to call “mature software” 😉. libsass already has everything I need. If it does eventually fail to be maintained or I need new features, it’s not a problem:

• Switch to Dart Sass, which can be installed as a standalone binary.

• Set your django-compressor settings like this:

  COMPRESS_PRECOMPILERS = [
      ("text/x-scss", "sass {infile} {outfile}"),
  ]
  

This covers the basic case. If you want all the features of django-libsass, which includes looking in your other static file folders for SCSS, you’ll probably need to fork the code and make it work by calling Dart Sass using subprocess — a small amount of work, and nothing that will fundamentally break this approach.

June 01, 2023 07:54 PM UTC

PyCharm

PyCharm 2023.2 EAP 2: Live Templates for Django Forms and Models, Support for Polars DataFrames

The second Early Access Program build brings a bunch of features for both web developers and data scientists. Try new, time-saving live templates for Django forms, models, and views, as well as support for a super-fast Polars DataFrame library and initial GitLab integration. 

You can get the latest build from our website, the free Toolbox App, or via snaps for Ubuntu.

If you want to catch up on the updates from the previous EAP build, you can refer to this blog post for more details.

Download PyCharm 2023.2 EAP

UX

Text search in Search Everywhere

The Search Everywhere (Double ⇧ / Double Shift) functionality, primarily utilized for searching through files, classes, methods, actions, and settings, now includes text search capabilities similar to Find in Files. With this enhancement, text search results are displayed when there are few or no other search results available for a given query. The feature is enabled by default and can be managed in Settings/Preferences | Advanced Settings | Search Everywhere.

Dedicated syntax highlighting for Python local variables

PyCharm 2023.2 will provide a dedicated syntax highlighting option for local variables. To use it, go to Settings | Editor | Color Scheme | Python and choose Local variables from the list of available options. 

By default, the highlighting is set to inherit values from the Language Defaults identifiers. By unchecking this checkbox, you can choose the highlighting scheme that works best for you. 

Syntax highlighting in inspection descriptions 

In Settings / Preferences | Editor | Inspections, we’ve implemented syntax highlighting for code samples, which facilitates comprehension of any given inspection and its purpose.

Support for Polars DataFrames

PyCharm 2023.2 will allow you to work with a new, blazingly fast DataFrame library written in Rust – Polars. 

In PyCharm, you can work with interactive Polars tables in Jupyter notebooks. In the Python console, you can inspect Polars DataFrames via the View as DataFrame option in the Special Variables list. Both Python and Jupyter debuggers work with Polars as well.  

PyCharm will provide information about the type and dimensions of the tables, complete names and types of the columns, and allow you to use sorting for the tables. 

Note that Polars DataFrames are not supported in Scientific mode.

Please try Polars support and share your feedback with us in the comments section, on Twitter, or in our issue tracker.

Web development

New live templates for Django forms and models

As part of Django support, PyCharm has traditionally provided a list of live templates for Django template files. PyCharm 2023.2 will extend this functionality to Django forms, models, generic views, and admin. Live templates will let you insert common fields for Django views, forms, and models by typing short abbreviations.

You can find the new templates and settings for them in Settings | Editor | Live Templates | Django. To edit the existing templates or create a new one, refer to the PyCharm help page.

The list of live templates that can be used to quickly create Django tags in the template files has also been enlarged. You can find the updated list via Settings | Editor | Live Templates | Django Templates.

Frontend development

Volar support for Vue

We have some great news for those using Vue in PyCharm! We’ve implemented Volar support for Vue to support the changes in TypeScript 5.0. This should provide more accurate error detection, aligned with the Vue compiler. The new integration is still in early development and we would appreciate it if you could give it a try and provide us with any feedback you have.

To set the Vue service to use Volar integration on all TypeScript versions, go to Settings | Languages & Frameworks | TypeScript | Vue. By default, Volar will be used for TypeScript versions 5.0 and higher, and our own implementation will be used for TypeScript versions lower than 5.0.

In the future, we’ll consider enabling the Volar integration by default instead of our own implementation used for Vue and TypeScript.

CSS: Convert color to LCH and OKLCH

In PyCharm 2022.3, we added support for the new CSS color modification functions. This provided PyCharm users with a number of color conversion actions. For instance, you can change RGB to HSL, and vice versa. We are expanding this support in PyCharm 2023.2 to include conversion of LCH and OKLCH with other color functions.

Next.js custom documentation support

Next.js 13.1 now includes a plugin for the TypeScript Language Service specifically for the new app directory. This plugin offers suggestions for configuring pages and layouts, as well as helpful hints for using both Server and Client Components. It also comes with custom documentation, which adds extra information to the output of the TypeScript Language Service. It’s now possible to view this custom documentation in PyCharm.

VCS: GitLab integration

PyCharm 2023.2 EAP 2 introduces initial integration with GitLab, allowing you to work with the Merge Request functionality right from the IDE, streamlining your development workflow. To add your GitLab account go to Settings | Version Control | GitLab.

Notable bug fixes

We fixed the issue with debugging multiprocessing code on MacOS ARM that was caused by a missing dylib file. [PY-48163]

For PowerShell 7, venv is now activated correctly in the Terminal. [PY-58019]

These are the most notable updates for this week. To see the full list of changes in this EAP build, please refer to the release notes.

If you encounter any bugs while working with this build, please submit a report using our issue tracker. If you have any questions or feedback, let us know in the comments below or get in touch with our team on Twitter.

June 01, 2023 03:06 PM UTC

Stack Abuse

Simple NLP in Python with TextBlob: Lemmatization

Introduction

TextBlob is a package built on top of two other packages, one of them is called Natural Language Toolkit, known mainly in its abbreviated form as NLTK, and the other is Pattern. NLTK is a traditional package used for text processing or Natural Language Processing (NLP), and Pattern is built mainly for web mining.

TextBlob is designed to be easier to learn and manipulate than NLTK, while maintaining the same important NLP tasks such as lemmatization, sentiment analysis, stemming, POS-tagging, noun phrase extraction, classification, translation, and more. You can see a complete list of tasks on the PyPI's TextBlob page.

If you are looking for a practical overview of many NLP tasks that can be executed with TextBlob, take a look at our "Python for NLP: Introduction to the TextBlob Library" guide.

There are no special technical prerequisites needed for employing TextBlob. For instance, the package is applicable for both Python 2 and 3 (Python >= 2.7 or >= 3.5).

Also, in case you don't have any textual information at hand, TextBlob provides the necessary collections of language data (usually texts), called corpora, from the NLTK database.

Installing TextBlob

Let's start by installing TextBlob. If you are using a terminal, command-line, or command prompt, you can enter:

$ pip install textblob

Otherwise, if you are using a Jupyter Notebook, you can execute the command directly from the notebook by adding an exclamation mark ! at the beginning of the instruction:

!pip install textblob

Note: This process can take some time due to the broad number of algorithms and corpora that this library contains.

After installing TextBlob, in order to have text examples, you can download the corpora by executing the python -m textblob.download_corpora command. Once again, you can execute it directly in the command line or in a notebook by preceding it with an exclamation mark.

When running the command, you should see the output below:

$ python -m textblob.download_corpora
[nltk_data] Downloading package brown to /Users/csamp/nltk_data...
[nltk_data]   Package brown is already up-to-date!
[nltk_data] Downloading package punkt to /Users/csamp/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to /Users/csamp/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /Users/csamp/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package conll2000 to /Users/csamp/nltk_data...
[nltk_data]   Unzipping corpora/conll2000.zip.
[nltk_data] Downloading package movie_reviews to
[nltk_data]     /Users/csamp/nltk_data...
[nltk_data]   Unzipping corpora/movie_reviews.zip.
Finished.

We have already installed the TextBlob package and its corpora. Now, let's understand more about lemmatization.

For more TextBlob content, check out our Simple NLP in Python with TextBlob: Tokenization, Simple NLP in Python with TextBlob: N-Grams Detection, and Sentiment Analysis in Python with TextBlob guides.

What is Lemmatization?

Before going deeper into the field of NLP, you should be able to recognize some key terms:

Corpus (or corpora in plural) - is a specific collection of language data (e.g., texts). Corpora are typically used for training various models of text classification or sentiment analysis, for instance.

Lemma - is the word you would look for in a dictionary. For instance, if you want to look at the definition for the verb "runs", you would search for "run".

Stem - is a part of a word that never changes.

What is lemmatization itself?

Lemmatization is the process of obtaining the lemmas of words from a corpus.

An illustration of this could be the following sentence:

• Input (corpus): Alice thinks she is lost, but then starts to find herself
• Output (lemmas): | Alice | think | she | is | lost | but | then | start | to | find | herself |

Notice that each word in the input sentence is lemmatized according to its context in the original sentence. For instance, "Alice" is a proper noun, so it stays the same, and the verbs "thinks" and "starts" are referenced in their base forms of "think" and "start".

Lemmatization is one of the basic stages of language processing. It brings words to their root forms or lemmas, which we would find if we were looking for them in a dictionary.

In the case of TextBlob, lemmatization is based on a database called WordNet, which is developed and maintained by Princeton University. Behind the scenes, TextBlob uses WordNet's morphy processor to obtain the lemma for a word.

Note: For further reference on how lemmatization works in TextBlob, you can take a peek at the documentation.

You probably won't notice significant changes with lemmatization unless you're working with large amounts of text. In that case, lemmatization helps reduce the size of words we might be searching for while trying to preserve their context in the sentence. It can be applied further in developing models of machine translation, search engine optimization, or various business inquiries.

Implementing Lemmatization in Code

First of all, it's necessary to establish a TextBlob object and define a sample corpus that will be lemmatized later. In this initial step, you can either write or define a string of text to use (as in this guide), or we can use an example from the NLTK corpus we have downloaded. Let's go with the latter.

Choosing a Review from the NLTK Corpus

For example, let's try to obtain the lemmas for a movie review that is in the corpus. To do this, we import both the TextBlob library and the movie_reviews from the nltk.corpus package:

# importing necessary libraries
from textblob import TextBlob
from nltk.corpus import movie_reviews

After importing, we can take a look at the movie reviews files with the fileids() method. Since this code is running in a Jupyter Notebook, we can directly execute:

movie_reviews.fileids()

This will return a list of 2,000 text file names containing negative and positive reviews:

['neg/cv000_29416.txt',
 'neg/cv001_19502.txt',
 'neg/cv002_17424.txt',
 'neg/cv003_12683.txt',
 'neg/cv004_12641.txt',
 'neg/cv005_29357.txt',
 'neg/cv006_17022.txt',
 'neg/cv007_4992.txt',
 'neg/cv008_29326.txt',
 'neg/cv009_29417.txt',
 ...]

Note: If you are running the code in another way, for instance, in a terminal or IDE, you can print the response by executing print(movie_reviews.fileids()).

By looking at the neg in the name of the file, we can assume that the list starts with the negative reviews and ends with the positive ones. We can look at a positive review by indexing from the end of the list. Here, we are choosing the 1,989th review:

movie_reviews.fileids()[-10]

This results in:

'pos/cv990_11591.txt'

To examine the review sentences, we can pass the name of the review to the .sents() method, which outputs a list of all review sentences:

movie_reviews.sents('pos/cv990_11591.txt')

[['the', 'relaxed', 'dude', 'rides', 'a', 'roller', 'coaster',
'the', 'big', 'lebowski', 'a', 'film', 'review', 'by', 'michael',
'redman', 'copyright', '1998', 'by', 'michael', 'redman', 'the',
'most', 'surreal', 'situations', 'are', 'ordinary', 'everyday',
'life', 'as', 'viewed', 'by', 'an', 'outsider', '.'], ['when',
'those', 'observers', 'are', 'joel', 'and', 'ethan', 'coen', ',',
'the', 'surreal', 'becomes', 'bizarre', '.'], ...]

Let's store this list in a variable called pos_review:

pos_review = movie_reviews.sents("pos/cv990_11591.txt")
len(pos_review) #returns 63

Here, we can see that there are 63 sentences. Now, we can select one sentence to lemmatize, for instance, the 15th sentence:

sentence = pos_review[16]
type(sentence) # returns list

Creating a TextBlob Object

After selecting the sentence, we need to create a TextBlob object to be able to access the .lemmatize() method. TextBlob objects need to be created from strings. Since we have a list, we can convert it to a string with the string.join() method, joining based on blank spaces:

sentence_string = ' '.join(sentence)

Now that we have our sentence string, we can pass it to the TextBlob constructor:

blob_object = TextBlob(sentence_string)

Once we have the TextBlob object, we can perform various operations, such as lemmatization.

Lemmatization of a Sentence

Finally, to get the lemmatized words, we simply retrieve the words attribute of the created blob_object. This gives us a list containing Word objects that behave very similarly to string objects:

# Word tokenization of the sentence corpus
corpus_words = blob_object.words
# To see all tokens
print('sentence:', corpus_words)
# To count the number of tokens
number_of_tokens = len(corpus_words)
print('\nnumber of tokens:', number_of_tokens)

The output commands should give you the following:

sentence: ['the', 'carpet', 'is', 'important', 'to', 'him', 'because', 'it', 'pulls', 'the', 'room', 'together', 'not', 'surprisingly', 'since', 'it', 's', 'virtually', 'the', 'only', 'object', 'there']

number of tokens: 22

To lemmatize the words, we can just use the .lemmatize() method:

corpus_words.lemmatize()

This gives us a lemmatized WordList object:

WordList(['the', 'carpet', 'is', 'important', 'to', 'him', 'because', 'it', 'pull', 'the',
'room', 'together', 'not', 'surprisingly', 'since', 'it', 's', 'virtually', 'the', 'only',
'object', 'there'])

Since this might be a little difficult to read, we can do a loop and print each word before and after lemmatization:

for word in corpus_words:
    print(f'{word} | {word.lemmatize()}')

This results in:

the | the
carpet | carpet
is | is
important | important
to | to
him | him
because | because
it | it
pulls | pull
the | the
room | room
together | together
not | not
surprisingly | surprisingly
since | since
it | it
s | s
virtually | virtually
the | the
only | only
object | object
there | there

Notice how "pulls" changed to "pull"; the other words, besides "it's," were also lemmatized as expected. We can also see that "it's" has been separated due to the apostrophe. This indicates we can further pre-process the sentence so that "it's" is considered a word instead of "it" and an "s".

Difference Between Lemmatization and Stemming

Lemmatization is often confused with another technique called stemming. This confusion occurs because both techniques are usually employed to reduce words. While lemmatization uses dictionaries and focuses on the context of words in a sentence, attempting to preserve it, stemming uses rules to remove word affixes, focusing on obtaining the stem of a word.

Let's quickly modify our for loop to look at these differences:

print('word | lemma | stem\n')
for word in corpus_words:
    print(f'{word} | {word.lemmatize()} | {word.stem()}')

This outputs:

the | the | the
carpet | carpet | carpet
is | is | is
important | important | import
to | to | to
him | him | him
because | because | becaus
it | it | it
pulls | pull | pull
the | the | the
room | room | room
together | together | togeth
not | not | not
surprisingly | surprisingly | surprisingli
since | since | sinc
it | it | it
s | s | s
virtually | virtually | virtual
the | the | the
only | only | onli
object | object | object
there | there | there

When looking at the above output, we can see how stemming can be problematic. It reduces "important" to "import", losing all the meaning of the word, which can even be considered a verb now; "because" to "becaus", which is a word that doesn't exist, same for "togeth", "surprisingli", "sinc", "onli".

There are clear differences between lemmatization and stemming. Understanding when to utilize each technique is the key. Suppose you are optimizing a word search and the focus is on being able to suggest the maximum amount of similar words, which technique would you use? When word context doesn't matter, and we could retrieve "important" with "import", the clear choice is stemming. On the other hand, if you are working on document text comparison, in which the position of the words in a sentence matters, and the context "importance" needs to be maintained and not confused with the verb "import", the best choice is lemmatization.

In the last scenario, suppose you are working on a word search followed by a retrieved document text comparison, what will you use? Both stemming and lemmatization.

We have understood the differences between stemming and lemmatization; now let's see how we can lemmatize the whole review instead of just a sentence.

Lemmatization of a Review

To lemmatize the entire review, we only need to modify the .join(). Instead of joining words in a sentence, we will join sentences in a review:

# joining each sentence with a new line between them, and a space between each word
corpus_words = '\n'.join(' '.join(sentence) for sentence in pos_review)

After transforming the corpus into a string, we can proceed in the same way as it was for the sentence to lemmatize it:

blob_object = TextBlob(pos_rev)
corpus_words = blob_object.words
corpus_words.lemmatize()

This generates a WordList object with the full review text lemmatized. Here, we are omitting some parts with an ellipsis (...) since the review is large, but you will be able to see it in its integral form. We can spot our sentence in the middle of it:

WordList(['the', 'relaxed', 'dude', 'rides', 'a', 'roller', 'coaster', 'the', 'big',
'lebowski', 'a', 'film', 'review', 'by', 'michael', 'redman', 'copyright', '1998', 'by',
'michael', 'redman', 'the', 'most', 'surreal', 'situations', 'are', 'ordinary', 'everyday',
'life', 'as', 'viewed', 'by', 'an', 'outsider', 'when', 'those', 'observers', 'are', 'joel',
(...) 

'the', 'carpet', 'is', 'important', 'to', 'him', 'because', 'it', 'pulls', 'the', 'room',
'together', 'not', 'surprisingly', 'since', 'it', 's', 'virtually', 'the', 'only', 'object',
'there'

(...)
'com', 'is', 'the', 'eaddress', 'for', 'estuff'])

Conclusion

After lemmatizing the sentence and the review, we can see that both extract the corpus words first. This means lemmatization occurs at a word level, which also implies that it can be applied to a word, a sentence, or a full text. It works for a word or any collection of words.

This also suggests that it might be slower since it is necessary to break the text first into tokens to later apply it. And since lemmatization is context-specific, as we have seen, it is also crucial to have a good pre-processing of the text before using it, ensuring the correct breakdown into tokens and the appropriate part of speech tagging. Both will enhance results.

If you are not familiar with Part of Speech tagging (POS-tagging), check our Python for NLP: Parts of Speech Tagging and Named Entity Recognition guide.

We have also seen how lemmatization is different from stemming, another technique for reducing words that doesn't preserve their context. For this reason, it is usually faster.

There are many ways to perform lemmatization, and TextBlob is a great library for getting started with NLP. It offers a simple API that allows users to quickly begin working on NLP tasks. Leave a comment if you have used lemmatization in a project or plan to use it.

Happy coding!

June 01, 2023 12:24 PM UTC

Tryton News

Newsletter June 2023

Changes for the User

In CSV exports with the option selected to use locale format, we now use the local time zone for date-time fields.

In the project_invoice module we now show the invoice line field on the time sheet line form. This is useful to get an understanding of the invoiced hours. The field is not shown if the user doesn’t have read access to invoice lines.

To have a clearer picture of the debts of a company, we’ve added receivables and payables to it.

Invoice lines now have pay, block and unblock payment buttons to be able to directly create payments from within the invoice form.

Changes for the System Administrator

Now, when saving a CSV export definition, the options to export listed records or selected records and to ignore search limit are also saved. Printing an export as a report will also make use of the new saved options.

We removed migrations prior to 5.0.

Changes for the Developer

We moved the representation of sum in XML from the bottom row of the Tryton client into the column header. And the sum attribute is now converted into a boolean type.

Authors: @udono @dave @pokoli

1 post - 1 participant

Read full topic

June 01, 2023 08:00 AM UTC

Matt Layman

New SaaS Signup - Building SaaS with Python and Django #161

In this episode, we dug into the first portion of the journey SaaS. I acquired the domain name of journeyinbox.com for this service. That service is not live yet. We started at the beginning to set up users and sign up features.

June 01, 2023 12:00 AM UTC

May 31, 2023

Test and Code

202: Using Towncrier to Keep a Changelog

Hynek joins the show to discuss towncrier.

At the top of the towncrier documentation, it says "towncrier is a utility to produce useful, summarized news files (also known as changelogs) for your project."

Towncrier is used by "Twisted, pytest, pip, BuildBot, and attrs, among others."

This is the last of 3 episodes focused on keeping a CHANGELOG.

Episode 200 kicked off the series with keepachangelog.com and Olivier Lacan
In 201 we had Ned Batchelder discussing scriv.

Special Guest: Hynek Schlawack.

Links:

• Towncrier docs
• How to Keep a Changelog in Markdown - Towncrier docs
• Keep a Changelog
• structlog/CHANGELOG.md — Example of manually edited changelog.
• hatch-fancy-pypi-readme
• MyST Markdown
• hatchling

May 31, 2023 09:15 PM UTC

Python Software Foundation

The Python Language Summit 2023: What is the Standard Library for?

 Brett Cannon came to the Python Language Summit this year with a fundamental question for the assembled core developers: What is the standard library for?

According to a quick python -c "import sys; print(len(sys.stdlib_module_names))" call on my laptop, the standard library in Python 3.11 consists of 305 importable modules. Many of these are implementation details that, if you’re a good citizen, you really shouldn’t be importing – but the point stands that the Python standard library is perhaps now larger than it should be.

But the goal of his question, Cannon explained, wasn’t to decide which modules to get rid of. Instead, it was to create guidelines on when and why new modules should be accepted into the standard library.

"We need to audit the standard library, and not deprecate it, but decide which bits should probably not have been added if we had perfect hindsight. 

-- Guido van Rossum, CPython Core Developer and former BDFL

Carol Willing agreed that the core dev team shouldn’t be looking to remove modules en masse, but should decide what kinds of modules they wanted to admit in the future. Łukasz Langa agreed, and pointed out that it was often hard removing modules even when we wanted to, due to the fact that “the standard library is a huge import cycle”.

Where do we go now?

Cannon himself put forward two possible answers to his question, before tossing it out to the audience:

1. The standard library should contain everything required to bootstrap an installer.
2. The standard library should make it easy for beginners to be able to write scripts without installing anything.

The conversation was free-flowing, but a common point of consensus among the attendees was that the standard library should focus on tools and utilities that allow users to write better Python code. Hynek Schlawack cited dataclasses as an example of a module that made writing classes much less painful, and generally led to them writing better code as a result. (Schlawack is the author of the attrs library, the third-party inspiration for dataclasses, which itself is still going strong.) Filipe Laíns agreed, arguing that the core dev team should focus on building business implementations for third-party libraries to build on top of.

“The default answer for ‘Should this be in the standard library?’ should be ‘No’, but we should bless smaller utilities that help people write better Python code” 

-- Antonio Cuni, HPy Core Developer

There was a certain amount of regret in the air about modules that perhaps should never have been added to the standard library, and had proved themselves to be significant maintenance burdens in the years since, but could now never be removed. tkinter, it was universally agreed, was the primary example here; possibly multiprocessing also.

Guido van Rossum pondered whether asyncio should ever have been added to the standard library, remarking that it had been difficult to evolve asyncio while it was in the standard library, and had possibly been added before it was “fully baked”. The ssl integration had probably been a mistake, he said, and should have been left to third parties.

Łukasz Langa noted that modules such as asyncio and typing, which had continued to evolve rapidly after being added to the standard library, had helped spur new syntax changes to Python that had been to the language’s betterment. Without asyncio in the standard library, Langa argued, we would probably never have adopted the async/await syntax that is now the foundation of asynchronous Python programming.

Zac Hatfield-Dods, maintainer of several prominent third-party packages, said that different standard-library packages had different impacts on the Python ecosystem. Pytest, one of the libraries he maintains, had managed to flourish and find success despite the existence of unittest in the standard library. But another library he helps out with, the asynchronous Trio framework, had struggled to attract users while asyncio had been part of the standard library. “Nobody supports alternative async implementations,” he complained, despite Trio’s development often being years ahead of where asyncio is. (In the coffee break afterwards, Hatfield-Dods was keen to emphasise that he is, in fact, a fan of asyncio and the work of the asyncio maintainers.)

Zac Hatfield-Dods (left), speaking at the Language Summit (Photo by Hugo van Kemenade)

Cannon brought up the question of whether a module like pathlib belonged. “It’s just sugar,” he remarked – i.e., hardly a “core utility” or a protocol that allowed people to write better code. But it has nonetheless been one of the more popular additions to the standard library in recent years. Langa again pushed back, arguing that without the addition of pathlib to the standard library, we would never have added os.PathLike, a protocol that had allowed a common interface for describing file-system paths in Python. “A third-party PyPI package wouldn’t have convinced us to make that change,” Langa argued.

Several attendees noted that adding a module to the standard library often made it hard for users to use features added to the module in newer versions of Python, due to CPython’s slow development cycle. One solution could to provide third-party versions of standard-library modules on PyPI, backporting the latest features of a module to older versions of Python. Thomas Wouters argued that previous attempts at providing these backport modules had often been disastrous. However, Jelle Zijlstra noted that typing_extensions, which backports features from the latest version of the typing module, had been incredibly successful (though it was sometimes hard to maintain).

Overall, there was agreement that the original motivations for a large, “batteries-included” standard library no longer held up to scrutiny. “In the good old days,” Ned Deily reminisced, “We said ‘batteries-included’ because we didn’t have a good story for third-party installation.” But in 2023, installing third-party packages from PyPI is much easier.

Often, Thomas Wouters noted, people preferred using standard-library modules in a corporate setting due to the fact that the installation of any third-party package would require approval from their company’s IT department. But, he noted, this was hardly Python’s problem.

May 31, 2023 06:21 PM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Brasil
  • Planet Python Poland
• Python Libraries
  • OLPC
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Mathspp Pydon'ts"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Paolo Amoroso's Journal"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Alec Munro
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • Armin Ronacher
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • BeDjango
  • Ben Bass
  • Ben Cook
  • Ben Rousch
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • BioPython News
  • Bit of Cheese
  • Bojan Mihelac
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Okken
  • Bruno Oliveira
  • Bruno Ponne / Coding The Past
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Düvel
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Casey Duncan
  • Catherine Devlin
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Miles
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • CodeGrades
  • CodeSnipers
  • CodersLegacy
  • Coding Diet
  • Corey Gallon
  • Corey Goldberg
  • Cormoran Project
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dallas Fort Worth Pythoneers
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Data Community DC
  • Data School
  • Dave Beazley
  • David Amos
  • David Caron
  • David Goodger
  • David Lindelof
  • David MacIver
  • David Malcolm
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Declassed Art
  • Denis Kurov
  • Django Weblog
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Dougal Matthews
  • Duncan McGreggor
  • Ed Crewe
  • Edward K. Ream
  • Eli Bendersky
  • Eniram Ltd.
  • Eray Özkural (examachine)
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evennia
  • Everyday Superpowers
  • Fabio Zadrozny
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • Gocept Weblog
  • Godson Gera
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Gustavo Narea
  • Gustavo Niemeyer
  • Gökhan Sever
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Inspired Python
  • Ionel Cristian Maries
  • IronPython-URLs
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Velichko
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • Janusworx
  • Jarrod Millman
  • Jean-Paul Calderone
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Burns
  • John Cook
  • John Jacobsen
  • John Ludhi/nbshare.io
  • Jon Parise
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan Street
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Julien Danjou
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lintel Technologies
  • Linux Stans
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mats Kindahl
  • Matt Layman
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Matthew Wright
  • Mattias Brändström
  • Mauveweb
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Nelson
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mirek Długosz
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Naomi Ceder
  • Natan Zabkar
  • Ned Batchelder
  • Neil Schemenauer
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • Nsukami Patrick
  • Obey the Testing Goat
  • Ofosos
  • Omaha Python Users Group
  • Ondřej Čertík
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patrick Kennedy
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • Pedro Lima
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyPy
  • PyTennessee
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python GUIs
  • Python Insider
  • Python Morsels
  • Python Piedmont Triad User Group
  • Python Pool
  • Python Software Foundation
  • Python Sweetness
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • Pythonicity
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Rene Dudfield
  • Reuven Lerner
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rishi Maker
  • Rob Galanakis
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Rodrigo Araúj
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Sandipan Dey
  • Sandro Tosi
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Senthil Kumaran
  • Shannon -jj Behrens
  • ShiningPanda
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Holden
  • Steven Klass
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Suresh Dasari/Tutlane.com
  • Swisscom ICT
  • Talk Python to Me
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Terri Oda
  • Terry Jones
  • Test and Code
  • TestDriven.io
  • The Data Scientist
  • The Digital Cat
  • The Lunar Cowboy
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Open Sourcerer
  • The Parcon Blog
  • The Python Coding Blog
  • The Python Papers
  • The Three of Wands
  • Thibauld Nion
  • ThisHosting.Rocks
  • Thomas Guest
  • Thomas Vander Stichele
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Knapp
  • Tim Lesher
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Virgil Dupras
  • Vladimir Perić
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • Wing Tips
  • Wingware
  • Wyatt Baldwin
  • Yaniv Aknin
  • Yann Larrivée
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • death and gravity
  • eGenix.com
  • hypothesis.works articles
  • kdev-python
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • scikit-learn
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github