Planet Python

Last update: January 08, 2018 01:46 PM

January 08, 2018

Mike Driscoll

PyDev of the Week: Nicholas Hunt-Walker

January 08, 2018 01:30 PM

PyPy Development

Leysin Winter sprint: 17-24 March 2018

The next PyPy sprint will be in Leysin, Switzerland, for the thirteenth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. (Note: this sprint is independent from the suggested April-May sprint in Poland.) Goals and topics of the sprint The list of topics is open, but here is our current list:

• cffi tutorial/overview rewrite
• py3 test runners are too complicated
• make win32 builds green
• make packaging more like cpython/portable builds
• get CI builders for PyPy into mainstream projects (Numpy, Scipy, lxml, uwsgi)
• get more of scientific stack working (tensorflow?)
• cpyext performance improvements
• General 3.5 and 3.6 improvements
• JIT topics: guard-compatible, and the subsequent research project to save and reuse traces across processes
• finish unicode-utf8
• update www.pypy.org, speed.pypy.org (web devs needed)

As usual, the main side goal is to have fun in winter sports :-) We can take a day off (for ski or anything else).

Exact times

Work days: starting March 18th (~noon), ending March 24th (~noon).

Please see announcement.txt for more information.

January 08, 2018 11:33 AM

Programiz

Reading CSV files in Python

In this article, we will learn how to read data from csv files in python of different formats.

January 08, 2018 09:40 AM

Working with CSV files in Python

In this article we will look into how to read and write CSV files in python.

January 08, 2018 08:22 AM

Python List vs. Tuples

In this article we will learn key differences between the List and Tuples and how to use these two data structure.

January 08, 2018 04:48 AM

January 07, 2018

Glyph Lefkowitz

Tips And Tricks for Shipping a PyGame App on the Mac

I’ve written and spoken at some length about shipping software in the abstract. Sometimes I’ve even had the occasional concrete tidbit, but that advice wasn’t really complete.

In honor of Eevee’s delightful Games Made Quick???, I’d like to help you package your games even quicker than you made them.

Who is this for?

About ten years ago I made a prototype of a little PyGame thing which I wanted to share with a few friends. Building said prototype was quick and fun, and very different from the usual sort of work I do. But then, the project got just big enough that I started to wonder if it would be possible to share the result, and thus began the long winter of my discontent with packaging tools.

I might be the only one, but... I don’t think so. The history of PyWeek, for example, looks to be a history of games distributed as Github repositories, or, at best, apps which don’t launch. It seems like people who participate in game jams with Unity push a button and publish their games to Steam; people who participate in game jams with Python wander away once the build toolchain defeats them.

So: perhaps you’re also a Python programmer, and you’ve built something with PyGame, and you want to put it on your website so your friends can download it. Perhaps many or most of your friends and family are Mac users. Perhaps you tried to make a thing with py2app once, and got nothing but inscrutable tracebacks or corrupt app bundles for your trouble.

If so, read on and enjoy.

What changed?

If things didn’t work for me when I first tried to do this, what’s different now?

• the packaging ecosystem in general is far less buggy, and py2app’s dependencies, like setuptools, have become far more reliable as well. Many thanks to Donald Stufft and the whole PyPA for that.
• Binary wheels exist, and the community has been getting better and better at building self-contained wheels which include any necessary C libraries, relieving the burden on application authors to figure out gnarly C toolchain issues.
• The PyGame project now ships just such wheels for a variety of Python versions on Mac, Windows, and Linux, which removes a whole huge pile of complexity both in generally understanding the C toolchain and specifically understanding the SDL build process.
• py2app has been actively maintained and many bugs have been fixed - many thanks to Ronald Oussoren et. al. for that.
• I finally broke down and gave Apple a hundred dollars so I can produce an app that normal humans might actually be able to run.

There are still weird little corner cases you have to work around — hence this post – but mostly this is the story of how years of effort by the Python packaging community have resulted in tools that are pretty close to working out of the box now.

Step 0: Development Setup

Get a good Python. Use Homebrew, and brew install python3. If you need python 2, brew install python2. Don’t use the System python. Probably nothing will work.

You probably also want to use a virtualenv for development. This post is about how to build a for-real thing that other people can download, but part of the magic of Python is the interactive, real-time dynamic nature of everything. Running the full build pipeline every time you change a file or an asset is slow and annoying. However, there’s a weird thing where certain parts of the macOS GUI won’t work right (in PyGame’s case, mostly keyboard focus) unless your code appears to be in an application bundle.

I made this dumb little thing which lets you fake out enough of this that the OS won’t hassle you: you just need to pip install venvdotapp; venvdotapp inside the virtualenv where you’re making your pygame app.

Finally: pip install all your requirements into your virtualenv, including PyGame itself.

Step 1: Make an icon

All good apps need an icon, right?

When I was young, you just popped over into ResEdit Resorcerer MPW CodeWarrior Project Builder Icon Composer Xcode and created a new ICON resource cicn resource .tiff file .icns file. Nowadays there’s some weird opaque stuff with xcassets files and Contents.json and “Copy Bundle Resources” in the default Swift and Objective C project templates and honestly I can’t be bothered to keep track of what’s going on with this nonsense any more.

Luckily the OS ships with the macOS-specific “scriptable image processing system”, which can helpfully convert an icon for you. Make yourself a 512x512 PNG file in your favorite image editor (with an alpha channel!) that you want to use as your icon, then run it something like this:

1

$ sips -s format icns Icon.png --out Icon.icns

somewhere in your build process, to produce an icon in the appropriate format.

There’s also one additional wrinkle with PyGame: once you’ve launched the game, PyGame helpfully assigns the cute, but ugly, default PyGame icon to your running process. To avoid this, you’ll need these two lines somewhere in your initialization code, somewhere before pygame.display.init (or, for that matter, pygame.display.<anything>):

1
2

from pygame.sdlmain_osx import InstallNSApplication
InstallNSApplication()

Obviously this is pretty Mac-specific so you probably want this under some kind of platform-detection conditional, perhaps this one.

Step 2: Just Include All The Dang Files, I Don’t Care About Performance

Unfortunately py2app still tries really hard to jam all your code into a .zip file, which breaks the world in various hilarious ways. Your app will probably have some resources you want to load, as will PyGame itself.

Supposedly, packages=["your_package"] in your setup.py should address this, and it comes with a “pygame” recipe, but neither of these things worked for me. Instead, I convinced py2app to just splat out all the files by using the not-quite-public “recipe” plugin API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

import py2app.recipes
import py2app.build_app

from setuptools import find_packages, setup

pkgs = find_packages(".")

class recipe_plugin(object):
    @staticmethod
    def check(py2app_cmd, modulegraph):
        local_packages = pkgs[:]
        local_packages += ['pygame']
        return {
            "packages": local_packages,
        }

py2app.recipes.my_recipe = recipe_plugin

APP = ['my_main_file.py']
DATA_FILES = []
OPTIONS = {}
OPTIONS.update(
    iconfile="Icon.icns",
    plist=dict(CFBundleIdentifier='com.example.yourdomain.notmine')
)

setup(
    name="Your Game",
    app=APP,
    data_files=DATA_FILES,
    include_package_data=True,
    options={'py2app': OPTIONS},
    setup_requires=['py2app'],
    packages=pkgs,
    package_data={
        "": ["*.gal" , "*.gif" , "*.html" , "*.jar" , "*.js" , "*.mid" ,
             "*.png" , "*.py" , "*.pyc" , "*.sh" , "*.tmx" , "*.ttf" ,
             # "*.xcf"
        ]
    },
)

This is definitely somewhat less efficient than py2app’s default of stuffing the code into a single zip file, but, as a counterpoint to that: it actually works.

Step 3: Build it

Hopefully, at this point you can just do python setup.py py2app and get a shiny new app bundle in dist/$NAME.app. We haven’t had to go through the hell of quarantine just yet, so it should launch at this point. If it doesn’t, sorry :-(.

You can often debug more obvious fail-to-launch issues by running the executable in the command line, by running ./dist/$NAME.app/Contents/MacOS/$NAME. Although this will run in a slightly different environment than double clicking (it will have all your shell’s env vars, for example, so if your app needs an env var to work it might mysteriously work there) it will also print out any tracebacks to your terminal, where they’ll be slightly easier to find than in Console.app.

Once your app at least runs locally, it’s time to...

Step 4: Code sign it

All the tutorials that I’ve found on how to do this involve doing Xcode project goop where it’s not clear what’s happening underneath. But despite the fact that the introductory docs aren’t quite there, the underlying model for codesigning stuff is totally common across GUI and command-line cases. However, actually getting your cert requires Xcode, an apple ID, and a credit card.

After paying your hundred dollars, go into Xcode, go to Accounts, hit “+”, “Apple ID”, then log in. Then, in your shiny new account, go to “Manage Certificates”, hit the little “+”, and (assuming, like me, you want to put something up on your own website, and not submit to the Mac App Store), and choose Developer ID Application. You probably think you want “mac app distribution” because you are wanting to distribute a mac app! But you don’t.

Next, before you do anything else, make sure you have backups of your certificate and private key. You really don’t want to lose the private key associated with that cert.

Now quit Xcode; you’re done with the GUI.

You will need to know the identifier of your signing key though, which should be output from the command:

1

$ security find-identity -v -p codesigning | grep 'Developer ID' | sed -e 's/.*"\(.*\)"/\1/'

You probably want to put that in your build script, since you want to sign with the same identity every time. The command to do the signing is:

1

$ codesign -fs "${YOUR_DEVELOPER_ID_IDENTITY}" --deep "dist/${NAME}.app"

Step 5: Archive it

The right way to do this is probably to use dmgbuild or something like it, but what I promised here was quick and dirty, not beautiful and best practices.

You have to make an archive that preserves symbolic links. There are a few options for this:

• open dist/, then in the Finder window that comes up, right click on the app and “compress” it
• cd dist; tar czf $NAME.app.tgz $NAME.app
• cd dist; zip -yr $NAME.app.zip $NAME.app

Most importantly, if you use the zip command line tool, you must use the -y option. Without it, your downloadable app bundle will be somewhat mysteriously broken even though the one before you zipped it will be fine.

Step 6: Download it

Ideally, at this point, everything should be working. But to make sure that code-signing and archiving went correctly, you should have either a pristine virtual machine with no dev tools and no Python installed, or a non-programmer friend’s machine that can serve the same purpose. They probably need a relatively recent macOS - I know that apps made using the above technique will definitely work on High Sierra and will definitely break on Yosemite; they probably start working at some OS version between those.

There’s no tooling that I know of that can clearly tell you whether your mac app depends on some detail of your local machine. Even for your dependencies, there’s no auditwheel for macOS. So it’s always a good idea to check your final app build on a fresh computer before you announce it.

Coda

If you were expecting to get to the end and download my cool game, sorry to disappoint! It really is a half-broken prototype that is in no way ready for public consumption, and given my current load of personal and professional responsibilities, you definitely shouldn’t expect anything from me in this area any time soon, or, you know, ever.

But, from years of experience, I know that it’s nearly impossible to summon any motivation to work on small projects like this without the knowledge that the end result will be usable in some way, so I hope that this helps someone else set up their Python game-dev pipeline.

I’d really like to turn this into a 3-part series, with a part for Linux (perhaps using flatpak? is that a good thing?) and a part for Windows. However, given my aforementioned time constraints, I don’t think I’m going to have the time or energy to do that research, so if you’ve got the appropriate knowledge, I’d love to host a guest post on this blog, or even just a link to yours.

If this post helped you, if you have questions or corrections, or if you’d like to write the Linux or Windows version of this post, let me know.

January 07, 2018 08:48 PM

Low Kian Seong

Easily staying afloat on High Sierra

January 07, 2018 07:06 PM

Filipe Saraiva

Discussing the future of Cantor

Hello devs! Happy new year!

It is common to use the new year date to start new projects or give new directions for old ones. The last one is the case for Cantor.

Since when I got the maintainer status for Cantor, I was working to improve the community around the software. Because the great plugins systems of Qt, it is easy to write new backends for Cantor, and in fact in last years Cantor reached the number of 11 backends.

If in a hand it is a nice thing because Cantor can run different mathematical engines, in other hand it is very common developers create backends, release them with Cantor upstream, and forget this piece of software after some months. The consequence of this is a lot of unsolved bugs in Bugzilla, unexpected behaviours of some backends, and more.

For instance, R backend is broken from some years right now (thanks Rishabh it was fixed during his GSoC/KDE Edu Sprint 2017 but not released yet). Sage backend breaks for each new release of Sage.

Different backends use different technologies. Scilab and Octave backends use QProcess + Standard Streams; Python 2 uses Python/C API; Python 3, R, and Julia use D-Bus.

In addition to these, remember each programming language used as mathematical engine for Cantor has their respective release schedule and it is very common new versions break the way as backends are implemented.

So, yes, the mainternhip of Cantor is a hell.

In order to remedy it I invited developers to be co-maintainer of these respective backends, but it does not have the effect I was suposed to. I implemented a way to present the versions of programming languages supported in the backend but it does not work well too.

So, my main work in Cantor during these years was try to solve bugs of backends I don’t use and, sometimes, I don’t know how they work, while new features were impossible to be planned and implemented.

If we give a look to Jupyter, the main software for notebook-based mathematical computation, it is possible to see this software supports several programming languages. But, in fact, this support is provide by the community – Jupyter focus effort in Python support only (named the ipython kernel) and in new features for Jupyter itself.

So, I would like to hear the KDE and Cantor community about the future of Cantor. My proposal is split the code of the others backends and put them as third-party plugins, maintained by their respective community. Only the Python 3 backend would be “officially” maintaned and delivered in KDE Applications bundle.

This way I could focus in provide new features and I could to say “well, this bug with X backend must be reported to the X backend community because they are accountable for this piece of software”.

So, what do you think about?

January 07, 2018 02:07 PM

Import Python

#157 - Pandas, TensorFlow, Extending Python 3 in Go and more

Worthy Read GoCD - Open Source Continuous Delivery Server GoCD is a continuous delivery tool supporting modern infrastructure with elastic on-demand agents and cloud deployments. With GoCD, you can easily model, orchestrate and visualize complex workflows from end to end. It’s open source, free to use and download. advert Pandas for beginners: How to handle real-life data Handling real life datasets can be painful when you are used to cleaned and ‘ready-to-use’ datasets that are used in books, tutorials and beginner challenges in Data Science. This tutorial aims to provide some useful tips and codes to get started with the Pandas library and data provided by your company or client. pandas Recognizing Handwritten digits with TensorFlow DeepLearning is a subfield of machine learning that is a set of algorithms and functions inspired by the structure and fucntioning of the brain. TensorFlow is a machine learning framework that Google created and is used to design, build and train deep learning models. This tutorial is an attempt on the MNIST dataset from this Kaggle competition while also explaining the basics of writing TensorFlow code. deep learning , tensorflow fleep Determining file format using Python project , user submission HelloSign eSign API The Industry’s Fastest eSignature API Integration. Embed docs directly on your website with a few lines of code. Test the API for free. advert A SQLite Tutorial with Python This tutorial will cover using SQLite in combination with Python's sqlite3 interface. sqlite3 Extending Python 3 in Go Extending Python has been a core feature of the platform for decades, the Python runtime provides a “C API”, which is a set of headers and core types for writing extensions in C and compiling them into Python modules. But, do you really have to write extensions to Python in C? Why can’t we use something a tad more modern, like Go. golang A $1200 Deep Learning Rig Inspired by several other system builds ($1000, $1700, and forum posts), I decided to have a go and build one. I was a Sr Director of Data Science for a large travel company at the time and was a bit envious of the work being done by the individual scientists. I was also contemplating a change in employment (with some downtime)?—?I wanted to ensure I had access to resources to continue my deep learning leveling-up. Finally, I wanted to make sure I could demonstrate to my kids the Internet’s most important task: distinguishing between “cat” and “not cat”. deep learning , offtopic Retiring Python as a Teaching Language For the last ten years, my standard advice to someone looking for a programming language to teach beginners has been start with Python. And now I'm changing that recommendation. teaching Talk Python to Me: #145 2017 Python Year in Review podcast Building a simple Redis-like server in Python The other day the idea occurred to me that it would be neat to write a simple Redis-like database server. While I've had plenty of experience with WSGI applications, a database server presented a novel challenge and proved to be a nice practical way of learning how to work with sockets in Python. In this post I'll share what I learned along the way. When and how to use Django TemplateView Django provides several class based generic views to accomplish common tasks. Simplest among them is TemplateView. TemplateView should be used when you want to present some information in a html page. TemplateView shouldn't be used when your page has forms and does creation or update of objects. django , template Jobs Full Stack Web Developer at Beauhurst London, United Kingdom If you’re a fearless generalist with at least a year of commercial experience who loves working with Django and Python, then this could be the perfect opportunity for you. Projects Parris - 237 Stars, 13 Fork Parris, the automated infrastructure setup tool for machine learning algorithms. SafeText - 55 Stars, 3 Fork Script to remove homoglyphs and zero-width characters to allow for safe distribution of documents from anonymous sources. xkeysnail - 29 Stars, 1 Fork Yet another keyboard remapping tool for X environment. vidDistill - 23 Stars, 3 Fork Automated (YouTube) Video Summarization Using Captions. flask-docker - 16 Stars, 2 Fork Fastest way to ship Python web apps, anywhere. Be shipping (using Docker, Flask, Gunicorn, Whitenoise). CryptoTracker - 15 Stars, 2 Fork Create and track your crypto currency portfolio from the terminal. tensorflow-grad-cam - 12 Stars, 0 Fork Tensorflow Slim Grad-Cam to Explain Neural Network Predictions with Heatmap or Shading Flask-Person-Detector - 10 Stars, 2 Fork Flask based web application that provides a REST endpoint using OpenCV's Deep Neural Network method for Object Detection. LineDistiller - 9 Stars, 3 Fork A data-driven line extractor for 2D anime, manga and illustration using Keras. 24-game - 5 Stars, 2 Fork 24 point game implemented in Python, just for fun!. flask-validates - 5 Stars, 1 Fork Painless form validation using view decorators. mastering-pycharm-course - 4 Stars, 0 Fork Course demos and handouts for Talk Python's Mastering PyCharm course. GitHub-contributions - 4 Stars, 0 Fork Get details about all the projects to which you have contributed to on GitHub. tweetsender - 3 Stars, 0 Fork Get tweets of particular users into your gmail inbox without using twitter app. ExpenseManager - 3 Stars, 0 Fork An application to manage all your personal expenditures. Add all your expenses along with the amount spent, the date and time of your spending and other additional description.

January 07, 2018 12:03 PM

Sandipan Dey

Classifying a Face as Happy/Unhappy and Face Recognition using Deep Convolution Net with Keras in Python

In this article couple of problems are going to be described. Both the problems appeared as assignments in the Coursera course Convolution Neural Network (a part of deeplearning specialization) by the Stanford Prof. Andrew Ng. (deeplearning.ai). The problem descriptions are taken from the course itself. 1. Classifying a Face Image as Happy/Unhappy Given: 600 64×64 … Continue reading Classifying a Face as Happy/Unhappy and Face Recognition using Deep Convolution Net with Keras in Python

January 07, 2018 12:15 AM

January 06, 2018

Weekly Python StackOverflow Report

(cvii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2018-01-06 20:26:43 GMT

1. Weird behaviour of non-ASCII Python identifiers - [7/1]
2. Python multiple inheritance is not showing class variables or method of second inherited base class - [6/5]
3. How to reliably check if a domain has been registered or is available? - [6/1]
4. What's the best way to enumerate permutations of deck of cards? - [6/1]
5. Vectorization to achieve performance - [5/3]
6. Comparing dates in python, == works but <= produces error - [5/3]
7. why I don't get clean data when i use cleaned_data - [5/2]
8. Scrapy: downloader/response_count vs response_received_count - [5/1]
9. scrapy: Middleware/Pipeline single instance - [5/1]
10. Handling large numbers python 2.7(runtime error) - [5/1]

January 06, 2018 08:27 PM

January 05, 2018

Peter Bengtsson

Whatsdeployed facelift

tl;dr; Whatsdeployed.io is an impressively simple web app to help web developers and web ops people quickly see what GitHub commits have made it into your Dev, Stage or Prod environment. Today it got a facelift.

The code is now more than 5 years old and has served me well. It's weird to talk too positively about the app because I actually wrote it but because it's so simple in terms of design and effort it feels less personal to talk about it.

Here's what's in the facelift

• Upgraded to Bootstrap 4.
• Instead of relying on downloading a heavy Glyphicon web font, just to display a single checkmark, that's now a simple image.
• Ability to use a GitHub developer personal token to avoid rate limitations on GitHub's API.
• The first lookup to get all commits is now done via the Flask app to use my auth token to avoid the rate limit.
• Much better error handling if any of the underlying requests.get() that the Flask app does, fails. Also includes which URL it failed on.
• Basic validation to prevent submitting the main form without typing anything in.
• You can hack on it with Docker. Thanks @willkg.
• Improved the code that extracts Bugzilla bug numbers out of commit messages. Thanks @edmorely.
• Refreshed screenshots in the README.md
• A brand new introduction text on the home page for people who end up on the site not knowing what it is.
• If any XHR errors happen figuring out the "culprits", you now get a pretty error describing this instead of swallowing it all.

Please let me know if there's anything broken or missing.

January 05, 2018 06:37 PM

Mike Driscoll

Top Ten Python Articles of 2017

January 05, 2018 06:15 PM

Codementor

How to Build RESTful APIs with Python and Flask

For sometime I have been working with Python but I just got to try Flask recently, so I felt it would be nice to write about it! Flask is a Python-based microframework that enables you to quickly...

January 05, 2018 05:27 PM

Evennia

New year, new stuff

January 05, 2018 10:39 AM

Tryton News

Newsletter January 2018

As new year resolution, we plan to make a monthly post about what happened in the Tryton project.

January 05, 2018 10:00 AM

Kushal Das

Using split ssh in QubesOS 4.0

The idea behind Qubes OS is known as security by compartmentalization. You create different Qubes (VMs or domains) to compartmentalize your digital data. So that even if one of the VMs is compromised, the attacker will not be able to access data stored in other VMs.

If we look into a typical GNU/Linux user’s daily routine, ssh is a regular tool everyday. We do login to various systems, or access files over ssh. But, if you keep the ssh keys in the place where you are also running the browser, there is a chance that someone will try to access the files by attacking through the browser. Yesterday we all read many things which can be done by attacking through the browsers (Yay! SECURITY!!!).

In this tutorial, we will learn about split-ssh and how we can keep the actual ssh keys safe in QubesOS. At the time of writing this article (2018-01-05), the commit in the master branch is 1b1786f5bac9d06af704b5fb3dd2c59f988767cb.

Modify the template VM

Because we will be adding things to /etc directory of our VMs, we will have to do this in the template VM. Because in the normal VMs the /etc directory will be a fresh copy every time we restart the VM. I modified fedora-26 as that is my default template.

First, add the following code in the /etc/qubes-rpc/qubes.SshAgent file in the template VM and then shut it down.

#!/bin/sh
notify-send "[`qubesdb-read /name`] SSH agent access from: $QREXEC_REMOTE_DOMAIN"
ncat -U $SSH_AUTH_SOCK

Creating the actual ssh-vault VM

Next task is to create a new VM, I named it ssh-vault. The name is important to remember as the code/configuration will access the ssh keys based on the vault VM name. You can have as many ssh vaults as you want. Remember to open the configuration after creation and set the networking to None.

Start the vault VM, either create a new pair of ssh key, or copy your existing key in there. Remember to use qvm-copy command to copy the files, no network is available.

[Desktop Entry]
Name=ssh-add
Exec=ssh-add
Type=Application

Then add the above content to the ~/.config/autostart/ssh-add.desktop file. You may have to create the autostart directory.

$ mkdir -p .config/autostart
# vim ~/.config/autostart/ssh-add.desktop

Configuring the client VM

Client VM is the VM in which you use the ssh key. Add the following to the /rw/config/rc.local file, and then make the file executable. Remember to use sudo for the same.

SSH_VAULT_VM="ssh-vault"

if [ "$SSH_VAULT_VM" != "" ]; then
	export SSH_SOCK=~user/.SSH_AGENT_$SSH_VAULT_VM
	rm -f "$SSH_SOCK"
	sudo -u user /bin/sh -c "umask 177 && ncat -k -l -U '$SSH_SOCK' -c 'qrexec-client-vm $SSH_VAULT_VM qubes.SshAgent' &"
fi

If you look carefully at the shell scrip above, you will find we are setting the vault VM name using a variable called SSH_VAULT_VM. Change this name to whatever VM you want to use as the vault.

$ sudo vim /rw/config/rc.local
$ sudo chmod +x /rw/config/rc.local

Next, we will add the following to the ~/.bashrc file, so that ssh can find the right socket file.

# Set next line to the ssh key vault you want to use
SSH_VAULT_VM="ssh-vault"

if [ "$SSH_VAULT_VM" != "" ]; then
	export SSH_AUTH_SOCK=~user/.SSH_AGENT_$SSH_VAULT_VM
fi

Then I restarted the vault and client VMs. Because my ssh key also has a passphrase, I entered that using ssh-add command in the ssh-vault VM.

Configuring the policy in dom0

In QubesOS you will have to define a policy in the dom0, based on that the VMs can talk to each other (using QubeOS’ internal). In my case I want only the emails VM should be able to ask to get access to the ssh keys. So, I added the following in /etc/qubes-rpc/policy/qubes.SshAgent file.

emails ssh-vault ask

The above policy rule says that when the emails VM tries to contact ssh-vault VM, it has to ask for permission to do so from the user.

Using ssh (finally!)

At this moment you can safely start the client VM, and try to ssh into anywhere. It will open up an authentication dialog, you will have to select and click on Okay button to give access to the ssh keys. You will also see a notification in the top notification area.

There is an active IRC channel #qubesin the Freenode server. Join there and ask any doubts you have.

January 05, 2018 09:59 AM

Python Bytes

#59 Instagram disregards Python's GC (again)

January 05, 2018 08:00 AM

Amit Saha

Compilation and Interpretation in C and CPython

It is common knowledge that programs written in high level languages have to be translated into a low level language using programs referred to as translators. This low level language is either in a native form, in the sense that it is understood by the operating system itself, or in an intermediate form which is understood by an intermediate program such as the bytecode interpreter. It is also known that C is a compiled language, whereas CPython is both first compiled and then interpreted.

In this article, I will try to illustrate the difference between the two languages by carrying out some simple experiments (on Linux).

Consider a program, helloworld.c:

# include<stdio.h>

int main(int argc, char **argv)
{
   printf("Hello World!\n");
   return 0;
}

We can compile and execute this program as follows:

$ gcc -o helloworld helloworld.c
$ ./helloworld
Hello World

The executable file helloworld is the low level language equivalent of the high level language program, helloworld.c. This is what the operating system on your computer understands and hence when it is executed, it prints Hello World on the screen. This process of converting helloworld.c to helloworld represents the translation process. In the case of C, this translation process is performed by the compiler, gcc (this translation process is really a process pipeline and involves two other processes preprocessing and linking, which are carried out by separate programs, automatically invoked by gcc). Nevertheless, compilation is at the core of the translation process of C programs and is responsible for converting a high level language to its low level equivalent - a version readily executable by the operating system on your computer. It is important to note that this executable file is composed of the instructions you wrote in your C program, along with a other details which are necessary to execute your program. These details are specific to the architecture and operating system you created the executable file on and hence if you copy a executable file you created on a computer with an Intel processor, it will not work at all on a computer which has an ARM processor, for example. Hence, you will have to recompile the program on the new computer before you can execute it. Note that your C program still remains the same, but the low level machine language equivalent is different and gcc takes care of this translation.

Now, let us consider our first CPython program:

# Print Hello world
if __name__=='__main__':
    print 'Hello World!'

You executed this program as follows

$ python helloworld.py
Hello World!

Unlike C, where you compiled the program first to get a separate executable file and then executed it (a two step process), here you executed the program in a single step - your program is directly executed on-the-fly. This is how traditionally interpreters (interpreted languages) worked. However, modern day interpreted languages like CPython (and others) also involve a compilation step. Your program helloworld.py is first converted to an intermediate representation which is a low level equivalent of your high level language program. The difference from C is that the instructions in these low level language equivalents are not meant to be executed by a real computer, but a process virtual machine [1]. In the case of CPython, the intermediate representation is known as bytecodes and the virtual machine referred to as the bytecode interpreter or the CPython virtual machine. Hence, the CPython code is first converted into its bytecode equivalent which is then executed by the bytecode interpreter. When you run a CPython program using python helloworld.py, both these steps happen in the background.

While discussing the section on C compilation I mentioned that the executable you create on an Intel computer will not run on an ARM computer, because of the architecture specific instructions embedded into the executable required for executing the program. In the case of CPython the bytecodes (result of compilation of the CPython program) are executed by the CPython virtual machine, instead of the real computer. This extra layer of abstraction allows you to execute the same bytecodes (without recompiling) on an Intel computer and an ARM computer, for example.

Let us understand this better with a real example. I will use two computers for the experiments: System1 and System2, with both running Fedora Linux. However, System1's instruction set architecture is x86_64 (Intel) where as System2 is a RaspberryPi with an armv6l (ARM) instruction set.

$ arch
x86_64
$ gcc -o helloworld helloworld.c
$ file ./helloworld
./helloworld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0xc50d74290927cb25ef9e34055af6c437e89ed5eb, not stripped

$ arch
armv6l
$ file helloworld
helloworld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0xc50d74290927cb25ef9e34055af6c437e89ed5eb, not stripped
$ ./helloworld
-bash: ./helloworld: cannot execute binary file

$ gcc -o helloworld helloworld.c
$ file helloworld
helloworld: ELF 32-bit LSB executable, ARM, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0xba57691af19ff94f894645398e66e263c8f57a9b, not stripped
$ ./helloworld
Hello World!

$ python -c "import py_compile;py_compile.compile('helloworld.py')"

$ file helloworld.pyc
helloworld.pyc: python 2.7 byte-compiled

$ arch
armv6l
$ file helloworld.pyc
helloworld.pyc: python 2.7 byte-compiled
$ python helloworld.pyc
Hello World!

$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0x9d8a414b778ff11ec075995248c43cdf5b67f17a, stripped

$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 32-bit LSB executable, ARM, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0x63fd81d3591769d6be0619b7273935ab9521010c, stripped

January 05, 2018 12:00 AM

January 04, 2018

Stack Abuse

Scheduling Jobs with python-crontab

What is Crontab

Cron is a software utility that allows us to schedule tasks on Unix-like systems. The name is derived from the Greek word "Chronos", which means "time".

The tasks in Cron are defined in a crontab, which is a text file containing the commands to be executed. The syntax used in a crontab is described below in this article.

Python presents us with the crontab module to manage scheduled jobs via Cron. The functions available in it allow us to access Cron, create jobs, set restrictions, remove jobs, and more. In this article we will show how to use these operations from within yhour Python code.

For the interested reader, the official help page can be found at https://pypi.python.org/pypi/python-crontab.

Crontab Syntax

Cron uses a specific syntax to define the time schedules. It consists of five fields, which are separated by white spaces. The fields are:

Minute Hour Day Month Day_of_the_Week  

The fields can have the following values:

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23) 
│ │ ┌───────────── day of month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday;
│ │ │ │ │                                       7 is also Sunday on some systems)
│ │ │ │ │
│ │ │ │ │
* * * * *  command to execute

Source: Wikipedia. Cron. Available at https://en.wikipedia.org/wiki/Cron

Cron also acccepts special characters so you can create more complex time schedules. The special characters have the following meanings:

Let's see some examples:

1. * * * * * means: every minute of every hour of every day of the month for every month for every day of the week.
2. 0 16 1,10,22 * * tells cron to run a task at 4 PM (which is the 16th hour) on the 1st, 10th and 22nd day of every month.

Installing Crontab

Crontab is not included in the standard Python installation. Thus, the first thing we have to do is to install it.

This is done with the pip command. The only thing to consider is that the name of the module is 'python-crontab', and not just 'crontab'. The following command will install the package in our machine:

$ pip install python-crontab

Getting Access to Crontab

According to the crontab help page, there are five ways to include a job in cron. Of them, three work on Linux only, and two can also be used on Windows.

The first way to access cron is by using the username. The syntax is as follows:

cron = CronTab(user='username')  

The other two Linux ways are:

cron = CronTab()

# or

cron = CronTab(user=True)  

There are two more syntaxes that will also work on Windows.

In the first one, we call a task defined in the file "filename.tab":

cron = CronTab(tabfile='filename.tab')  

In the second one, we define the task according to cron's syntax:

cron = CronTab(tab="""* * * * * command""")  

Creating a New Job

Once we have accessed cron, we can create a new task by using the following command:

cron.new(command='my command')  

Here, my command defines the task to be executed via the command line.

We can also add a comment to our task. The syntax is as follows:

cron.new(command='my command', comment='my comment')  

Let's see this in an example:

from crontab import CronTab

cron = CronTab(user='username')  
job = cron.new(command='python example1.py')  
job.minute.every(1)

cron.write()  

In the above code we have first accessed cron via the username, and then created a job that consists of running a Python script named example1.py. In addition, we have set the task to be run every 1 minute. The write() function adds our job to cron.

The example1.py script is as follows:

from datetime import datetime  
myFile = open('append.txt', 'a')  
myFile.write('\nAccessed on ' + str(datetime.now()))  

As we can see from the above code, the program will open and append the phrase "Accessed on" with the access date and time added.

The result is as follows:

Figure 1

As we expected, Figure 1 shows that the file was accessed by the program. It will continue to do the assigned task while the example1.py program is running on cron.

Once cron is accessed, we can add more than one job. For example the following line in above example would add a second task to be managed by cron:

job2 = cron.new(command='python example2.py')  

Once a new task is added, we can set restrictions for each of them.

Setting Restrictions

One of the main advantages of using Python's crontab module is that we can set up time restrictions without having to use cron's syntax.

In the example above, we have already seen how to set running the job every minute. The syntax is as follows:

job.minute.every(minutes)  

Similarly we could set up the hours:

job.hour.every(hours)  

We can also set up the task to be run on certain days of the week. For example:

job.dow.on('SUN')  

The above code will tell cron to run the task on Sundays, and the following code will tell cron to schedule the task on Sundays and Fridays:

job.dow.on('SUN', 'FRI')  

Similarly, we can tell cron to run the task in specific months. For example:

job.month.during('APR', 'NOV')  

This will tell cron to run the program in the months of April and November.

An important thing to consider is that each time we set a time restriction, we nullify the previous one. Thus, for example:

job.hour.every(5)  
job.hour.every(7)  

The above code will set the final schedule to run every seven hours, cancelling the previous schedule of five hours.

Unless, we append a schedule to a previous one, like this:

job.hour.every(15)  
job.hour.also.on(3)  

This will set the schedule as every 15 hours, and at 3 AM.

The 'every' condition can be a bit confusing at times. If we write job.hour.every(15), this will be equivalent to * */15 * * *. As we can see, the minutes have not been modified.

If we want to set the minutes field to zero, we can use the following syntax:

job.every(15).hours()  

This will set the schedule to 0 */4 * * *. Similarly for the 'day of the month', 'month' and 'day of the week' fields.

Examples:

1. job.every(2).month is equivalent to 0 0 0 */2 * and job.month.every(2) is equivalent to * * * */2 *
2. job.every(2).dows is equivalent to 0 0 * * */2 and job.dows.every(2) is equivalent to * * * * */2

We can see the differences in the following example:

from crontab import CronTab

cron = CronTab(user='username')

job1 = cron.new(command='python example1.py')

job1.hour.every(2)

job2 = cron.new(command='python example1.py')  
job2.every(2).hours()

for item in cron:  
    print item

cron.write()  

After running the program, the result is as follows:

$ python cron2.py
* */2 * * * python /home/eca/cron/example1.py
0 */2 * * * python /home/eca/cron/example1.py  
$

Figure 2

As we can see in Figure 2, the program has set the second task's minutes to zero, and defined the first task minutes' to its default value.

Finally, we can set the task to be run every time we boot our machine. The syntax is as follows:

job.every_reboot()  

Clearing Restrictions

We can clear all task's restrictions with the following command:

job.clear()  

The following code shows how to use the above command:

from crontab import CronTab

cron = CronTab(user='username')

job = cron.new(command='python example1.py', comment='comment')  
job.minute.every(5)

for item in cron:  
    print item

job.clear()

for item in cron:  
    print item
cron.write()  

After running the code we get the following result:

$ python cron3.py
*/5 * * * * python /home/eca/cron/example1.py # comment
* * * * * python /home/eca/cron/example1.py # comment

Figure 3

As we can see in Figure 3, the schedule has changed from every 5 minutes to the default setting.

Enabling and Disabling a Job

A task can be enabled or disabled using the following commands:

To enable a job:

job.enable()  

To disable a job:

job.enable(False)  

In order to verify whether a task is enabled or disabled, we can use the following command:

job.is_enabled()  

The following example shows how to enable and disable a previously created job, and verify both states:

from crontab import CronTab

cron = CronTab(user='username')

job = cron.new(command='python example1.py', comment='comment')  
job.minute.every(1)

cron.write()

print job.enable()  
print job.enable(False)  

The result is as follows:

$ python cron4.py
True  
False  

Figure 4

Checking Validity

We can easily check whether a task is valid or not with the following command:

job.is_valid()  

The following example shows how to use this command:

from crontab import CronTab

cron = CronTab(user='username')

job = cron.new(command='python example1.py', comment='comment')  
job.minute.every(1)

cron.write()

print job.is_valid()  

After running the above program, we obtain the validation, as seen in the following figure:

$ python cron5.py
True  

Figure 5

Listing All Cron Jobs

All cron jobs, including disabled jobs can be listed with the following code:

for job in cron:  
    print job

Adding those lines of code to our first example will show our task by printing on the screen the following:

$ python cron6.py
* * * * * python /home/eca/cron/example1.py

Figure 6

Finding a Job

The Python crontab module also allows us to search for tasks based on a selection criterion, which can be based on a command, a comment, or a scheduled time. The syntaxes are different for each case.

Find according to command:

cron.find_command("command name")  

Here 'command name' can be a sub-match or a regular expression.

Find according to comment:

cron.find_comment("comment")  

Find according to time:

cron.find_time(time schedule)  

The following example shows how to find a previously defined task, according to the three criteria previously mentioned:

from crontab import CronTab

cron = CronTab(user='username')

job = cron.new(command='python example1.py', comment='comment')  
job.minute.every(1)

cron.write()

iter1 = cron.find_command('exam')  
iter2 = cron.find_comment('comment')  
iter3 = cron.find_time("*/1 * * * *")

for item1 in iter1:  
    print item1

for item2 in iter2:  
    print item2

for item3 in iter3:  
    print item3

The result is the listing of the same job three times:

$ python cron7.py
* * * * * python /home/eca/cron/example1.py # comment
* * * * * python /home/eca/cron/example1.py # comment
* * * * * python /home/eca/cron/example1.py # comment

Figure 7

As you can see, it correctly finds the cron command each time.

Removing Jobs

Each job can be removed separately. The syntax is as follows:

cron.remove(job)  

The following code shows how to remove a task that was previously created. The program first creates the task. Then, it lists all tasks, showing the one just created. After this, it removes the task, and shows the resulting empty list.

from crontab import CronTab

cron = CronTab(user='username')

job = cron.new(command='python example1.py')  
job.minute.every(1)

cron.write()  
print "Job created"

# list all cron jobs (including disabled ones)
for job in cron:  
    print job

cron.remove(job)  
print "Job removed"

# list all cron jobs (including disabled ones)
for job in cron:  
    print job

The result is as follows:

$ python cron8.py
Job created  
* * * * * python /home/eca/cron/example1.py
Job removed  

Figure 8

Jobs can also be removed based on a condition. For example:

cron.remove_all(comment='my comment')  

This will remove all jobs where comment='my comment'.

Clearing All Jobs

All cron jobs can be removed at once by using the following command:

cron.remove_all()  

The following example will remove all cron jobs and show an empty list.

from crontab import CronTab

cron = CronTab(user='username')  
cron.remove_all()

# list all cron jobs (including disabled ones)
for job in cron:  
    print job

Environmental Variables

We can also define environmental variables specific to our scheduled task and show them on the screen. The variables are saved in a dictionary. The syntax to define a new environmental variable is as follows:

job.env['VARIABLE_NAME'] = 'Value'  

If we want to get the values for all the environmental variables, we can use the following syntax:

job.env  

The example below defines two new environmental variables for the task 'user', and shows their value on the screen. The code is as follows:

from crontab import CronTab

cron = CronTab(user='username')

job = cron.new(command='python example1.py')  
job.minute.every(1)  
job.env['MY_ENV1'] = 'A'  
job.env['MY_ENV2'] = 'B'

cron.write()

print job.env  

After running the above program, we get the following result:

$ python cron9.py
MY_ENV1=A  
MY_ENV2=B  

Figure 9

In addition, Cron-level environment variables are stored in 'cron.env'.

Wrapping Up

The Python module crontab provides us with a handy tool to programmatically manage our cron application, which is available to Unix-like systems. By using it, instead of having to rely on creating crontabs, we can use Python code to manage frequent tasks.

The module is quite complete. Although there have been some criticisms about its behavior, it contains functions to connect to cron, create scheduled tasks, and manage them. As shown in the above examples, their use is quite direct. Thus, it provides a tool that allows for complex scripts with the main Python characteristic: simplicity.

January 04, 2018 02:45 PM

Kushal Das

Using diceware to generate passwords

Choosing a new password is always an interesting topic. When I started using computers for the first time, my idea was to find some useful words which I can remember, maybe 2-3 of those words together. With time I found that the websites have different requirements when it comes to choosing a new password. But, in the last few years we also saw many examples where brute forcing a password is a rather simple thing. The modern powerful computers enable anyone to find a right combination of characters in a decent time frame.

What is a diceware password?

Diceware passwords are normal passwords (a few words together) generated from a list of words by either rolling a dice, or by computer. You can read more in the original Diceware website.

Using diceware project to generate your passphrases

If you notice, I have written passphrase instead of password. This is because passphrases are not only easier to remember than a complex password, but they also provide better security from bruteforce attacks. The following comics from XKCD explains it better than any words.

Installing diceware

diceware is a very simple command line tool written in Python. It can help you to choose a diceware passphrase easily. It was already packaged for Debian, last week I have packaged it for Fedora (thank you Parag for the review). Yesterday night it was pushed to stable. So, now you can install it using dnf.

$ sudo dnf install diceware

Using diceware

$ diceware 
MotorBolsterFountainThrowerPorridgeBattered

By default it is creating passphrases with 6 words in it, but you can increase by using -n command line argument. You should use at least 7 words in your passphrase. Read the story from Micah Lee to understand how this helps to increase the strength of your passphrases in many folds.

The man page of the diceware has more details about usage.

Start using a password manager

Now is a good time to start using a password manager. Save all the passwords/passphrases in one place, and secure it with a super long passphrase which you can remember. This article from Martin Shelton has many examples. The members of Fedora engineering team uses a command line tool called pass which uses gpg to encrypt the passwords.

January 04, 2018 09:11 AM

Codementor

An Introduction to this New Kid Called MACHINE LEARNING

A beginner's Introduction to the world of machine learning and Artificial Intelligence.

January 04, 2018 08:45 AM

Catalin George Festila

Python 2.7 : InsecurePlatformWarning error.

This is not a common error and can be solve it easy like any python issue.
The result of this error can be show like into the next example:

c:\python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\util\ssl_.py:318: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension 
to TLS is not available on this platform. This may cause the server to present an incorrect TLS 
certificate, which can cause validation failures. You can upgrade to a newer version of Python to
 solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html
#snimissingwarning.
  SNIMissingWarning
c:\python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\util\ssl_.py:122: 
InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from
 configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade 
to a newer version of Python to solve this. For more information, see https://urllib3.readthe
docs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

pip install urllib3 
pip install requests

Successfully installed certifi-2017.11.5 chardet-3.0.4 idna-2.6 requests-2.18.4

C:\Python27>python
Python 2.7 (r27:82525, Jul  4 2010, 07:43:08) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> help()

Welcome to Python 2.7!  This is the online help utility.

If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.

Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules.  To quit this help utility and
return to the interpreter, just type "quit".

To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics".  Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".

help> modules requests

Here is a list of matching modules.  Enter any module name to get more help.

pip._vendor.cachecontrol.controller - The httplib2 algorithms ported for use with requests.
pip._vendor.requests - Requests HTTP library
pip._vendor.requests.adapters - requests.adapters
pip._vendor.requests.api - requests.api
pip._vendor.requests.auth - requests.auth
pip._vendor.requests.certs - requests.certs
pip._vendor.requests.compat - requests.compat
pip._vendor.requests.cookies - requests.cookies
pip._vendor.requests.exceptions - requests.exceptions
pip._vendor.requests.hooks - requests.hooks
pip._vendor.requests.models - requests.models
pip._vendor.requests.packages
pip._vendor.requests.packages.chardet
pip._vendor.requests.packages.chardet.big5freq
pip._vendor.requests.packages.chardet.big5prober
pip._vendor.requests.packages.chardet.chardetect - Script which takes one or more file paths 
and reports on their detected
pip._vendor.requests.packages.chardet.chardistribution
pip._vendor.requests.packages.chardet.charsetgroupprober
pip._vendor.requests.packages.chardet.charsetprober
pip._vendor.requests.packages.chardet.codingstatemachine
pip._vendor.requests.packages.chardet.compat
pip._vendor.requests.packages.chardet.constants
pip._vendor.requests.packages.chardet.cp949prober
pip._vendor.requests.packages.chardet.escprober
pip._vendor.requests.packages.chardet.escsm
pip._vendor.requests.packages.chardet.eucjpprober
pip._vendor.requests.packages.chardet.euckrfreq
pip._vendor.requests.packages.chardet.euckrprober
pip._vendor.requests.packages.chardet.euctwfreq
pip._vendor.requests.packages.chardet.euctwprober
pip._vendor.requests.packages.chardet.gb2312freq
pip._vendor.requests.packages.chardet.gb2312prober
pip._vendor.requests.packages.chardet.hebrewprober
pip._vendor.requests.packages.chardet.jisfreq
pip._vendor.requests.packages.chardet.jpcntx
pip._vendor.requests.packages.chardet.langbulgarianmodel
pip._vendor.requests.packages.chardet.langcyrillicmodel
pip._vendor.requests.packages.chardet.langgreekmodel
pip._vendor.requests.packages.chardet.langhebrewmodel
pip._vendor.requests.packages.chardet.langhungarianmodel
pip._vendor.requests.packages.chardet.langthaimodel
pip._vendor.requests.packages.chardet.latin1prober
pip._vendor.requests.packages.chardet.mbcharsetprober
pip._vendor.requests.packages.chardet.mbcsgroupprober
pip._vendor.requests.packages.chardet.mbcssm
pip._vendor.requests.packages.chardet.sbcharsetprober
pip._vendor.requests.packages.chardet.sbcsgroupprober
pip._vendor.requests.packages.chardet.sjisprober
pip._vendor.requests.packages.chardet.universaldetector
pip._vendor.requests.packages.chardet.utf8prober
pip._vendor.requests.packages.urllib3 - urllib3 - Thread-safe connection pooling and re-using.
pip._vendor.requests.packages.urllib3._collections
pip._vendor.requests.packages.urllib3.connection
pip._vendor.requests.packages.urllib3.connectionpool
pip._vendor.requests.packages.urllib3.contrib
pip._vendor.requests.packages.urllib3.contrib.appengine
pip._vendor.requests.packages.urllib3.contrib.ntlmpool - NTLM authenticating pool, 
contributed by erikcederstran
pip._vendor.requests.packages.urllib3.contrib.pyopenssl
pip._vendor.requests.packages.urllib3.contrib.socks - SOCKS support for urllib3
pip._vendor.requests.packages.urllib3.exceptions
pip._vendor.requests.packages.urllib3.fields
pip._vendor.requests.packages.urllib3.filepost
pip._vendor.requests.packages.urllib3.packages
pip._vendor.requests.packages.urllib3.packages.ordered_dict
pip._vendor.requests.packages.urllib3.packages.six - Utilities for writing code that runs on 
Python 2 and 3
pip._vendor.requests.packages.urllib3.packages.ssl_match_hostname
pip._vendor.requests.packages.urllib3.packages.ssl_match_hostname._implementation - The match_hostname() 
function from Python 3.3.3, essential when using SSL.
pip._vendor.requests.packages.urllib3.poolmanager
pip._vendor.requests.packages.urllib3.request
pip._vendor.requests.packages.urllib3.response
pip._vendor.requests.packages.urllib3.util
pip._vendor.requests.packages.urllib3.util.connection
pip._vendor.requests.packages.urllib3.util.request
pip._vendor.requests.packages.urllib3.util.response
pip._vendor.requests.packages.urllib3.util.retry
pip._vendor.requests.packages.urllib3.util.ssl_
pip._vendor.requests.packages.urllib3.util.timeout
pip._vendor.requests.packages.urllib3.util.url
pip._vendor.requests.sessions - requests.session
pip._vendor.requests.status_codes
pip._vendor.requests.structures - requests.structures
pip._vendor.requests.utils - requests.utils
requests - Requests HTTP Library
requests.__version__
requests._internal_utils - requests._internal_utils
requests.adapters - requests.adapters
requests.api - requests.api
requests.auth - requests.auth
requests.certs - requests.certs
requests.compat - requests.compat
requests.cookies - requests.cookies
requests.exceptions - requests.exceptions
requests.help - Module containing bug report helper(s).
requests.hooks - requests.hooks
requests.models - requests.models
requests.packages
requests.sessions - requests.session
requests.status_codes
requests.structures - requests.structures
requests.utils - requests.utils
help>
You are now leaving help and returning to the Python interpreter.
If you want to ask for help on a particular object directly from the
interpreter, you can type "help(object)".  Executing "help('string')"
has the same effect as typing a particular string at the help> prompt.
>>>
...

January 04, 2018 05:58 AM

January 03, 2018

Yasoob Khalid

2017 was good, 2018 will be even better

This is not a programming or Python related post. This is more of my personal reflection on the year 2017. I decided to reflect and evaluate my year and see what I got done and what could have been better. Below I have listed some of the stuff which I was able to get done this year.

Visited UK and Bahrain 

I love travelling and exploring new places. This year I got a chance to visit UK (Cambridge, Newcastle and London) and Bahrain (Manama). I visited these places in January 2017 and learned a ton about both of these places. Next year (2018) I plan on exploring USA.

Participated in a robotics competition 

I participated in a robotics competition for the first time ever! I had never worked with robots before. We programmed our very own Lego robot for the FIRST Global Pakistan regional competition. We won the best design award for our “Aquabot”. It was an amazing experience and I got a chance to make some really good friends.

Formed a new programming club

After coming to Colgate I realized that there was a lack of programming events at the institution. The main reason was the absense of a pure programming focused club. I gathered a team and formed a club called . I did it in my first semester at Colgate and till now we have organized four different events/workshops. We even brought in a speaker from Microsoft who gave a really engaging talk about the future of AI and Machine Learning.

Learned Racquetball 

I used to play football till 2013 but then stopped because of a back-injury. This year I decided to start playing a sport again. I spent three months (Sep-Nov) learning how to play Racquetball. In the next year I am planning on participating in various inter-university Racquetball tournaments. Keeping my fingers crossed!

100 days of running challenge

This year I decided to start running. I had always read about the benefits of running and how it helps in keeping one fit but I had never actually done it. After experiencing an extremely painful emotional breakdown I decided to focus all of my energy into doing something productive, hence I started running. I decided to run 100 days in a row so that I could form a habit of running. Initially it was hard as I couldn’t even run 1Km non-stop. I had to walk after a couple of meters. But now I can successfully run 5Km+ non-stop without any problem. There were a lot of missed days as well but I am glad that I was able to complete my 100 days. My next aim is to take part in a 5k or a 10k run during next year (2018).

Lost 5kgs

This one ties back to the 100 days of running challenge. I was getting fat by not doing any physical exercise. When your whole day revolves around computers there is little to no need to get up as everything can be done on your laptop/pc. I decided to control my diet a bit and bounce back and lose some fat. I am glad that it payed off and I was able to lose 5Kgs. Although a minute amount, it completely changed my personality. Now I talk to people with more confidence. I don’t know how both of these are related but they surely affect each other. A fit body makes you more confident and improves your personality.

Learned ballroom dancing

I had never danced in my life till three months ago. It is actually surprising for a lot of people but yeah its true. I always wanted to dance but never got around to doing it because I was afraid. I was afraid that I was fat and that I couldn’t dance and if I did try it I would end up making a fool out of myself. Well this year I was able to forego that idea/mentality and actually learned how to do ballroom dancing and I am glad that I was able to successfully perform in the biggest dancing event at Colgate University.

Did 2 camping trips

I had never done “real” camping before. As-in I had never gone out on any camping trip where we had to setup our own tents and cook our own food. This year I did exactly that. I went on two camping trips and in one of these trips we actually set up our own tents. We ate hot dogs, marshmallows, smores and played different outdoor games. They were all-around fun trips. Next year I plan on camping in snow. Lets see how it goes!

Learned a new language Arabic

I love learning new languages. I already knew 6 languages (English, Urdu, Punjabi, Siraiki, Hindi, German) but I wanted to learn more! This year I took up elementary Arabic and successfully completed the Elementary level 1 course of Arabic. Now I can converse in around 7 languages

I wrote this using a marker. It says “Bismillah” meaning “in the name of God/Allah”

Became the founding member of a startup

I became the founding member of a food delivery startup called Xpedite. It does not have any web presence right now but will soon launch in Hamilton, NY. It was fun, enlightening and tiring to plan our trial runs at the end of this year right before my final exams but my team was awesome and we had tons of fun along the way!

Participated in a hackathon

I participated in UB Hacking at University of Buffalo. When I went to the States my plan was to participate in as many hackathons as I possibly could but due to certain circumstances I could not do that but still I ended up participating in one hackathon. I did not win anything but I was able to mentor someone at his very first hackathon and I helped him submit his first ever hack. It was a fulfilling experience.

Crossed 100,000 monthly views on my blog

When I started blogging I always had a dream to cross 100,000 monthly page views. I started from 2000-3000 monthly page-views. But I am glad that constant hard-work paid off and now my blog gets 100,000+ page-views each month. I like to think of it as impacting the life of around 100,000 individuals each month.

Blog Views

1.5 Million total page-views for my book

My book successfully crossed the 1.5 Million total page-views this year and I also celebrated the two year anniversary of my book. The book is going strong and so am I

Book Views

Attended my first American Football Game

Although I am still not 100% sure about how the scoring system in American Football actually works I actually enjoyed the game.

Successfully completed the first semester in university

Considering that I am from Pakistan, this was more of a necessity and requirement than an achievement  I am glad that I was able to successfully complete the first semester of my BA in Comp Sci. 1 down 7 to go!

One of my most important takeaway from this year is that never make someone else the reason for your success. Make your own-self the reason for whatever you achieve.

These are the major highlights from the year 2017 and inshAllah I hope to achieve a lot more in 2018! My main aim is to learn how to play a violin. What were the highlights of your 2017? Do share them with me in the comments below

January 03, 2018 06:51 PM

Stack Abuse

A SQLite Tutorial with Python

Introduction

This tutorial will cover using SQLite in combination with Python's sqlite3 interface. SQLite is a single file relational database bundled with most standard Python installs. SQLite is often the technology of choice for small applications, particularly those of embedded systems and devices like phones and tablets, smart appliances, and instruments. However, it is not uncommon to hear it being used for small to medium web and desktop applications.

Creating a Database and Making a Connection

Creating a new SQLite database is as simple as creating a connection using the sqlite3 module in the Python standard library. To establish a connection all you need to do is pass a file path to the connect(...) method in the sqlite3 module, and if the database represented by the file does not exists one will be created at that path.

import sqlite3  
con = sqlite3.connect('/path/to/file/db.sqlite3')  

You will find that in everyday database programming you will be constantly creating connections to your database, so it is a good idea to wrap this simple connection statement into a reusable generalized function.

# db_utils.py
import os  
import sqlite3

# create a default path to connect to and create (if necessary) a database
# called 'database.sqlite3' in the same directory as this script
DEFAULT_PATH = os.path.join(os.path.dirname(__file__), 'database.sqlite3')

def db_connect(db_path=DEFAULT_PATH):  
    con = sqlite3.connect(db_path)
    return con

Creating Tables

In order to create database tables you need to have an idea of the structure of the data you are interested in storing. There are many design considerations that go into defining the tables of a relational database, which entire books have been written about. I will not be going into the details of this practice and will instead leave it up to reader to further investigate.

However, to aid in our discussion of SQLite database programming with Python I will be working off the premise that a database needs to be created for a fictitious book store that has the below data already collected on book sales.

Upon inspecting this data it is evident that it contains information about customers, products, and orders. A common pattern in database design for transactional systems of this type are to break the orders into two additional tables, orders and line items (sometimes referred to as order details) to achieve greater normalization.

In a Python interpreter, in the same directory as the db_utils.py module defined previously, enter the SQL for creating the customers and products tables follows:

>>> from db_utils import db_connect
>>> con = db_connect() # connect to the database
>>> cur = con.cursor() # instantiate a cursor obj
>>> customers_sql = """
... CREATE TABLE customers (
...     id integer PRIMARY KEY,
...     first_name text NOT NULL,
...     last_name text NOT NULL)"""
>>> cur.execute(customers_sql)
>>> products_sql = """
... CREATE TABLE products (
...     id integer PRIMARY KEY,
...     name text NOT NULL,
...     price real NOT NULL)"""
>>> cur.execute(products_sql)

The above code creates a connection object then uses it to instantiate a cursor object. The cursor object is used to execute SQL statements on the SQLite database.

With the cursor created I then wrote the SQL to create the customers table, giving it a primary key along with a first and last name text field and assign it to a variable called customers_sql. I then call the execute(...) method of the cursor object passing it the customers_sql variable. I then create a products table in a similar way.

You can query the sqlite_master table, a built-in SQLite metadata table, to verify that the above commands were successful.

To see all the tables in the currently connected database query the name column of the sqlite_master table where the type is equal to "table".

>>> cur.execute("SELECT name FROM sqlite_master WHERE type='table'")
<sqlite3.Cursor object at 0x104ff7ce0>  
>>> print(cur.fetchall())
[('customers',), ('products',)]

To get a look at the schema of the tables query the sql column of the same table where the type is still "table" and the name is equal to "customers" and/or "products".

>>> cur.execute("""SELECT sql FROM sqlite_master WHERE type='table'
… AND name='customers'""")
<sqlite3.Cursor object at 0x104ff7ce0>  
>>> print(cur.fetchone()[0])
CREATE TABLE customers (  
    id integer PRIMARY KEY,
    first_name text NOT NULL,
    last_name text NOT NULL)

The next table to define will be the orders table which associates customers to orders via a foreign key and the date of their purchase. Since SQLite does not support an actual date/time data type (or data class to be consistent with the SQLite vernacular) all dates will be represented as text values.

>>> orders_sql = """
... CREATE TABLE orders (
...     id integer PRIMARY KEY,
...     date text NOT NULL,
...     customer_id integer,
...     FOREIGN KEY (customer_id) REFERENCES customers (id))"""
>>> cur.execute(orders_sql)

The final table to define will be the line items table which gives a detailed accounting of the products in each order.

lineitems_sql = """  
... CREATE TABLE lineitems (
...     id integer PRIMARY KEY,
...     quantity integer NOT NULL,
...     total real NOT NULL,
...     product_id integer,
...     order_id integer,
...     FOREIGN KEY (product_id) REFERENCES products (id),
...     FOREIGN KEY (order_id) REFERENCES orders (id))"""
>>> cur.execute(lineitems_sql)

Loading the Data

In this section I will be demonstrating how to INSERT our sample data into the tables just created. A natural starting place would be to populate the products table first because without products we cannot have a sale and thus would not have the foreign keys to relate to the line items and orders. Looking at the sample data I see that there are four products:

• Introduction to Combinatorics ($7.99)
• A Guide to Writing Short Stories ($17.99)
• Data Structures and Algorithms ($11.99)
• Advanced Set Theory ($16.99)

The workflow for executing INSERT statements is simply:

1. Connect to the database
2. Create a cursor object
3. Write a parameterized insert SQL statement and store as a variable
4. Call the execute method on the cursor object passing it the sql variable and the values, as a tuple, to be inserted into the table

Given this general outline let us write some more code.

>>> con = db_connect()
>>> cur = con.cursor()
>>> product_sql = "INSERT INTO products (name, price) VALUES (?, ?)"
>>> cur.execute(product_sql, ('Introduction to Combinatorics', 7.99))
>>> cur.execute(product_sql, ('A Guide to Writing Short Stories', 17.99))
>>> cur.execute(product_sql, ('Data Structures and Algorithms', 11.99))
>>> cur.execute(product_sql, ('Advanced Set Theory', 16.99))

The above code probably seems pretty obvious, but let me discuss it a bit as there are some important things going on here. The insert statement follows the standard SQL syntax except for the ? bit. The ?'s are actually placeholders in what is known as a "parameterized query".

Parameterized queries are an important feature of essentially all database interfaces to modern high level programming languages such as the sqlite3 module in Python. This type of query serves to improve the efficiency of queries that are repeated several times. Perhaps more important, they also sanitize inputs that take the place of the ? placeholders which are passed in during the call to the execute method of the cursor object to prevent nefarious inputs leading to SQL injection. The following is a comic from the popular xkcd.com blog describing the dangers of SQL injection.

To populate the remaining tables we are going to follow a slightly different pattern to change things up a bit. The workflow for each order, identified by a combination of customer first and last name and the purchase date, will be:

1. Insert the new customer into the customers table and retrieve its primary key id
2. Create an order entry based off the customer id and the purchase date then retrieve its primary key id
3. For each product in the order determine its primary key id and create a line item entry associating the order and the product

To make things simpler on ourselves let us do a quick look up of all our products. For now do not worry too much about the mechanics of the SELECT SQL statement as we will devote a section to it shortly.

>>> cur.execute("SELECT id, name, price FROM products")
>>> formatted_result = [f"{id:<5}{name:<35}{price:>5}" for id, name, price in cur.fetchall()]
>>> id, product, price = "Id", "Product", "Price"
>>> print('\n'.join([f"{id:<5}{product:<35}{price:>5}"] + formatted_result))
Id   Product                            Price  
1    Introduction to Combinatorics       7.99  
2    A Guide to Writing Short Stories   17.99  
3    Data Structures and Algorithms     11.99  
4    Advanced Set Theory                16.99  

The first order was placed on Feb 22, 1944 by Alan Turing who purchased Introduction to Combinatorics for $7.99.

Start by making a new customer record for Mr. Turing then determine his primary key id by accessing the lastrowid field of the cursor object.

>>> customer_sql = "INSERT INTO customers (first_name, last_name) VALUES (?, ?)"
>>> cur.execute(customer_sql, ('Alan', 'Turing'))
>>> customer_id = cur.lastrowid
>>> print(customer_id)
1  

We can now create an order entry, collect the new order id value and associate it to a line item entry along with the product Mr. Turing ordered.

>>> order_sql = "INSERT INTO orders (date, customer_id) VALUES (?, ?)"
>>> date = "1944-02-22" # ISO formatted date 
>>> cur.execute(order_sql, (date, customer_id))
>>> order_id = cur.lastrowid
>>> print(order_id)
1  
>>> li_sql = """INSERT INTO lineitems 
...       (order_id, product_id, quantity, total)
...     VALUES (?, ?, ?, ?)"""
>>> product_id = 1
>>> cur.execute(li_sql, (order_id, 1, 1, 7.99))

The remaining records are loaded exactly the same except for the order made to Donald Knuth, which will receive two line item entries. However, the repetitive nature of such a task is crying out the need to wrap these functionalities into reusable functions. In the db_utils.py module add the following code:

def create_customer(con, first_name, last_name):  
    sql = """
        INSERT INTO customers (first_name, last_name)
        VALUES (?, ?)"""
    cur = con.cursor()
    cur.execute(sql, (first_name, last_name))
    return cur.lastrowid

def create_order(con, customer_id, date):  
    sql = """
        INSERT INTO orders (customer_id, date)
        VALUES (?, ?)"""
    cur = con.cursor()
    cur.execute(sql, (customer_id, date))
    return cur.lastrowid

def create_lineitem(con, order_id, product_id, qty, total):  
    sql = """
        INSERT INTO lineitems
            (order_id, product_id, quantity, total)
        VALUES (?, ?, ?, ?)"""
    cur = con.cursor()
    cur.execute(sql, (order_id, product_id, qty, total))
    return cur.lastrowid

Awh, now we can work with some efficiency!

You will need to exit() your Python interpreter and reload it to get your new functions to become accessible in the interpreter.

>>> from db_utils import db_connect, create_customer, create_order, create_lineitem
>>> con = db_connect()
>>> knuth_id = create_customer(con, 'Donald', 'Knuth')
>>> knuth_order = create_order(con, knuth_id, '1967-07-03')
>>> knuth_li1 = create_lineitem(con, knuth_order, 2, 1, 17.99)
>>> knuth_li2 = create_lineitem(con, knuth_order, 3, 1, 11.99)
>>> codd_id = create_customer(con, 'Edgar', 'Codd')
>>> codd_order = create_order(con, codd_id, '1969-01-12')
>>> codd_li = create_lineitem(con, codd_order, 4, 1, 16.99)

I feel compelled to give one additional piece of advice as a student of software craftsmanship. When you find yourself doing multiple database manipulations (INSERTs in this case) in order to accomplish what is actually one cumulative task (ie, creating an order) it is best to wrap the subtasks (creating customer, order, then line items) into a single database transaction so you can either commit on success or rollback if an error occurs along the way.

This would look something like this:

try:  
    codd_id = create_customer(con, 'Edgar', 'Codd')
    codd_order = create_order(con, codd_id, '1969-01-12')
    codd_li = create_lineitem(con, codd_order, 4, 1, 16.99)

    # commit the statements
    con.commit()
except:  
    # rollback all database actions since last commit
    con.rollback()
    raise RuntimeError("Uh oh, an error occurred ...")

I want to finish this section with a quick demonstration of how to UPDATE an existing record in the database. Let's update the Guide to Writing Short Stories' price to 10.99 (going on sale).

>>> update_sql = "UPDATE products SET price = ? WHERE id = ?"
>>> cur.execute(update_sql, (10.99, 2))

Querying the Database

Generally the most common action performed on a database is a retrieval of some of the data stored in it via a SELECT statement. For this section I will be demonstrating how to use the sqlite3 interface to perform simple SELECT queries.

To perform a basic multirow query of the customers table you pass a SELECT statement to the execute(...) method of the cursor object. After this you can iterate over the results of the query by calling the fetchall() method of the same cursor object.

>>> cur.execute("SELECT id, first_name, last_name FROM customers")
>>> results = cur.fetchall()
>>> for row in results:
...     print(row)
(1, 'Alan', 'Turing')
(2, 'Donald', 'Knuth')
(3, 'Edgar', 'Codd')

Lets say you would like to instead just retrieve one record from the database. You can do this by writing a more specific query, say for Donald Knuth's id of 2, and following that up by calling fetchone() method of the cursor object.

>>> cur.execute("SELECT id, first_name, last_name FROM customers WHERE id = 2")
>>> result = cur.fetchone()
>>> print(result)
(2, 'Donald', 'Knuth')

See how the individual row of each result is in the form of a tuple? Well while tuples are a very useful Pythonic data structure for some programming use cases many people find them a bit hindering when it comes to the task of data retrieval. It just so happens that there is a way to represent the data in a way that is perhaps more flexible to some. All you need to do is set the row_factory method of the connection object to something more suitable such as sqlite3.Row. This will give you the ability to access the individual items of a row by position or keyword value.

>>> import sqlite3
>>> con.row_factory = sqlite3.Row
>>> cur = con.cursor()
>>> cur.execute("SELECT id, first_name, last_name FROM customers WHERE id = 2")
>>> result = cur.fetchone()
>>> id, first_name, last_name = result['id'], result[1], result['last_name']
>>> print(f"Customer: {first_name} {last_name}'s id is {id}")
Customer: Donald Knuth's id is 2  

Conclusion

In this article I gave a brief demonstration of what I feel are the most important features and functionalities of the sqlite3 Python interface to the lightweight single file SQLite database that comes pre-bundled with most Python installs. I also tried to give a few bits of advices regarding best practices when it comes to database programming, but I do caution the new-comer that the intricacies of database programming is generally one of the most prone to security holes at the enterprise level and further knowledge is necessary before such an undertaking.

As always I thank you for reading and welcome comments and criticisms below.

January 03, 2018 04:53 PM

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
  • Planet Python Brasil
  • Planet Python Indonesia
  • Planet Python Poland
• Python Libraries
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Captain DeadBones'' Chronicles"
  • "Control F'd"
  • "Eric Wu's Pythonfilter"
  • "Fredrik Håård's Blaag"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "Thomas Ballinger's Blog"
  • "William's Journal"
  • 2General
  • 2degrees
  • 4teamwork
  • A Little Bit of Python
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AWeber Engineering
  • Aahz
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Adrián Deccico
  • Agendaless Consulting
  • Al-Ahmadgaid Asaad
  • Albert Hopkins
  • Alec Munro
  • Alex Clark
  • Alex Gaynor
  • Alex Grönholm
  • Alex Morozov
  • Alex Smirnov
  • Alexander Limi
  • Alexandre Bourget
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Ali Afshar
  • Allison Kaptur
  • Amir Rachum
  • Amit Saha
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Drozdyuk
  • Andriy Kornatskyy
  • Andrzej Skupień
  • Andy Dustman
  • Andy R. Terrel
  • Andy Todd
  • Ankit Chandawala
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • AppNeta Blog
  • Armin Ronacher
  • Arnav Khare
  • Ashish Dutt
  • Ashish Vidyarthi
  • Ashley Camba
  • Astro Code School
  • Atul Varma -- Toolness
  • Audrey Roy Greenfeld
  • Automating OSINT
  • Baiju Muthukadan
  • Bajusz Tamás
  • Balthazar Rouberol
  • BangPypers
  • Barbara Shaurette
  • Barry Warsaw
  • BeDjango
  • Ben Bangert
  • Ben Bass
  • Ben Rousch
  • Ben Tappin
  • Benjamin Peterson
  • Benjamin W. Smith
  • Benji York
  • Bertrand Mathieu
  • Bhishan Bhandari
  • Bill Mill
  • BioPython News
  • Bit of Cheese
  • Björn Tillenius
  • Blended Technologies
  • Blue Yonder Tech
  • BlueBream
  • BlueDynamics Alliance
  • Bojan Mihelac
  • Brad Lucas
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Curtin
  • Brian Harring
  • Brian Jones
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Bryce Verdier
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catalin George Festila
  • Catherine Devlin
  • Chad Cooper
  • Chad Whitacre
  • Checking and Sharing
  • Chris Arndt
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Mitchell
  • Chris Moffitt
  • Chris Perkins
  • Chris Rose
  • Chris Warrick
  • Christian Barra
  • Christian Heimes
  • Christian Joergensen
  • Christian Scholz
  • Christoph Zwerschke
  • Christopher Denter
  • Christopher Lenz
  • Christopher Perkins
  • Chuck Thier
  • Chui Tey
  • Claudio Sparpaglione
  • Cloudnumbers
  • ClusterHQ
  • Cobe.io
  • CodeSnipers
  • Codementor
  • Coding Diet
  • Collin Winter
  • Concrete Clouds
  • Continuum Analytics Blog
  • Continuum Analytics News
  • Continuum Blog
  • Core Software
  • Corey Goldberg
  • Corey Oordt
  • Cormoran Project
  • Craig Kerstiens
  • Creative Commons
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • Daily Tech Video (Python)
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Damián Avila
  • Damyan Bogoev
  • Dan Crosta
  • Dan Stromberg
  • Daniel Bader
  • Daniel Brown
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Dariusz Suchojad
  • Data Community DC
  • Data School
  • DataCamp
  • Dataquest
  • Dave Beazley
  • Dave Behnke
  • Dave Haynes
  • David Goodger
  • David Grant
  • David J C Beach
  • David Janes
  • David MacIver
  • David Malcolm
  • David Marte
  • David Stanek
  • David Szotten
  • David Winterbottom
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • Dazza
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diane Trout
  • Diego Garcia
  • Django Weblog
  • Django Weekly
  • Django Zone
  • Djangocon
  • Djangofeed
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • DreamHost
  • Duncan McGreggor
  • Dusty Phillips
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eko S. Wibowo
  • Eli Bendersky
  • Emmanuel Leblond
  • Eniram Ltd.
  • Enthought
  • Eray Özkural (examachine)
  • Eric Florenzano
  • Eric Holscher
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Evan Fosmark
  • Eve REST Framework
  • Even Wiik Thomassen
  • Evennia
  • Everyday Python
  • Experienced Django
  • Fabio Zadrozny
  • Fazal Majid
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • Fredrik Lundh
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Gabriel Pettier
  • Galvanize
  • Gary Bernhardt
  • Gary Robinson
  • Gary Wilson
  • Gavrie Philipson
  • Geek Scrap
  • Geert Vanderkelen
  • Gene Campbell
  • Georg Brandl
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Gilad Raphaelli
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • GoDjango
  • Gocept Weblog
  • Godson Gera
  • Gonçalo Margalho
  • Graeme Cross
  • Graham Dumpleton
  • Gramps
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Greg Turnquist
  • Greg Wilson
  • Gregory Pinero
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido Kollerie
  • Guido van Rossum
  • Guilherme Toti
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hanno Schlichting
  • Hany Fahim
  • Harrison Erd
  • Hector Garcia
  • Heikki Toivonen
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Huy Nguyen
  • Hynek Schlawack
  • IO Digital Sec
  • Ian Bicking
  • Ian Ozsvald
  • Ilian Iliev
  • Imaginary Landscape
  • Import Python
  • Intellimath blog
  • Invent with Python
  • Investing using Python
  • Ionel Cristian Maries
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • Isotoma
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Krstic
  • Jack Diederich
  • Jacob Perkins
  • Jaime Buelta
  • Jamal Moir
  • James Mills
  • James Polera
  • James Tauber
  • Jan-Carel Brand
  • Janusworx
  • Jared Forsyth
  • Jarrod Millman
  • Jason Baker
  • Jason Meyers
  • Jason Webb
  • Jaysinh Shukla
  • Jean-Louis Fuchs
  • Jean-Paul Calderone
  • Jeet Sukumaran
  • Jeethu Rao
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Knupp
  • Jeff McNeil
  • Jeff Rush
  • Jeff Shell
  • Jeff Winkler
  • Jeremy Epstein
  • Jeremy Hylton
  • Jessie Anderson
  • Jim Baker
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Gagliardi
  • John Jacobsen
  • John Paulett
  • Jon Parise
  • Jonathan Dobson
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan Street
  • Jordan Dimov
  • Jorge Niedbalski
  • Jorge Puente Sarrín
  • Jorgen Schäfer
  • João Laia
  • Juan Manuel Contreras
  • Juan Rivas
  • Juan Rodríguez Monti
  • Juho Vepsäläinen
  • Julien Danjou
  • Julien Palard
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Diefenbach
  • Kai Lautaportti
  • Kamon Ayeva
  • Karen Rustad
  • Kate Editor
  • Katie Cunningham
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Konrad Delong
  • Koodaamo
  • Kracekumar Ramaraju
  • Kristján Valur Jónsson
  • Kriti Godey
  • Krys Wilken
  • Krzysztof Żuraw
  • Kulbir Saini
  • Kumar McMillan
  • Kun Xi
  • Kurt Grandis
  • Kushal Das
  • Károly Nagy
  • Laurence Rowe
  • Laurent Luce
  • Laurent Szyster
  • Lawrence Oluyede
  • Lee Braiden
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lesscode.org
  • Levi Velázquez
  • Lightning Fast CMS
  • Lightning Fast Shop
  • Lintel Technologies
  • Lion Kimbro
  • Lionel Tan
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Ludovic Gasc
  • Ludovico Fischer
  • Ludvig Ericson
  • Luke Macken
  • Luke Plant
  • Machinalis
  • Maciej Fijalkowsk
  • Mahdi Yusuf
  • Mahmoud Hashemi
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marc Kerins
  • Marc-André Lemburg
  • Marcin Kuźmiński
  • Marcos Dione
  • Marcus Whybrow
  • Mario Boikov
  • Marius Gedminas
  • Mark Dennehy
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Mark Mruss
  • Mark Paschal
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martijn Pieters
  • Martin Blais
  • Martin Fitzpatrick
  • Mathieu Fenniak
  • Mathieu Virbel
  • Mats Kindahl
  • Matt Goodall
  • Matt Harrison
  • Matt Reid
  • Matt Wilkes
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Mattias Brändström
  • Mauveweb
  • Max Ischenko
  • Max Klymyshyn
  • Michael Bayer
  • Michael Becker
  • Michael Crute
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael Martinez
  • Michael Nelson
  • Michael Schurter
  • Michael Sparks
  • Michael Watkins
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michał Domański
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mike Naberezny
  • Mike Pirnat
  • Mikeal Rogers
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Morten W Petersen
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Nadia Alramli
  • Naomi Ceder
  • Natan Zabkar
  • Nathan Lemoine
  • Neckbeard Republic
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Amorim
  • Nicholas Piël
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikhil Gopal
  • Nikola
  • Noah Gift
  • NumFOCUS
  • Obey the Testing Goat
  • Oliver Andrich
  • Omaha Python Users Group
  • Omar Abo-Namous
  • Ondřej Čertík
  • OpenHatch Python posts
  • Orestis Markou
  • Partecs
  • Pathwright
  • Patrice Neff
  • Patricio Paez
  • Patrick Kennedy
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paulo Nuin
  • Paweł Fertyk
  • PayPal Engineering Blog
  • Pedro Kroger
  • Pedro Lima
  • Pete Hunt
  • Petecode
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Halliday
  • Peter Harkins
  • Peter Hoffmann
  • Peter Parente
  • Petro Verkhogliad
  • Phil Hassey
  • Philip Jenvey
  • Philip Semanchuk
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Pierre Carbonnelle
  • Pietro Abate
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programando Ciência
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyAMF Blog
  • PyBites
  • PyCarolinas
  • PyCharm
  • PyCon
  • PyCon 2008 on YouTube
  • PyCon Australia
  • PyCon PL Conference
  • PyCon Podcast
  • PyCon Pune
  • PyLadies
  • PyPy Development
  • PyTennessee
  • PyTexas
  • Pylons News Feed
  • Pypix
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python News
  • Python Open Mike
  • Python Piedmont Triad User Group
  • Python Software Foundation
  • Python Sprints
  • Python Sweetness
  • Python Testing Cookbook
  • Python User Groups
  • Python for biologists
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • PythonThreads
  • Pythonology
  • Péter Szabó
  • Péter Zsoldos
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Ramana
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Raymond Li
  • Reach Tim
  • Real Python
  • Red Hat Developers
  • Reinout van Rees
  • Rene Dudfield
  • Reuven Lerner
  • Richard Gomes
  • Richard Jones
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Cakebread
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Lujo
  • Robert Picard
  • Robert Zaremba
  • Roberto Alsina
  • Robin Dunn
  • Robin Parmar
  • Robin Wilson
  • Roche Compaan
  • Rocky Burt
  • Rodrigo Araúj
  • Rok Garbas
  • Roman Imankulov
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • Ryan Tomayko
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Salman Haq
  • Samuel Sutch
  • Sandipan Dey
  • Sandro Tosi
  • Sayan Chowdhury
  • Scripting the web with Python
  • Sean McGrath
  • Sean Reifschneider
  • Selena Deckelmann
  • Semaphore Community
  • Senthil Kumaran
  • Server Density
  • Shannon -jj Behrens
  • ShiningPanda
  • Shopkick Tech Blog
  • Shriphani Palakodety
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Willison
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Spike ekipS
  • Spyder IDE
  • St James Software Development
  • Stack Abuse
  • Stacks
  • Starzel.de
  • Stefan Behnel
  • Stefan Petrea
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Dower
  • Steve Holden
  • Steven Klass
  • Steven Kryskalla
  • Steven Wilcox
  • Stijn Debrouwere
  • Stories in My Pocket
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumith - Blog about SymPy/Python
  • Swaroop C H
  • SwiftStack
  • Swisscom ICT
  • Sylvain Hellegouarch
  • Talk Python to Me
  • Tarek Ziade
  • Taylor Edmiston
  • Teach Me Python
  • Techiediaries - Django
  • Ted Leung
  • Ted Nyman
  • Teemu Harju
  • Tennessee Leeuwenburg
  • Terri Oda
  • Terry Jones
  • Terry Peppers
  • Test and Code
  • The Artificial Intelligence Cookbook
  • The Changelog
  • The Data Scientist
  • The Digital Cat
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • The Three of Wands
  • Thiago Avelino
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Thomi Richards
  • Tibo Beijen
  • Tim Gilbert
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim Parkin
  • Tim van der Linden
  • Titus Brown
  • Tobias Ivarsson
  • Tom Christie
  • Tomasz Ducin
  • Tomasz Früboes
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Virgil Dupras
  • Viva La Chipperfish
  • Vladimir Iakolev
  • Vladimir Perić
  • Wallaroo Labs
  • Wallix
  • Washington Times OpenSource
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Wichert Akkerman
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • William Thompson
  • Wingware Blog
  • Wingware News
  • Wolfram Kriesing
  • Wraithan
  • Wyatt Baldwin
  • Xavier Spriet
  • Yaco
  • Yaniv Aknin
  • Yann Larrivée
  • Yannick Gingras
  • Yasoob Khalid
  • Yoong Kang Lim
  • Yuval Greenfield
  • Zachary Voase
  • Zaki Akhmad
  • Zato Blog
  • ZeroDB
  • Zeth
  • bottlepy-dev
  • codeboje
  • eGenix.com
  • eRambler
  • hypothesis.works articles
  • kdev-python
  • nl-project
  • pgcli
  • pythonwise
  • qutebrowser development blog
  • tryexceptpass
  • wiredobjects
  • zombofant.net
  • Éric Araujo
  • Łukasz Langa
  • بايثون العربي
  • To request addition or removal:
    Open an issue on github
    or e-mail planet at python.org
    (note, responses can take up to a few days)