Planet Python
Last update: April 14, 2018 01:46 AM
April 14, 2018
Doug Hellmann
daily-tweeter 0.1.0
daily-tweeter is a command line tool for posting scheduled messages to Twitter This is the first public release.
April 13, 2018
Roberto Alsina
I have written half a book
LIke mentioned before I am trying to write a book and ... well, I may be actually making progress? At least the generated PDF is about 170 pages long, which means I have written a bunch in this past month.
I have finished the second of four planned parts, which means I have done about half of it. Since I expect the next two parts to be shorter, it's actually more than that.
The target audience are people who have finished the python tutorial but are not exactly programmers yet. They have the syntax more or less in their heads, but how do you turn that into an actual piece of code?
- Part 1 is about "prototyping", the process of dumping an idea into rough code.
- Part 2 is about polishing that rough code into ... not so rough code. Includes a gentle introduction to testing, for example.
- Part 3 (to be written) is about things that are not code:
- Git / Gitlab
- Issues
- Packaging
- Setting up a website
- CI
- Lots more
- Part 4 is still to be thought but basically it will cover implementing a large feature from the ground up.
I much appreciate comments about it.
PD: Si, va a haber una traducciń al castellano. O mas bien al argentino. Una vez que lo termine.
Stack Abuse
Basic Socket Programming in Python
In general, network services follow the traditional client/server model. One computer acts as a server to provide a certain service and another computer represents the client side which makes use of this service. In order to communicate over the network a network socket comes into play, mostly only referred to as a socket. This kind of socket communication can even be used internally in a computer for inter-process communication (IPC).
This article explains how to write a simple client/server application that communicates via network socket using the Python programming language. For simplicity, our example server only outputs the received data to stdout. The idea behind the client/server application is a sensor in a weather station, which collects temperature data over time and sends the collected data to a server application, where the data gets processed further.
What is a Socket?
A network socket is an endpoint of a two-way communication link between two programs or processes - client and server in our case - which are running on the network. This can be on the same computer as well as on different systems which are connected via the network.
Both parties communicate with each other by writing to or reading from the network socket. The technical equivalent in reality is a telephone communication between two participants. The network socket represents the corresponding number of the telephone line, or a contract in case of cell phones.
Example
In order to make use of the socket functionality, only the Python socket module is necessary. In the example code shown below the Python time module is imported as well in order to simulate the weather station and to simplify time calculations.
In this case both the client and the server run on the same computer. A socket has a corresponding port number, which is 23456 in our case. If desired, you may choose a different port number from the unrestricted number range between 1024 and 65535.
The Server
Having loaded the additional Python socket module an Internet streaming socket is created using the socket.socket class with the two parameters socket.AF_INET and socket.SOCK_STREAM. The retrieval of the hostname, the fully qualified domain name, and the IP address is done by the methods gethostname(), getfqdn(), and gethostbyname(), respectively. Next, the socket is bound to the IP address and the port number 23456 with the help of the bind() method.
With the help of the listen() method the server listens for incoming connections on the specified port. In the while loop the server waits for incoming requests and accepts them using the accept() method. The data submitted by the client is read via recv() method as chunks of 64 bytes, and simply output to stdout. Finally, the current connection is closed if no further data is sent from the client.
# load additional Python module
import socket
# create TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# retrieve local hostname
local_hostname = socket.gethostname()
# get fully qualified hostname
local_fqdn = socket.getfqdn()
# get the according IP address
ip_address = socket.gethostbyname(local_hostname)
# output hostname, domain name and IP address
print ("working on %s (%s) with %s" % (local_hostname, local_fqdn, ip_address))
# bind the socket to the port 23456
server_address = (ip_address, 23456)
print ('starting up on %s port %s' % server_address)
sock.bind(server_address)
# listen for incoming connections (server mode) with one connection at a time
sock.listen(1)
while True:
# wait for a connection
print ('waiting for a connection')
connection, client_address = sock.accept()
try:
# show who connected to us
print ('connection from', client_address)
# receive the data in small chunks and print it
while True:
data = connection.recv(64)
if data:
# output received data
print ("Data: %s" % data)
else:
# no more data -- quit the loop
print ("no more data.")
break
finally:
# Clean up the connection
connection.close()
The Client
Now we will have a look at the client side. The Python code is mostly similar to the server side, except for the usage of the socket - the client uses the connect() method, instead. In a for loop the temperature data is sent to the server using the sendall() method. The call of the time.sleep(2) method pauses the client for two seconds before it sends another temperature reading. After all the temperature data is sent from the list the connection is finally closed using the close() method.
# load additional Python modules
import socket
import time
# create TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# retrieve local hostname
local_hostname = socket.gethostname()
# get fully qualified hostname
local_fqdn = socket.getfqdn()
# get the according IP address
ip_address = socket.gethostbyname(local_hostname)
# bind the socket to the port 23456, and connect
server_address = (ip_address, 23456)
sock.connect(server_address)
print ("connecting to %s (%s) with %s" % (local_hostname, local_fqdn, ip_address))
# define example data to be sent to the server
temperature_data = ["15", "22", "21", "26", "25", "19"]
for entry in temperature_data:
print ("data: %s" % entry)
new_data = str("temperature: %s\n" % entry).encode("utf-8")
sock.sendall(new_data)
# wait for two seconds
time.sleep(2)
# close connection
sock.close()
Running the Server and Client
To run both the server and the client program, open two terminal windows and issue the following commands - one per terminal window and in the following order:
$ python3 echo-server.py
and
$ python3 echo-client.py
The two figures below show the corresponding output of the example program:
Figure 1
Figure 2
Conclusion
Writing Python programs that use IPC with sockets is rather simple. The example given above can certainly be extended to handle soemthing more complex. For further information and additional methods you may have a look at some great Python socket programming resources available.
Weekly Python Chat
Tuple Unpacking
Tuple unpacking is one of those Python features that is under-used frequently. Many uses for tuple unpacking are obvious and well-encouraged but some uses are frequently overlooked.
In this chat we'll talk about both tuples and tuple unpacking. We'll discuss the difference between the two and then we'll discuss how and where to use tuple unpacking.
This will be a Q&A-heavy chat so come prepared to ask questions!
All experience levels are welcome in this chat. Don't be afraid of asking bad/silly/easy/hard questions. This chat is for you and your questions belong here.
Graham Dumpleton
Book #2: Deploying to OpenShift
I have been more than a bit busy over the past year and this blog has become somewhat neglected. One of the reasons for being so busy was that I was working on a second book. As with the first book, which I co-authored, this book is on OpenShift. This time I am the sole author, and the book somewhat thicker, so you can imagine it has taken a fair bit of time and effort.For those who may not know
Kushal Das
Latest attempt to censor Internet and curb press freedom in India
A branch of the Indian government, the Ministry of Information and Broadcasting, is trying once again to censor Internet and Freedom of Speech. This time, it ordered to form a committee of 10 members who will frame regulations for online media/ news portals and online content.
This order includes these following Terms of Reference for the committee.
- To delineate the sphere of online information dissemination which needs to be brought under regulation, on the lines applicable to print and electronic media.
- To recommend appropriate policy formulation for online media / news portals and online content platforms including digital broadcasting which encompasses entertainment / infotainment and news/media aggregators keeping in mind the extant FDI norms, Programme & Advertising Code for TV Channels, norms circulated by PCI, code of ethics framed by NBA and norms prescribed by IBF; and
- To analyze the international scenario on such existing regulatory mechanisms with a view to incorporate the best practices.
What are the immediate problems posed by this order?
If one reads carefully, one can see how vague are the terms, and specifically how they added the term online content into it.
online content means everything we can see/read/listen do over cyberspace. In the last few years, a number of new news organizations came up in India, whose fearless reporting have caused a lot of problems for the government and their friends. Even though they managed to censor publishing (sometimes self censored) news in the mainstream Indian media, but all of these new online media houses and individual bloggers and security researchers and activists kept informing the mass about the wrongdoings of the people in power.
With this latest attempt to restrict free speech over the internet, the government is trying to increase its reach even more. Broad terms like online content platforms or online media or news/media aggregators will include every person and websites under its watch. One of the impacts of mass indiscriminate surveillance like this is that people are shamed into reading and thinking only what is in line with the government, or popular thought .
How do you determine if some blog post or update in a social media platform is news or not? For me, most of things I read on the internet are news to me. I learn, I communicate my thoughts over these various platforms on cyberspace. To all those computer people reading this blog post, think about the moment when you will try to search about “how to do X in Y programming language?” on Internet, but, you can not see the result because that is blocked by this censorship.
India is also known for random blockades of different sites over the years. The Government also ordered to kill Internet for entire states for many days. For the majority of internet blockages, we, the citizens of India were neither informed the reasons nor given a chance to question the legality of those bans. India has been marked as acountry under surveillance by Reporters Without Borders back in 2012.
Also remember that this is the same Government, which was trying to fight at its best in the Supreme Court of India last year, to curb the privacy of every Indian citizen. They said that Indian citizens do not have any right to privacy. Thankfully the bench declared the following:
The right to privacy is protected as an intrinsic part of the right to life and personal liberty under Article 21 and as a part of the freedoms guaranteed by Part III of the Constitution.
Privacy is a fundamental right of every Indian citizen.
However, that fundamental right is still under attack in the name of another draconian law The Aadhaar act. A case is currently going on in the Supreme Court of India to determine the constitutional validity of Aadhaar. In the recent past, when journalists reported how the Aadhaar data can be breached, instead of fixing the problems, the government is criminally investigating the journalists.
A Declaration of the Independence of Cyberspace
Different governments across the world kept trying (and they will keep trying again and again) to curb free speech and press freedom. They are trying to draw borders and boundaries inside of cyberspace, and restrict the true nature of what is it referring to here?.
In 1996, late John Perry Barlow wrote A Declaration of the Independence of Cyberspace, and I think that fits in naturally in the current discussion.
Governments of the Industrial World, you weary giants of flesh and steel, I come from Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us alone. You are not welcome among us. You have no sovereignty where we gather. -- John Perry Barlow
How can you help to fight back censorship?
Each and every one of us are affected by this, and we all can help to fight back and resist censorship. The simplest thing you can do is start talking about the problems. Discuss them with your neighbor, talk about it while commuting to the office. Explain the problem to your children or to your parents. Write about it, write blog posts, share across all the different social media platforms. Many of your friends (from other fields than computer technology) may be using Internet daily, but might not know about the destruction these laws can cause and the censorship imposed on the citizens of India.
Educate people, learn from others about the problems arising. If you are giving a talk about a FOSS technology, also talk about how a free and open Internet is helping all of us to stay connected. If that freedom goes away, we will lose everything. At any programming workshop you attend, share these knowledge with other participants.
In many cases, using tools to bypass censorship altogether is also very helpful (avoiding any direct confrontation). The Tor Project is a free software and open network which helps to keep freedom and privacy of the users. By circumventing surveillance and censorship, one can use it more for daily Internet browsing. The increase in Tor traffic will help all of the Tor network users together. This makes any attempt of tracking individuals even more expensive for any nation state actors. So, download the Tor Browser today and start using it for everything.
In this era of Public private partnership from hell, Cory Doctorow beautifully explained how internet is the nervous system of 21st century, and how we all can join together to save the freedom of internet. Listen to him, do your part.
Header image copyright: Peter Massas (CC-BY-SA)
April 12, 2018
Continuum Analytics Blog
What You Missed on Day Three of AnacondaCON 2018
And that’s a wrap! Yesterday was the third and final day of AnacondaCON 2018, and what a ride it’s been. Read some highlights from what you missed, and stay tuned for our comprehensive AnacondaCON 2018 recap, coming soon! Improving Your Anaconda Distribution User Experience Anaconda Product Manager Crystal Soja presented a roadmap of upcoming plans …
Read more →
Will Kahn-Greene
AWS Lambda dev with Python
A story of a pigeon
I work on Socorro which is the crash ingestion pipeline for Mozilla's products.
The pipeline starts at the collector which handles incoming HTTP POST requests, pulls out the payload, futzes with it a little, and then saves it to AWS S3. Socorro then processes some of those crashes in the processor. The part that connects the two is called Pigeon. It was intended as a short-term solution to bridge the collector and the processor, but it's still around a year later and the green grass grows all around all around and the green grass grows all around.
Pigeon is an AWS Lambda function that triggers on S3 ObjectCreated:Put events, looks at the filename, and then adds things to the processing queue depending on the filename structure. We called it Pigeon for various hilarious reasons that are too mundane to go into in this blog post.
It's pretty basic. It doesn't do much. It was a short term solution we thought we'd throw away pretty quickly. I wrote some unit tests for the individual parts of it and a "client" that invoked the function in a faux AWS Lambda like way. That was good enough.
But then some problems
Pigeon was written with Python 2 because at the time AWS Lambda didn't have a Python 3 runtime. That changed--now there's one with Python 3.6.
In January, I decided to update Pigeon to work with Python 3.6. I tweaked the code, tweaked the unit tests, and voila--it was done! Then we deployed it to our -stage environment where it failed epically in technicolor glory (but no sound!) and we had to back it out and return to the Python 2 version.
What happened? I'll tell you what happened--we had a shit testing environment. Sure, we had tests, but they lacked several things:
- At no point do we test against the build artifact for Pigeon. The build artifact for AWS Lambda jobs in Python is a .zip file that includes the code and all the libraries that it uses.
- The tests "invoke" Pigeon with a "client", but it was pretty unlike the AWS Lambda Python 3.6 runtime.
- Turns out I had completely misunderstood how I should be doing exception handling in AWS Lambda.
So our tests tested some things, but missed some important things and a big bug didn't get caught before going to -stage.
It sucked. I felt chagrinned. I like to think I have a tolerance for failure since I do it a lot, but this felt particularly faily and some basic safeguards would have prevented it from happening.
Fleshing out AWS Lambda in Python project
We were thinking of converting another part of the Socorro pipeline to AWS Lambda, but I put that on hold until I had wrapped my head around how to build a development environment that included scaffolding for testing AWS Lambda functions in a real runtime.
Miles or Brian mentioned aws-sam-local. I looked into that. It's written in Go, they suggest installing it with npm, it does a bunch of things, and it has some event generation code. But for the things I needed, it seemed like it would just be a convenience cli for docker-lambda.
I had been aware of docker-lambda for a while, but hadn't looked at the project recently. They added support for passing events via stdin. Their docs have examples of invoking Lambda functions. That seemed like what I needed.
I took that and built the developer environment scaffolding that we've got in Pigeon now. Further, I decided to use this same model for future AWS Lambda function development.
How does it work?
Pigeon is a Python project, so it uses Python libraries. I maintain those requirements in a requirements.txt file.
I install the requirements into a ./build directory:
$ pip install --ignore-installed --no-cache-dir -r requirements.txt -t build/
I copy the Pigeon source into that directory, too:
$ cp pigeon.py build/
That's all I need for the runtime to use.
The tests are in the tests/ directory. I'm using pytest and in the conftest.py file have this at the top:
import os import sys # Insert build/ directory in sys.path so we can import pigeon sys.path.insert( 0, os.path.join( os.path.dirname(os.path.dirname(__file__)), 'build' ) )
I'm using Docker and docker-compose to aid development. I use a test container which is a python:3.6 image with the test requirements installed in it.
In this way, tests run against the ./build directory.
Now I want to be able to invoke Pigeon in an AWS Lambda runtime so I can debug issues and also write an integration test.
I set up a lambda-run container that uses the lambci/lambda:python3.6 image. I mount ./build as /var/task since that's where the AWS Lambda runtime expects things to be.
I created a shell script for invoking Pigeon:
#!/bin/bash docker-compose run \ --rm \ -v "$PWD/build":/var/task \ --service-ports \ -e DOCKER_LAMBDA_USE_STDIN=1 \ lambda-run pigeon.handler $@
That's based on the docker-lambda invoke examples.
Let's walk through that:
- It runs the lambda-run container with the services it depends on as defined in my docker-compose.yml file.
- It mounts the ./build directory as /var/task because that's where the runtime expectes the code it's running to be.
- The DOCKER_LAMBDA_USE_STDIN=1 environment variable causes it to look at stdin for the event. That's pretty convenient.
- It runs invokes pigeon.handler which is the handler function in the pigeon Python module.
I have another script that generates fake AWS S3 ObjectCreated:Put events. I cat the result of that into the invoke shell script. That runs everything nicely:
$ ./bin/generate_event.py --key v2/raw_crash/000/20180313/00007bd0-2d1c-4865-af09-80bc00180313 > event.json $ cat event.json | ./bin/run_invoke.sh Starting socorropigeon_rabbitmq_1 ... done START RequestId: 921b4ecf-6e3f-4bc1-adf6-7d58e4d41f47 Version: $LATEST {"Timestamp": 1523588759480920064, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"msg": "Please set PIGEON_AWS_REGION. Returning original unencrypted data."}} {"Timestamp": 1523588759481024512, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"msg": "Please set PIGEON_AWS_REGION. Returning original unencrypted data."}} {"Timestamp": 1523588759481599232, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "number of records: 1"}} {"Timestamp": 1523588759481796864, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "looking at key: v2/raw_crash/000/20180313/00007bd0-2d1c-4865-af09-80bc00180313"}} {"Timestamp": 1523588759481933056, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "crash id: 00007bd0-2d1c-4865-af09-80bc00180313 in dev_bucket"}} MONITORING|1523588759|1|count|socorro.pigeon.accept|#env:test {"Timestamp": 1523588759497482240, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "00007bd0-2d1c-4865-af09-80bc00180313: publishing to socorrodev.normal"}} END RequestId: 921b4ecf-6e3f-4bc1-adf6-7d58e4d41f47 REPORT RequestId: 921b4ecf-6e3f-4bc1-adf6-7d58e4d41f47 Duration: 101 ms Billed Duration: 200 ms Memory Size: 1536 MB Max Memory Used: 28 MB null
Then I wrote an integration test that cleared RabbitMQ queue, ran the invoke script with a bunch of different keys, and then checked what was in the processor queue.
Now I've got:
- tests that test the individual bits of Pigeon
- a way to run Pigeon in the same environment as -stage and -prod
- an integration test that runs the whole setup
A thing I hadn't mentioned was that Pigeon's documentation is entirely in the README. The docs cover setup and development well enough that I can hand this off to normal people and future me. I like simple docs. Building scaffolding such that docs are simple makes me happy.
Summary
You can see the project at https://github.com/mozilla-services/socorro-pigeon.
py.CheckIO
What is Python capable of?
This article shows how Python programming language is being used in the the different spheres, such as web-development, desktop and mobile applications, artificial intelligence and data analysis, testing, robotics and so on, for entertainment purposes as well as for serious and important objectives.
Kushal Das
dgplug summer training 2018
dgplug summer training 2018 will start at 13:30 UTC, 17th June. This will be the 11th edition. Like every year, we have modified the training based on the feedback and, of course, there will be more experiments to try and make it better.
What happened differently in 2017?
We did not manage to get all the guest sessions mentioned, but, we moved the guest sessions at the later stage of the training. This ensured that only the really interested people were attending, so there was a better chance of having an actual conversation during the sessions. As we received mostly positive feedback on that, we are going to do the same this year.
We had much more discussions among the participants in general than in previous years. Anwesha and I wrote an article about the history of the Free Software and we had a lot of discussion about the political motivation and freedom in general during the training.
We also had an amazing detailed session on Aadhaar and how it is affecting (read destroying) India, by Kiran Jonnalagadda.
Beside, we started writing a new book to introduce the participants to Linux command line. We tried to cover the basics of Linux command line and the tools we use on a day to day basis.
Shakthi Kannan started Operation Blue Moon where he is helping individuals to get things done by managing their own sprints. All information on this project can be found in the aforementioned Github link.
What are the new plans in 2018?
We are living in an era of surveillance and the people in power are trying to hide facts from the people who are being governed. There are a number of Free Software projects which are helping the citizens of cyberspace to resist and bypass the blockades. This year we will focus on these applications and how one can start contributing to the same projects in upstream. A special focus will be given to The Tor project, both from users’ and developers’ point of views.
In 2017, a lot of people asked help to start learning Go. So, this year we will do a basic introduction to Go in the training. Though, Python will remain the primary choice for teaching.
How to join the training?
First, join our mailing list, and then join the IRC channel #dgplug on Freenode.
Python Software Foundation
Python Software Foundation Fellow Members for Q1 2018
Congratulations! Thank you for your continued contributions. We have added you to our Fellow roster online.
The above members have contributed to the Python ecosystem by maintaining popular frameworks, maintaining critical Python infrastructure, organizing Python events, hosting Python podcasts, teaching classes, contributing to CPython, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.
If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. Here is the nomination review schedule for 2018:
- Q2: April to the end of June (01/04 - 30/06) Cut-off for quarter two will be May 20. New fellows will be announced before June 30.
- Q3: July to the end of September (01/07 - 30/09) Cut-off for quarter three will be August 20. New fellows will be announced before end of September.
- Q4: October to the end of December (01/10 - 31/12) Cut-off for quarter four will be November 20. New fellows will be announced before December 31.
We are still looking for a few more voting members to join the Work Group to help review nominations. If you are a PSF Fellow and would like to join, please write to psf-fellow at python.org.
Python Bytes
#73 This podcast comes in any color you want, as long as it's black
Mike Driscoll
Adding SVGs to PDFs with Python and ReportLab
ReportLab has native support for generating SVGs, but not for embedding SVGs in their PDFs. Fortunately, Dinu Gherman created the svglib package, a pure-Python package that can read SVG files and convert them to other formats that ReportLab can use. The official website for svglib is on Github.
The svglib package will work on Linux, Mac OS and Windows. The website states that it works with Python 2.7 – 3.5, but it should work in newer versions of Python as well.
You can use svglib to read your existing SVG giles and convert them into ReportLab Drawing objects. The svglib package also has a command-line tool, svg2pdf, that can convert SVG files to PDFs.
Dependencies
The svglib package depends on ReportLab and lxml. You can install both of these packages using pip:
pip install reportlab lxml
Installation
The svglib package can be installed using one of three methods.
Install the latest release
If you’d like to install the latest release from the Python Packaging Index, then you can just use pip the normal way:
pip install svglib
Install from latest version from source control
On the off chance that you want to use the latest version of the code (i.e. the bleeding edge / alpha builds), then you can install directly from Github using pip like this:
pip install git+https://github.com/deeplook/svglib
Manual installation
Most of the time, using pip is the way to go. But you can also download the tarball from the Python Packaging Index and do all the steps that pip does for you automatically if you want to. Just run the following three commands in your terminal in order:
tar xfz svglib-0.8.1.tar.gz cd svglib-0.8.1 python setup.py install
Now that we have svglib installed, let’s learn how to use it!
Usage
Using svglib with ReportLab is actually quite easy. All you need to do is import svg2rlg from svglib.svglib and give it the path to your SVG file. Let’s take a look:
# svg_demo.py from reportlab.graphics import renderPDF, renderPM from svglib.svglib import svg2rlg def svg_demo(image_path, output_path): drawing = svg2rlg(image_path) renderPDF.drawToFile(drawing, output_path) renderPM.drawToFile(drawing, 'svg_demo.png', 'PNG') if __name__ == '__main__': svg_demo('snakehead.svg', 'svg_demo.pdf')
After giving svg2rlg your path to the SVG file, it will return a drawing object. Then you can use this object to write it out as a PDF or a PNG. You could go on to use this script to create your own personal SVG to PNG converting utility!
Drawing on the Canvas
Personally, I don’t like to create one-off PDFs with just an image in them like in the previous example. Instead, I want to be able to insert the image and write out text and other things. Fortunately, you can do this very easily by painting your canvas with the drawing object. Here’s an example:
# svg_on_canvas.py from reportlab.graphics import renderPDF from reportlab.pdfgen import canvas from svglib.svglib import svg2rlg def add_image(image_path): my_canvas = canvas.Canvas('svg_on_canvas.pdf') drawing = svg2rlg(image_path) renderPDF.draw(drawing, my_canvas, 0, 40) my_canvas.drawString(50, 30, 'My SVG Image') my_canvas.save() if __name__ == '__main__': image_path = 'snakehead.svg' add_image(image_path)
Here we create a canvas.Canvas object and then create our SVG drawing object. Now you can use renderPDF.draw to draw your drawing on your canvas at a specific x/y coordinate. We go ahead and draw out some small text underneath our image and then save it off. The result should look something like this:
Adding an SVG to a Flowable
Drawings in ReportLab can usually be added as a list of Flowables and built with a document template. The svglib’s website says that its drawing objects are compatible with ReportLab’s Flowable system. Let’s use a different SVG for this example. We will be using the Flag of Cuba from Wikipedia. The svglib tests download a bunch of flag SVGs in their tests, so we will try one of the images that they use. You can get it here:
https://upload.wikimedia.org/wikipedia/commons/b/bd/Flag_of_Cuba.svg
Once you have the image saved off, we can take a look at the code:
# svg_demo2.py import os from reportlab.graphics import renderPDF, renderPM from reportlab.platypus import SimpleDocTemplate from svglib.svglib import svg2rlg def svg_demo(image_path, output_path): drawing = svg2rlg(image_path) doc = SimpleDocTemplate(output_path) story = [] story.append(drawing) doc.build(story) if __name__ == '__main__': svg_demo('Flag_of_Cuba.svg', 'svg_demo2.pdf')
This worked pretty well, although the flag is cut off on the right side. Here’s the output:
I actually had some trouble with this example. ReportLab or svglib seems to be really picky about the way the SVG is formatted or its size. Depending on the SVG I used, I would end up with an AttributeError or a blank document or I would be successful. So your mileage will probably vary. I will say that I spoke with some of the core developers and they mentioned that **SimpleDocTemplate** doesn’t give you enough control over the frame that the drawing goes into, so you may need to create your own Frame or PageTemplate to make the SVG show up correctly. A workaround to get the snakehead.svg to work was to set the left and right margins to zero:
# svg_demo3.py from reportlab.platypus import SimpleDocTemplate from svglib.svglib import svg2rlg def svg_demo(image_path, output_path): drawing = svg2rlg(image_path) doc = SimpleDocTemplate(output_path, rightMargin=0, leftMargin=0) story = [] story.append(drawing) doc.build(story) if __name__ == '__main__': svg_demo('snakehead.svg', 'svg_demo3.pdf')
Scaling SVGs in ReportLab
The SVG drawings you create with svglib are not scaled by default. So you will need to write a function to do that for you. Let’s take a look:
# svg_scaled_on_canvas.py from reportlab.graphics import renderPDF from reportlab.pdfgen import canvas from svglib.svglib import svg2rlg def scale(drawing, scaling_factor): """ Scale a reportlab.graphics.shapes.Drawing() object while maintaining the aspect ratio """ scaling_x = scaling_factor scaling_y = scaling_factor drawing.width = drawing.minWidth() * scaling_x drawing.height = drawing.height * scaling_y drawing.scale(scaling_x, scaling_y) return drawing def add_image(image_path, scaling_factor): my_canvas = canvas.Canvas('svg_scaled_on_canvas.pdf') drawing = svg2rlg(image_path) scaled_drawing = scale(drawing, scaling_factor=scaling_factor) renderPDF.draw(scaled_drawing, my_canvas, 0, 40) my_canvas.drawString(50, 30, 'My SVG Image') my_canvas.save() if __name__ == '__main__': image_path = 'snakehead.svg' add_image(image_path, scaling_factor=0.5)
Here we have two functions. The first function will scale our image using a scaling factor. In this case, we use 0.5 as our scaling factor. Then we do some math against our drawing object and tell it to scale itself. Finally we draw it back out in much the same way as we did in the previous example.
Here is the result:
Using SVG Plots from matplotlib in ReportLab
In a previous article, we learned how to create graphs using just the ReportLab toolkit. One of the most popular 2D graphing packages for Python is matplotlib though. You can read all about matplotlib here: https://matplotlib.org/. The reason I am mentioning matplotlib in this article is that it supports SVG as one of its output formats. So we will look at how to take a plot created with matplotlib and insert it into ReportLab.
To install matplotlib, the most popular method is to use pip:
pip install matplotlib
Now that we have matplotlib installed, we can create a simple plot and export it as SVG. Let’s see how this works:
import matplotlib.pyplot as pyplot def create_matplotlib_svg(plot_path): pyplot.plot(list(range(5))) pyplot.title = 'matplotlib SVG + ReportLab' pyplot.ylabel = 'Increasing numbers' pyplot.savefig(plot_path, format='svg') if __name__ == '__main__': from svg_demo import svg_demo svg_path = 'matplot.svg' create_matplotlib_svg(svg_path) svg_demo(svg_path, 'matplot.pdf')
In this code, we import the pyplot sub-library from matplotlib. Next we create a simple function that takes the path to where we want to save our plot. For this simple plot, we create a simple range of five numbers for one of the axes. hen we add a title and a y-label. Finally we save the plot to disk as an SVG.
The last step is in the if statement at the bottom of the code. Here we import our svg_demo code from earlier in this article. We create oru SVG image and then we run it through our demo code to turn it into a PDF.
The result looks like this:
Using svg2pdf
When you install svglib, you also get a command-line tool called svg2pdf. As the name implies, you can use this tool to convert SVG files to PDF files. Let’s look at a couple of examples:
svg2pdf /path/to/plot.svg
This command just takes the path to the SVG file that you want to turn into a PDF. It will automatically rename the output to the same name as the input file, but with the PDF extension. You can specify the output name though:
svg2pdf -o /path/to/output.pdf /path/to/plot.svg
The -o flag tells svg2pdf requires that you pass in the output PDF path followed by the input SVG path.
The documentation also mentions that you can convert all the SVG files to PDFs using a command like the following:
svg2pdf -o "%(base)s.pdf" path/to/file*.svg
This will rename the output PDF to the same name as the input SVG file for each SVG file in the specified folder.
Wrapping Up
The svglib is the primary method to add SVGs to ReportLab at the time of writing this book. While it isn’t full featured, it works pretty well and the API is quite nice. We also learned how to insert a plot SVG created via the popular matplotlib package. Finally we looked at how to turn SVGs to PDFs using the svg2pdf command line tool.
Related Reading
- A Simple Step-by-Step Reportlab Tutorial
- ReportLab 101: The textobject
- ReportLab – How to add Charts and Graphs
- Extracting PDF Metadata and Text with Python
Sumana Harihareswara - Cogito, Ergo Sumana
My LWN Story Summarizing PyPI's Overhaul
This coming Monday, April 16th, we plan to flip the switch on the new PyPI and redirect https://pypi.python.org web browser requests and pip install requests so the codebase serving them is Warehouse (which is in beta right now at https://pypi.org). I'm proud of our team's work and hope you find it useful.
I haven't blogged here in a while, but I've been writing a lot, mostly announcements and explanations listed on, or a few hyperlinks away from, the onwiki index to my PyPI work. When I can't give people choices (and, unless your organization sets up a private package index/repository, PyPI can feel like the only game in town), I want to give them a lot of lead time to test, file bug reports, and migrate, and I want to provide backstory.
So: today LWN publishes a new article by me, "A new package index for Python". In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
This summary should help occasional Python programmers understand why a new PyPI codebase is necessary, what's new, what features are going away, and what to expect in the near future.
If you aren't already a subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. Thanks to LWN for the venue and the subscriber links, and thanks to Jake Edge in particular for thorough editing. Thanks to my Warehouse team for fact-checking me.
April 11, 2018
PyCharm
PyCharm Edu 2018.1: Going Beyond Python
Back in 2014, we launched PyCharm Educational Edition with the vision to provide a free, open-source tool that would familiarize Python learners with real developer experience from the very start, and would offer teachers an easy way to share code practice exercises. Since then, we’ve received a lot of positive feedback from both students and teachers, which helped us improve PyCharm Edu a lot. Now, it’s time to go beyond Python.
Please welcome Java and Kotlin learning and teaching support available inside IntelliJ IDEA and Android Studio!
What’s new in PyCharm Edu 2018.1
Educators can now use Markdown not only in task descriptions but also in answer placeholder hints:
Educators can also add Markdown tables and local .png images to task descriptions:
Learner’s progress is now counted for theory tasks and tasks with subtasks:
Learners can now hide all the solved lessons with the Course View Settings button, to stay more focused on lessons they need to take next:
—
Download PyCharm Edu 2018.1, check out the new features and give it a try. Don’t forget to share your feedback!
Your PyCharm Edu Team
PyCharm Scientific Mode with Code Cells
You can use code cells to divide a Python script into chunks that you can individually execute, maintaining the state between them. This means you can re-run only the part of the script you’re developing right now, without having to wait for reloading your data. Code cells were added to PyCharm 2018.1 Professional Edition’s scientific mode.
To try this out, let’s have a look at the raw data from the Python Developer Survey 2017 that was jointly conducted by JetBrains and the Python Software Foundation.
To start, let’s create a scientific project. After opening PyCharm Professional Edition (Scientific mode is not available in the Community Edition), choose to create a new project, and then select ‘Scientific’ as the project type:
A scientific project will by default be created with a new Conda environment. Of course, for this to work you need to have Anaconda installed on your computer. The scientific project also creates a folder structure for your data.
If we want to analyze data, we’ll first need to go get some data. Please download the CSV file from the ‘Raw Data’ section of the developer survey results page. Afterward, place it in the data folder that was created in the scientific project’s scaffold.
Extract, Transform, Load
Our first challenge will be to load the file. The easiest way to do this would be to run:
import pandas as pd pd.read_csv(‘data/survey.csv’)
So let’s run this. After writing this code in the main.py file that was created for us with the project, right click anywhere in the file and choose ‘Run’. We should see a Python console appear at the bottom of our screen after the script completes execution. On the right-hand side, we should see the variable overview with our dataframe. Click ‘View as DataFrame’ to inspect the DataFrame:
We can see the structure of the CSV file here. The columns have headings like “Java:What other language(s) do you use?”. These columns are the result of multiple-choice answers: respondents were asked ‘What other language(s) do you use?’ and could select multiple answers. If an answer was selected, that string is inserted. Otherwise the string ‘NA’ is inserted (if you open the CSV file directly, you’ll be able to see a lot of ‘NA’ values).
If you scroll through the DataFrame a little more, you’ll see that in some cases Pandas was able to correctly infer some data, but in many cases it would be fairly unwieldy to work with the data in this shape.
To make the data easier to work with, we could recode columns after the read_csv call, and fix things. A better way is to configure the read_csv call with various parameters.
In this step, we’d like to make sure that our columns will be named in a way that’s easier to work with, and to make sure that the data types are all correct. To do this, we can use several parameters of read_csv:
- names – allows us to specify the names of the columns (instead of reading them from the CSV file). We need to pay attention to the fact that if we specify this parameter, Pandas will import the header column as a data row by default. We can prevent that by explicitly specifying
header=0, which indicates that the 0th row (the first row) is a header row.
- dtype – enables us to specify a datatype per column, as a dict. Pandas will cast values in these columns to the specified datatype.
- converters – functions that receive the raw value of the cell for a specified column, and return the desired value (with the desired datatype).
- usecols – allows us to specify exactly which columns to import.
For more details, see the documentation for read_csv. Or, just write pd.read_csv in PyCharm to see it in the documentation tool window (this works if you have scientific mode enabled; if not, use Ctrl+Q to see the documentation).
The disadvantage of these parameters is that they take lists and dicts, which become very unwieldy for datasets with many columns. As our dataset has over 150 columns, it would be a pain to write them inline. Also, the information for one column would be spread among these parameters, making it hard to see what is being done to a column.
One great thing about analyzing data with Pandas is that we can use all features of the Python language. So let’s create a data dictionary with plain Python objects, and then use some Python magic to transform these to the structures Pandas needs.
The Data Dictionary
To recap, for every column we want to know what name we will want to give it, and how to encode the values. We also want to have the ability to drop a column.
Let’s create a separate file to hold our data dictionary: survey_data_dictionary.py. In this file, we define a class that describes what we want to do with a column:
class ColumnDescription:
def __init__(self, full_name, name='', dtype=None, converter=None, usecol=True):
self.full_name = full_name
if name:
self.name = name
else:
self.name = full_name
self.dtype = dtype
self.converter = converter
if self.dtype is not None and self.converter is not None:
raise ValueError("Define either a dtype or a converter, not both")
self.usecol = usecolNow we can make a big list of all of our columns, and describe one-by-one what to do with them. To make our lives easier, we can use Pandas to get a list of the current names of the columns. Run in the Python console:
for col in df.columns:
print(‘#{}’.format(col))This will print the full name of every column as a Python comment. Copy & paste the full list into the data dictionary Python file after the class definition. We can now use regex replacement to create instances of our ColumnDescription class.
Open the Replace tool (Ctrl+R or Edit | Find | Replace), and make sure to check ‘Regex’ to enable regex mode. Enter #([^\n]+) as the regex to find. This looks for the ‘#’ character, and then multiple characters that are *not* a newline. Everything between the parentheses is captured into a group, which we can then use in the replacement (use $1 for the first capture group).
As a replacement type (use Ctrl+Shift+Enter to create newlines):
ColumnDescription(
full_name=”$1”
),Make sure you’ve indented the middle line, and used double quotes, and then click “Replace all”. Now we’ve created a lot of ColumnDescription objects. We’ll need them in a list for Pandas, so let’s wrap it with a list constructor now. Write DATA_DICTIONARY = [ before the first ColumnDescription call, and ] at the end of the file. Choose Code | Reformat Code to properly indent all of the ColumnDescription calls.
At this point, we can go back to our main.py and feed this data structure into the read_csv call. Let’s start by adding the ability to rename columns – we do this with the names parameter. This should be a list of strings, with as its length the number of columns in the CSV file.
We can use a Python list comprehension to extract the names from our list of ColumnDescription objects:
from survey_data_dictionary import DATA_DICTIONARY names = [x.name for x in DATA_DICTIONARY]
At this point we can provide this list to read_csv. We also need to remember to specify the header row explicitly so that Pandas doesn’t import the header row as a data row:
df = pd.read_csv(‘data/survey.csv’, names=names, header=0)
If we run this code, we should see nothing has changed. To see if it worked, let’s go back to our data dictionary and add a name to the first column:
ColumnDescription( full_name="Is Python the main language you use for your current projects?", name="python_main" ),
After re-running main.py, we should now see that the first column has been renamed. Crack open a bottle of champagne to celebrate your success!
Let’s provide the other metadata from DATA_DICTIONARY to read_csv with a combination of list comprehensions and dict comprehensions:
# Generate the list of names to import
usecols = [x.name for x in DATA_DICTIONARY if x.usecol]
# dtypes should be a dict of 'col_name' : dtype
dtypes = {x.name: x.dtype for x in DATA_DICTIONARY if x.dtype}
# same for converters
converters = {x.name: x.converter for x in DATA_DICTIONARY if x.converter}
df = pd.read_csv('data/survey.csv',
header=0,
names=names,
dtype=dtypes,
converters=converters,
usecols=usecols)Now all that’s left to do is to populate the rest of the data dictionary. Unfortunately, this is manual work; there’s no way for Pandas to know the design of our survey. If you want to follow along with the rest of the blog post without writing the entire data dictionary, you can grab a complete one from the GitHub repo.
For those columns that are either ‘NA’ or the name of the selected columns, we can create a small helper function that will convert these to booleans:
def notNA(text): return text != ‘NA’
We can then specify this helper function as the converter for a column like this:
ColumnDescription( full_name="Java:What other language(s) do you use?", name="otherlang_java", converter=notNA ),
Another type of data that’s fairly common in surveys is categorical data: multiple choice, single answer. We can specify Pandas’ CategoricalDType as the data type for those columns:
ColumnDescription(
full_name="What do you think is the ratio of these two numbers?:Please think about the total number of Python Web Developers in the world and the total number of Data Scientists using Python.",
name="webdev_science_ratio_self",
dtype=CategoricalDtype(
["10:1", "5:1", "2:1", "1:1", "1:2", "1:5", "1:10"],
ordered=True
)
),
Cleaning up our Data
Although our columns are now looking good, we may want to make some additional changes to our data. In the Python developer survey, the first question was:
Is Python the main language you use for your current projects?
- Yes
- No, I use Python as a secondary language
- No, I don’t use Python for my current projects
All respondents who selected they don’t use Python were excluded from most of the rest of the survey. So we should drop these data points for our analysis. Let’s create a new code cell, and start cleaning up our data.
Code cells are defined simply by creating a comment that starts with #%%. The rest of the comment is the header of the cell, which you see when you collapse it:
#%% Cleaning the data
As long as you have the scientific mode enabled in PyCharm Professional, you should see a dividing line appear, and a green ‘play’ icon to run the cell.
It’s fairly easy to select data in Pandas, so let’s complete our cell:
#%% Cleaning the data df = df[df['python_main'] != "No, I don’t use Python for my current projects"]
As the remaining choices are basically “Yes” and “No”, we can also turn the remaining data into a boolean:
df[‘python_main’] = df[‘python_main’] == ‘Yes’
Analyzing our Data
In our survey, users were asked what they thought the ratio is between the number of Python developers creating web applications, and the number of developers that do data science. To make things interesting, they were also asked what they thought other people thought this ratio was. Let’s see now if people think they agree with the rest of the world.
The questions look like this:
Please think about the total number of Python Web Developers in the world and the total number of Data Scientists using Python.
What do you think is the ratio of these two numbers?
Python Web Developers ( ) 10:1 ( ) 5:1 ( ) 2:1 ( ) 1:1 ( ) 1:2 ( ) 1:5 ( ) 1:10 Python data scientists
What do you think would be the most popular opinion?
Python Web Developers ( ) 10:1 ( ) 5:1 ( ) 2:1 ( ) 1:1 ( ) 1:2 ( ) 1:5 ( ) 1:10 Python data scientists
Make sure you’ve specified categorical data types in the data dictionary for both questions.
We can now go ahead and create a new code cell to start our analysis. Let’s start by getting the value counts:
ratio_self = df['webdev_science_ratio_self'].value_counts(sort=False) ratio_others = df['webdev_science_ratio_others'].value_counts(sort=False)
We’re disabling sorting here to maintain the order that we’ve specified using the categorical data type. If we run this cell with the green play icon, we can then click ‘View as Series’ in the variable overview to have a glance at our data.
We can also use Matplotlib to get a graphical overview of the data:
See the GitHub repository for the exact code used to generate the plot.
We can see in the plot that there’s a difference between what individual respondents thought the ratio was, and what they thought the most popular opinion was. So let’s dive a little deeper: how big is this difference?
Exploring Further
Although the data points are categorical, they represent numbers, so we can see what the numeric difference would be if we turn them into numbers. Let’s create a new code cell, and calculate the difference:
CONVERSION = {
'10:1': 10,
'5:1' : 5,
'2:1' : 2,
'1:1' : 1,
'1:2' : 0.5,
'1:5' : 0.2,
'1:10': 0.1
}
self_numeric = df['webdev_science_ratio_self'] \
.replace(CONVERSION.keys(), CONVERSION.values())
others_numeric = df['webdev_science_ratio_others'] \
.replace(CONVERSION.keys(), CONVERSION.values())
print(f'Self:\t\t{self_numeric.mean().round(2)} web devs / scientist')
print(f'Others:\t\t{others_numeric.mean().round(2)} web devs / scientist')After running this cell, we see:
Self: 3.23 web devs / scientist Others: 3.02 web devs / scientist
Turns out the difference in means isn’t very large. However, the distributions are fairly different. We can see in the plot that the ‘self’ distribution has a peak at the 5:1 web dev:data scientist point, whereas the ‘Most popular’ distribution trades some votes from 5:1 to 1:1. Fun fact: this same survey found about a 1:1 distribution between web developers and data scientists with its Python usage questions.
To see whether or not we have a significant difference, we can use a Chi-Square test. The scipy.stats package contains a method to calculate this statistic. So let’s create a last code cell to finish this investigation:
#%% Is the difference statistically significant? result = scipy.stats.chisquare(ratio_self, ratio_others) # The null hypothesis is that they're the same. Let's see if we can reject it print(result)
This results in: Power_divergenceResult(statistic=294.72519972505006, pvalue=1.1037599850410103e-60).
In other words, there’s a 1-60 chance that this is the result of random chance, and we can conclude this is a statistically significant difference.
What’s Next?
We’ve just shown how to ingest a fairly large CSV file into Pandas, and how to handle the conversion of data from its raw form to a form that’s easier to analyze. For the example, we looked into what respondents think the Python ecosystem looks like. And we’ve confirmed that people think that others have a different opinion from themselves (also, water is wet).
Now it’s your turn! Download the CSV (and you may want to grab the data dictionary from this blog post’s repo) and let us know what interesting things you discover in the Python developer survey! It contains many interesting data points: what people use Python for, what their job roles are, what packages they use, and more.
Peter Bengtsson
Efficient many-to-many field lookup in Django REST Framework
The basic setup
Suppose you have these models:
from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
class Blogpost(models.Model):
title = models.CharField(max_length=100)
categories = models.ManyToManyField(Category)
Suppose you hook these up Django REST Framework and list all Blogpost items. Something like this:
# urls.py
from rest_framework import routers
from . import views
router = routers.DefaultRouter()
router.register(r'blogposts', views.BlogpostViewSet)
# views.py
from rest_framework import viewsets
class BlogpostViewSet(viewsets.ModelViewSet):
queryset = Blogpost.objects.all().order_by('date')
serializer_class = serializers.BlogpostSerializer
What's the problem?
Then, if you execute this list (e.g. curl http://localhost:8000/api/blogposts/) what will happen, on the database, is something like this:
SELECT "app_blogpost"."id", "app_blogpost"."title" FROM "app_blogpost"; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 1025; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 193; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 757; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 853; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 1116; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 1126; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 964; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 591; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 1112; SELECT "app_category"."id", "app_category"."name" FROM "app_category" INNER JOIN "app_blogpost_categories" ON ("app_category"."id" = "app_blogpost_categories"."category_id") WHERE "app_blogpost_categories"."blogpost_id" = 1034; ...
Obviously, it depends on how you define that serializers.BlogpostSerializer class, but basically, as it loops over the Blogpost, for each and every one, it needs to make a query to the many-to-many table (app_blogpost_categories in this example).
That's not going to be performant. In fact, it might be dangerous on your database if the query of blogposts gets big, like requesting a 100 or 1,000 records. Fetching 1,000 rows from the app_blogpost table might be cheap'ish but doing 1,000 selects with JOIN is never going to be cheap. It adds up horribly.
How you solve it
The trick is to only do 1 query on the many-to-many field's table, 1 query on the app_blogpost table and 1 query on the app_category table.
First you have to override the ViewSet.list method. Then, in there you can do exactly what you need.
Here's the framework for this change:
# views.py
from rest_framework import viewsets
class BlogpostViewSet(viewsets.ModelViewSet):
# queryset = Blogpost.objects.all().order_by('date')
serializer_class = serializers.BlogpostSerializer
def get_queryset(self):
# Chances are, you're doing something more advanced here
# like filtering.
Blogpost.objects.all().order_by('date')
def list(self, request, *args, **kwargs):
response = super().list(request, *args, **kwargs)
# Where the magic happens!
return response
Next, we need to make a mapping of all Category.id -1-> Category.name. But we want to make sure we do only on the categories that are involved in the Blogpost records that matter. You could do something like this:
category_names = {} for category in Category.objects.all(): category_names[category.id] = category.name
But to avoid doing a lookup of category names for those you never need, use the query set on Blogpost. I.e.
qs = self.get_queryset() all_categories = Category.objects.filter( id__in=Blogpost.categories.through.objects.filter( blogpost__in=qs ).values('category_id') ) category_names = {} for category in all_categories: category_names[category.id] = category.name
Now you have a dictionary of all the Category IDs that matter.
Note! The above "optimization" assumes that it's worth it. Meaning, if the number of Category records in your database is huge, and the Blogpost queryset is very filtered, then it's worth only extracting a subset. Alternatively, if you only have like 100 different categories in your database, just do the first variant were you look them up "simplestly" without any fancy joins.
Next, is the mapping of Blogpost.id -N-> Category.name. To do that you need to build up a dictionary (int to list of strings). Like this:
categories_map = defaultdict(list) for m2m in Blogpost.categories.through.objects.filter(blogpost__in=qs): categories_map[m2m.blogpost_id].append( category_names[m2m.category_id] )
So what we have now is a dictionary whose keys are the IDs in self.get_queryset() and each value is a list of a strings. E.g. ['Category X', 'Category Z'] etc.
Lastly, we need to put these back into the serialized response. This feels a little hackish but it works:
for each in response.data: each['categories'] = categories_map.get(each['id'], [])
The whole solution looks something like this:
# views.py from rest_framework import viewsets class BlogpostViewSet(viewsets.ModelViewSet): # queryset = Blogpost.objects.all().order_by('date') serializer_class = serializers.BlogpostSerializer def get_queryset(self): # Chances are, you're doing something more advanced here # like filtering. Blogpost.objects.all().order_by('date') def list(self, request, *args, **kwargs): response = super().list(request, *args, **kwargs) qs = self.get_queryset() all_categories = Category.objects.filter( id__in=Blogpost.categories.through.objects.filter( blogpost__in=qs ).values('category_id') ) category_names = {} for category in all_categories: category_names[category.id] = category.name categories_map = defaultdict(list) for m2m in Blogpost.categories.through.objects.filter(blogpost__in=qs): categories_map[m2m.blogpost_id].append( category_names[m2m.category_id] ) for each in response.data: each['categories'] = categories_map.get(each['id'], []) return response
It's arguably not very pretty but doing 3 tight queries instead of doing as many queries as you have records is much better. O(c) is better than O(n).
Discussion
Perhaps the best solution is to not run into this problem. Like, don't serialize any many-to-many fields.
Or, if you use pagination very conservatively, and only allow like 10 items per page then it won't be so expensive to do one query per every many-to-many field.
Codementor
How and why I built A Flask-powered web app
About me I am a newbie to coding, and have been dabbling in Javascript + HTML/CSS, and Bootstrap to create beautiful, unique web apps. The problem I wanted to solve I basically wanted to learn the...
PyCharm
PyCharm Hotfix for Pip 10.0 and IPython 6.3.0 compatibility
Pip 10.0 is close to being released, and changes parts of its API. Several older versions of PyCharm are incompatible with the newer version. If you’d like to use PyCharm with the new version of pip, please update PyCharm.
In addition, IPython 6.3.0 causes a ValueError if run after a script by using the “Show command line afterwards” option. This bug has also been resolved.
The new versions of PyCharm are:
- 2016.3.5
- 2017.1.7
- 2017.2.6
- 2017.3.5
If you’re using a version with a minor update (last number) lower than those mentioned, please update PyCharm. If you wish to update to the latest version (2018.1.1 at the time of writing), get it here. If you wish to get one of the hotfixed older versions, you can find the appropriate release on our previous versions page.
Mike Driscoll
Splitting and Merging PDFs with Python
The PyPDF2 package allows you to do a lot of useful operations on existing PDFs. In this article, we will learn how to split a single PDF into multiple smaller ones. We will also learn how to take a series of PDFs and join them back together into a single PDF.
Getting Started
PyPDF2 doesn’t come as a part of the Python Standard Library, so you will need to install it yourself. The preferred way to do so is to use pip.
pip install pypdf2
Now that we have PyPDF2 installed, let’s learn how to split and merge PDFs!
Splitting PDFs
The PyPDF2 package gives you the ability to split up a single PDF into multiple ones. You just need to tell it how many pages you want. For this example, we will download a W9 form from the IRS and loop over all six of its pages. We will split off each page and turn it into its own standalone PDF.
Let’s find out how:
# pdf_splitter.py import os from PyPDF2 import PdfFileReader, PdfFileWriter def pdf_splitter(path): fname = os.path.splitext(os.path.basename(path))[0] pdf = PdfFileReader(path) for page in range(pdf.getNumPages()): pdf_writer = PdfFileWriter() pdf_writer.addPage(pdf.getPage(page)) output_filename = '{}_page_{}.pdf'.format( fname, page+1) with open(output_filename, 'wb') as out: pdf_writer.write(out) print('Created: {}'.format(output_filename)) if __name__ == '__main__': path = 'w9.pdf' pdf_splitter(path)
For this example, we need to import both the PdfFileReader and the PdfFileWriter. Then we create a fun little function called pdf_splitter. It accepts the path of the input PDF. The first line of this function will grab the name of the input file, minus the extension. Next we open the PDF up and create a reader object. Then we loop over all the pages using the reader object’s getNumPages method.
Inside of the for loop, we create an instance of PdfFileWriter. We then add a page to our writer object using its addPage method. This method accepts a page object, so to get the page object, we call the reader object’s getPage method. Now we had added one page to our writer object. The next step is to create a unique file name which we do by using the original file name plus the word “page” plus the page number + 1. We add the one because PyPDF2’s page numbers are zero-based, so page 0 is actually page 1.
Finally we open the new file name in write-binary mode and use the PDF writer object’s write method to write the object’s contents to disk.
Merging Multiple PDFs Together
Now that we have a bunch of PDFs, let’s learn how we might take them and merge them back together. One useful use case for doing this is for businesses to merge their dailies into a single PDF. I have needed to merge PDFs for work and for fun. One project that sticks out in my mind is scanning documents in. Depending on the scanner you have, you might end up scanning a document into multiple PDFs, so being able to join them together again can be wonderful.
When the original PyPdf came out, the only way to get it to merge multiple PDFs together was like this:
# pdf_merger.py import glob from PyPDF2 import PdfFileWriter, PdfFileReader def merger(output_path, input_paths): pdf_writer = PdfFileWriter() for path in input_paths: pdf_reader = PdfFileReader(path) for page in range(pdf_reader.getNumPages()): pdf_writer.addPage(pdf_reader.getPage(page)) with open(output_path, 'wb') as fh: pdf_writer.write(fh) if __name__ == '__main__': paths = glob.glob('w9_*.pdf') paths.sort() merger('pdf_merger.pdf', paths)
Here we create a PdfFileWriter object and several PdfFileReader objects. For each PDF path, we create a PdfFileReader object and then loop over its pages, adding each and every page to our writer object. Then we write out the writer object’s contents to disk.
PyPDF2 made this a bit simpler by creating a PdfFileMerger object:
# pdf_merger2.py import glob from PyPDF2 import PdfFileMerger def merger(output_path, input_paths): pdf_merger = PdfFileMerger() file_handles = [] for path in input_paths: pdf_merger.append(path) with open(output_path, 'wb') as fileobj: pdf_merger.write(fileobj) if __name__ == '__main__': paths = glob.glob('w9_*.pdf') paths.sort() merger('pdf_merger2.pdf', paths)
Here we just need to create the PdfFileMerger object and then loop through the PDF paths, appending them to our merging object. PyPDF2 will automatically append the entire document so you don’t need to loop through all the pages of each document yourself. Then we just write it out to disk.
The PdfFileMerger class also has a merge method that you can use. Its code definition looks like this:
def merge(self, position, fileobj, bookmark=None, pages=None, import_bookmarks=True): """ Merges the pages from the given file into the output file at the specified page number. :param int position: The *page number* to insert this file. File will be inserted after the given number. :param fileobj: A File Object or an object that supports the standard read and seek methods similar to a File Object. Could also be a string representing a path to a PDF file. :param str bookmark: Optionally, you may specify a bookmark to be applied at the beginning of the included file by supplying the text of the bookmark. :param pages: can be a :ref:`Page Range <page-range>` or a ``(start, stop[, step])`` tuple to merge only the specified range of pages from the source document into the output document. :param bool import_bookmarks: You may prevent the source document's bookmarks from being imported by specifying this as ``False``. """
Basically the merge method allows you to tell PyPDF where to merge a page by page number. So if you have created a merging object with 3 pages in it, you can tell the merging object to merge the next document in at a specific position. This allows the developer to do some pretty complex merging operations. Give it a try and see what you can do!
Wrapping Up
PyPDF2 is a powerful and useful package. I have been using it off and on for years to work on various home and work projects. If you need to manipulate existing PDFs, then this package might be right up your alley!
Related Reading
- A Simple Step-by-Step Reportlab Tutorial
- ReportLab 101: The textobject
- ReportLab – How to add Charts and Graphs
- Extracting PDF Metadata and Text with Python
Michael Droettboom
Profiling WebAssembly
Summary: Tips for profiling WebAssembly
I couldn't find a comprehensive guide to profiling WebAssembly, so I thought I'd share my own limited experience here. In my last post, I talked about benchmarking a WebAssembly port of the scientific Python stack. I knew which benchmarks were doing better than others and had some theories about why, but since I didn't yet know how to profile WebAssembly, I couldn't really answer that with any certainty.
It turns out that profiling WebAssembly is quite easy.
Rebuilding with the --profiling flag
The first step is to rebuild the application with the --profiling flag passed to both the compiler and the linker for every object. This makes sure that all of the information necessary for profiling is available in the output and makes the code more readable. The typical way to do this would be to set CFLAGS and LDFLAGS and let the ./configure script for your project pick those up. In the case of pyodide, the Python cross-compiling setup makes that tricky, or at least I couldn't figure it out in a short amount of time. Fortunately, emscripten provides a handy backdoor to just force this on everything: the EMCC_CFLAGS environment variable. Therefore, to make a profiling-friendly build of pyodide:
make clean
EMCC_CFLAGS=--profiling make
Setting start and stop points for profiling
You generally don't want to profile an entire run, which would include initialization and other things. It turns out there's a handy Javascript API to turn the profiler on and off.
- console.profile() turns the profiler on.
- console.profileEnd() turns the profiler off.
If you wanted to call these from C/C++, you could use the EM_ASM macro, which allows you to insert literal Javascript into the C application:
EM_ASM(
console.profile();
);
In my case, I wanted to turn the profiler on and off from Python, so I can do:
from js import console
console.profile()
Profiling
The actual profiling is performed within the development tools of your browser. When you load an .html file that runs the WebAssembly built and instrumented as above, it will record a set of profiling data available from the Performance tab.
I'll refer you to the Performance Tools documentation for more information. Suffice it to say that profiling WebAssembly is almost exactly like profiling vanilla Javascript in the browser.
Case study
For pyodide, I created a profiling build to look into the julia benchmark that I knew was performing poorly. Right away, I noticed from the Call Tree that 50% of the time was spent in this function:
for (var named in NAMED_GLOBALS) {
(function(named) {
Module['g$_' + named] = function() {
return Module['_' + named] // <- 50% of runtime HERE
};
})(named);
}
This code is actually part of the boilerplate that emscripten emits. It helps dynamically loaded modules (such as Numpy in my case) access symbols in the main module. Since these symbols don't change at runtime, we don't actually need to do the dictionary lookup for Module['_' + named] every time, we can cache (memoize) it at startup and then just use that:
for (var named in NAMED_GLOBALS) {
(function(named) {
var func = Module['_' + named];
Module['g$_' + named] = function() {
return func;
};
})(named);
}
This 2-line change to emscripten resulted in significant speedups in my pyodide benchmarks across the board.
descriptionHere, the x-axis is the number of times slower that WebAssembly runs vs. native code. The grey bars are the timings before this change, and blue bars are the timings after this change.
More details about this changes are in the pull request.
Scientific Python in the Browser
Summary: An early report on getting the scientific Python stack compiled to WebAssembly.
Data Science in the Browser
Shortly after starting at Mozilla in January, I became aware of Hamilton Ulmer and Brendan Colloran's Iodide project, an experiment to build a data science notebook based on web technologies. Unlike Jupyter notebooks, the computation happens in the browser, with direct access to Web API technologies like the DOM. Sharing a notebook is as simple as passing around a single HTML file, since there's no server side to worry about. It's not out to replace Jupyter notebooks, but rather to exist in a different design tradeoff space that makes it more suitable the sharing and collaboration.
Since it targets the browser, the programming language of Iodide is, of course, Javascript. While there are a number of libraries for doing data science in Javascript, such as numjs and scijs, they aren't as widely used or as battle-tested as the scientific Python or R ecosystems. Nonetheless, I think "data science in Javascript" is an interesting area to explore, particularly since Javascript has some of the best JIT compilers of any dynamic language. This advantage allows writing both high-level orchestration and low-level numeric code in the same language, side-stepping the notorious "two language problem" in scientific Python. (In Python land, most of the core scientific libraries have significant chunks of code in lower level languages such as C, FORTRAN or Cython for performance reasons.) Combining Javascript's great compiler technology, and perhaps adding a smattering of transpilation to fix some syntactic issues, is really promising, and Iodide as a project is exploring that space.
Nonetheless, we received frequent feedback that Iodide "looks really cool, but I wish I could use the Python (or R) tools I'm familiar with." I understood in theory that it should be possible to compile the Python interpreter into WebAssembly in order to run it in the browser. There are already a few projects that do this: (cpython-emscripten, micropython javascript support, pypyjs). Unfortunately, I couldn't find a project that included a practical scientific Python stack including Numpy and friends. I was concerned about the amount of effort it would take to build such a thing, and also whether the result would be performant enough to be useful. In February, we had a conversation with some folks who work on WebAssembly tooling at Mozilla, and they were pretty bullish that it wouldn't be too hard. Based on their optimism, I gave it a shot, and starting with dgym's cpython-emscripten as a basis, I had the basic parts of a Python interpreter working in WebAssembly in a couple of days. Of course, going from that to a working Numpy took much longer, but thanks to some help from Alon Zakai and others, Numpy is working, too. With that done, it has been much easier getting other libraries higher up the stack to work, including preliminary support for Pandas.
Tight integration
One thing that sets this implementation apart from other Python-in-the-browser projects I've come across is the ability to easily pass and share objects between Python and Javascript.
The basic Python data types (None, bool, int, float, str, bytes, list and dict) are transparently converted to and from their Javascript equivalents. Other types, including Numpy arrays, are wrapped in a proxy that allows Javascript to call their methods and access their items and attributes. Vice versa, Javascript objects are wrapped in a Python proxy. These proxies allow objects to be shared on both sides of the language barrier without copying, which is particularly important for large Numpy arrays.
Say, for example, you had a value in Javascript:
// javascript
secret = "Wklv#lv#olnh#pdjlf$"
You could use it from Python by using the from js import ... syntax:
# python
from js import secret
decoded = ''.join(chr(ord(x) - 3) for x in secret)
And then send data back to the Javascript side using pyodide.pyimport:
// javascript
var decoded = pyodide.pyimport("decoded")
One of the coolest side effects of this design is that Python has complete access to the Web API, so it can manipulate the DOM, use HTML Canvas, access webcams or audio and all the other cool things you can do from Javascript in a browser.
For example, changing the browser tab's title is as simple as importing window and setting an attribute:
from js import window
window.title = "My mind is blown"
What works
Most of the Python standard library works. The most notable exceptions are:
- subprocess: since the browser isn't an OS, it can't spawn new processes.
- socket: access to raw network sockets would break the browser security model. There are a lot of networking-related things in the standard library built on socket that therefore also don't work.
- All of the browser sandboxing still applies, so you can't access the local filesystem. However, by calling through Javascript, you do have access to XMLHttpRequest and browser local storage. Eventually, Python wrappers around this functionality should be written to make those operations feel more like they do in native Python.
Within Numpy, all of the core functionality works, but there's no support for long double (but those are pretty niche). There are still some low-level compiler bugs that prevent the FFT stuff from compiling, but that should eventually resolve.
How fast is it?
To answer this question, I reached for a few existing Python and Numpy benchmarks:
- The venerable pystone, which ships with CPython.
- Serge Guelton's set of numpy benchmarks.
These benchmarks probably fall into the trap of being a little too "synthetic". I would have preferred to also use the Python Performance Benchmark Suite, which aims to be a little closer to "real world", but it has a significant number of dependencies and would need to be adapted to work on a platform without subprocess before it could be used in this context. Nonetheless, I think these benchmarks offer a useful approximation for now.
The benchmarks were run on the same machine in the native CPython implementation and in Firefox Nightly using selenium. The following figure shows how many times slower the WebAssembly implementation is.
descriptionEDIT 2018-04-10: The original results posted here inadvertently included Numpy import time in the WebAssembly times (but not in the native times). These have now been corrected above. There is some improvement in the results, but not in a best or worst case. You can see the original results here.
The results are interesting. For benchmarks that spend most of their time in Numpy routines, such as harris or rosen, runtime is at par with the native-compiled Python. When WebAssembly rocks, it really, really rocks. Unfortunately, for other benchmarks that spend a lot of time looping or making function calls in Python, runtimes can be as much as 35 times slower. I have an unsubstantiated hunch that this is due to the use of Emscripten's EMULATE_FUNCTION_POINTER_CASTS option which is required to make all of the function pointer calls that CPython does work correctly.
UPDATE 2018-04-11: My hunch was wrong, and I was able to get to the bottom of the root cause and significantly speed up these benchmarks. See my post Profiling WebAssembly for more info.
Future directions
I'd love to see improvements to the toolchain that close the performance gap. At this point, I don't personally know enough to anticipate how much work is involved.
Another current limitation is that all of the packages you anticipate you might need must be compiled and wrapped into a single large data file that is downloaded in its entirety to your browser before anything can start. It would be great to modularize that, so that packages are downloaded on demand. Related to that, it would also be helpful to modularize the build system so that individual packages can be added more independently. Conda build could potentially serve as a basis for that.
Check it out
The easiest way to play with this is to visit the example Pyodide notebook (EDIT: This link was fixed to a working version). (Note that this only works on Firefox right now. Chrome support is pending).
You can also get involved at pyodide github repository. Note that while Pyodide grew out of the needs of Iodide, there's nothing Iodide-specific about it, and it should be useful in other contexts where you want to embed a scientific Python stack in the browser. I'm pretty new to WebAssembly and I'd love any help, advice or comments to make this better.
Continuum Analytics Blog
What You Missed on Day Two of AnacondaCON 2018
What a day! On Tuesday we got started bright and early, then partied our way into the night. Here are some highlights from Day Two of AnacondaCON 2018. Opening Keynote: John Kim John Kim, President of HomeAway, kicked things off for us with a personal, touching keynote on Love in the Age of Machine Learning. …
Read more →
April 10, 2018
Test and Code
40: On Podcasting - Adam Clark
Adam is the host of The Gently Mad podcast, and teaches the steps in creating and growing a podcast in his course Irresistible Podcasting.
He was one of the people who inspired Brian to get the Test & Code podcast started in the first place. Brian took his course in 2015. Adam is in the process of updating the course, and building a community around it.
Warning: This may be an episode to listen to with headphones if you have kids around. There is swearing.
I wanted to get Adam's help to convince many of you to either come on this show as a guest, or start your own podcast. We did some of that. But we also cover a lot of issues like self doubt and the importance of community.
Special Guest: Adam Clark.
Sponsored By:
- Python Testing with pytest: Simple, Rapid, Effective, and Scalable The fastest way to learn pytest. From 0 to expert in under 200 pages.
- Patreon Supporters: Help support the show with as little as $1 per month. Funds help pay for expenses associated with the show.
Links:
- Irresistible Podcasting – A Step by Step Guide to Launching and Growing a Podcast that Can’t Be Ignored
- The Gently Mad – Life, business & entrepreneurship without the bullshit.
- avclark.com
- Justin Jacksons episodes on minimal podcasting | Build and Launch
- Dan Benjamin's Podcast Method podcast
- Python Bytes Podcast
- ATR2100 Mic
- Shure SM57 Mic
- Shure SM7B mic
- dbx 286s Microphone Preamp Processor
- Focusrite Scarlett 2i2 USB Audio Interface
Reinout van Rees
First Rotterdam (NL) Python meetup: my summaries
Welcome to the first Python Rotterdam meetup - Thijs Damsma
He searched for python meetups in Rotterdam, but didn't find any. So he clicked on the "do you want to start one?" button.
Today's meetup is at Van Oord (a big family owned marine contractor), but anywhere else in Rotterdam is fine. And you can help out. Everything is fine, from talks to workshops, as long as it is python-related.
The goal is to have a few meetups per year.
Jupyter lab - Joost Dobken
Joost is a data engineer. He uses python daily, mostly jupyter notebooks. Jupyter notebooks are ideal for getting to know python, btw.
Jupyter lab is the next iteration of notebooks. It is in beta, but usable.
Normally, you still work in notebooks, but you can also start a regular python prompt or a terminal. It starts to look like an IDE: file browser, generic editor, etc. And very nice for data analysis, for instance with a fast CSV browser.
He demoed the markdown integration. A source file with an (live updating) output window side by side. And a terminal window at the bottom.
Very fancy: there is an extension that works with google docs. So you can collaboratively work on the same document (via google docs). And everything just works. He demoed it with a colleague: fancy!
Interactive geospatial analysis with python, notebooks and geopandas - Erna Oudman
Erna gave a nice presentation on geospatial data, using trash containers in Rotterdam as an example. She uses geopandas, a spatial add-on to pandas (=timeseries).
With geopandas, it is easy to read geospatial datasets, like all the administrative areas in the Netherlands. You can display them directly in the jupyter notebook (or rather jupyter lab, as she's now using that on a day-to-day basis).
Nice features: you can easily give all areas a random color so that the areas are better recognizable. But of course you can also do geo calculations like "how big is the area".
The trash container info was in an excel file. Geopandas could read that one without problems, too. The coordinates were regular numbers in two separate columns. Of course python can convert that to proper coordinates. Also a transformation from one projection to another wasn't a problem.
She used various handy python packages, like an openstreetmap importer, a geolocator. And "folium" for inserting an interactive javascript "leaflet" map into the notebook.
In the end, she could color-code the areas depending on the number of trash containers (relative to the number of people living in the area).
Python adaptive - Bas Nijholt
Bas works on Python adaptive, "a tool for adaptive and parallel evaluation of functions". He's a PhD student that works on something fancy quantum-computer related. "I use half a year of CPU time per day for my research".
With such a huge amount of calculations, it makes sense to optimize it. "Adaptive" is a strategy he uses: figure out where more calculation is useful by sampling a function. Sampling is to calculate a couple of points and to try to detect where the biggest changes happen and concentrate on those areas.
That's where he wrote python-adaptive for: a library that handles it generally.
It works based on a "Learner" object that takes the function to learn and the bounds. With three methods, you can get it to learn and improve and add new points and re-evaluate etc etc.
He showed some sample code. A problem with it was that, while running, it blocks the CPU. And it only uses one thread. For that, there's a separate runner that can run the calculation on multiple computer cores at the same time.
He also demoed it with a 2D figure: great. The difference with a non-adaptive (homogeneous) figure was striking.
How does the multiprocessor stuff work? Simply from concurrent.futures import ProcessPoolExecutor, from the standard library. It even works on the university's supercomputer. Standard python!
Automating our offshore engineering work - Daan Scheltens
They do lots of calculations on ship movements. How does a ship move in waves. How when there's a crane with a heavy load on the deck?
He showed a short movie about an offshore windmill farm being build. Those ships and cranes and wind turbines are absolutely huge. Not exactly the same movie, but this one is pretty similar. It is the same vessel he used in his talk.
When lifting the big yellow pipes, there are various variables you have to take into account. Such a pipe musn't descent too fast. And not move too fast in the horizontal plane. And the stresses on the cables keeping the pipe in place shouldn't get too big.
With jupyter notebook, he showed the effect of some alternative rigging configurations they were researching: with a few extra lines, they could improve the resistance to interference quite effectively. The actual calculation happens in a specific program, but they can interface with it with python.
With python they also made graphs (based on weather forecasts) for the installation vessels with info on whether they can actually do the work, based on the time of day and the heading of the vessel. Such a simple graph can then easily be used on the vessel.
Techniques for speeding up your python: numpy, cython and numba - Thijs Damsma
He used run length encode as an example. You take a sequence ('a a a c b b`) and transform it to '3xa, 1xc, 2xb'. Often you can get quite a good compression out of this with the right kind of data.
He showed a couple of implementations, starting with a pure python one. Next came a numpy implementation. Drawback: the numpy code is unreadable. Advantage: it is (in his example) 34 times faster than the pure python code.
Next: compile the pure python code with cython. It was a 50% improvement over regular python. But when you specify data types, cython can do a much better job: it even beats numpy with a factor of 2.
Dirty, dirty, dirty: you can tell cython to switch off lots of python safety features (for memory allocation and so).... Another factor of 8. In the end: 18x faster than numpy and 628x faster than the original python code.
Another go: numba, just in time compilation via a decorator. Twice at fast as numpy. Almost as fast as most of the cython efforts, but with only a simple decorator.
