Category Archives: Software

System wide python profiling

I recently had to profile an embedded system that runs a bunch of python code. The goal was to improve the start-up speed and identify bottlenecks. This is a large and complex system that has many process starting and stopping and forking all over the place. It looked to be a difficult task to hook the profiler into each and every process that would get launched. I came up with a solution to temporarily hook the profiler into the interpreter itself and dump the stats at exit using the atexit module.

It turns out, python has the nifty ability to run code at start-up by adding it to the sitecustomize.py file. This file should be put in your python’s lib directory, in my case: /usr/lib/python2.7/sitecustomize.py. Anything you add to this file will get run when the python interpreter gets started. For example, if your file has only:

print "Hello World"

You will get this on starting python:

Hello World
Python 2.7.2 (default, Jan 30 2013, 17:47:27) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

So, in order to get the interpreter to fire up the profiler and dump the stats at exit, I just need to add some code to this file to do just that. Here is what I used:

import cProfile
import atexit
import os,sys
def ProfExit(p):
  p.disable()
  prof_f_name = "%d"%os.getpid()
  p.dump_stats("/root/profiles/%s"%prof_f_name)
profile_hook = cProfile.Profile()
atexit.register(ProfExit, profile_hook)
profile_hook.enable()

So what does this do? It imports the profiler and the atexit module. It creates an instance of the profiler, registers with atexit to stop the profiler and dump the stats to a file named with the process ID of that python process, and finally starts the profiler. So every python process run on the system will now be profiled! FYI, the stats won’t get dumped until the process exits, so make sure you stop all of them.

What do I do will all of these stats files now that I have them. Well, you could go inspect them with the pstats module… or you could use this other great tool I found gprof2dot to convert them into call graphs. You must also install Graphviz to convert the dot files into images (apt-get install graphviz). But the results are worth it! The image show an example of the output, each block is a function called, and color codded by how much time was spent in that call. This makes it pretty easy to identify slow paths and easy optimizations. To generate call graphs for each stats file, I used something like this, from the directory where my stats files were:

for i in *; do ../gprof2dot.py -f pstats $i | dot -Tsvg -o ../callgraphs/$i.svg; done;

 

Call Graph

Call Graph

 

 

The need for a universal Virtual Machine

Lately I have been working on software, a lot, from low level embedded Linux stuff to ruby on rails applications. One idea that keeps coming back to me is how great it would be if there was a standard open source virtual machine platform on which to do (almost) all development on. Not just on the internet, or the desktop or mobile, but everywhere. I could choose my language of choice, instead of being stuck with say javascript in the browser or java on android or objective C on iOS. Each language would be treated equally, not an afterthought hacked in with reduced functionality. Applications could be written to work both online and off and the interface would look identical everywhere.

A VM that I have been watching closely lately is Parrot VM with the goal to run all languages. Under the hood, you can write libraries in an intermediate language that are then easily exposed to languages implemented on top of it. While exciting, development is slow, and my favorite language ruby, currently has only an unusable implementation. I keep imagining how cool it could be if this VM had every language you can imagine available and it was available in the browser, my computer and my phone. Also, how great it would be if every library was available for it, and therefor also available to every language on top of it.

I know there are many existing VMs, like the JVM, CLR and others, but almost all have been fatally flawed, often not by technical reasons but for political ones. I think the idea of a virtual machine not controlled by corporations, but by open standards, that could run everywhere is very powerful. It would make it much easier for new vendors to enter the mobile market, since all you really need to do is get the VM running on your hardware. It would ease compatibility and lock in issues, reduce how much code has to be written to support multiple platforms and give developers the flexibility they want.

I think there are many developers today who are held back by a lack of flexibility, not being able to use their language of choice at different layers of the stack. While its certainly true that some languages are better than others for certain tasks, I don’t think that judgement should be forced upon us.

Installing applications would be as easy as requesting a URI, there would be no cross platform issues, compiler issues, porting, and updating your app or your OS would be simple. Obviously there would be many objections to this idea, many may cite performance issues, but considering 90+% of what most people do today is in the browser, I don’t think this is great argument. Also, modern VMs can approach the performance of native code. Also, parsing HTML + JS is slow compared to reading in bytecode. Finally, the problems it would solve I believe would be immense and well worth the performance hit for most tasks.

Here are some ways I imagine it could be used:

Browser: The browser could either work as it does today, but new standards could require support for the universal VM. OR a new type of browser could be built around the concept. Users would then download bytecode files directly instead of html + script files. Compatibility modules could be created so that existing HTML + JS websites could be converted into bytecode and run in the VM Browser using HTML & JS languages on top of the VM. This would allow for easy support for existing websites, while allowing incremental migration and still supporting old browsers.

Desktop: A linux kernel with a thin hardware abstraction layer. Above that, the VM runs the show, with all software running in the VM. Imagine how cool it would be if all aspects of your system were available from libraries in every language. This would basically be just Linux with a browser running on top and there would be no distinction between web and local applications.

Mobile: Pretty much the same as the desktop.

 

In summary, what I am imaging is a sort of unified web and OS, where everything is run in a VM and most all languages are available and on an equal footing. I have many more ideas why this would be awesome as well as more potential criticisms, but I could not list them all!