System wide python profiling

I recently had to profile an embedded system that runs a bunch of python code. The goal was to improve the start-up speed and identify bottlenecks. This is a large and complex system that has many process starting and stopping and forking all over the place. It looked to be a difficult task to hook the profiler into each and every process that would get launched. I came up with a solution to temporarily hook the profiler into the interpreter itself and dump the stats at exit using the atexit module.

It turns out, python has the nifty ability to run code at start-up by adding it to the sitecustomize.py file. This file should be put in your python’s lib directory, in my case: /usr/lib/python2.7/sitecustomize.py. Anything you add to this file will get run when the python interpreter gets started. For example, if your file has only:

print "Hello World"

You will get this on starting python:

Hello World
Python 2.7.2 (default, Jan 30 2013, 17:47:27) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

So, in order to get the interpreter to fire up the profiler and dump the stats at exit, I just need to add some code to this file to do just that. Here is what I used:

import cProfile
import atexit
import os,sys
def ProfExit(p):
  p.disable()
  prof_f_name = "%d"%os.getpid()
  p.dump_stats("/root/profiles/%s"%prof_f_name)
profile_hook = cProfile.Profile()
atexit.register(ProfExit, profile_hook)
profile_hook.enable()

So what does this do? It imports the profiler and the atexit module. It creates an instance of the profiler, registers with atexit to stop the profiler and dump the stats to a file named with the process ID of that python process, and finally starts the profiler. So every python process run on the system will now be profiled! FYI, the stats won’t get dumped until the process exits, so make sure you stop all of them.

What do I do will all of these stats files now that I have them. Well, you could go inspect them with the pstats module… or you could use this other great tool I found gprof2dot to convert them into call graphs. You must also install Graphviz to convert the dot files into images (apt-get install graphviz). But the results are worth it! The image show an example of the output, each block is a function called, and color codded by how much time was spent in that call. This makes it pretty easy to identify slow paths and easy optimizations. To generate call graphs for each stats file, I used something like this, from the directory where my stats files were:

for i in *; do ../gprof2dot.py -f pstats $i | dot -Tsvg -o ../callgraphs/$i.svg; done;

 

Call Graph

Call Graph

 

 

The need for a universal Virtual Machine

Lately I have been working on software, a lot, from low level embedded Linux stuff to ruby on rails applications. One idea that keeps coming back to me is how great it would be if there was a standard open source virtual machine platform on which to do (almost) all development on. Not just on the internet, or the desktop or mobile, but everywhere. I could choose my language of choice, instead of being stuck with say javascript in the browser or java on android or objective C on iOS. Each language would be treated equally, not an afterthought hacked in with reduced functionality. Applications could be written to work both online and off and the interface would look identical everywhere.

A VM that I have been watching closely lately is Parrot VM with the goal to run all languages. Under the hood, you can write libraries in an intermediate language that are then easily exposed to languages implemented on top of it. While exciting, development is slow, and my favorite language ruby, currently has only an unusable implementation. I keep imagining how cool it could be if this VM had every language you can imagine available and it was available in the browser, my computer and my phone. Also, how great it would be if every library was available for it, and therefor also available to every language on top of it.

I know there are many existing VMs, like the JVM, CLR and others, but almost all have been fatally flawed, often not by technical reasons but for political ones. I think the idea of a virtual machine not controlled by corporations, but by open standards, that could run everywhere is very powerful. It would make it much easier for new vendors to enter the mobile market, since all you really need to do is get the VM running on your hardware. It would ease compatibility and lock in issues, reduce how much code has to be written to support multiple platforms and give developers the flexibility they want.

I think there are many developers today who are held back by a lack of flexibility, not being able to use their language of choice at different layers of the stack. While its certainly true that some languages are better than others for certain tasks, I don’t think that judgement should be forced upon us.

Installing applications would be as easy as requesting a URI, there would be no cross platform issues, compiler issues, porting, and updating your app or your OS would be simple. Obviously there would be many objections to this idea, many may cite performance issues, but considering 90+% of what most people do today is in the browser, I don’t think this is great argument. Also, modern VMs can approach the performance of native code. Also, parsing HTML + JS is slow compared to reading in bytecode. Finally, the problems it would solve I believe would be immense and well worth the performance hit for most tasks.

Here are some ways I imagine it could be used:

Browser: The browser could either work as it does today, but new standards could require support for the universal VM. OR a new type of browser could be built around the concept. Users would then download bytecode files directly instead of html + script files. Compatibility modules could be created so that existing HTML + JS websites could be converted into bytecode and run in the VM Browser using HTML & JS languages on top of the VM. This would allow for easy support for existing websites, while allowing incremental migration and still supporting old browsers.

Desktop: A linux kernel with a thin hardware abstraction layer. Above that, the VM runs the show, with all software running in the VM. Imagine how cool it would be if all aspects of your system were available from libraries in every language. This would basically be just Linux with a browser running on top and there would be no distinction between web and local applications.

Mobile: Pretty much the same as the desktop.

 

In summary, what I am imaging is a sort of unified web and OS, where everything is run in a VM and most all languages are available and on an equal footing. I have many more ideas why this would be awesome as well as more potential criticisms, but I could not list them all!

Pan/Tilt Servo Camera, and updated Servo Controller Code

I built this crude Pan/Tilt Camera mount and used my servo controller and a joystick to control it. Pretty cool… Also, I completely re-wrote my servo controller code since the original code was pretty messy, new code looks a lot better. Also, I included a couple Perl scripts demonstrating how to control it from your computer, using a joystick or whatever. Download the new code on the previous servo controller post.

3 Axis CNC Machine

One tool thats becoming a must have for the home hacker is the CNC machine. I have been looking into these machines for the past few months and have been trying to come up with a design that will suite my needs. People use these machines for many different applications, but my main reason for wanting one is for milling PCB’s, while also wanting to make some small parts from plastic/wood. This being the goal, I finally came to the conclusion that accuracy/precision is more important to me than size. So my final design has only a 4″x4″x4″ x/y/z range with 1 mil/step theoretical accuracy (or 1/8 mil/step if using micro-stepping. This should be large enough for most of the prototype PCBs I make while still large enough to make some parts and things from plastic and wood.

Initially I was trying to build the machine from scratch using wood. Ultimately I was not satisfied with the accuracy I was (not) able to achieve. Also, I was trying to go the route of using of the shelf parts from hardware stores, such as threaded rod and nuts for the lead screw and stainless steel rod for the linear rails. I found that it was very difficult to avoid binding and the friction from the threaded rod and nut seemed too much. Later I found I could use some high quality drawer sliders for the rails and this worked quit well, yet I still had the lead screw problem.

Finally I found some linear actuators at HSC Electronics that suited my needs perfectly. As much as it hurts me to give in and buy pre-made parts, I could not resist the $40 price :) Also its better than anything I could build and I have been quit happy with them. Ultimately, the goal for this project is not the project itself, but to have a working CNC machine that I can use to facilitate my other projects, so I don’t mind using some pre-made parts.

The drivers I am using are the EasyDriver’s from sparkfun. These are okay, but really kind of under powered for these steppers I am using. The Linear axis have a homing swith and an encoder that I am not currently utilizing, but I plan to in the future.

Check out these videos of my progress.

Servo Controller

Top of Revision 2.0 Board

Top of Revision 2.0 Board

I wanted to create a stand alone board that could drive servos and that could be controlled over USB. I used an ATMega328 and the

FT232RL. This final board as you can see is mostly all surface mount components. Below you can see my first prototype board, all through hole and could only control four servos, while this one can do eight.

This final board can be powered by either the USB or the barrel jack connector that powers the servos. I have noticed an issue that if the servos move to fast, it seems to be resetting the FT232RL, which is not such a big deal though since it is not plugged into the USB if it is being powered only by the barrel jack. The servo PWM is all done by my own interrupt driven code instead of using the hardware PWM for several reasons. The board runs at 16MHz and is capable of producing PWM signals from .5ms to 3ms (this is limited in software but can be changed to smaller or larger values). I used an ATMega328 for this, which is probably overkill for controlling servos (and man are they hard to find these days!), ATMega168 could e a drop in replacement, and many lesser micros could do the job as well.

This board also served as an experimental board to learn more about Atmel micros, since before I mostly used PIC (see senior project). Also, I wanted to practice more with surface mount components which is a bit intimidating, but turned out to be not such a big deal. I actually much prefer surface mount now. Again I used Eagle and sparkfun’s library. Also, I did all of the software development in in Linux which is sweet and the main reason I moved from PIC to atmel. You can install a GCC toolchain for the ATMega and us make and other build tools as you would normally. I even got a Makfile to have a “make install” rule to automatically program the chip with avrdude! Awesome!

The source code and Eagle project fies are available below. The code is not fully complete for the eight channel Rev. 2.0 board, only the first six servos work right now. I screwed up on the aove board and connected the last two servos to the wrong pins on the uController, this is fixed in the Eagle project. The menu and commend interface has not been fully updated to support all eight servos, but I will complete this soon and should not be too hard to figure out on your own.

Download Code + Eagle Files Here

Code: https://github.com/seiferteric/servo-controller

Eagle Project: https://github.com/seiferteric/servo-controller-eagle

Note:

If you want to compile this code, you will need a few things, if you are running Ubuntu/Debian install these packages:

  • gcc-avr
  • binutils-avr
  • avrdude
  • avr-libc
  • (build-essential … etc standard c dev stuff)
Schematic of Revision 2.0 Board

Schematic of Revision 2.0 Board

Top of Revision 1.0 Board

Top of Revision 1.0 Board

Senior Project: Automatic Pointing and Tracking Telescope (APTT)

I just graduated and my senior project is complete! Here are the details…

Telescope

Telescope

Concept

My project is called the A.P.T.T. or Automatic Pointing and Tracking Telescope. I wanted to build a telescope that would automatically point at and track objects (stars and planets) in the sky so you would not have to spend all night trying to find them. There are already commercial versions of systems like this from many companies, but they are fairly expensive. I bought a regular telescope (from Orion) with an altazimuth mount and I modified the mount by adding bearings and servos for motion control. I used a PIC 18F2550 microcontroller on a custom designed and built board to make it all work.

Goals/Features

  • Calculate object positions like star software does
  • Point at and track objects
  • Implement USB for updating catalog list and advanced control
  • Make it easy to use

The PCB

Main Board Top

Main Board Top

I decided from the beginning that after having made several printed circuit boards using the toner transfer method, I wanted to make my senior project board this way. I designed the board using Cadsoft Eagle, a schematic and PCB layout tool available for free (with some restrictions). Along the way I realized that there was no way I was going to be able to route the board on a single layer, so I went ahead and tried my hand at a double layer board. Also, along with the main controller board, I had to design and build three other boards. There is what I call the sensor board, that has the GPS and the magnetic sensor on it that physically attaches to the telescope base, the hand-held remote control board that has the LCD and the buttons for i/o and the power board that connects to the battery and charger and has dual linear regulators for the servos and circuits. I had to use two regulators because the microcontroller would reset ever time the servos moved!

Main, Servo/GPS/Compass and Remote Control Boards

Main, Servo/GPS/Compass and Remote Control Boards

The dual layer PCB is actually two single layer PCBs strapped back to back because I was worried I would not be able to align the toner masks well enough on a real two layer board. This worked okay… they are essentially held together by all the vias that I put wire through and soldered on both sides as well as the component leads. I ran into difficulties with the connectors, the USB and the serial, because their leads were too short and I had to extend them with some wire, but they don’t feel very secure. Also, I had to do some various hacks and tricks to get eagle to do what I wanted. It automatically assumes that all leads can be soldered both on the top and bottom layers, even for things that have them completely covered on top like the USB connector, so I had to route those manually.

How it works

To calculate the position of a star in the sky, you must have three pieces of information: The current time, your current position (latitude & longitude) and the coordinate of the star. Also, you must be aligned with true north. In my system, I get time and position from the on-board GPS module. I store a list of stars in the micro controller memory that has their coordinates as right ascension and declination. Finally, I find true north by finding magnetic north using my on-board magnetic sensor and then add the magnetic declination for the area, which unfortunately is hard coded in the system right now since I have no way of getting it other than looking it up online. As it works right now, you simply put the scope down on a level surface and it finds north and adds that as an offset in the servo position so it does not actually have to be aligned with north, it does so automatically. To actually get the angles you need to point the servos, you must calculate sidereal time, find the hour angle, and finally convert from the equatorial coordinates to altitude and azimuth. For my system, since I somewhat stupidly decided to use servos for some reason, there is an extra step since the servos only go about 180 degrees. Since, both servos can go 180 degrees, you can point in all directions, but if you hit the limit on the azimuth for instance, it must rotate 180 degrees, while the altitude servo must also rotate to the opposite side to keep pointing in the same direction. This is kind of goofy but it does work. Also, I am not using any position feedback, and so my system is open loop. I calibrated it pretty well though and so it has not been too much of a problem.

Remote Control Board

Remote Control Board

The board also has USB that allows the user to control it and hopefully update the star catalog from a computer (although the updating does not work yet). It also has a serial port for testing that allows the user to look at the current state of the system as well as GPS data. I implemented a menu system for the user I/O that displays the object, its magnitude (brightness) and the current time. The magnetic sensor on board is a two axis sensor that I initially hoped I would be able to read the values and do a four quadrant arctan on two find the angle of magnetic north. This however proved to be difficult since the sensitivity in each axis seems to be different, and each time you turned it on, there was a different offset in the output. Since I only need to find north, I actually put the sensor on the rotating part of the scope itself and when you turn it on, it goes through a calibration routine that turns the bottom servo through 180 degrees and finds where the maximum field strength occurs. As long as the front side of the scope is within 180 degrees of north, it works.

Problems/Future Improvements

Schematic of Main Board

Schematic of Main Board

The main thing I would have done differently, and probably will since I plan on using this is not to use servos. I used high torque metal gear hobby servos because I knew how to use them and I thought they would be good enough for my purposes. Actually, they do work, but they are the main source of error in my system. Instead I would like to use just regular DC motors or steppers geared way down to get high torque and better positional accuracy with an optical encoder or just a potentiometer. Secondly, I probably would have chosen an Atmel controller instead of the PIC for many reasons… Finally, implementing the USB on my USB PIC was a huge pain in the ass and almost certainly not worth it. I should have just used one of these and been done with it. It provides a virtual COM device on the PC side, exactly like what I am doing with my PIC using the CDC device firmware from microchip, and you get regular serial I/O to your micro, all with no overhead on your micro.

Code

If any one wants to look at my code to find out how to do things like calculate sidereal time (kind of tricky), here it is (On Github)

Code: https://github.com/seiferteric/senior-project-aptt-code

Eagle Project: https://github.com/seiferteric/senior-project-aptt-eagle-pcb

Radio Modem!

Receiver + Transmitter

Receiver + Transmitter

Using an “iPod” FM Transmitter and a standard radio receiver with a headphone jack, I set out to make a simple radio modem. The idea is that I have here a transmitter and receiver and all I have to do is connect the transmitter to the audio out on one computer, and the receiver to the microphone or line-in jack on another, modulate the input to the transmitter and demodulate it on the other side to transmit binary data. Initially, I wanted to use AFSK (audio frequency shift keying) but, to get a working proof of concept, I decided to use Amplitude shift keying since I thought it would be easier to get off the ground.

To start, I wrote a separate program for the sending and receiving side so I could work the bugs out of both, then later I will integrate the two. For the transmitting side, I used the SDL library to create and play the output signal. The program will be a sort of one way “chat”, where the one user will type in a string, hit return and then it will be transmitted. The sending program loops through the string bit by bit and sends a tone for a 1 and silence for a 0. The period of each tone or silence is 1/(BAUD rate) and I arbitrarily chose the frequency of the tone to be 12kHz, and I initially set the BAUD rate slow, so that I could hear it and detect problems, but now I have it running well at 9600 BAUD. Both, the capture and playback operate at 48,000 samples/second, since this is the fastest (and native) speed of my sound card’s clock.

The receiving side uses the ALSA library for capture since SDL does not natively support audio capture. The program constantly reads in and process data from the sound card. It takes an average power over the last 5 samples and tries to discriminate between a 0 and 1. When the program first starts, it reads in 500 average power samples and uses this as the noise level. After that, every 1/BAUD seconds it tests the current average power, if it is greater than the noise offset + a (arbitrary for now) threshold, then it is a one, otherwise, it is a zero. The program is constantly shifting in bits until it reads a 0xAB, which is the start sequence. The next 2 bytes it reads in is taken as an unsigned 16 bit value (everything is lsb first) representing the length of the incoming message (so there is no stop sequence). After this value is read in, it reads in that many bytes and then prints the message on screen.

Receiver + Transmitter

Receiver + Transmitter

The programs for sending and receiving are a hack at the moment. Some problems I am having right now include a non-robust protocol (no error checking) and problems sending large files. On the receiving side, there are two possible errors that can occur. Misinterpreting a bit and getting out of sync by one bit. The first problem simply changes the bytes value, which could theoretically be handled if I had some error checking. The latter issue of getting off by one bit, causes chaos for the remainder of the message. Especially bad is if the length value at the beginning of the message is bad, then it tries to read in an arbitrarily large number of bytes. If the start sequence is missed, then the message is simply ignored. Also, the current setup is finicky about the volume of the input. For testing, I use an audio patch cable between the output and input of my computer, and I keep the volume around %54. When I use the radio, I have to play around with the volume knob a bit to find a good level for reliable transmitting.

Initially, the range of the transmitter was very short, 15 feet or less depending on the surroundings. I popped open the case and saw the “antenna” which was a coil printed on the PCB. At the end of the coil was a small through hole that I soldered on a copper wire antenna that was about 1/4 wavelength. This dramatically improved the range to several hundred feet! So you may want to try this if you are thinking about doing it.

Download the code + binaries here

This requires a Linux system with the SDL and ALSA development libraries to compile. I have compiled the sender program on Windows as well since it uses SDL, but it had some issues and did not work correctly. To get this working for you, it might take a bit of fiddling with the volume of the output, the radio and the recording level on the input. Also, it is important to get the tuner on the radio as close as possible. The way I tested it was to play some song over the radio since it was easier to tell if it was distorted by being slightly out of tune.

PID Motor Controller (Servo!)

PID Board Motor and Display/Control Knob

PID Board Motor and Display/Control Knob

This is a project I did for my control systems class. A potentiometer acts as the input and can be turned to indicate an angle from 0 to 180 degrees. The feedback potentiometer is mechanically linked (by a worm gear) to the motor. The control board uses five operational amplifiers to create a Proportional, Derivative and Integral controller.

The feedback from the potentiometer on the motor is subtracted from the input voltage to provide the proportional signal. The derivative and the integral of that signal is taken for the derivative and integral signals. Each of these signals is amplified and summed to create the output signal for the motor. Before it can be fed to the motor, the signal goes through a push-pull power amplifier to drive the motor since the op amp alone cannot provide enough current.

Motor & Controller Board

Motor & Controller Board

The whole setup was mounted on a Plexiglas sheet so that the output could be projected on an overhead projector. I connected a dowel on the end of the motor to point in the direction of the output. The motor gears and mounting I used were all taken from my old Lego Mindstorm kit. Two switches on the board allow you to disable the derivative and/or integral feedback to see the change in response. Each signal has an adjustable gain via a trimpot, to change the pole and zero locations of the system. The system is powered by two nine volt batteries two provide +/-9V for the opamps. In the end, this worked very well, just like the theory suggests! I was able to do things like increase the gain to move the system into an unstable region and it would begin oscillating out of control!

Back of Motor with Potentiometer and Control Knob

Back of Motor with Potentiometer and Control Knob