Programming on galvanist

Non-privileged Python Package Installs on OS X

Wed, 10 Dec 2014 04:35:00 +0000

Sometimes a virtualenv isn’t what you want. I often want my account to have a particular python package (like bpython), but I also like to keep things clean and compartmentalized. One answer is installing the package to a standard place within your user account’s home folder. Python is already setup for this, as you can see if you run python -m site.

sys.path = [
    '/Users/someuser',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
    '/Library/Python/2.7/site-packages',
]
USER_BASE: '/Users/someuser/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/someuser/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

You’ll see that python is looking for USER_BASE and USER_SITE directories in particular locations. Let’s create them:

Giving Your Ancient Robot Arm A Storage Upgrade

Thu, 09 Oct 2014 02:06:04 +0000

I’ve been hacking on a Giant Robot Arm at a great hackerspace called sudoroom. I’m doing so under the expert tutelage of an amazing genius or two.

The Enemy

Below you will see the YASNAC FC1 pictured. It is an incredibly slow of speed, minuscule of capacity, short of reliability, and large of footprint external floppy drive. It connects to the various robot brain parts by a standard RS-232 serial connection. That whole giant device is just the floppy drive.

json2csv

Sun, 15 Jun 2014 19:26:50 +0000

Do you have a JSON file, like the staticsitegenerators.net list of static site generator software, which you want to convert into CSV format? I created json2csv for just this purpose. Check it out.

More Web Scraping with Phantomjs & CoffeeScript

Thu, 20 Mar 2014 03:33:00 +0000

I’ve been playing with Phantomjs and CoffeeScript a bit more. In this example I load the first page of google news, extract all the links and link text, and write them to stdout in markdown format.

# headlines: generate markdown-formatted output with just the headlines
# and links from the main google news page (no sections, nothing fancy)
page_url = 'http://news.google.com/'
jquery_url = 'http://ajax.googleapis.com/ajax/libs/jquery/2.1.0/jquery.min.js'
page = require("webpage").create()
	
# this is a handler that relays console messages from inside our page to
# the phantomjs console. I've made it ignore everything that isn't
# prefixed with the word 'headline:'
page.onConsoleMessage = (msg) ->
    if msg.substring(0, 9) is "headline:"
        console.log(" - " + msg.substring(9))
	
	
page.open page_url, (status) ->
    if status isnt "success"
        console.log "# Error: couldn't load the page"
        phantom.exit()
    console.log "# The News\n"
    page.includeJs jquery_url, ->
        page.evaluate ->
            $('a.article').has('span.titletext').each (indx, elem) ->
                title = $(elem).text()
                href = $(elem).attr 'href'
                console.log "headline:[#{title}](#{href})"
        phantom.exit()  # I need this here to let the evaluate finish

Getting to know CoffeeScript

The combination of CoffeeScript an jQuery feels natural. But the overall program feels pretty brittle, and the phantomjs error messages that were printed while I was writing this little script weren’t very helpful. To be fair, part of this relates to CoffeeScript. I’m still getting used to it, and I’m in the phase where overlooked unbalanced parens or unintentionally ambiguous expressions cost me lots of time to debug.

Switch Statement Emulation in Python

Sat, 08 Mar 2014 02:39:00 +0000

I’ve typically enjoyed having fall through switch statements in my programming languages. Not everyone loves em. They’re not always the right answer and they can be a bit dangerous; you miss a “break” and you “goto fail”. But they are easy to read and easy to understand – for me the switch statement is about readability (and to a lesser extent reducing the number of times you have to type the switched variable).

Javascript Duration Seconds to Years, Days, Hours, Minutes, Seconds

Thu, 06 Mar 2014 06:24:00 +0000

For a recent project I needed to express the difference between date objects as a string. Here’s an algorightm I’ve been using for a while. This time in JavaScript. JavaScript isn’t my best language, so if you have any suggestions for improvement, please let me know.

function fmt_duration(seconds, detail) {
  // Return a string containing the number of years, days, hours,
  // minutes, and seconds in the given numeric seconds argument.
  // The optional detail argument can limit the about of detail.
  // Note: 1 year is treated as 365.25 days to approximate "leap
  // years" TAGS: secToYMDHMS, secToDHMS
  //
  // Some Examples:
  // 
  // fmt_duration(35000000)
  // returns "1 year, 39 days, 20 hours, 13 minutes, 20 seconds"
  //
  // fmt_duration(24825601)
  // returns "287 days, 8 hours, 1 second"
  //
  // fmt_duration(24825601, 3)
  // returns "287 days, 8 hours"
  //
  // fmt_duration(24825601, 1)
  // returns "less than one year"
  // 
  "use strict";
  var labels = ['years', 'days', 'hours', 'minutes', 'seconds'],
    increments = [31557600, 86400, 3600, 60, 1],
    result = "",
    i,
    increment,
    label,
    quantity;
  detail = detail === undefined ? increments.length : detail;
  detail = Math.min(detail, increments.length);
	
  for (i = 0; i < detail; i += 1) {
    increment = increments[i];
    label = labels[i];
	
    if (seconds >= increment) {
      quantity = Math.floor(seconds / increment);
      if (quantity === 1) {
        // if singular, strip the 's' off the end of the label
        label = label.slice(0, -1);
      }
      seconds -= quantity * increment;
      result = result + " " + quantity + " " + label + ",";
    }
  }
	
  result = result.slice(1, -1);
  if (result === "") {
    result = "less than one " + labels[detail - 1].slice(0, -1);
  }
	
  return result;
}

Javascript Date from Python datetime string representation

Thu, 06 Mar 2014 06:17:00 +0000

In a recent project I needed to convert the string representation of a Python datetime.datetime into a Javascript Date object. Here’s what I came up with. JavaScript isn’t my best language so if you have any suggestions for improvement, please let me know.

function datetimeToDate(datetime_str, use_utc) {
  // Return a JS Date object from the given python datetime_str string.
  // if the optional boolean use_utc argument is given, the datetime_str
  // will be treated as a UTC datetime_str
  "use strict";
  var datetime_parts = datetime_str.split(" ", 2),
    date_parts = datetime_parts[0].split("-"),
    time_parts = datetime_parts[1].split(":"),
    second_parts = time_parts[2].split('.'),
    mkdate,
    s2i,
    smu2ims;
  use_utc = use_utc === undefined ? false : use_utc;
	
  // The Date constructors don't work with .apply(), so lambdas
  if (use_utc) {
    mkdate = function (y, m, d, hr, min, sec, ms) {
      return new Date(Date.UTC(y, m, d, hr, min, sec, ms));
    };
  } else {
    mkdate = function (y, m, d, hr, min, sec, ms) {
      return new Date(y, m, d, hr, min, sec, ms);
    };
  }
	
  // string to int with radix 10
  s2i = function (i_str) { return parseInt(i_str, 10); };
	
  // string microsecond (aka mu) to int millisecond (aka ms)
  smu2ims = function (f_str) {
    return Math.round(parseFloat(f_str) * 1000);
  };
	
  return mkdate(s2i(date_parts[0]), // year
                s2i(date_parts[1]) - 1, // month 
                s2i(date_parts[2]), // day
                s2i(time_parts[0]), // hour
                s2i(time_parts[1]), // minute
                s2i(second_parts[0]), // second
                smu2ims("." + second_parts[1])); // millisecond
}

Fixing bpython's delete

Fri, 21 Feb 2014 00:21:00 +0000

A guy named Shironoo found one solution (which I read in english) to a problem I’d been having with bpython over SSH. Namely that using the delete key ruins the world, or at least kills the current line.

My slightly modified (temporary) solution is to make this change:

in bpython/cli.py:
- in def p_key(self, key):
  - replace if platform.system() == 'Windows':
  - with if platform.system() in ('Windows', 'Linux'):

You can find your bpython install with this kinda thing:

txtscan

Mon, 27 Jan 2014 22:56:12 +0000

I’ve created a small command-line program called “txtscan” for scanning the contexts of text files and reporting on the interesting characters and the line endings. It is useful for finding and identifying damage/garbage/exploit attempts in files that are basically just supposed to be ASCII.

“txtscan” is available on github.

wikitext

Thu, 23 Jan 2014 21:35:28 +0000

I’ve written a small command-line program that downloads the raw wikipedia markup text of the specified article. It does not download images or other supporting media. It does not follow links.

It is designed as a shortcut to using curl or wget in situations where you want article markup. For example, to get the raw markup for the “Abraham Lincoln” article, you’d normally run wget 'http://en.wikipedia.org/w/index.php?title=Abraham_Lincoln&action=raw'; that’s lame. Instead use wikitext, and just run wikitext "Abraham Lincoln" If you want the markup for the “Baberaham Lincoln” article, you’ll have to write your own program.

taskpaper2busycal

Mon, 13 Jan 2014 13:53:10 +0000

I’ve created a small command-line utility called taskpaper2busycal for converting TaskPaper todo lists into BusyCal events. It’s incredibly small and easy because it leverages BusyCal’s nice URL handlers and OS X’s “open” command. It’s nice when things just work. You can find it on github (at the link above).

"Delete Above" for Inkscape

Tue, 07 Jan 2014 11:37:20 +0000

Just before Christmas, I created an Inkscape extension called “Delete Above” to help solve a problem that I was having converting a print-formatted map into something more suitable for a GPS app. The extension makes it easier to delete items and everything that appears to be on top of them while preserving what appears to be below. You can find it and more information about how it works on github.

POSIX Testing my FUSE filesystem

Mon, 25 Nov 2013 17:41:00 +0000

I’m working away at my FUSE filesystem. I’ve found it incredibly helpful to run the Dawidek/Tuxera POSIX Test Suite on it. There are lots of interesting edge cases to account for. One helpful bit of info came from comp.file-systems.fuse.devel:

It seems that the options “allow_other, default_permissions, use_ino, attr_timeout=0, entry_timeout=0” fix this issue.

My Current Testing Status

My Thoughts On That

(Originally From gifrific.com)

OS X Quick Look Preview on the Command-line

Mon, 18 Nov 2013 20:56:41 +0000

The qlmanage command has a -p argument which lets you create Quick Look previews on the command line. So something like this can be nice to have in your shell profile: alias ql='qlmanage -p'.

I decided I needed something a little more featurey than an alias (because qlmanage -p README is useless). So, I made a quick python implementation of ql which has special treatment for README, LICENSE, Makefile and probably eventually other extension-less file names.

Revisiting Shell Concurrency (this time in ZSH)

Fri, 15 Nov 2013 23:08:00 +0000

I’ve been thinking about concurrency in the command shell again. This was prompted by my ongoing transition from Bash to ZSH. I’ve decided to re-implement my conc and xconc functions in a slightly different way.

Intro

Lets say you have 50 data files to compress. Here’s one way:

% xz *.dat
<< completed in 2 minutes, 4 seconds >>          
%

That’s not bad, but it doesn’t really take advantage of all those fancy cores we have in our computers these days. Let’s run multiple xz jobs as-parallel-as-possible, so that each file gets its own xz process.

Sort du intelligently

Tue, 22 Oct 2013 17:14:00 +0000

The du command is a nice tool for telling you the disk usage of your files:

$ du *
[...]
1928	system.log
136	system.log.0.bz2
112	system.log.1.bz2
0	uucp
552	vnetlib
[...]

I find that adding the -h flag (which produces more human-readable output) is often very helpful for me:

$ du -h *
[...]
964K	system.log
 68K	system.log.0.bz2
 56K	system.log.1.bz2
  0B	uucp
276K	vnetlib
[...]

Now if I could just sort that…

Solving The Secret Santa Selection "Problem"

Mon, 21 Oct 2013 20:05:00 +0000

My girlfriend’s family does a variation of secret santa for Christmas every year instead of the more traditional American “gift armageddon” strategy. I’m a fan because it encourages gift creativity and thoughtfulness while eliminating the crazy holiday consumerism that is prevalent here.

Every year the selection of giver-receiver pairs must be made and this can be a challenge because there are strict rules:

everyone gives once, everyone receives once
you can’t give to your partner
you can’t give a gift to someone who is giving a gift to you
it should be a secret until gift-giving time

Last year the selections were made by my girlfriend’s sister and Christmas was absolutely delightful. But then at some point afterward I opened my big fat mouth and said that a computer program should do the selections. The response was more or less “make it so.” Here’s my Pythonic attempt. For the web I’ve blanked out the PARTNERS map and my SMTP information. The rest is as I ran it.

Tail-Recursive Function Transformation in Python

Thu, 22 Aug 2013 19:57:00 +0000

Intro

In 1986, Hal Abelson and Gerald Jay Sussman recorded some videos for their M.I.T. EECS course on the Structure and Interpretation of Computer Programs. Twenty-seven years later, these videos are still a great resource. But the language featured is the Scheme¹ dialect of Lisp and while I can/do follow along in “edwin”, sometimes it’s nice to play in a (language (with (less (parens)))), like python.

One of the early challenges that I faced while trying to follow a scheme course using python was Scheme’s heavily tail-recursive programming idiom. Here’s a toy example from the videos where we define a Peano addition operation that only uses increment and decrement.

MapStack

Thu, 08 Aug 2013 01:02:30 +0000

In the San Francisco Bay Area, you have to write a bunch of code to even apply for I.T. jobs. It doesn’t bother me because there is no time pressure and I usually get to work in a language I enjoy on jobs that have tidy solutions.

The annoying part is that your work just goes into the bit-bucket, even if (lucky you) you get the job. You can’t just publish your solution because then the company would have to come up with a new code challenge. But if you’re using the proper level of abstraction, you should be able to publish the reusable bits of your code without spoiling their code challenge.

chroot jails on OS X

Wed, 31 Jul 2013 01:09:00 +0000

Recently my free time has mostly been dedicated to learning more about parallel computing. But I did take a little detour into the world of old-timey chroot(8) jails on OS X. They work, even on 10.8.4, even running a shell. The big trick on OS X is that you need to include the dynamic link editor /usr/lib/dyld in the jail.

Of course you also need to include the programs you want to run in the jail and their dependencies. The otool -L operation on OS X and the ldd command on linux are key to finding these dependencies. There are a lot, and you’ll need to recursively resolve them. Additionally on OS X, when you are recursively finding the dependencies, you’ll need to account for Mach-O’s @executable_path, @loader_path, and @rpath variables. This can get a little tricky. On Linux, the libraries are easier but things like PAM can be tricky. This is why you want software like jailkit, which can help automate the process of creating and running jails.

Python Reverse Enumerate

Fri, 21 Jun 2013 00:51:00 +0000

I want to save this thing I came across for future reference… a nice way to enumerate over an iterable in python in reverse order while retaining the proper indexes.

Here’s the issue. This code:

#!/usr/bin/python
    
WORDS = ['zero', 'one', 'two']

for index, string in enumerate(reversed(WORDS)):
    print index, string

… prints this:

0 two
1 one
2 zero

… which is “wrong” for our purposes. words[0] is not “two”, it is “zero”. So a quick web search for an elegant solution turned up a comment from “ΤΖΩΤΖΙΟΥ” on Christophe Simonis’ 2008 blog entry about reverse enumeration that has some code I really like:

Task Focus The Dock

Sat, 08 Jun 2013 04:52:00 +0000

The Problem

I was just thinking about my signal-to-noise ratio during coding and I noticed that my OS X dock has lots of stuff in it. I like this stuff. I want to keep this stuff. Also I hate a dock that hides. But I thought this: it would be nice to have an icon to click on that switches the dock between “only show running apps”-mode and normal mode.

Benchmarking Python Algorithms

Fri, 31 May 2013 00:39:00 +0000

I’m trying to figure out a fast way to compare the speed of different implementations of a single python algorithm. One option is the timeit module.

I’m not sure the best way to setup these comparisons, so I made a function for each implementation: divceil1 … divceil6. Then, I made a function to run each and check its output: test1 … test6. Finally, I’m running the test functions with timeit, e.g.: timeit.timeit(test6, number=5000). The output goes to STDOUT, which I’m redirecting to a CSV file, for processing in a spreadsheet tool. (I know, I know).

collect - a shell function

Fri, 24 May 2013 22:41:00 +0000

In OS X’s Finder, if you select some things then right-click you have an option called “New Folder with Selection”. ilikeitverymuch. So a while back, I though it would be alsonice to have something analogous in the command shell. I’ve been using just such a command for about six months now and let me assure you that it is very alsonice.

What does it do, under the hood?

Managed concurrency in the bash shell

Thu, 23 May 2013 07:14:00 +0000

Concurrency is all the rage these days. I started to wonder about using it in my command shell (which is regrettably still bash) because so much of the infrastructure for job control is already in place. I’ve been using xargs -P for a while, but I wondered about doing it all in the shell.

“conc”

I came up with a super-simple function to do some experiments:

export CONC_MAX=2
    
conc() {
	local procs=(`jobs -p`)
	local proc_count=${#procs[*]}
    
	# Block until there is an open slot
	if ((proc_count >= CONC_MAX)); then
		wait
	fi
    
	# Start our task
	(eval "$@") &
}
export -f conc

To use it, just throw the word ‘conc’ in front of your commands. It will only use up to CONC_MAX processes. I matched that to the number of cores I have. It is especially useful in loops. For example, while this does some things:

Some Saw-Sharpening Python Resources

Mon, 25 Feb 2013 07:04:01 +0000

Some Saw-Sharpening Python Resources

I’m coding more lately. This weekend I knocked off some rust using these resources on python hacking:

A “Hidden features of Python” question on Stack Overflow doesn’t fit the site well but the answers are so much geeky fun.
The Hitchhiker’s Guide to Python is a work in progress, but is already a valuable resource.
Starting a Django 1.4 Project the Right Way involves using git/hg, virtualenv, virtualenvwrapper, pip, south, and fabric.