Python on galvanist

Django SECRET_KEY generation

Wed, 13 May 2020 22:13:52 +0000

Generate django `SECRET_KEY`s on the command line using this “one-liner” (django not required)

Background

Django is a Python web framework with a lot of nice features including an exceptional ORM and high quality documentation. Most of my experience with django has been deploying and debugging other people’s projects - during the course of which I often want to regenerate the SECRET_KEY that underpins most of the security of every deployment.

The django settings docs describe the SECRET_KEY:

Better Documentation of Data with Markdown Tables

Sun, 28 Apr 2019 10:16:33 -0700

Documentation & Markdown Tables

Documentation is a vital part of almost any technical project. However, maintaining good documentation is hard for many reasons, one being that the time needed to document things is time you’re not spending on the “building / fixing / operating / assisting” aspects of your work – this is especially painful when you have lots of other urgent or important work to do.

Therefore, an important value of any documentation system is to make it as easy as possible for the would-be documenter to access, create, and edit content at any time. This includes enabling the documenter to express their ideas without requiring excessive authentication, cumbersome markup, or slow / complex UI interactions.

Non-privileged Python Package Installs on OS X

Wed, 10 Dec 2014 04:35:00 +0000

Sometimes a virtualenv isn’t what you want. I often want my account to have a particular python package (like bpython), but I also like to keep things clean and compartmentalized. One answer is installing the package to a standard place within your user account’s home folder. Python is already setup for this, as you can see if you run python -m site.

sys.path = [
    '/Users/someuser',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
    '/Library/Python/2.7/site-packages',
]
USER_BASE: '/Users/someuser/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/someuser/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

You’ll see that python is looking for USER_BASE and USER_SITE directories in particular locations. Let’s create them:

Giving Your Ancient Robot Arm A Storage Upgrade

Thu, 09 Oct 2014 02:06:04 +0000

I’ve been hacking on a Giant Robot Arm at a great hackerspace called sudoroom. I’m doing so under the expert tutelage of an amazing genius or two.

The Enemy

Below you will see the YASNAC FC1 pictured. It is an incredibly slow of speed, minuscule of capacity, short of reliability, and large of footprint external floppy drive. It connects to the various robot brain parts by a standard RS-232 serial connection. That whole giant device is just the floppy drive.

json2csv

Sun, 15 Jun 2014 19:26:50 +0000

Do you have a JSON file, like the staticsitegenerators.net list of static site generator software, which you want to convert into CSV format? I created json2csv for just this purpose. Check it out.

Switch Statement Emulation in Python

Sat, 08 Mar 2014 02:39:00 +0000

I’ve typically enjoyed having fall through switch statements in my programming languages. Not everyone loves em. They’re not always the right answer and they can be a bit dangerous; you miss a “break” and you “goto fail”. But they are easy to read and easy to understand – for me the switch statement is about readability (and to a lesser extent reducing the number of times you have to type the switched variable).

Javascript Date from Python datetime string representation

Thu, 06 Mar 2014 06:17:00 +0000

In a recent project I needed to convert the string representation of a Python datetime.datetime into a Javascript Date object. Here’s what I came up with. JavaScript isn’t my best language so if you have any suggestions for improvement, please let me know.

function datetimeToDate(datetime_str, use_utc) {
  // Return a JS Date object from the given python datetime_str string.
  // if the optional boolean use_utc argument is given, the datetime_str
  // will be treated as a UTC datetime_str
  "use strict";
  var datetime_parts = datetime_str.split(" ", 2),
    date_parts = datetime_parts[0].split("-"),
    time_parts = datetime_parts[1].split(":"),
    second_parts = time_parts[2].split('.'),
    mkdate,
    s2i,
    smu2ims;
  use_utc = use_utc === undefined ? false : use_utc;
	
  // The Date constructors don't work with .apply(), so lambdas
  if (use_utc) {
    mkdate = function (y, m, d, hr, min, sec, ms) {
      return new Date(Date.UTC(y, m, d, hr, min, sec, ms));
    };
  } else {
    mkdate = function (y, m, d, hr, min, sec, ms) {
      return new Date(y, m, d, hr, min, sec, ms);
    };
  }
	
  // string to int with radix 10
  s2i = function (i_str) { return parseInt(i_str, 10); };
	
  // string microsecond (aka mu) to int millisecond (aka ms)
  smu2ims = function (f_str) {
    return Math.round(parseFloat(f_str) * 1000);
  };
	
  return mkdate(s2i(date_parts[0]), // year
                s2i(date_parts[1]) - 1, // month 
                s2i(date_parts[2]), // day
                s2i(time_parts[0]), // hour
                s2i(time_parts[1]), // minute
                s2i(second_parts[0]), // second
                smu2ims("." + second_parts[1])); // millisecond
}

Fixing bpython's delete

Fri, 21 Feb 2014 00:21:00 +0000

A guy named Shironoo found one solution (which I read in english) to a problem I’d been having with bpython over SSH. Namely that using the delete key ruins the world, or at least kills the current line.

My slightly modified (temporary) solution is to make this change:

in bpython/cli.py:
- in def p_key(self, key):
  - replace if platform.system() == 'Windows':
  - with if platform.system() in ('Windows', 'Linux'):

You can find your bpython install with this kinda thing:

txtscan

Mon, 27 Jan 2014 22:56:12 +0000

I’ve created a small command-line program called “txtscan” for scanning the contexts of text files and reporting on the interesting characters and the line endings. It is useful for finding and identifying damage/garbage/exploit attempts in files that are basically just supposed to be ASCII.

“txtscan” is available on github.

wikitext

Thu, 23 Jan 2014 21:35:28 +0000

I’ve written a small command-line program that downloads the raw wikipedia markup text of the specified article. It does not download images or other supporting media. It does not follow links.

It is designed as a shortcut to using curl or wget in situations where you want article markup. For example, to get the raw markup for the “Abraham Lincoln” article, you’d normally run wget 'http://en.wikipedia.org/w/index.php?title=Abraham_Lincoln&action=raw'; that’s lame. Instead use wikitext, and just run wikitext "Abraham Lincoln" If you want the markup for the “Baberaham Lincoln” article, you’ll have to write your own program.

taskpaper2busycal

Mon, 13 Jan 2014 13:53:10 +0000

I’ve created a small command-line utility called taskpaper2busycal for converting TaskPaper todo lists into BusyCal events. It’s incredibly small and easy because it leverages BusyCal’s nice URL handlers and OS X’s “open” command. It’s nice when things just work. You can find it on github (at the link above).

"Delete Above" for Inkscape

Tue, 07 Jan 2014 11:37:20 +0000

Just before Christmas, I created an Inkscape extension called “Delete Above” to help solve a problem that I was having converting a print-formatted map into something more suitable for a GPS app. The extension makes it easier to delete items and everything that appears to be on top of them while preserving what appears to be below. You can find it and more information about how it works on github.

OS X Quick Look Preview on the Command-line

Mon, 18 Nov 2013 20:56:41 +0000

The qlmanage command has a -p argument which lets you create Quick Look previews on the command line. So something like this can be nice to have in your shell profile: alias ql='qlmanage -p'.

I decided I needed something a little more featurey than an alias (because qlmanage -p README is useless). So, I made a quick python implementation of ql which has special treatment for README, LICENSE, Makefile and probably eventually other extension-less file names.

Revisiting Shell Concurrency (this time in ZSH)

Fri, 15 Nov 2013 23:08:00 +0000

I’ve been thinking about concurrency in the command shell again. This was prompted by my ongoing transition from Bash to ZSH. I’ve decided to re-implement my conc and xconc functions in a slightly different way.

Intro

Lets say you have 50 data files to compress. Here’s one way:

% xz *.dat
<< completed in 2 minutes, 4 seconds >>          
%

That’s not bad, but it doesn’t really take advantage of all those fancy cores we have in our computers these days. Let’s run multiple xz jobs as-parallel-as-possible, so that each file gets its own xz process.

Sort du intelligently

Tue, 22 Oct 2013 17:14:00 +0000

The du command is a nice tool for telling you the disk usage of your files:

$ du *
[...]
1928	system.log
136	system.log.0.bz2
112	system.log.1.bz2
0	uucp
552	vnetlib
[...]

I find that adding the -h flag (which produces more human-readable output) is often very helpful for me:

$ du -h *
[...]
964K	system.log
 68K	system.log.0.bz2
 56K	system.log.1.bz2
  0B	uucp
276K	vnetlib
[...]

Now if I could just sort that…

Solving The Secret Santa Selection "Problem"

Mon, 21 Oct 2013 20:05:00 +0000

My girlfriend’s family does a variation of secret santa for Christmas every year instead of the more traditional American “gift armageddon” strategy. I’m a fan because it encourages gift creativity and thoughtfulness while eliminating the crazy holiday consumerism that is prevalent here.

Every year the selection of giver-receiver pairs must be made and this can be a challenge because there are strict rules:

everyone gives once, everyone receives once
you can’t give to your partner
you can’t give a gift to someone who is giving a gift to you
it should be a secret until gift-giving time

Last year the selections were made by my girlfriend’s sister and Christmas was absolutely delightful. But then at some point afterward I opened my big fat mouth and said that a computer program should do the selections. The response was more or less “make it so.” Here’s my Pythonic attempt. For the web I’ve blanked out the PARTNERS map and my SMTP information. The rest is as I ran it.

Tail-Recursive Function Transformation in Python

Thu, 22 Aug 2013 19:57:00 +0000

Intro

In 1986, Hal Abelson and Gerald Jay Sussman recorded some videos for their M.I.T. EECS course on the Structure and Interpretation of Computer Programs. Twenty-seven years later, these videos are still a great resource. But the language featured is the Scheme¹ dialect of Lisp and while I can/do follow along in “edwin”, sometimes it’s nice to play in a (language (with (less (parens)))), like python.

One of the early challenges that I faced while trying to follow a scheme course using python was Scheme’s heavily tail-recursive programming idiom. Here’s a toy example from the videos where we define a Peano addition operation that only uses increment and decrement.

The Birdometer v0.1a

Wed, 14 Aug 2013 02:18:02 +0000

The Birdometer v0.1a

Introduction

There’s an old Russian proverb: “Make a birdometer, it will be really fun.”

So I did.

My amazing girlfriend and I like birds. We have some hummingbird feeders. We have two Arduinos. We want to start taking data about our bird guests. So, we bought some sensors from the amazing Al Lasher’s Electronics Store in Berkeley. Then we made our “birdometer” by connecting an arduino to a passive infrared sensor, a small microphone, a Canon EOS 5D camera, and a PC. We then wrote some code to make the arduino trigger the camera shutter when the infrared sensor detects motion. We are also recording audio samples of the birds from the mic and sending them to the PC via USB.

MapStack

Thu, 08 Aug 2013 01:02:30 +0000

In the San Francisco Bay Area, you have to write a bunch of code to even apply for I.T. jobs. It doesn’t bother me because there is no time pressure and I usually get to work in a language I enjoy on jobs that have tidy solutions.

The annoying part is that your work just goes into the bit-bucket, even if (lucky you) you get the job. You can’t just publish your solution because then the company would have to come up with a new code challenge. But if you’re using the proper level of abstraction, you should be able to publish the reusable bits of your code without spoiling their code challenge.

Arduino & OS X Notes

Mon, 05 Aug 2013 23:25:46 +0000

My girlfriend and I spent last weekend building Arduino-controlled devices. Here are some random notes about working with Arduino (in an OSX environment)

splurge on a long USB device cable. also a breadboard and jumper wires
use stty to change baud & other settings on a device file - having access to the usual unix tools e.g. cat /dev/ttyUSB0 is a really nice thing
pyserial is always your friend. seriously it is fantastic.
the official Arduino docs are really good
don’t miss the docs for your specific model, for example, these are the specs for my old-timey diecimila
don’t miss the part about Arduino ints being 16-bits. Arduinos are little-endian but it rarely comes into play. the ADC produces 10-bit values.
this sparkfun article about using Arduino with relays will be useful to me soon
the OS X drivers for the FTDI chip on old Arduinos like mine are terrible. I’ve had 6 kernel panics in the last three days. i finally uninstalled them. luckily ubuntu has native support, so I currently program my old Arduino on the linux machine. My girlfriend’s Arduino uno doesn’t use this chip and working with it on OS X is fantastic.
the digital pin’s output mode high/low is enough to operate the coil in small low-voltage relays like this
this guy is doing interesting things with Arduino data acquisition
now i want one of these.

chroot jails on OS X

Wed, 31 Jul 2013 01:09:00 +0000

Recently my free time has mostly been dedicated to learning more about parallel computing. But I did take a little detour into the world of old-timey chroot(8) jails on OS X. They work, even on 10.8.4, even running a shell. The big trick on OS X is that you need to include the dynamic link editor /usr/lib/dyld in the jail.

Of course you also need to include the programs you want to run in the jail and their dependencies. The otool -L operation on OS X and the ldd command on linux are key to finding these dependencies. There are a lot, and you’ll need to recursively resolve them. Additionally on OS X, when you are recursively finding the dependencies, you’ll need to account for Mach-O’s @executable_path, @loader_path, and @rpath variables. This can get a little tricky. On Linux, the libraries are easier but things like PAM can be tricky. This is why you want software like jailkit, which can help automate the process of creating and running jails.

Python Reverse Enumerate

Fri, 21 Jun 2013 00:51:00 +0000

I want to save this thing I came across for future reference… a nice way to enumerate over an iterable in python in reverse order while retaining the proper indexes.

Here’s the issue. This code:

#!/usr/bin/python
    
WORDS = ['zero', 'one', 'two']

for index, string in enumerate(reversed(WORDS)):
    print index, string

… prints this:

0 two
1 one
2 zero

… which is “wrong” for our purposes. words[0] is not “two”, it is “zero”. So a quick web search for an elegant solution turned up a comment from “ΤΖΩΤΖΙΟΥ” on Christophe Simonis’ 2008 blog entry about reverse enumeration that has some code I really like:

Task Focus The Dock

Sat, 08 Jun 2013 04:52:00 +0000

The Problem

I was just thinking about my signal-to-noise ratio during coding and I noticed that my OS X dock has lots of stuff in it. I like this stuff. I want to keep this stuff. Also I hate a dock that hides. But I thought this: it would be nice to have an icon to click on that switches the dock between “only show running apps”-mode and normal mode.

Benchmarking Python Algorithms

Fri, 31 May 2013 00:39:00 +0000

I’m trying to figure out a fast way to compare the speed of different implementations of a single python algorithm. One option is the timeit module.

I’m not sure the best way to setup these comparisons, so I made a function for each implementation: divceil1 … divceil6. Then, I made a function to run each and check its output: test1 … test6. Finally, I’m running the test functions with timeit, e.g.: timeit.timeit(test6, number=5000). The output goes to STDOUT, which I’m redirecting to a CSV file, for processing in a spreadsheet tool. (I know, I know).

PIL/PIP/Homebrew Problems

Wed, 24 Apr 2013 22:42:00 +0000

I have PIL installed via pip. I have libjpeg installed via homebrew. I have the system python.

I was having problems with PIL’s jpeg module. It was giving the error message: “ImportError: The _imaging C module is not installed”. It was installed, but it was broken. Telling python to import _imaging gave me the better message: “Symbol not found: _jpeg_resync_to_restart”.

So I had a look: dyldinfo -dylibs /Library/Python/2.7/site-packages/PIL/_imaging.so and saw this:

The VAX MTH$RANDOM Generator in Python

Tue, 23 Apr 2013 15:36:00 +0000

So I needed the ole’ VAX MTH$RANDOM for a puzzle. I like python. Here’s what I came up with after reading some gsl docs:

def vax_rand(seed):
    a = 69069
    c = 1
    m = 2**32
    while True:
        seed = divmod(a * seed + c, m)[1] 
        yield seed

To use it (with a seed of 6), do this: v = vax_rand(6) and to get a number out of it, you could do print v.next().

Some Saw-Sharpening Python Resources

Mon, 25 Feb 2013 07:04:01 +0000

Some Saw-Sharpening Python Resources

I’m coding more lately. This weekend I knocked off some rust using these resources on python hacking:

A “Hidden features of Python” question on Stack Overflow doesn’t fit the site well but the answers are so much geeky fun.
The Hitchhiker’s Guide to Python is a work in progress, but is already a valuable resource.
Starting a Django 1.4 Project the Right Way involves using git/hg, virtualenv, virtualenvwrapper, pip, south, and fabric.

VectorDesigner: The Resurrection

Sat, 15 Dec 2012 06:11:00 +0000

The Problem: VectorDesigner Dead

A few years ago I bought a vector drawing app called VectorDesigner and then I used it to make a bunch of stuff. Beautiful. Stuff. Then VectorDesigner and it’s developer kinda disappeared from the Internet. Time moved on, MacOS 10.7 and 10.8 came out, and the old VectorDesigner app that I still have on my Mac stopped working. It would just fail to launch, bouncing in the dock for a few moments then sagging into an apparent coma without ever showing a window or menubar. This new behavior meant that the many works of beautiful stuff that I lovingly crafted in VectorDesigner were completely inaccessible to me.

Easy JPEG Transcoding

Fri, 07 Oct 2011 07:00:00 +0000

2011-10-07 : Amundsen-Scott South Pole Station

The Independent JPEG Group is responsible for a C-based JPEG library that is used by lots of software. Their library distribution also includes the very useful binary executables “cjpeg” and “djpeg”. When combined, they can be used to transcode JPEGs. I’ve used these programs gainfully to transcode JPEGs to 14% of their original size with no visible change in picture quality.

To prove the point, I’m including the difference between the original seven megabyte version and the transcoded version. A completely black image means there are zero differences. In this one you might be able to make out faint shades of color:

Disjointed & Rebound

Fri, 16 Sep 2011 07:00:00 +0000

2011-09-16 : Amundsen-Scott South Pole Station

So I’ve been at the South Pole for a while. In fact, it has almost been a year now. It has been a challenging winter in some ways and I’m feeling more than a little “toasty”. I’m not sure what else to say about it, so I’m going to write about some other stuff…

I’m switching from Fraise to MacVim. I’m also changing shells from bash to zsh. I’m changing from MacFuse to OSXFuse. I’m sticking to Firefox for most of my browsing (NoScript and AdBlock Plus are entirely indispensable). I’m giving GPGTools a try. Boxee has been really nice to have in my berthing room. I’ve never been a big fan of GNU screen, but I’m interested in tmux.

Python on galvanist

Django SECRET_KEY generation

Generate django SECRET_KEYs on the command line using this “one-liner” (django not required)

Background

Better Documentation of Data with Markdown Tables

Documentation & Markdown Tables

Non-privileged Python Package Installs on OS X

Giving Your Ancient Robot Arm A Storage Upgrade

The Enemy

json2csv

Switch Statement Emulation in Python

Javascript Date from Python datetime string representation

Fixing bpython's delete

txtscan

wikitext

taskpaper2busycal

"Delete Above" for Inkscape

OS X Quick Look Preview on the Command-line

Revisiting Shell Concurrency (this time in ZSH)

Intro

Sort du intelligently

Solving The Secret Santa Selection "Problem"

Tail-Recursive Function Transformation in Python

Intro

The Birdometer v0.1a

The Birdometer v0.1a

Introduction

MapStack

Arduino & OS X Notes

chroot jails on OS X

Python Reverse Enumerate

Task Focus The Dock

The Problem

Benchmarking Python Algorithms

PIL/PIP/Homebrew Problems

The VAX MTH$RANDOM Generator in Python

Some Saw-Sharpening Python Resources

Some Saw-Sharpening Python Resources

VectorDesigner: The Resurrection

The Problem: VectorDesigner Dead

Easy JPEG Transcoding

Disjointed & Rebound

Generate django `SECRET_KEY`s on the command line using this “one-liner” (django not required)