Tools

Script of the week: linkdups

It's been a while since I've posted a script; life has been distracting lately. I also wanted to let this current script mature a lot more before sharing it, as it has the potential to be destructive. Use wisely!

It's name is linkdups, and it's a Python program to recursively walk through a directory tree and hard-links any files together whose contents match exactly. That means that if you have two files, each taking up 10 Kb, afterwards they will be linked to the same contents for a total savings of 10 Kb. Read More...
|

Script of the week: verify

This week's script uses Leopard's new "xattr" tool to store MD5 checksum information alongside any files you wish. Later, you can run the same script to ensure that this checksum has not changed. Read More...
|

Script of the week: redirect

This week’s script of the week is so simple, it doesn’t really deserve to be called a script. But since it’s highly useful and comes as a surprise to many people that it can be done so easily, here it is.

The purpose of this script is to create momentary TCP routes. TCP routing is also called Layer 4 routing. That is, one machine momentarily serves as a transparent gateway between two TCP ports on two other machines. The advantages to layer 4 routing are:

  1. It’s “port to port” (you aren’t opening up subnets to each other, or even whole machines).
  2. It doesn’t require complicated routing tables entries, or IP forwarding, or NAT.
  3. It can be done entirely in user space. No strange kernel drivers required!

Here’s an example: Let’s say you use a web server sitting on a private network which you access over VPN. You can see the server just fine by typing it’s address in your web browser. One day, however, you find a bug on the server, but it only happen on that server like for your friend — who knows about such servers — to see what’s happening, but you obviously can’t grant him access to your secured network.

What would be really cool is if your friend could connect to your machine instead, and have your machine transparently proxy the connection into the VPN and over to that web server. It would also proxy responses back, so that from your friend’s point of view: your machine becomes the web server for as long as you keep the link up.

Here’s the command to do this, assuming I expose port 8080 on my machine for my friend to connect to, and I’m linking him to port 80 on the VPN’s web server:

$ tcpserver <MY-PUBLIC-IP> 8080 nc <VPN-WEB-SERVER-IP> 80

Did I mention that this doesn’t even require root privileges to work?

Note: If you have the socat utility installed, things get even simpler. In that case, the above command is just this:

$ socat tcp-listen:8080 tcp:<VPN-WEB-SERVER-IP>:80

Now you have a transparent route from port 8080 on your machine to your secured web server. After your friend is done checking things out, just cancel the command and the tunnel is destroyed. This is the best way I can think of to temporary and easily create transparent tunnels into otherwise inaccessible networks.

For this scriptlet to work, you’ll need ucspi-tcp installed (for the tcpserver command), and netcat, which comes pre-installed on OS X 10.5.

|

Script of the week: bzdmg

I haven't written much this past week because I've been upgrading all the home's machines to Leopard. So far it's gone very smoothly, and I like the new OS!

The script for this week is about disk images... Read More...
|

Script of the week: sizes

For the next few months, I will have a "script of the week" each week: just some tiny little scripts I've developed over the years that I happen to find particularly useful.

Today's is a shell script called sizes. It's a fairly simplistic interface to the du commands which just shows you all files and directories in the current directory that are larger than one megabyte. Read More...
|

Serving up Mercurial using mod_python

The following article resulted from several hours of battling with SELinux and Apache, attempting to find some way of serving up my Mercurial repository (now at http://hg.newartisans.com) over HTTP. Now I'm happy to bring you the fruits of that research, even though I'm still getting errors from Mercurial when trying to push (I'm using ssh at the moment). More to come on that front later... Read More...
|

Applescript and UTF-8 arguments

Today's brief article describes how to pass UTF-8 arguments to an Applescript from the command-line. If you've ever tried saying "olé" to osascript before, you'll know what I mean. Read More...
|

OpenSSH connection mastering

I just discovered a very cool feature of SSH today: control mastering. It lets you multiplex a single ssh connection so you don't have to open multiple TCP connections to the remote host; instead, all your SSH/SCP commands "share" the initial connection. This speeds up subsequent connections to the same host, and also means you don't have to enter your password more than once for hosts who don't know your public key yet. I use this feature to implement a script for setting up new remote accounts. Read More...
|

Stateful directory scanning in Python

About a half year ago I wrote a little Python module for myself to do "stateful" directory scans. This means keeping watch on the state of a directory so that you can act on changes, like files added or removed, files changed, etc. Now that I've been using this library every hour for that entire period -- with only a few minor bug fixes to cover some exceptional cases -- I believe that version 1.0 is ready for consumption. Today's article reviews the structure of this module and how to use it in your own, since I designed it with the full of intention of others being able to use it with their own scripts. Read More...
|

An SVK primer

Today’s entry is a little primer I wrote for some co-workers at CEG, on setting up SVK to do remote development. We continue to use a central Subversion repository, but I often find myself working in cafés where I don’t have immediate access to the server. Also, I like to branch and check-in much more frequently than would be sane to do with Subversion — I also like the distinction between a “check-in” being a simple, quick snapshot, and an "svk push" as the real deal. Read More...
|
© 2008 John Wiegley