Category Archives: File formats

Programatic submission of Australia Post’s CN23 customs form

A number of major international destinations of packages now will only accept packages with electronic CN23 customs declaration. Normally, you’d do this by rocking up to the Post Office with your pre-addressed parcel, filling in a CN23 paper form, and have that transcribed into Australia Post’s computer system by the postal worker behind the counter. You can elect to receive SMS notifications of change of status (landed, delivered, etc) for 50c.

Australia Post also allows you to fill in the appropriate details on their website; if you do this, then you get a QR code sent to you via SMS (free) and email (free) which the postal worker scans in and all the details (your name and address, destination name and address, contents, etc) are attached to your package’s details without any error-prone re-keying. The downside of going down this path is the dismal website Aussie Post provides, a JavaScript heavy, painfully slow dog of a site that doesn’t cache your own address.

Once the QR code is scanned, and the postal worker checks everything with you, they’ll print out the CN23, get you to sign it , and then it gets attached to your parcel. Because the To and From addresses are on the CN23 form (and those details are in electronic form, associated with the barcode for the package), it’s perfectly acceptable to present an unaddressed package to the post office (make sure you can tell which package is which, if you go down this route).

One thing you need to be aware of: Australia Post hasn’t heard of Unicode. You absolutely can’t use any characters not in the ASCII character set, and even then a very limited range of them. Certain fields allow some characters, which in turn aren’t allowed in other fields.

One of the fields you can supply is the HS tariff code, which is an international standard group of codes to describe “stuff” – the Harmonised System Tariff code. The sourcecode below uses the code for “Toy, plastic construction” – you should use the code for what you’re actually sending. You can specify multiple HS codes. Dollar values are in decimal dollars, weights are in decimal kilograms.

After calling the Australia Post website with your customs declaration, it returns to you a base-64 encoded PNG of the QR code to present at the counter, and a base-64 encoded PDF of the CN23 form – there’s no point printing this out, because it’s not paid for yet; let the Post Office print it out with the postage on it. You’ll also get the PNG via email and SMS (free).

Here’s some Python to make this submission:


    AP_session = requests.Session()
    jsonFormData =  {"customDeclaration":{
      "label":{"source":"AEM","postagePaidIndicator":False,"eadIndicator":False},
      "parcelCharacteristics":{
        "productClassification":11,
        "dangerousGoodsIndicator":False,
        "returnInstructions":"Return By Most Economical Route",
        "confirmationMobileNumber":"0411111111",
        "content":[{
          "content":"HS traffic code name for your stuff",
          "contentQuantity":1,
          "contentUnitValue":subtotal,
          "totalContentValue":subtotal,
          "contentWeight":int(order["total_weight"])/1000,
          "hsTariff":"95030039",
          "contentCountryOfOrigin":"DK"
          }],
        "totalConsignmentValue":subtotal},
      "senderAddress":{"firstName":"Josh","lastName":"FromGeekrant.org",
        "addressLine":["11 Example St"],"suburb":"YourSuburbName","state":"VIC",
        "postcode":"3000","email":"addr@example.com",
        "phone":"0411111111","smsConfirmation":False,"countryCode":"AU"},
      "receiverAddress":{"firstName":CustomsString(order["label_address_name_first"]),
        "lastName":CustomsString(order["label_address_name_last"]),
        "countryCode":order["label_address_two_char_country_code"],
        "addressLine":CustomsAddress(order),
        "suburb":CustomsString(order["label_address_city"]),
        "state":CustomsString(order["label_address_state"]),
        "postcode":CustomsString(order["label_address_postal_code"]),
        "email":order["buyer_email"]}
    }}

    stopact = {"jsonFormData":json.dumps(jsonFormData) }
    result = AP_session.post(url='https://auspost.com.au/bin/form/stopact', 
      data=stopact, timeout=2)
    response = json.loads(result.text)
    result.raise_for_status()
    filename = "{}-customsQRcode.png".format(orderid)
    with open(filename, "wb") as fh:
      fh.write(base64.b64decode(response['qrCode']))
    filename = "{}-CN23.pdf".format(orderid)
    with open(filename, "wb") as fh:
      fh.write(base64.b64decode(response['label']))

Clone to a bigger drive, and convert MBR to GPT

I wanted to partly upgrade Windows to a new drive.

Currently, Windows itself and Program Files are on C: drive, which is an SSD (which I meant to blog about in detail, but never got around to) and documents are on D: drive (which was the tricky bit of the SSD upgrade — to do it properly involves using SysPrep with an Unattend.xml configuration file that tells Windows that documents will live on D: not C:. This article describes it in detail.

Anyway that’s really irrelevant to the problem at hand, which is that D: drive had run out of space. Here’s a brief description of what I did:

  • The new drive is a 4 Tb drive, replacing a 1 Tb drive.
  • Plug the new drive in, use Clonezilla to clone the old D: onto the new drive. Following the detailed instructions, this all went pretty smoothly.
  • But… the catch is the old drive was formatted in MBR, which has a limitation of 2 Tb. For beyond that, you need GPT.
  • I looked around for tools to convert the drive. It’s easy if you’re prepared to wipe it, but I wanted to preserve the data I’d just moved across. Finding ways to do it without wiping everything was tricky, but I settled on the free version of Minitools Partition Wizard — this has an easy-to-understand interface, and did the job
  • Once that MBR is converted to GPT, you can enlarge the partition to make the whole drive available.
  • Unplug the old drive, move the new one into the same slot as the old (this is on a Mac Pro booting in Windows Bootcamp) and it works. Done!

PS. Similar exercise afterwards shuffling the OS X partition from a 320 Gb drive to the old 1 Tb. That required GParted, as it seems the GPT partition couldn’t be expanded due to a formatting issue (which GParted helpfully offered to fix as it started up) and another small 600 Mb partition being in the way — not sure what it is, but it seems to be essential for booting OS X — GParted was able to move it to the end of the disk.

Replace a missing remote control with an Arduino and a laptop

I recently found myself without a remote for my WDTV Live media player, and limited resources to do anything about it – but I did have an Arduino, a breadboard and the local Jaycar had an IR LED.  Controlling IR devices is common practice with an Arduino. I would even be able to hack in functions that didn’t exist on the manufacturer’s remote – like creating a three minute skip by switching to 16x speed for 12 seconds.

The first port of call was to obtain Ken Shirriff’s Arduino IR remote control protocol library – as opposed to communications protocols, of which there are quite a number; did you know the first cut of WiFi included an infrared version? Without the remote, I wasn’t able to record and playback the IR signals sent to the WDTVLive, as you would with a learning remote. I had to find what to transmit from my custom remote. I little googling and I found the WD TV Live infrared remote control codes, which also helpfully reveals that the protocol is NEC.

I knocked up a quick proof of concept, installed it and watched it not work. Given I can’t see in infrared, I didn’t know if my circuit was working. I hooked a red LED up in parallel, and it didn’t light up; I thought I had cathode and anode swapped around, so flipped the red LED – and it didn’t light up. I pulled the IR LED, and then the red LED worked… I was shorting out the red LED. I couldn’t – with the bits I had lying around – confirm the device was transmitting anything. Rather than put the LEDs in series, I got a cheap camera-phone with video function, and it could see IR just fine. And it turns out the IR LED was transmitting something, but the WD TV Live media player wasn’t listening. Why?

The NEC infrared control protocol transmits 32 bits in one of two formats, one old (as in elderly) format encodes for 256 devices with 256 commands each, and the other encodes for ~64K devices with 256 commands each. The first 16 bits encode the device, and the second 16 bits encode the command. 16 bits for one of 256 commands, you ask? Well, one byte of the second 16 bits is the command, and the other is – for error checking purposes – the one’s complement of that. Further details of the pulse timing and protocol contents are available in various places, but they neglect to mention the extended addressing format. There are many IR control protocols. To use Ken’s IR library you need to know which protocol is used (which the google search revealed), and you can determine the protocol from the timing data found in the LIRC definition of a protocol, in this case the LIRC infrared control protocol for WDTV Live media player remote. The LIRC protocol defintion format is described by WinLIRC, so you can see what the timings are. In this case, the NEC protocol is revealed by the header, one and zero definitions, along with the fact that each code has 16 bits of ‘pre-data’ and 16 bits of data (a 32 bit package). Everything I could see was showing that the two, separately arrived at sets of command codes that were empirically sampled from the real world were compliant with the spec. One of the things the spec taught me was to transmit the NEC code twice, and to wait 70ms between re-transmissions.

I wasted time finding other codes for the remote, in other formats; I checked for byte ordering issues. Nothing worked.

The actual problem was the unsigned long for the command was previously an int; failing to notice this simple error led me to spend a long time trying to figure out why nothing was happening when I transmitted a command. One of the problems with the C language is the guarantees about data sizes aren’t worth much.  My entire life has been spent programming on architectures that have 32 bit data words; C compilers on these machines have all defined an int as 32 bits, but I’ve always been aware that the language spec says that an int is at least as wide as a short, which is at least as wide as a char with actual widths being up to the compiler implementation (although why you’d have different words for things of the same size is beyond me).  The AVR microcontroller in question has an 8 bit word; mathematical operands typically yield an 8 bit result (multiply is an exception) with compilers needing to implement more instructions to yield greater data widths. The defines express the codes as four byte values, which were then wrangled into a two byte int, and then again into unsigned four byte integer when passed to the IR library. Truncated bits in a protocol like this were the cause of inactivity.

Even with this fundamental problem solved, confusion was added by the fact that one of the memory cells in my Arduino is faulty. Once IR control code transmission was working, I noticed that sometimes it didn’t work. I decided to echo the command to the serial port, and the command being transmitted didn’t match that for the key pressed – the second byte was wrong. I added code to work around this memory corruption (not shown in the code below, because this is a pretty unusual). I’ve never come across this kind of problem before, recognising and then solving something like that is pretty old-school.

/*
Pin 3 is hard-wired into the IR library as the emitter
 */
#include <IRremote.h>
//#define DEBUG

IRsend irsend;

#define btn_enter  0x219E10EF
#define btn_right  0x219E906F
#define btn_left   0x219EE01F
#define btn_down   0x219E00FF
#define btn_up     0x219EA05F
#define btn_option 0x219E58A7
#define btn_back   0x219ED827
#define btn_stop   0x219E20DF
#define btn_rew    0x219EF807
#define btn_ff     0x219E7887
#define btn_play   0x219E50AF
#define btn_prev   0x219E40BF
#define btn_next   0x219E807F
#define btn_eject  0x219E08F7
#define btn_search 0x219EF00F
#define btn_home   0x219E609F
#define btn_power  0x219E48B7

// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
const int onboard_led = 13;
const int retransmit=2;
unsigned long play_after=0;

void setup()
{
  pinMode(3, OUTPUT);     
  pinMode(onboard_led, OUTPUT);     
  Serial.begin(9600);
  Serial.println("WDTV Live serial controlled IR remote");
  Serial.println("~ Power    Eject ^ & Search   Rew - + FF");
  Serial.println("  w         Back q e Enter   Play  P");
  Serial.println("a s d (Arrows)     x Stop    Last < > Next");
  Serial.println("3 - FastForward three minutes");
}

void loop() {
  unsigned long cmd=0;
  if (Serial.available()) {
    switch (Serial.read()) {
      case 'E':
      case 'e':
      case ')':
      case '0':
      case 'O':
      case 'o': cmd=btn_enter; break;
      case 'q':
      case 'Q': cmd=btn_back; break;
      case 'P':
      case 'p':
      case ' ': cmd=btn_play; break;
      case 'S':
      case 's': cmd=btn_down; break;
      case 'W':
      case 'w': cmd=btn_up; break;
      case 'A':
      case 'a': cmd=btn_left; break;
      case 'D':
      case 'd': cmd=btn_right; break;
      case '-':
      case '_': cmd=btn_rew; break;
      case '=':
      case '+': cmd=btn_ff; break;
      case ',':
      case '< ': cmd=btn_prev; break;
      case '.':       
      case '>': cmd=btn_next; break;
      case '/':
      case '?': cmd=btn_option; break;
      case '~': cmd=btn_power; break;
      case '!':
      case '1': cmd=btn_home; break;
      case '^':
      case '6': cmd=btn_eject; break;
      case '*':
      case '8': cmd=btn_search; break;
      case 'x':
      case 'X': cmd=btn_stop; break;
      case '3': 
        if (!play_after) play_after=4; break;
    }
  }
  if (play_after > 0) {
    if (cmd) {
      play_after=0;
    }
    else if (play_after > 5) {
      if (play_after < millis()) {
        cmd=btn_play;
        play_after=0;
      }
    }
    else {
      cmd=btn_ff;
      if (--play_after == 0) {
        play_after=millis()+12000;
      }
    }
  }
  if (cmd) {
    digitalWrite(onboard_led, HIGH);   // turn the LED on to indicate activity
    for (int i = 0; i < retransmit; i++) {
      irsend.sendNEC(cmd, 32);
      delay(70);
    }
#ifdef DEBUG
    Serial.println(cmd, HEX);
#endif
    digitalWrite(onboard_led, LOW);    // turn the LED off - we're done transmitting
  }
}

In other links, How-To: IR Remote Control your Computer

Disney: evil, but defeatably evil

Disney DVD’s slogan is Moves, Magic and More.  They got the more part right for sure.

There I am trying to back up my copy of Wall_e_lic2_d1 so that once the kids have scratched the living bejesus out of the playing version, a new one can be generated from the master. And also as to avoid the annoying ads, language selection and other remote-control-based activity at the start – just shove it in the DVD player and walk away. Thankfully Australian copyright law lets me do this.

The studio have been dicking around with the disk’s table of contents, giving it over seventy files it claims are five gig in size – which, giving the DVD specification, is not possible. What you need to do in circumstances like this is play it in some player that will tell you what the magic track that actually contains the movie, not some hacked version of it. Then back that one up.

In the case of this particular disk it’s track 53, 1:33:26 long weighting in at 5425.95MB in size.

Thing is, DVDShrink barfs on it. Like it does Cars, but for different reasons. Thankfully I’ve recently discovered that Linux has an equivalent to DVDShrink, but this one is still being maintained. K9copy is it’s name; Cars was processed with no problems, and it was only the tomfoolery on Wall-E that caused a pause in activity.

So there’s one less application that I need a copy of Windows to run.