Author Archives: josh

Sanden Heat Pump water tank outlet size

The outlet size on a Sanden Heat Pump’s water tank is 3/4 inch (DIN20) with a BSP thread, so be aware you’ll have to use a reducer to get down to the normal 1/2 inch (DIN15) copper pipes.

However, 3/4″ pipes are able to transport a greater volume of water per unit time. If you’re running warm water down them for bathing, rather than hot for subsequent mixing, greater carrying capacity means better water pressure when running multiple showers, for example. If one was replumbing a whole house, and doing so with pre-mixed warm water, replumbing with 3/4″ copper would be a reasonable thing to do.

Refactoring the tax code

Our tax system is broken, properly.  It is insanely complex.  It’s a messy mix of transfers (e.g. the old age pension) and taxes (e.g. income tax).  Normally, when code ends up as a huge ball of mud someone steps in and re-writes it, or re-expresses it bit-by-bit (refactoring) until it’s much better – clearer, faster with a smaller footprint.

Before you refactor, you’ve got to figure out what the system is meant to do – what kind of country do we want Australia to be?


Generally I imagine Australians agree they want a progressive taxation system, where poorer taxpayers pay a smaller proportion of their income in tax compared to the richer members of society.  They want one where tax can’t be evaded (the system has high integrity), and the proportion of the economy devoted to the evaluation, collection and remittance of tax is low (it is efficient).  There seems to be an appetite for a system that operates, over the long term, neutrally – no Greek-style runaway spending, nor perpetual budgetary surpluses.  Equally, while Australians like the idea of a small government, they actually want a large one – one that intervenes to take the rough edges off of life, that provides a safety net no matter what misfortune befalls you; but they don’t want their government’s generosity to be exploited by those without need.  I look at as: I don’t want to look at poor people; make it better.


We’ve got unemployment benefits, single parent benefits, old age benefits, disability benefits and more.  But the nub of all of these payments is that a civilised society doesn’t leave anyone in abject poverty, and that problems that aren’t your fault ought to be covered by broader society. So what makes the unemployed less in need of support than a single parent?  Free money reduces the incentive to work, if one can.  How can a government tell if you’re merely free-loading?  Should it care, or should we – as a society – deem that if you don’t want to work, you shouldn’t have to?  What should we be doing when there’s a worldwide depression, and there just isn’t any work to be had?  If there’s no work for you locally, ought to be compelled to move, potentially away from friends and family – it reasonable to say that if your friends and family are so great that you refuse to move, they (instead of the government) can look after you?  If you bought your own home while you were working, ought the government give you have an easier go of it once you retire?  If you retire in the city, ought payments be made to you to compensate for the higher cost of rents than rents in the country?

The recent budget has decided that tertiary students ought to contribute more towards their education (by way of paying for a greater proportion of the cost of their education, and then paying more for the associated debt), and do so more quickly (by way of lowering the repayment threshold to basically the minimum wage).  Which is all good and well, but it seems that all those who got their education at lower rates or even free ought to shell out too – so if you got your education in the late 1970s, you ought to have a retrospective charge levied against you today.  I’m not clear on what use art degrees are to our economy, but if fewer are undertaken, is that so bad?  Don’t we want a well-educated citizenry?  It’s been pointed out that graduates earn more, so they ought to pay their way – but don’t they do that in higher tax brackets? What, if anything, should we do about the perpetual student – gaining education but never applying it to the benefit of society? What of those who build up a substantial education debt and then move to another country to apply said education?


The tax system is a little more complex – there are some taxes (sin taxes) that try to discourage legal but morally undesirable things – drinking, gambling, smoking, greenhouse gas emissions.  Other taxes discourage consumption – taxes on insurance, land transfer, Goods and Services Tax, excise on fuel.  Still more taxes try to level income inequality; our income tax system taxes are proportionally higher on higher incomes. In Australia we don’t have much in the way of asset taxes to level out disparity in asset ownership, but there are the odd example here and there.

If you design a tax system wrong, it discourages desirable behaviour and relatively encourages undesirable behaviour.  These behaviours ought to be enumerated somewhere.  Is saving better than consumption?  Can you have too much of one or the other?  Societal happiness increases with greater income and asset uniformity, but communist societies have shown that reward must follow effort or work becomes demoralising. So how much equality is enough, and how much inequality is too much? Should sin taxes be eliminated by eliminating the associated sin, as New Zealand is doing with tobacco?


I see a lot of our tax expenditures don’t mesh with any reasonable model of how the world ought to be, or commonly held views.

FBT – what the hell? Weddings are funded by the taxpayer? Allow me to refactor FBT for you: Organisations can spend money on whatever they like.  If they want to deduct that expenditure from their taxable income, they need to either: show it was a legitimate business expense, or attribute it to another taxpayer for whom it will be income.  Thousands of pages of legislation replaced with two sentences.

There’s been a lot of rhetoric lately, and from it I’ve learned that apparently debt is bad – especially being indebted to foreigners.  If it’s bad, make interest payments to foreign entities deductible at 98% rather than 100%, and keep lowering the proportion of the international interest bill that Australian taxpayers will subsidise until an acceptable mix of domestic-international debt is reached.

The Howard government decided that we needed more children, and women in the home, so paid for giving birth and underfunded childcare.  Payments were made for birthing and having children, almost regardless of income.  Welfare payments blew out to be by far the largest part of the budget, and importing children (via adoption) just got harder, slower and more expensive. I guess they were the wrong colour or something – bloody protectionists. Anyway, we ought not be growing babies locally, we ought to be importing them.  The demographic issues have been well studied.  Global population is expected to top out after the next couple of billion people, and then start dropping, but as I understand things Australia has no plan to balance its population growth. Is that the kind of country we want, covered in population centres, with cities that smear across hundreds of kilometres?

The biggest expenditure is on the Department of Defence, but I’m unclear what that department is tasked with or why it costs so much.  Apparently its role is to “protect and advance Australia’s strategic interests by providing military forces“, but that could be any armed force in the world.  Is it meant to repel a foreign invasion?  Is it meant to protect our exclusive economic zone?  Is it meant to provide an acceptable contribution to UN interventions?  Is it a coiled spring, ready to train up millions of soldiers in case of emergency?  Why does it need tanks, or fighter aircraft?  The USA is clear what their DoD is for, it’s the employer of last resort and a jobs program for domestic companies that can’t find other purchasers elsewhere. What is our DoD for?

There’s currently a bunch of whining about how Australian median Real Disposable Income has been stagnant for the last decade. The Liberal party is convinced the only way to get it moving is with Trickle-down economics – dropping the corporate tax rate. Labour seems to think if we get rid of enterprise bargaining, roll-back anti-union legislation and raise the minimum wage everything will be sorted. I think it’s globalisation at work – everything will level out, with wages in the developing world rising, and those in the developed world dropping. If you want less of something: tax it; if you want more of something: tax it less. If you want employees to be paid more, make wages deductible at more than 100%, rather than taxing corporate profits less.

So, tax system broken, needs more income (and perhaps dramatically less air superiority fighter jets).


Increasing and broadening the GST would dramatically increase the tax take, but that comes at a cost.  The GST is a regressive tax.  Taxing things like fresh food, health-care and financial payments (interest, insurance) is complex, in that there are a large number of interacting considerations. On the whole, I’m in favouring of taxing everything (broadening that tax) and hiking the rate.  But this is where the complicating considerations come in.

Push up GST and a lot of high-value purchases will go overseas – cameras, phones, breast implant surgery, etc. – because international transactions are not taxed.  This no-tax on international transactions has the effect of pushing multinational companies to bill from foreign countries and avoid GST on their sales (e.g. Google, Apple, et al). So: just tax international financial transactions; credit cards and PayPal to start with, that will catch 99% of low-value transactions, but you’re going to have to keep moving to catch whatever the latest work-around becomes. I can see health insurance funds moving overseas, for example. Or just tax all international transactions, with a simple piece of paperwork to fill in if that wasn’t a payment for goods or services, but was in fact a transfer.

There’s a complaint that if taxes are set too high, the taxpayers will leave.  High income earning English speaking workers are highly mobile, and can move to whatever tax jurisdiction they like, and they can move their money there too.  But if they’re living in Australia, consuming Australian government services, they’ll get taxed here.

Anyway, once you’ve hiked up the GST tax take, you’re going to have to compensate the poor – consumption taxes are regressive (i.e. the proportion of your income gobbled up by them is higher the lower your income is).  This is where negative income taxes come in; everybody (children included) becomes a taxpayer, and gets a cut of the negative income tax goodness.  And thus we’ve closed off this inequity – yes, more GST is paid, but if you don’t have any income the government hands over money that ought to cover the additional GST you’ll be shelling out.


Companies and Trusts are taxed as a different rate to the rest of the population, although I’m not clear as to why that is.  In fact, I’m not clear as to why companies pay tax at all.  Companies and Trusts ought to be legally obligated to hand over their profits to their owners each tax year, and have the owners deal with the tax liability arising from that.  Doing so would remove an enormous world of complexity and opportunity to manipulate the tax system.  Naturally taxpayers without TFNs will have to have tax withheld at the highest marginal rates, for later imputation.  Admittedly, this will lead to an increase in unemployment for accountants and lawyers, but I’m really not upset by that.

Although donations to a registered charitable cause are wholly tax deductible, for some weird reason there’s a company tax exemption on companies owned by charities – why not just donate all of this year’s profits to the owning entity?  In the same breath, government-owned businesses pay tax, but the tax paid by businesses owned by state governments are refunded to those same state governments. No doubt this is some relic from the transfer of taxation power from the States to the Commonwealth, but enough is enough.

Often companies don’t pay out all of their profits as dividends – some of that money is retained to fund growth.  This can continue to happen under my proposed system; the liability for the full profit is transferred to owner, but the amount of cash transferred is up to the directors.

So, by taking companies out of the tax-paying regime an enormous amount of fiddling and pissing around will be removed – creating a simpler, more straight-forward and transparent tax system.


Capital Gains Tax makes a poor attempt at smearing the real (i.e. inflation-adjusted) realised capital gain over period it was made and taxing it as income, but while treasurer Peter Costello introduced concessional taxation and since then Australian housing has become unaffordable for a large section of the population.  The ATO is perfectly capable of applying capital gains over the entire earning history, and ought to do so.  50% CGT concession ought to be removed, as it doesn’t encourage investment but instead speculation.

CGT exemption for housing is a hard problem, but one I think I have a solution for.  Read below.


Houses have proven to be a failing in our taxation system.  Concessional taxation treatment means CGT is not payable on homes.  State governments have become addicted to the revenue of turnover in the property market, which is a volatile income stream.  Stamp duty on property is a substantial impediment to transactions – it costs a lot of money to move home, which means labour will be much less willing to move to follow work. Inflexible labour markets drive up employment/business costs. Instead of stamp duty on property transfers, state governments ought to do as the ACT is doing and move from stamp duty to levying annual property taxes.


Tony Abbott said in 1995, “The basic objective of compulsory superannuation is that the government is taking our money now so that it does not have to pay us a pension when we retire.

“The government is making us worse off now so that it will be better off in the future.”

This is a laudable goal. Choosing to not work merely because you’re old is foreseeable, and the government ought not be expected to insure you, or the broader populous, against foreseeable certitudes.

Super is not a con as Abbott asserted, and the rates need to be increased.  Given that people can generally be expected to earn income for 45 years and be retired for 20, putting away a third of your income for old age wouldn’t be outrageous.

Superannuation is fabulous idea, tucking away income for the future, and is concessionally taxed on the way in, and also while in the super system – enabling growth of superannuation savings.  On the way out – if you wait long enough (60 years of age) – the money is tax-free.  Why tax free?  Isn’t it income?  This is madness, and needs to stop.

Because of the tax-free status, super is seen as a fantastic mechanism for inheritance planning, and that’s outrageous.

It’s time to rework superannuation; make it an income-smoothing scheme:

Make all contributions 100% tax deductible, and split each contribution into two – one part vested until retirement, the equivalent of today’s super system.  Money transferred into super, or earned while in super, will be non-taxable. If you inherit super money, it’s income – but you can shove it straight into your own superannuation account and avoid paying tax on it.  When you pull money out of super, it’s income. Employers won’t pay their employee any more – it will all go to their super fund, who withhold appropriate amounts for tax and old age, and pay the employee whatever they ask for – more or less than they earned, it’s up to them, because it is all smoothed out and tax accounted for. There’s only hundreds or thousands of super funds, so push the tax paperwork onto them rather than the millions of businesses around Australia.

If you want to buy a house, they’re considered superannuation investments and can be bought with your superannuation money. When you sell it the sale proceeds are wholly income – which you can pay tax on or shove back into your super account.

When not working, for example if you’ve retired, you draw down on your super account and pay tax on it like everyone else.

This set-up will demolish the distinction between assets and income. For too long you’ve been able to build up enormous assets and hand them on to subsequent generations without being clipped for a contribution to running the country.


Of course, much of this was considered by the Henry Tax Review,  The Australia’s Future Tax System Review couldn’t consider super or GST, and they’re two areas that need reform too.  But this stuff is simultaneously obvious and too hard for our politicians.

Replacement Hot Water Service

All my electricity is green: my retailer buys RECS sufficient to back my electricity purchases. As such, I don’t care how much electricity I use, except such that it costs me money.

Breakeven analysis is fun.

My house has a twelve year old 160 litre resistive electric hot water service (HWS). General opinion seems to be that a HWS will last perhaps as long as 12 years before failing (my last house had one that was 30 years old and still going strong). It’s currently inside the house (taking up valuable floorspace), and may not survive being moved outside in the coming renovation. I want an instantaneous gas HWS (unlimited hot water at exactly the temperature I dictate), but refuse to use gas.

Instantaneous electric HWS exist and are only about $1000, but require three-phase power (an upgrade costing a surprisingly small $1000, plus electrical work on my side of the divide).  How much power will it consume?  Turns out, the same as resistive heating the water, but it’s all peak electricity.  That pushes its daily cost quite high.  There are other HWS options; reusing the existing tank (free-ish), replacing it with a larger tank (still resistive, $1000), sucking up all the spare electricity from the PV solar system (perhaps $1000), or a heat pump ($3400), and combinations of the above.

I was able to figure out how much electricity we’re using to power our HWS by virtue of it being on a separate meter to the rest of the house – 4.6kWh/day, costing about $0.85/day because it runs off peak. Hot water consumption is expected to increase after the renovations.  It turns out that how much hot water is consumed, and when, is very important for accurately pricing electricity consumption. I’ve a fairly complex spreadsheet modeling current and projected consumption patterns, and the resultant energy requirement timing and costs. We have a PV solar system, which is how I thought I could push our cost of hot water down – heat it from the panels during the day, when electricity is cheap for me ($0.119/kWh).  If your tank is too small (which ours will be/nearly is already) then you’ve got to heat using electricity other than cheap solar electricity.

I calculated the Total Cost of Ownership at the 5 year mark, and the average daily cost of hot water for the various options (note, this is for my projected hot water consumption profile – yours will differ, altering the values):

Option TCO Daily Cost
Keep existing HWS, peak electricity $4,340 $2.27
Keep existing HWS, off peak only, coupled to instantaneous electric HWS $7,411 $2.69
Buy 315L resistive HWS, off peak only $5,497 $2.46
Keep existing resistive HWS, run off PV solar and off-peak $3,764 $1.51
Buy 315L resistive HWS, run off PV solar $4.968 $1.63
Buy 315L heat-pump HWS, run off PV solar $3,888 $0.27

The heat pump can run off solar using its built-in clock, saving $1000 in diverter costs. You can see that its TCO is a little more than a salvage job on the existing HWS run from solar power, but the daily cost means the heat pump is going to pull away at a mad rate. Hot water that cheap is making me think of grand ways to heat my house.

Ditching gas

All my electricity is green: my retailer buys RECS sufficient to back my electricity purchases. When I calculated my household’s Green House Gas emissions equivalence, we pulled in emissions below 15% of that of the average Australian household. In fact, our emissions were down to two sources: our car (7000 km/year @ 9l/100km – emitting a quarter of that of the average Australian household) and our natural gas consumption (20300MJ/year – home heating emissions 57% of that of the average Australian household).  Apparently emissions can vary from 3 to 30 tonnes/year – I’ve calculated my household at around 2.5 tonnes per year at the moment. I think we can do better. How about 1.5 tonnes per year?

I’ve had a poke around the non-hydrocarbon motorised transport market. There isn’t much there for me, cars are north of $50,000, lifespans are limited. If I could buy an electric car for $20,000 that was going to last 20 years, I’d be up for it. Because I can’t, hydrocarbons will continue to be used for this form of transport. Will revisit when car fails, I’m guessing in less than a decade. Besides, I’m pretty convinced “car ownership” will end up being something people did in the 20th century, not the 21st.

We don’t cook with gas; we have an induction cooktop. I hate electric cooking – resistive electric cooking. It’s inefficient, slow, too cold, too hot, ugly and messy and too expensive. I’ve always cooked on gas. Induction cooking has turned me around; it’s everything gas cooking is, without the explosions, burnt-on gunk, poisoning and GHG emissions.  However, it is fussy (it only works with ferrous cookware) but that’s inconvenient, not a showstopper (example: coffee pecolators are almost all aluminium, and those that aren’t have a very small base. The pecolator has to go in a small pot to be used).

Our gas consumption is purely for space heating via a ducted heating system. It costs less than $770/year to heat our house, so an electric replacement will need to be competitive with that. Having run the numbers, I’ve calculated our gas consumption produced 4000kWh of heat in the house each year. Doing that with air conditioners would (assuming 400% efficiency, which is pretty middle-of-the-road) require 1000kWh of electricity. I seem to pay about 30 cents/kWh (if you can figure out what your electricity actually costs you, I’d love to hear what you did to get that number), so that’s $300/year to run air conditioners instead of ducted heating. Payback is less than 10 years if $4000 is spent on adding aircon units.

From an environmental and financial perspective it’s time to ditch gas, so I’m off. Each gas bill raises the daily connection fee.  It’s about $1/day now, so if you don’t use much gas there’s an increasing incentive to use no gas at all. Nearly half my bill is for the privilege of having a gas supply.

But wait! I love wok cooking, and there’s almost no way to wok-fry stuff without gas. What to do? For a couple of years we’ve used a butane camp stove as a stop-gap until we got around to plumbing in our dedicated wok burner, but if we’re cutting off gas we’ll continue living like animals for the rest of our squalid lives! No fear, says my plumber: convert to LPG – like used for BBQ cooking. And so, we now happily wok-fry on gas, which I figure will cost us $30 – $60 a year to refill the bottle. $38 for the LPG conversion kit for the burner, which would have been avoided if I’d thought this all through a couple of years ago when buying the wok burner.

The next problem is: what to do about hot water?

Securely run a low memory/low CPU Minecraft server

If you’ve got next to no memory and CPU available to run a Minecraft server, don’t fret. Cuberite is what you’re after. At them moment, Cuberite isn’t bug-free, nor indiscernible from a genuine Minecraft server, but it’s quite usable – and instead of needing 4GB+ of RAM, it needs less than 300MB. And it needs next to no processing power: some people run Cuberite on their Raspberry Pi and have plenty of CPU available.

I would at this point go on about how this is a significant point of difference between C++ and Java, but Java optimizes for something different to C++.  I got into an interesting discussion with Cathy about this after reading a question someone had about what Java was trying to be good at. I used to think that VB was the new COBOL, but clearly Java is the new COBOL; those Java programs are going nowhere, they’re verbose and easy to understand and maintain.

A point to note: The Minecraft protocols are bandwidth heavy, I found if I wanted to run a server at home I’d be able to have one, perhaps two players. Thus is Internet in Australia. Instead I’ve dropped this onto a free AWS VPS instance and bandwidth is no problem.

Still, it’s a random piece of software off the Internet, so we’re going to give it its own user account for our own safety. Let’s install the software:

curl -sSfL https://download.cuberite.org | sh
sudo mv Server /usr/local/cuberite
cd /usr/local/cuberite

Cuberite allows configuration through a web interface.  We now enable webadmin.ini
[User:admin]
; Please restart Cuberite to apply changes made in this file!
Password=yourstrongpassword
[WebAdmin]
Ports=8080
Enabled=1

Port 8080 is the alternative html port (http/https).  You could cd into webadmin and run GenerateSelfSignedHTTPSCertUsingOpenssl.sh and get https serving, but your browser will barf on the certificate. Instead, let’s use a LetsEncrypt certificate, one that we installed earlier. First we make our one-line shell script for running the daemon:

sudo useradd -c "Cuberite server" -f -1 -M -r cuberite
chown -R cuberite:`whoami` /usr/local/cuberite/
sudo nano /etc/init.d/cuberite.sh

#!/bin/sh
### BEGIN INIT INFO
# Provides: cuberite
# Required-Start: $local_fs $network
# Required-Stop: $local_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: cuberite
# Description: Cuberite server, a Minecraft server lookalike
### END INIT INFO
cd /usr/local/cuberite
sudo -u cuberite /usr/local/cuberite/Cuberite -d &

Next we set it going when the box starts up:

sudo chmod +x /etc/init.d/cuberite.sh
sudo update-rc.d cuberite.sh defaults

Before we can go to the website we need to allow user cuberite to get to the certificates:

sudo groupadd privkey_users
sudo usermod -aG privkey_users cuberite
sudo sudo chmod g+rx /etc/letsencrypt/live/
sudo sudo chmod g+rx /etc/letsencrypt/archive/
sudo chown root:privkey_users /etc/letsencrypt/archive/
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/cert1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/chain1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/privkey1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/fullchain1.pem
sudo chown root:privkey_users /etc/letsencrypt/live/
sudo chown root:privkey_users /etc/letsencrypt/live/example.com/
sudo -u cuberite ln -s /etc/letsencrypt/live/example.com/cert.pem /usr/local/cuberite/webadmin/httpscert.crt
sudo -u cuberite ln -s /etc/letsencrypt/live/example.com/privkey.pem /usr/local/cuberite/webadmin/httpskey.pem

Changing these permissions doesn’t affect apache2’s ability to get them.
The reason we’ve used a group here is to allow both Cuberite and any other app (for example, exim) to access the private keys; just add any other user that needs to use the private keys to the privkey_users group.

Remember to punch a firewall hole for port 8080. Fire up Cuberite now:

sudo service cuberite restart

And check if that worked, there should be about one entry:

ps -aux | grep cuberitps -aux | grep cuberit

If not, you can check in the logs directory to see what’s wrong.

So now:

sudo lsof -i :8080
https://example.com:8080/

should be secure.  Note the https is mandatory, as your browser will use http if you fail to specify a protocol.

Where do you wake up from a bed in Minecraft?

After issuing many /time set night commands, I can tell you the waking-location algorithm for Minecraft. This presumably also affects your spawn point.

Two locations are checked, and if they fail to select an acceptable location the pillow-location is used regardless of consequences of picking this location. An acceptable location is on the same level as the bed, and has two transparent-non-solid blocks above it (i.e. you will be standing next to the bed without your head or body embedded in something that’s killing you).

The process is the same for each of the two locations:

Sweep x-1 to x+1:
  Sweep z-1 to z+1:
    if the location is acceptable, we're done

The locations are checked in the order: pillow-part-of-bed, non-pillow-part-of-bed. The effect is:
From the Minecraft wiki:

For a bed to be usable as a spawn point, the player must be able to stand next to the bed at the same level as it. There must be a solid block at the same ‘floor’ level as the bed, with 2 transparent blocks of space (for example, air) for the player to stand in, in one of the ten blocks that surround the bed. It doesn’t matter if the bed itself has blocks above it.

Netgear Stora upgrade v3: 2-disk-JBOD to 1-disk-JBOD

So, we’re butting heads up against the storage capacity of our Netgear Stora again (93% full). The NAS currently has 2 x 2TB drives and no more free bays to drop drives into, so whatever the next arrangement is it has to involve getting rid of at least one of the current drives. The Stora is currently backed up to an external drive enclosure with a 4TB drive mounted in it. Other things are also backed up on that external drive, so it’s more pressed for space than the Stora.

So here’s the plan:

  • collect underpants
    This was a flippant comment, but it’s upgrade season and we recently acquired a computer second hand, which had an i5-3470S CPU, the most powerful thing in the house by a significant margin. I wanted the dual Display Port outputs, but unfortunately it could only be upgraded to 8GB of RAM, so instead the CPU got swapped into our primary desktop (and a graphics card acquired to run dual digital displays). Dropping in a replacement CPU required replacing the thermal grease, and that meant a rag to wipe off the old grease, thus the underpants.
  • backup the Stora to the 4TB drive
  • acquire a cheap 8TB disk because this is for backing up, not primary storage
  • clone the 4TB drive onto it using Clonezilla
  • expand the cloned 4TB partition to the full 8TB of drive space
    Well, that didn’t work.  Clonezilla didn’t seem to copy the data reliably, but admittedly I was running a stupidly old version.  Several hours of mucking around with SATA connectors and Ubuntu NTFS drivers later, I gave up and copied the disk using Windows.  It took several days, even using USB3 HDD enclosures, which is why I spent so much time mucking around trying to avoid it.
  • backup the Stora to the 8TB drive
  • remove the 2 x 2TB drives from the Stora
  • insert the 4TB drive into the Stora
  • allow the Stora to format the 4TB drive
  • pull the 4TB drive
  • mount the 4TB and 2 x 2TB drives in a not-otherwise-busy machine
  • copy the data from the 2 x 2TB drives onto the 4TB drive
  • reinsert the 4TB drive into the Stora
  • profit!

And, by profit, I mean cascade the 2TB drives into desktop machines that have 90% full 1TB drives… further rounds of disk duplication ensue. 1TB drives then cascade to other desktop machines, further rounds of disk duplication ensue.

At the end of this process, the entire fleet will have been upgraded. But the original problem of butting heads against the Stora will not have been addressed; this will hopefully a simple matter of dropping another drive in.

The last time we did this, we paid $49.50/TB for storage.  This time around, it was $44.35; a 10% drop in storage prices isn’t anything to write home about in a four-and-a-half year window.

Trustworthy email: authentication using exim4, SPF, DKIM and DMARC

The email authentication technologies we’re about to implement are, according to the authentication authorities, all you need to be regarded as being from your domain when you send email, and someone else not being from your domain.  Effect: your emails can be considered trustworthy by email receivers who use these technologies. If they don’t use these technologies, they can’t tell.

At the very least, Google will be less likely to think your email is spam.

PTR record

A PTR record can be obtained from your host’s nameserver – it’s a reverse DNS record for your IP address. If the PTR record points at ec2-23-65-53-221.ap-southeast-2.compute.amazonaws.com rather than example.com (your domain), and you’re claiming to be sending mail from example.com, what’s the email recipient meant to think?

host 23.65.53.221

will tell you what the host for that IP is. Lodge a ticket with your hosting provider and get that PTR record changed to example.com. This might take about a day.

SPF record

Create a Sender Policy Framework record on your nameserver:

TXT @ "v=spf1 a mx -all"

This says “for my domain, I will only send email from IP addresses listed on the nameserver”.  Nameserver changes take time to propagate.

After your nameserver changes have propagated, you can go to https://dmarcian.com/spf-survey/ to check out if you got it right.

DKIM

DomainKeys Identified Mail is where things get more involved.  We’re doing this on a Debian Linux, like Ubuntu for exim4. We’re making our signing key 2048 bits, which is long enough to make life slightly unpleasant for us. Fortunately for you I’ve written a bash script that outputs the TXT record we need to create on the nameserver – because some nameservers (I’m looking at you, Gandi) can’t hold “long” strings – it’s broken into “small” strings:

sudo apt install openssl
cd /etc/exim4
sudo openssl genrsa -out dkim.private 2048
sudo openssl rsa -in dkim.private -out dkim.public -pubout -outform PEM
echo $(echo $(date -u +%Y%m%d && echo '.domainkey.example.com') | sed -e 's/[ ]//g' && echo $(echo ' TXT "v=DKIM1; p="' && echo $(grep 'PUBLIC KEY' -v dkim.public) | sed -e 's/[ ]//g' | fold -w200 | sed -e 's/\(.*\)/"\1"/g'))

which gives something like
20170419._domainkey TXT “v=DKIM1; p=” “MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvCNqU0Njd4YQ4e89T3FNc+uyOS2JwUqynGk7uwcSYHjIE2MGRuTxi56s4JgPKSnCVlBkJlUnXQHXFp2UGnLm8SADtjRMfWwpNxz6TmzXBpMnNZV1zvuoBBdcxh0Qg1TtCSACtWM6ehml0BmOHVA8Ippqj9iRlP2HMjuVMxZXewN9eJl”
“c6zsyOwQPvVKpJ+Rdvr+pPkDztAVTw7mNSeyy+TL6O/3L9sl7A19Yx8jLHKuGUh9LutVuv1VP16e7GwlnA3Zqn5C1jyY5Qvr2SEHZMcE3VzD7XKZtZWbpkGh+A5S15NrOH4k9tbVfNbjft6Y1jUJRTT+4DD0ZEVlr4zO+WQIDAQAB”

That all goes into one nameserver TXT record, spaces and all.  The world will join up the ” ” and get one big string. Note the bold number up there? That’s the selector. That a number needs to get larger with each new key.  Periodically you’re going to have to reissue your key because security.  You know what gets larger as time goes by?  The date.  Use the date.  If you screw up, use tomorrow’s date, etc.

Once you’ve got our public key out to the public via our public nameserver, we need to get exim to sign the payloads:

sudo nano conf.d/main/01_exim4-config_listmacrosdefs

After the line CONFDIR = /etc/exim4, add:

#DKIM loading
DKIM_CANON = relaxed
DKIM_DOMAIN = ${sender_address_domain}
DKIM_PRIVATE_KEY = CONFDIR/dkim.private
DKIM_SELECTOR = 20170419

and reload the mail server

sudo service exim4 restart

After an appropriate delay for nameserver propagation, use https://protodave.com/tools/dkim-key-checker/?selector=20170419&domain=example.com to check your work.
If that works out, mailto:check-auth@verifier.port25.com from example.com to ensure everything checks out:

echo -e "Test my DKIM plz\nMsg Body\n.\n\n" | mail -v check-auth@verifier.port25.com

DMARC

Domain-based Message Authentication, Reporting and Conformance is where the wheels can come off if you screwed anything up.  We’re going to set things up so that when you screw it up, computers scold you rather than putting your emails in the bin.

You will need to create two dmarc reporting accounts.  Servers will email you a (surprisingly detailed) report card on how you’re doing with your implementation. It’s best if these accounts are on the same domain, because technically you need to be or it’ll be ignored (Google will happily mail reports off-domain even if the other domain hasn’t said that’s okay).  Yours are dmarc_failures@example.com and dmarc_summary@example.com, according to the following nameserver entry:

_dmarc.example.com. 1800 IN TXT "v=DMARC1;p=none;pct=100;ruf=mailto:dmarc_failures@example.com;rua=mailto:dmarc_summary@example.com"

none is the consequence for screwing up. none is where we’ll start at, and see what the reporting records say to us.  After a while, you’ll be comfortable that everything is ticking along nicely, and you’ll up the consequent to quarantine (shove it in spam) or reject (burn it).

After your nameserver changes have propagated, you can go to https://dmarcian.com/dmarc-inspector/ to check out if you got it right.

As a human, to read the records sent to you, upload the files to https://dmarcian.com/dmarc-xml/

Making a captcha deamon for spamgourmet installations

For those of you following along at home, this is part of a cookbook style instruction set for getting spamgourmet going, but because of screwed up permission logic I can’t post this section there.

The captcha is for validating humanity when creating spamgourmet accounts. We’re going to limit what parts of the OS it can tromp over:

sudo useradd -c "captcha server for spamgourmet" -f -1 -M -r captcha
sudo /bin/mkdir -p /var/www-spamgourmet/captchasrv/
sudo chown -R captcha /usr/local/lib/spamgourmet/captchasrv/
sudo chown -R captcha /var/www-spamgourmet/captcha

Now we make our one-line shell script for running the daemon

sudo nano /etc/init.d/captcha.sh

#!/bin/sh
### BEGIN INIT INFO
# Provides:          captchasrv
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: captchasrv
# Description:       captcha daemon for spamgourmet
### END INIT INFO
sudo -u captcha perl /usr/local/lib/spamgourmet/captchasrv/captchasrv.pl &

Next we get it going

sudo chmod +x /etc/init.d/captcha.sh
sudo update-rc.d captcha.sh defaults

And check if that worked, there should be about four entries:

ps -aux | grep captc

Now the captcha server will start whenever the computer starts.

How to to install the Crypt::Eksblowfish::Bcrypt module, and Crypt::Random

Have you gotten the error
Can't locate Crypt/Eksblowfish/Bcrypt.pm in @INC (you may need to install the Crypt::Eksblowfish::Bcrypt module)
and wondered what to do? Wonder no more!

sudo apt install libcrypt-eksblowfish-perl

and perhaps

sudo apt install libdigest-bcrypt-perl

What about
Can't locate Crypt/Random.pm in @INC (you may need to install the Crypt::Random module)
Easy!

sudo apt install unzip make gcc
wget http://search.cpan.org/CPAN/authors/id/I/IL/ILYAZ/modules/Math-Pari-2.01080900.zip
cd Math-Pari-2.01080900/
perl Makefile.PL
sed -i 's/CLK_TCK/CLOCKS_PER_SEC/g' pari-2.1.7/src/language/init.c
make
make test
sudo make install
cd ..
wget http://search.cpan.org/CPAN/authors/id/V/VI/VIPUL/Crypt-Random-1.25.tar.gz
tar zxvf Crypt-Random-1.25.tar.gz
cd Crypt-Rando1.25.tar
perl Makefile.PL

Easy! Only takes a few hours if you don’t know what you’re doing.

Installing a secure Apache webserver to run Perl

So, you want to run Perl on the web, because it’s the 90s all over again. You want HTTPS, because… no, there’s no because.  You want HTTPS.  Who wouldn’t?  Here’s what you do on a Debian Linux such as Ubuntu:
sudo apt-get install apache2 libapache2-mod-perl2
mod-perl is an Apache module that allows Perl programs to be executed from Apache.

Our goal is to get /var/www/html/index.pl running at http://www.example.com/index.pl:

#!/usr/bin/perl
print "Hello World"

Disable the default Apache virtual host:

sudo a2dissite 000-default.conf

Create an example.com.conf file in /etc/apache2/sites-available with your text editor, replacing instances of example.com with your own domain name in both the configuration file and in the file name /etc/apache2/sites-available/example.com.conf

<VirtualHost *:80>
     ServerName example.com
     ServerAlias www.example.com
     ErrorLog ${APACHE_LOG_DIR}/error.log
     CustomLog ${APACHE_LOG_DIR}/access.log combined
     <Directory /var/www/>
              Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
              AllowOverride None
              AddHandler cgi-script .pl
              Require all granted
     </Directory>
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
     ServerName example.com
     ServerAlias www.example.com
     ErrorLog ${APACHE_LOG_DIR}/error.log
     CustomLog ${APACHE_LOG_DIR}/access.log combined
     <Directory /var/www/>
              Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
              AllowOverride None
              AddHandler cgi-script .pl
              Require all granted
     </Directory>
</VirtualHost>
</IfModule>

If you have multiple sites, you’ll want to do things with DocumentRoot to isolate them from each other. But that’s for another post.

You might add in DirectoryIndex /index.pl to make http://www.example.com/ execute your program.

The Directory section above allows you to isolate executable code from served code, which is good practice. For this example we’re dumping the executable in with everything else, which is questionable.

Repeat this process for any other domains you host.

sudo a2ensite example.com.conf
sudo ln -r -s /etc/apache2/sites-available/example.com.conf /etc/apache2/sites-enabled/example.com.conf
sudo service apache2 restart

Punch holes in your firewall for ports 80 and 443.  Navigate to http://www.example.com/index.pl to check all is okay. You ought to see Hello World displayed for your website.

Having security used to be a pain.  SSL certificates signed by a recognised CA cost money, and then you’d have to keep them up to date, and there wasn’t process automation, so you’d do all that stuff by hand.  LetsEncrypt address all these problems, handing out free certificates and scripted everything.

Now it’s time for the S part of HTTPS:
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-apache
sudo certbot --apache

certbot renew
If that works, we’ll automatically renew our 90-day certificates every month:
echo '@monthly root /usr/bin/certbot renew >> /var/log/letsencrypt/letsencrypt-auto-update.log' | sudo tee --append /etc/crontab

Done.  You’ll never have to worry about certificates again. Now alter your Apache sites-available file (look in /etc/apache2/sites-available/) to include the (optional) redirect HTTP to HTTPS and the mandatory location of the SSL certificates:

<VirtualHost *:80>
....
# Only allow HTTPS
RewriteEngine on
RewriteCond %{SERVER_NAME} = example.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
...
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Now make the secure version live, and in the process the insecure one… shy? When a request is made for a http page, like http://example.com/index.html, the response will be “Here’s https://example.com/index.html where what you asked for has moved to… forever!”:
sudo service apache2 restart
Now requesting http://www.example.com/index.pl ought to deliver you to https://www.example.com/index.pl

Install exim4 STARTTLS using a free LetsEncrypt certificate

Here we are on a Debian Linux, such as Ubuntu and we want to run a mail server. Exim4 is currently the most popular email server, but getting it up and working for free is a hassle – who wants to pay for a SSL certificate, on an ongoing basis? And then there’s the maintenance of the security of it – constant renewal, renouncing and re-installation of the certificates.

Wherever you see example.com, swap in your Fully Qualified Domain Name. That may be mail.example.com
It’s assumed you’re not logged in as root, but user ubuntu
Wherever you see 1.2.3.4, swap in your machine’s local IP address, from
ifconfig | grep "inet addr" | grep -v "127.0.0.1"

Security is all handled automatically by LetsEncrypt’s certbot. I’ll let you look that one up yourself. Run it up and get your certificate for example.com

Once you’ve got that handled, punch a hole in your firewall so that port 25 can get through from the outside world to your machine. Be aware: the outside world is filled full of botnets trying to hack into your machine.  After installing exim, keep an eye on the logs in /var/log/exim4/ for a while.

Let’s install exim4:
sudo apt-get install exim4
sudo dpkg-reconfigure exim4-config

  • pick “Internet site”
  • system mail name is example.com
  • IP address is 1.2.3.4 (the one returned by ifconfig, not the externally accessable one)
  • Other destinations: example.com
  • No relays
  • No smarthost
  • No Dial-on-Demand
  • mbox format (or whatever)
  • Split the files
  • ubuntu for postmaster mail

Check we’re now running a mail server:
sudo netstat -napt
should show
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 1.2.3.4:25 0.0.0.0:* LISTEN 25700/exim4

Now we have a mail server, the world needs to find it. Check your nameserver setting to ensure mail is destined this machine.  You probably want only one MX record.

Check the Internet can send mail to our server. After allowing for the appropriate propagation delay for your nameserver changes, use gmail or something to send an email to ubuntu@example.com – you should be able to read it by typing
mail

Now it’s time to enable MTA-MTA link encryption for secure transport of mail, by enabling STARTTLS on exim4 using our LetsEncrypt certificate
sudo nano /etc/exim4/conf.d/main/03_exim4-config_tlsoptions
Enable STARTTLS by adding/setting in the tlsoptions section:
MAIN_TLS_ENABLE = yes
MAIN_TLS_CERTKEY = no

before any of the IF shenanigans. Also add/replace pointers to the certificates:
tls_certificate = /etc/letsencrypt/live/example.com/fullchain.pem
tls_privatekey = /etc/letsencrypt/live/example.com/privkey.pem

The MAIN_TLS_CERTKEY = no entry fixes an exim4 log message
2017-04-16 09:13:24 TLS error on connection from your.home.ip.com (IcePlanet) [5.6.7.8] (cert/key setup: cert=/etc/exim4/exim.crt key=/etc/exim4/exim.key): Error while reading file.
You will see this when testing with swaks:
$ swaks -a -tls -q HELO -s example.com -au test -ap '<>'
=== Trying example.com:25...
=== Connected to example.com.
< - 220 your.vps.host.com ESMTP Exim 4.86_2 Ubuntu Sun, 16 Apr 2017 09:13:24 +0000 -> EHLO IcePlanet
< - 250-your.vps.host.com Hello your.home.ip.com [5.6.7.8]
STARTTLS
< ** 454 TLS currently unavailable *** STARTTLS attempted but failed -> QUIT
< - 221 your.vps.host.com closing connection
=== Connection closed with remote host.

Allow exim (which when running runs as user Debian-exim) to get to the certificates:

sudo groupadd privkey_users
sudo usermod -aG privkey_users Debian-exim
sudo sudo chmod g+rx /etc/letsencrypt/live/
sudo sudo chmod g+rx /etc/letsencrypt/archive/
sudo chown root:privkey_users /etc/letsencrypt/archive/
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/cert1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/chain1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/privkey1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/fullchain1.pem
sudo chown root:privkey_users /etc/letsencrypt/live/
sudo chown root:privkey_users /etc/letsencrypt/live/example.com/

Changing these permissions doesn’t affect apache2’s ability to get them.
The reason we’ve used a group here is to allow both exim and any other app (for example, a secondary service that wants to use 8080 to serve up a configuration page) to access the private keys; just add any other user that needs to use the private keys to the privkey_users group.

These permission changes prevent the following error message in your log file:
2008-06-03 08:27:35 TLS error on connection from me.at.home.com ([1.2.3.4]) [5.6.7.8] (cert/key setup: cert=/etc/ssl/certs/server.pem key=/etc/ssl/private/server.key): Error while reading file.

Restart the service and the TLS settings ought to be working
sudo service exim4 restart
Test STARTTLS is working from another machine
swaks -a -tls -q HELO -s example.com -au test -ap '<>'
There shouldn’t be any obvious complaining.

Done!