paulry

bloody internet...

GarryP

People before today were saying the ABS wasn't ready. In fact the people saying it were ready with news articles on the screwup, ready to run.

Census 2016: The $10 million online census – What went wrong? – EFTM

They paid IBM $10m for this 'solution'. They paid another company $470k for 'load testing'.

Only they only designed it for 1m form submissions per hour.

10m households, maybe 1m paper forms, maybe another 1-2m for outside this evening. That leaves 7m over the course of perhaps 4 hours. In short they didn't even spec it for the right rate number.

And when you get to these types of rates, you can't do simple load testing anyway. Other things start breaking - not least the load balancing, the DNS, and the routers into the servers in places like Telstra.

This time it appears the server for the images in the page was underspeced and started dying at lunch time. Then late this afternoon the base webservers that lead to the census servers were going down. And finally the actual form submission stopped working - pulling it all down.

Hopefully they will sack the entire board of the ABS over this - none of it was a surprise.

Pollyana

Did mine about 13.30 and it ran ok,took 10 minutes Colleague tried about 16.00 -very very slow, took about 30 minutes to get through it.Both very glad we didn't wait till tonight

paulry

It's government doing what government is good at: Blowing huge sums of money for little benefit/project failure. The mind boggles at the incompetence.

GarryP

This is basically the same solution they had for 2011 - if anything they didn't pay enough money and tried to do it on the cheap. In particular, giving it to IBM wasn't a great move.

Once you are in to the '1m+ users per hour' region you need to consider many more aspects - like the absolute size of router tables in the infrastructure heading towards your servers. It's hard and failure modes are weird.

And you could tell they didn't have a clue how bad it could get - since they were saying they were well prepared for it....

-----

11pm and its still dead Jim.

I wonder if the databases have crashed and they have lost everything? ;-)

paulry

That would be so embarrassing.

7am and still no sign of life...

spouse of scouse

Bloody typical. I leave the country and all the fun starts

knockoff nige

I think eithbthings like this as well as an electronic voting system, it should be possible for people to verify their details after submission. That would satisfy any doubters in a voting system and would be useful to also see that our data is not being compromised for the census.

GarryP

Stranger and stranger. Trunbull admitted that the reason the plug was pulled was because the geoblocking IP servers failed. But if that had happened the other servers should have been impossible to reach anyway - so what is this "an abundance of caution"?

I think, as expected, it's actually the routers which died due to the predictable, but unpredicted, level of traffic from normal punters. In other words they screwed up.

Swerv-o

You believe the overseas DoS attacks story to be a flight of fancy then?

Network engineer at work had this to say:


S

GarryP

I'm sure there were some DDOS script kiddies pointing at the census sites, but if it had been enough to seriously worry something that had been scoped for 1m+, it would have shown up on the alert services - and you would have found sites globally going down for Oz, as traffic couldn't get through.

No such evidence has been found.

Nope, it was the reality that they scoped for 750k per hour, when they should have expected 3m per hour and scoped for 10m per hour. And when you do that you need to look at not only the immediate servers, but the wider infrastructure leading up to it.

Cockup, in other words.

knockoff nige

I think that's all speculation as there could easily have been a denial of service attack which sends millions of requests to the site and then bring it down. No website is really safe from that but the government and the ABS messed up by not providing enough education on the changes and dealing with security worries from the public a long time ago.

GarryP

Take a look for yourself :

Digital Attack Map

Pollyana

People not in the country are to be counted through using their passenger exit cards (according to the census site) - caused great hilarity when I read this out at work as I have been filling those in as 'leaving permanently' since 2004, and it turns out most other people write rubbish on them too

knockoff nige

It'll never take off.