Linux - No space left on device, yet plenty of free space

My little server ran into an issue, and started reporting the error:

No space left on device

No worries, lest figure out which disk has full and clean up…

Using the df command with the -h (for human-readable output) it should be easy to find the issue:

root@server:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 483M 0 483M 0% /dev
tmpfs 100M 3.1M 97M 4% /run
/dev/vda 20G 9.3G 9.4G 50% /
tmpfs 500M 0 500M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 500M 0 500M 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 100M 0 100M 0% /run/user/1000

Strange. Notice who the /dev/vda is 50% fillled and all other disk devices seems to be finde too. Well after a little digging, thinking and googling, it turns out device space consists of two things - space (for data) on the device and iNodes (the stuff used to mange the space - where the data go - simplified).

DNSSEC and switching nameservers

I’ve switched nameservers for all my domains yesterday. During the past many years I’ve been free-riding on GratisDNS and enjoying their free DNS service (and luckily never needed support in their forums).

Yesterday I switched to Cloudflare and I’m using them for DNS for this (and other domains). I don’t have any particular requirements, and the switch was mostly easy and automated to the extent possible. Two domains went smooth, but the last my mahler.io domain went a stray a few hours during the switch.

Updates...

It’s been quiet here for a while, but be things have been happening behind the scenes. In case your wondering the site (and surroundings) have been seeing a number of updates which eventually may make it into separate posts.

  • I’m running on a Digital Ocean droplet. It was provisioned as an Ubuntu 12.04 LTS, which is dead by now (as in no more updates including security updates). The server has now been roll up to an Ubuntu 16.04 LTS in place.
  • As I was messing around with the server, I’ve added IPv6 support.
  • The DNS has been updated to have full support for DNSSEC.
  • My Let’s Encrypt Certificates now has automated certificate renewals and I’ve upgraded to CAA support.
  • The Webserver has been switched from Apache to NGINX.
  • The PHP has been switched from PHP 5.6 series to a modern 7.0.
  • I’m adopting full Git-backed backup of all server setup and configuration using BitBucket.org. It’s not complete but most config files have been added and managed using GitHub.

These was the majority of changes on the site and server the past few months. With these updates in place, I might get back to producing content for the site.

Devops: You build it; you run it… sort of

DevOps seems to be sweeping through IT departments these years and for most developers it seems to be sen as a way of getting those pesky gatekeepers from Operations away and ship code whenever any developers feels like it.

The problem is however, that in the eagerness to be a modern DevOps operation, the focus is often solely on the benefits of faster releases (on the short term) the “DevOps” provide over “Dev to Ops”, and many developers do seem to forget the virtues Operations (should) bring to the party.

Have your IT systems joined Social Media?

No, your servers should (probably) not have a facebook profile, nor should your servicebus have a twitter profile, but as the work tools change and evolve, you should probably consider updating the stream of status mails to more modern “social media” used at work.

When you’re in DevOps you probably get a steady stream of emails from various systems checking in. It may be alert emails, health checks or backup completed emails. It’s been more “fun” getting these mails with the rise of unlimited mail storage and powerful email-search tools should you ever need to find something in the endless stream of server-generated mails.