Category Archives: Linux

Linux – as a desktop, as a server or any other way

Bulk conversion of webp files to png format

Google has come up with a nice new image format called webp. Currently support for this format is fairly limited, so if you need to use webp images else where it might be nice to convert them to a more widely supported format. To do the conversion, Google has made a small tool available called dwebp. The tool however does only seem to support conversion of a single image, not a batch of images.

Using regular command line magic it’s easy though. Download the tool a pair it with the find and xargs command and you should quickly be on you way. If all the webp files needing conversion to png is in a single directory, simply do this:

find . -name "*.webp" | xargs -I {} dwebp {} -o {}.png

It findes all webp files, and converts them one by one. If the initial files name was image.webp the resulting file will be called image.webp.png (as the command above doesn’t remove the .webp but only appends .png at the end.

The command assumes the dwebp program is available in you include path. If this isn’t the case, you need to specify the complete path to the program.

Beware of DNS

For some time the server running this site had been acting up. Page loads were slow, access through SSH seemed lagging and something was absolutely misbehaving.

I’ve been trying to figure out what exactly was going on, but nothing really made sense. there were plenty of disk space, memory was reasonable utilized (no swapping) and the CPU load seemed to be less than 0.1 at any time – there were no good reason the server was “turtling” along at such a perceived slow pace.

Thanks to a tip from Henrik Schack, the server is now running at full speed again. it turned out that one of the DNS resolvers used by the machine was in a bad state and slow, unreliable or dysfunctional DNS causes trouble all sorts of places. The fix was quite easy, the file /etc/resolv.conf was updated to contain the IPs of the Google Public DNS servers, and once the file was saved things were back to the rapid normal.

All computers really need solid, fast DNS servers these days – be it servers or workstations – as the auto-updates and the utilization of “cloud-resources” of various kind much have DNS working to reach out to the destinations they need. If your system starts acting up without any reasonable explanation, checking DNS could be an easy place to start checking things.

Sending mail from a droplet

As stated earlier this site is now running on a DigitalOcean droplet. A droplet is basically the same as having a “real server”, and when running a bare bones machine, it isn’t born with the ability to handle email – receiving nor sending. As a number of web apps require the ability to handle mail, I had to setup facilities on the server (or droplet) to handle mail.

The “default” way to do this would probably be to install sendmail or postfix, as they are full-featured mail server, but configuring a mail-server, keeping it secure and updated is a nightmare I’d like to avoid. Therefore it was time to look for another option.

Enter msmtp

msmtp is an open-source, light-weight solution, which allows you to get your server to send email, or as the project itself describes it:

In the default mode, it transmits a mail to an SMTP server (for example at a free mail provider) which takes care of further delivery.

msmtp project homepage

There are several ways msmtp can be setup, but in this post I’ll just cover the two basic scenarios.

Configuration

msmtp can handle mail delivery different ways. I’ll just cover two basic scenarios here.

If you have a smtp-server available. Your hosting provider or someone else may provide you with access to a full-featured SMTP-server. If this is the case, you can configure msmtp to pass all mail on to that server like this:

# smtp server configuration
account  smtp
host   smtp.example.com
from   example@example.com
port   25
 
# Default account to use
account default : smtp

As you’re talking to a “real” SMTP server all options and features should (potentially) be available to you.

If you have a Google account – either a regular Gmail account or Google Apps account will do just fine. To configure msmtp to use the Gmail SMTP server use this configuration:

# Gmail/Google Apps
account  gmail 
host   smtp.gmail.com 
port   587 
from   example@gmail.com
user   example@gmail.com
password  enter-password-here!
auth   on 
tls   on 
tls_trust_file /etc/ssl/certs/ca-certificates.crt 
 
# Default account to use
account default : gmail

In the above example you need to change “example@gmail.com” to an actual GMail account, and you need to change “enter-password-here!” to the password belonging to the specified Gmail addresss.

Using Gmail, all mail passed on from msmtp, will be sent from the account credentials used in the configuration, and there doesn’t seem to be a way to override this. You may therefore opt to create a specific mail-account for this use. You can set a custom Reply-To header in the mails passed through Gmail SMTP, which in many cases may help secure the replies get to a proper recipient.

If your site has adopted DMARC, this may not be a suitable option (at least not on the free tier), as they don’t support signing and do not offer dedicated IP-addresses for you SPF-records.

Testing 1, 2, 3…

Once you’ve set up the mstmp configuration file, it’s time to do some testing. Create at text file called “testmail.txt” with this content:

To: example@example.com
From: example@example.com
Subject: Subject for test mail
 
This is the body content for the test mail.

Change example@example.com to your own actual email address. Then enter from the command line:

cat testmail.txt | msmtp example@example.com

You should recieve your test mail shortly.

Setting up an alias

Many unix/linux tools and apps seems to assume, that you have sendmail installed and that it is available at /usr/bin/sendmail or a few other locations in the file system. To handle these cases easily, you can create an alias pointing the sendmail name to the msmtp binary like this (the examples should cover most cases):

ln -s /usr/bin/msmtp /usr/sbin/sendmail
ln -s /usr/bin/msmtp /usr/bin/sendmail
ln -s /usr/bin/msmtp /usr/lib/sendmail

Depending on which package manager your installation use, it may automatically setup these aliases, so do check if they exist before trying to create them.

Setting up with PHP

if you made the aliases as suggested above, it may already work, but you should make the following changes, just keep things clean and transparent.
Find all php.ini files applicable (you probably have one for the web-server and another for the Command Line):

Add or change the line:

sendmail_path = "/usr/bin/msmtp -t"

Now for some testing. Add a file with the following content (change the example-address to your own):

<!--?php mail("example@example.com","test","test","-fexample@example.com"); ?-->

Now, call the file from the command line using the php cli, and then call the file through the webserver. In both cases you should receive an email shortly.

 Another suggestion…

Apart from running sendmail or postfix, there also seems to an application similar to mstmp called ssmtp, which offers many of the same features as msmtp.

Server setup: Setting up a firewall

A firewall is a basic filter that can provide an efficient protection to your server by only allowing the traffic in and out as the rules of the firewall allows it. Setting up a firewall on a Ubuntu Linux server does not need to be complicated – in fact the one used in this example is called “uncomplicated firewall”.

To get the firewall up and running make sure it’s installed through the package manager. Login and switch to a root shell, then install the firewall with this command:

apt-get install ufw

If everything goes okay, the firewall is installed but not configured nor enabled.

Firewall Configuration

I find the easiest way to mange the firewall is through a little script in the root home directory. The beginning script could look something like this:

1
2
3
4
5
6
#!/bin/sh
ufw reset
ufw allow from 127.0.0.1
#ufw allow ssh
ufw enable
ufw status

Line 2 resets any existing configuration rules in the firewall.

In line 3 you should change the 127.0.0.1 to you own fixed IP address if you have one (you really ought to). This line will allow any traffic from you ip-number into the server (assuming there is something able to receive it naturally).

If you haven’t a fixed IP number line 3 should be removed and line 4 used instead. It allows SSH connections from any outside IP-number to knock on the door – then well rely on the SSH daemon (and the configuration of this) to reject any unwanted visitors knocking on the server.

Line 5 enables the firewall and line 6 prints a list of the current status and configuration of the firewall.

Depending on what you are using your server to do, you’ll probably need a few more lines in the firewall script. If you’re running a webserver, you should at least add a line (just above the “ufw enable” line) allowing web traffic to pass through the server:

utf enable www

Are you using https on you’re webserver? – then you need to allow that too:

utf enable www

The simple enable lines above are suitable for “publicly accessible services”. If you’re running something the whole world should be able to use, UFW allows for that too. The Community documentation on UFW over at the Ubuntu site is quite helpful.

Server setup: A user account

So, I’ve been moving the site to a VPS – a Virtual Private Server. A VPS is basically the same as a physical server to which you can’t have physical access. When you get your virtual server, most likely it will be setup with a basic disk image with an Operating System and a root account. In my case at DigitalOcean I choose to setup an Ubuntu Linux image and here are the first moves you should take after creating the VPS to get the basic security in place.

Setting up a user account

At DigitalOcean the server images is deployed and once it’s ready you get a mail with the root password. Letting root login over the internet is pretty bad practice, so the first step you should do is login (over SSH) and setup a new user. Creating the new user is done with the adduser command and follow the instructions, then start visudo to grant your new user some special powers:

adduser newuser
visudo

In the visudo file you want to add copy of an existing line. Find this line:

root    ALL=(ALL:ALL) ALL

… and make a copy of the line. Change the “root” to your newly created login name to grant you new user the right to become root.
Save and exit the file. Check out can be come root from you new account (first switch to the new user with the command “su – newuser” (change newuser to you new username), then try to switch back to root by writing “sudo su -” and enter the password to your new user account (not the root password, and surely you didn’t use the same right?). If this success enter “exit” twice to get back to the initial root shell. The new account is setup and has the rights to become root.

Setting up SSH

Next step is preventing root from login in from remote locations (we only want the newly created account from above to be able to login remotely and then change to root if needed).

Setup the .ssh directory

Assuming you have an existing SSH key set start up creating a “.ssh” directory in you new users directory.
Add your public key to the directory (it’s probably called “id_rsa.pub”) and name it “authorized_keys”.

Make sure…

  • the .ssh directory and the file in it is owned by your newuser-account (not root).
  • the directory is set to 0700 and the file to 0600 (using the chmod command).

You should now be able to login to the “newuser” account remotely using SSH.

Reconfiguring the SSH daemon

Asuming your new account is setup and able to login from remote with SSH the next step should be reconfiguring the SSH daemon to a more secyre setup, open the sshd-configuration file with this command (as root):

vi /etc/ssh/sshd_config

The changes you should make are these two:

PasswordAuthentication no
PermitRootLogin no

The first requires we only allow logins using public-key authentication – no password-only logins. The second denies root to login from remote. If we need root access, we must login with the regular account and then change to root.

Once the changes are med, make sure they take effect by reloading the SSH daemon with this command (as root):

reload ssh

Once this is completed, please move on and setup a firewall.

The emergency hatch

Should you get into trouble and not be able to get back in to your server using SSH, DigitalOcean offers an emergency hatch. If you log into the backend (where you created the VPS) there’s an option to get “console” access to your server. Using this console is as close as you can get to actually sitting with a console next to the machine, and could be the access you needed to fix any misconfiguration or problem preventing you getting in through regular SSH.

Moving the site

This site (and my other site in Danish) have been hosted on a cheap shared hosting site a few years. As shared hosting platforms go, the service and features at GigaHost was quite reasonable, but their servers seemed continuously overloaded and the site had a few issues from time to time. I’ve been moving everything from the shared hosting platform to the smallest available VPS server at DigitalOcean.

Why the move?

  • Performance on shared hosting platforms never seems to amaze.
  • Limited set of features – no shell access, dummy selfcare interface, reasonable features – but limited.
  • Was dirt cheap when I moved in, but not as much – the VPS is actually priced lower.

How did I move the site?

The various parts of the move will probably be described in details in further posts on the site in the foreseeable future, but basically the steps included:

  • setting up an account on Digital Ocean and creating a droplet.
  • setting up a user acount, getting a firewall up and running, securing a few items.
  • installing a webserver and mysql.
  • moving the data from the shared hosting platform (databases and code) to the new webserver.
  • testing everything works by hacking the local hosts-file.
  • redirecting DNS to point to the new site.
  • deleting all stuff from the shared hosting platform once everything has been verified to work as expected.

What comes next…

Running my own server opens a lot of interesting new possibilities. I’m no longer running Apache (which was mandatory previously). Now I’m running nginx which seems much more light-weight.  I’m also running NewRelic which seems to provide amazing insights into how the server resources are utilized.

My first experiments on this server, has been focused on getting the old stuff up and running. You might notice, that the site is running somewhat faster (and I’m still tweaking things).

I expect to be able to use this server to experiment with node.js, ruby and other interesting stuff… and the Comunity help pages at Digital Ocean seems quite amazing.

 

Caution: Here be dragons!

Running your on server (virtual or real) is slightly more complicated than being just another guest on a shared hosting platform. While I do feel reasonable fit on a Linux platform (and run it as my daily desktop), I’ve been blessed with a hints and help from a friend throughout the process which made the move considerably faster (and the settings far more secure from the outset.

I’m sure I’ll run into some trouble along the way – I even managed to -amost – shut myself out of my virtual server once, as I only allowed for SSH access,  but seemed to have deleted all public keys needed on the server to allow my self to get back in.

Linux Mint: OpenSSH Daemon

I’m in the process of reinstalling my work desktop. One of the mandatory packages which I install once the core system is up and running is a SSH Daemon.
Setting it up (on Linux Mint which I’m running) is pretty easy. To install the OpenSSH daemon go to the shell and write:

sudo apt-get install openssh-server

It’s a fairly small install, so in a few seconds it ought to be up and running. Next step is editing the default config file and change a few things.
Editing the config file is done by entering:

sudo vi /etc/ssh/sshd_config

The cofiguration options I usually edit these parameters:

PermitRootLogin no
#Banner /etc/issue.net
 
AllowUsers <username>
  • PermitRootLogin – The default option is yes, but frankly root should never be allowed to login remote unless absolutely needed.
  • Banner – Can allow a custom message be displayed at login (if needed).
  • AllowUsers – A space separated list of users allowed to login remotely.

Once the edits are done and saved, the openSSH Daemon needs to restarted which is done by:

sudo service ssh restart

Which packages are installed (on Ubuntu)

If you’re using a Debian based Linux Desktop such as Ubuntu, Linux Mint – or Debian itself naturally – you can easily create a simple text file of all packages installed on the machine.

I have a habit of removing a lot of the “extras” (junk) which comes with the basic install – not religiously, but just to keep it reasonably tidy and not have too much stuff eating up the harddisk, requiring updates (without providing any value).

Creating a list of the currently installed packages is quite simple – just open a terminal and enter:

dkpg --get-selections

(you will probably want to redirect the output into a file – or at least pipe it through more).

PHP 5.4 built-in webserver & Linux (mint/ubuntu)

PHP 5.4 comes with a built-in webserver, which can be useful for development and quick tests. It easily launched from the command-line, but if you’re running Linux Mint or Ubuntu, the PHP version, isn’t 5.4 but 5.3.x. If you don’t have the time/courage/energy to compile PHP 5.4 yourself, some nice fellow on the internet has done the work and made it available through a package repository which makes it a breeze to install.

To install PHP 5.4 on your Ubuntu or Linux Mint simply do this:

1
2
3
sudo add-apt-repository ppa:ondrej/php5
sudo apt-get update
sudo apt-get install php5

(answer yes to any questions asked).

then you should go to go. Verify the update with:

php --version

.. and the "answer" should be something like:

PHP 5.4.4-1~precise+1 (cli) (built: Jun 17 2012 13:01:09)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies

(version numbers and dates are probably subject to change).

To use the webserver, go to the directory you want to be the document root, and launch the webserver with:

php -S localhost:8000

and you can also add a custom php.ini file with the configuration you want with:

php -c ./php.ini -S local:8000

Please remember, that the built-in webserver is only suited for development, but for a quick hack, it sure beats installing Apache or any other webserver.