Have your IT systems joined Social Media?

No, your servers should (probably) not have a facebook profile, nor should your servicebus have a twitter profile, but as the work tools change and evolve, you should probably consider updating the stream of status mails to more modern “social media” used at work.

When you’re in DevOps you probably get a steady stream of emails from various systems checking in. It may be alert emails, health checks or backup completed emails. It’s been more “fun” getting these mails with the rise of unlimited mail storage and powerful email-search tools should you ever need to find something in the endless stream of server-generated mails.

As we’ve adopted new tools, the automatic messaging from the servers has more or less stayed the same – until recently. We’ve been drinking the Kool-Aid with Slack, Jira and other fancy tools, and recently we considered moving the IT systems along… and so we did.

Slack has a very nice API and even with free tier, you can enable a “robot” (a robot.sh shell script that is) to emit messages on the various slack channels. We’ve integrated the slackbot into our various (automated)workflows in our Continuous integration / Continuous Deployment  pipeline, so that releases which move from environment to the next – and finally into production, emits a message to a #devops channel . We’ve also made a #operations channel, and when our monitoring registers “alert events”, it emits messages onto this channel. Using this setup anyone on the team can effectively and easily subscribe to push messages.

As a sanity measure and not to have “yet another mailbox”, we’ve tried to preach that everything on Slack should be considered ephemeral (that is short lived), and if anything worth remembering is said, it must be captured elsewhere.
releasetrain
As many other companies we use Jira to manage our software development. A task is tracked from idea to delivery and then closed and archived. As a Jira task is long-lived we’ve also integrated the same CI/CD pipeline into Jira. Once the release train starts rolling – a ticket in Jira is created (by the scripting tools through the Jira API) and updated as it pass through the environments – and closed automatically when the solution has been deployed to production.

The ticket created in Jira contains a changelog generated from pulling the contained commits from Git included in the pull request and if possible (assuming the commit comments are formatted correctly) linked to the Jira issues contained in the release (build from the pull request).

The jira tickets created by the release train is collected in a kanban board where each environment/stage have a separate column giving a quick overview of the complete state of releases (what is where right now).

A future move we’ve considered was if we should have blogging by servers. Assuming agile developers is able to create reasonable commit comments which may be readable (and comprehensible) by non-developers, it might me interesting utilizing a blogging tool such as wordpress to provide a historical log of releases.

As you adopt new tools for communication, remember to also think of automated communication, which may have a useful place in the new tools. Often new platforms have readily available APIs which allows your IT platforms to provide – or receive -information much more efficiently than pagers, email or whatever was available once you set it up in times gone by.

(This post originally appeared on Linked)

No more pets in IT

Remember the good old days, when IT got a new server. It was a special event – and naturally the naming game. Finding that special name for the server, which neatly fitted into the naming scheme adopted (be it Indian cities, Nordic mythology or cartoon characters).

This ought to be then, but the ceremony still happens regularly in many IT departments, where servers are treated with the same affection as with pets – and with all the bad side effects it may bring along.

Remember the “superman” server – it must not die – superman will live on forever- nor matter how much patching, maintenance and expensive parts replacements it need, we will care for that special pet server… and we would be wrong to do so.

Modern IT should not be that cozy farm from the 1950ies, but find their reflection in modern farming.

From pets to cattle

In a modern farm the cattle isn’t named individually and harsh at may seem – when one of the cows doesn’t perform, it replaced as the performance of the farm as a whole matter much more, than the individual care and nurture of the individual animals on the farm… and rarely are the individual animals named – perhaps in recognition that they will be replaced and the number of animals will be adjusted to align with the requirements of the farm.

Modern IT does have all the technology to adopt the modern farm metaphor and should do so as we move to virtual servers, containers, micro services and cloud-based infrastructure.

All these technologies (along with others) have enabled us to care much less about a specific server or service, and instead focus on what “server templates” are needed to support the services provided by IT – and mange the number of instances needed to support the requirements posed to IT.

Hardware as Software – From care to control

As servers move from a special gem to a commodity and we may need 10, 50 or 100 small servers in the cloud instead of a single huge “enterprise” spaceship in the server room, a key challenge is our ability to manage and control them – and the tools to do that is also readily available.

Using tools like Chef, Puppet or Docker(files) – as the enabler for our server templates from above – developers are able to describe a specific server configuration and use this template to produce as many identical copies as may be needed. Further more, as we’re moving to manage the herd of servers, the server templates should easily be manged using the standard version control software used to mange your source code already.

Using this template model, the developers take control (and responsibility) of making sure the complete stack needed to run the service, is coherent, and the operations can make sure to size (and resize) the resources available as needed.

Finally as we move to a “cattle” perception of servers, no one should ever need to login to a specific server and make changes – it all needs to go through the configuration management tools and tracking changes all changes to the production environment. If a server starts acting up, kill that the server and spin a new server up in your infrastructure.

(This post originally appeared on Linked)

Ephemeral Feature Toggles

At its simplest  feature toggle is a small if-statement, which may be toggled without deploying code, and used to enable features (new or changed functionality) in your software.

In an agile setup most developers love  having feature toggles available, as they often allow for a continuous delivery setup  with very little friction and obstacles. While this is true, it often seems developers forget to think of feature toggles as ephemeral, and doesn’t realize what a terrible mess, this can cause – given they don’t remove the toggles once the feature is launched and part of the product.

While feature toggles often is an extremely lean way to  introduce public features in software – short circuiting normal release ceremony, as it has already happened before the launch of the feature – which often literally is a button press in a admin/back end interface.

Feature toggles must be ephemeral when used to support Continuous Delivery.

Introducing feature toggles in source code is often just allowing an if-statement and toggling between two states. While it ought to be a single place a toggle occur in the source code, it may often be in views, models, libraries and/or templates – and thus leads to (potentially) a lot of clutter in the source code.

If feature toggles are allowed to live on, the next issue is often that feature toggles may become nested or unintentional crossed as various functions, components in the code, may depend on other features.

Feature toggles is a wonderful thing, but all toggles should not have a lifespan beyond the two sprints forward – allowing for the wins of the feature toggles, yet avoiding long term clutter and complications in the source code.

Other uses for feature toggles…

Feature toggles may exist as hooks for operations or to define various class-of-service for your users.

When used for operations, they may be used as hooks to disable especially heavy functionality during times of heavy load – black Friday, Christmas or whenever it may be applicable.

When used for Class-of-Service feature toggles can be used to determine which feature a basic and a premium user – or any other classes you may choose to have – have access to.

If you’re applying feature toggles as Operations hooks or class-of-service, don’t make them ephemeral, but have your developers mark them clearly in source code.

Baking Audiobooks with m4baker

Building audiobooks on (Debian) Linux in the m4b format is actually possible and doesn’t have to be a pain. I’ve found numerous recipes with shell instructions, but having a nice simple app to handle the building of the books seems much easier.

Most of the apps available for Linux seemed to be in a pre-alpha state, but after a few experiments I’ve settled on m4baker, which – while a bit rough – actually seems to do the job just fine.

Getting the m4baker running on my Debian Testing took a few steps:

sudo apt-get install python-qt4
 sudo apt-get install libcanberra-gtk-module
 sudo apt-get install faac
 sudo apt-get install libmp4v2-2
 sudo apt-get install mp4v2-utils
 sudo apt-get install sox
 sudo apt-get install libsox-fmt-mp3

Once these steps have completed successfully the final step is getting m4baker installed and running:

  • Download the source from https://github.com/crabmanX/m4baker/releases
  • Unpack the file and from the unpacked directory run the install script:
    python setup.py install --optimize=1
    

This should have successfully installed M4Baker and all the required files and libraries to build m4b-audiobooks (suitable for iTunes and other m4b-supporting audio players).

You launch  m4baker either through the (start) menu or simply with the m4Baker command from the shell.

m4Baker is an open source project available on GitHub.

 

Why DevOps works…

I’m digging through a backlog of podcasts and the gem of the day goes to SE-Radio podcast. In episode #247 they talk about DevOps and while I’ve preached and practiced DevOps for years, as mainly as common sense, the podcast did present a more reasonable argument why it works.

Developers are praised and appreciated for short time to market; the number of new features they introduce and changes they make to the running system.

Operations are praised for stability and up time, and thus change is bad; many changes is really bad.

A DevOps fuse the two roles and responsibilities, the developers need to balance the business value new development may cause with the risks it introduce and balance what and when things are released into production.

If you’re into Software Engineering and DevOps curious give it a listen at SE-Radio #247.

(This post originally appeared on Linked)

Get your DMARC going

Get your company implementing DMARC now…

During the past 5-6 years email industry efforts have been pushing the DMARC standard along. It provides the best widely supported and seemingly efficient way to – as a domain-owner – protect the domain from misuse and abuse in terms of spam and phishing attacks.

postkasse_japanAs sending email has often been a wild-west, and knowing who is a valid sender of email may prove a challenge for many companies – and as most IT developers does seem to care too much about the finer details of email (and production just as bad email headers as HTML markup 🙂 ), implementing DMARC protection on your domain may actually be a challenge.

The DMARC standard provide you 3 powerful tools:

  • Using DMARC you have the power (through) DNS to declare which mail-servers are valid senders of email from your domain.
  • The DKIM signing of mails allows your to prove to recipients it was sent from a valid server.
  • Finally DMARC  provides a way for the email receiver to report back to the sender about messages that pass and/or fail DMARC evaluation.

In summary, you have the option to protect the credibility of your domain (by not exposing it to spam and phishing), and you should care now, as Google through Gmail seems to be starting to push harder to signal which email is “safe” (or legitimate at least).

This latter effort will not only remove fake emails pretending to be from your domain, but it will likely also promote your legitimate emails and make them more likely to reach their audience.

Here are a few articles on how to get on with DMARC implementation:

 

 

How not to become the maintenance developer

As a developer it seems, you always seem to strive towards producing ever more complicated code. Utilizing new frameworks, adopting an ever evolving “convention before configuration”, pushing object-oriented programming – maybe Domain Driven Development – are practices introduced, refined and explored in the quest to prove yourself as a steadily better developer with rising skills.

Workstation

Yet to what point?

While the intricate complications may impress fellow developers, doing so often digs a hole which may be pretty hard to get out of. Every complication – not matter if it is in the design, the architecture or the structure of the code – often provides the opposite effect of the desired outcome. The whims of your current self to impress developers with the latest fashionable technique, is short termed reward for a long term pain.

I accept some fashions and whims do change and solidify to become better practice, often the urge to over use the latest and greatest frameworks, techniques and paradigms, often only leads to a painful maintenance in years to come – and as you’ve complicated things beyond comprehension you’ll probably be the trapped in the maintenance for the lifetime of the developed code.

Keep It Simple

As new development challenges often are much more fun, than maintaining legacy code, there are a few basic things you can do to keep yourself from being the eventual eternal maintenance developer and they are straightforward:
Keep it simple
When solving a problem, make as simple and as little code as needed. Don’t wrap everything in object hierarchies, split configuration, views, models and controllers in individual files – do it only, when it provide clear and apparent value.

Names matter

Choose short, but descriptive names for variables, functions, classes and objects in your code. An object containing content for an invoice should be named $invoice, not just $x, $y or $z. Using naming of artifacts in the code provide clues to content and functionality makes it much easier for anyone – including your future self – to comprehend and understand the code when doing maintenance.

Don’t fear comments

Comments does not slow down your code. Al code is compiled at the latest at run-time and a few well placed comments may often be very valuable for the eventual maintainer. Don’t comment obvious code constructs (“don’t state the obvious”) , but do reference business rules or why something tricky is going on.

Be consistent

Find a consistent way to solve the same task/pattern every time, as it will help you focus on what the code is doing instead of the syntax expressed. If you during development find a better way to do something, remember to go back and fix the instances where applicable.

Move on…

Every developer will eventually be stuck with some development, but making your code accessible and easy to understand – the odds of you being stuck forever on maintenance duty is much lower. Great code move on from the parent and have a life on it’s own in the care of other developers if you’ve done your job right.

Are you ready for transparency?

Running a modern IT platform is rarely an easy nor isolated task. Most platforms consist of a fairly large number of components ranging from OS level to 3. party libraries and components added in the user interfacing layers – and adding numerous integrations does make it an interesting challenge to quickly identify and correct bugs and errors.

While the system complexity does pose a challenge is surely not an impossible task, as several tools exists for most – if not all – platforms to allow instrumentation of the platform and utilize the instrumentation tools to handle the platform and identify issues quickly.

Instrumentation provides insight…

Instrumentation tools are generic tools which allows you to identify errors in your production environment and provide sufficient context and debug information to allow developers to diagnose, understand and fix the issues identified. Examples of such tools include AppDynamics, New Relic, Stackify and many others. If you’re the Do-It-Yourself type, it’s not unfeasible to build a tool yourself by hooking into error handlers and other hooks exposed by the specific platform due to be instrumented.

Having worked with various degrees of instrumentation for 10+ years – homebuild and purchased tools, I can certainly confirm that such tools works and allows you to mature a complex IT platform much quicker, as the insights provided from a live production environment allows you to attack the most occurring errors experienced by real users of the system.

Test suites are great for minimizing risk during development, but the test suites are based on assumptions on how users and data acts in your platform,  and while the identified errors experienced over time certainly help minimizing risks in new development, it is “theory” as opposed to instrumentation which is much more “practice”.

Heimbach - power plant 12 ies

 

Transparency not needed

While the tools to do instrumentation for most platforms may readily be available, the “natural use”  – even in an enterprise setting – seems surprisingly low, and I suspect numerous reasons exists.

We do not need it is often the most common. As set procedures exists and they seem to work, why would we need to introduce a new tool to provide data we already have. Error logs, end-user descriptions and screenshots have been used for decades and why should there be a better method?

It introduces risk is another often cited concern. As instrumentation tools are not considered a need tool in the IT platform, operations may oppose to adding it to the already complicated stack – especially if the value of the instrumentation is not known or recognized.

It is expensive is another misconception. Instrumentation often don’t provide any direct business value (assuming your IT platform isn’t burning and the users is leaving rapidly). Most of the value offered by instrumentation tools is fixing issues faster and the scope of issues  being smaller, and as such it’s often hard to prove the value offered by issues not occurring.

Transparency not desired

Apparently many people believe firmly, that issues not seen nor reported are not real issues, and does not exist. Gaining insights into one instrumented platform and running a black-box platform next to it, may  cause the false belief that the black box system is running more stable and with fewer issues than the transparent system.

The reason is simply that on black box systems (that is systems without any instrumentation tools to monitor their actual performance) it is rare to proactively examine logs files and other places where the black box might emit issues. Only when an issue is reported, developers are assigned to examine these sources to resolve the issue.

Gaining insights into an IT platform though instrumentation and being able to resolve “real” issues as experienced by your users should be a fantastic thing, but beware that many people implicitly seems to believe, that with you don’t monitor for errors and issues, they probably doesn’t exist – however false it is.
See No Evil, Hear No Evil, Speak No Evil

Accidental Architecture

Most IT departments have the best intentions of providing the best quality, coherent solutions, yet market conditions, projects running in parallel and various constraints on budgets, resources or time, often causes what might be defined as Accidental Architecture.

The easiest way to identify cases where you’ve been hit by accidental architecture is when describing your it architecture and look for the works “except” or “but not”. Typical examples include – we have a single-sign system utilized everywhere except…”, “We update all systems to current versions, but not this old system…”.

The accidental architecture seem to be caused by a few main drivers:

  1. Lack of overview
  2. Lack of time
  3. Lack of resources

Lack of overview

When the root cause is lack of overview, the case is often, that decisions and designs are implemented without understanding the entire scope of the problem – or existing procedures and architecture in place. While a good coherent architecture seems to have been designed, it turns out that existing complexities which wasn’t known or addressed causes issues.

Lack of time

Deadlines often seem to be the cause of many issues – and no matter who much time you may a assign to a project, you’ll often need just a little more. As a deadline approaches often shortcuts a made to make the deadline and the shortcuts – which is assumed to be fixed in the next version, but often forgotten and abandoned – until issues arise.

Lack of resources

The issues caused by lack of time and lack of resource may seem similar, but are different. When lack of time causes the issue, the problem could have been solved, whereas lack of resources often happens when budget constraints or lack of knowledge causes an architecture to be chosen which may not be the right solution of the problem at hand.

The lack of resource issue may often occur, when projects are expected to drive enterprise changes – merging billing systems, decommissioning legacy platforms and other issues, which should be done, but often a product development project may not be able to facilitate.

The first step, is to realize there is a problem…

While many organizations fail to realize the existence and the volume, most actually seem to have a fair share of them – and if not handled – probably a growing volume.

Once you’ve realized you have cases of accidental architecture, make sure you address them on your technical debt list and have plan for what to do about the system. While “pure” technical debt most often may cause operational issues, the accidental architecture usually cause customer facing issues and are not recognized as severely as the operational issues caused by technical debt.

The issues introduced by accidental architecture is often complexity, slowly rising operational costs and increased user-support costs. To keep your IT domain alive and moving forward, time and resources must be found continuously to address and resolve the accidents.

Three points on the costs of COTS

It seems to be quite popular to move away from custom build IT solutions to so called COTS – commercial of the shelf solutions. The idea being, that to software solution fulfil a functionality which long has been commoditized and standardized to such an extent that it offers no “competitive edge” nor core value to the business.

For most companies and organizations the office suite would be a pretty safe bet for a piece of software which is magnificently suited for a COTS solution. Finding someone who develops an internal word processor in-house seems crazy as so many fully capable solutions exists in the market.

As time passes more software seem to be included in the parts which may be commoditized and custom solutions be replaced by standard solutions to provide an adequate and capable solution to areas served by custom solutions.

The drive to COTS software seem to be a hard challenge to many organizations, as the primary driver in most COTS adoption projects seems to be a drive from the accountants and a mistrust to the IT department to choose and deliver the best fit solutions to the rest of the business.

When listening for failed Microsoft Office implementations it sems fairly small, yet the number of failed ERP projects seem endless. The scope of this post is not to address when nor how to choose COTS solutions, but just make the point, that the choice of COTS is often naive and not fully understood ahead of the decision itself.

  • When adopting COTS you’re tied to the options and customizations offered by the chosen COTS software. You should never expect to be able to force the solution to be adapted to your organization and processes, but instead be prepared to adapt the organization and processes to fit within options offered by the chosen software.
  • Choosing COTS is a strategic commitment to the vendor of the software within the scope the COTS solution is adapted to fit within the organization. Once implemented within an organization, the adopting organization is often committed to follow the roadmap and direction the vendor chooses – as the cost of switching to another solution often is large and challenging project.
  • When adopting COTS you’re committing to follow along. All versions of software has a limited “life cycle” and as new versions are released you’re expected to follow along – in the pace that’s suitable for your organization and within the roadmap offered by the vendor (in terms of support and upgrade paths).

While COTS software seems like a cheap and easy solution to many areas within an organization, the three points above seems to be forgotten too often and causes problems with the stand COTS solutions again and again.

Coming back to Microsoft Office it seems all organizations are more than capable to restrain within the possibilities offered by Word, Excel and “friends”. As the Office documents seems to be the standard exchange format, there is an implicit drive to move the organization to current versions of the software and the new options offered by new versions.

When COTS implementations fail it seems often seems, that organizations are unwilling to adopt within the options offered by the COTS software chosen – and thus breaking the core idea of COTS as a commoditized solution.

It also seems many organizations seem to forget the commitment to follow the COTS vendor, and often end up using software versions dangerously outdated, as no budget exists to update or too many customizations have been made (see paragraph above) to make it easy to upgrade to current versions.

While COTS may offer solutions to many areas in the organization, please be warned – there is no free lunch. COTS does not only come with an initial implementation price – it also comes with commitment.

Pioneering the Internet….