The Linux Philosophy for SysAdmins, Tenet 13 — Make programs portable

0

Image by Opensource.com: CC-by-SA 4.0

Author’s note: This article is excerpted in part from chapter 15 of my book, The Linux Philosophy for SysAdmins, with some changes to update the information in it and to better fit this format.

Portable programs make life much easier for the lazy SysAdmin. Portability is an important consideration because it allows programs to be used on a wide range of operating system and hardware platforms. Using shell languages such as bash and Perl that can run on many types of systems can save loads of work.

Programs written in compiled languages such as C must be recompiled at the very least when porting from one platform to another. In many cases, platform specific code must be maintained in the sources in order to support the different hardware platforms that the binaries are expected to run on. This generates a lot of extra work, both writing and testing the programs.

Perl, Bash, and many other shells are available in most environments, and with very few exceptions, can run unchanged on many different platforms.

Intel PC to IBM mainframe

At one place I worked I was responsible for one Intel host that ran Linux and a fairly large Apache web site with an internally built database engine. We wrote a large number of Perl and Bash programs that were used as CGI to generate web pages based on the data retrieved from the database. Even the database software was written in Perl by one of our SysAdmins.

As part of our disaster recover plan, all of the mainframe and Unix programs that we were running — our entire inventory of software — was supposed to be migrated to a recovery service with a location in Philadelphia. The service did not provide Intel based computers so our web site could not be directly supported with identical hardware.

However part of the hardware we had at my organization, and which was also replicated at the disaster recovery site, was an IBM Z series mainframe that could support a large number of Red Hat instances. We decided it would be wise to test our software and see if it could be migrated to an IBM Z series box. We were hoping that we would not need to make too many changes to make it work. I was provided with access to a dedicated Red Hat instance on the Z series mainframe and told to report my results.

I started by identifying the software and related data that would have to be moved. This was easy because we used standard directory locations for files and data as defined by the Linux Filesystem Hierarchical Standard.

It took less than five minutes to create a tarball of the files we needed to transfer, and a few seconds to scp (secure copy) the tarball to the Red Hat instance on the mainframe. I extracted the files from the tarball, started the various servers using a startup shell program, Automate Everything, and started testing. Everything worked flawlessly. Total time from start to finish to transfer and up and running — except the testing itself — twelve minutes.

This was in part due to the fact that our database was in fact a flat ASCII text file, in accordance with tenet 11, Store data in open formats. No magical incantations were required to modify it, convert from one binary format to another, or ASCII to EBCDIC, or export it from one system and import it on another. It just worked.

But this easy migration was also made possible by the fact that we used Perl and Bash which made for portable programs.

Architectures

Linux runs on a number of architectures. Actually quite a lot of hardware architectures1. Wikipedia maintains a long list of hardware architectures supported by Linux, but here are just a few.

Of course Linux supports Intel and AMD. It also supports 32- and 64-bit ARM architectures that are found in practically every mobile phone on the planet and devices such as the Raspberry Pi2. Most mobile phones use a form of Linux called Android. Also supported are Freescale (formerly Motorola) 68K architecture, Texas Instruments 320 family, Qualcomm Hexagon, Hewlett Packard’s PA-RISC, IBM’s S390 and Z series, MIPS, IBM’s Power, PowerPC, SPARC, and many more.

Each of these architectures is different at the hardware instruction set3 level. Each architecture needs different compilers, or at least a compiler that is capable of supporting their respective instruction sets. This in turn means that any programs using a compiled language for any of these many architectures must be recompiled when migrated from one to another. This is a form of portability despite the fact that the programs need to be recompiled.

The sense that I mean here is that programs written by and for SysAdmins should just work when moved from one architecture to another. No recompiling or rewriting should be required. Only shell scripts can do that.

Portability restrictions

When I first heard the term portable in connection with software, it was in the sense of making a copy of a program that could be moved from one computer to another of the same architecture and operating system and run it there. Searching Google results in a large number of hits that all relate to moving software from one Windows computer to another using various techniques including things like running the program from a USB drive that can be inserted into any computer. Other techniques were described a little less clearly.

Licensing

Other results referred to simply installing programs on more than one computer. Vendors can attempt to prevent this for various reasons and in some cases it is quite illegal. End user license agreements (EULA) may explicitly state that you have the right to install and use a program on only one computer. More lenient ones may allow you to install it on multiple computers — with some specified limit — but to only use it on one computer at a time.

I do not intend to become embroiled in a discussion about licensing agreements. But true portability is affected by licensing so some consideration of it is necessary. Sometimes technology is not the restrictive factor in portability.

Technology

However, sometimes technology is the limiting factor in software is portability.

We have already looked at portability with respect to supported platforms. For compiled programs this means that compilers must be available to create binaries compatible with the supported platforms. We have already seen that Linux is supported on a wide range of hardware platforms so there are clearly compilers that support those platforms.

We can say that there is a certain level of compatibility with these platforms, and that code will be semi-portable between them. That basically means that the code can be placed in a single code base, if desired, but that considerations need to be made within the code in order to support targeted platforms. These differences are due to the inherent differences in the hardware instruction sets of each platform.

The good news is that the the GNU Compiler Collection4 (GCC) used by Linux contains compilers for the C, C++, Object C, Fortran, Java and Ada programming languages. GCC can run on more than sixty operating system platforms including Linux, DOS, Windows, many Unix variants, MIPS, NeXT, and a bunch I never heard of before finding the GCC Definition referenced in Footnote 4. We can also see in that document that GCC supports a wide range of processors for which it can compile binary code.

This all means that we have some level of portability in the compiled binary world. The drawback is that code compiled for one hardware platform will not run on a different hardware platform so it must be recompiled. Sometimes significant changes must be made to the code in order to get it to compile. This takes a lot of effort and most developers won’t bother with trying to make their code compile on all or even most of the hardware platforms that they could. They usually pick one or two that together have the most potential customers and don’t go beyond that.

If that source code is open source, then some programmer with a need or desire to make this code run on one of the less common hardware platforms, they can do that. If they do, it will certainly take a good bit of work and knowledge to make it happen.

This is definitely not the appropriate choice for we lazy admins. We must make our code much more portable right from the start and eliminate most of this extra work. Compiled code, rates low on my portability scale because of the amount of work required to move it from one platform to another. It can be done, but I don’t want to do it myself.

LibreOffice

LibreOffice5 is a good example of compiled code that is portable. I use LibreOffice extensively for various projects including writing the book from which I excerpted parts of this article. LibreOffice is available for many operating system platforms including Linux, various Windows releases, Mac OS, and Android. There is even a “Portable”6 version packaged using PortableApps.com7. This packaging enables the application to be used from one’s own USB stick, for example, on any Windows computer.

So LibreOffice is portable in multiple senses. It is also open source so that you can download the source code from the LibreOffice web site and modify it to suit your own needs. Most of us would never do that, but the code is available so we can view it or change it if we need or want to. LibreOffice is distributed under the Mozilla Public License Version 2.08.

Shell scripts

Here we are, back at shell scripts. Why? Because the vast majority of shell scripts work on any hardware platform under Linux. In most cases they will also work on other Unix and Unix like operating systems.

The shell is actually a programming language: it has variables, loops, decision-making, and so on.

— The Unix Programming Environment9

This statement applies to every shell I have ever used. In previous chapters you have already seen how short shell programs can be written directly at the command line in order to facilitate the rapid solution to a problem. We have also covered creating executable files in which to store these ad hoc programs so they will be available in the future and for other SysAdmins who might need the same solution.

I prefer to use bash because it is the default shell for all Linux distributions and it is available for Unix as well. Other shells are also widespread, like ksh, csh, tcsh, and zsh, but they may need to be installed because they may not be by default.

The fact that the Bash shell is nearly 100% compliant10 with the Portable Operating System Interface (POSIX11) standard means that you can expect Bash shell scripts that run on one operating system and hardware platform to also run on all others on which bash is supported. That does not mean that you might not run into some issues. For example the mymotd script that I wrote to display some host system hardware and software data looks for some specific hardware data that might not be available or might be available but in a different manner than our script assumes. The script will run but you might encounter some anomalous results.

Portability with Windows

So far we have concentrated on compatibility within Linux and Unix operating systems. But what about Windows? Although my book and the Linux Philosophy for SysAdmins is about the Linux environment, this article would not be complete if we did not look — however briefly — at Windows, too.

As discussed above, it is possible to create source code that can be compiled on Linux, various versions of Unix, Windows, and other operating systems. It takes a lot of work to do this, but it can and has been done. The real question is how can we run our shell scripts on both Linux and Windows.

There are a couple ways to provide script portability between Linux and Windows.

Cygwin

Cygwin is a free open source product that can be downloaded and installed on your Windows computer. Cygwin supports Windows Vista and later and installs a very flexible Linux environment and a nearly complete set of programs, utilities and desktop environments that have been ported from Linux and the GNU Utilities.

It is possible to use Cygwin to install bash, tcsh, other shells, the KDE and other Linux desktops, and many Linux utilities that we SysAdmins have become accustomed to. Not only is it possible to have a Linux experience on Windows but also our bash and other scripts are now portable to Windows. The Cygwin environment even extends to imposing the /dev directory and the usual device special files that we expect to find on any Linux host.

This portability does have its limits, though. For example hardware and operating system specific functions may not work correctly. Therefore, it may be necessary to add some code to shell scripts to determine the operating system environment and act accordingly to allow for the differences. This is nothing new and has been done between different Linux distributions and between Linux and various versions of Unix. Adding a little bit of additional code to a script to allow it to run on multiple operating systems is a very easy way to ensure a higher level of portability.

In other cases bash scripts may port in the sense that they will run, but it would make no sense to so so. For example, the post-installation script I have written to handle all of tasks that the Fedora Linux installation does not would run but it would generate many errors.

I have spent a little time to install and learn a bit about Cygwin but I don’t normally use Windows except in a VM for a few tests. The Cygwin bash shell is familiar and offers a good opportunity to use Linux commands and scripts that are not operating system dependent.

PowerShell

Microsoft released the first version of their PowerShell12 in 2006. In January of 2018, they made PowerShell available under the MIT license13. The code itself is now available for many platforms including Linux. PowerShell is an object oriented scripting language and shell that is – among other things – intended to provide script portability between the Windows and Linux platforms.

I have not used PowerShell although I have played with it just a bit to see what it’s about. It is very different from any of the Linux and Unix shells I have used. I suspect if I spent some time with it I could learn to use it as well as I use the bash shell. With all I still have to learn about Linux I am probably not going to use PowerShell myself. However, if you need script portability between Linux and Windows operating systems, you should definitely check it out.

Creating web pages

Here is something a little different. We have been looking at shell scripts in terms of running them from the command line. What happens when we use other programs to run our scripts?

Back in the stone age of the Internet when I first created my first business web site, life was good. I installed Apache and created a few simple HTML pages that stated a few important things about my business and provided information like an overview of my product and how to contact me. It was a static web site because the content seldom changed. Maintenance was simple because of the unchanging nature of my site.

CGI (Common Gateway Interface) code can be used for a very dynamic web site. In this context, dynamic means that the HTML needed to produce the web page on a browser was generated from data that could be different every time the page was accessed. This includes input from the user on a web form that is used to look up data in a database. The resulting data is surrounded by appropriate HTML and displayed on the requesting browser. But it does not need to be that complex.

Using CGI scripts for a web site allows you to create simple or complex interactive programs that can be run to provide a dynamic web page that can change based on input, calculations, current conditions in the server, and so on. There are many languages that can be used for CGI scripts. Perl, Bash, PHP, and Python are all excellent choices for CGI.

Although the most common way we think about scripts is in terms of running them from the command line, they can also be used with other software to perform some very interesting tasks. CGI scripts written in common languages are a fine example of this.

Because the languages used to create our CGI programs are supported on many operating systems, these programs are portable. You may need to install bash on a Windows web server, but that too is possible. Other languages like Python and PHP, for example, can also be used to generate dynamic web pages and, along with Perl, are easily available on most platforms, both operating system and hardware.

WordPress

WordPress14 is a powerful open source program that allows creation and management of web pages. WordPress is written in PHP so is easily portable to any platform that runs PHP15.

PHP is a programming language especially well-suited to writing dynamic web pages. I sometimes forgot to install PHP because it is not always installed by default. I added the PHP installation to my post-installation script so it will always be there. But if you have trouble running WordPress, check to ensure PHP is installed.

WordPress is extremely flexible because it uses themes to generate the look and feel of a web site. By changing themes it is possible to change how a web site looks in a few seconds with a few mouse clicks. I use WordPress on all of my web sites because it is so easy and flexible. I have even taught non-technical people how to use the word processing like interface to create new web pages and posts.

Although many aspects of WordPress themes can be altered through its own web-based administration interface, some things require working directly with the CSS style sheets that, along with theme specific PHP code, define the look and feel of each theme. It is possible to work with the CSS through the WordPress interface, but I find using Vi or Vim in a terminal session works best for me.

Before I modify anything, however, I always make a new copy and leave the original intact. I usually rename the copy to something like, “my-wordpress-theme” to differentiate it from the original. Then I use the WordPress admin interface to switch to my new theme. Now I can modify the new theme and not need to be concerned that updates to the original theme will wipe out my changes.

Of course I can modify the CSS to change things like colors and fonts. I can also modify the PHP code for the themes, as well, in order to change the page structure a bit. I have done this on a few occasions when the theme needed a bit of tweaking. I have also modified the PHP code for a theme when installing some of the many plug-ins available for WordPress.

The only reason that all of this is possible – in terms of both portability and the ability to change anything about it – is that WordPress and the themes available for it are all open and accessible. The files that make up this application are all stored as ASCII text files. And it is open source, which means that the GPLv216 license under which WordPress is distributed allows all of this.

Final thoughts

Portability is a range of values and not just a binary response — yes, it is portable or no, it is not portable. And portable as they are, shell scripts still may need to be tweaked in order to produce the desired results when run on different operating systems and hardware platforms.

Portability is a key ingredient in reducing our workload as SysAdmins. Writing code that is portable – or at least as portable as possible – is an excellent method for only needing to do a job once. Why write code for several different platforms when it can be done once to run on all of those platforms with shell scripts?

Command line scripts are where we spend most of our time as SysAdmins and making these portable is important. Fortunately most shell scripts, especially those written in shells like Bash, are high on the portability scale. Writing portable CGI code for web sites we manage is another good step, where it is applicable.

More time saving comes from using open source code that is portable and has been tested and created for many environments. We looked at WordPress as one example of this. Just because we can write our own amazing CGI scripts to drive a web site does not mean that it is efficient to do so. WordPress is already written, it is open source, and it does a great job. And if you don’t like WordPress there are many other options available as well.


  1. Wikipedia, List of Linux-supported computer architectures, https://en.wikipedia.org/wiki/List_of_Linux-supported_computer_architectures ↩︎
  2. Raspberry Pi Foundation, https://www.raspberrypi.org/ ↩︎
  3. Free On-Line Dictionary of Computing, Instruction Set, http://foldoc.org/instruction+set ↩︎
  4. The Linux Information Project, GCC Definition, http://www.linfo.org/gcc.html ↩︎
  5. LibreOffice, Home Page, https://www.libreoffice.org/ ↩︎
  6. LibreOffice, Portable Versions, https://www.libreoffice.org/download/portable-versions/ ↩︎
  7. PortableApps.com, Home page, https://portableapps.com/ ↩︎
  8. LibreOffice, Licenses, https://www.libreoffice.org/about-us/licenses/ ↩︎
  9. Kernighan, Brian W.; Pike, Rob (1984), “3. Using the Shell”, The UNIX Programming Environment, Prentice Hall, Inc., ISBN 0-13-937699-2, 94 ↩︎
  10. Newham and Rosenblatt, Learning the bash shell, (O’Reilly 1998), ISBN 1-56592-347-2, 248. ↩︎
  11. Wikipedia, POSIX, https://en.wikipedia.org/wiki/POSIX ↩︎
  12. Opensource.com, Feb 6, 2018, Power(Shell) to the people, https://opensource.com/article/18/2/powershell-people ↩︎
  13. Linux Foudation, MIT License, https://spdx.org/licenses/MIT ↩︎
  14. WordPress, Home page, https://wordpress.org/ ↩︎
  15. Wikipedia, PHP, https://en.wikipedia.org/wiki/PHP ↩︎
  16. Free Software Foundation, Free Software Licensing Resources, https://www.fsf.org/licensing/education ↩︎

Leave a Reply