{"id":10800,"date":"2025-05-29T03:00:00","date_gmt":"2025-05-29T07:00:00","guid":{"rendered":"https:\/\/www.both.org\/?p=10800"},"modified":"2025-05-27T21:02:36","modified_gmt":"2025-05-28T01:02:36","slug":"my-first-sysadmin-mistake","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=10800","title":{"rendered":"My first sysadmin mistake"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n 1 <\/span>\r\n<\/div><\/div>\n

If you work in IT, you know that things never go completely as you think they will. At some point, you\u2019ll hit an error, or something will go wrong, and you\u2019ll end up having to fix things. That\u2019s the job of a systems administrator.<\/p>\n\n\n\n

As humans, we all make mistakes. So sometimes, we are the error in the process, or we are what went wrong. As a result, we end up having to fix our own mistakes. That happens. We all make mistakes, typos, or errors.<\/p>\n\n\n\n

At my very first job as a young systems administrator, I learned this lesson the hard way. I made a huge blunder. But thanks to some coaching from my supervisor, I learned not to dwell on my errors, but to create a \u201cmistake strategy\u201d to set things right. Learn from your mistakes. Get over it, and move on.<\/p>\n\n\n\n

I deleted what?<\/h2>\n\n\n\n

My first job was Unix systems administrator for a small company. Really, I was a junior systems administrator, but I worked alone most of the time. We were a small IT team, just the three of us. I was the only systems administrator for twenty or thirty Unix workstations and servers. The other two systems administrators supported the Windows servers and desktops.<\/p>\n\n\n\n

Any experienced systems administrators reading this probably won\u2019t be surprised to know that, as an unseasoned junior systems administrator, I eventually ran the rm<\/strong> command in the wrong directory. As root. I thought I was deleting some stale cache files for a program of ours. Instead, I wiped out all files in the \/etc<\/strong> directory by mistake. Ouch.<\/p>\n\n\n\n

My clue that I\u2019d done something wrong was an error message that rm<\/strong> couldn\u2019t delete certain subdirectories. But the cache directory should only contain files! I immediately stopped the rm<\/strong> command and looked at what I\u2019d done. And then I panicked. All at once, a million thoughts ran through my head. Did I just destroy an important server? What was going to happen to the system? And, would I get fired?<\/em><\/p>\n\n\n\n

Fortunately, I\u2019d run rm *<\/strong> and not rm -rf *<\/strong> so I\u2019d only deleted files. The subdirectories were still there. But that didn\u2019t make me feel any better.<\/p>\n\n\n\n

First, recover from the mistake<\/h2>\n\n\n\n

Immediately, I went to my supervisor and told her what I\u2019d done. She saw that I felt really dumb about my mistake, but I owned it. Despite the urgency, she took a few minutes to do some coaching with me. \u201cYou\u2019re not the first person to do this,<\/em>\u201d she said. \u201cWhat would someone else do in your situation?<\/em>\u201d That helped me to calm down and focus. I started to think less about the stupid thing I had just done, and more about what I was going to do next.<\/p>\n\n\n\n

I put together a simple strategy: Don\u2019t reboot the server. Use an identical system as a template, and re-create the \/etc<\/strong> directory.<\/p>\n\n\n\n

Once I had my plan of action, the rest was easy. It was just a matter of running the right commands to copy the \/etc<\/strong> files from another server, and edit the configuration so it matched the system. Thanks to my practice of documenting everything, I used my existing documentation to make any final adjustments. I avoided having to completely restore the server, which would have meant a huge disruption.<\/p>\n\n\n\n

Then, learn from the mistake<\/h2>\n\n\n\n

To be sure, I learned from that mistake. For the rest of my years as a systems administrator, I always confirmed what directory I was in before running any command, especially the rm<\/strong> command. Even now, on my Raspberry Pi that I use for personal experiments, I’ll run echo rm *<\/strong> before using rm *<\/strong> to delete files on the command line. That way, I can double check what the wildcards will pick up.<\/p>\n\n\n\n

I also learned the value of building a \u201cmistake strategy.\u201d When things go wrong, it\u2019s natural to panic, and to think about all the bad things that might happen next. That\u2019s human nature. But creating a mistake strategy helps me stop worrying about what just went wrong, and focus on making things better. I may still think about it, but knowing my next steps allows me to \u201cget over it.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

1 If you work in IT, you know that things never go completely as you think they will.<\/p>\n","protected":false},"author":33,"featured_media":4654,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[78,79,83,89],"tags":[162,97],"class_list":["post-10800","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fixing-problems","category-i-realy-borked-it-this-time","category-problem-solving","category-system-administration","tag-problem-determination","tag-sysadmin"],"modified_by":"David Both","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/10800","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10800"}],"version-history":[{"count":1,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/10800\/revisions"}],"predecessor-version":[{"id":10801,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/10800\/revisions\/10801"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/4654"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10800"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10800"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10800"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}