{"id":4787,"date":"2024-04-10T03:00:00","date_gmt":"2024-04-10T07:00:00","guid":{"rendered":"https:\/\/www.both.org\/?p=4787"},"modified":"2024-04-05T12:55:48","modified_gmt":"2024-04-05T16:55:48","slug":"how-to-debug-a-web-server-error-with-wget","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=4787","title":{"rendered":"How to debug a web server error with \u2018wget\u2019"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

If you manage a website, you know that sometimes things can get a little messed up. You might remove some stale content and replace it with a redirect to other web pages. Or you might update a website to require a username and password to access certain pages. After making enough changes, you might discover that your website has stopped working the way you want it to. For example, you might see an error like this in your browser to indicate that the website is redirecting too many times:<\/p>\n\n\n\n

\"screenshot<\/figure>\n\n\n\n

One way to debug this is by using the wget<\/code> command line tool. While you might already know how to use wget<\/code> to fetch individual web pages or files from websites, you can also use wget<\/code> as a handy tool to keep in your system adminstrator \u201ctoolkit\u201d to help troubleshoot web server errors.<\/p>\n\n\n\n

For example, when I need to debug a website, I rely on the -S<\/code> option to show all server responses. When using wget<\/code> for debugging, I also use the -O<\/code> option to save the output to a file in case I need to review it later. Here is how I use these options with wget<\/code> to debug a website issue.<\/p>\n\n\n\n

A problem website<\/h1>\n\n\n\n

Let\u2019s say we have updated a website to add a new feature to edit content. But when we access the website, we get the message from earlier in this article about too many redirects.<\/p>\n\n\n\n

The first step to debugging the problem is to understand what is happening behind the scenes. The web browser doesn\u2019t provide any details about where it has been redirected to<\/em> so we need to use wget -S<\/code> to get more detail about how the client is being bounced around the website:<\/p>\n\n\n\n

$ wget -S http:\/\/10.0.0.11\/edit\/\n--2024-04-05 11:30:57--  http:\/\/10.0.0.11\/edit\/\nConnecting to 10.0.0.11:80... connected.\nHTTP request sent, awaiting response... \n  HTTP\/1.1 302 Found\n  Date: Fri, 05 Apr 2024 16:30:57 GMT\n  Server: Apache\/2.4.58 (Fedora Linux)\n  X-Powered-By: PHP\/8.2.17\n  Location: \/login\n  Content-Length: 0\n  Keep-Alive: timeout=5, max=100\n  Connection: Keep-Alive\n  Content-Type: text\/html; charset=UTF-8\nLocation: \/login [following]\n--2024-04-05 11:30:57--  http:\/\/10.0.0.11\/login\nReusing existing connection to 10.0.0.11:80.\nHTTP request sent, awaiting response... \n  HTTP\/1.1 301 Moved Permanently\n  Date: Fri, 05 Apr 2024 16:30:57 GMT\n  Server: Apache\/2.4.58 (Fedora Linux)\n  Location: http:\/\/10.0.0.11\/login\/\n  Content-Length: 231\n  Keep-Alive: timeout=5, max=99\n  Connection: Keep-Alive\n  Content-Type: text\/html; charset=iso-8859-1\nLocation: http:\/\/10.0.0.11\/login\/ [following]\n--2024-04-05 11:30:57--  http:\/\/10.0.0.11\/login\/\nReusing existing connection to 10.0.0.11:80.\nHTTP request sent, awaiting response... \n[..]\nLocation: \/edit [following]\n--2024-04-05 11:30:58--  http:\/\/10.0.0.11\/edit\nReusing existing connection to 10.0.0.11:80.\nHTTP request sent, awaiting response... \n  HTTP\/1.1 301 Moved Permanently\n  Date: Fri, 05 Apr 2024 16:30:58 GMT\n  Server: Apache\/2.4.58 (Fedora Linux)\n  Location: http:\/\/10.0.0.11\/edit\/\n  Content-Length: 230\n  Keep-Alive: timeout=5, max=81\n  Connection: Keep-Alive\n  Content-Type: text\/html; charset=iso-8859-1\nLocation: http:\/\/10.0.0.11\/edit\/ [following]\n--2024-04-05 11:30:58--  http:\/\/10.0.0.11\/edit\/\nReusing existing connection to 10.0.0.11:80.\nHTTP request sent, awaiting response... \n  HTTP\/1.1 302 Found\n  Date: Fri, 05 Apr 2024 16:30:58 GMT\n  Server: Apache\/2.4.58 (Fedora Linux)\n  X-Powered-By: PHP\/8.2.17\n  Location: \/login\n  Content-Length: 0\n  Keep-Alive: timeout=5, max=80\n  Connection: Keep-Alive\n  Content-Type: text\/html; charset=UTF-8\nLocation: \/login [following]\n20 redirections exceeded.<\/code><\/pre>\n\n\n\n
I\u2019ve omitted most of the output from this command, but you can already see that accessing \/edit<\/code> redirects to \/login<\/code>, which then immediately redirects back to \/edit<\/code>, and so on. This is an endless loop, and wget<\/code> will automatically exit after 20 iterations.<\/p>\n\n\n\n
By seeing these details, the webmaster can make a determination that the login system is not working properly, and focus their attention on the \/login<\/code> page instead of trying to debug a (working) \/edit<\/code> page.<\/p>\n\n\n\n
Saving output<\/h1>\n\n\n\n
In this case, the website redirects too many times, so it never produces a valid web page. But if you had an example where the web page printed an error that you needed to save for later, you can dump a copy of a web page to a file.<\/p>\n\n\n\n
Let\u2019s say you manage a website with an RSS feed, and the feed has stopped working. The feed displays an error code that you need to debug the problem, so the first step is to get a copy of the RSS feed. Use the -O<\/code> option to save the output to a file:<\/p>\n\n\n\n
$ wget -O t.rss http:\/\/10.0.0.11\/rss\/\n--2024-04-05 11:44:09--  http:\/\/10.0.0.11\/rss\/\nConnecting to 10.0.0.11:80... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: unspecified [text\/html]\nSaving to: \u2018t.rss\u2019\n\nt.rss                 [ <=>          ]     273  --.-KB\/s    in 0s      \n\n2024-04-05 11:44:09 (26.2 MB\/s) - \u2018t.rss\u2019 saved [273]<\/code><\/pre>\n\n\n\n
This saves a copy of the feed from http:\/\/10.0.0.11\/rss\/<\/code> to the temporary file named t.rss<\/code>:<\/p>\n\n\n\n
$ cat t.rss \n<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<rss version=\"2.0\">\n<channel>\n  <title>Latest News<\/title>\n  <description>A summary of the latest news from XYZ.<\/description>\n  <link>http:\/\/example.com\/news\/<\/link>\n\n  *ERROR: Cannot read news items (ERR 8675309)\n<\/channel>\n<\/rss><\/code><\/pre>\n\n\n\n
With this information, we can see that the RSS failed with the error code 8675309<\/strong>, which is likely an important error number on the system that produces the news feed. As the website administrator, you would then have the information you need to track down and fix the error.<\/p>\n\n\n\n
Other useful options to \u2018wget\u2019<\/h1>\n\n\n\n
The wget<\/code> command has lots of other useful options. Use man wget<\/code> to read the online manual to learn what other options might be useful to you. For example, the -o<\/code> (lowercase) will save the wget<\/code> command output to a file, rather than printed to the screen. And the -T<\/code> option can set a timeout, after which wget<\/code> will stop waiting on the web server.<\/p>\n","protected":false},"excerpt":{"rendered":"
The wget command line tool can provide detailed information about what\u2019s happening on a web server.<\/p>\n","protected":false},"author":33,"featured_media":4395,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[78,315],"tags":[104,162],"class_list":["post-4787","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fixing-problems","category-web-server","tag-command-line","tag-problem-determination"],"modified_by":"Jim Hall","_links":{"self":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/4787","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4787"}],"version-history":[{"count":2,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/4787\/revisions"}],"predecessor-version":[{"id":4790,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/4787\/revisions\/4790"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/4395"}],"wp:attachment":[{"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4787"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4787"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4787"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}