You might have wondered why our website got redirected to my personal page beginning Sunday afternoon (CEST) and lasting until Monday morning. To make it short: Fab had another server admin fail. While trying to make our server more secure, I must have messed up Apache’s subdomain redirection somewhere which caused it to route all requests for linuxoutlaws.com to lamerk.org which is located on the same server. I did test the new configuration and could see nothing wrong with it so I happily proceeded to have a BBQ with some friends out in the sun, leaving my iPhone, and with it all connections to the virtual world, at home. I also busted my ankle that night, but that’s a whole ‘nother story…

Something must have happened on the server after I left because shortly after I had tested the setup and went outside, people began having problems with the site. I have no clue as to what changed, maybe I just didn’t test everything correctly in my hurry to get out into the sun. In any case, the site and with it the RSS feeds were offline. If you were using gPodder as your podcast client, you might have experienced something else: In a weird twist, gPodder decided to put some “Tiger Records” stuff in our feeds and tried to subsequently download 600+ MB of files. Apparently this is a “feature” of the program; if it does not find one of your subscribed RSS feeds on the provided URL, it crawls that site and all embedded links looking for something that looks like RSS with rich media enclosures. If it finds anything, it helpfully uses that as the new podcast feed. Quite a handy feature that means you can just pop linuxoutlaws.com in the new subscription field and get one of our feeds auto-magically but it also messed with my brain royally when I woke up Monday morning (slightly hangovered) and started gPodder to check if some new podcasts were out for me to listen to. As it were, the “spam” in our feed made me believe our server had been hacked. The fact that nobody, including Dan, hadn’t been able to reach me until then just complicated matters.

Long story short: I have checked the server thoroughly (just in case) and am quite confident that we have not been hacked but that everything was due to me being an idiot. I have also since repaired the subdomain problem and all of the site (including the forums) should be in 100% working order again. If you are using gPodder (a fantastic podcatcher, by the way) and you still have weird stuff in our feed, please re-subscribe and everything should be cleared up. You can also manually remove the non-LO entries from the list. I am really sorry for any inconvenience or frustration this might have caused and I am also very thankful to the great people in our forums, #linuxoutlaws on Freenode and our identi.ca group for their help in figuring this out. Also thanks to everyone who emailed telling us the site was broke, we got a lot of those as well.

And so we prove once again that the life of an outlaw is everything, but it ain’t boring…

Tonight, shortly after 02:00 CET, a process on our server stalled out the CPU and shot down the machine. I only noticed it this afternoon via text message from Dan, who was alerted by some helpful souls on identi.ca. I promptly rebooted the server, but I am sorry to have only realised the problem that late. I was pretty busy at work and never even thought about checking my servers all day. I appologise wholeheartedly for any inconvenience this might have caused, especially with the delivery of the podcast files.

It seems like something went wrong with our Laconica installation (which is also hosted on the machine in question) shortly before the server died, or so my log files would indicate. At least the latest errors are from that process. I will investigate this issue further and keep you posted. It seems like Laconica both caused this issue and kind of helped to solve it at the same time. How very Zen.

Well, as you might have noticed, the server went down once more. Sorry again for any inconvenience the latest outage might have caused. This time it wasn’t my fault. Our server exceeded another limit, in this case its memory allowance. Dan actually had to upgrade us to a business plan. Seems like we were not only pushing considerable bandwith, but there is also quite a bit of load on the server. As you can see from the graphs, our downloads as well as the feed subscriptions are constantly increasing. At this rate we are gonna need dedicated hardware pretty soon…

Don’t get me wrong, I think this is great. When we started this little project, I would have never dreamed of even getting 1,000 downloads per week and we have well exceeded that by now. This whole thing is really amazing and the support we are getting from you on a day-to-day basis makes it well worth it. As for the server, we’ll get something worked out. I just hope we can avoid further downtime from now on. Doing the show is mad fun and I won’t let any silly technical difficulties stop us. As Dan said on the forums, these seem to be minor growing pains that will just make us stronger in the long run. Just like the real outlaws we are, we won’t give up without a last stance! Just bear with us here…