Exocomm Technologies
LOGIN
e-mail
password
Don't have an account? Register now!

Zack rants about Amazon AWS

Sometimes I get asked about AWS -- and I don't do a very good job of explaining the wonders of AWS and what I've done with it. Well, I do a lot of different things -- AWS doesn't really dominate my attention at the moment, no. At various companies sure, we use a TON of different tools and approaches. As nerds we play with just about every tool we can find. Every programmer wants to make something other people will use. As a system administrator trying to keep everything running smoothly, I'd prefer to not have new tools constantly popping up all over my network, disrupting things. But sometimes one has to compromise, and we conduct tests of new technologies to see if we like them. That's fair enough. That's why there's some stuff on my resume that I don't like, and some stuff that I've pretty much forgotten all about. Don't be suprised if you ask me about some software tool that's on my resume somewhere and find I've forgotten about it -- then later go "AH, you mean THAT". This happens all the time. That's what experience brings. You know stuff, but often you forget that you know it -- because you're constantly learning more and more, you forget what you did a year or two ago. Maybe that's because we have brains instead of computers in our heads.

If something is in the experience section of my resume, that means I used it at a company somewhere, to some extent. It doesn't mean I mastered the that something; it just means it was one of the toys we played with. Stuff like Jenkins or Maven or node.js -- I've used those things once or twice over the years (usually at someone else's request), but then I go on to other things and forget about them. 10 years ago I was using Atlassian products exclusively, because back then the average web developer didn't really care about version control and automated testing and all that. That was the domain of actual serious software companies, and they'd pay big bucks for our software. It was good software. But these days, we have GitHub and various other low-cost cloud solutions. It's the same old thing, done a different way.

(When's the last time I did any PERL? I used to write PERL all day long. 20 years later we have Python, which I'm only now learning. Funny how that works!)

Now AWS however, is not rocket science. There's not all that much to learn. It boggles my mind that there are specialists running around town that do ONLY AWS, all day long -- and it's even more weird that these are SMALL companies hiring them.

(GitHub wouldn't work for most of my needs exactly as AWS wouldn't -- I run Git, not neccessarily GitHub; I run servers, not neccessarily AWS. For one thing, we can't store 100 GB of code in GitHub. For another, I don't trust GitHub or any third-party service enough to put my code there. My code is valuable; it matters to me. I don't just upload my customer's code to the cloud all willy-nilly. Some of this code could be worth millions to someone -- we cannot just toss it into the cloud! Around here, we have high standards when it comes to that sort of thing.)

In the skills section, that's where I actually make claims about what I'm GOOD at. It usually takes a lot to say I've "mastered" a technology. I mean, a LOT. Have I mastered c programming after 30+ years? Well, I feel like I'm getting pretty good at it -- but no, I am not a "master". How about something like bash. Sure, I can say I've pretty much mastered it -- maybe 95 percent. I'm still finding new little features and tricks in bash. Then we come to AWS. Yes, I've mastered AWS. I've used all of it's features and there's nothing to it that concerns me in the slightest. There's probably some new stuff they've come out with in the past 18 months or so that I've been doing other things outside of AWS, but who cares.

But we go around talking about the tools we use instead of the actual work we plan to do with the tools. My brain just doesn't work that way. I've probably learned and then forgotten hundreds of different tools, protocols, languages, approaches. What I remember is what we achieved with the tools -- that part doesn't change very much. You know -- automation, security, performance, scalability, all that good stuff.

I really don't get what it is about AWS, that makes people so religious about it. I suspect AWS is popular for the same reason as Amazon shopping -- it's quick, easy, and everybody else is doing it. Just like they didn't INVENT online shopping. They just did it well and got super popular. Just like with AWS. I might call the Amazon shopping "just a shopping service" except in that area they probably innovate a lot more than in hosting. I've stated that AWS is "just another hosting provider", and I get odd looks of doubt and suspicion. To me, this is a fact as true as the sun coming up in the morning.

Look, I've done systems and networks for three decades now. For pretty much everybody. Everywhere I look, I see something I've worked on at some time or other. Everybody from the makers of your mobile phone, to the makers of your shampoo, to the processors of your health data at your doctor's office. Just trust me, I've been around.

I've spent countless hours in data centers of every kind. I was CTO of a downtown Toronto data center for a number of years, selling cloud services before Amazon came up with AWS. I know what it's like to have this PERCEPTION of the cloud -- a fantasy about these super perfect data centers somewhere, maybe with barbed wire and guards and firewalls and professionals trained to deal with your systems -- that's the PERCEPTION -- but I KNOW what goes on in data centers around the world, I KNOW things don't always work perfectly, and that EVERYBODY has failures. It's just a silly computer sitting there. Heck I remember one time at a major data center, an electrician dropped a tool into a junction box, frying himself and a number of cabinets full of switching equipment. I'm a guy who spent my 20's crawling under the raised tiles in the data center floor, running cat6 and power. So don't try and tell me that AWS is different.

(Think about certain VERY EXPENSIVE data centers with the guards, man-traps and retina scans -- yes, I've worked in those places too, and well, I know what goes on there behind the scenes. They aren't perfect either, although they do a great job of looking like it. They're actually quite good, but they are "just a data center" after all. Some people think the guards and iris scanners are worth the expense -- me, I just find it takes a lot longer to get anything done because the security procedures are so strict. This is the kind of DC where someone's job is to walk around every cabinet every few hours and make sure it's secure. There isn't even a speck of dust on the floor. The entire environment is perfectly controlled. And yet -- I don't think they are as much fun to work in as the more "everyday" data centers. It's only a very rare type of customer who would want to pay that much money for these types of services. Banks, government. Anyway yes, screwups even occur at these super-high-end data centers.)

Because I tend to specialize in IP network / UNIX system stuff, I've done some very large scale deployments. About as big as it gets. Yes, there are government agencies and private companies that have VAST networks which have nothing to do with Amazon. Are you suprised? Then you haven't been in this business very long.

I mention anycast, and nobody knows what that is. We talk of scalability and everyone wants to be Google-scale, but they don't even want to think about what that kind of thing really involves. I've been doing this stuff WAY before Google, Facebook or Amazon existed. Who knows, I might even be around after they are gone.

(It's true that my focus has shifted somewhat over the years. Markets change. AWS is part of the reason guys like me don't build as much infrastructure as we used to.)

I've worked on big-name firewalls, storage, AAA appliances, routers -- you name it. I mean I'm writing the firmware for these devices, not just USING them.

I've seen so many fads, technologies and trends come and go. These days, everybody talks about scalability and load balancing and such as if it's some brand new thing. It isn't. We've been doing it for decades. Nothing Amazon offers is in any way different than countless (hundreds of thousands, maybe more) of quality service providers in the world.

You get your control panel credentials to control who can do what -- Amazon's is OKAY in this regard, but I've seen better. You get a bit of firewall control, actually NAT by default. Block this port, allow that IP. Basic everyday firewall. Now sure, Amazon's management interface in this area is pretty decent. They do a pretty good job of making it easy for the users. But still -- it's a basic firewall. It's nothing you can't do on any other host. It's not some Big Impressive Fancy Security Solution.

You've got a nice selection of AMI images to choose from, and some of them even work. But let's be serious, people. Don't try and talk to me about security when you suck in an untrusted operating system that some anonymous person made and published an AMI of. Yeah, you got your web server up in 45 seconds -- but what about when it's discovered later that malicious code was present in that AMI which has been sucking sensitive data out of your instances? Would the administrator KNOW such a thing had happened? What -- the AMI is safe because it's on Amazon? That's like saying every product on Amazon shopping is going to be top quality goods. What about a few years from now when that AMI is no longer available and you need to scale something, but you got stuck with some silly architecture because everbody built things specifically for one type of AWS instance and you can't figure out how to escape? Secure networks don't suck in untrusted code from anywhere. That will never change.

The ability to build your own AMIs, and to convert physical machines to virtual and vice-versa, is really the fun and powerful part of AWS. But then, if you know how to build operating system images, then you probably don't really need Amazon, do you.

Then you have some load balancing and auto-scaling features. Again, nothing new. Define some health checks and load balancing methods, and there you go. It's pretty quick and easy.

But guess what -- you could be doing the same thing 10, 20 years ago without AWS. It's really nothing to get excited about. Now from a developer perspective, this whole AWS cult would be a concern to me. Why do I have to care about where my code goes when I'm done? The user puts it where he pleases.

Sure, I've worked quite a bit with AWS. I've completely mastered all of it's API -- it's really not that big of a deal. I've used their API with a few different languages, even the PHP one -- I think I prefer REST, though.

What I see in the real world is that companies who do NOT understand infrastructure, jump on this AWS bandwagon as if it's some kind of magic savior solution. Some crazy people even believe, SOMEHOW, that Amazon makes them "more secure". No people, AWS is JUST A HOST. It is not a security solution. It's a basic everyday hosting control panel with some firewall control. It has absolutely no "security" features per se (although of course you can use their features as part of a comprehensive security plan). Amazon is NOT a managed services provider -- they have no clue what goes on inside your hosts, and they do not magically make your application more secure. Yes, they do a pretty good job of what they do.

I get asked about "large scale" AWS deployments. This is ridiculous. No. There aren't any. It's true that many large companies do use AWS (Samsung for example), but NOT in a large scale fashion. To me, 100 nodes is not large scale. Large scale to me, is tens to hundreds of thousands of nodes spread all around the planet. Millions to billions of data transactions per day. Yes, I've managed AWS clusters that get fairly big -- sometimes I create them from scratch, and sometimes I get called in to fix things that others have built. But they never get truly "large scale". Sometimes guys think they are -- and that's fine.

(If you can actually count the number of servers you have to manage just by looking around the room, or in an AWS console, hey -- in my line of work you're doing pretty good -- guess what, it's probably not large-scale.)

True large scale often involves anycast routing, for example -- which is not something AWS can handle. AWS isn't present in many parts of the world where people need infrastructure.

Don't try and tell me that your little app or web site is seriously large-scale, when it isn't. Your AWS clusters may seem impressive to you, but trust me -- they aren't large-scale. Never will be. Sorry.

AWS is GENERALLY most popular with tech start-ups. Now I really enjoy working with start-ups and small businesses of all kinds -- and I can tell you, there are TINY COMPANIES who do BIG DATA and there are BIG COMPANIES that do LITTLE DATA. You can't just judge these companies based on the SIZE of their network. Sometimes AWS works for them. Sometimes not.

AWS is basically a slightly more robust, much more expensive, GoDaddy. There. I said it. Look guys, when you actually make something large-scale -- millions of devices running around in the real world, you realize that there are a LOT of different aspects to things. Security? Don't even get me started.

Complexity is the enemy. Never add complexity to a system unless you're SURE you're getting a good pay-off. This is basic engineering.

AWS is not a magic solution for anything, it's just an easy popular way to get started hosting. Being Amazon -- sure, as long as your credit card is valid, they can probably handle your capacity needs. But then, so can many other quality service providers on the planet.

I have seen AWS instances disappear without cause or warning. Sometimes there are routing or infrastructure issues or performance degradation. Their Linux distro is pretty terrible. Behind all the smoke and mirrors, is a computer. A COMPUTER. That sits in a data center somewhere, that will one day overheat and fail or require replacement and maintenance. You just don't see it, but you still pay for it.

(Actually you could get private hosts from AWS at one point, I'm pretty sure you can't get that anymore, just like local discs? I don't know, it's been a couple of months since I've even logged into AWS.)

Most of all, don't talk to me about fault tolerance when 100 percent of your systems are hosted by one single company. That's not scalable, that's just stupid. Forget hackers, you've got major scalability issues pal, sorry.

The honest truth is that AWS is ABSURDLY expensive. It is pretty much THE most expensive way to host anything these days. That's why you generally can't scale it into the stratosphere.

Toronto businesses, I love working on your AWS setups. It's quite a lot of fun to use their API. As long as it's not on my credit card, I can make AWS do anything you want it to. Then again, I can do that with real hardware, or any other host in the world. I'll even make your GoDaddy host more secure, and scalable. It shouldn't really matter all that much where you host. And if you're married to one particular hosting provider -- guess what, you're not agile, you're not scalable, and you're not secure. Period.

AWS is not a particularly scalable, economical or secure service when looked at in the context of all the different options available. And it's mostly the little guys who are there. 10 years ago, they might have been with a small service provider like Exocomm.

(10 years ago I would possibly have been ranting about how expensive those VMware licenses were, because people were going around selling THAT as the magic solution to everything. You'd get a bunch of "high availability" virtual machines stuck on one physical server, and the provider makes 10 times more money. But we can make snapshots, and do load balancing, and this and that! Great, but by and large, customers didn't really need it. As with Amazon, plenty of companies just wasted resources unneccessarily. A lot of big shops like banks of course still use these types of solutions to this day -- they can afford it!)

I don't give a crap if my opinion is unpopular, or makes me less money. It's my job to give you the straight truth about your network and systems. It's what I do. It's totally fine for people to have different opinions, but some things are simple facts -- AWS is a hosting provider, one of many, not magic.

Want to be agile? Be agile! Don't get all trapped into a particular ecosystem. Be flexible. Most likely your app doesn't care where it's hosted, and your developers don't really need to be thinking about being the next Google, when most likely they already have a long list of things to do -- making your application. Don't get all narrow-minded and think that AWS is the only option. When the time comes to scale, scale.

Don't drink the kool-aid. AWS is nothing special, and it probably won't change the way you do things. If it does, there was probably something wrong to begin with. To really be flexible, portable, secure, scalable and all those other wonderful things -- get ready for it -- you have to do the work. You have to think about how your app works, and how your users interact with it. Where your data is born, and where it dies. Security and scalability should be part of your everyday process. You cannot slap on ANY technology on top and suddenly become secure or increase performance. This is physics. It is not really up for debate -- although feel free to try, I suppose.

If your app performance sucks or your code is insecure, AWS isn't going to fix that for you. In fact it may make things worse. You can go ahead and blow $20,000 a month to make a basic everyday web app "scalable". But that's not really scalable, is it?

If somebody convinced you to drink the kool-aid and now you're wondering why your app is slow and your systems are getting hacked and your AWS bill just keeps growing -- heck, give me a call. Let's unravel that silly mess and build you something that works. It'll work so well that you'll stop thinking about your hosting provider and maybe think more about your own business.

If your server isn't performing well, trying to scale that poor performance is just silly. Sure you CAN do it, but you'll just end up with an expensive mess. People will tell you it's all so cool that your app auto-scales, and maybe it does to some extent -- but what if the performance still sucks?

People ask me about these things, because they know I'll give them an honest answer and not hype.

How many transactions can you process per day with a single host? What, nobody knows? Nobody even is aware of the performance capacity of a single node, and yet they're going off and auto-scaling into the stratosphere. They didn't bother to actually test their capacity planning. Sometimes a 1-liner configuration change is all you need to work some magic. Companies have totally different data flows.

One of my customers for example has systems all over North American and Europe -- they need the data to be highly available, but the data can't just leave the EU since it contains personally identifiable information. Therefore they have a firm requirement that data may never be replicated outside of it's originating country. Amazon would not work for this.

Another customer operates many small networks all over the world. They use RADIUS, and they need to be FAST. Even 10ms of latency is too much. Amazon would not work for this either, because we need that geographic distribution.

Let's take "security groups" for example. Fanboys get all into this and think it's something special. It's not. We're basically talking about VLANs and a bit of NAT. Yes, of course you can use this between sites too if you want to, if you can live with the latency which most can. If you had a couple of physical servers that needed to talk together, you give them the same VLAN ID, and maybe some routing rules between subnets, that's it. In the cloud, it's basically the same thing. All it does is direct traffic around your clusters and control who talks to whom. We've been doing this for decades; nothing really interesting to see here.

The customers who have had SUCCESS with AWS hosting, clearly, are the ones who "keep it simple". That's what I do when I'm able to build an AWS setup from scratch. With a sensible approach, you can get acceptable bang for the buck. I guess the same could be said for GoDaddy.

It's the ones that go charging in, auto-scaling and micro-servicing everything without understanding what they're doing -- that's who gets that WOAH bill from AWS.

But keep in mind, in AWS you pay for EVERYTHING. That poorly-indexed database query might have just cost 12 cents to execute -- that's not normal. In the past with physical systems or even hybrid clouds, you don't pay for bytes on the wire or CPU cycles or storage (depending on your deal). In AWS, you do. Forgot to minify that script? That costs money. Doing a big build or even just a backup that utilizes a bunch of storage capacity and CPU time? Yep, that's going to cost.

And I get funny looks -- like what, it's not normal to spend 20 grand a month for a couple of cloud instances? What, are you suggesting we can't afford it? Isn't everyone else doing it? Isn't this the way to solve all those problems we keep having?

(10 years ago I was saving companies money on their PHYSICAL infrastructure costs. And the snooty data centers were mostly pushing VMware and private cloud services. When containerization came out it really opened up the market, too.)

As a programmer, it just means that resources which were once effectively "free" now cost actual dollars. In the old days you'd buy some storage or servers, and then utilize that capacity forever (until obsolete). In AWS, you pay as you go. For every single packet, every single CPU cycle. Forever and ever.

We programmers tend not to care about performance matters that don't affect functionality or the user experience -- for example if my database query is fast, generally we wouldn't care how much disc I/O or CPU load each query causes (until something starts to "feel" slow). But in AWS, your organization is going to be paying the price for poorly-performing code, even if it doesn't affect the user at all. For example just creating a couple of indexes on some popular tables might save thousands of dollars per month -- it does happen (but this is still not magic; with database indexing you are trading storage for CPU load)! A rather extreme example would be ElasticSearch. Sure, it's a pretty fast way to store and query data. But it's Java-based -- it's not very efficient -- it uses piles of memory and storage -- it's basically like a SQL database that has indexes on EVERYTHING. Sure it's fast -- it's got the entire database in memory! But it actually costs more to run on a daily basis because under the hood, it consumes more resources than a conventional SQL database. Run such stuff in AWS at your peril. There's no free ride in technology. There are always trade-offs.

But of course that doesn't concern you if you've got a small instance that doesn't do all that much. Sure, a basic company web site is perfectly happy in such an instance, and your bill won't be anything to worry about.

That means that physical server with the load average of 30 is going to cost a BUNDLE to operate in AWS. That's how it works. How much I/O are you doing? How much CPU time do you burn? What, nobody knows? Then sorry, you're not scalable no matter how many instances you create or how big your AWS bill gets. Sorry.

Sure, it can be a lot easier to start up with a cloud service than to manage your own servers -- of course! Maybe it's a great place to host for a while, while you build your business and get enough users/customers to actually need scalability.

There are also many companies who have failed to achieve what they had hoped. They hire teams of people just to manage AWS, it doesn't work, and eventually the company either goes broke or limps along burning way more money on infrastructure than we did in the old days -- when you actually owned the server running your application. Systems are still getting hacked, unstable code still breaks, and basically shit still happens. That's life. But those shilling these solutions will tell you it'll make everything better -- faster, cheaper, scalable, secure. Yeah, yeah. Sure.

Most of this rant can be applied to the cloud in general, or to Google Cloud, or VMWare or containers or Docker or Puppet or Chef or Kubernetes or any of the trendy cults that will emerge in the future.

Want a really good deal in hosting? Come talk to me. Excellent pipe and top quality infrastructure, dirt cheap. We built AWS before Amazon did (it's Xen virtualization technology if you didn't know), we have better locations and better security and most of all, better pricing. We run all our own systems here -- public cloud, hybrid cloud, private cloud, shared, real machines that you own -- we love being "network agile" around here, and I think you will too.

We have servers in Panama that are FANTASTIC! Yes, Panama. What, why are you giving me that look? The pipe is fantastic (it can be lower latency than within some US cities, think about it), and we get the latest servers fully spec'ed for dirt cheap. The "not in the US" aspect is just a bonus some people like. Look, some nice places on earth have shit networks. Some shit places have nice networks. In Europe you're talking Amsterdam and maybe Sweden over the UK and Germany, just for example -- in terms of how much quality hosting you can get for the least money (for those companies who actually care about the rest of the world outside the US/Canada). Or go ahead and pay 10 times more in AWS -- your call, HOWEVER don't try and tell me this is "scalable". You use it because you don't know what else to do. You wouldn't need me to point it out for you. If you use AWS but have a strategy that ensures your success no matter what happens to any one resource -- there you go, you'll do well and survive any disaster.

Look, I know how it is. Anything other than Amazon, and people wonder -- will it really stay up? Well the thing is, if you're locked into ANY hosting provider, that's a bad thing for you and a good thing for the provider. Trust me, I've worked both sides.

Sure, it's probably true that Amazon's more likely to stay up than some bargain-basement hosting operation. HOWEVER, that's not really the issue here, you're missing the point. The fact is that it's dumb to be stuck with any one vendor for anything, especially hosting.

There's NOTHING WRONG with having, say, a write-intensive database hosted in one place and a read-intensive database in another. Maybe one provider has fast storage, or a good backbone link -- and the other is just cheap. Maybe use a CDN for static hosting to reduce the load on the expensive cloud option. Whatever. Scalability is about being able to mix and match those vendors at will based on any criteria you choose.

The whole name of the game is to be agile and portable. Be EVERYWHERE, ANYWHERE -- you can mix and match! If your loads are acceptable for AWS and that's what you want, great -- but a good effective tech company would never get STUCK in any one provider.

I'm not an "AWS specialist". I don't need to be, because there's really not all that much to it. There's nothing there I need to learn. And when you do what I do, there's never a shortage of things to do or learn. AWS just isn't one of them.

(When Cisco firewalls were first coming out, nobody knew anything about them and there were no such things as certifications! As a matter of fact Cisco didn't even create "Cisco" firewalls, or much of anything really, they just buy small companies who create desirable technologies -- like Network Translation or Meraki. These days their firewall technology is lagging a bit behind, though. The ASA is just a rebranded PIX which is decades-old technology and firmware. Yep, it's just a crappy Celeron chip in there, believe it or not. But you'd never think that by looking at the appliances -- because of the PERCEPTION they have. Same goes for expensive storage.)

I just don't understand how someone who has a CHOICE will CHOOSE to waste $20,000 a month in AWS when they could have just as well hosted a rack full of real servers that they own -- for say $2,000 per month. You can actually get a TON of infrastructure for $2,000 per month! Heck, you might even be able to slash that to $200/month. Some might be even lower than that -- scary thought (for Amazon), but completely true. Yes, I'm completely serious.

(But to have a choice you have to be versatile enough to move your application around without disruption, and there's a whole art to that. That's real scalability there.)

I work on embedded systems a lot, it's my main area of expertise, and we do a LOT of builds. I mean real, actual machine code builds. To build firmwares for 10 different platforms (say phones, tablets, sensors, whatever), about 100 gigs of source code must be compiled. So if I just fix one bug, if all of this were hosted in AWS it might cost hundreds of dollars just to do one single build.

Just last night I did about 10 builds just investigating one bug. Can you imagine if I had to pay for all that in AWS? Impossible! Can't be done. Does not compute.

Continuous Integration? We probably couldn't live without it, because the code changes so frequently and the builds consume so many resources, we can't wait around for builds -- the sooner they finish, the sooner the programmer can go on to other things (no you can't cheat -- one test has to pass before the next executes). But we build real software here, not just web sites. We have to test our code on many different platforms. It's pretty serious stuff, very processor-intensive, and easy to screw up.

(These days, the web developers talk like software developers -- we talk about "builds" like they are actually building compiled software. Sorry kids. Your Javascript and Python apps don't really "build", do they. But we pretend that pulling in a bunch of scripts from GitHub and pushing them to your AWS instances constitutes a "build" -- fine, I can go with that. Just be aware that that is not what actual software does when it "builds".)

If any AWS fanboy out there would like to debate this, please feel free to get in touch! Explain to me how I'm wrong in stating that Amazon is "just" another hosting provider.

Security doesn't come from your hosting provider. Scalability can't just be slapped on top of a broken application like a band-aid. Neither does performance. If you do what everybody else does, you can expect the same results. AWS works pretty well for simple systems -- arguably, so does GoDaddy. Just don't try and tell me it's some kind of magic solution for everyone. If you think that, you probably don't know a whole lot about systems.

I can help you build truly scalable, truly secure, truly fault-tolerant systems. AWS might even be a part of that solution. Have a network that is NOA -- Not Only Amazon. You can do it. It's not that hard. Maybe your company's stuff doesn't fit the cookie-cutter AWS solution -- maybe you're non-x86, maybe you run embedded devices, or do some sort of highly intensive data processing of some sort, or just enjoy being able to reach over and touch a physical server now and then.

On the one hand system requirements are generally unique, but the modules we use to meet those requirements are not -- your needs may change, but AWS is just a shared computer. The specifics of using AWS's API to do things are outside the scope of this rant -- use one of my open-source scripts for a starting point if you like. But they are the same things one does with any automation. Create nodes, destroy nodes, configure nodes, blah blah blah.

I mean, it wouldn't completely amaze me to one day wake up and find all of your AWS nodes deleted without explanation. You furiously call Amazon and guess what, nobody gives a shit about you or your application or your customers. Network engineers have to plan for disasters to happen. It's what we do. I've seen this sort of thing happen before, and I have to plan for it to happen again. It boggles my mind people have job titles like Site Reliability Engineer, when they choose to host 100 percent of their resources in one single place with one single vendor and talk about how reliable and scalable they are. It isn't. They aren't.

If you're gonna host in AWS, you'd better have a strategy for what to do if it ever blows up. What if some phisher gets hold of your company credit card and messes up your AWS account -- or gains access to the administrative privileges of AWS and proceeds to exploit your entire company because you depended on AWS "security features" to keep you safe? Maybe it's a malicious app that infected an employee phone which got the AWS credentials and voila, your entire company is screwed. Now your customer's database has is being torrented around the planet and is in the hands of criminals because you made AWS your golden key to everything. Added to the long list of companies this has happened to. The more successful the company, the more people will try this sort of thing.

Most people don't even bother to delete the TEMPORARY KEYS installed by AWS and install their own. They just pass these (usually passwordless) keys around the office so it's easier to manage. Employee left the team? Ah well. We have a shared key and can't change it, but it'll all be okay. Generally they don't IP-restrict administrative access either (often they don't know their IP address in the office or how to find it), so basically you're one stolen laptop away from total 0wnage.

This is the real world, people. Real security isn't a couple of clicks on a web site and "done", it's never "done". We design things to be robust, so that one single point of failure takes out the whole system. That's the whole idea here, people. Don't say it can't happen. It happens all the time. It's just that few have the experience with it to know, or the balls to point it out when there's money to be made shilling AWS. No. We plan for these disasters ahead of time, even if it seems unthinkable.

SOME quality providers out there will give you everything that AWS does and SO much more. You can get security people, people to manage any aspect of your infrastructure, people who will work tirelessly to help you and your organization succeed. Some offer superbly performing machines with ultra-fast storage and compute resources at incredibly low pricing. And you can get this anywhere in the world. Just sayin'.

It must really boggle people's minds to think that even decades ago, we still did all these same things. Believe it or not, automation was a thing long before we talked about "the cloud" and all that. You can still automate the deployment, management and maintenance of actual physical servers pretty much exactly the same way you do it in the cloud. No really, you can! A bit of PXE and some RADIUS with a DHCP server and you're in business. Plug in a bunch of servers fresh out of the boxes; mix and match with VMs or other remote cloud nodes -- and auto-scale your application automagically. Wait a minute -- that's pretty much what Amazon provides today, you just don't notice because you aren't experienced in these things, and that's why you're using AWS. You spend your time fussing about auto-scaling your app because you want it all point-and-click. You AWS fanboys are just like the outsourcers of the 90s, who would bankrupt their companies by hiring IBM consulting to look after every little thing instead of doing some actual diagnostics or improvements to their company's infrastructure. It's just that now, every company is a tech company.

I realize my way of speaking and thinking might be a bit different than what you're used to. Generally "AWS experts" go around evangelizing it and doing ONLY AWS work.

I'm honestly telling you that it is FACTUAL that AWS is just another hosting service. That's all it is. Can we agree on that? If someone goes around claiming to be an expert in High Availability, Security, Scalability or any of that jazz yet cannot explain how one would do it without Amazon, you should probably find another expert.

If all I did was AWS all day, I would have a TON of time on my hands. But of course -- there are always lots of things to be done and the queue always fills up fast.

One thing my bosses tell me is that they like how I can reduce all this crazy nerd gobbledeygook talk into simple, straightforward terminology. That's not always easy. When the sales guy comes running in with an order for something he's not quite sure we're able to build -- I can run with that and translate it into actual code or infrastructure. When a programmer finds a brick wall and everybody's pressuring him to finish something on a deadline and we need a solution -- well that's the sort of thing I tend to work on.

Once people get that I never say "no, we can't do it" but rather "well, we COULD do THIS" -- they just start coming to me. It's problem-solving. All programming is basically just problem-solving too. And when I don't answer that THE ONE TRUE WAY TO DO X IS Y -- don't be so suprised. Some things are facts, and some are opinions. Systems are complex and full of entropy; decisions get based on probabilities and the exact way to we achieve something is always going to vary somewhat. That's what experience gives you -- you've seen bad things happen, you've learned tricks to come out on top.

(That's why the entire planet doesn't host every server in the world in AWS. Although yes they're growing, that's for sure!)

If you get a big contract or whatever and maybe they don't like AWS and want to run -- say, in VMWare somewhere or a private cloud -- or they want to run it as a container within a Kubernetes cluster -- or whatever. I'm the guy who'll say yep, no problemo. Your apps should be able to run anywhere you want them to. I bet I could make it run on a toaster.

Hey, I'm not the only one. Here are some articles supporting everything I've just ranted about.

  • 5 reasons AWS sucks
  • how to keep Amazon from eating your business
  • why AWS is bad for the Internet
  • GNU
    Linux
    X
    Terms of Service
    |
    contact Exocomm
    |
    about Exocomm software
    |
    download Exocomm tools
    Exocomm Technologies | Toronto, Ontario, Canada
    Copyright 1992-2020 by Exocomm Technologies. All rights reserved.