xenmaster's devops systems
This list is inspired by the Kamran Ahmed's "2020 Developer Roadmap," now at roadmap.sh
Without further ado, my take on the most important technologies to learn for dev-ops in 2020!
Basics for SysAdmin
These are the basic tools you will need to start your path to being a sysadmin. BASH is the scripting language used in the command terminal while vim will be your go-to text editor and PuTTY your primary solution for remote server access.
Bash is the shell, or command language interpreter, that will appear in the GNU operating system. Bash is an sh-compatible shell that incorporates useful features from the Korn shell (ksh) and C shell (csh). It is intended to conform to the IEEE POSIX P1003.2/ISO 9945.2 Shell and Tools standard. It offers functional improvements over sh for both programming and interactive use. In addition, most sh scripts can be run by Bash without modification.
The improvements offered by BASH include:
- Command line editing
- Unlimited size command history
- Job Control
- Shell Functions and Aliases
- Indexed arrays of unlimited size
- Integer arithmetic in any base from two to sixty-four
Terminal is a modern terminal emulator for the Unix/Linux desktop - primarily for the Xfce desktop environment. We developed it because we saw the need for a lightweight and easy to use terminal emulator in the Xfce desktop environment, that doesn’t require the user to install the GNOME platform, but still provides a worthy alternative to the GNOME terminal emulator.
tmux is a terminal multiplexer: it enables a number of terminals (or windows), each running a separate program, to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.
tmux uses a client-server model. The server holds multiple sessions and each window is a independent entity which may be freely linked to multiple sessions, moved between sessions and otherwise manipulated. Each session may be attached to (display and accept keyboard input from) multiple clients.
tmux is intended to be a modern, BSD-licensed alternative to programs such as GNU screen. Major features include:
A powerful, consistent, well-documented and easily scriptable command interface. A window may be split horizontally and vertically into panes. Panes can be freely moved and resized, or arranged into one of four preset layouts. Support for UTF-8 and 256-colour terminals. Copy and paste with multiple buffers. Interactive menus to select windows, sessions or clients. Change the current window by searching for text in the target. Terminal locking, manually or after a timeout. A clean, easily extended, BSD-licensed codebase, under active development.
Vim ("Vi IMproved") is an advanced text editor that allows syntax highlighting, word completion and has a huge amount of contributed content.
Vim offers several “modes” for editing with efficiency. This makes vim a non-user-friendly application but it is also a strength. The normal mode binds alphanumeric keys to task-oriented commands. The visual mode highlights text. The command-line mode offers more tools (for search&replace, defining functions, etc.)
Vim comes with complete help.
PuTTY is a free and open-source terminal emulator, serial console and network file transfer application. It supports several network protocols, including SCP, SSH, Telnet, rlogin, and raw socket connection. It can also connect to a serial port.
A simple, lightweight client for connecting to a Linux shell from your Windows machine!
Linux Servers
Redhat is the most common server seen in the field. CentOS is the freeware equivalent.
Red Hat is the leader in development, deployment, and management of Linux and open source solutions for Internet infrastructure - ranging from embedded devices to secure Web servers. Red Hat was founded in 1994 by visionary entrepreneurs Bob Young and Marc Ewing. Open source is the foundation of our business model. It represents a fundamental shift in how software is created. The code that makes up the software is available to anyone. Developers who use the software are free to improve the software. The result: rapid innovation. Red Hat solutions combine Red Hat Linux, developer and embedded technologies, training, management services, technical support. We deliver this open source innovation to our customers via an Internet platform called Red Hat Network. Red Hat is headquartered in Raleigh, North Carolina, USA.
CentOS is a community-supported, free and open source operating system based on Red Hat Enterprise Linux. It exists to provide a free enterprise class computing platform and strives to maintain 100% binary compatibility with its upstream distribution. CentOS stands for Community ENTerprise Operating System.
The leading platform for scale-out computing, Ubuntu Server helps you make the most of your infrastructure. Whether you want to deploy an OpenStack cloud, a Hadoop cluster or a 50,000-node render farm, Ubuntu Server delivers the best value scale-out performance available.
Compiling Tools
For compiling apps from source.
Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.
The GNU Compiler Collection (GCC) is a compiler system produced by the GNU Project supporting various programming languages. GCC is a key component of the GNU toolchain. As well as being the official compiler of the unfinished GNU operating system, GCC has been adopted as the standard compiler by most other modern Unix-like computer operating systems, including Linux, and the BSD family.
Virtualization
I don't see virtualization as much as I used to, but these are the best free options I've come across. Note that in the Corporate environment, VMWare dominates from what I've seen.
Xen.org, home of the Xen hypervisor, the powerful open source industry standard for virtualization. It is a native (bare-metal) hypervisor providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently.
VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers, it is also the only professional solution that is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2.
Presently, VirtualBox runs on Windows, Linux, Macintosh, and Solaris hosts and supports a large number of guest operating systems including but not limited to Windows (NT 4.0, 2000, XP, Server 2003, Vista, Windows 7, Windows 8, Windows 10), DOS/Windows 3.x, Linux (2.4, 2.6, 3.x and 4.x), Solaris and OpenSolaris, OS/2, and OpenBSD.
Virtualbox is available in 28 languages.
Create and manage virtualized development environments. Vagrant is a tool for building complete development environments. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases development/production parity, and makes the "works on my machine" excuse a relic of the past.
Networking + Security
This is the longest section and some of these solutions overlap. Some additional notes:
- Wireguard is looking to become the faster, more secure version of OpenVPN, but is still in the experimental stage at this point.
- Snort is an alternative to Suricata I've seen a lot in the field as well.
Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF and its supporting vendors. Suricata is a fork of snort.
Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator.
HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net.
HAProxy is used by a number of high-profile websites including GitHub, Bitbucket, Stack Overflow, Reddit, Speedtest.net, Tumblr, Twitter and Tuenti and is used in the OpsWorks product from Amazon Web Services.
Netfilter software comprises iptables, ipset, conntrack-tools, libnetfilter and more. It is the core of Linux firewalls and NAT. It consists of a set of hooks inside the Linux kernel and a number of utilities to manage callback functions.
Dnsmasq is a lightweight, easy to configure DNS forwarder and DHCP server. It is designed to provide DNS and, optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS. The DHCP server integrates with the DNS server and allows machines with DHCP-allocated addresses to appear in the DNS with names configured either in each host or in a central configuration file. Dnsmasq supports static and dynamic DHCP leases and BOOTP/TFTP/PXE for network booting of diskless machines.
Unbound is a validating, recursive, and caching DNS resolver.
The C implementation of Unbound is developed and maintained by NLnet Labs. It is based on ideas and algorithms taken from a java prototype developed by Verisign labs, Nominet, Kirei and ep.net.
Unbound is designed as a set of modular components, so that also DNSSEC (secure DNS) validation and stub-resolvers (that do not run as a server, but are linked into an application) are easily possible.
Automatically enable HTTPS on your website with EFF's Certbot, deploying Let's Encrypt certificates.
OpenVPN is a full-featured open source SSL VPN solution that accommodates a wide range of configurations, including remote access, site-to-site VPNs, Wi-Fi security, and enterprise-scale remote access solutions with load balancing, failover, and fine-grained access-controls. Starting with the fundamental premise that complexity is the enemy of security, OpenVPN offers a cost-effective, lightweight alternative to other VPN technologies that is well-targeted for the SME and enterprise markets.
WireGuard is an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPSec, while avoiding the massive headache. It intends to be considerably more performant than OpenVPN. WireGuard is designed as a general purpose VPN for running on embedded interfaces and super computers alike, fit for many different circumstances. Initially released for the Linux kernel, it plans to be cross-platform and widely deployable. It is currently under heavy development, but already it might be regarded as the most secure, easiest to use, and simplest VPN solution in the industry.
Web Servers
Two common webservers, each with its own user case and differences.
Apache, is a web server notable for playing a key role in the initial growth of the World Wide Web and in 2009 became the first web server to surpass the 100 million web site milestone. Apache was the first viable alternative to the Netscape Communications Corporation web server, and has since evolved to rival other Unix-based web servers in terms of functionality and performance. The majority of all web servers using Apache are Linux web servers. Apache is developed and maintained by an open community of developers under the auspices of the Apache Software Foundation.
nginx [engine x] is a HTTP and reverse proxy server, as well as a mail proxy server written by Igor Sysoev. It has been running for more than five years on many heavily loaded Russian sites including Rambler (RamblerMedia.com).
Basic HTTP server features
Serving static and index files, autoindexing; open file descriptor cache; Accelerated reverse proxying with caching; simple load balancing and fault tolerance; Accelerated support with caching of FastCGI, uwsgi, SCGI, and memcached servers; simple load balancing and fault tolerance; Modular architecture. Filters include gzipping, byte ranges, chunked responses, XSLT, SSI, and image transformation filter. Multiple SSI inclusions within a single page can be processed in parallel if they are handled by proxied or FastCGI/uwsgi/SCGI servers; SSL and TLS SNI support.
Code Management + Deployment
Jenkins is the Continuous Integration / Deployment tool I've seen the most, both the free and enterprise versions. I should note that github is more popular than Gitlab right now, but I see a lot companies making the switch.
Jenkins is a fork of Hudson and is a continuous build system for software projects.
The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project.
Jenkins is an application that monitors executions of repeated jobs, such as building a software project or jobs run by cron. Among those things, current Jenkins focuses on the following two jobs:
Building/testing software projects continuously, just like CruiseControl or DamageControl. In a nutshell, Jenkins provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity. Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.
Git is a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Git tracks changes in any set of computer files, usually used for coordinating work among collaborators.
GitLab is a on-premise or hosted Git repository management tool. It also includes code reviews, issue tracking, wikis, and continuous integration.
Infrastructure as Code
Docker will be your container software whereas kubernetes is the management platform for orchestrating multiple containers in production at once. Ansible is an up and coming configuration manager used to manage setups across many servers and is gaining popularity fast. Terraform is used for infrastructure provisioning where istio is used for service mesh monitoring (microservices).
Docker is an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.
Docker containers can encapsulate any payload, and will run consistently on and between virtually any server. The same container that a developer builds and tests on a laptop will run at scale, in production*, on VMs, bare-metal servers, OpenStack clusters, public instances, or combinations of the above.
Common use cases for Docker include:
• Automating the packaging and deployment of applications • Creation of lightweight, private PAAS environments • Automated testing and continuous integration/deployment • Deploying and scaling web apps, databases and backend services
Docker is available as Community Edition (CE) for free and an Enterprise Edition (EE) subscription with software, support and certification.
Kubernetes is an open source orchestration system for Docker containers.
It handles scheduling onto nodes in a compute cluster and actively manages workloads to ensure that their state matches the users declared intentions. Using the concepts of "labels" and "pods", it groups the containers which make up an application into logical units for easy management and discovery.
Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
Being designed for multi-tier deployments since day one, Ansible models your IT infrastructure by describing how all of your systems inter-relate, rather than just managing one system at a time.
It uses no agents and no additional custom security infrastructure, so it’s easy to deploy — and most importantly, it uses a very simple language (YAML, in the form of Ansible Playbooks) that allow you to describe your automation jobs in a way that approaches plain English.
Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.
The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.
Istio is an open platform for providing a uniform way to integrate microservices, manage traffic flow across microservices, enforce policies and aggregate telemetry data. Istio's control plane provides an abstraction layer over the underlying cluster management platform, such as Kubernetes, Mesos, etc.
Visit istio.io for in-depth information about using Istio.
Cockpit makes it easy to administer your GNU/Linux servers via a web browser.
Easy to use Cockpit is perfect for new sysadmins, allowing them to easily perform simple tasks such as storage administration, inspecting journals and starting and stopping services.
No interference Jumping between the terminal and the web tool is no problem. A service started via Cockpit can be stopped via the terminal. Likewise, if an error occurs in the terminal, it can be seen in the Cockpit journal interface.
Multi-server You can monitor and administer several servers at the same time. Just add it easily and your server will look after its buddies.
Monitoring Tools
Netdata is a free NewRelic / AppDynamic alternative, but it is important to note that both of these solutions are pretty common in the field. Same goes for the ElasticSearch - Logstash - Kibana (ELK) stack, which I've seen is being replaced by Splunk in many places, especially those looking to deploy their production infrastructure to the cloud.
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company.
Prometheus's main features are:
- A multi-dimensional data model (time series identified by metric name and key/value pairs).
- A flexible query language to leverage this dimensionality.
- No reliance on distributed storage; single server nodes are autonomous.
- Time series collection happens via a pull model over HTTP.
- Pushing time series is supported via an intermediary gateway.
- Targets are discovered via service discovery or static configuration.
- Multiple modes of graphing and dashboarding support.
Grafana provides a powerful and elegant way to create, explore, and share dashboards and data with your team and the world.
Grafana is most commonly used for visualizing time series data for Internet infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control.
Grafana works with Graphite, Elasticsearch, Cloudwatch, Prometheus, InfluxDB & More.
Grafana features pluggable panels and data sources allowing easy extensibility and a variety of panels, including fully featured graph panels with rich visualization options. There is built in support for many of the most popular time series data sources.
Netdata collects metrics per second and presents them in beautiful low-latency dashboards. It is designed to run on all of your physical and virtual servers, cloud deployments, Kubernetes clusters, and edge/IoT devices, to monitor your systems, containers, and applications.
It scales nicely from just a single server to thousands of servers, even in complex multi/mixed/hybrid cloud environments, and given enough disk space it can keep your metrics for years.
KEY FEATURES:
💥 Collects metrics from 800+ integrations Operating system metrics, container metrics, virtual machines, hardware sensors, applications metrics, OpenMetrics exporters, StatsD, and logs.
💪 Real-Time, Low-Latency, High-Resolution All metrics are collected per second and are on the dashboard immediately after data collection. Netdata is designed to be fast.
😶?🌫? Unsupervised Anomaly Detection Trains multiple Machine-Learning (ML) models for each metric collected and detects anomalies based on the past behavior of each metric individually.
🔥 Powerful Visualization Clear and precise visualization that allows you to quickly understand any dataset, but also to filter, slice and dice the data directly on the dashboard, without the need to learn any query language.
🔔 Out of box Alerts Comes with hundreds of alerts out of the box to detect common issues and pitfalls, revealing issues that can easily go unnoticed. It supports several notification methods to let you know when your attention is needed.
📖 systemd Journals Explorer Explore and analyze logs in systemd journals, on individual hosts and infrastructure-wide logs centralization servers. Find the needle in the haystack of your systems and applications errors and exceptions, with an intuitive and interactive dashboard.
😎 Low Maintenance Fully automated in every aspect: automated dashboards, out-of-the-box alerts, auto-detection and auto-discovery of metrics, zero-touch machine-learning, easy scalability and high availability, and CI/CD friendly.
? Open and Extensible Netdata is a modular platform that can be extended in all possible ways and it also integrates nicely with other monitoring solutions.
ElasticSearch is a distributed, RESTful, source-available search server based on Java
Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, Logstash comes with a web interface for searching and drilling into all of your logs.
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
Open source data visualization platform to interact with your data through stunning, powerful graphics. Download for free and give shape to your data.
Cloud Infrastructure
Two major players. There are other options out there, but these are the two I've seen the most in production based heavy use companies.
In 2006, Amazon Web Services LLC (“AWS”), an Amazon.com company, officially began offering developer customers access to in-the-cloud infrastructure services based on Amazon's own back-end technology platform. Today, AWS is a comprehensive cloud services platform, offering compute power, storage, content delivery, and other functionality that enables businesses to cost-effectively deploy applications and services with greater flexibility, scalability, and reliability.
The Azure cloud platform is more than 200 products and cloud services designed to help you bring new solutions to life—to solve today’s challenges and create the future. Build, run, and manage applications across multiple clouds, on-premises, and at the edge, with the tools and frameworks of your choice.
Find answers to the business challenges you face with an Azure solution that brings together everything you need—related products, services, and third-party applications. From Azure DevOps to business analytics to the Internet of Things, you’ll be up and running quickly with a scalable, cost-effective solution that works with your existing investments.
Deploy, manage, and scale cloud applications faster and more efficiently on DigitalOcean. We make managing infrastructure easy for teams and businesses, whether you’re running one virtual machine or ten thousand.
The purpose of this list is to provide a listing of solutions available and commonly used by system administrators in the field in 2019 for managing production systems and support. I encourage everyone and anyone reading this list to also check out the site "https://github.com/kamranahmedse/developer-roadmap" as it has additional information on a logical order to run and use these apps as well as additional topics that are not software solutions, but concepts that are important to learn.
Comments
Programmes like Putty are not needed in Bash. I just type
ssh server.com
(or more probably some nickname I given to that server) and I am connected.
great list, it helped me