WHO DID WHAT WITH ROOT?!

When you are not sure who is using SUDO on a server, and you really need to know who keeps making that annoying change.  You can install something to watch them, and maintain that software and related logs. Keep it setup in your package management system, and make sure it doesn’t have any patches.

OR

You could use the little-known (at least those I have asked in the field) modifications I will list below.  They are two fold.  One, you will enable to record who logs in and uses SUDO, and records their session. Much like many pieces of software out there today.  The one catch to my method is simple.  You already have the software installed, yup this has been a feature of SUDO since version 1.7.4p4.  So nothing else to install, worry about, or maintain.  It is also very easy to setup, see below:


/etc/sudoers modifcation:
All you need to do is to add 2 tags to all required sudoers entries.
*(where "su" specified, either with command or alias). 
LOG_INPUT and LOG_OUTPUT
Example: 
%admins ALL=(ALL) NOPASSWD: LOG_INPUT: LOG_OUTPUT: ALL

It will add the following default log dir structure to sudoers: Defaults iolog_dir=/var/log/sudo-io/%{user}
Note:
Output is logged to the directory specified by the iolog_dir option (/var/log/sudo-io by default) using a unique session ID that is included in the normal sudo log line, prefixed with TSID=.  The iolog_file option may be used to control the format of the session ID.  Output logs may be viewed with the
sudoreplay(8) utility, which can also be used to list or search the available logs.   Keeping in mind that if the user has a really long session you will be viewing it like a movie, it will replay as if he is sitting there typing.  With this in mind, sudoreplay gives you the ability to play back at faster speeds.  This makes it easier to find where things happened in a long recording.

So that is one good method to help find a culprit, but what if you are just looking at history of root?  Can you tell me who ran what? Can you tell me when they ran the commands you see when you type ‘history’?  By default, no.  The next tidbit of info is very useful, and extremely easy to add to your machines.  Simply add the following to your /etc/profile:

export HISTTIMEFORMAT="%m.%d.%y %T "

Yes, that is a space at the end.  If you do not put that in there you will end up with it running together with the actual command typed in history.  So your history should look like the example below:

1995 06.10.15 13:08:05 top
1996 06.10.15 13:08:05 clear
1997 06.10.15 13:08:05 df -h
1998 06.10.15 13:08:05 umount /media
1999 06.10.15 13:08:05 sudo umount /media
2000 06.10.15 13:08:05 sudo su –
2001 06.10.15 13:08:07 history

I hope this helps someone save some time, as it has me.  Please feel free to share with others.

-M

 

Why is everyone so mad at Redhat about CentOS?

First, what is a Rolling Release, and why is everyone so mad about it?

Well Wikipedia defines a rolling release as follows:

Rolling release, rolling update, or continuous delivery, in software development, is the concept of frequently delivering updates to applications. This is in contrast to a standard or point release development model which uses software versions that must be reinstalled over the previous version. An example of this difference would be the multiple versions of Ubuntu Linux versus the single, constantly updated version of Arch Linux.

Well, now we know what it is… why is everyone so mad?

It is because a rolling release, even though it is constantly being fixed can be quite unstable. This is not a huge deal for applications running in a desktop environment, but in a real production environment it is not acceptable.

To give an example, if you were running an application in a production. A library underneath could get updated, and break your application without any notice. In fact, it is common practice for many enterprise applications to “holdback” a version of an application; or even host the install files in their own repository to ensure nothing malicious makes its way in.

So in short, and in my opinion, Red Hat bought CentOS a few years back as they were becoming the competition. As everyone feared then, they are essentially making it a non-enterprise product. This is likely due to the large chunk of the market they are loosing to Ubuntu. Which is well deserved in my opinion.

Hope this was informational… Have a good rest of your 2020!

Abandoned by GoDaddy…

I have been a GoDaddy.com customer for hosting for several years now. I also have over the years used their products. Granted, this was more due to it just being there more than anything else.

However, after many years hosting this blog they have decided to no longer support updating libraries that keep WordPress working and safe (This refers to classic vps customers, was told to buy all new hosting). This has led me to have no choice but to abandon them as a whole and move hosting.

That being said, please be patient if there is any down time or service interruptions.

PS: Godaddy… if you are reading this….so are others…

-M

Azure is selling a BROKEN CLOUD. K8s.

Recently, I created a kubernetes cluster in Azure as a POC. I did this using Terraform to ensure it was infrastructure as code. This way it could be easily stood up again.

After jumping through tons of hoops to turn on a service that was no longer in preview, I was able to add a node pool for windows machines… In short, after a lot of hoops I was able to do so. This included getting cores quotas/etc extended. That took a few days because of the response time from Microsoft.

So, at this point I am heavily invested with my time (over a week of waiting and back/forth with MSFT). Now I have a K8s cluster up and running with a windows pool and Linux pool. It appears to be working…but this was a facade.

Once I started using the K8s cluster, I noticed a problem with all my deployments that had one of the following features:

  • Several Mounts (PVCs)
    • I found more than 3
  • Mounts Over 5-10gb

I tried reaching out to Microsoft via an azure support ticket. I was basically just given the run around, and asked to go through more hoops; all of which were for no more of a reason than to close the question in the techs queue.

Finally, I was able to get more information (only because of GitHub), see below:

This, again was a lie (even if not on purpose); as it has been well more than the original two weeks referred to. The original issue was opened March 2019! Also, after I tried to rebuild (per their suggestion) I was told they are out of cores; and “would I like to rebuild in another region?”, this started the quota requests again. That added 3 more days of waiting.

(Probably due to large government contract)

None of this behavior is enterprise grade, and quite frankly I don’t know why anyone would ever use this cloud. Please reference my previous post on their uptime. Keep in mind they are slightly more expensive that AWS, less robust, and less reliable.

This is the most classic case of “This is always how we have done it.”, and the nature of people to avoid change.

Enterprise Cloud? Not Azure…

Azure has only 99.95% uptime (Four 9s is standard, 6 is my personal minimum). The Azure cloud also has an incomplete UI, and a large portion of its services are either NOT theirs ( hosted in their “Marketplace”), or they are a part of their API which is constantly changing/invalidating infrastructure as code, such as Terraform.

However, Gartner says the following about Azure:

“Gartner finds fault with some of the platform’s imperfections. “While Microsoft Azure is an enterprise-ready platform, Gartner clients report that the service experience feels less enterprise-ready than they expected, given Microsoft’s long history as an enterprise vendor,” it said. “Customers cite issues with technical support, documentation, training and breadth of the ISV partner ecosystem.”

Root ZFS Soon, but now we know how soon (sort of)…

The following is a link to the current project in GitHub. This will give us what is hopefully an accurate picture of the timeline.

https://github.com/orgs/ubuntu/projects/1

Given the hopeful nature of devs and people like me, this will keep us in check for a realistic timeline. Wish I could help guys, but kernel programming is not in my wheel house (yet).

Root ZFS for Ubuntu… soon..

Dev at Ubuntu Dimitri John Ledkov who has previously been involved in this effort commented on the Ubuntu mailing list with this update: 

Zstd patches have not made it into the upstream kernel yet. 

As used by mkinitramfs: 
– lz4 is faster to compress than gzip 
– lz4 is blazingly fast to decompress 
– lzma is dog slow to compress and decompress, but is tiny 
– lz4 size weight over gzip is marginal (14%) but imho worth the improved boot time & initrd creation time 
– xz is potentially even slower and even smaller than lzma 

In places where size is an absolute premium (tiny embedded iot devices) and performance is irrelevant, xz or lzma should be used. 

In all other places, our performance profile is in favor of lz4. 

Imho that includes the kernel image itself, thus we should consider switching: 
– initramfs tools to default to lz4 
– livecd-rootfs to default to lz4 
– kernels to compress kernel image with lz4 
– grub to include lz4 support 

I shall proceed with changing the defaults on the above to improve our responsiveness experience on installer, cloud, core and classic devices. If our firstboot & subsequent boot speed degrades or disk space becomes a concern, we can look into tweaking these changes further.

So, we can hope for this in Ubuntu 19.10 for testing/updates before (hopefully) being released in Ubuntu 20.04 LTS.

Linux Gaming just got way better!

Since the release of ProtonDB, there have been over 2500 games added to Linux!

What is Proton?

Proton is a new tool released by Valve Software that has been integrated with Steam Play to make playing Windows games on Linux as simple as hitting the Play button within Steam. Underneath the hood, Proton comprises other popular tools like Wine and DXVK among others that a gamer would otherwise have to install and maintain themselves. This greatly eases the burden for users to switch to Linux without having to learn the underlying systems or losing access to a large part of their library of games. Proton is still in its infancy so support is inconsistent, but regularly improving.

ProtonDB is used to collaborate on performance on different hardware/software configurations. Allowing the community to work together to get the games working.

Setting Up Proton in STEAM – Tested Ubuntu 18.10

Deploy Large Stacks Easily in ANY Cloud with Juju

First, a lot of people will ask, “What is Juju?”.  I had also not heard of it; however after using it, I can say it is an amazing piece of software.  It is made by Canonical, and this is their description of it:

“Juju is a state-of-the-art, open source modelling tool for operating software in the cloud. Juju allows you to deploy, configure, manage, maintain, and scale cloud applications quickly and efficiently on public clouds, as well as on physical servers, OpenStack, and containers. You can use Juju from the command line or through its beautiful GUI.”  

Essentially, JuJu makes deploying large, complicated, and difficult piece of software a breeze.  It also has a great GUI, and CLI tools.  It also supports many clouds as you can see below. 

I am currently testing this for my use case.  However, if you have to manage several clouds/customers/etc, it has the potential to save tons of time from an operational perspective.

 

Want to install it? Super easy on Ubuntu via a SNAP:

sudo snap install juju --classic

This will setup a new controller on your system behind a NAT’d interface.  It will then walk you through the setup.  Also, I highly recommend a run through of the install documentation; especially if you are going to run it on your own hardware; or localhost (LXD).  Recommended for ‘easy’ setup are the following: Ubuntu 18.04+, LXD, and ZFS (if on your own servers). ZFS is highly  recommended by myself for many other reasons we can get into later, see ZFSonLinux.org.