Update Linux Packages (including major services) in a Method Rather than Using Yum Daily
Open Discussion
As a server administrator I would like to see cPanel manage RPMs for all services that it supports in order to help reduce the likelihood that a major or critical bug is allowed to effect the users on my cPanel & WHM server.
- Original Request
- This information is shared privately in spoiler and not visible!
I like this idea 
cPanel, WebHost Manager and WHM are registered trademarks of cPanel, L.L.C. for providing its computer software that facilitates the management and configuration of Internet web servers.
Update Linux Packages (including major services) in a Method Rather than Using Yum Daily
Cpanel is moving all packages from their RPM repository over to yum, and they will all update daily.
Currently most major packages are NOT updated daily by Yum, an exception being MariaDB.
MariaDB recently released 10.1.15 with a major bug that crashed mysql, rendering sites useless, and also affected mod_security creating a security concern.
MariaDB developers quickly pulled 10.1.15, but for cpanel servers running 10.1.x it was too late, because yum had already updated to this non-sufficiently tested version.
Running yum daily has created problems previously, for example a few years ago, a 64-bit rpm was accidentally uploaded to the 32-bit repository (or it may have been a 32-bit posted to a 64-bit), which resulted in crashed servers.
By introducing several critical new services to daily yum updates, instead of through the Cpanel RPM respository, it only increases the likely hood of major problems with these releases which haven't been tested, and have had little exposure to the open source community as whole.
Note: For serious security issues, I still recommend running yum right away.
Updating everything daily, no matter what, would be very risky for a production environment, and there should be at least a minimal delay and preferably at least minimal testing before installing new minor version of at least the major services (except for serious security issues).
Otherwise history will surely repeat itself and cause a major catastrophe, if not with MariaDB, then with another service.
Please post your feedback on this.
Update Linux Packages (including major services) in a Method Rather than Using Yum Daily
Cpanel is moving all packages from their RPM repository over to yum, and they will all update daily.
Currently most major packages are NOT updated daily by Yum, an exception being MariaDB.
MariaDB recently released 10.1.15 with a major bug that crashed mysql, rendering sites useless, and also affected mod_security creating a security concern.
MariaDB developers quickly pulled 10.1.15, but for cpanel servers running 10.1.x it was too late, because yum had already updated to this non-sufficiently tested version.
Running yum daily has created problems previously, for example a few years ago, a 64-bit rpm was accidentally uploaded to the 32-bit repository (or it may have been a 32-bit posted to a 64-bit), which resulted in crashed servers.
By introducing several critical new services to daily yum updates, instead of through the Cpanel RPM respository, it only increases the likely hood of major problems with these releases which haven't been tested, and have had little exposure to the open source community as whole.
Note: For serious security issues, I still recommend running yum right away.
Updating everything daily, no matter what, would be very risky for a production environment, and there should be at least a minimal delay and preferably at least minimal testing before installing new minor version of at least the major services (except for serious security issues).
Otherwise history will surely repeat itself and cause a major catastrophe, if not with MariaDB, then with another service.
Please post your feedback on this.
Yes, there should be a minimal delay before adding new minor versions of the services that are critical in a cPanel server and they should not push it through yum daily.
Yes, there should be a minimal delay before adding new minor versions of the services that are critical in a cPanel server and they should not push it through yum daily.
I wanted to clarify a few things here, just to make sure everyone reading this request is on the same page.
The situation that triggered this request was a bug case making it through MariaDB's testing into a public release. When cPanel added support for MariaDB we decided against distributing our own RPMs, and have been using the system provided ones. These are distributed outside of cPanel's control, but were installed overnight as the system updated automatically.
This differs from how cPanel manages some other service RPMs (MySQL, for example), which is what triggered this request. There's another feature request that is similar: Operating System / Installs Only Security Package Updates
I wanted to clarify a few things here, just to make sure everyone reading this request is on the same page.
The situation that triggered this request was a bug case making it through MariaDB's testing into a public release. When cPanel added support for MariaDB we decided against distributing our own RPMs, and have been using the system provided ones. These are distributed outside of cPanel's control, but were installed overnight as the system updated automatically.
This differs from how cPanel manages some other service RPMs (MySQL, for example), which is what triggered this request. There's another feature request that is similar: Operating System / Installs Only Security Package Updates
Thank you for that clarification, I'd like to further clarify that it was that bug plus previous poor experiences when updating daily, plus the fact you're getting rid of the cpanel RPM management altogether (according to Jesse A.) which will expand the number of critical services that are updated daily.
Regarding the cpanel RPM repository you have already removed Perl from it and over to yum, and going forward all(most) packages will be removed and managed by (and updated daily* by) yum including exim, dovecot, pure-ftp, spamassasin, etc. and possibly PHP and Apache as well. (*if there was an update released that day).
The feature request: Operating System / Installs Only Security Package Updates is just short of never updating, where as this request is more about updating responsibly, a happy medium between that and updating daily.
I'm all for receiving updates, including non-security bug fixes and new features, but I don't think it would be appropriate for a production environment to update everything daily, and certainly don't need new features on the very same day they are released with little testing and use by the open source community.
Thank you for that clarification, I'd like to further clarify that it was that bug plus previous poor experiences when updating daily, plus the fact you're getting rid of the cpanel RPM management altogether (according to Jesse A.) which will expand the number of critical services that are updated daily.
Regarding the cpanel RPM repository you have already removed Perl from it and over to yum, and going forward all(most) packages will be removed and managed by (and updated daily* by) yum including exim, dovecot, pure-ftp, spamassasin, etc. and possibly PHP and Apache as well. (*if there was an update released that day).
The feature request: Operating System / Installs Only Security Package Updates is just short of never updating, where as this request is more about updating responsibly, a happy medium between that and updating daily.
I'm all for receiving updates, including non-security bug fixes and new features, but I don't think it would be appropriate for a production environment to update everything daily, and certainly don't need new features on the very same day they are released with little testing and use by the open source community.
In my humble opinion, automatic updates are a bad thing to begin with. But you also have to keep your system up to date. The way to do that is to monitor for updates and then apply updates on your own. But the sad truth is that if most people don't turn on automatic updates, then things will never get updated. People are either lazy, don't understand how to monitor and apply updates, or just don't care.
I think a better solution for all of this, would be to allow for version rollback. I'm not entirely sure how this would be done with yum, I know it can be done with apt on Debian (although I would have to look up instructions on how to do it, it's been a while). For example for MariaDB (which I'm not using, so forgive me if I have the versions wrong), when MariaDB 10.1.15 is released leave MariaDB 10.1.14 and MariaDB 10.1.13 in the cPanel repositories, and allow administrators to overwrite MariaDB 10.1.15 with MariaDB 10.1.14 or MariaDB 10.1.13.
Then if you update to MariaDB 10.1.15 (whether automatic updates or manual updates) and see problems, you can downgrade back to MariaDB 10.1.14 without having to raise this issue with cPanel. That would allow people to resolve issues themselves quicker. I think a lot of the trouble regarding this issue is the time it take to get cPanel's attention regarding a matter and then getting rollbacks or fixes dispatched into the repository. The more individual administrators can do to resolve their own issues, the faster those administrators can resolve the issues themselves.
A rollback should exist for every package that cPanel provides (should be done for every package from every repository, but I'm not sure if that's the case).
What would also be helpful would be an always up to date page on cPanel's website that lists the latest version of every cPanel provided application and what versions also exist in the repository. It might say that MariaDB 10.1.15 is the latest version of MariaDB but that MariaDB 10.1.14 and MariaDB 10.1.13 also exist in the repository.
In my humble opinion, automatic updates are a bad thing to begin with. But you also have to keep your system up to date. The way to do that is to monitor for updates and then apply updates on your own. But the sad truth is that if most people don't turn on automatic updates, then things will never get updated. People are either lazy, don't understand how to monitor and apply updates, or just don't care.
I think a better solution for all of this, would be to allow for version rollback. I'm not entirely sure how this would be done with yum, I know it can be done with apt on Debian (although I would have to look up instructions on how to do it, it's been a while). For example for MariaDB (which I'm not using, so forgive me if I have the versions wrong), when MariaDB 10.1.15 is released leave MariaDB 10.1.14 and MariaDB 10.1.13 in the cPanel repositories, and allow administrators to overwrite MariaDB 10.1.15 with MariaDB 10.1.14 or MariaDB 10.1.13.
Then if you update to MariaDB 10.1.15 (whether automatic updates or manual updates) and see problems, you can downgrade back to MariaDB 10.1.14 without having to raise this issue with cPanel. That would allow people to resolve issues themselves quicker. I think a lot of the trouble regarding this issue is the time it take to get cPanel's attention regarding a matter and then getting rollbacks or fixes dispatched into the repository. The more individual administrators can do to resolve their own issues, the faster those administrators can resolve the issues themselves.
A rollback should exist for every package that cPanel provides (should be done for every package from every repository, but I'm not sure if that's the case).
What would also be helpful would be an always up to date page on cPanel's website that lists the latest version of every cPanel provided application and what versions also exist in the repository. It might say that MariaDB 10.1.15 is the latest version of MariaDB but that MariaDB 10.1.14 and MariaDB 10.1.13 also exist in the repository.
I also think there is some confusion over RPM and yum.
RPM is just a packaging system for applications. It is designed to install precompiled binary versions of an application.
Yum is a system of installing RPMs from a yum repository. When you execute yum install application you are essentially telling yum to download the application RPM from it's defined list of repositories and install the RPM.
Before yum, Redhat based systems installed application by manually downloading RPMs and using the rpm command to install RPMs on a system. But when you went to install an RPM it would complain "you don't have such and such appplication installed, install it first", then you'd have to go find that RPM install it - unless you got another "you don't have another such and such application installed" and you'd have to go get it. This was known as dependency hell. Because pretty soon you were downloading levels and levels of RPMs and installing them and you'd forget what application you were trying to install. Because there was no repositories of RPMs (just various FTP sites with RPMs) there was no system for knowing when an update to the package was released. So you have to stay in-tune with the development of the application and know when an updated RPM was released and manually update the RPM. This is how Linux operated in the 90s.
Yum simplifies this. The idea being that you keep RPMs in a particular repository, so when you go to install an application through yum, it searches for dependencies, sees what you have installed on your system, if you don't have a certain dependency it downloads and installs those dependencies as well.
Yum and RPM are two different things, but they both fell from the same tree.
I think where the confusion is coming is that the OP believe that yum has to run an update daily. That's false. Typically yum runs an update everyday because people are lazy and unable to monitor their system and keep their packages up to date. You can just as easily disable daily yum update and manually run yum check-update or yum update to check for and apply system updates.
cPanel may have removed the perl RPM from being directly downloadable from cPanel. But yum install cpanel-perl-514 does the exact same thing as downloading the RPM and manually installing it.
I also think there is some confusion over RPM and yum.
RPM is just a packaging system for applications. It is designed to install precompiled binary versions of an application.
Yum is a system of installing RPMs from a yum repository. When you execute yum install application you are essentially telling yum to download the application RPM from it's defined list of repositories and install the RPM.
Before yum, Redhat based systems installed application by manually downloading RPMs and using the rpm command to install RPMs on a system. But when you went to install an RPM it would complain "you don't have such and such appplication installed, install it first", then you'd have to go find that RPM install it - unless you got another "you don't have another such and such application installed" and you'd have to go get it. This was known as dependency hell. Because pretty soon you were downloading levels and levels of RPMs and installing them and you'd forget what application you were trying to install. Because there was no repositories of RPMs (just various FTP sites with RPMs) there was no system for knowing when an update to the package was released. So you have to stay in-tune with the development of the application and know when an updated RPM was released and manually update the RPM. This is how Linux operated in the 90s.
Yum simplifies this. The idea being that you keep RPMs in a particular repository, so when you go to install an application through yum, it searches for dependencies, sees what you have installed on your system, if you don't have a certain dependency it downloads and installs those dependencies as well.
Yum and RPM are two different things, but they both fell from the same tree.
I think where the confusion is coming is that the OP believe that yum has to run an update daily. That's false. Typically yum runs an update everyday because people are lazy and unable to monitor their system and keep their packages up to date. You can just as easily disable daily yum update and manually run yum check-update or yum update to check for and apply system updates.
cPanel may have removed the perl RPM from being directly downloadable from cPanel. But yum install cpanel-perl-514 does the exact same thing as downloading the RPM and manually installing it.
Replies have been locked on this page!