Enhance FPM support

I've been thinking about this feature a bit differently from the other SAPIs that we offer. The primary reason being that php-fpm isn't the same as the other SAPIs we offer.

The experience that I have had in mind is that you have a *global* SAPI (dso, cgi, suphp, etc) and then you have the ability to set certain sites or users as "fpm pools" in the WHM MultiPHP interface. This allows for per-user processing, but not on a global level, if you want it on a global level it shouldn't be difficult to iterate on this plan to allow for a "on account creation, create fpm pool" type option.

The idea of a shared multi-user pool terrifies me for various security reasons.

with your php-fpm support implementation in EA 4, you are doomed to failure ... just installed latest version, tested and i see no way it will work properly...

some developers have prototypes for EA3, like this one https://github.com/magenx/cPanel-php-fpm,

you wasting time with unnecessary functions rather than moving forward with todays technology....

> with your php-fpm support implementation in EA 4, you are doomed to failure ... just installed latest version, tested and i see no way it will work properly...

I can tell you that it does work properly, you just have to know how to configure it.

I've attached a document that we haven't released yet (still awaiting validation internally), however it covers in detail how to configure EA4 + php-fpm. There are probably some steps missing (like enabling the service).

The implementation we deliver with the next iteration will use these same methods (in an automated fashion, of course). We are actively working towards getting these sorts of features in to EA4 however we have to make some feature parity with EA3 first, and that takes time.

no, you are missing a lot more, like creating php-fpm master service per user account, with its own master php-fpm config, and chrooting its pool to home directory... otherwise this account will be able to see/hack entier server....

php-fpm => php-fpm-{cpaneluser}

php-fpm.conf => php-fpm-{cpaneluser}.conf

www.conf => {cpaneluser}.conf

this way you can run completely isolated instance with personal opcode cache,

and stop/start/debug it without breaking other users...

The plan is per-user pools, the config specified in that article is enough to get you started, it's certainly not the entire config that we will be delivering when we release this feature.

Now, you mention copying over "php-fpm" which file in specific are you referring to there? I'm not entirely sure why you're breaking out php-fpm.conf and www.conf in this configuration, at that point I'd just provide a single configuration file for the user and manage the pool without the init script.

chroot is certainly an option we will be exploring as we go into this implementation. We've also considered putting each FPM pool inside it's own container so that it has as much isolation as we can reasonably give from the rest of the system.

php-fpm => php-fpm-{cpaneluser} is an init script in CentOS6, that will pick up those cloned config files.

in CentOS7 it will be systemd configuration.

but anyway, you can not start one master and fork any child process/pool from the same in shared environment. this will lead to some serious issues. you need to run all isolated.

thats why every cpanel user must have its own master php-fpm process/config and own pool in it.

master process takes few mb and no cpu, but from other side of course, memory consumption will depend on the number of active pools, every pool will use dedicated realpath and opcode caches.

googling around, and searching github, probably i see now the full picture.

thats why i was confused and found your config a bit jerky, after i tested lots of configurations, i was expecting to see in cPanel something really promising, but in fact you just loaded defaults.

Yes, PHP-FPM pools don't isolate users, for instance with pools all users share the same APCu cache and can access each other's cache.

The only way to fix this security issue on shared hosting is to have a full php-fpm instance per user. Seperate pools don't help from a security perspective and are only there to provide different settings to different users.

That is certainly a workable approach. So we're looking at single fpm service instances per user, with perhaps setting up pools on a per-domain basis (if doing so has any benefit).

At this point, memory management becomes a concern. Being able to do an ondemand setup on a per-user basis has big advantages when you're looking at scaling to a whole lot of users, but I see the security implications here. One work around that occurs to me is using socket activation (or a similar concepts for c6) where the pool isn't started until the first request and shut down after a certain number of minutes being idle. This approach has issues when you consider that:

1. the first request may be a bit slow

2. it still can have scaling issues if too many sites get hit at the same time

Thoughts?

My biggest concern with this feature is how it will scale. I can verifiably state that this will have HUGE benefits for users on single domain / few domain systems. I think we are going to have to develop a formula for determine whether we recommend using FPM to an administrator or not.

Unless I'm mistaken, Linux has quite a good memory management system and I don't think that alot of idle processes would be an issue, I also don't think that memory would be an issue.

We currently use Litespeed in Process Group mode and it doesn't use that much ram on a modern server (we have 256GB per server and only use about 100GB excluding file cache.

Under a competently administered system, you will be ok. What worries me is the people who buy 1gb VPSes and then put 1000 accounts on it.

Litespeed allows to choose between deamon mode (one since process that is forked) and process group mode (one master process per user that is forked only for that user). The advantage with Litspeed is you can set the expire time, we have set ours to 30 minutes so if no further calls are made the owner process is stopped.

I think like litespeed this should be up to the administrator to decide which mode he want's. Either using one pool per user (recommended for low to medium memory) or one instance containing a single pool for each user (recommanded for high memory servers).

I don't think the individual php-fpm instance should be the default recommended option, but an option with a warning that says to only use if you know what you are doing and you have enough memory on the serveur.

A VPS with only 1 GB ram and 1000 accounts would probably crash anyway due to other limits like maxing out the CPU.

Currently we are planning to do this in the 58 cycle (note: delays do occur, planning is the operative word there). Development should begin in the next week.

Good to hear Matt. This is a big one, as everyone already knows....

Thanks for the quick reply.

i gess good old chroot will save you from many problems:

1. chroot = /home/user
2. chdir = /public_html
3. php_admin_value[doc_root] = /public_html
4. php_admin_value[open_basedir] = /public_html
5. php_admin_value[upload_tmp_dir] = /public_html/tmp

thanks @backblaze but would user and group run as nobody for the process requiring all user file permissions as nobody?

Yes I agree, cPanel musn't forget that pools don't isolate users and only allow to have different settings for different users. One php-fpm instance per user is require for things like APCu cache to be only accessible by that user. Otherwise all users can read, modify and empty other user's cache.

When is the planned release for v60?

Okey. Will the PHP-FPM UI be available in that EDGE build?

It is already available in Edge inside of the MultiPHP manager when using EA4.

No, this feature requires being enabled on a per-domain basis. In the future we may iterate towards being able to enable FPM pools on every new account/domain however there are memory concerns with this approach for FPM.

Okey, what PHP handler will other account's use? FCGI?

How much memory would you say it would take per account if you use PHP-FPM?

Whatever the SAPI is set to for the server. I believe that fcgid will be available.

The amount of memory depends on a LOT of factors, if they are idle accounts, not very much at all. If they are semi-active, it can be quite a memory hit.

Nope, no particular reason. It's definitely in EDGE, I just completely failed to mention it. I'll address it more in the next update!

Very nice.

Some questions from what I see in the image:

- FPM will be able to be turned on/off independently from what PHP version you select as default? - FPM will be able to be turned on/off on a per account basis?

That's starting to look pretty good ! will it have options for other FPM settings or would we just change them in the correspinding php.ini ?

@Kent: Yup! Enabling FPM will be independent from the PHP version, and is enabled on a per-domain basis.

@Greg: For now it's just those that are adjustable in WHM, and the rest will need to be adjusted in the php.ini.

Thanks for responding, thats cool, if possible would be great to have the additional / slightly more advanced options there in the future would be great and save lots of time when making the same changes across all pools.

Greg, what sort of options would you like to see? the idea was to put in the options we knew everyone would use then recieve feedback on what else to add. We very purposefully made this UI to not have every single PHP-FPM options available as that would be rather overwhelming to users.

This is a great question, I personally used this file from cPanel to manually create my FPM pools - https://features.cpanel.net/attachments/1210

like kdean asked will this be read in v60 or is it likely to break the sites and require the removal of the manually created FPM pools, and then recreate using the GUI?

(obviously if the GUI could just scan and import the predefined settings from the manual FPM pools that would be much easier)

Hey @kdean and @greg. In version 60 we're implementing using FileMatch, rather than ProxyPass. Additionally, our implementation in version 60 will not work with the old manual directions. The old version should still work after you upgrade, but if you want to use the Multi-PHP manager in WHM you will need to remove those customizations and then implement again in 60.

Would you provide instruction on how to deactivate userpool FPM we did with build 58 and allow it to work with Multi-PHP FPM?

Thanks Benny, it's not as much of an issue knowing this ahead of time ! I can make a few preparations for this when v60 comes out :)

@JayPK: Sure, I can work to get something put together for users that need to convert over, but it should just be reverting your changes and then implementing through WHM. I'll take a look and see what we can do!

There's currently not a plan to allow users to enable this on the cPanel side, only through WHM.

hmmm okay, so we can enable users to use FPM by default right?

@hrace009 that unfortunately won't be possible in the initial release of our PHP-FPM feature. It may be possible with a post account creation hook, but I'll know more about that next week after we hit code freeze. I'll let you know once this is live!

finally there is something,

but please dont forget that you have to use php-fpm in chroot mode

otherwise every account can be hacked easily.

im sure cpanel has users chroot by default too.

also make sure in this case document root can be set properly, and php.ini per pool aswell

cause im not sure how multiple subfolders in the same home dir will work

like

/home/user/public_html/

/home/user/public_html/project_1/

/home/user/public_html/other_user_test_project/

waiting for release to test it.

thank you

chroot'ing each user's FPM pool into their /home/virtfs/%user% directory would be ideal. Except, /home/virtfs/%user% doesn't exist for every user and I'm not sure by what logic cPanel determines if a user gets a /home/virtfs/%user% directory.

Benny should have some more information on this shortly but the approach we are going with which is off by default is that if a user is in jailshell or noshell then they will be jailed automatically providing that /var/cpanel/feature_toggles/apachefpmjail exists.

The one thing to note is that this works based off a patch to fpm itself which performs a getpwnam whenever a new pool is forked off. This is done to avoid extra restarts to PHP-FPM which can be problematic for a variety of reasons.

You may view the patch at https://raw.githubusercontent.com/CpanelInc/scl-php70/master/SOURCES/php-7.0.x-fpm-jailshell.patch

I haven't followed all of this and I'm not entirely sure how you are going to implement all of this, but couldn't most (all?) of this be done in the php-fpm init script?

For example, modify /etc/init.d/ea-php56-php-fpm so that it checks to see if the virtfs directory exists for the user, and create it if it needs it. Something like

1. for u in `/bin/ls -A /var/cpanel/users`
2. do
3. if [ ! -d "/home/virtfs/${u}" ]
4. then
5. /usr/local/cpanel/bin/jailmount ${u}
6. fi
7. done

And automatically set each /opt/cpanel/ea-php56/root/etc/php-fpm.d/%user%.conf to chroot to

/home/virtfs/%user%

I'm sure it's not completely as simple as that, but this would avoid the need to patch the PHP code with custom code for FPM.

sparek: The problem with that approach is that FPM restarts would have to wait on thousands of executions of jailmount before coming back up, meaning that PHP will be unavailable for an undetermined amount of time while it loops through every user on the system.

Yea, but how many times does the master php-fpm process have to be restarted? And how long does it take to mount a user's virtfs?

Doesn't patching the fpm source to do this on each instantiation of an FPM pool create startup cost, per process? Seems to me, it would be more cost effective, process-wise, to do this at the beginning rather than during each individual process's start.

Plus, this gives you the advantage of unsetting an FPM pool's chroot - through the user's /opt/cpanel/ea-php56/root/etc/php-fpm.d/%user%.conf file - if you so desire. Which gives you a bit more configuration flexibility.

An addition to this would be to create a process, possible something in tailwatchd, that checks and insures that all user's have a virtfs jailmount all the time. Add a hook after account creation for creating the jailmount virtfs. Just something that insures that /home/virtfs/%user% always exists. Or is there a reason why every user shouldn't have their own virtfs mount? This kind of goes beyond the scope of this feature request, but if every user always has a virtfs mount, then patching the PHP FPM source is moot and really adding my tidbit of code to /etc/init.d/ea-php56-php-fpm is also moot.

The bit of code I posted only runs the jailmount command if a user's virtfs doesn't exist. If every user's virtfs mount always exists, the the process cost of checking this in the init script would seem to be negligible. Certainly less than doing it on a per pool startup check.

I'm not saying my way is the best, but I think it's a lot more simple. I think patching the PHP source with this code is kind of over complicating the issue.

I actually tested the timing on this.

An unmodified /etc/init.d/ea-php56-php-fpm - restarting it on one server took on average 0.4526 seconds

Then I modified /etc/init.d/ea-php56-php-fpm to jailmount users if the /home/virtfs/%user% directory didn't exist. Restarting it on the same server took on average 0.715 seconds.

That's a difference of about 0.26233 seconds.

I conducted this test by restarting /etc/init.d/ea-php56-php-fpm, sleeping for 60 seconds, restarting /etc/init.d/ea-php56-php-fpm, sleeping for 60 seconds, and restarting /etc/init.d/ea-php56-php-fpm. Then I took that total time needed to complete this, subtracted the 120 seconds that I slept and divided by 3

The modified /etc/init.d/ea-php56-php-fpm mounted 81 users. But it really only mounted those the first time. Each subsequent /etc/init.d/ea-php56-php-fpm restart would have found that /home/virtfs/%user% already existed and would have skipped over them.

Granted, this is a small test. But I think it shows that this is feasible. Especially if you consider doing a jailmount of each user after account creations and/or have a separate daemon on check that works to keep jailmounts mounted.

I suppose all of this basically boils down to: Why isn't every user automatically given a /home/virtfs/%user% jailmount? Why doesn't this stay consistently mounted? And is there any reason why every user shouldn't have a /home/virtfs/%user% jailmount? If every user consistently has a /home/virtfs/%user% jailmount then all of this is moot and chrooting can be done in each user's FPM pool.

I posed this question on the forums about a month ago, but didn't get a lot of traction.

@sparek-3 to answer your questions about virtfs: virtfs is tied to Jailshell. Unless a user's assigned jailshell (or noshell), then we don’t create virtfs.

But why?

That's the question I've never gotten an answer to.

I understand that virtfs was initially used for jailshell, but you can expand on this so much more and offer a jailed PHP environment. cPanel even created the /usr/local/cpanel/bin/jailmount binary at some point to create this virtfs for a user.

This is essentially what the php-7.0.x-fpm-jailshell.patch file is doing.

1. if ( 0 != system( jail_command ) ) {

is running the /usr/local/cpanel/bin/jailmount command for that user.

If every user already has a virtfs jailmount, this isn't needed. And I think it makes more sense to do this in the /etc/init.d/ea-php56-php-fpm init script anyway.

Unless there is a reason why every user shouldn't have their own /home/virtfs/%user% mount point.

Some interesting suggestions but I'm not agree with them all.

The chroot is really interesting and important, but it should be optional.

Most os cPanel shared servers (almost all mines) uses CageFS or similar solutions that do the same in better way.

The FPM chroot obviously changes the base dir, this will be a big problem in some ugly PHP systems that use absolute path in theirs scripts (/home/{user}/public_html), so in most of cases the sysadms will prefer a CageFS/Jailshell like solution instead a chroot.

About put all config in one or multi files, to be honest, I don't care. In the execution time the effect will be the same.

About the systemd, remember that this won't be only for RHEL 7. The 6 versions will need to run it too.

I'm also don't understand the socket locations and names. Thinking like a hacker I don't care about the sockets name, if I have access to some one the only think that I need to know is that is a FPM socket, I will try exploit it no matter the name they are.

To be honest, as the cPanel has becoming a RPM based system, it will be more interesting if it respect the POSIX and put the sockets in /var/run using the user names in socket name to help the identify. The ACLs of the socket make the security part.

I don't understand what do you trying to say about php-fpm.confs. The normal user don't need to had access to such file, I don't understand how "someone made a typo in one of the pool configs" without be a root?

These files only have the FPM limits and locations, I don't want that any users had access to it, so I don't see any these problems that you like.

And please forget about that crazy idea of make a service for each user. This will be terrible hard to keep, imagine you make some php.ini change and you have to make 1000 restarts?

This strange approach will also waste memory/cpu with excessive overhead for nothing.

Each php-fpm.conf for each user already their own configs to make logs, limits and locations without the need to make a separate service for them or even if all they are in the same conf file, and if you (because your users, even the resellers don't have access to these files) miss a typo, the service log or the stderr will say where the tipo is when you try to restart it.

Sorry for any bad english

For chroot support, touch /var/cpanel/feature_toggles/apachefpmjail and make sure the user has noshell or jailshell set as their shell.

As for the master process issue - the development team was adamant on a single master process per version with pools per domain. What are the security issues you are concerned with? We were not able to identify any major issues with the shared master process that we were unable to patch or fix with chrooting.

touched apachefpmjail, what exactly i should see after?

one php-fpm master config and process leads to security issues

if you enable opcache one account from one pool will be able to see and extract data from another account pool using opcode cache.

in case of keys collision, exploited wp-config.php from one account will be loaded into another account.

also again , all accounts are hostages of one process.

@Yuri we are talking about default setup, i think CageFS is from CloudLinux.

and there is no problem to restart multiple services in one command, you should know that.

looks like you never had experience hacking one account from another on shared hosting, and believe me CageFS wont save you if i see your files and have code access...

I agree with the opcache issue, sharing files is very bad, I'm currently using FPM on a few sites they have the individual pools configured as per the cpanel docs (https://confluence2.cpanel.net/display/CKB/Configure+PHP-FPM+with+User+Pools+for+EasyApache+4) and they share the same cache, if I load an opcache (domain.com/opcache.php) script onto one of the domains and look at the cached files I Can see all the other sites cached files and their location. Meaning a single user could view everything in the pool from other users.

I'm not sure if I'm allowed to post this (If not I'm sorry !) but I have created a request already asking for a WHM opcache integration with separate opcache pools for each user perhaps this could be of use... https://features.cpanel.net/topic/zend-opcache-user-interface-for-whm-cpanel-with-separate-caches-for-each-cpanel-account

As far as the opcache issue goes, would using Xcache instead of PHP's opcache be a solution? At least for PHP 5.6. You could set xcache.var_namespace_mode = 2 and xcache.var_namespace="uid" would that work to separate virtualhosts? Or at least separate users on a shared server. Granted, xcache hasn't been updated in 2 years and it doesn't appear to support PHP7 (is it a dead project?)

From what I understand its basically recommended for everyone using PHP5.5+ to move to opcache from the other ones such as xcache, APC etc and then in terms of PHP7 I'm not sure the other ones are even fully compatible but could be wrong there

Perhaps I'm not the best one to comments on this. As I understand it Xcache is an opcode cacher, just like Zend OpCache that is included with PHP. It is true, that the general consensus is that Zend OpCache works better than Xcache. And, Zend OpCache seems to work with PHP 7, something XCache doesn't. BUT, if Zend OpCache is going to reveal information about other VirtualHosts, then it really doesn't make sense in a shared hosting environment, correct?

So the ability to use Zend OpCache - at least in terms of with PHP-FPM - becomes moot if the security implications of it are that great. So while Zend OpCache may perform best, it's useless if it doesn't provide security.

That's where I'm suggesting (or inquiring about) Xcache. XCache is an opcode cacher, like Zend OpCache. It does not work as well as Zend OpCache and it does not work for PHP 7. But with the namespace feature in XCache, can you isolate the caching per VirtualHost (or per user)? That's what I'm asking.

Ideally, Zend OpCache would include the namespace feature in XCache allowing the opcode cache to be isolated per VirtualHost (or per user). But it doesn't.

XCache, while it doesn't perform as well as Zend OpCache, it's better than nothing, right? (Assuming you are using PHP 5.6 or PHP 5.5). And if XCache can take care of the security implications of isolating opcode cache per VirtualHost (or per user), then isn't it a better choice?

Granted, this does nothing for users that want to use PHP 7 or PHP 7.1.

Zend OpCache might work with FCGI, but then don't you lose the ability to chroot your PHP environment as you do in FPM?

I'm not completely advocating XCache, I'm just asking if it can be an alternative that might provide solution to some of the issues being addressed here.

I've been testing the FPM jail chroot (/var/cpanel/feature_toggles/apachefpmjail). I seem to have run into a snag.

It would appear that there is a limit to the number of jailmounts you can have, or at least something that stops further jailmounts after a certain period. At least in an OpenVZ environment. This may be a limitation or conflict with something in OpenVZ.

It would seem that after mounting 157 virtfs jailmounts, further jailmount attempts are not successful. /home/virtfs/%user%/home/%user%/public_html doesn't populate. I doubt that 157 is an arbitrary limit, in fact I have as many as 219 succesful jailmounts in a KVM environment. I suspect that the limit has to do with something else and I think it may have something to with some restrictions in OpenVZ.

This is not an issue specifically with /var/cpanel/feature_toggles/apachefpmjail but instead is an issue with the ability to create jailmounts. I am not able to successfully create jailmounts using the /usr/local/cpanel/bin/jailmount command either.

I will see if I can do some further testing on this when I get caught up on some other work. But in the mean time, I thought I would mention this and see if anybody else has been able to replicate this issue. Specifically within an OpenVZ environment and with more than 157 users (or at some point, do the users stop being successfully jailmounted?). Of note, the /usr/local/cpanel/bin/jailmount command may create the /home/virtfs/%user%/home/%user% structure, but fails to create the /home/virtfs/%user%/home/%user%/public_html structure.

After more investigation on this, the mount failures are caused by a restriction on the number of mountpoints inside OpenVZ.

By default (I assume), OpenVZ containers are limited to 4096 mount points (of which, I can only get 4091 mounts, not sure what happened to the other 5). This is controlled by the variable - /proc/sys/fs/ve-mount-nr

You can check the total number of mounts by:

cat /proc/mounts | wc -l

And if you are in an OpenVZ environment, you can check your max mount points by:

cat /proc/sys/fs/ve-mount-nr

This value can only be increased by the host node, either by directly increasing the limit:

echo 10000 >/proc/sys/fs/ve-mount-nr

Or, probably more appropriately through the /etc/sysctl.conf file on the host node, by adding or modifying the variable:

fs.ve-mount-nr = 10000

And saving the file and running sysctl -p

(10000 is just an example in this case, I'm not sure if there is an upper limit that this can be set to)

But this will be dependent on how willing your VPS provider is to make this change on their host node. So bottom line, some OpenVZ servers may experience problems with this PHP-FPM chroot if they are unable to get fs.ve-mount-nr variable increased. It's hard to say exactly what the user limit would be for this, as it depends on the total number of mountpoints you have and how many (if any) additional jailshell mountpoints you have. But I'd say the limit is going to commonly be some where between 150 and 200 users.

Nothing really cPanel can do about this as it's not a cPanel issue, but is an OpenVZ issue. But it might be worth putting an advisory of some sort attached to this PHP-FPM chroot if the environment is OpenVZ.

Does this limit affect any other server environments? I'm not seeing the issue on standard, KVM, or Xen based environments (at least not yet) but might be worth further testing.

The FPM pool should be automatically chrooted after touching this file, ignoring configuration file settings.