Our features site is undergoing a refresh! Be sure to explore the revamped site and discover our latest product roadmap launching here on Monday, March 18th.

Allow users to install websites from version control

Jacob Brown shared this idea 9 years ago
Completed

Just a small feature request we think would be useful for clients utilizing web files that get version controlled through svn, git or mercurial.


Our proposed implementation would be to have a VC control tools (ie a GIT button) in the File Manager that allows users to pull/push/checkout/checkin repositories to the directory they choose, and keep it updated.


The current method is to download the zip from git and upload it via the file manager, over-writing files as it goes however we propose the VC method to allow users to roll back after a failed deployment.

Best Answer
photo

Hey folks! This feature was added to cPanel & WHM in Version 72, which is available in the CURRENT tier now! Take a look at the release notes, and see all of the highlights for this version on the 72 Release Site.

https://documentation.cpanel.net/display/72Docs/72+Release+Notes#id-72ReleaseNotes-NewGit%E2%84%A2VersionControlfeature

Replies (31)

photo
2

Very interesting. How would you expect database migrations (upgrades to the schema, etc) or cache flushes to be handled?

photo
1

Very good point. In terms of CI integration and the like, I'm not totally sure however I imagine that if you're utilizing something some sort of automation, you wouldn't be working with cPanel. Especially now that people are moving towards Docker and it's friends.


Majority of scripts do release an "update" script which handles database alterations. Most scripts I've worked with also don't VC config files, and handle updates to the config file through a script designed to non-destructively change it. (ie wp-config.php is wp-config.php.sample)

photo
1

We could look for a Makefile and run make if we find it. Developers could lump in whatever their needs are into the default make task and dispatch the required tasks from there.

photo
1

Symfony/doctrine and Laravel use the so called migrations that are updated or rolled back through a console command.

Another command also rebuilds the composer autoload cache.


Perhaps along with the remote URL in each site the user could define a couple of commands to be executed after the synchronization.


Alternatively, something like the travis yml could be loaded in the remote rep, to define the commands that need to be executed.

photo
3

This is a feature that I personally feel is extremely valuable to our more technical users, as this is how modern developers interact with code and push code around. When looking the ability to deploy from version control I can see quite a few advantages and disadvantages. When discussing this feature I will use 'git' however that does not mean we are unwilling to look into mercurial or svn support.


I do feel that just building this into the file manager, while possibly a great minimum viable product for this feature, is not harnessing the entire power of git for this feature.


Below are some of the various use cases that I have come up with while grokking this feature:

  • Ability to clone a 3rd party software repository and deploy/update a web app via git
  • Ability to maintain one's own web content from a 3rd party hosted git repository
  • Ability for the host to maintain a list of allowed git repositories that users can deploy software from.

The usual feedback I hear on this is that this is too technical of a feature for use by your average cPanel user.

photo
2

It is definitely how most devs push code around. It's how we've done it at all my jobs, it's how I do it for my personal projects and it's how my friends prefer to share code with each other.

Ideally, you would be able to:


  • Clone
  • Commit/add
  • Push
  • Pull
  • Revert
  • Switch branch

I like the idea of whitelisting/blacklisting repositories. Whether this should be a WHM or cPanel managed list, I'm not sure of. That might be getting a bit too crazy, but it would be good for companies (like the one I work for) who utilize cPanel for "microsites or blogs", but want to maintain that extra security where people can only download from our repos.


It is definitely too technical for the lower level cPanel users (guys who pay $30/month for 2GB of webspace because the hosting company will do most changes to their account, like creating emails, databases, subdomains, etc), but for the mid-level and higher-level cPanel users (casual devs, guys who aren't blase about computers, and professional devs), it would be a welcomed featured.


Most low level guys will just ignore it anyway :P

photo
2

I would better encourage some version controlling system (both remote and on server) so any changes under specific repository over a specific branch can be used for this.


I personally don't feel the cPanel will be interested as this feature involves moderator controls and understanding however for cPanel, I would love to see this as simple as it can i.e. some options where user can change his mode to developer mode and than all such settings may start appearing.


Anyways, I am strongly supporting this feature as in many cases with me, I have to alter system configurations and have to provide shell access to my clients requiring this feature with cpanel which makes the system unstable farmost time.

photo
1

Version control from the shell can already be accomplished. After reading this I decided to check and can confirm that I can checkout files from my SVN repo using a "normal shell" and "jailed shell" using a cPanel accounts login.


What I would really like to see is it implemented into the file manager so that SSH isn't required.


Going by Adams response, I feel that they might be interested and I also feel that it would be a nice additional feature for their hosts to be able to boast about.

photo
1

Honestly, this feature is not necessary. There are more important features that cPanel needs to focus their time on than something the "more technical users" as one puts it, can simply setup themselves. SNV is just about obsolete, and who uses Mercurial? Git is what most everyone uses, and the ones that don't, should.

BUT... What would be nice is if cPanel installed Git by default with itself, like it does everything else, that way developers working on cPanel servers that don't have root access can setup Git repos per project/user.

IF... you do have root access, it is fairly simple to setup Git on cPanel servers. I currently am and have been using Git for my projects for some time now on cPanel servers with no issues. The setup isn't much different, just requires a few extra steps to get it working the "cPanel" way.

As Root:

1. Verify that the /etc/yum.conf file does not contain perl* in the exclude line.

2. yum install git

3. nano /opt/suphp/etc/suphp.conf

change: allow_file_group_writeable=true

change: allow_directory_group_writeable=true


As User: (not root, or you will run into all kinds of permission errors)

1. Enable shell access for user account - Jailed Shell or Normal Shell (Doesn't matter, they both work)

2. Login As User, and setup your repo...

I personally like to initialize my repo before the public_html directory, and follow the cPanel structure in my repo for better compatibility with cPanel severs. But, everyone has their own weird way of doing things, so do whatever makes the most sense to you.

My Way example:

SSH to /home/some-user-account

Type: git init

Type: git config --global user.name "FirstName LastName"

Type: git config --global user.email "youremail.com"

Type: git config --global core.excludesfile ~/.gitignore

Type: git remote add origin https://somerepo.git

/home/some-user-account/.gitignore: (ignore everything but public_html directory)


  1. /*
  2. !/public_html

This .gitignore file is basically for cpanel stuff, you can add another to the public_html directory to ignore project files. Git supports multiple .gitignore files and can be placed just about anywhere. They work the same exact way robots.txt files do. So if you want Git to ignore for example, your cache folders and all xml files, just create another .gitignore file and place it inside your public_html directory.


  1. *.xml
  2. cache/

Anytime you want to deploy, just login as user and type: git pull origin master (or whatever branch u want to pull).

photo
1

As I stated earlier (but after you posted this), you can do it from a shell provided that the web host sets it up on their server. But are webhosts going to invest the time and money into securely setting it up on all their nodes for their clients? Are they going to want to invest in that risk?


Larger ones; sure. Mid-tier and smaller companies; less likely.


Adding this feature would just be yet a nicer thing for cPanel to provide to their websites. At no point have I stated it's necessary, I just said that me - and some other devs I work with - would think it would be nice.

photo
1

I agree that it would be nice. Why I voted thumbs up, even though I highly doubt cPanel will invest the time to implement it.

photo
2

While highly technical in nature, this idea has potential for end-users if it's executed properly. The idea of designers providing a GIT repo URL to customers...and all the customer has to do is register it within cPanel and hit deploy...that has sexy written all over it.

photo
1

It's something cPanel could roll out in multiple stages over time. A good start would be to just have Git installed at server level by default, so its available at the user level without having to have the server admin install/setup Git. This quick-fix solution would still require the admin to enable shell access for users, but at least Git would already be available to use once a shell is enabled.


Ideally and interface of some sort would be preferred, especially for the non-tech users or in cases where a server admin refuses to enable shell access. Again, this part of it could be rolled out over time. At least the quick-fix solution I suggested would solve the problem for devs that currently have shell access, but not root level access.


I know someone suggested building it into File Manager, but I personally think it would make more sense to build a separate Interface for this, e.g. like Cron Jobs, etc. More simple to maintain and scale since you would essentially be building a stand-alone Git client out of JavaScript.

photo
1

@bruce - your thinking is right in line with how we'd approach this project. As of right now git is already available to end-user cPanel accounts if they have shell access. Beyond that, though, I'm definitely thinking a dedicated interface for configuring git repositories at a file location. (Links to this interface could totally exist inside File Manager so that you could select a folder and start from there...but the feature itself would be outside of File Manager).

photo
1

I'm really curious about this feature. Some of my customers have been asking for this as they are mostly designers and do not have a lot of experience using SSH. Any comments on the status?

photo
1

We haven't considered adding this to the roadmap since I started in this position in March, but I can definitely bring it up again. With only 17 votes over nearly a year, it doesn't seem like this has much interest in the community.

photo
2

I have to use cPanel for clients on a regular basis and I find this extremely frustrating, so much so that I ended up using Plesk instead which has GIT control baked in and can be set up in about 3 clicks. Now I just use subdomains with protected directories that are all set up to automatically pull any changes from GitHub and everything works seamlessly - clients can see me update things in real time.


To confirm cPanelAdamF's statement: It is absolutely sexy.

photo
1

Staging / Development / Production sites with GIT through cPanel. That'd be awesome :)

photo
1

My team and I just had a serious discussion about this with intent to start planning it out. I'll have more to say after we explore the idea more...including a call for focus group members...when we get done with that exploration.


We're excited!

photo
1

Good news!!


A nice feature I like in other services is the ability to configure a webhook (for example at github) so that a pull is triggered whenever a new version is pushed on the master branch.

photo
2

yup, we're looking at that as the main way to be notified of new changes that need to be pulled in. We've also considered a scheduled pull daily (probably during the day rather than overnight).


As far as development goes, first thing is to make the first clones work. Then we'd tackle an 'update now' button...then start talking about triggered updates or regularly scheduled updates or both.

photo
1

Cool :)

Selecting which branch to probe will also be very useful for test/staging environments.The "overnight" thing is a bit dangerous.

What is overnight for one site might be working hours for another.

I would make the execution time configurable per case.

BTW, is this going to be a WHM or Cpanel feature?

Is composer included in installations?

photo
2

Right now our thinking is that this should be a cPanel feature at first. Keeping a directory up to date could be useful at the WHM layer, but we'd like to keep it simple at first.


re working hours - yeah, definitely. We'd rather updates to the site happen automatically & conveniently...and not cause panic at 1am during upcp.

photo
2

Something for your checklist if you are still gathering specs.

It would be nice to be able to configure pre and post sync scripts. That way things like cache clear, composer installations or updates and the like can be automated too.

Also, it would be nice to have a way to define environment variables and .env files (used by laravel and symfony), although this can be probably handled manually from the file manager.

photo
2

On it. We were already thinking about pre-sync/post-sync scripts. Hadn't thought of ENV variables, though, glad you said something about that!

photo
1

@dimkasta - Something to keep in mind is that the work for https://features.cpanel.net/topic/ruby-app-support-through-phusion-passenger will already have the ability to specify ENV variables for the launch of processes. Are you wanting additional set of ENV vars for the pre-flight and post-flight scripts too?

photo
1

Thanks for the tip


I will check it, although I do not use ruby


In any case, the file manager works ok for those files

photo
2

This is now officially planned for v68, baring unforeseen circumstances!

photo
1

Nice!

photo
1

A lot of this is already do-able in v64 and newer. We're prepping some documentation for advanced users comfortable at the command line that will describe the way to achieve it right now.


Don't worry. Still building out a nice new feature in the cpanel interface for it too ;-)

photo
1

New documentation for how to do it right now (v64 or newer) is now available.

https://documentation.cpanel.net/display/CKB/How+to+Host+Git+Repositories+on+a+cPanel+Account

As a result of working up this doc, the team and I identified three major scenarios that we need to plan for in the development of this feature:


  1. "Single Remote without automatic deployment" - basically just repository hosting. For people who don't want to use Github or one of the other special purpose hosts.
  2. "Single Remote with automatic deployment" - hosting a repo on your cpanel account with the expectation that new changes are automatically checked-out upon push
  3. "Multiple Remote with automatic deployment" - hosting a clone of a repository hosted elsewhere (eg Github) and automatically monitoring that remote for new updates.

I've got those roughly ordered in terms of complexity. 1 and 2 are already possible right now, and the new document explains how to do it via command-line. 3 is where the fun is ;-)

photo
1

Question for the room: How do you plan on marketing this feature? What would advertisements on your hosting company's website look like for this?

I ask because this will help guide us in how best to chop up this feature's capabilities and how best to schedule out which parts of the project we do first.

photo
4

"Dev like it's 2017, not '97" — perhaps? :p

photo
1

Laravel forge and envoyer might give you nice ideas

photo
1

What would your hosting Packages (like, WHM Packages) look like? Do we need to slice-up the feature into more than one 'feature manager'-entry so that you can fine-tune your offerings?

photo
1

Dev update for everyone:

We've broken ground on the interfaces to create and list git repositories. I'll have some early screenshots to share here pretty soon.

We did identify a couple of things now that we're getting started.

First, we know people will create git repositories from the shell since they have that ability so we're going to implement a 'discover repositories' button which will trigger a scan of the user's home directory for .git folders and update the list of repositories in the interface. (Pushing it more than once in rapid succession will only trigger one scan for that user at one time. Scanning file structures is slow and expensive so we're building it in such a way that it can't really be abused.)

Second, we've discussed performing a 'git stash --include-untracked' before receiving any git-push in order to clear out any local changes that might exist on the cpanel-remote. In the occasion where we end up git-stashing something, we'll shoot a note to the end-user saying "hey, we cleaned up something in order to perform your deployment. Here's the git-stash-apply command to restore it". What do you all think of this idea? Should we simply fail the deployment instead?

photo
2

Adam, someone that uses the shell, probably doesn't use cPanel and I suspect most don't even give shell access for security reasons on shared accounts. The idea and concept of cPanel is having a web GUI for site/server tasks, click and point and as easy as possible for non-computer persons. If this is the initial concept, great, but users will expect to do this from a visual interface as well eventually.

photo
1

The stash idea nice is in cases when someone makes an experimental change on the envronment and does not want to lose it

About the scanning, it really depends on how you are designing the interface. You could just add a folder browser and let the user detect for a git repository or create a new one just on the selected/working folder

photo
1

Stash is indeed good and should be there for sure, other wise any local changes would break the repo pull.

However, I think an auto pull would be great to have in place as well.


For instance on Github, BitBucket and Gitlab is possible to setup a hook when the repo gets updated, in that case a HTTP request will be fired to the given endpoint, the service behind the endpoint can be execute the necessary actions, in this case pulling the repo.


So I guess the implementation of this on cPanel would be, having an endpoint/webhook for each repo so it can pull the repo once it has been accessed.


In this way anytime we push new changes to git repository on github/bitbucket/gitllab, the repo on the cPanel will be updated automatically as well.


The flow is same as any CI is using to run the tests against the code changes, Heroku uses the same flow as well to keep the apps updated with the latest changes on app git repositories.

photo
1

Re shell access - Yup, we understand that too. We're not intending to require shell-access in order to set up a deployment...but we have to account for use-cases where they DO have it and they meddle (because users will meddle in our experience). Plus all of this is happening in user's home-directories so there's potential for meddling via the File Manager as well.


The first versions of this feature will require shell access due simply to the fact that it makes developing it a bit easier on our end. Later versions will remove that requirement and do as much on behalf of a no-shell user as appropriate.


Re webhooks - Already planning to provide webhook URLs for repositories which enable deployment! We're intending to offer them on all repositories that enable deployment. If you don't want to use them in to integrate deployments, you can still bookmark them and they'll work all the same.


As a behind the scenes, here's a chart we came up with that shows what's going to happen during a deployment and when it'll happen.

photo
1

Another feature that would push this to an entirely new level, is to allow the user to change the working root public directory.

That way, we can create folders to host different repos, versions and/or environments, make and test all building tasks and easily switch between them by pointing the site to some other directory.

This can make deployments, migrations, cache clears, cache warmups and total rollbacks much faster and allow us to minimize downtime.

This can already be done by using subdomains, but we cannot point the root domain to some other directory to make environment switches completely effortless and allow for instant rollbacks that allow you to troubleshoot the new environment right at the prod server

photo
2

@dimkasta - I hesitate to stray into the management of the root domain's docroot for this particular project (because other teams are in that arena...plus it's a major systemic change that is going to be rough to do for unrelated reasons) HOWEVER we can certainly prepare docs for how to do the "docroot-swap" dance with the git deployment feature for addon domains or subdomains.


I'd like to get this feature out the door and used in the real world some before we go about making the 'docroot swap' easier to do...but that's not to say no to the idea. Only 'not yet'. We're currently trying to design this in such a way that it's agnostic to whether your repository is in a docroot or not. (Meaning it'll act the same whether you're deploying a website or a set of files and relying on Indexes to list them OR whether you just want to use git over SSH or HTTPS). Once we get the basics down and implemented, we can refine it further.

photo
1

Yep, perfectly understandable approach.

Just for the record and to give you a few ideas, two deployment systems that I am currently experimenting with, are both using a symlink for the docroot.


You have a "releases" folder where you can clone your repo in folders named 1, 2, 3 etc

Once cloned, you hit your build script for the new folder (composer install, cache warmup etc) and the final step is to symlink your docroot to the latest folder inside "releases"


If anything goes sideways, you can "rollback" by just symlinking docroot to the previous release folder.


I don't know if you can tell, but I am really excited about all this :)

photo
1

Short back-end updates for everyone:


  • We are updating the version of git available at the command line to git 2.13, the version we ship in /usr/local/cpanel/3rdparty. This version adds support for a few git repo configurations we want to use. If users already have a non-system git in their path, we won't touch it.
  • We're introducing a new namespace in UAPI called VersionControl and designing it in such away that adding support for Mercurial (or, shudders, SVN) should be pretty straightforward to do when/if there's demand for it.


I'm hoping to have a public 'not even EDGE yet' build available for everyone to play with by the end of next week. (Don't hold me to that, lots can happen between now and then which might delay it). Such a build will have the new feature available in cPanel and provide the means to create and list git repositories as well as discover repositories that were created outside of the feature.

photo
1

Update on the release of a 'not even edge yet' build: not happening this week due to having to delay for the targeted security release we issued earlier this week, as well as putting some finishing touches on the creating and listing repositories portions of the feature. We want to include showing the "clone-able" URLs after you create a repository in the first preview release of it so that you can see the whole "create => clone => push/pull" lifecycle.


I'll set the new expectation to middle of next week and I'll have more details about what's in that preview release later on.

photo
6

Alright, here we go. The first incremental delivery of the feature is about to go into our upstream and be built for release in a "not even edge yet" build. Here's a brief about what it can currently do and some known issues to keep in mind:


  • Adds a "Git Version Control and Deployment" feature to the cPanel end-user interface, with access controlled via a checkbox in the Feature Manager
  • Provides the means to create new bare git repositories on your cpanel end-user account, with the means of specifying the file location in one's home directory where the git repo should go
  • Provides the SSH clone URL for each repository you create so that you can clone those repositories
  • Enables the full "clone, push, pull" development lifecycle once your SSH Access and SSH Keys are configured correctly
  • Provides the means to scan your home directory for git repositories that might already exist that the feature doesn't already know about
  • Shows a searchable sortable paginate-able list of repositories
  • Establishes a new VersionControl namespace of APIs for UAPI. The create method is already capable of creating a git repository by cloning from a third-party though the UI for doing so isn't ready yet.

Feature index page

6dea54ded08e5e08cbbcb0dd04804250

Repository Creation screen

c79a40258f4d98d123481b1b3741f66d

Some known issues:


  • Access to the feature currently requires the end-user to have SSH access enabled for their account and an optional valid authorized SSH key in place. We have plans to remove this restriction in future iterations.
  • If you delete a repository manually and hit 'scan', your deleted repository will remain in the list.
  • Clone URLs are composed using your primary domain, not any addon or subdomains. We aren't sure if this is a bad thing or not, but FYI.
  • Clicking on the 'Deployment Path' will launch the file manager at that location. Right now, because we're supporting only bare repositories, this means end-users will be dropped into a .git folder itself. If they muck about in here, it can potentially damage git's version history enough to cause repository corruption. The deployment path link will eventually lead them directly into their deployed working tree once the deployment scope gets started.
  • The feature icon is a placeholder. Expect a better one soon!

I"ll have build information available for everyone once the merge and build process finishes up.

Next up, we dive head-long into the deployment related scope of the project!

photo
2

Until it's possible for users to choose the folder where their main domain points to, we create "fake" main domains that don't point anywhere, and create their main domain as an addon domain. Enforcing the use of the main domain in the final version would make that part of the tool unusable.


The use of a fake main domain applies mostly to users who are likely to use git on a daily basis. I don't think we have many users who use git and work in the public_html folder.

photo
1

Could git be added to the limited shell for this to work? Another option with this option is setting the key only work for that set of commands.

photo
1

@denver - that is currently the plan, however we're operating under the assumption that the user has shell access for the first round of implementation. We hope to make this happen before we call it done!

photo
4

For those interested in installing a "not even edge yet" build and playing with this feature, please hit me up! adam.french@cpanel.net - I've got to send you install instructions!

photo
2

We were actually able to get the "clone from external repo" stable for this "not even edge yet" build as well. Here's an updated screenshot of the create page now:

b6dd09361f7dfd9e6b46e9376e112db2

Known Issues:


  • We designed it to work with http/s URLs but it *should* work with ssh:// clone urls as well. If you want to do this, make sure to generate an SSH public key for your server and offer it to wherever the third-party repo is located as authentication.
  • By default, we 'check out" the master branch. In future iterations we will provide the means to specify which branch you would like to deploy.

photo
1

Small update for everyone - our release team is updating git to 2.14.0 in the "not even edge yet" builds. If you're curious what's new between 2.13 and 2.14: https://github.com/git/git/blob/master/Documentation/RelNotes/2.14.0.txt - Doesn't really have much of an effect for our particular project, but figured I'd FYI anyway.

photo
3

Alright so we're coming up on a release deadline internally so I have to make some hard decisions about what makes it into v68 and what waits until the next release.

Right now, the feature provides for...


  • creating bare repositories
  • cloning an existing remote repository into your cpanel account
  • enabling deployment (AKA "ensuring that the working-tree of the repository on your cpanel account instantly stays up to date whenever new changes arrive"...it will also convert a Bare repository into a non-bare repository if needed)
  • disabling deployment (AKA "detaching the HEAD pointer for the current deployed branch so that nothing gets updated when new changes arrive")
  • Updating a repositories settings, including name and deployment preferences like what branch to deploy
  • triggering a deployment right now (AKA "if a remote repository is registered, performing a git fetch and checkout of the latest on the deployed branch. If there is no remote repository registered, ensuring that the working tree's state is correct compared to HEAD")
  • offering "SSH Clone URLs" to users with shell access so that they can clone a repository on their cPanel end-user account locally and then push changes. (This also accounts for those of you who run SSH on non-standard ports.)
  • deleting a repository (by moving it into the end-user's Trash folder for later disposal or automatic purge)
  • starting a new repository from within the File Manager, and autocompleting the repository path as you type it in the "Create a new repository" screens

As far as I can tell, that's what you can expect in the v68 version of this feature. I know many of you were looking forward to the deployment tasks portion of the scope as well as the "make it work under noshell" support but it looks like that's about as far as we can get for v68 and still have time to stabilize and fight bugs without introducing new ones.

The good news, though, is that this isn't the end of this feature's development. I've gotten go-ahead to continue work on this feature for at least another release after the holidays are over. The leadership of cPanel product development feel as strongly as you do on the potential for version control integration inside cPanel & WHM AND I can tell you from personal experience that my team and I ADORE working on this project. (Git is so freakin' cool when you pop the hood and have a good look.)

In another update, I'll post our thinking as far as what's next on our TODO list for the project. It would be incredibly helpful to get everyone's opinion on what's most important, and what can wait until later, for the next update to this feature!

photo
2

Personally this couldn't have come at a more opportune time. I have a WHM instance, but was recently looking into alternatives that allowed me to do this very thing, now i don't have to! Not sure what the release schedule is / when v68 is supposed to come out, but i for one can't wait. This will make it easy to replicate instances of my web application and keep them all up to date. Hoping that deployment tasks isn't too far off. Good work!

photo
2

@christopher It just arrived last night on our EDGE tier, if you'd like to try it out (somewhere non-production, ideally) right now! https://blog.cpanel.com/how-and-why-to-build-an-edge-server/

photo
3

We'll be updating the feature in the upcoming EDGE push of v68 soon. When we do, that'll be the debut of the new deployment set of features I've described previously.

We need your help and evaluation notes for that work to make sure we're meeting your expectations!

I do have some rather technical notes to share with regard to how we did it so that you understand some of the design decisions we've had to make. Here goes:

Git repositories take up space

All files hosted in a user's home directory are counted against any disk usage quota. This includes the ".git" directories of any git repositories they create. Keep this in mind when you're designing your web hosting packages for your customers. Git repositories can grow pretty quickly depending on end-user activity.

Deployment means "deploy right now"

When deployment is enabled, the net effect is that any new commits that arrive to the deployed branch get deployed RIGHT NOW. There are two scenarios to consider here:

First, if the cpanel-hosted repository is playing the classical git convention of a "origin repository", the end-user is performing a 'git push' in order to push changes up to it. When that git-push happens, any changes to the branch configured as the deployment branch will be deployed RIGHT NOW. No delay. If something in the cpanel-hosted repository working-tree conflicts with something in their push, the push will fail and they'll be told about it. It's an atomic transaction so there's no such thing as a "partially successful push" (thank you, git, for being awesome like that).

Second, if the cpanel-hosted repository is a "clone" of a remote repository elsewhere (like if the code is on github and they want it deployed to their cpanel account), new changes will arrive via a 'git fetch' and 'git pull'. You'll only see available branches from the REMOTE repository when you go to choose a branch to deploy. This is intentional as we want to treat the cpanel-hosted repository as much like a mirror of the remotely hosted repo as possible.

When new changes are 'git pull'-ed into the cpanel-hosted repo, we will only perform the deployment if it can be 'fast-forwarded' into place. This is to, again, avoid any local changes to the cpanel-hosted repository that could eventually diverge from the remotely hosted repository.

Remember: you can push/pull to your hearts content to any branch OTHER than the deployed branch at any time. We pretty much ignore everything that happens to your repo beyond changes on the deployed branch.

Use git submodules? We got you fam.

All of the git operations cPanel performs on your repositories are submodule aware. Want to clone a remotely hosted repository onto your cpanel account? If it has submodules, those get cloned and initialized too. Was there new changes upstream on a submodule you're using? Just trigger a deployment and the git commands we apply to your repository will recursively retrieve any new changes for your submodules.

Enabling deployment on a bare repository means we have to convert it into a non-bare one

This is just a housekeeping thing. If you create a 'bare' repository...and then later on decide to enable deployment on it...cPanel will happily convert it into a 'non-bare' repository for you. This involves temporarily moving your repository out of the way and then relocating its contents into a .git/ folder inside folder at your repositories previous location...and then perform a checkout of the most recent change on your deployment branch.

"But Adam, doesn't this mess up my clone URLs if I have that repository cloned locally?" you ask. Nope, it doesn't. Git has some magic in place that handles this cleanly. You won't have to update your git-remotes locally because it'll find what it needs at the URL provided just fine. Git's kinda neat that way ;-)

Deleting a repo means sending it to the trash

So you can create and deploy git repositories all you want now, but what about deleting it? When you delete a repository, cPanel will move the git repository AND its working tree contents into your "~/.trash" directory for the trash system to purge. Depending on your settings in "WHM » Tweak Settings", the "~/.trash" folder will get purged instantly or on a regular schedule for your end-users.

If you need to recover a git repository for an end-user, definitely check their "~/.trash" directory first. Hitting the magic 'Restore' button in File Manager sets everything back the way it was!

Don't muck about in a .git folder

So all of this git-foo is happening in the end-user's home directory...which means they're going to see the ".git" directory if they have "Show Hidden Files" enabled in File Manager. PLEASE discourage your end-users from mucking about in that directory because they can do some real damage to its integrity.

ALSO let us know about how frequently you see trouble tickets from your customers where they've gotten themselves confused about whatever-the-heck the ".git" directory is. If this happens often enough that it's causing you headaches, we'll re-evaluate our implementation and hide these further. We know end-users "fiddle" and get themselves into trouble and this could potentially be an avenue for that.

Ok so that's it for this update. Definitely hit me up in the replies with questions!

photo
1

Thanks very much for this fantastic looking feature! I do a lot of work with the flat-file CMS Grav using Git repositories to hold site content and the existing Git Sync Plugin to push/pull changes automatically.

I've been able to clone an existing Grav CMS site to my test cPanel server using the new Git Feature, but I cannot seem to request changes to either the local files or to the remote Git repository are synced - will this type of process be possible once the feature gets further developed? Please pardon my lack of technical knowledge, I am not a developer myself :-)

Thanks very much,

Paul

photo
1

@paul - so it's a bit complicated, but I suspect the easiest answer would be "yes...with caveats"


I admit I'm not a Grav user, but from what I can tell from the it's git-sync plugin documentation, git plays a couple of roles when it comes to Grav:


  • git is used as version control locally while you're working on your website on your computer
  • git is used as the transport to push new changes around between your local computer and whatever git hosting solution you are using (looks like git-sync offers easy integration with github, gitlab, and bitbucket)


Once cPanel's deployment features hit (and we're fighting to get them released in EDGE right now; found a few stability related things we need to address just yesterday), you should be able to "clone a repository" from those three locations down to space on your cpanel account for use.


One caveat...the step in the setup tutorial for git-sync that has you taking a "payload URL" and registering it with github (or gitlab or bitbucket) won't be available in the first release of this feature. After you save your work in Grav...and it's changes are pushed up to github, you'll have to log into cpanel and hit the "deploy now" button. This is something we're hoping to bring to the feature in the next update to it.


The only other caveat I can offer is that I haven't attempted this yet...so YMMV. Definitely report back after you play with it so that the community can see how it went!

photo
2

So I would be remiss in my product manager duties if I didn't update you all on what's going on. Don't worry. Everything's fine!


At the end of last week, my team and I tidied up the remaining loose ends and submitted the deployment features to v68. As a result of the review process, one of our coworkers who had never played with the feature BUT uses git deployment for their current websites attempted to convert their setup to use the new Git Version Control and Deployment features.


While the feature worked just fine, a few of their repositories had git-remotes which either no longer existed or responded very slowly. As a result, a few of the Git Version Control and Deployment interfaces locked up waiting for the 'git fetch' commands to finish cleanly.


Not good. I'm glad we caught this before release! It would have resulted in your customers staring at blank screens for ages waiting for the interface to show up.


We're now making a few tweaks to the network code around retrieving changes from other repositories, as well as related API calls, to handle this situation better. We are updating the offending interfaces not to require so much information about each repository and to avoid network activity entirely wherever possible. There will be some visual changes to the interfaces I posted screenshots of earlier but hopefully just minor tweaks.


So that's what's going on. If you're interested in playing with the feature before it hits EDGE, definitely reach out to me here or shoot me a note at adam.french@cpanel.net. I'll need to send you beta installation instructions!

photo
1

oh, I forgot to mention, check out the new feature icon:


f5cd6321c335d0ce185dcbb3cb5f9ce0

photo
1

Catch-up update for everyone now that our issues with the feature request site have been resolved. Here's where everything stands on the feature.

First off, this feature will debut in cPanel & WHM v72, slated for release very soon (after 70 makes it through our release schedule). The v72 release includes:

  • Hosting repositories on your cpanel account
  • Creating new blank repositories
  • Cloning down repositories from other VCS hosts (like Github or Bitbucket)
  • Selecting the branch which should be considered 'active' and checked-out locally
  • Browsing your repository (using a history browser as well as the file manager)
  • Automatically maintaining your repository's "working tree" when new changes arrive (basically it's "dumb deployment" with more to come in v74)
  • An updated UI for managing your repositories, with a new expandable summary view of each repository.

The feature will be enabled by default (which is our usual procedure), but you can control access to it using a new Feature Manager entry titled "Git Version Control". We've got considerable amounts of documentation prepped for this feature (our docs specialist, Sarah, deserves particular commendations for the heroics she's pulled for this feature) that will get published around the time of v72's release that'll expand (greatly) on my quick description here.

Ok so v74 is also very far along at this point as well. Once v72 is moving through our release schedule, I'll be able to share 'bleeding edge' builds of new deployment focused features!

Right now, v74's deployment focus consists of...

  • Adding "pull now" and "deploy now" buttons to each repository so that you can manually manage updates and deployments of each repository
  • When you push the "deploy now" button, the system will look for a .cpanel.yml file in your repository (which you need to check-in to your project!) and look for deployment tasks (CLI commands) and execute them in the order you specify. If anything goes wrong, we'll halt and tell you about it.
  • The system will now show you the current SHA we deployed last so you can square-up with your expectations.
  • When you create, clone or import a repository into your account, we'll now install a deployment post-receive git-hook that will automatically trigger deployments (as described above) when new changes arrive on the active branch you set. (We'll have documentation of our hook and how you can integrate it into your own use of git-hooks if you need that.)
  • All the typical trappings of .gitignore will be in play and the system won't attempt a deployment if it detects that your repository's working-tree is dirty.

Soon we'll be turning our attention to improving management of SSH Keys. I'll have more conversation about that when plans enter the "ok, let's figure out specifics" phase of planning.

photo
1

Beautiful work folks! :)

Looking forward to wielding this heavily.

photo
1

I've been thinking about setting up some kind of GIT deployment system myself, so this is a great addition.

photo
1

For those interested, v72 (under the name of v71) is now available in the EDGE tier and has the feature available to play with. Now would be a great time to create an edge server and check it out!

photo
1

Nice! Works well on our test server.

photo
1

Hey folks! This feature was added to cPanel & WHM in Version 72, which is available in the CURRENT tier now! Take a look at the release notes, and see all of the highlights for this version on the 72 Release Site.

https://documentation.cpanel.net/display/72Docs/72+Release+Notes#id-72ReleaseNotes-NewGit%E2%84%A2VersionControlfeature

Replies have been locked on this page!