Modern development can be a complicated process, between build tools, deployment management, dependency management, version control, etc. There are a lot of options out there, and we've spent a reasonable amount of time evaluating existing options and creating a process that works for us, for now. In the end we reached the conclusion that a server-based development process works for us. This post explains some of our challenges with other popular workflows, how a server-based workflow solves several of these challenges, and gives some insight about how our process works.
Local Development
The most common alternative that I am aware of is local development (duh), which generally involves running some local server(s) installed on a computes. Vargrant is a popular option, giving the developer access to a variety of development environments with a simple command "vagrant up". There are other options for local development, some purpose built solutions, like Aquia's Dev Desktop, primarily aimed at Drupal development, and even some IDE's include some basic internal server capabilities.
For our situation, local development presents some challenges. I'll go more in depth into each of these.
- Local servers often don't look like production servers.
- Spinning up a new project or switching quickly between projects is either slow or uses a lot of disk space.
- It's more difficult to get multiple sets of eyes on a problem.
Local Server Differences
Solutions like Dev Desktop ship with a particular configuration. In Dev Desktop, Apache is the server. Well, we deploy to Nginx. For basic coding, it doesn't really matter, but there are subtle differences that can affect your deployment if they haven't been addressed. I don't know if it's possible to configure Dev Desktop to use Nginx locally, but that is not really what I need to do with my time.
One might argue that Vagrant can solve this problem, because when using Vagrant, you can configure your local environment exactly like your production server. That may be true, but that also means one might need to spend a significant amount of time tweaking your Vagrant configuration to get it as close as possible to production. We don't have a "dev ops" team in-house ( though, Kyle might argue that point ), so spending a lot of time working out Vagrant configurations is time we're not writing code.
There is also the challenge that once you deploy to the Web, you have a host of new considerations that you don't have in local development. Network latency, SSL certs, CORS, etc, can all work differently on a local box than on a live server living on the Web.
Project Switching
I think a lot of development workflows have been defined by large companies where developers might spend the bulk of their time working on a small number of projects at any given time. In such a case, using a local server makes a ton of sense. We, on the other hand, tend to jump back and forth between projects, often multiple times in a single day.
Setting up a Vagrant box takes a fair amount of your local resource, usually there is an operating system image with RAM and disk space allocated to each box you need to run. Even when idle, those Vagrant boxes are soaking up disk space, maybe to the tune of 20 GB each. To get through a week's work, I would need 10 - 15 boxes ready to go at any given time, the problem is, I don't know which 10-15. So to cover my bases, I would want somewhere around 30-40 boxes at the ready.
Essentially, I see three options:
- Keep a box on your local computer for every site that we've ever built, or
- Every time I need to work on a different project, install configured box, or
- Reuse one or two boxes and just wipe the current project and clone the repo for the next project.
The first one doesn't make a lot of sense once you've built more than about 50 sites. Using up more than 100gb of local storage with servers that aren't getting updated regularly doesn't make sense to me.
The second option could make more sense, store a project's Vagrant config in git and spin up a new box to work on a project. But the process is slow, maybe taking longer to spin up the environment than the actual dev time might take.
Option three might be the best option, if you kept a fairly generic Vagrant box ( like, just a LEMP stack and some basic tools ) and just clone the repo of the project that needs attention. But then, you run the risk of making a change that just doesn't work the same way in your production environment. Overall, it's probably the best for making small changes from time to time.
Collaboration
How many times have you worked on a problem and just needed someone else to look at it to tell you that you missed a semicolon? Hate to admit it, happens to me all the time. Well, if you are working locally, that simple collaboration is really hard with a remote team.
What does that process look like? Commit broken code, get the other person to clone your broken code ( maybe needs to spin up a box just to see what is happening ), explain how to recreate your issue, etc. Sounds inefficient to me.
Remote Development
Before I get into how remote development solves some of these problems for us, I want to explain how we got here.
When we started building Websites, we had no corporate experience. All of our tools and practices grew from trial and error, reading blogs, etc. When we needed to work on a site, we would fire up an FTP client, download a file or two, upload, and cross our fingers. Ok, bad. Can't tell you how many times I broke something and started sweating.
Back then, we primarily deployed to shared hosts, and mostly on one provider. Configuration didn't really matter, we knew what we were dealing with. And most of our projects were very small, even when we totally borked something, we could fix it in a few minutes and know one would ever know the site was offline for 30 minutes.
Now, our projects are bigger, the demands and expectations are higher. We need a high degree of confidence that our code is going to work before deploying. We typically deploy to VPS now, and configuration matters.
Our process usually looks something like this.
- Separate Dev, Staging, and Production environments ( all similarly or identically configured VPS ).
- Git to manage code and deployment.
- Work is done over SSH.
We use Digital Ocean for most of our production and almost all of our dev work. With Digital Ocean, we can configure a Production server exactly how we want it, clone the server and spin up an identical Dev environment. We can store images of each environment so we're not starting from scratch each time, and we can install our dev tools and clone that separately.
For us, this totally solves the problem of managing a bunch of different environments. Since it's all on Digital Ocean it uses zero local disk space.
It's fast. I can spin up a full stack from image faster than I can install a new Vagrant box. And if I mothball an old dev server, I can bring it back up in seconds. Everything is there as I left it.
We configure our VPS environments with everyone on our team's ssh key, so getting help is just a matter of giving out an IP address.
There is another benefit for me. I am often times out of the office, and while I do most of my work on a desktop, when in the field I can use my laptop to work on any project, seamlessly.
How To Develop Remotely
If you are coming from a local development process, there are a few things that you'll want to get somewhat familiar with.
Either an FTP client like Filezilla ( remember that? ), or sshfs. I prefer sshfs. I don't really bother setting up sFTP on my dev servers, but if you have ssh enabled ( which you almost have to on Digital Ocean ), you can use sshfs to mount the remote file system to your local machine. Once this is done, you can edit files locally, and when you save, they save directly to the server.
Get familiar with the commandline. I've learned to use the commandline for most everything. I don't know much about Git clients, since I am always connected to a server, I just "cd" into my project and run commands through the terminal.
A lot of tools you might be accustomed to on your Macbook, may not work so well on the server. If you want Sass, you need to install it on the server. Basically, everything lives on the server, so whatever tools you want to use, whether nodejs or xdebug, you need to install it on your dev server. I run gulp, webpack, whatever, right there on a live server.
There can be some latency, depending on your connection. If you have a fast network, it's not much, just be aware.
Limitations
Obviously I like working this way, or I wouldn't have bothered with this post, but I do have to contend with some limitations. They aren't show stoppers, for me, but I don't always get to have my cake and eat it too.
For starters, some IDEs do not like this workflow. PHPStorm in particular expects a local dev environment. I recently download an evaluation version of PHPStorm, I really like the UI, heard such great things, but it demands that I create local projects in order to use it. I just don't see myself with trying to manage over 100 projects locally.
It does cost money, not much, but multiple servers, server images, etc, has a monthly cost associated with it.
Dev / Staging environments need a little extra consideration. You don't want a cron job on a dev box sending emails. For Drupal sites, we use maillog to capture outgoing emails, but everytime we sync the database back to dev, we need to delete the maillog table and re-enable maillog. It's an extra step, but it's quick. You just have to remember to do it.
You don't want search engines and other bots crawling around. We usually throw an htpasswd file on dev just to keep out the cruft.
Overall, there is just a slightly different mindset working on a live server on the Web. It's liberating, not worrying about my local machine. It's just a portal to my dev servers.