Dropfort

• What is Dropfort?
  • Dropfort is a suite of tools to develop and manage Drupal applications.

On the development side, it integrates with GitHub and GitLab to track commits, issues and tags. It then packages releases based on those tags and lets you share those releases with your Drupal sites. Same way you tag and create releases on Drupal.org. The only difference is the released modules are private. Meaning a site that wants to use those modules needs to authenticate to download them.

For example, if you want to download a custom module to your site from dropfort, you can just do a “drush dl mymodule --source=https://app.dropfort.com/fserver/release-history”. Dropfort generates the same XML data for its modules as does Drupal.org for contrib modules. Meaning the Update module works with Dropfort, all your Drush commands work and Drush make works too. It’s all pretty seamless. The only difference with our XML is that it's not publicly available. Your site has to be allowed to see the update feed which is what you configure in the Dropfort web app itself.

The other half of Dropfort are the operations or “ops” tools. You connect your sites to Dropfort using the Dropfort Update module (which is available on Drupal.org) and it will start doing a few things. The most obvious is tracking the update status of your site. Being a Drupal shop, we monitor a few dozen Drupal sites at once and so logging into each site to see what modules need updating and the status of those sites is time consuming. What Dropfort let’s us do is see all those sites in one dashboard. I can login and see the update status and status report data from all the sites in one place. Dropfort then uses this data to generate some graphs about your sites. For example it can tell you how many dev modules you’re using across all your sites, it shows you a list of what sites have security updates and it does some fancy calculations to grade your site’s health as well. Lots of metrics to know what’s going on and how things are changing over time.

The last part, and this is what I’ve been working on mostly these last few months, is an Environment manager. It’s still pretty fresh and there are some rough edges but it does work. You can create a set of environments (dev, testing or production) to store machine configurations to both do your development and run your Drupal applications. You can say “I want a server running apache, with MySQL and the Commerce Kickstart distro” on it. Then you can either download a Vagrant file which will provision a vm, or you can download a docker container which will do the same. Or you can run a bash script on an existing machine to link the server to Dropfort and have that configuration deployed. It’s pretty neat stuff. Basically any server anywhere can be turned into a managed Drupal cloud.

Like I said, there’s a whole suite of stuff in here.

• Why did you create Dropfort? Where did it start?

It all really came from using Feature Server for Drupal 6. When we incorporated Coldfront in 2011 and really turned it into a full time job, we wanted a way to distribute code to all our clients (even though at the time I think we only had one). We setup FServer to deploy code to your client sites. But manually creating the releases and the pages and stuff was kind of a pain. So we came up with a special commit syntax we could add to Subversion (this was before Git was a big thing). So could make our svn commit and in the message we’d write [release:full] and some post-commit scripts would run on the svn server. They’d look at the commit message and then take the code, create a tgz file, create a tag, commit the tag and then upload the tgz file to FServer using some REST web service endpoints we create on a Drupal 6 site with Services. That would create the release page, the release notes, add the files and generate the update XML. It was pretty well a mini Drupal.org but with Subversion (instead of CVS which D.O was still using at the time). It actually worked really well. So well in fact that the University of Ottawa asked if we could install a version for them to manage their Drupal stuff (which they’re actually still using today until their git migration is done). That’s when the lightbulb went off I guess. We had built this stuff for us but as it turns out, other people want to deploy custom modules too! Who’da thunk it?

And that’s when the idea for a more “web app” version of our SVN workflow came to be. At the time I thought “Yeah we can totally rewrite this in no time, I give is 6 months and we’ll have a web app ready to go”. That was 3 years ago I think? It took a bit longer than expected. Mostly I kept adding features… Yay scope creep. But now we’ve got a pretty awesome suite of tools and we’re focusing on the polish now. I’ve been told I’m not allowed to make feature requests for at least a month. We’ll see how that goes.

• Can this be used to compete with Drupal.org? Meaning can people share public releases here instead of on d.o?

Nope. Public modules should be on Drupal.org. No module can be downloaded from Dropfort without authenticating first. We don’t want to supersede d.o in any way. We actually looked into writing a feature to automatically move a project from bieng private to a public one on d.o but since Drupal.org doesnt’ have an API we couldn’t do that. But yeah, this is for privately distributed modules only.

• Does the monitoring dashboard check both the private projects as well as public ones on d.o?

  • Yes

• What did you build it with? Is Dropfort open-source?

The original SVN workflow using Subversion, CLI PHP, Drupal 6, 2 custom modules and some REST / Services stuff.

Dropfort uses Git, Drupal 7, Services, about 20 or so custom modules, Puppet and our Drupal 7 port of FServer. So Dropfort is a Drupal application itself. We actually use Dropfort to manage Dropfort. Meaning we track it’s own updates and status using itself, and it packages releases for itself. A little inceptiony but it works.

Most of the parts which make up Dropfort are open. Some of the custom modules aren’t openly available. But that’s mostly because we don’t have the bandwidth to help and support the distro on D.o. Especially the stuff involving setting up a Puppet master. We’d spend more of our time debugging that than actually making things work. Doesn’t mean we won’t share everything eventually, just not right now.

• What’s the plan for Dropfort? Is it a paid service or is it free?

Right now it’s free to use. The main reason for that is we haven’t written the commerce component yet so we can’t actually charge for it so… yeah. But we’re looking at different ways of monetizing. It’s tricky cause we want people to use it but at the same time we don’t really know what people will use. There’s such a variety of things in there it’s tough to decide what should be charged for. For example Github is pretty straightforward in their pay structure. If the code is open, your repo is free. If your code is closed, you pay for the repo. For us, I think it’s a question of usage. We’re leaning towards all tools are free to use with any account, it’s just a question of how much storage or how many sites you’re using. But regardless, anyone who uses it now is free to use it as much as they want. And we’ll have some special plans for early adopters as a thanks for their feedback. More than likely a bunch of free stuff.

• How does this compare to other tools like Platform.sh, Pantheon, Acquia Dev Cloud?

The big difference is that they’re primarily a hosting platform. Dropfort is a management platform. You can connect a Pantheon site or an Acquia Dev cloud site to Dropfort and use most of the features no problem. You’d probably skip the release packaging stuff and environment management (for now) but the stats tracking and collaboration tools would work just fine. Dropfort doesn’t care where or how you run your Drupal site. As long as it can reach the internet, you can use Dropfort.

But you can use Dropfort with GitHub or GitLab or neither. You can use Vagrant or Docker or both. We do our best to integrate with anything which might make building Drupal application easier. It’s all about choice.

As for the hosting side of things, we give you tools to deploy your own server or cloud of servers. Meaning you can run an optimized network of Drupal web servers on whatever provider you want. It’s a philosophical difference. We let you host your code and sites wherever you want whereas with others you live on their machines. Which can have a lot of advantages and for the majority of folks out there, that’s fine with them.

But for us we’ve found it difficult for some enterprises here in Canada to get hosting on services in the US which are bound by the Patriot Act. We have FIPA, the Freedom of Information and Privacy Act which states that we can’t share information about users unless the user has explicitly allowed that agency access. The Patriot Act is pretty much the exact opposite of that. So we figured we’d bring most if not all of the advantages of a cloud solution (the optimized configuration, automated deployments / scaling, generated environments) to anyone’s infrastructure. You just supply the hardware, Dropfort does the rest.

I see it as just another option in how you can host your Drupal site. You can choose how much or how little you want to be involved in managing the hosting environment. Whichever way works best for you is the one you should go with.

• How does this handle dev/staging/live scenarios?
• How about local?

Use Cases

• Let’s talk about the current use cases for Dropfort.
  • Managing several sites in one place
  • Create custom, shareable development environments
  • Create releases of projects destined for more than one application
• Why would you use Dropfort instead of just Git to manage deployments?
  • We use Drush Make for just about everything. We control our releases using Drush make. We apply patches with Drush make. We really like Drush make. And we really don’t like merge conflicts. The number of times I’ve come into a project where the entirety of Drupal core and all the contrib modules are in a single repo with a team of people trying to all work on it at once has taught me that’s not the way to work. Treat your projects like d.o does, as self contained sets of functionality. Use make files to build your application and drush to manage updates. This is how Drupal is designed to work. Drupal is a collection of modules. When you all of a sudden lump it all together into a single repo you’re breaking how Drupal was meant to be managed.
• Can you explain a bit more about how Drush Make works?
• You just mentioned automated updates.
• What’s in the near, and far future for Dropfort?

Training

While at DrupalCon recently, someone mentioned my recent podcasts about training and hiring Drupal talent, and mentioned that Cameron had been a part of a somewhat unique program to train new talent through a two month training program, with the intent to hire some of them. I was intrigued (and based on the great response of recent episodes on similar topics) I decided to get ahold of him, so here we are.

• What company were you working for when you did this? And what was your role there?
  • ProPeople (Now FFW)
• How did the idea come about?
  • We were inspired by a similar program they did in the Moldova office.
• Who were you targeting?
  • Current college students, and recent grads
• What did you hope to accomplish?
  • There were two goals. One: We wanted to be able to hire people in the Boise area. Two: To get our name out there, to university students, as well as the broader tech community in the Boise area so that when people were looking for work in the future, they’d look at ProPeople.
  • We weren’t looking for Drupal experts, we wanted them to come out with the minimum ability necessary to become a competent member of a team within ProPeople
• How many people did you have participating at the beginning?
  • 70-80 applied
  • 12 were accepted (2 dropped out)
• How many did you end up hiring?
  • 1 full time junior developer
  • 1 contractor
• How did you promote it?
  • Craigslist, Boise State University CS department, Random encounter at the coffee shop
• What did they participants do?
  • Clone Netflix and Basecamp
  • Wanted them to work as a team
  • Learn Features (getting config into code - turns out it’s more difficult than we thought)
  • Issues lead to Skype calls to talk about how something might be done better
• What kind of interaction did you have with the participants?
  • Daily checking call with each member
  • Weekly check-in call with each team

Should others do this?

• Was there enough of an ROI that you think Drupal shops should be doing more of this?
  • Fairly expensive in number of dollars spent (About $4,000 - from that came one junior developer, and one consultant, who have been there ever since). Compare that to a recruitment agency, where you can pay well over $4,000 each.
  • Great to see who is actually going to go out and learn what they need to know in order to accomplish their project vs. someone you might hire, and then later find out that they’re not a good fit.
• Do you have any suggestions for companies that are interested in looking into running a program like this?
  • If you’re going to do this, you need to have someone committed to running the program. It’s not a no-commitment project. Having more than one would be even better.
  • One reason we chose to go the training route, is because ProPeople didn’t have a standard classroom style training program. I’d recommend having more structured content: What knowledge we want them to come out of the program with. The hands on stuff is great, but throwing someone into the deep end is a bit hit or miss. (Use Blink Reactions stuff)

Writing a Book for D8

• What’s it like writing a book for a piece of software that isn’t even officially released yet?
• How long did the writing process take?
  • Packt publishing sent us a proposal to write this book in December of 2013. We got started quickly, sending them an outline of the chapters and an estimated page count in the same month. The original estimated page count was 150, it turned out to be around 120. We received a pretty strict time line, having to finish a chapter every two weeks, starting in December of 2013.
  • We managed to finish most chapters in two weeks, but some of the longer ones took a little longer since we also started one of our biggest projects we had had until then, also in January. That was pretty tough because that project took up way more than a regular full time job, so we ended up having to write all of the chapters late at night and on the weekends. In May, all of our chapters then went to the editors and we didn’t hear back from the publisher for a really long time.
  • We also told them that we will have to rewrite a lot of the chapters since there was so much work in progress with the Configuration Management Initiative and they were changing a lot about how it worked, like going from the file based default to the database default. I think it was in January of 2015 when chapters came back with some feedback and we started rewriting every chapter, which was pretty painful at the time. We were able to update some of the documentation at drupal.org with the changes we found. It felt good to contribution at least a small part, when with our project and the book we had no time left to contribute code to Drupal 8 like we usually do.
  • We spent around 40 days on the book between the two of us.
  • In December, Packt asked the first publisher to review the book. We had recommended them one of our team members at undpaul, Tom, who has a similar amount of Drupal knowledge as Stefan. We really wanted to have someone from CMI to review the book, like Greg Dunlap. They had turned down reviewing the book after the first chapters were written, because too much would still change. Then after the changes went in we kept recommending Greg but I never heard anything back, maybe he was busy or they didn’t bother to ask. At the beginning of this year they told us the book was planned to be published by March. We recommended waiting because we didn’t expect a release candidate before the European Drupalcon and we would have rather had someone like Greg take the time to review, but Packt had another opinion :) Since most of CMI changes were finished, we didn’t feel too uncomfortable about the time of publishing, and it was also kind of nice to finally be done with this thing :) So it took a little over a year from start to finish. It was published on March 24th.
• Do you expect to need to rewrite anything between now and when 8.0 is released?

The Book: Drupal 8 Configuration Management

• What do you cover in the book?
  • We start of with a basic introduction to what Configuration Management in Drupal means, because it is a thing in Software development in general, that doesn’t usually refer to what it is in Drupal, where it basically just means that configuration is saved in files which makes deployment easier. In the first chapters, we make sure the reader understands what Configuration Management means and why version control is so important. We mention some best practices and then show how to use it for non-coders as well, since there’s a nice backend non-technical folks can use, even if you don’t use version control (which of course we don’t recommend). We also have a part that describes how managing configuration works in Drupal 7 (Features!) and then dive into code examples, explaining schema files, showing how to add configuration to a custom module, how to upgrade Drupal 7 variables to the new system and cover configuration management for multilingual sites.
• Who is the target audience of the book?
• Why did you decide to write about Configuration Management?
  • We have used Features to deploy configuration changes for a very long time, I don’t recall not using it since we started the company 5 years ago. We have talked about it at several DrupalCamps and Drupal User Groups and always tried to convince everyone to use it. We were really excited about the Configuration Management Initiative and thought it was a very good fit for us.
• Before we started recording, you mentioned that there is a companion website to the book. Can you talk about what content we’ll find there, and what purpose that serves?
• Are you building any sites in D8 at Undpaul?

d8upgrade.org

• What is d8upgrade.org?
• When did you put the site together?
  • Today
• How were you offering the service before the site was up?
  • Talk about MVP
• Why did you guys create this service?
  • Build a more direct communication channel for module maintainers - Mailing list (module devs talk to people using module
  • Initiative to get out the catch 22 - modules aren’t ready, so you can’t upgrade, you don’t upgrade so the modules don’t get money to be upgraded.
• Does it cost anything?
  • Nope - it’s free
• Are you planning to monitize the site in any way?
  • We expect a lot of site owners will be looking for a way to upgrade their website, if they don’t yet have an agency, we can help them find a good agency. If this evolves how we imagine, this could fuel the sales pipeline for our consultancy and a few selected partners. We are not going to go hardcore salesy on this, but put in links for module work and 2for1 discounts and hopefully some of those will convert into customers for us or our partners.
• How does it work? What do I have to do to track a site?
  • As simple as possible, conscious choice to not make it a module - just copy paste your module page
• How many people have signed up so far?
  • We’ve got a total of 73 sites now, some from the same people - after Drupalcon LA we expect that to become a few hundred.
• What do you have planned for the future of the site?
  • The community misses a permission marketing channel for module developers. We would like to let site owners sign up for important information about the modules they use through a push communication channel. There is a lot of talk in the community that we are not getting any support from site owners for the development of D8, but we’ve made it almost impossible for them to stay up to date. There is no way they are going to start reading the issue queues of modules on regular basis.
• I took a look at the site and saw something called 2for1 - Give to Get. What’s that?
  • This is part of the strategy we want to use to monetise the platform: we offer our customers a 2 dollar discount on their future D8 project for every dollar they invest in helping Drupal 8 get out of the door. This gets us more business and brings outside funding into the community. For too long we’ve been giving money from one developer’s pocket to the next developer’s pocket. We hope we can change that this way. We are inviting other consultancies to join us in this campaign, if you want to join, you can become a partner in the campaign.

Special Thanks!

Thank you FourKitchens for the DrupalCon Ticket and Kalamuna for helping offset my travel expenses! You guys (and gals) are all awesome!