Mikamai developers blog
The last time I worked on OpsWorks I needed to implement a recipe to handle the setup and configuration of cron jobs.
Actually I had already written some code to handle this functionality in the past. However, the recipe I ended up with wasn’t particularly flexible.
4th part of the serie titled “Wordpress and OpsWorks“ (here you can find the 1st, 2nd and 3rd part).
Despite the title, this time I’m not going to talk about an aspect of OpsWorks specifically related to WordPress, since I’ll be presenting a brute force way to modify the default attributes merging behavior of OpsWorks.
Continue reading “WordPress and OpsWorks, “Pride and Prejudice” (Part 4)”
Last time I described a general approach I used to decrease the amount of duplication across the recipes I developed in different cookbooks. The goals of these latter is to handle the deploy of specific PHP application via OpsWorks.
The PHP application I’m talking about are based in particular on WordPress (WP) and PrestaShop (PS).
Last time I also cited a “mysterious” recipe upon which I relied for setting the default attributes to specialize its behavior.
Continue reading “Clean the kitchen, aka refactor cookbooks and recipes (Part 2)”
I was planning to continue my series of posts entitled “Wordpress and OpsWorks, Pride and Prejudice”, but today I want to talk about something a little bit less specific. Anyway, for the curious readers, here’s the link with the last post of the aforementioned series.
The subject of this post doesn’t relate in particular to Opsworks or WordPress but it is still connected to these topics as it discusses “something” I found myself in need while working on a new cookbook containing some new recipes.
This something is the wish to refactor and it comes from the fact that the recipes of the new cookbook have a lot in common with the ones of the WordPress one. Just for the record, the purpose of the new recipes is to handle the deploy of another PHP framework (if WordPress can be considered a framework) called PrestaShop.
Anyway, pushed by the will to improve and dry out the structure of my cookbooks and recipes I started to delve more and more into the Chef docs and to study the other open source cookbooks.
Among all the sources I stumbled upon let me report this one as I consider it a great summary of the best practices regarding cookbooks and recipes development.
One of this best practices is to identify the type of the cookbooks you’re writing. This indeed will let you structure them in a more appropriate way and avoid, in many cases, a lot of duplication.
In my case for example, the WordPress and PrestaShop cookbooks can be considered as wrapper cookbooks for a more general one.
This last, i.e. the mysterious general cookbook, can be considered instead like an application cookbook. You can roughly think it like a sort of parent cookbook whose aim is to reduce the possible duplication between his children.
An example of these duplication is the recipe needed to handle the import of a database dump.
In this case the recipe is pretty general as it doesn’t rely on any specific data or behavior concerning the type of the application beeing deployed.
Here it is:
node['deploy'].each do |application, deploy|
attributes = deploy['set_db']
next if deploy['application_type'] != 'php' or deploy['application'] != application
import_db_dump do
db deploy['database']
db_dump attributes['db_dump']
end
end
end
The check I’m doing here (i.e. if deploy['application_type'] != 'php' or or deploy['application'] != application) is something I need in order to ensure that the recipe gets executed only for the current deployed application and not for the others allocated on the same OpsWorks stack. Remember, we’re using OpsWorks to handle all the operations and their related configurations.
The import_db_dump is just a definition I’ve written to add a little bit of magic and hide the details related to the import. Maybe one day I’ll show you those details… ;P
Now the important thing to remember about the recipe is that it resides inside a (general) application cookbook. I called it phpapps. So whenever I need to use the recipe inside a PHP layer defined inside an OpsWorks stack I must remember to specify the correct “path” to the recipe, in this case, phpapps::set_db.
This is really nothing special. The magic here (except from the one related to the import_db_dump definition 😉 ) is that OpsWorks will handle for us all the information we need in our recipe by automatically merge them not only with all the attributes specified inside the JSONs (i.e. the stack and the customs one) but also with the defaults attributes written inside the recipe cookbook.
In this way you can rely on these attributes in order to specialize the recipes.
As stated before this isn’t particularly the case as the recipe doesn’t use any application specific information.
Anyway in another general recipe I’ve written I rely on this feature and it seems to work pretty well.
Next time I’ll maybe describe the particularities of this misterious recipe as it need some tweaks to function properly.
Moreover it strongly relies on the default attributes of the cookbooks. These elements are indeed from my point of view the real cornerstone to properly structure the cookbooks as wrapper and application specific ones.
To the next time! 😉
Cheers!
Updated on 2015-02-19, changed the stack JSON with new interpolation syntax
When I was younger I always wanted to be a cool kid. Growing up as computer enthusiast that meant using the latest window manager, and the most cutting edge versions of all software. Who hasn’t gone through a recompile-kernel-every-day-several-times-a-day phase in their lives?
Luckily I was able to survive that phase (except a 120GB Raid0 setup lost, but that’s another story), and maturity gave me the usual old guy-esque fear of change.
That’s the main reason I waited a bit before approaching Docker. All the cool kids where raving about it, and I am a cool kid no more, just a calm, collected, young-ish man.
You might have guessed it from the title of this article, but Docker blew my mind. It solves in a simple way lots of common development and deployment problems.
After learning to rebuild my application to work from a Docker container, more about it in the next weeks, my focus went to deploying Docker. We run our application on the AWS cloud, and our Amazon service of choice for that is Opsworks. Easy to setup, easy to maintain, easy to scale.
Unfortunately Opsworks does not support Docker, and ECS, Amazon’s own container running solution, is not production ready. I needed to migrate my complicated application to Docker, and I wanted to do it now. The only solution was convincing Opsworks to support Docker.
Googling around I found a tutorial and some github repos, but none were up to the task for me.
My application needs several containers to work, and I was aiming for something that allowed me to orchestrate the following:
In addition to that I wanted everything to scale horizontally based on load. It seemed a daunting task at first, but as I started writing my chef recipes everything started to make sense. My inspiration was fig. I like the way fig allows you to specify relations between containers, and I wanted to do something like that on Opsworks.
The result of my work can be found here. At the time of this writing the README.md file is still blank, but I promise some documentation should appear there sooner or later… for now use this article as a reference 🙂
The first thing we’ll do to deploy our dockerized app is login on AWS, access Opsworks and click on create a stack. Fill in the form like this:
Once you have your stack you have to add a layer. Click on add a layer and fill in the form:
After you create the layer go Edit its configuration and click the EBS volumes tab. We’ll need to add ad 120GB volume for each instance we add to the stack. Why you ask? Unfortunately on Amazon Linux/EC2 docker will use devicemapper to manage your containers, and devicemapper creates a file that will grow with normal use to up to 100GB. The extra 20GB are used for your images. You can go with less than that, or even no EBS volume, but know that sooner or later you’ll hit that limit.
After that let’s edit our layer to add our custom recipes:
What do our custom recipes do?
Now let’s add an application. From the Opsworks menu on the left click Apps and add a new one.
Now add just one Env variable to the app:
Everything else will be configured via the (enormous) Stack JSON. Go to your stack settings and edit them. You will need to compile a stack json for your containers. Here’s an example one:
{
"logrotate": {
"forwarder": "logspout0"
},
"deploy": {
"amazing application": {
"data_volumes": [
{
"socks": {
"volumes": ["/var/run", "/var/lib/haproxy/socks"]
},
"static": {
"volumes": ["/var/static"]
}
}
],
"containers": [
{
"app": {
"deploy": "auto",
"image": "quay.io/mikamai/awesome-app",
"database": true,
"containers": 2,
"volumes_from": ["socks", "static"],
"entrypoint": "/app/bin/entrypoint.sh",
"command": "bundle exec unicorn -c config/unicorn.rb -l /var/lib/haproxy/socks/%{app_name}.sock",
"migration": "bundle exec rake db:migrate",
"startup_time": 60,
"env": {
"RANDOM_VAR": "foo"
},
"notifications": {
"rollbar" : {
"access_token": "",
"env_var": "RACK_ENV",
"rev_var": "GIT_REVISION"
}
}
}
},
{
"cron": {
"deploy": "cron",
"image": "quay.io/mikamai/awesome-app",
"database": true,
"containers": 1,
"env_from": "app",
"command": "bundle exec rake cron:run",
"cron": {"minute": 59}
}
},
{
"sidekiq": {
"deploy": "auto",
"image": "quay.io/mikamai/awesome-app",
"database": true,
"containers": 1,
"env_from": "app",
"command": "bundle exec sidekiq"
}
},
{
"haproxy": {
"deploy": "manual",
"hostname": "opsworks",
"image": "quay.io/mikamai/configured-haproxy",
"volumes_from": ["socks"],
"env": {
"REMOTE_SYSLOG": "logs.papertrailapp.com:1234"
}
}
},
{
"nginx": {
"deploy": "manual",
"image": "quay.io/mikamai/configured-nginx",
"ports": [
"80:80",
"443:443"
],
"volumes_from": [
"socks",
"static"
]
}
},
{
"logspout": {
"deploy": "manual",
"hostname": "%{opsworks}",
"image": "progrium/logspout",
"volumes": [
"/var/run/docker.sock:/tmp/docker.sock"
],
"command": "syslog://logs.papertrailapp.com:1234"
}
}
]
}
},
"docker": {
"registries": {
"quay.io": {
"password": "",
"username": ""
}
}
}
}WOW! That’s a lot to digest, isn’t it? In the next article we’ll go through the Stack JSON and see what each of the keys mean and what they enable you to do.
Thanks for reading through this, see you soon!
I know, this third part of the series regarding WordPress and OpsWorks comes a little bit late.
Between this article and the previous one of the series I wrote a lot of other gems spanning many different topics.
But now I’m here to continue right from where I stopped 😉
Before we begin, here’s the first and the second article.
Now, just to wrap up, last time I tried to explain a quirk about the compilation and the convergence phase of a chef node.
To present it I relied on a recipe I wrote to “set up” the database during the deploy of a WP (WordPress) application.
Everyone accustomed with WP development knows that, due the characteristics of WP itself, it is very often difficult to keep everything updated and working as expected.
Many functionalities depend not only on the code but also on the state of the database where most of the configurations are stored. The problem with WP is that there isn’t a recognized and out-of-the-box standard that can be followed to handle the aformentioned database state.
No Rails-like migrations, sorry.
A simple (but bad) way to solve this problem is to push, together with the code, a dump of the database directly inside the repository of the project. In this way it’s state can also be tracked and most of all restored (imported) whenever needed.
Obviously this solution doesn’t fit the case in which there are sensitive information that can’t be simply shared between all the project contributors.
Anyway, assuming this is not the case, if you plan to deploy a WP project through OpsWorks you may end up with the need to automatically import a dump just during a deploy.
This is exactly the purpose of the recipe taken as an example in the last article of this series.
But hey, as L.T. says, “Talk is cheap. Show me the code”. So, here it is:
script 'load_database' do
only_if { File.file?(db_path) and Chef::Log.info('Load PrestaShop database...') }
interpreter 'bash'
user 'root'
code <<-MIKA
mysql -h #{deploy['database']['host']} -u #{deploy['database']['username']} #{deploy['database']['password'].blank? ? '' : "-p#{deploy['database']['password']}"} #{deploy['database']['database']} < #{db_path};
MIKA
endWhat I do here is simply to rely on the script resource to invoke the mysql Command Line Interface (CLI) and tell it to import the dump located at the path stored inside the db_path variable inside the proper database.
This is done by relying on the bash interpreter and by using all the info that OpsWorks gives us through a JSON object (one per deployed application) embedded inside the deploy attribute of the JSON associated with the deploy event.
{
"deploy" : {
"app1" : {
"database" : {
"database" : "...",
"host" : "...",
"username" : "...",
"password" : "...",
}
},
"app2" : {
...
},
...
}
}The overmentioned info are picked up by OpsWorks (and underline, Chef) right from the app configuration that can be accessed from the OpsWorks dashboard.
The complete list of the deploy attributes can be found right here.
As a side note and as already stated in the previous article, the import of the database gets triggered only if the the dump is actually present and if a proper flag (i.e. “import_database”) is present inside the overmentioned JSON.
Next time I will talk about…well…I don’t know! There are really many things to talk about OpsWorks (and WP) so just stay tuned! 😉
Cheers!
Here we are for Part 2 🙂
In my last post about WP and OpsWorks I tried to explain the general setup for the recipes that I’ve developed to automate the deploy of WP applications through OpsWorks. If you missed it here’s the link.
Today I’m gonna talk about a particular problem that I encountered while writing the recipe that currently handles the database importation.
Continue reading “WordPress and OpsWorks, “Pride and Prejudice” (Part 2)”
OpsWorks is an impressive piece of software, but sometimes it lacks the comfort zone we developers love so much.
One feature that I really miss, is the ability to configure my application using ENV variables.
I’m not aware of any easy way (ie: Heroku like) to create environment variables in OpsWorks that the application can consume.
Fortunately OpsWorks is based on Chef and can be customized in any way you want.
Be warned, it’s not always an easy path, basic Chef knowledge is required, the interface is quite convoluted, but in the end it gets the job done.
So, we were saying environment!
We know environment is not supported in OpsWorks, so what we really need is to simulate it in some way.
A common solution among Rails developers, is the Dotenv gem which load the file .env in the root of you app and create the correspondent keys in the ENV object.
I will assume you already have created a Stack in OpsWorks with a Rails application layer.
Continue reading “Use the ENV Luke! aka: simulate the ENV in OpsWorks using Chef and Dotenv”
is a property of
