"cap production deploy" can erroneously deploy in development

After running "cap production deploy" I had a mysterious case of "Action Controller Exception: Mysql2::Error Can't connect to MySQL server on '127.0.0.1' (111)". The settings in database.yml were correct. I could verify that the database was, indeed, accessible by using these settings in MySQL Workbench to ssh tunnel to the database on the VPS from my laptop. So what was going on?

It turned out, that the application was deployed in development, not production despite the capistrano command and the settings in deploy/production.rb. And in database.yml I had incorrect settings in the development block - they were supposed to work only on my laptop but not at the VPS.

It was a great relief to find the explanation of that mysterious error. But how could be that "cap production deploy" produced a deployment in a development environment, rather than in production? I don't have 100% certainty as I changed a number of things while trying to find the culprit but the most likely reason was the following line in /etc/nginx/nginx.conf:

server {
    rack_env           development;

Once I changed that setting and re-started nginx the problem disappeared. I am relieved but I still find the whole thing a bit troubling. Somehow I cannot escape the feeling that the rails stack is really messy and fragile - you can specify the same setting (in this case the environment) in a number of different places (in fact in different layers of the stack) so that you get inconsistency and unexpected behaviour. God knows what other settings are now over-ridden but haven't yet manifested themselves in an error. This is a bit frightening.

Anyway - you have to live with it if you want to use rails. For those who are suspecting a similar problem in their deployment:

The best way to tell which environment your application is running in is to look at the logs (/home/deploy_user/my_app/shared/log). In my case the file development.log had a current timestamp while production.log was stale (a few weeks old). That sort of nailed it. However, the situation could be a bit more complicated if you are running different instances of the application at different ports - they could run in different environments. Than you need to look up the logs and find which one has the error message that you are getting via the browser.

Sorting out 403 and "directory index of * is forbidden" in a nginx deployment of a rails application

I deployed a rails 4 application on Digital Ocean Ubuntu instance using Capistrano, Nginx, Passenger by following this recipe.

I set up the server block to listen on a non-standard port:


server {

        listen                        3001;
        server_name            178.62.17.94;
        root                          /home/deploy/appmate/current/public;
        passenger_enabled  on;
        passenger_ruby       /home/deploy/.rvm/gems/ruby-2.1.2/wrappers/ruby;
        rails_env production;

       }


This worked okay.

However, when I tried to set the same application on port 80 I stumbled upon HTTP 403.

The server block for port 80 was set up automatically and I just added/changed the root and the passenger/rails lines. I left the other original content there:

server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        passenger_enabled on;
        passenger_ruby     /home/deploy/.rvm/gems/ruby-2.1.2/wrappers/ruby;
        rails_env production;

        root   /home/deploy/appmate/current/public;
        ##root /usr/share/nginx/html;
        ##index index.html index.htm;

        # Make site accessible from http://localhost/
        server_name localhost;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
               try_files $uri $uri/ =404;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules
        }

}

The error message I found in

/var/log/nginx/error.log

was the same I found earler: "directory index of "/home/deploy/appmate/current/public/" is forbidden". Back than I resolved the issue by changing the privileges of the parent directories. So there was no reason I had to do that again!

With a bit of googling I found the culprit in the location block. By following this recipe. I removed  $uri/ from the try_files command. I started getting 404 instead of 403. When I removed the whole location block all error messages disappeared.

Deploying Rails application on Digital Ocean Ubuntu VPS using Capistrano, Passenger and Nginx

As a part of a New Year's resolution I started learning Ruby on Rails (it was more than 5 years since I last picked up a new programming language!). Ruby and Rails are really fun to learn especially if you start with a toy application. But I found Rails deployment to be a bit more involved than Java deployment. I don't mean using a platform such as Heroku - this really removes all the grunt out of it. I mean deploying Rails while keeping control over the entire stack - i.e. starting from bare metal.

Deploying Rails

There are tons of tutorials on this subject but the clearest I could find is the one written by Chris Oliver. I followed it quite literally. I chose the Ubuntu 12.04 and rvm and I've been using the most recent versions of the other packages as available from the Ubuntu 12.04 repositories:

rnginx/1.6.0
mysql/5.5.38-0ubuntu0.12.04.1
Phusion Passenger version 4.0.48
*** LOCAL GEMS ***
bigdecimal (1.2.4)
bundler (1.6.2)
bundler-unload (1.0.2)
executable-hooks (1.3.2)
gem-wrappers (1.2.4)
io-console (0.4.2)
json (1.8.1)
minitest (4.7.5)
psych (2.0.5)
rake (10.1.0)
rdoc (4.1.0)
rubygems-bundler (1.4.4)
rvm (1.11.3.9)
test-unit (2.1.2.0)

On the dev machine I installed, among others,

capistrano (3.2.1)

Despite following the tutorial to the letter I was getting HTTP 500 after deployment.

Setting the correct path for passenger_ruby

I posted a question on Chris's blog and he kindly suggested that Nginx is the likely culprit. After inspecting

/var/log/nginx/error.log

I found the following error message

You've set the `PassengerRuby` (Apache) or `passenger_ruby` (Nginx) option to '/home/deploy/.rvm/rubies/ruby-2.1.2/bin/ruby'. However, because you are using RVM, this is not allowed: the option must point to an RVM wrapper script, not a raw Ruby binary. This is because RVM is implemented through various environment variables, which are set through the wrapper script. 

To find out the correct value for `PassengerRuby`/`passenger_ruby`, please read:
  https://www.phusionpassenger.com/documentation/Users%20guide%20Apache.html#PassengerRuby
  https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#PassengerRuby

The error message is handily descriptive and checking the above links makes it clear that I had to execute

$ which passenger-config

to find the path to the passenger-config executable. Running it produced:

$ deploy@drop1:~$ /usr/bin/passenger-config --ruby-command

passenger-config was invoked through the following Ruby interpreter:
  Command: /home/deploy/.rvm/gems/ruby-2.1.2/wrappers/ruby

Relaxing the privileges for the parent directories of the app 
Fixing the ruby path did a change - I started getting HTTP404 instead of HTTP500. Checking the log again I found

directory index of {public directory of the app} is forbidden


After a bit of googling it turned out that relaxing the privileges to ALL parent directories is what is needed. I executed


$ sudo chmod g+x,o+x

for each of the parent directories.Than re-starting nginx eliminated the problem. Voila!