Translated from 日本語 • View original The Ruby On Rails app that was running normally until yesterday, until mid-morning, and just a while ago, suddenly became an error called "Routing Error : Uninitialized constant XXXXX Controller" no matter what I did, and I fell into a mysterious symptom that I couldn't say yes or no. I had a numb experience that I was about to cry for about a month until the cause was investigated, so here are the tips for solving the problem (originally, a story of struggle?) I thought I would write about it.
Searching for the Culprit
First of all, this Rails error screen is already close to trauma...
First of all, when you get caught in this error, Rails will directly specify the URL and switch the controller that is called, or nothing else, it will just keep spitting out the same "Routing Error" with the error that only the controller name is different. I only say that as if I knew the error message anymore. However, the Rails route configuration file 'config/routes.rb' is fine (because it was running until just now and I haven't changed the route, so it's natural). Even if you look at the Rails log, you can only get the same amount of information as the error screen, and if you look at the error log on the server side, it only says that there is a 404 error every time, and there is no content in the route (URL) (the Rails controller is not working). Even if you go back the error log processing stack, there is nothing particularly strange... What **the hell is this!? ** My heart is already broken, I've broken a lot, I hate Rails, but I can't throw it away, it's my job, so I can't move forward unless I find out the cause (T_T)
If you search StackOverflow or other support forums, you will find several questions from people who are suffering from the same symptoms... But I didn't have an answer. **Seriously~ No one has solved it~!? ** Someone please help me~... That's the state.
And the even more mysterious part of this symptom is that it can be resolved by rebuilding the Rails environment (reinstalling everything from scratch) (or rather, at that time there was only one way to recover it). Therefore, at first, I did a lot of research to see if the problem was the configuration and cache on the server side (such as Passenger that relayed Apache and Rails), but everything was off the mark (when the bug fix was wrong, it would dent if the intuition of "Isn't it here?" I think I'm old too).
Now, in my first Rails rebuild, I tried changing the Ruby version and the Rails version. At first, it was Ruby 1.9.3 + Rails 4.1.0, but I changed it to Ruby 2.0.0 + Rails 4.2.0. There is quite a bit of compatibility with the version, so I tried to attack from that side. There was no problem immediately after the rebuild, but the same symptoms occurred again at noon the next day. If the Rails SQLite database is broken, it is useless to try switching back to the DB that was backed up immediately after the rebuild. Initialize the DB again and put the data back in. … Hmm? seeds.rb doesn't work (in Rails, when registering the initial data in the database, you run the 'rake db:seed' command using 'db/seeds.rb', which gives an error). Is this a broken Rake binary?
──Well, this was the place where I felt like I was going to cry the most... I feel like I have nothing but a sense of futility and despair. I don't even have a flash.
I continued to do the same research for nearly a month, but finally the light came into it. Well, moss's thoughts... I wonder if it's a guy~. I found such a site!
Ruby suddenly stopped working "/usr/bin/ruby: No such file or directory" prelink? In the CentOS environment, it is installed as a mandatory and runs on its own with cron.daily (a cron job that runs once a day). As a bad thing, there are cases where Ruby binary files are destroyed. It's quite suspicious. This time, I searched for prelink+ruby and finally came up... It feels like I've reached the golden vein.
It's too suspicious prelink .
cron.daily is run randomly every morning between 3:00 ~ 4:00. By the way, the time zone of the Rails environment is UTC, so if you +9:00, the prelink will run between 12:00~13:00 in Japan. I guess the only time the problem occurred was at lunchtime... That means...
**prelink, is it you~? **
That's why I actually verified it. In the middle of writing to the SQLite database in the rebuilt Rails environment (this timing is quite important). When the main application was processing changes to the binary, the prelink did not overlap and the symptoms did not occur) I tried to run the prelink manually...
**After all, you were the culprit, prelink! **
Congratulations on "Routing Error"! No, I don't want to be happy that the Rails app is broken, but I want to be very happy ♪ for everyone involved, including me.
So, since I took the Rails checksum before running prelink, when I compared it to the checksum after running, it seems that the binaries of Gemfile.lock and db/development.sqlite3 are broken. Even if you return only the files in the SQLite database, you cannot recover them.
In particular, if Gemfile.lock is broken, we don't know how Rails will behave because it is a binary that centrally manages gem dependencies, versions, and destinations (file paths). This time, it was uniformly "Routing Error", but depending on the environment, Rails may end up saying only another error.
Now, this is the story of the struggle to find out the cause. From here on, it will be TIPS for fixing bugs.
Disable Prelink
First, what is a prelink? ──It's true that he's a suspect so far, but maybe he's not the one who does anything that bad. Upon investigation, prelink is a middleware that analyzes the dynamic link processing generated by the binary file in advance and embeds it in the binary body before executing the binary file in order to optimize the performance of the application. Hmm, even though it's for performance improvement, it's probably okay to rewrite it binary on its own. "Sometimes I fail to rewrite and break the binary, but Tehepero ♥ " or something like that**! ** ── When I looked it up, it seems to be an app that is harmful and has no benefits. Why does CentOS install such a bomb by default?
Oh, and CentOS has finally noticed. To his danger (laughs)
Well, that's why I want such a dangerous guy to leave quickly. Now, how to disable prelinks. There are several of them, as follows.
- Uninstall prelink from cron.daily
- Specify as a blacklist of files and paths that you do not want to binary rewrite to /etc/prelink.conf
- Set PRELINKING=no in /etc/sysconfig/prelink
First of all, I don't want to mess around with cron.daily because it also contains settings for other middleware (because I don't want it if it affects other things). Next, number 2, it is defamatory and uncomfortable to call apps that are running normally without a blacklist even though prelink itself is not on the blacklist. … So, this is a rejection (laughs) The last setting is the most appropriate. CentOS 7 also disables it in this way, and if you want to change it again, you just revert it back to yes (well, that won't come anymore...).
So, let's disable prelink in method 3. After turning it off, restart the server just in case, and you can sleep with your pillow high.
Conclusion
Thank you for reading to the end, and for sticking with the long and miscellaneous articles. The summary of this long article can be summarized in one word as follows.
If your Ruby on Rails environment suddenly stops, it's likely that prelink is corrupting your Rails binary files
However, even in an environment that does not use Rails, it is basically safer to disable prelink in a CentOS environment.