Dec 17, 2015
Till Toenshoff
D2iQ
In this post, we will show how to simplify your development workflows by adding RTags to your code for source code navigation. RTags is an Abstract Syntax Tree (AST)-based symbol navigation tool, that you can use directly from within your favorite text editor. RTags enables C++ code completion as well as exact symbol navigation throughout your project. You can use this tool to quickly find references of any symbol within the Apache Mesos source code.
Most text editors offer only grep-based navigation which can be very inaccurate. Many IDE's include robust symbol-based navigation within a project. With RTags you can close this gap, by adding AST features to your text editor.
Prerequisites
Create a compilation database
To set up RTags, you must first create a compilation database out of the source tree. Because Apache Mesos relies on autotools as its build system, you must create the database manually. Future versions of Apache Mesos will support CMake as a build system, which will remove the requirement to create a compilation database.
1. Install Bear
Build EAR (Bear) is a compilation database generator. It attaches itself to the build process and captures the compilation commands. Bear writes a JSON blob containing the exact compilation command used for achieving build results off of each and every source file.
Install via homebrew
$ brew install bear
2. Do a clean sweep of your build directory.
Note that ccache, distcc, and others must be disabled for this step to succeed. You can re-enable these things after the compilation database is created.
$ rm -rf build
3. Prepare a complete, new build.
Note: Bootstrapping is needed for builds out of the current master.
$ ./bootstrap $ mkdir build $ cd build $ ../configure
4. Attach bear to the process for a complete build including the tests while disabling test runs (for speeding up the process).
$ GTEST_FILTER="" bear make check -j8
The result should be a more than 500kb sized file named compile_commands.json, residing in your build folder.
~/Development/mesos/build (master ✓ ) $ ls -la compile_commands.json -rw-r--r-- 1 till staff 807653 Oct 10 16:38 compile_commands.json
If that file was suspiciously small, chances are that you are running OS X El Capitan (10.11) which disables code injection and runtime attachments from non Apple signed executables. You will have to boot into recovery mode (Command + R) so that you can re-allow at least the debugging specific runtime modifications which prevent bear from doing its job properly.
$ csrutil enable --without debug
We do not want to disable the system integrity protection entirely as it is a good thing to have.
Setup RTags
Use the most recent RTags master because important things have been added since the 2.0 release that specifically address the OS X limitations and features.
1. Install RTags via homebrew
$ brew install --HEAD rtags
You can also build manually from the RTags sources.
2. Start the RTags server
For a first test, start the RTags server manually and leave it running until you reach the end of this post.
$ rdm -v <br />listening /Users/till/.rdm Running with 8 jobs, using args: -ferror-limit=0 -Wall -fspell-checking -Wno-unknown-warning-optionIncludepaths: -isystem /usr/local/Cellar/llvm/3.6.2/lib/clang/3.6.2/include/
3. Prepare the project
The compilation database can now get loaded. Run the client tool from within the build directory.
$ rc -J . <br />[Server] Compilation database loaded
You should see plenty of output from the RTags server. That server will spin up a bunch of indexer processes named rp.
$ top<br />PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE97756 rp 80.2 00:02.80 2/1 0 12 58M+ 0B 0B 97612 97612 running97754 rp 76.1 00:04.51 2/1 0 12 104M+ 0B 0B 97612 97612 running97751 rp 76.4 00:06.38 2/1 0 12 140M+ 0B 0B 97612 97612 running97749 rp 78.8 00:07.34 2/1 0 12 155M+ 0B 0B 97612 97612 running97745 rp 78.0 00:12.45 2/1 0 12 212M+ 0B 0B 97612 97612 running97719 rp 77.0 00:32.66 2/1 0 12 484M+ 0B 0B 97612 97612 running97714 rp 70.8 00:37.56 2/1 0 12 544M+ 0B 0B 97612 97612 running97612 *rdm* 4.3 00:11.71 4 0 32 18M 0B 0B 97612 60329 sleeping
Limitations
Due to some issues of RTags underlying implementation and glue code, some of the Apache Mesos files are currently failing on this step, specifically units that receive the build and source directories via build environment variables.
[ 62%] 284/457 17:12:13 ~/Development/mesos/src/tests/containerizer/docker_tests.cpp in 45721ms. (767 syms, 1376 symNames, 13 includes, 1 of 1337 files, symbols: 4634 of 55726, 0 queried) (45513/192/10ms). (priority 0)Output from /Users/till/Development/mesos/src/tests/containerizer/docker_tests.cpp:/Users/till/Development/mesos/build/src/../../src/tests/flags.hpp:60:45: error: expected expression/Users/till/Development/mesos/build/src/../../src/tests/flags.hpp:60:45: error: use of undeclared identifier 'Users'
Let's try it!
Note that this example is highly volatile as it will break with changes in stout. For the given example, the number 62 is a line number and 8 is the column number (both 1 based) at a type reference of hashset. The command is run from the mesos source root folder.
$ rc -f 3rdparty/libprocess/3rdparty/stout/tests/os_tests.cpp:62:83rdparty/libprocess/3rdparty/stout/include/stout/hashset.hpp:34:7: class hashset : public boost::unordered_set<Elem, Hash, Equal>
Success, RTags correctly resolved the type hashset to a class declaration in stout's hashset.hpp.
Integrate RTags with a text editor
Sublime Text 3
RTagsComplete aka SublimeRTags is available via package control. RTagsComplete in its current form needs a bit of tweaking to fit.
1. Go to your Sublime package directory and clone a patched version of RTagsComplete, then restart Sublime.
$ cd ~/Library/Application Support/Sublime Text 3/Packages $ git clone https://github.com/tillt/sublime-rtags.git
2. Optional: You might want to edit the settings via Sublime Text > Preferences > Package Settings > SublimeRTags > Settings - User. Specifically, if you have tweaked your Sublime to support C++ as something else but source.c++ - e.g. source.c++.11 for extended syntax highlighting via the C++11 package, you must extend the file_types setting.
{ "rc_path": "/usr/local/bin/rc", "rdm_path": "/usr/local/bin/rdm", "file_types": [ "source.c", "source.c++", "source.c++.11" ], "jump_limit": 10 }
3. Add these keyboard shortcuts via Sublime Text > Preferences > Package Settings > SublimeRTags > Key Bindings - User.
[ # Note that you will want need to merge this with whatever shows up by default. # [..] {"keys": ["ctrl+r"], "command": "rtags_location", "args": {"switches": ["--absolute-path", "-r"]} }, {"keys": ["ctrl+period"], "command": "rtags_location", "args": {"switches": ["--absolute-path", "-f"]} }, {"keys": ["ctrl+shift+period"], "command": "rtags_go_backward" } ]
You should now be able to follow to the declaration of a symbol using [CTRL + .], jump back using [CTRL + SHIFT + .]. For listing all references, press [CTRL + R].
Vim
1. Install the vim-rtags plugin through your preferred channel, see the README for instructions.
The authoritative source for the provided default keybindings is .
Mappings are provided to e.g. jump-to-definition, dumping of symbol information, finding of various kinds of references, or symbol renaming.
Run the RTags server production
For running the RTags server automatically, the recommended way is using Apple's launchd. Create a file with the following contents. Save it as ~/Library/LaunchAgents/com.andersbakken.rtags.agent.plist
Note that you must replace YOUR-USER-NAME-GOES-HERE with your username.
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"> <dict> <key>Label</key> <string>com.andersbakken.rtags.agent</string> <key>ProgramArguments</key> <array> <string>sh</string> <string>-c</string> <string>/usr/local/bin/rdm -v --launchd --inactivity-timeout 300 --log-file ~/Library/Logs/rtags.launchd.log</string> </array> <key>Sockets</key> <dict> <key>Listener</key> <dict> <key>SockPathName</key> <string>/Users/YOUR-USER-NAME-GOES-HERE/.rdm</string> </dict> </dict> </dict></plist>
For safely aborting the previously manually loaded RTags server, press [CTRL + C]. You may now setup automatic loading via launchd.
$ launchctl load ~/Library/LaunchAgents/com.andersbakken.rtags.agent.plist
The RTags server will now run on demand. You may always inspect the logs for further troubleshooting.
$ tail -f ~/Library/Logs/rtags.launchd.log
Credits
Compiled by Benjamin Bannier and Till Toenshoff
RTags by Jan Erik Hanssen and Anders Bakken
Bear by László Nagy
SublimeRTags by Sergei Turukin
vim-rtags by lyuts
System Integrity Protection - Adding another layer to Apple's security model
