Compile mutrace on RHEL6

Why mutrace

Recently, I want to profile the linux user lock(pthread mutex/rwlock) performance of a project, and there are few options for this purpose:

  • valgrind(drd)
  • systemtap(futexes.stp)
  • lttng
  • mutrace

Finally, I selected the mutrace due to:

  1. valgrind(drd): It’s really slow, cannot provide credible information.
  2. systemtap(futexes.stp): It’s really great in profiling kernel, but the user layer profile need to additional efforts to setup the environment(some debuginfo pkg).
  3. lttng: There is only rpm pkg available on RHEL7, have to compile by myself. And there are more than one dependency lib, hard to fix the dependency issue.
  4. mutrace: There is no rpm pkg available on RHEL6, have to compile by myself, and fortunately it’s not very hard to pass the compilation.

Installation Steps

To get it done, need a few steps:

  1. Get the source code
    git clone git://
  2. Modify the autoconf required version from 2.68 to 2.63
    diff --git a/ b/
    index fcb1397..0d36e41 100644
    --- a/
    +++ b/
    @@ -18,7 +18,7 @@
    # You should have received a copy of the GNU Lesser General Public
    # License along with mutrace. If not, see <>.
    AC_INIT([mutrace], [0.2], [mzzhgenpr (at) 0pointer (dot) net])
  3. Upgrade gcc version to at least gcc 4.6

  4. Do some code changes for backtrace-symbols.c to pass the compilation

    diff --git a/backtrace-symbols.c b/backtrace-symbols.c
    index 0a0d751..6f84c56 100644
    --- a/backtrace-symbols.c
    +++ b/backtrace-symbols.c
    @@ -34,6 +34,8 @@
      along with this program; if not, write to the Free Software
      Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.  */
    +#include "config.h"
    #define fatal(a, b) exit(1)
    #define bfd_fatal(a) exit(1)
    #define bfd_nonfatal(a) exit(1)
    @@ -44,13 +46,13 @@
    #define true 1
    #define false 0
    -#define _GNU_SOURCE
    +//#define _GNU_SOURCE
    #include <string.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <execinfo.h>
    #include <bfd.h>
    -#include <libiberty.h>
    +//#include <libiberty.h>
    #include <dlfcn.h>
    #include <link.h>
    #if 0
  5. Run ./ to generate Makefile

  6. Change ‘-O0′ to ‘-O2′ from CFLAGS in the Makefile
  7. Run make and make install

In the end

Now, mutrace, matrace and the other related libraries have been generated, profile your program by:

mutrace -r $test_app ...

And big thanks to the original author @Lennart Poettering


have a fun :)

Flibs 0.7.4 Released

The flibs 0.7.4 has been released, which include the following changes:

1. Refactor all the makefiles, now it can be built in parallel
2. Refactor the header files folder structure, to make them more user friendly
3. Refactor some apis, to make them more user friendly
4. Replace malloc with calloc, to avoid potential un-initialization issue
5. Fix some type conversion incorrect issue, to make it more stablization
6. Fix the valgrind errors when building the libs on 32 bit platform

For example, if you want to compile the static-link libraries, now just run:

make -j4
make install

And all the header files have been moved to include/flibs, so whatever you include it from the submodule of your project or /usr/local, just write it one time:

#include <flibs/xxx.h>

Ok, so for now, all the api prefix are starting with “f”, and you can identify them easier than before.

Just enjoy and have a fun :)

Summary of 2014

In the past year, for my personal projects, I did some achievements on them, and created some new projects as well, here I just summarize them for the recording purposes.

Exist Projects

  • flibs: upgraded to 0.6.6, including:
    • The code base refactor
    • fhash refactor
    • flog optimization
    • other bug fixing
  • fenv: more user friendly, and add more than 30+ commits

New Projects

  • ftracer: C/C++ program tracer, which is useful in deep diving into a complex project quickly
  • openwrt-scripts: now, it contain a dnsmasq config generator

The Next

And, in the next year, the draft plan as following:

  1. Separate fmem and fpcap from flibs
  2. Refine the makefile for flibs, make it more user friendly, faster and portable
  3. Create a fstr in flibs, which will be great useful for many C programs
  4. Continue add more scripts in openwrt-scripts project, such as:
    • Geo based ip record selector for dnsmasq or pdnsd
    • Secure dns reply policy for dnsmasq
  5. A new server side framework will be announced

Good bye 2014, good luck 2015 :)

Refactor fhash is done


Aha, After some days, I have finished the refactor task for fhash, it’s really really big improvement for flibs, since many many components depend on it, such as:

  • thread pool
  • event framework
  • pcap conversion lib
  • log

Why need to do refactor

The old fhash have some defects, such as:

  • hard to iterate
  • hard to extend
  • hard to modify fhash when iteration
  • some performance issues during iteration and fhash_set interface

so finally, I decided to rewrite it. Before we go through the new design, let’s take a look on the old design at below section.

Graph of Old/New design

mind mapping software

The new design

The new design fix all the issues in old design, which much clean and user friendly.

Let’s take a look the core APIs:

fhash* fhash_create(uint32_t init_size, fhash_opt opt, uint32_t flags);
void fhash_delete(fhash* table);

void fhash_set(fhash* table,
               const void* key, key_sz_t key_sz,
               const void* value, value_sz_t value_sz);
void* fhash_get(fhash* table, const void* key, key_sz_t key_sz,
               value_sz_t* value_sz);
void* fhash_fetch_and_del(fhash* table,
               const void* key, key_sz_t key_sz,
               void* value, value_sz_t value_sz);

fhash_iter fhash_iter_new(fhash* table);
void fhash_iter_release(fhash_iter* iter);
void* fhash_next(fhash_iter* iter);
void fhash_foreach(fhash* table, fhash_each_cb cb, void* ud);

int fhash_rehash(fhash* table, uint32_t new_size);
void fhash_profile(fhash* table, int flags, fhash_profile_data* data);

Full API and Example Documents

Please refer to wiki

The benchmark

And for this time, I just add a benchmark tool for fhash, so that I can review the performance when I need it, let’s take a look one result I ran it on my virtual box:

========= fhash testing without auto rehash =========
fhash_set x100000 spend time: 9189450 usec
fhash_get x100000 spend time: 6535296 usec
fhash_next x100000 spend time: 1111 usec
fhash_rehash (index double), ret: 0, spend time: 42825 usec
[index]: used: 63226, total: 100000, usage rate: 0.632260
[slots]: used: 100000, total: 107241, usage rate: 0.932479
fhash_del x100000 spend time: 32075 usec
========= fhash testing with auto rehash =========
fhash_set x100000 spend time: 112542 usec
fhash_get x100000 spend time: 35542 usec
fhash_next x100000 spend time: 3333 usec
fhash_rehash (index double), ret: 0, spend time: 57153 usec
[index]: used: 63226, total: 100000, usage rate: 0.632260
[slots]: used: 100000, total: 107241, usage rate: 0.932479
fhash_del x100000 spend time: 37410 usec

So, from above, we can see the performance comparison of disabling/enabling auto rehash, the result with auto rehash is winner, that means in a normal case(if user cannot sure how many items will be putted into hash table), the best solution is enable auto rehash feature, it will avoid the hash table convert to a list.

The End

After rewrite fhash, I realized that:

  • To create a user friendly, extendable program is more important than the performance, since the most important thing is that how can we get start with a new library? If the library is hard to use, user will give up and try to find a another library.
  • Another hand, the documentation is also a very important part in the project, since that’s the easiest way to tell user: what it is and how to use it.

Next Step

In the future, I have some actions to optimize it:

  • I can add a counter in every hash node, to record the get/set actions frequency, so that fhash can have ability to optimize the hash node list in every rehash or some other trigger point, so the highest hotspot node will be putted at the front of list.
  • Use a list instead of array, and compare the performance impact.

Ok, let’s hacking begin!~

The Steps of Creating a New Open Source Library

Recently, I’m doing the refactoring job of fhash lib, which is a part of flibs, during the refactoring, I realised that to create a new open source project, we should have some basic steps, and follow the steps we will keep moving to right forward without lost.


Ok, let’s check the core steps:

  1. API Design Lock Down
  2. Write the code
  3. Write the UT
  4. Write the benchmarking tools
  5. Create a document of these APIs
  6. Announce your library

The benefit of Writing Documents

Most of developers don’t like to write the document, it’s so boring. But actually the document is not written only for yourself, it will be a great help for all the people who will maintain this project. Try to think a about it, if there is a open source library without any API document or comments in the code, will you use it in your project? The answer is NO obviously, so let me brief the benefits of writing document:

  • Document will guide people to understand what is this quickly
  • Document will help people to go to the right forward
  • Document will help people who want to use the project

In The End

There are many many open source project without any document, and all of them almost failed, no one knows what is this, no one knows how to use it. So let’s create the open source project with more documents :D

Ftracer: The C/C++ callgraph generator

Sometimes, we need the callgraph to help us reading the source code, especially the big C++ project, it’s really hard to understand the program quickly/easily.

So, I wrote a tracer to record the whole code path, and then generate the callgraph by the trace file. It helped me to follow the correct code path, save a lot of time.

The full introduce page at:

Have a fun

A Tip of Writing Makefile — Using @

Ok, let’s talk about one of the tips for writing Makefile, the magic character ‘@’

Before continue, let’s think about one thing: when you want to build a project, write a shell script or use the Make for building your project, which one would you prefer?

Ok, I’ll choose Make, the reasons are:

  1. The shell script may contain the code which is specific for the target shell, for example: bash. So, if some users DO NOT use bash, the build process may stop unexpected. And the Make is stand alone, which can run on all UNIX like system, not depend on a specific shell environment.
  2. You need to handle all the building actions, for example, entry the target folder, return back to last folder, do the dependence actions before actual do the building actions, etc, and the Make has already supported all of the above.

If you agree with me, follow me to the next step — Using ‘@’ in your Makefile.

Let’s take a look at a example Makefile:

	echo "hello Makefile"

When we run it, the output is:

bash $ make
echo "hello Makefile"
hello Makefile

From above, we can see the original instructions are shown as well, they are useless and not beautiful, so let’s using ‘@’ to rewrite the Makefile:

	@echo "hello Makefile"

And run it again:

bash $ make
hello Makefile

From above, we can see the original instructions(echo xxx) have gone, it only shows the output of this command. So if you build a very large project, it may contain a lot of instructions, in this way you can using ‘@’ to avoid outputing the command, only show its result.


Compile Clang3.4

As we know, llvm is a great project, and it also include the clang compiler. And for Clang 3.4, it support dumping the format configuration file, so that people can generate the self-style format for their own project. Detail: ClangFormat  and ClangFormatStyleOptions

And for now, Clang 3.4 has not been released, but we can pull its latest code and compile it.
For RHEL6.x, I remind to compile it, and for ubuntu/debian, there are already have the nightly building package(llvm apt source) , people can install it directly.

1. Ok, the compiling instructions as following, I tested it on a rhel6.x, it worked.

git clone
cd llvm/tools
git clone
cd ../projects
git clone
cd ../../
mkdir build
cd build
../llvm/configure --prefix=~/bin/clang34 --enable-optimized --enable-targets=x86_64
make -j4
make install

2. now, add the clang3.4’s binaries into your PATH.

vim ~/.bash_profile
export $PATH

3. If you are using vim, add the following line into your .vimrc (replace the $path-to-this-file with your real path)

map <C-K> :pyf $path-to-this-file/<CR>
imap <C-K> <ESC>:pyf $path-to-this-file/<CR>i

4. Ok, finally, you can try it with clang-format or open a file with vim and using the Control+K to format your code.

[Other References]:

Final :)

Migration WordPress from 5.2.1 to 5.7.1

Ok guys, I just made a mistake and broken my blog few days ago, but finally, as it get back to normal, so let me show you what’s happening.

It is beginning from a mad idea, I upgraded my vps OS to ubuntu 12.04, after that, I found that the php-fpm process has broken(due to the php has upgraded too), and this is just a little bit problem, but I also downloaded the latest version of wordpress, so the disaster has coming to me.

During the upgrading to the latest wordpress, I just follow the instructions from its official website, maybe I did a wrong way, I deleted the wp-admin folder, so when I re-try to access my blog from browser, it shown me nothing, yes, there is nothing I can see. I checked the access log, it’s 500 error in there, but I don’t know what’s the exactly error happened.

God help me, I have not too much time to figure it out what’s the root reason, I only guess the old version of wordpress is not compatible with the latest. After a while, I install the latest wordpress and play with it. it’s more beautiful that the old one, so I decide to use this one.

Ok, the next thing I need to do is migration the old data to the new one, at first I dump all the old wordpress mysql db, and restore to the new one, unfortunately, it failed, after restoring, I cannot see anything from browser. I guess the wp-metadata table is not suitable for the new one or some other reasons I don’t care. And when I check the wp-posts table, I didn’t saw there is any special field depend on other tables, it is only contain the pure data, so I only dumped this table’s data and restored it to the new one, that’s great, it works, aha, so excited for that. The exact command as follow:
1. dump the wp-posts table from old wordpress

mysqldump --add-drop-table -u mysqlusername -p old_databasename wp-posts > old_posts.sql

2. restore the wp-posts table to the new wordpress

mysql -u mysqlusername
 -p new_databasename < old_posts.sql

Finally, as you saw, all the things come back except the comments, if you also care about the comments, you should dump the wp-comments table as well.

Have a fun :D

Final_Libs 0.4.3 released


This is a huge update, I just refine all the filenames and api names, and also support both 32 and 64bit libraries. So that, if someone who want to use the new version, need to be migrated your code to the new apis. If you don’t want to do it directly, I suggest that people should use the 0.3.8 tag, the detail please refer to

have a fun