Pahole

From bwHPC Wiki
Jump to: navigation, search

The main documentation is available via module help devel/pahole on the cluster. Most software modules for applications provide working example batch scripts.


Description Content
module load devel/pahole
Availability bwUniCluster
License GPL
Citing n/a
Links Homepage | Releases
Graphical Interface No


1 Introduction

The poke-a-hole, or short pahole tool is part of the dwarves tool-set. It dissects data structures in binary object files, showing (otherwise useless) padding and data crossing cachelines, allowing optimization and performance analysis of data structures in user- or kernel-code. This tool is worthy to be known by any C and Fortran developer -- the article The lost Art of Structure Packingg by Eric S. Raymond provides an in-detail answer, why.

2 Documentation

There currently is no web documentation, or tutorial. After loading the module, documentation is provided in the man page.

$ man pahole


3 Usage

3.1 Dissecting data structures

You may receive information on the padding and alignment of data structures using pahole. If interested in the data layout pass the option to expand data structures, e.g. the kernel's data structure for every single task, use pahole -E task_struct

struct task_struct {
  struct thread_info {
    long unsigned int  flags;         /*     0     8 */
    unsigned int       status;        /*     8     4 */
  } thread_info;                      /*     0    16 */

  /* XXX last struct has 4 bytes of padding */

  volatile long int    state;         /*    16     8 */
  void *   stack;                     /*    24     8 */
  struct {
    int    counter;                   /*    32     4 */
  } usage;                            /*    32     4 */
  unsigned int         flags;         /*    36     4 */
  unsigned int         ptrace;        /*    40     4 */

  /* XXX 4 bytes hole, try to pack */

  struct llist_node {
    struct llist_node * next;         /*    48     8 */
  } wake_entry;                       /*    48     8 */
  int      on_cpu;                    /*    56     4 */
  unsigned int         cpu;           /*    60     4 */
  /* --- cacheline 1 boundary (64 bytes) --- */
  unsigned int         wakee_flips;   /*    64     4 */
  ...
}

This nicely shows, where the compiler needed to insert padding to adhere to the architecture's alignment requirements specified by the ABI. Additionally it layouts the crossing of past cacheline boundaries, which might be problematic in false sharing of cachelines in multi-threaded programming. Both of this information may be used by You to re-layout your data-structures to minimize them and limit cache-thrashing.

3.2 Usage in own application

To employ this in your own application, recompile with compiler option -g. For the following code (part of the Open MPI implementation) pahole $MPI_LIB_DIR/libmpi.so:

struct ompi_communicator_t {
        opal_infosubscriber_t      super;                /*     0    96 */
        /* --- cacheline 1 boundary (64 bytes) was 32 bytes ago --- */
        opal_mutex_t               c_lock;               /*    96    64 */
        /* --- cacheline 2 boundary (128 bytes) was 32 bytes ago --- */
        char                       c_name[64];           /*   160    64 */
        /* --- cacheline 3 boundary (192 bytes) was 32 bytes ago --- */
        ompi_comm_extended_cid_t   c_contextid;          /*   224    16 */
        ompi_comm_extended_cid_block_t c_contextidb;     /*   240    32 */
        /* --- cacheline 4 boundary (256 bytes) was 16 bytes ago --- */
        uint32_t                   c_index;              /*   272     4 */
        int                        c_my_rank;            /*   276     4 */
        uint32_t                   c_flags;              /*   280     4 */
        uint32_t                   c_assertions;         /*   284     4 */
        int                        c_id_available;       /*   288     4 */
        int                        c_id_start_index;     /*   292     4 */
        uint32_t                   c_epoch;              /*   296     4 */

        /* XXX 4 bytes hole, try to pack */

        ompi_group_t *             c_local_group;        /*   304     8 */
        ompi_group_t *             c_remote_group;       /*   312     8 */
        /* --- cacheline 5 boundary (320 bytes) --- */
        struct ompi_communicator_t * c_local_comm;       /*   320     8 */
        struct opal_hash_table_t * c_keyhash;            /*   328     8 */
        int                        c_cube_dim;           /*   336     4 */

        /* XXX 4 bytes hole, try to pack */

        struct mca_topo_base_module_t * c_topo;          /*   344     8 */
        int                        c_f_to_c_index;       /*   352     4 */

        /* XXX 4 bytes hole, try to pack */

        struct ompi_peruse_handle_t * * c_peruse_handles; /*   360     8 */
        ompi_errhandler_t *        error_handler;        /*   368     8 */
        ompi_errhandler_type_t     errhandler_type;      /*   376     4 */

        /* XXX 4 bytes hole, try to pack */

        /* --- cacheline 6 boundary (384 bytes) --- */
        struct mca_pml_comm_t *    c_pml_comm;           /*   384     8 */
        struct mca_mtl_comm_t *    c_mtl_comm;           /*   392     8 */
        mca_coll_base_comm_coll_t * c_coll;              /*   400     8 */
        int32_t                    c_nbc_tag;            /*   408     4 */

        /* XXX 4 bytes hole, try to pack */

        ompi_instance_t *          instance;             /*   416     8 */
        int                        any_source_offset;    /*   424     4 */

        /* XXX 4 bytes hole, try to pack */

        opal_object_t *            agreement_specific;   /*   432     8 */
        _Bool                      any_source_enabled;   /*   440     1 */
        _Bool                      comm_revoked;         /*   441     1 */
        _Bool                      coll_revoked;         /*   442     1 */

        /* size: 448, cachelines: 7, members: 32 */
        /* sum members: 419, holes: 6, sum holes: 24 */
        /* padding: 5 */
};

The output shows, any single MPI_Comm requires a total of 448 Bytes, extending to over 7 64-Byte cache lines, with 6 holes of 4 Byte each. Rearranging the above data structure would reduce the amount of holes (e.g. move above any_source_offset to after agreement_specific), taking into account data access in the most-common code paths would also optimize cache line usage. Low hanging fruits such as eliminating (or reducing or moving to the end) c_name, which is solely used for application/ompi debugging, might help as well...