C Custom Warnings

By Nikolay Merinov, SW Team Lead

Motivation

We had a task to investigate а big amount of a C code in order to estimate how many issues we can face in future work with this code. This task had the following properties:

  1. It was a huge amount of code. And it was not possible to investigate it manually or semi-automatically. All the statistics should be gathered automatically.
  2. Big part of this code was obsolete and was not compiled at all. This included two cases: there were files that were not compiled and there was a code in the compiled files that was disabled with preprocessor directives. Our estimations should have been only for the active code, i.e. the one, that is currently compiled and used.
  3. It was not possible to create a run-time test with a full (or at least a big enough) code coverage. All our estimations should have been made at the compile time.

In order to solve this task we decided to add custom compile time warnings through system headers. Only when the work was finished, we’ve discovered that the GNU Libc widely uses the same technique for a -D_FORTIFY_SOURCE support.

 

Implementation Technique

There is a broadly known technique to stop a compilation on any of the compile-time conditions:

// Starting from C11 there is _Static_assert construction
#define non_zero_constant_argument(x) \
  _Static_assert(x, "Argument should be non zero")
// Before C11 you can declare an impossible array with a constant integer expression
#defime non_zero_constant_argument_pre_C11(x) \
  char static_assert_argument_should_be_not_zero[(x) != 0 ? 1 : -1]

But there is no standard technique to add a compilation warning instead of a compilation error. Luckily, there are compiler-specific ways to generate warnings instead of errors.

 

GCC Compilation Warnings

Our environment was designed to work with the GCC compiler and we worked only with the GCC to implement custom warnings. The GCC among other function attributes supports the next attributes that are suitable to emit warnings during the compilation:

  1. void deprected_function() __attribute__((deprecated("custom message"))); will emit a compilation warning on each deprecated_function usage in code, except function declarations.

    void __attribute__((deprecated("custom message"))) warning() { }
    int main () {
      // warning: ‘warning’ is deprecated: custom message [-Wdeprecated-declarations]
      warning();
    }
    

    Unfortunately, this attribute does not allow to add any conditions for warnings.

  2. void warning_if_false(void *arg) __attribute__((nonnull(1))); will emit a warning if an integer constant expression in the first argument is calculated as zero:

    void __attribute__((nonnull(1))) warning_if_false(void * arg) { }
    int main () {
      // warning: null argument where non-null required (argument 1) [-Wnonnull]
      warning_if_false((void *)(sizeof(long) == 7));
    }
    

    Such conditions allow us to define conditional warnings, but it does not provide the possibility to emit any message related to this warning.

  3. void warning_if_true(int unused, ...) __attribute__((sentinel)); will emit a warning, if an integer constant expression in the last argument is calculated as non-zero value:

    void __attribute__((sentinel)) warning_if_true(int unused, ...) { }
    int main () {
      // warning: missing sentinel in a function call [-Wformat=]
      warning_if_true(0, (void *)(sizeof(long) != 7));
    }
    

    This solution in the same manner as the previous one does not allow to write a custom warning message.

  4. int warning() __attribute__((warn_unused_result)); will emit a warning if the function was compiled and the function result was not used in any way:

    int __attribute__((warn_unused_result)) warning() { return 0; }
    int main () {
      // warning: ignoring return value of a ‘warning, declared with an attribute warn_unused_result [-Wunused-result]
      (void)((sizeof(long) != 7) ? warning() : 0);
    }
    

    This attribute allows to emit warnings based on a condition, but if we want to provide a custom warning message, we should embed this message into the function name:

    int __attribute__((warn_unused_result)) warning_64bit_support_is_unstable() { return 0; }
    // warning: ignoring the return value of a ‘warning_64bit_support_is_unstable’, declared with an attribute warn_unused_result [-Wunused-result]
    (void)((sizeof(void *) == 8) ? warning_64bit_support_is_unstable() : 0);
    
  5. int warning() __attribute__((warning("custom message"))); will emit a warning with a custom message if a call to such function is not eliminated through the dead code elimination or other optimizations:

    void __attribute__((warning("custom message"))) warning() { }
    int main () {
      // warning: call to ‘warning’ declared with an attribute warning: custom message
      (void)((sizeof(void *) != 7) ? warning() : 0);
    }
    

    This attribute allows both custom messages and compile-time conditions. And even better: this attribute emits a warning after the dead code elimination and we can use this warning-attributed function both in a conditional block and as integer constant expressions. Consider the following code to understand what benefits it provides us:

    void __attribute__((warning("..."))) warning1() { }
    int __attribute__((warn_unused_result)) warning2() { return 0; }
    if (sizeof(void *) == 7) {
      // Will not emit a warning: dead code elimination will remove if block
      warning1();
      // Will emit a warning: return value is ignored
      warning2();
    }
    // Will not emit a warning: warning2 will be removed on the GCC constant folding
    (void) ((sizeof(void *) == 7) ? warning2() : 0);
    

 

Obtaining Usefull Compile-Time Information

First of all, in order to use this warnings technique for your code, you should have wrappers for all your APIs:

library.c:
void __null_warning() { }
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
extern void __null_warning() __attribute__((warning("function called with a NULL argument")))
extern int __internal_do_stuff(int *arg);

// A function can be wrapped with an inlined function to make sure that the GCC will propagate the argument values
// inline qualification does not guarantee that the function will be inlined; use an attribute additionally
static inline int __attribute__((always_inline)) do_stuff(int *arg) {
  if (arg == NULL) {
    __null_warning();
  }
  return __internal_do_stuff(arg);
}

// OR a function can be wrapped with a macro with the GCC Compound Expression.
// https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html
#define do_stuff(arg) ({    \
  if (arg == NULL) {        \
    __null_warning();       \
  }                         \
  __internal_do_stuff(arg); \
})

Suggested code will emit a warning on every do_stuff(0) call. Unfortunately, this code will also emit warnings for the calls like do_stuff(non_constant_argument). In order to create more useful warnings you should consider to use additional GCC built-in functions from the https://gcc.gnu.org/onlinedocs/gcc/Object-Size-Checking.html and the https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html#Other-Builtins.

This builtins allowed us to emit the required information from the compiler:

#define do_stuff(arg) ({                          \
  if (__builtin_constant_p(arg) && arg == NULL) { \
    __called_with_a_null_constant_warning();      \
  }                                               \
  if (!__builtin_types_compatible_p(typeof(arg), int *)) { \
    __called_with_an_int_pointer();               \
  }                                               \
  if (__builtin_object_size(arg, 0) == -1) {      \
    __called_with_a_pointer_to_an_unknown_object_warning(); \
  }                                               \
  if (__builtin_object_size(arg, 0) != -1         \
      && __builtin_object_size(arg, 0) < 20 ) {   \
    __called_with_a_pointer_to_an_object_smaller_then_20_bytes_warning(); \
  }                                               \
  if (!__builtin_types_compatible_p(typeof(&((arg)[0])), typeof(arg))) { \
    __called_with_an_array_argument_warning();    \
  }                                               \
  __internal_do_stuff(arg);                       \
})

This technique can be adjusted for any purposes: the GNU Libc used this technique to find the memory bounds violations in the string functions when compiled with the -D_FORTIFY_SOURCE=1, this technique can be used to warn about a non-recommended API usage (e.g. “in next versions calls with NULL argument will be forbidden”).

 

Issues and Workarounds

  1. To use the __attribute__((warning("message"))) you should ensure that the attributed function will not be optimized from the code. This can be achieved in two ways:

    • You can place a definition of a warning function to a library and call it from a library header:

      library.c:
      void library_warning() { }
      int __internal_do_stuff(int *arg) {
        // do usefull stuff
        return 0;
      }
      
      
      library.h:
      extern void library_warning() __attribute__((warning("Warning message")));
      extern int __internal_do_stuff(int *arg);
      #define do_stuff(arg) ({ \
        library_warning();     \
        __internal_do_stuff(); \
      })
      

      This solution is used by the GNU Libc for the __warn_memset_zero_len diagnostic function.

    • Or you can disable the optimization of a warning function directly in the header file with another GCC attribute:

      library.c:
      int __internal_do_stuff(int *arg) {
        // do usefull stuff
        return 0;
      }
      
      
      library.h:
      static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() {  }
      extern int __internal_do_stuff(int *arg);
      #define do_stuff(arg) ({ \
        library_warning();     \
        __internal_do_stuff(); \
      })
      

      For Inango code we have selected this solution.

  2. With the code described above all your warnings will suddenly disappear once you install your library.h header to the /usr/include directory. This happens because the GCC suppresses all warnings that were found in the code of system headers. You can bring back your warnings in two ways:

    • If you use a function wrapper you can use the “artificial” GCC attribute:

      library.c:
      void library_warning() { }
      int __internal_do_stuff(int *arg) {
        // do usefull stuff
        return 0;
      }
      
      
      library.h:
      extern void library_warning() __attribute__((warning("Warning message")));
      extern int __internal_do_stuff(int *arg);
      static inline void __attribute__ ((always_inline, artificial)) do_stuff() {
        library_warning();
        __internal_do_stuff();
      }
      

      This solution used by GNU Libc.

    • If you use a macro wrapper you should enable “-Wsystem-headers” warnings for your code:

      library.c:
      int __internal_do_stuff(int *arg) {
        // do usefull stuff
        return 0;
      }
      
      
      library.h:
      static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() {  }
      extern int __internal_do_stuff(int *arg);
      #pragma GCC diagnostic push
      #pragma GCC diagnostic warning "-Wsystem-headers"
      #define do_stuff(arg) ({ \
        library_warning();     \
        __internal_do_stuff(); \
      })
      #pragma GCC diagnostic pop
      
  3. The solution with warnings will conflict with the usage of the “-Werror” compilation flag. As on December 2018 all the GCC versions will fail the compilation with an error if the __attribute__((warning(""))) code will be compiled with the “-Werror” flag. There is no way to change this behavior. The possible solutions to this issue:

    • You can change to use the __attribute__((warn_unused_result)) instead:

      library.c:
      int __internal_do_stuff(int *arg) {
        // do usefull stuff
        return 0;
      }
      
      
      library.h:
      // this pragma should be global to force moving this diagnostics from errors to warnings
      #pragma GCC diagnostic warning "-Wunused-result"
      #pragma GCC diagnostic push
      #pragma GCC diagnostic warning "-Wsystem-headers"
      static inline void __attribute__((warn_unused_result, optimize("O0"))) library_warning() {  }
      extern int __internal_do_stuff(int *arg);
      #define do_stuff(arg) ({ \
        (void)((integer_constant_expression_condition) ? library_warning() : 0);     \
        __internal_do_stuff(); \
      })
      #pragma GCC diagnostic pop
      
    • Or you can wait for the next GCC upstream release where we added the solution for the __attribute__((warning(""))) itself and use a new “-Wno-error=attribute-warning” that we submitted to the GCC as a result of the described work.

 

Final Code Template

Finally, let’s combine all the described work together and show how the custom warnings can be created for a GCC-based code:

        library.c:
        int __internal_do_stuff(int *arg) {
          // do usefull stuff
          return 0;
        }

        library.h:
        #ifndef LIBRARY_H
        #define LIBRARY_H

        /* Interpret all the "warning" attributes in a code compiled with this library as warnings,
           regardless of -Werror or -Wno-attribute-warning flags.
           As on Dec 2018, this flag is supported only in GCC trunk */
        #pragma GCC diagnostic warning "-Wattribute-warning"

        /* Declare the internal functions that we will wrap */
        extern int __internal_do_stuff(int *arg);

        /* Create the warning functions with a specific messages.
           Make sure that these functions would not be optimized out from the code.
           Define the function as a static to not export the name from the resulting binaries. */
        static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() {  }

        /* Print these warnings even if the library.h is installed to the system headers directory.
           This behavior will be applied only to the library.h code.
           Other system headers will not print the warnings */
        #pragma GCC diagnostic push
        #pragma GCC diagnostic warning "-Wsystem-headers"

        /* Wrap all the functions in the macro definitions with the GCC Compound expressions inside.
           Make sure that all the `if' conditions are integer constant expressions. Otherwise,
           there is no guarantees that the compiler will be able to optimize out the failed checks. */
        #define do_stuff(arg) ({                       \
          if (integer_constant_expression_condition) { \
            library_warning();                         \
          }                                            \
          __internal_do_stuff();                       \
        })

        #pragma GCC diagnostic pop

        #endif /* LIBRARY_H */

 

Clang Based Solution

There is an alternate solution for the described issue supported in the Clang compiler. In order to emit compile-time warnings in Clang, you can just specify the attribute on the relevant function:

library.h:
int do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
int do_stuff(int *arg) __attribute__((diagnose_if(arg == NULL, "arg is NULL", "warning")));
int do_stuff(int *arg) 
  __attribute__((diagnose_if(__builtin_object_size(arg, 0) == -1, "arg of unknown size", "warning")));

With the Clang __attribute__((diagnose_if(...))) there is no need in additional wrappers (you still should create a wrapper if you want to extract the information about the original type of the argument) and new warnings can be simply added with additional declaration lines.

But the GNU GCC is still the main compiler in GNU/Linux systems and we think that the proposed article will be usable for all the GCC users.

 

Yocto Source Mirrors Mechanism

By Nikolay Merinov, Software Team Lead

 

Foreword

The BitBake build utility, adapted by the Yocto Project, allows for flexible configuration of source mirrors for all the remote files required during the build of Yocto distribution. This functionality allows to speed up the distribution build, as well as organize a local backup with sources of all the packages used.
However, this mechanism is rather poorly documented, and we are going to fix this situation in the current article.

 

Bitbake Source Mirrors

The BitBake mirrors mechanism is supported at least in three places:
 
1. PREMIRRORS are the primary mirrors that will be used before the upstream URI from SRC_URI. It can be used to provide quick mirrors to speed-up the sources downloading.
2. MIRRORS are the additional mirrors that are used when an upstream URI from SRC_URI is not accessible. Creating MIRRORS is a good idea for long-living distributions, when a distribution can out-live the upstream sources of a used software.
3. SSTATE_MIRRORS are the mirrors for a prebuilt cache data objects. SSTATE_MIRRORS can be used to share the pre-built objects from a CI builds.
 
The Yocto Project additionally provides the support for the next mechanisms:
 
1. own-mirrors.bbclass is a standard way to provide a single pre-mirror for all the supported fetchers:

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "http://example.com/my-source-mirror"

2. BB_GENERATE_MIRROR_TARBALLS variable can be set to "1" to allow reusing DL_DIR from the current build as a mirror for other builds. Without a BB_GENERATE_MIRROR_TARBALLS variable the resulting DL_DIR would not provide suitable mirrors for sources fetched from VCS repositories.
3. BB_FETCH_PREMIRRORONLY and SSTATE_MIRROR_ALLOW_NETWORK variables can be used to configure BitBake to download sources and build artifacts only from the configured mirrors. These variables will be usable for all the BB_NO_NETWORK configurations.
 
 

Yocto Documentation on Mirror Syntax

All things described in the previous part are correctly documented and can be found in the official documentation. But a syntax for mirror rules itself is poorly documented. An official documentation suggests only three examples how mirror rules can be created:

1. In the example[1] for a PREMIRRORS variable we can see that we should use .*/.* regexp to match all URI for a specific fetcher:

PREMIRRORS_prepend = "\
git://.*/.* http://www.Yoctoproject.org/sources/ \n \
ftp://.*/.* http://www.Yoctoproject.org/sources/ \n \
http://.*/.* http://www.Yoctoproject.org/sources/ \n \
https://.*/.* http://www.Yoctoproject.org/sources/ \n"

2. In the example[2] for a SSTATE_MIRRORS variable we can find that there is also a support for a PATH string in the matching:

SSTATE_MIRRORS ?= "\
file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
file://.* file:///some-local-dir/sstate/PATH"

3. In the same SSTATE_MIRROS variable from example[2] we can additionally find an information that a mirror syntax supports a more sophisticated regex usage:

SSTATE_MIRRORS ?= "file://universal-4.9/(.*) http://server_url_sstate_path/universal-4.8/\1 \n"

But there is no full description of the mirror matching mechanism. We offer the following as a way of filling this gap.

 

Yocto Mirror Syntax

All mirrors are defined as a “pattern replacement” pair. The BitBake attempts to match a source URI against each “pattern” and on a successful match the BitBake generates a new mirror URI based on an original URI, by using the “pattern” and “replacement” definitions.
 

URI matching

Each URI, including the “pattern” and “replacement” strings, is split into six components before matching as follows:

scheme://user:password@hostname/path;parameters.

In an upstream URI this parts is understood as strings, in a “pattern” this parts is interpreted as a Python re regexps (parameters is the only field that is not regexp); in a “replacement” this parts is interpreted as regexp replacements in terms of a Python re.sub function.
 
All parts from an upstream URI are matched against regexps from the corresponding part of a “pattern” with the next additional rules:
 
1. The BitBake always adds a $ sign to the end of a scheme regex. It was done to ensure that the “http” scheme would not match the “https” string.
2. If a scheme is a “file” then the hostname is assumed to be empty. Otherwise the hostname is a text between “://” and next “/” characters.
3. parameters are interpreted as a list of “param=value” pairs. For each such pair specified in a “pattern” there should be a full match in an original SRC_URI. It is the only field that is matched with a string equality instead of a regexp matching.
NOTE: A parameters matching was broken up to Yocto 2.6 release. This issue was found and fixed as a part of work on this document.
 

URI Replacement

If a URI was successfully matched, then the BitBake creates a new mirror URI from an upstream URI using a “replacement” string. The BitBake uses the next rules to make a replacement:
1. A replacement is made for each part of the URI separately.
2. Before replacement is performed, we replace the next substrings in a corresponding part of a “replacement” string:

    1. “TYPE” — this substring in a “replacement” string will be replaced with an original URI scheme.
    2. “HOST” — will be replaced with an original URI hostname.
    3. “PATH” — will be replaced with an original URI path.
    4. “BASENAME” — will be replaced with a basename of a “PATH” (with a substring after the last “/” symbol).
    5. “MIRRORNAME” — will be replaced with a special string obtained as concatenation of a “HOST” with “:” symbols replaced with “.” symbols, and a “PATH” with “/” and “*” symbols replaced with a “.” symbols.

NOTE: In the parameters list this replacement is made only in a value part of the param=value pairs.
 
3. Each part except the parameters is replaced with re.sub(part_from_pattern, part_from_replacement, part_from_original_uri, 1) Python command. This means that the BitBake will replace only a first match from a current part and that you can use a Python regex replacement syntax including "\1" syntax to insert parts of a matched URI to a result.
4. If an original URI scheme differs from a “replacement” scheme then the original parameters will be wiped off.
5. The parameters from a “replacement” should be added to a resulting URI. If there were parameters with the same names in an original URI then the value for these parameters will be overridden.
6. Finally, the BitBake checks that a path part upon replacement is finished with a suggested “basename”. If a resulting path ends with any other string then a resulting path will be concatenated with a “/basename” string. A suggested “basename” is created according to the next rules:
 
1. If a scheme was not changed then the “basename” is just a basename of the path from an original URI.
2. If a scheme was changed and an original URI points to a single file or a tarball with sources, then the same basename will be used.
3. If a scheme was changed and an original URI points to some kind of a repository then the mirrortarball name will be used. This name is a fetcher specific. e.g. for a git repository a tarball name will be "git2_hostname.path.to.repo.git.tar.gz".
 
NOTE: The last step means that it is impossible to use the "file:///some/path/b.tar.gz" as a mirror path for the "http://other/path/a.tar.gz", but you still can use the "file:///some/path/a.tar.gz" or the "file:///some/path/prefix_a.tar.gz".

 

Step by Step Process Description

Let’s check how the next example will be processed:

SRC_URI = "git://git.invalid.infradead.org/foo/mtd-utils.git;branch=master;tag=1234567890123456789012345678901234567890"
PREMIRRORS = "git://.*/.*;branch=master git://somewhere.org/somedir/MIRRORNAME;protocol=http;branch=master_backup \n"

 
First of all, we will split all the three URI’s into the next parts:
 

upstream:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "git.invalid.infradead.org"
    path       = "/foo/mtd-utils.git"
    parameters = {"branch": "master", "tag": "1234567890123456789012345678901234567890"}
pattern:
    scheme     = "git$"
    user       = ""
    password   = ""
    host       = ".*"
    path       = "/.*"
    parameters = {"branch": "master"}
replacement:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "somewhere.org"
    path       = "/somedir/MIRRORNAME"
    parameters = {"branch": "master_backup", "protocol": "http"}

On the next step the BitBake will match the “upstream” URI parts against the corresponding parts of a “pattern” with a re.match(pattern.part, upstream.part) command for all the parts except the parameters. For the “parameters” the BitBake will check that the value for a “branch” parameter in an “upstrem” URI and a “pattern” URI are equal.
When these checks pass, the BitBake will start a replacement process. In each part of the “replacement” BitBake will make the replacements for the special strings:
 

replacement:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "somewhere.org"
    path       = "/somedir/git.invalid.infradead.org.foo.mtd-utils.git"
    parameters = {"protocol": "http", "branch": "master_backup"}

Then the BitBake will make the actual replacements with a re.sub(pattern.part, replacement.part, upstream.part, 1) command for all parts except the parameters. This means that if you want to replace the string completely, the full pattern should be a “.*”, and not “”. For the parameters the new keys will be added to the list and the old values will be replaced. As the result we will get as follows:
 

result:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "somewhere.org"
    path       = "/somedir/git.invalid.infradead.org.foo.mtd-utils.git"
    parameters = {"branch": "master_backup", "protocol": "http", "tag": "1234567890123456789012345678901234567890"}

On the last step the BitBake will check that the result.path is finished with a basename of an upstream.path. Since the result.scheme and the upstream.scheme are the same, the basename will be defined as an mtd-utils.git (if a scheme will be different, then the basename would be defined to a "git2_git.invalid.infradead.org.foo.mtd-utils.git.tar.gz").
 
When the last check passes, the BitBake will combine a result to a new mirror URI that will be used as an alternative source for the files:
 
git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;branch=master_backup;protocol=http;tag=1234567890123456789012345678901234567890.
 
You can find additional replacements examples in the BitBake unit tests code[3].

 
 

Used Documentation

[1]: Simple example
[2]: Extended example
[3]: Additional replacement examples
[4]: BitBake mirror mechanism implementation
 
 

Download a PDF