We had a task to investigate а big amount of a C code in order to estimate how many issues we can face in future work with this code. This task had the following properties:
In order to solve this task we decided to add custom compile time warnings through system headers. Only when the work was finished, we’ve discovered that the GNU Libc widely uses the same technique for a -D_FORTIFY_SOURCE
support.
There is a broadly known technique to stop a compilation on any of the compile-time conditions:
// Starting from C11 there is _Static_assert construction
#define non_zero_constant_argument(x) \
_Static_assert(x, "Argument should be non zero")
// Before C11 you can declare an impossible array with a constant integer expression
#defime non_zero_constant_argument_pre_C11(x) \
char static_assert_argument_should_be_not_zero[(x) != 0 ? 1 : -1]
But there is no standard technique to add a compilation warning instead of a compilation error. Luckily, there are compiler-specific ways to generate warnings instead of errors.
Our environment was designed to work with the GCC compiler and we worked only with the GCC to implement custom warnings. The GCC among other function attributes supports the next attributes that are suitable to emit warnings during the compilation:
void deprected_function() __attribute__((deprecated("custom message")));
will emit a compilation warning on each deprecated_function
usage in code, except function declarations.
void __attribute__((deprecated("custom message"))) warning() { }
int main () {
// warning: ‘warning’ is deprecated: custom message [-Wdeprecated-declarations]
warning();
}
Unfortunately, this attribute does not allow to add any conditions for warnings.
void warning_if_false(void *arg) __attribute__((nonnull(1)));
will emit a warning if an integer constant expression in the first argument is calculated as zero:
void __attribute__((nonnull(1))) warning_if_false(void * arg) { }
int main () {
// warning: null argument where non-null required (argument 1) [-Wnonnull]
warning_if_false((void *)(sizeof(long) == 7));
}
Such conditions allow us to define conditional warnings, but it does not provide the possibility to emit any message related to this warning.
void warning_if_true(int unused, ...) __attribute__((sentinel));
will emit a warning, if an integer constant expression in the last argument is calculated as non-zero value:
void __attribute__((sentinel)) warning_if_true(int unused, ...) { }
int main () {
// warning: missing sentinel in a function call [-Wformat=]
warning_if_true(0, (void *)(sizeof(long) != 7));
}
This solution in the same manner as the previous one does not allow to write a custom warning message.
int warning() __attribute__((warn_unused_result));
will emit a warning if the function was compiled and the function result was not used in any way:
int __attribute__((warn_unused_result)) warning() { return 0; }
int main () {
// warning: ignoring return value of a ‘warning, declared with an attribute warn_unused_result [-Wunused-result]
(void)((sizeof(long) != 7) ? warning() : 0);
}
This attribute allows to emit warnings based on a condition, but if we want to provide a custom warning message, we should embed this message into the function name:
int __attribute__((warn_unused_result)) warning_64bit_support_is_unstable() { return 0; }
// warning: ignoring the return value of a ‘warning_64bit_support_is_unstable’, declared with an attribute warn_unused_result [-Wunused-result]
(void)((sizeof(void *) == 8) ? warning_64bit_support_is_unstable() : 0);
int warning() __attribute__((warning("custom message")));
will emit a warning with a custom message if a call to such function is not eliminated through the dead code elimination or other optimizations:
void __attribute__((warning("custom message"))) warning() { }
int main () {
// warning: call to ‘warning’ declared with an attribute warning: custom message
(void)((sizeof(void *) != 7) ? warning() : 0);
}
This attribute allows both custom messages and compile-time conditions. And even better: this attribute emits a warning after the dead code elimination and we can use this warning-attributed function both in a conditional block and as integer constant expressions. Consider the following code to understand what benefits it provides us:
void __attribute__((warning("..."))) warning1() { }
int __attribute__((warn_unused_result)) warning2() { return 0; }
if (sizeof(void *) == 7) {
// Will not emit a warning: dead code elimination will remove if block
warning1();
// Will emit a warning: return value is ignored
warning2();
}
// Will not emit a warning: warning2 will be removed on the GCC constant folding
(void) ((sizeof(void *) == 7) ? warning2() : 0);
First of all, in order to use this warnings technique for your code, you should have wrappers for all your APIs:
library.c:
void __null_warning() { }
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
extern void __null_warning() __attribute__((warning("function called with a NULL argument")))
extern int __internal_do_stuff(int *arg);
// A function can be wrapped with an inlined function to make sure that the GCC will propagate the argument values
// inline qualification does not guarantee that the function will be inlined; use an attribute additionally
static inline int __attribute__((always_inline)) do_stuff(int *arg) {
if (arg == NULL) {
__null_warning();
}
return __internal_do_stuff(arg);
}
// OR a function can be wrapped with a macro with the GCC Compound Expression.
// https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html
#define do_stuff(arg) ({ \
if (arg == NULL) { \
__null_warning(); \
} \
__internal_do_stuff(arg); \
})
Suggested code will emit a warning on every do_stuff(0)
call. Unfortunately, this code will also emit warnings for the calls like do_stuff(non_constant_argument)
. In order to create more useful warnings you should consider to use additional GCC built-in functions from the https://gcc.gnu.org/onlinedocs/gcc/Object-Size-Checking.html and the https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html#Other-Builtins.
This builtins allowed us to emit the required information from the compiler:
#define do_stuff(arg) ({ \
if (__builtin_constant_p(arg) && arg == NULL) { \
__called_with_a_null_constant_warning(); \
} \
if (!__builtin_types_compatible_p(typeof(arg), int *)) { \
__called_with_an_int_pointer(); \
} \
if (__builtin_object_size(arg, 0) == -1) { \
__called_with_a_pointer_to_an_unknown_object_warning(); \
} \
if (__builtin_object_size(arg, 0) != -1 \
&& __builtin_object_size(arg, 0) < 20 ) { \
__called_with_a_pointer_to_an_object_smaller_then_20_bytes_warning(); \
} \
if (!__builtin_types_compatible_p(typeof(&((arg)[0])), typeof(arg))) { \
__called_with_an_array_argument_warning(); \
} \
__internal_do_stuff(arg); \
})
This technique can be adjusted for any purposes: the GNU Libc used this technique to find the memory bounds violations in the string functions when compiled with the -D_FORTIFY_SOURCE=1
, this technique can be used to warn about a non-recommended API usage (e.g. “in next versions calls with NULL argument will be forbidden”).
To use the __attribute__((warning("message")))
you should ensure that the attributed function will not be optimized from the code. This can be achieved in two ways:
You can place a definition of a warning
function to a library and call it from a library header:
library.c:
void library_warning() { }
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
extern void library_warning() __attribute__((warning("Warning message")));
extern int __internal_do_stuff(int *arg);
#define do_stuff(arg) ({ \
library_warning(); \
__internal_do_stuff(); \
})
This solution is used by the GNU Libc for the __warn_memset_zero_len
diagnostic function.
Or you can disable the optimization of a warning function directly in the header file with another GCC attribute:
library.c:
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() { }
extern int __internal_do_stuff(int *arg);
#define do_stuff(arg) ({ \
library_warning(); \
__internal_do_stuff(); \
})
For Inango code we have selected this solution.
With the code described above all your warnings will suddenly disappear once you install your library.h
header to the /usr/include
directory. This happens because the GCC suppresses all warnings that were found in the code of system headers. You can bring back your warnings in two ways:
If you use a function wrapper you can use the “artificial” GCC attribute:
library.c:
void library_warning() { }
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
extern void library_warning() __attribute__((warning("Warning message")));
extern int __internal_do_stuff(int *arg);
static inline void __attribute__ ((always_inline, artificial)) do_stuff() {
library_warning();
__internal_do_stuff();
}
This solution used by GNU Libc.
If you use a macro wrapper you should enable “-Wsystem-headers” warnings for your code:
library.c:
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() { }
extern int __internal_do_stuff(int *arg);
#pragma GCC diagnostic push
#pragma GCC diagnostic warning "-Wsystem-headers"
#define do_stuff(arg) ({ \
library_warning(); \
__internal_do_stuff(); \
})
#pragma GCC diagnostic pop
The solution with warnings will conflict with the usage of the “-Werror” compilation flag. As on December 2018 all the GCC versions will fail the compilation with an error if the __attribute__((warning("")))
code will be compiled with the “-Werror” flag. There is no way to change this behavior. The possible solutions to this issue:
You can change to use the __attribute__((warn_unused_result))
instead:
library.c:
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
// this pragma should be global to force moving this diagnostics from errors to warnings
#pragma GCC diagnostic warning "-Wunused-result"
#pragma GCC diagnostic push
#pragma GCC diagnostic warning "-Wsystem-headers"
static inline void __attribute__((warn_unused_result, optimize("O0"))) library_warning() { }
extern int __internal_do_stuff(int *arg);
#define do_stuff(arg) ({ \
(void)((integer_constant_expression_condition) ? library_warning() : 0); \
__internal_do_stuff(); \
})
#pragma GCC diagnostic pop
__attribute__((warning("")))
itself and use a new “-Wno-error=attribute-warning” that we submitted to the GCC as a result of the described work.
Finally, let’s combine all the described work together and show how the custom warnings can be created for a GCC-based code:
library.c:
int __internal_do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
#ifndef LIBRARY_H
#define LIBRARY_H
/* Interpret all the "warning" attributes in a code compiled with this library as warnings,
regardless of -Werror or -Wno-attribute-warning flags.
As on Dec 2018, this flag is supported only in GCC trunk */
#pragma GCC diagnostic warning "-Wattribute-warning"
/* Declare the internal functions that we will wrap */
extern int __internal_do_stuff(int *arg);
/* Create the warning functions with a specific messages.
Make sure that these functions would not be optimized out from the code.
Define the function as a static to not export the name from the resulting binaries. */
static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() { }
/* Print these warnings even if the library.h is installed to the system headers directory.
This behavior will be applied only to the library.h code.
Other system headers will not print the warnings */
#pragma GCC diagnostic push
#pragma GCC diagnostic warning "-Wsystem-headers"
/* Wrap all the functions in the macro definitions with the GCC Compound expressions inside.
Make sure that all the `if' conditions are integer constant expressions. Otherwise,
there is no guarantees that the compiler will be able to optimize out the failed checks. */
#define do_stuff(arg) ({ \
if (integer_constant_expression_condition) { \
library_warning(); \
} \
__internal_do_stuff(); \
})
#pragma GCC diagnostic pop
#endif /* LIBRARY_H */
There is an alternate solution for the described issue supported in the Clang compiler. In order to emit compile-time warnings in Clang, you can just specify the attribute on the relevant function:
library.h:
int do_stuff(int *arg) {
// do usefull stuff
return 0;
}
library.h:
int do_stuff(int *arg) __attribute__((diagnose_if(arg == NULL, "arg is NULL", "warning")));
int do_stuff(int *arg)
__attribute__((diagnose_if(__builtin_object_size(arg, 0) == -1, "arg of unknown size", "warning")));
With the Clang __attribute__((diagnose_if(...)))
there is no need in additional wrappers (you still should create a wrapper if you want to extract the information about the original type of the argument) and new warnings can be simply added with additional declaration lines.
But the GNU GCC is still the main compiler in GNU/Linux systems and we think that the proposed article will be usable for all the GCC users.
The BitBake build utility, adapted by the Yocto Project, allows for flexible configuration of source mirrors for all the remote files required during the build of Yocto distribution. This functionality allows to speed up the distribution build, as well as organize a local backup with sources of all the packages used.
However, this mechanism is rather poorly documented, and we are going to fix this situation in the current article.
The BitBake mirrors mechanism is supported at least in three places:
1. PREMIRRORS
are the primary mirrors that will be used before the upstream URI from SRC_URI
. It can be used to provide quick mirrors to speed-up the sources downloading.
2. MIRRORS
are the additional mirrors that are used when an upstream URI from SRC_URI
is not accessible. Creating MIRRORS
is a good idea for long-living distributions, when a distribution can out-live the upstream sources of a used software.
3. SSTATE_MIRRORS
are the mirrors for a prebuilt cache data objects. SSTATE_MIRRORS
can be used to share the pre-built objects from a CI builds.
The Yocto Project additionally provides the support for the next mechanisms:
1. own-mirrors.bbclass is a standard way to provide a single pre-mirror for all the supported fetchers:
INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "http://example.com/my-source-mirror"
2. BB_GENERATE_MIRROR_TARBALLS
variable can be set to "1"
to allow reusing DL_DIR
from the current build as a mirror for other builds. Without a BB_GENERATE_MIRROR_TARBALLS
variable the resulting DL_DIR would not provide suitable mirrors for sources fetched from VCS repositories.
3. BB_FETCH_PREMIRRORONLY
and SSTATE_MIRROR_ALLOW_NETWORK
variables can be used to configure BitBake to download sources and build artifacts only from the configured mirrors. These variables will be usable for all the BB_NO_NETWORK
configurations.
All things described in the previous part are correctly documented and can be found in the official documentation. But a syntax for mirror rules itself is poorly documented. An official documentation suggests only three examples how mirror rules can be created:
1. In the example[1] for a PREMIRRORS
variable we can see that we should use .*/.*
regexp to match all URI for a specific fetcher:
PREMIRRORS_prepend = "\
git://.*/.* http://www.Yoctoproject.org/sources/ \n \
ftp://.*/.* http://www.Yoctoproject.org/sources/ \n \
http://.*/.* http://www.Yoctoproject.org/sources/ \n \
https://.*/.* http://www.Yoctoproject.org/sources/ \n"
2. In the example[2] for a SSTATE_MIRRORS
variable we can find that there is also a support for a PATH
string in the matching:
SSTATE_MIRRORS ?= "\
file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
file://.* file:///some-local-dir/sstate/PATH"
3. In the same SSTATE_MIRROS
variable from example[2] we can additionally find an information that a mirror syntax supports a more sophisticated regex usage:
SSTATE_MIRRORS ?= "file://universal-4.9/(.*) http://server_url_sstate_path/universal-4.8/\1 \n"
But there is no full description of the mirror matching mechanism. We offer the following as a way of filling this gap.
All mirrors are defined as a “pattern replacement” pair. The BitBake attempts to match a source URI against each “pattern” and on a successful match the BitBake generates a new mirror URI based on an original URI, by using the “pattern” and “replacement” definitions.
Each URI, including the “pattern” and “replacement” strings, is split into six components before matching as follows:
scheme://user:password@hostname/path;parameters
.
In an upstream URI this parts is understood as strings, in a “pattern” this parts is interpreted as a Python re
regexps (parameters
is the only field that is not regexp); in a “replacement” this parts is interpreted as regexp replacements in terms of a Python re.sub
function.
All parts from an upstream URI are matched against regexps from the corresponding part of a “pattern” with the next additional rules:
1. The BitBake always adds a $
sign to the end of a scheme
regex. It was done to ensure that the “http” scheme
would not match the “https” string.
2. If a scheme
is a “file” then the hostname
is assumed to be empty. Otherwise the hostname
is a text between “://” and next “/” characters.
3. parameters
are interpreted as a list of “param=value” pairs. For each such pair specified in a “pattern” there should be a full match in an original SRC_URI
. It is the only field that is matched with a string equality instead of a regexp matching.
NOTE: A parameters matching was broken up to Yocto 2.6 release. This issue was found and fixed as a part of work on this document.
If a URI was successfully matched, then the BitBake creates a new mirror URI from an upstream URI using a “replacement” string. The BitBake uses the next rules to make a replacement:
1. A replacement is made for each part of the URI separately.
2. Before replacement is performed, we replace the next substrings in a corresponding part of a “replacement” string:
NOTE: In the parameters
list this replacement is made only in a value
part of the param=value
pairs.
3. Each part except the parameters
is replaced with re.sub(part_from_pattern, part_from_replacement, part_from_original_uri, 1)
Python command. This means that the BitBake will replace only a first match from a current part and that you can use a Python regex replacement syntax including "\1"
syntax to insert parts of a matched URI to a result.
4. If an original URI scheme
differs from a “replacement” scheme
then the original parameters
will be wiped off.
5. The parameters
from a “replacement” should be added to a resulting URI. If there were parameters with the same names in an original URI then the value for these parameters will be overridden.
6. Finally, the BitBake checks that a path
part upon replacement is finished with a suggested “basename”. If a resulting path
ends with any other string then a resulting path
will be concatenated with a “/basename
” string. A suggested “basename” is created according to the next rules:
1. If a scheme was not changed then the “basename” is just a basename of the path
from an original URI.
2. If a scheme
was changed and an original URI points to a single file or a tarball with sources, then the same basename
will be used.
3. If a scheme
was changed and an original URI points to some kind of a repository then the mirrortarball
name will be used. This name is a fetcher specific. e.g. for a git repository a tarball name will be "git2_hostname.path.to.repo.git.tar.gz"
.
NOTE: The last step means that it is impossible to use the "file:///some/path/b.tar.gz"
as a mirror path for the "http://other/path/a.tar.gz"
, but you still can use the "file:///some/path/a.tar.gz"
or the "file:///some/path/prefix_a.tar.gz"
.
Let’s check how the next example will be processed:
SRC_URI = "git://git.invalid.infradead.org/foo/mtd-utils.git;branch=master;tag=1234567890123456789012345678901234567890"
PREMIRRORS = "git://.*/.*;branch=master git://somewhere.org/somedir/MIRRORNAME;protocol=http;branch=master_backup \n"
First of all, we will split all the three URI’s into the next parts:
upstream:
scheme = "git"
user = ""
password = ""
host = "git.invalid.infradead.org"
path = "/foo/mtd-utils.git"
parameters = {"branch": "master", "tag": "1234567890123456789012345678901234567890"}
pattern:
scheme = "git$"
user = ""
password = ""
host = ".*"
path = "/.*"
parameters = {"branch": "master"}
replacement:
scheme = "git"
user = ""
password = ""
host = "somewhere.org"
path = "/somedir/MIRRORNAME"
parameters = {"branch": "master_backup", "protocol": "http"}
On the next step the BitBake will match the “upstream” URI parts against the corresponding parts of a “pattern” with a re.match(pattern.part, upstream.part)
command for all the parts except the parameters. For the “parameters” the BitBake will check that the value for a “branch” parameter in an “upstrem” URI and a “pattern” URI are equal.
When these checks pass, the BitBake will start a replacement process. In each part of the “replacement” BitBake will make the replacements for the special strings:
replacement:
scheme = "git"
user = ""
password = ""
host = "somewhere.org"
path = "/somedir/git.invalid.infradead.org.foo.mtd-utils.git"
parameters = {"protocol": "http", "branch": "master_backup"}
Then the BitBake will make the actual replacements with a re.sub(pattern.part, replacement.part, upstream.part, 1)
command for all parts except the parameters
. This means that if you want to replace the string completely, the full pattern should be a “.*”, and not “”. For the parameters
the new keys will be added to the list and the old values will be replaced. As the result we will get as follows:
result:
scheme = "git"
user = ""
password = ""
host = "somewhere.org"
path = "/somedir/git.invalid.infradead.org.foo.mtd-utils.git"
parameters = {"branch": "master_backup", "protocol": "http", "tag": "1234567890123456789012345678901234567890"}
On the last step the BitBake will check that the result.path
is finished with a basename
of an upstream.path
. Since the result.scheme
and the upstream.scheme
are the same, the basename
will be defined as an mtd-utils.git
(if a scheme will be different, then the basename
would be defined to a "git2_git.invalid.infradead.org.foo.mtd-utils.git.tar.gz"
).
When the last check passes, the BitBake will combine a result to a new mirror URI that will be used as an alternative source for the files:
git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;branch=master_backup;protocol=http;tag=1234567890123456789012345678901234567890
.
You can find additional replacements examples in the BitBake unit tests code[3].
[1]: Simple example
[2]: Extended example
[3]: Additional replacement examples
[4]: BitBake mirror mechanism implementation