How to point CodeQL to source code for 3rd party/precompiled libraries?

[adriaanjacobs]

[C/C++]
For calls into 3rd party libraries for which no source code is available, it seems like CodeQL assumes the return value is tainted if any of the arguments are tainted (which is a good compromise I think). However, for calls into glibc for example, I could in theory provide the source code for every call and let CodeQL do a more extensive/detailed analysis on very common calls, which would increase the precision of the analysis I'm doing greatly. Is there a way to point CodeQL towards a specific source repository for specific libraries that are linked into the main code when building?

I'm trying to get it to work by setting up a database for CodeQL manually with glibc checked out in the repository and linking against that, but since glibc is still built before the rest (I'm testing this with coreutils), I don't have a lot of hope.

I could also modify the coreutils buildsystem to build glibc as part of it and then hope CodeQL picks up on it when it generates the database and compiles the repo?

What would you suggest in this case?

[p0]

Your best bet if you really want to go down this route is indeed to build all the dependencies you want to include in the database as part of the database creation. CodeQL will pick up the compiler invocations that happen while the build command/script you provide runs, and reflect all processed source in the database (with appropriate compiler settings, target platforms, include paths, and so on). Without that information, we can't create a sufficiently accurate model of C/C++ code.

Having said that, our default assumption about external calls is actually that they do not propagate taint -- our CodeQL libraries provide explicit models for many common APIs that are relevant to data flow analysis and taint tracking. It may be that you can achieve what you're after through explicit modelling of relevant APIs, if you can't build them as part of the same script.

[adriaanjacobs]

Oh okay, well I've only seen examples of libc functions, and they've always propagated taint (as far as I can tell) in a way that doesn't really "make sense" if you know what the function does (which CodeQL of course doesn't).

For example, the strtol is a common offender for me:

long strtol(const char *nptr, char **endptr, int base);

When the 2nd parameter endptr is tainted, the return value is considered tainted as well (which doesn't make sense to me since the 2nd parameter is really an 'output' parameter).

I'll see if I can find more examples later, I can't remember more.

[adriaanjacobs]

So then I'm assuming an explicit model was created for the libc API since it seems to propagate taint (at least sometimes)? I'd love to contribute to that (if possible) with things I stumble upon during my work on coreutils

[p0]

Yes, that's right -- and we're definitely always looking for further contributions.

In the case of strtol, it is modelled, alongside many others, as a PureStringFunction. The key method that makes taint propagate through it is hasTaintFlow.

[adriaanjacobs]

Actually @p0 , I was wondering where I could find some specifications about the dataflow analysis and taint tracking suite you provide. AFAIK it's an unsolved problem in its generality, so I was wondering where I could find other assumptions CodeQL makes when determining what taints what, like how it handles recursive calls and some more information about why external calls are assumed to introduce no taint. Is that documented somewhere?

[p0]

Hmm, I think the closest high-level docs on that we have are here. In general, the details are continuously evolving: We're improving precision and completeness of the inter-procedural data flow analysis, and we're adding models for additional library functions that allow things to propagate further, as a regular part of our work. That makes it a little hard to maintain a "specification" of what the analysis does, other than the code itself, of course.

Our general guiding principle is that we want the analysis to be "useful" -- so by default we tend to make low-noise assumptions (like "unmodelled functions do not propagate data"), and when we introduce complex features like context-sensitivity or path-sensitivity we always try to balance the precision gains with performance on large code bases.