Figure out ways to resolve function pointer casting in wasm

[qianxichen233]

In python (and probably some other softwares), it is using function pointer casting to store the python method function pointers. This sometimes forcely changed the signature of the function. One example is dict_keys function, which has the signature of PyObject *dict_keys(PyDictObject *mp). It is forcely casted to PyObject *(*PyCFunction)(PyObject *, PyObject *) in python and python invokes the dict_keys in form of (*meth)(self, NULL). This causes wasm to raise function type mismatch error.

Calling a function pointer with extra argument is an undefined behavior according to c specification, but it looks like many developer does not care about this as long as this looks like working fine. But apparently wasm will not allow this behavior so we are facing some issues here. There is an article that also discussed exactly the same issue here. So looks like we currently have two options here:

1. Use emulated function pointer casting. This would basically rewrite every occurence of indirect_call instruction by making it always calling a function that takes 60 arguments. However, according to the article, this method could introduce a huge overhead to the wasm module
2. Another method would be just modifying the source code and fixing the undefined behavior here. This could be manually or automatically. Though having a general-purpose tool automatically fixing the function pointer casting would probably be pretty hard.

[JustinCappos]

How does WASM handle this normally? I can't believe we're the first to run into this.

[rennergade]

How does WASM handle this normally? I can't believe we're the first to run into this.

They go over that in the article link Qianxi provided. Emscripten deals with it with option 1.

[rennergade]

I've posted on Zulip asking the wasmtime folks if they deal with this in any explicit way.

[rennergade]

https://bytecodealliance.zulipchat.com/#narrow/channel/206238-general/topic/Function.20pointer.20casting

It seems like we can use fpcast-emu with binaryen to deal with this, though it comes with the code bloat that Qianxi described.

[rennergade]

This also might be useful: https://bytecodealliance.zulipchat.com/#narrow/channel/206238-general/topic/Function.20pointer.20casting/near/492409381