How to translate gcc CLI arguments to correponding clang arguments?

[guest271314]

Problem

I successfully compile a C file to an executable using this

gcc permutations.c -O3 -I./build_release/lib/config -I./hermes-static_h/include -DNDEBUG -g -fno-strict-aliasing -fno-strict-overflow -L./build_release/lib -L./build_release/jsi -L./build_release/tools/shermes -lshermes_console -Wl,-rpath ./build_release/lib -Wl,-rpath ./build_release/jsi -Wl,-rpath ./build_release/tools/shermes -lm -lhermesvm -o permutations

When I try to compile with LLVM's clang or WASI-SDK's clang I get an error for the -Wl,-rpath option

wasm-ld: error: unknown argument: -rpath
wasm-ld: error: unknown argument: -rpath
wasm-ld: error: unknown argument: -rpath
wasm-ld: error: unable to find library -lshermes_console
wasm-ld: error: cannot open ./build_release/lib: Is a directory
wasm-ld: error: cannot open ./build_release/jsi: Is a directory
wasm-ld: error: cannot open ./build_release/tools/shermes: Is a directory
wasm-ld: error: unable to find library -lhermesvm
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Solution

clang compilation corresponding to gcc compilation.

Additional Context

I'm trying to compile with clang so I can target WASM.

[tmikov]

Clang, not GCC, is already the primary target of Shermes. With that said, they have compatible command line options.

You are seeing errors because compiling for Wasm is not exactly the same as compiling Elf binaries. Specifically, -rpath is an Elf specific option relevant to dynamic linking, so naturally it is not recognized by the Wasm toolchain, which supports neither.

[guest271314]

I'm thinking more of the source code is C, therefore it must be capable of cross-compilation, by some means.

JavaScript to C to WASM, C++, Rust, et al.

The reason is I was testing JavaScript to C, C++ code bases. ts2c, jsxx, TypeScriptCxx, compilets.

Then there's a random Web site that says it's "artificial intelligence" technology converts JavaScript to C, and does.

Therefore I am looking into to JavaScript engines and runtimes that could possibly achieve that, even if not within the scope of the repository. shermes has -emit-c, to start with. See what can be done with shermes to achieve my own requirement. Ask questions.

It can't be that there is no FOSS that can achieve that. That's my research interest for this project. Thanks.

[tmikov]

Shermes can be used for cross compilation or to compile JavaScript to Wasm. Both work, but may require some manual steps.

[guest271314]

I suspected that. Manual steps work. I have llvm-project's clang and WASI-SDK clang, build directory of static hermes, and hermes-static_h in the same directory.

[guest271314]

To provide a roadmap of what I'm doing here - without using a random Web site that converts JavaScript to C - consider starting out with some JavaScript.

test.js

// https://stackoverflow.com/a/34238979
function array_nth_permutation(input, n) {
    n = Number(n);
    let a = Array.from({ length: Number(input) }, (_, i) => i);
    let lex = n;
    let b = Array(); // copy of the set a.slice()
    for (let x = 0; x < a.length; x++) {
      b[x] = a[x];
    }
    let len = a.length; // length of the set
    const res = Array(); // return value, undefined
    let i = 1;
    let f = 1;

    // compute f = factorial(len)
    for (; i <= len; i++) {
      f *= i;
    }

    let fac = f;
    // if the permutation number is within range
    if (n >= 0 && n < f) {
      // start with the empty set, loop for len elements
      // let result_len = 0;
      for (; len > 0; len--) {
        // determine the next element:
        // there are f/len subsets for each possible element,
        f /= len;
        // a simple division gives the leading element index
        i = (n - n % f) / f; // Math.floor(n / f);
        // alternately: i = (n - n % f) / f;
        // res[(result_len)++] = b[i];
        // for (let j = i; j < len; j++) {
        //   b[j] = b[j + 1]; // shift elements left
        // }
        res.push(b.splice(i, 1)[0]);
        // reduce n for the remaining subset:
        // compute the remainder of the above division
        n %= f;
        // extract the i-th element from b and push it at the end of res
      }
      return `${lex} of ${fac - 1} (0-indexed, factorial ${fac}) => ${
        JSON.stringify(res)
      }`;
    } else {
      if (n === 0) {
        return `${JSON.stringify(res)}\n`;
      }
      return `${n} >= 0 && ${n} < ${f}: ${n >= 0 && n < f}`;
    }
  }
  
console.log(array_nth_permutation("4", "5"));

Using a random Web site https://products.codeporting.app/convert/ai/js-to-c/ we can get this C output

// This code snippet is a JavaScript to C translation of a function that calculates the nth permutation of an array.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

char* array_nth_permutation(const char* input, const char* n_str) {
    int n = atoi(n_str);
    int length = atoi(input);
    int* array = (int*)malloc(length * sizeof(int));
    int* b = (int*)malloc(length * sizeof(int)); // copy of the set array
    int lex = n;
    int result_length = 0;
    int factorial = 1;
    
    for (int i = 0; i < length; i++) {
        array[i] = i;
        b[i] = i;
    }
    
    // compute factorial
    for (int i = 1; i <= length; i++) {
        factorial *= i;
    }
    
    int fac = factorial;
    int* res = (int*)malloc(length * sizeof(int)); // return value
    
    // if the permutation number is within range
    if (n >= 0 && n < factorial) {
        // start with the empty set, loop for length elements
        for (; length > 0; length--) {
            factorial /= length;
            // a simple division gives the leading element index
            int i = (n - n % factorial) / factorial; 
            res[result_length++] = b[i];
            // reduce n for the remaining subset:
            n %= factorial;
            // extract the i-th element from b and shift elements left
            for (int j = i; j < length - 1; j++) {
                b[j] = b[j + 1]; // shift elements left
            }
        }
        
        // Prepare the result string
        char* result = (char*)malloc(256); // allocate enough space for the result
        sprintf(result, "%d of %d (0-indexed, factorial %d) => [", lex, fac - 1, fac);
        for (int i = 0; i < result_length; i++) {
            char buffer[10];
            sprintf(buffer, "%d", res[i]);
            strcat(result, buffer);
            if (i < result_length - 1) {
                strcat(result, ", ");
            }
        }
        strcat(result, "]");
        
        free(array);
        free(b);
        free(res);
        return result;
    } else {
        char* result = (char*)malloc(100);
        if (n == 0) {
            sprintf(result, "[]\n");
        } else {
            sprintf(result, "%d >= 0 && %d < %d: %d", n, n, factorial, (n >= 0 && n < factorial));
        }
        
        free(array);
        free(b);
        free(res);
        return result;
    }
}

int main() {
    char* result = array_nth_permutation("4", "5");
    printf("%s\n", result);
    free(result);
    return 0;
}

Modify the main function to pass and get arguments

int main(int argc, char *argv[]) {
    char* result = array_nth_permutation(argv[1], argv[2]);
    printf("%s\n", result);
    free(result);
    return 0;
}

Now we can do JavaScript <=> WASM | C | Native executable <=> JavaScript.

Build a native executable with clang

llvm-project/build/bin/clang test.c -o test

./test 9 362879
362879 of 362879 (0-indexed, factorial 362880) => [8, 7, 6, 5, 4, 3, 2, 1, 0]

Build WASM using WASI-SDK

wasi-sdk/bin/clang test.c --sysroot=wasi-sdk/share/wasi-sysroot -o test.wasm

wasmtime run test.wasm 9 362879
362879 of 362879 (0-indexed, factorial 362880) => [8, 7, 6, 5, 4, 3, 2, 1, 0]

Convert WASM back to JavaScript

binaryen/bin/wasm2js test.wasm -o test.wasm.js

Modify the resulting asm.js format slightly, using a minimal WASI runtime that does not define filesystem access https://gitlab.com/-/snippets/4782260

// import * as wasi_snapshot_preview1 from 'wasi_snapshot_preview1';

import process from "node:process";
import { WASI } from "./wasi-minimal.js";
import * as fs from "node:fs";
// ...
var memasmFunc = new ArrayBuffer(0);

var args = await (async() => {
  var xargs = process.argv.slice(-2);
  var bool = xargs.map(Number).some((n) => Number.isNaN(n));
  if (bool) {
    for await (const data of process.stdin) {
      var ret = new TextDecoder().decode(data).trim().split(" ");
      ret.unshift(" ");
      return ret;
    }
  }
  xargs.unshift(" ");
  return xargs;
})(); 

let wasi = new WASI({
  env: {},
  args,
  fds: [
    {
      type: 2,
      handle: fs,
    },
    {
      type: 2,
      handle: fs,
    },
    {
      type: 2,
      handle: fs,
    },
  ],
});

var retasmFunc = asmFunc({
  "wasi_snapshot_preview1": wasi.exports,
  memory: { buffer: memasmFunc }
});
export var memory = retasmFunc.memory;
wasi.memory = memory;
export var _start = retasmFunc._start;
_start();

echo '9 362879' | node --no-warnings test.wasm.js
362879 of 362879 (0-indexed, factorial 362880) => [8, 7, 6, 5, 4, 3, 2, 1, 0]

My thinking is that one of these modern JavaScript engine/runtime/compiler projects is capable of achieving true cross-compilation to/from JavaScript to C, WASM, native executable, and back to JavaScript from WASM, and possibly from C - without using an unobservable box such as

using AI involves utilizing natural language processing (NLP) techniques and machine learning algorithms to analyze and understand source code

that is, without using "artificial intelligence" or "machine learning"; rather using FOSS.

[tmikov]

To my knowledge, with the exception of GraalVM (which has a somewhat different focus), Static Hermes is the only production quality project that is able to compile JavaScript to native code ahead-of-time. SH is not a research or a hobby project, it is a production[1] quality compiler that can compile normal JS ES6 code to native and pass ECMAScript compliance tests (Test262).

Please, keep in mind that translating JavaScript to C is not really possible, at least not in any meaningful way and certainly not at scale. JS semantics are too complex and dynamic for that.

Static Hermes compiles to native, using C as a kind of high level assembly language (this is not a new technique). The generated C certainly looks nothing like the input JS. This is a shortcut, in lieu of generating LLVM IR directly, to make development easier.

It is definitely possible to use Static Hermes to compile JavaScript to Wasm, since for our purposes Wasm is just a LLVM backend. In a sense, Static Hermes was designed for this. See these tweets for some interesting results:

• https://x.com/tmikov/status/1837600409386734037
• https://x.com/tmikov/status/1706138872412074204

[1] While typed JS is still experimental, untyped JS is well supported.

[guest271314]

In the instructions I rely on the natively compiled shermes.

Not here

node ./build-wasm/bin/hermes.js hello.js

I'm just curious why the hermes.js compiled runtime executes the code, while the shermes.js compiled runtime doesn't.

I'm far more interested in building the WASM standalone, without the JavaScript "glue" that Emscripten produces, so the same WASM can be executed by wasmtime, wasmer, etc. Such as Bytecode Alliance's Javy can. I've been working on that since I first reproduced what you published.

[tmikov]

In the instructions I rely on the natively compiled shermes.

Not here

node ./build-wasm/bin/hermes.js hello.js

I'm just curious why the hermes.js compiled runtime executes the code, while the shermes.js compiled runtime doesn't.

shermes is not a runtime, it is a compiler and a compiler driver. It compiles, it does not execute. It has an --exec flag, but that simply invokes a compiled native binary.

[guest271314]

Got it. So executing shermes with-exec roughly translates to the following?

  build-host/bin/shermes ../shermes-permuations.c -g \
  -I./build-host/lib/config -I../hermes/include -DNDEBUG -g \ 
  -L./build-host/lib -L./build-host/jsi \
  -L./build-host/tools/shermes -lshermes_console \
  -Wl,-rpath ./build-host/lib \
  -Wl,-rpath ./build-host/jsi \
  -Wl,-rpath ./build-host/tools/shermes \
  -lm -lhermesvm \
  -O3 -flto -nostdlib -Wl,--no-entry -Wl,--export-all -Wl,--lto-O3 \
  -o shermes-permutations && ./shermes-permutations

[tmikov]

Yes, conceptually. For increased performance --exec actually creates a shared library (.so) and then loads it.

[guest271314]

FYI the shermes built with the current WASM build instructions doesn't work the same as shermes I had previously built.

build-host/bin/shermes hello.js  -o hello1
/usr/bin/ld: cannot find -lshermes_console: No such file or directory
collect2: error: ld returned 1 exit status
/usr/bin/cc: execution failed

[tmikov]

Converting to discussion.