Musl-gcc in rust based Netlify Functions

morenol · April 7, 2022, 10:42pm

Build problems? Link or paste the FULL build log & build settings screenshot
I am building a rust based netlify function that has as one of its dependencies the openssl-sys.

When I build that project locally, I don’t have any problems, but when deploying that to netlify I get the following output:

6:17:34 PM: ────────────────────────────────────────────────────────────────
6:17:34 PM:   2. Functions bundling                                         
6:17:34 PM: ────────────────────────────────────────────────────────────────
6:17:34 PM: 
6:17:34 PM: Packaging Functions from functions directory:
6:17:34 PM:  - function-name/src/main.rs
6:17:34 PM: 
6:19:01 PM: Could not compile Rust function function-name:
6:19:01 PM: 
6:19:01 PM: ────────────────────────────────────────────────────────────────
6:19:01 PM:   Bundling of Function "function-name" failed                          
6:19:01 PM: ────────────────────────────────────────────────────────────────
6:19:01 PM: 
6:19:01 PM:   Error message
6:19:01 PM:   Command failed with exit code 101: cargo build --target x86_64-unknown-linux-musl --release (https://ntl.fyi/exit-code-101)

...

6:19:01 PM:   error: failed to run custom build command for `openssl-sys v0.9.72`
6:19:01 PM: 
6:19:01 PM:   Caused by:
6:19:01 PM:     process didn't exit successfully: `/opt/build/repo/.netlify/rust-functions-cache/fluvio/release/build/openssl-sys-94e8951be1549fff/build-script-main` (exit status: 101)
    --- stdout
6:19:01 PM:     cargo:rustc-cfg=const_fn
6:19:01 PM:     cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_NO_VENDOR
6:19:01 PM:     X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_NO_VENDOR unset
6:19:01 PM:     cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
6:19:01 PM:     OPENSSL_NO_VENDOR unset
6:19:01 PM:     CC_x86_64-unknown-linux-musl = None
6:19:01 PM:     CC_x86_64_unknown_linux_musl = None
6:19:01 PM:     TARGET_CC = None
6:19:01 PM:     CC = None
6:19:01 PM:     CROSS_COMPILE = None
6:19:01 PM:     CFLAGS_x86_64-unknown-linux-musl = None
6:19:01 PM:     CFLAGS_x86_64_unknown_linux_musl = None
6:19:01 PM:     TARGET_CFLAGS = None
6:19:01 PM:     CFLAGS = None
6:19:01 PM:     CRATE_CC_NO_DEFAULTS = None
6:19:01 PM:     DEBUG = Some("false")
6:19:01 PM:     CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
6:19:01 PM:     running "perl" "./Configure" "--prefix=/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/install" "no-dso" "no-shared" "no-ssl3" "no-unit-test" "no-comp" "no-zlib" "no-zlib-dynamic" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-engine" "no-async" "linux-x86_64" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-DOPENSSL_NO_SECURE_MEMORY"
6:19:01 PM:     Configuring OpenSSL version 1.1.1n (0x101010efL) for linux-x86_64
6:19:01 PM:     Using os-specific seed configuration
6:19:01 PM:     Creating configdata.pm
6:19:01 PM:     Creating Makefile
6:19:01 PM: 
6:19:01 PM:     **********************************************************************
6:19:01 PM:     ***                                                                ***
6:19:01 PM:     ***   OpenSSL has been successfully configured                     ***
6:19:01 PM:     ***                                                                ***
6:19:01 PM:     ***   If you encounter a problem while building, please open an    ***
6:19:01 PM:     ***   issue on GitHub <https://github.com/openssl/openssl/issues>  ***
6:19:01 PM:     ***   and include the output from the following command:           ***
6:19:01 PM:     ***                                                                ***
6:19:01 PM:     ***       perl configdata.pm --dump                                ***
6:19:01 PM:     ***                                                                ***
6:19:01 PM:     ***   (If you are new to OpenSSL, you might want to consult the    ***
6:19:01 PM:     ***   'Troubleshooting' section in the INSTALL file first)         ***
6:19:01 PM:     ***                                                                ***
6:19:01 PM:     **********************************************************************
6:19:01 PM:     running "make" "depend"
6:19:01 PM:     running "make" "build_libs"
6:19:01 PM:     /usr/bin/perl "-I." -Mconfigdata "util/dofile.pl" \
6:19:01 PM:         "-oMakefile" include/crypto/bn_conf.h.in > include/crypto/bn_conf.h
6:19:01 PM:     /usr/bin/perl "-I." -Mconfigdata "util/dofile.pl" \
6:19:01 PM:         "-oMakefile" include/crypto/dso_conf.h.in > include/crypto/dso_conf.h
6:19:01 PM:     /usr/bin/perl "-I." -Mconfigdata "util/dofile.pl" \
6:19:01 PM:         "-oMakefile" include/openssl/opensslconf.h.in > include/openssl/opensslconf.h
6:19:01 PM:     make depend && make _build_libs
6:19:01 PM:     make[1]: Entering directory '/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/build/src'
6:19:01 PM:     make[1]: Leaving directory '/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/build/src'
6:19:01 PM:     make[1]: Entering directory '/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/build/src'
6:19:01 PM:     musl-gcc  -I. -Iinclude -fPIC -pthread -m64 -Wall -O3 -O2 -ffunction-sections -fdata-sections -fPIC -m64 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/install/ssl\"" -DENGINESDIR="\"/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/install/lib/engines-1.1\"" -DNDEBUG -DOPENSSL_NO_SECURE_MEMORY  -c -o apps/app_rand.o apps/app_rand.c
6:19:01 PM:     make[1]: Leaving directory '/opt/build/repo/.netlify/rust-functions-cache/fluvio/x86_64-unknown-linux-musl/release/build/openssl-sys-353a1c004da0cfab/out/openssl-build/build/src'
6:19:01 PM: 
6:19:01 PM:     --- stderr
6:19:01 PM:     /bin/sh: 1: musl-gcc: not found
6:19:01 PM:     make[1]: *** [Makefile:671: apps/app_rand.o] Error 127
6:19:01 PM:     make: *** [Makefile:175: build_libs] Error 2
6:19:01 PM:     thread 'main' panicked at '

6:19:01 PM:     Error building OpenSSL:
6:19:01 PM:         Command: "make" "build_libs"
6:19:01 PM:         Exit status: exit status: 2
6:19:01 PM: 
6:19:01 PM:         ', /opt/buildhome/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-src-111.18.0+1.1.1n/src/lib.rs:479:13
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
6:19:01 PM:   warning: build failed, waiting for other jobs to finish...
6:19:01 PM:   error: build failed

Is it possible to have installed musl-gcc in the host instance that runs the functions?

github.com/awslabs/aws-lambda-rust-runtime

difficulties using aws-sdk-rust with aws-lambda-rust-runtime

opened 11:55PM - 09 Sep 21 UTC

closed 12:16PM - 13 Sep 21 UTC

Franck-Q

Hello, I'm trying to compile a rust lambda function using the aws-sdk-rust but h…aving issues with it. Here's some info: rust code: ```use aws_sdk_dynamodb::{Client, Error}; use lambda::{handler_fn, Context}; use serde_json::Value; type Error = Box<dyn std::error::Error + Sync + Send + 'static>; #[tokio::main] async fn main() -> Result<(), Error> { lambda::run(handler_fn(handler)).await?; Ok(()) } async fn handler(event: Value, _: Context) -> Result<Value, Error> { let shared_config = aws_config::load_from_env().await; let client = Client::new(&shared_config); let req = client.list_tables().limit(10); let resp = req.send().await?; println!("Current DynamoDB tables: {:?}", resp.table_names); Ok(()) } ``` Cargo.toml: ``` [package] name = "lambda_sdk" version = "0.1.0" edition = "2018" [dependencies] lambda = { git = "https://github.com/awslabs/aws-lambda-rust-runtime/", branch = "master"} serde_json = "1.0" aws-config = { git = "https://github.com/awslabs/aws-sdk-rust", tag = "v0.0.17-alpha", package = "aws-config" } aws-sdk-dynamodb = { git = "https://github.com/awslabs/aws-sdk-rust", tag = "v0.0.17-alpha", package = "aws-sdk-dynamodb" } tokio = { version = "1", features = ["full"] } ``` compilation (all worked before changing the lambda function to add the SDK): 1) cross-compilation cargo build --release --target x86_64-unknown-linux-musl 2) docker linux rustup target add x86_64-unknown-linux-musl && cargo build --release --target x86_64-unknown-linux-musl 3) cdk synth myStack ``` import * as cdk from '@aws-cdk/core'; import * as lambda from '@aws-cdk/aws-lambda'; export class LookupIAMRoleStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) { super(scope, id, props); const target = 'x86_64-unknown-linux-musl'; new lambda.Function(this, 'lambdaSDK', { code: lambda.Code.fromAsset('lambda/lambda_sdk', { bundling: { command: [ 'bash', '-c', `rustup target add ${target} && cargo build --release --target ${target} && cp target/${target}/release/lambda_sdk /asset-output/bootstrap` ], image: cdk.DockerImage.fromRegistry('rust:1.55-slim') } }), functionName: 'TestLambdaSDK', handler: 'main', runtime: lambda.Runtime.PROVIDED_AL2 // defines an AWS Lambda resource }); } } ``` all generate the exact same error message (with different path obviously): ```.... Compiling syn-mid v0.5.0 error: failed to run custom build command for `ring v0.16.20` Caused by: process didn't exit successfully: `/mnt/target/release/build/ring-e3ac515dcadd1e70/build-script-build` (exit status: 101) --- stdout OPT_LEVEL = Some("3") TARGET = Some("x86_64-unknown-linux-musl") HOST = Some("x86_64-unknown-linux-gnu") CC_x86_64-unknown-linux-musl = None CC_x86_64_unknown_linux_musl = None TARGET_CC = None CC = None CROSS_COMPILE = None CFLAGS_x86_64-unknown-linux-musl = None CFLAGS_x86_64_unknown_linux_musl = None TARGET_CFLAGS = None CFLAGS = None CRATE_CC_NO_DEFAULTS = None DEBUG = Some("false") CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2") --- stderr running "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/mnt/target/x86_64-unknown-linux-musl/release/build/ring-d2b907aa8e6446d7/out/aesni-x86_64-elf.o" "/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S" thread 'main' panicked at 'failed to execute ["musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/mnt/target/x86_64-unknown-linux-musl/release/build/ring-d2b907aa8e6446d7/out/aesni-x86_64-elf.o" "/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"]: No such file or directory (os error 2)', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace```

Thanks

coelmay · April 7, 2022, 10:50pm

Hey @morenol

As best my knowledge, and as per Functions Documentation Rust is not (currently) a supported functions language (on Netlify)—only JS, TS, and Go are.

morenol · April 7, 2022, 11:27pm

There is some functionality that is experimental

morenol · April 7, 2022, 11:28pm

In any case, is it possible to install custom OS packages for the build process?

hrishikesh · April 10, 2022, 12:54pm

If there’s a brew package for that, then sure: