Now in experimental alpha: homebrew access in the build image

Features Updates
,
1

In an effort to support a wider range of languages and build tools, we are experimenting with adding Homebrew’s Linux variant to the Netlify build image. This allows you to select from a long list of supported tools to to be installed automatically before your build command runs.

If there’s a language or toolset you’ve been wanting to use in your build on Netlify but haven’t been able to because it’s not included in the build image, you can add it to a Brewfile.netlify file and we’ll take it from there. But be warned:

:warning: WARNING! :construction:

This feature is in early alpha, and functionality may change or be removed. Not recommended for production deploys.

You can still try it in a deploy preview or a test site, though! Scroll to the bottom of the build image included software list for details on using Homebrew in your build. Give it a test run, and let us know how it goes!

5 Likes
3

Whoa, that’s awesome. Right on!

1 Like
4

@verythorough This sounds very exciting. It would be great to show a couple Brewfile.netlify examples for those of us who have never dealt with this.

2 Likes
5

that would be a nice idea, @gregraven!

6

Perhaps some early testers could share their configs and explain their use cases!

7

@verythorough OK, I’ll start with something that doesn’t work.

I have a couple sites where I make extensive use of PHP include files for headers, footers, navigation, etc., along with a couple other simple PHP sections for “random” inclusion of content on each page. This is not WordPress, it’s basically a PHP version of what we used to be able to do with SHTML. (Aside: Why the powers who run HTML development have taken decades to add a simple tag to include HTML content the way we do images and other assets is beyond comprehension.)

If I was converting this to HTML locally, I would use something like httrack to crawl this site on my local server and output HTML files instead of PHP files. The terminal command would be:

httrack waterwedoing.local:8888

The trick is to duplicate this workflow remotely on Netlify.

The files are already on Github, so I spun up a new site in Netlify, linked it to my repository, and then added Brewfile.netlify with the following contents:

# 'brew install'
brew "php"

Netlify built this site, but did not translate the PHP files to HTML files, so I tried this:

# 'brew install'
brew "php"
brew "httrack"

I also included httrack as a build command.

This time, Netlify started a deploy (two of them, actually), and spent five minutes doing next to nothing before I cancelled it.

It seems obvious that I need to load PHP and then scan the files in the repository with something that will translate them. What’s missing in my build chain is the web server. I could add brew nginx so that httrack has something to crawl, but that opens port :8080 by default, and as far as I can tell, the repository does not have a URL at which I can point httrack. So, I think I can load each of the components, but I can’t figure out how to connect them so they talk to each other.

Unfortunately, I’ve never created a build command before (I create my static sites “by hand” – read “using BBEdit” – so they are ready to go no matter where I upload them), so I have no idea if I’m doing something wrong or if what I am trying to do is even possible.

8

Hi Greg,

Are you aware that we already have php installed in the build image? You should just brew "httrack", and I think that might work better.

I think you should then be able to fire up a server (on a port >1024 since you aren’t root, but you got that) and connect to it, but yes, we aren’t going to do that for you, you’d have to find one that is brew installable and easy to configure. I feel like this might be trying to fit a round peg into a square hole, to run a server in our container during build, but we won’t stop you anyway.

9

@fool I did not know that it was unnecessary to load PHP separate, but my first attempt did not include that step and it didn’t work, so I invoked PHP for the second go-round.

As I said, though, I think where it’s breaking down is not due to whether or not I am loading any particular cask, but rather where to point httrack (i.e., the URL of the internal server) to render the PHP files as HTML. For that matter, because I don’t understand how the build process works, I don’t even know that the files are accessible via a URL. Through the file system, certain, but why would there be a web server pointing at these files?

At any rate, I’m pretty happy with my current build process. I’m just intrigued by the possibilities created by incorporating brew into a hypothetic build process.

10

There’s also this now: The brew installation of httrack seems throw an error.

5:52:58 AM: Installing Homebrew dependencies from Brewfile.netlify
5:53:24 AM: Canceling deploy
6:00:38 AM: Error: No such file or directory - getcwd
6:00:38 AM: ==> Downloading https://curl.haxx.se/ca/cacert-2020-01-01.pem
6:00:38 AM: ==> Downloading https://cpan.metacpan.org/authors/id/L/LE/LEONT/Test-Harness-3.42.tar.gz
6:00:38 AM: ==> Downloading https://cpan.metacpan.org/authors/id/E/EX/EXODIST/Test-Simple-1.302175.tar.gz
6:00:38 AM: ==> Downloading https://cpan.metacpan.org/authors/id/B/BI/BINGOS/ExtUtils-MakeMaker-7.48.tar.gz
6:00:38 AM: ==> Downloading https://www.openssl.org/source/openssl-1.1.1h.tar.gz
6:00:38 AM: ==> Downloading https://mirror.httrack.com/historical/httrack-3.49.2.tar.gz
6:00:38 AM: ==> Installing dependencies for httrack: openssl@1.1
6:00:38 AM: ==> Installing httrack dependency: openssl@1.1
6:00:38 AM: ==> perl Makefile.PL INSTALL_BASE=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/libexec
6:00:38 AM: ==> make PERL5LIB=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/libexec/lib/perl5 CC=gcc-5
6:00:38 AM: ==> make install
6:00:38 AM: ==> perl Makefile.PL INSTALL_BASE=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/libexec
6:00:38 AM: ==> make PERL5LIB=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/libexec/lib/perl5 CC=gcc-5
6:00:38 AM: ==> make install
6:00:38 AM: ==> perl Makefile.PL INSTALL_BASE=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/libexec
6:00:38 AM: ==> make PERL5LIB=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/libexec/lib/perl5 CC=gcc-5
6:00:38 AM: ==> make install
6:00:38 AM: ==> perl ./Configure --prefix=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h --openssldir=/opt/buildhome/.linuxbrew/etc/openssl@1.1 no-ssl3 no-ssl3-method no-zlib  enable-md2 linux-x86_64
6:00:38 AM: ==> make
6:00:38 AM: ==> make install MANDIR=/opt/buildhome/.linuxbrew/Cellar/openssl@1.1/1.1.1h/share/man MANSUFFIX=ssl
6:00:38 AM: ==> make test
6:00:38 AM: Last 15 lines from /opt/buildhome/.cache/Homebrew/Logs/openssl@1.1/13.make:
6:00:38 AM: ../test/recipes/99-test_ecstress.t ................. ok
6:00:38 AM: ../test/recipes/99-test_fuzz.t ..................... ok
6:00:38 AM: Test Summary Report
6:00:38 AM: -------------------
6:00:38 AM: ../test/recipes/30-test_afalg.t                  (Wstat: 256 Tests: 1 Failed: 1)
6:00:38 AM:   Failed test:  1
6:00:38 AM:   Non-zero exit status: 1
6:00:38 AM: Files=158, Tests=2633, 145 wallclock secs ( 2.40 usr  0.28 sys + 104.90 cusr 28.59 csys = 136.17 CPU)
6:00:38 AM: Result: FAIL
6:00:38 AM: Makefile:207: recipe for target '_tests' failed
6:00:38 AM: make[1]: *** [_tests] Error 1
6:00:38 AM: make[1]: Leaving directory '/tmp/openssl@1.1-20201125-1675-1m3jo1g/openssl-1.1.1h'
6:00:38 AM: Makefile:205: recipe for target 'tests' failed
6:00:38 AM: make: *** [tests] Error 2
6:00:38 AM: Error: An exception occurred within a child process:
6:00:38 AM:   Errno::ENOENT: No such file or directory @ dir_chdir - /opt/build/repo
6:00:38 AM: Installing httrack has failed!
6:00:38 AM: Homebrew Bundle failed! 1 Brewfile dependency failed to install.
6:00:38 AM: Execution cancelled
6:00:38 AM: Error running command: Command was cancelled
6:00:38 AM: Failing build: Failed to build site
6:00:38 AM: Finished processing build request in 7m46.409820317s
12

@gregraven Thanks for giving it a shot! Your use case is a pretty complex one, with some challenges I’m not sure could be handled by Homebrew.

People who have requested this feature have been interested in using languages (such as Rust or Lua) or tools (such as Helm) in static build workflows they’re already familiar with. And for the early alpha, it makes troubleshooting easier if the only “unknown” we’re dealing with is whether Homebrew is working as expected in the build image.

If we get through some more alpha testing and determine that this change is ready for full release, we’ll definitely add more information and guidance to the docs. And maybe at that point you or someone else might make another attempt at the PHP-server-to-static workflow you describe!

13

Hi, @verythorough. Thanks for this awesome feature.
Looks like it has problems installing Deno. I’ve left this comment with more details: Support for Deno on Netlify - #14 by oscarotero

1 Like
14

@verythorough this is a nice addition, thanks! I was hoping that we could use this to install TeX Live, as we need LaTeX support in order to be able to render equations. However, it turns out that there is no bottle for this:

https://formulae.brew.sh/formula-linux/texlive

Hence we’d have to build from source and I’ve not attempted this as I suspect it would take way too long and burn up a lot of deploy minutes. I wondered if you might have any suggestions?

There is also an old PR to add TeX Live to the build image, but this doesn’t seem to have gone anywhere:

Perhaps because it would grow the image size a fair bit. May be that “texliveonfly” would help in this respect and seen an example for generating PDFs.

https://ctan.org/pkg/texliveonfly?lang=en

Our use for this is with Sphinx generated documentation, where we’d like to be able to take advantage of it being able to generate PDF downloads in addition to HTML, hence using something like MathJax to render client-side wouldn’t be any good.

15

@Andrew_Back That’s a bummer about texlive not having a bottle available.

You’re right that one of our biggest concerns with adding new software to the build image is the impact on image size. This is especially true for less commonly used packages because it slows the build system for everyone in order to benefit a small few.

However, if it can be configured to install on demand, that can be a great alternative, and one that we’ve used for some other dependencies such as Swift. Your suggestion to pursue “texliveonfly” sounds promising, though the implementation is outside my area of expertise. Perhaps inspecting the PR for converting Swift to on-demand installation would help?

It might be worth starting a new issue in the repo, referencing the old texlive PR, and proposing an on-demand alternative.

By the way, I also noticed that you’ve commented in the PR, and said you might explore tectonic for your needs. I’m curious if you ended up going down that path at all.