Compile and use custom build of Arduino ESP32

Leave a reply

For ESP32 Arduino project I needed to use PPPoS. Looking around I found several example and libs for using PPPoS, so I tried and boom! I started my journey into the compilation nightmare errors!

Preamble

Starting from 2.0.1, arduino-esp32 is not shipped with PPPoS support because of the output size (one of the several source I found: issue)

People suggest to use Arduino as ESP-IDF component (and I tried, it works! But you have to port your project with a good effort).

I built the arduino-esp32 with PPPoS by myself then used “as normal” to build my original Arduino project.

The following is certified working on my machine only.

Install build tools

Install the esp-idf build tools, for my project I need the version 4.4.6, the last supported for arduino-esp32 2.0.12 (You will understand your needed during the nightmare of compilations).

Get the esp32-build tools (I stay on release branch) (clone wherever you want)

clone --branch release/v4.4 https://github.com/espressif/esp32-arduino-lib-builder.git

Get ready to build

Enter the directory esp32-arduino-lib-builder and get ready with esp-idf tools (e.g. ‘. <path-to-esp-idf>/export.sh’ or the suggested alias ‘get_idf’)

cd <somewhere>/esp32-arduino-lib-builder
get_idf
# <output-stuff>.... idf.py build

Set the build flags you need in configs/defconfig.common (example). For PPPoS I added:

# configs/defconfig.common
CONFIG_PPP_SUPPORT=y
CONFIG_PPP_NOTIFY_PHASE_SUPPORT=y
CONFIG_PPP_PAP_SUPPORT=y
CONFIG_PPP_CHAP_SUPPORT=y
CONFIG_PPP_DEBUG_ON=y
CONFIG_LWIP_PPP_SUPPORT=y
CONFIG_LWIP_PPP_NOTIFY_PHASE_SUPPORT=y
CONFIG_LWIP_PPP_PAP_SUPPORT=y
CONFIG_LWIP_PPP_CHAP_SUPPORT=y
CONFIG_LWIP_PPP_DEBUG_ON=y
# ...

Build Arduino-ESP32

Then try to build. I want to build a specific target with specific android-esp32 module (2.0.12) and with specific esp-idf version (the latest compatible one with arduino-esp32, in my case v4.4.6).

./build.sh -t esp32 -A 2.0.12 -I v4.4.6

If you need to repeat the build command (and you will need) and you get some error about git and pulling stuff, add ‘-s’ like so:

./build.sh -t esp32 -A 2.0.12 -I v4.4.6 -s

If you are lucky you have the built arduino-esp32 ready in out/tools/sdk/. If you are not, maybe you have to play with the versions of arduino-esp32, esp-idf (-I argument) and the esp-idf installed in your system. I lost a lot of time tuning this, and be careful to clean everything every time you change some tool version.

Where and what to copy to build my project?

Oh you made it! Great!

So now, you have to build your project with the new beautiful android-esp32 custom build!

There are several different ways of doing this, let explore.

The “QUICK & DIRTY” way (Arduino IDE and arduino-cli

The dirty way consists of coping the out/tools folder to the location of your Arduino IDE or CLI.

You must have a previously installed version of the arduino-esp32 (of the same version) and set up with your project. In my setup I have the following:

Arduino IDE: $HOME/.arduino15/packages/esp32/hardware/esp32/2.0.12/tools/
Arduino CLI: $HOME/.arduino15/internal/esp32_esp32_2.0.12_5c4bba8bb6752c8d/tools/

You can try to compile, but maybe, you will get errors because of the missing includes: you should include header stuff during your project compilation by adding the same macro definitions (build flags) you used for building the sdk.

In my case I use the following:

arduino-cli compile \
    -m prod \
    --build-property "build.extra_flags=\"-DPPPOS_SUPPORT=1\" \"-DPPP_SUPPORT=1\" \"-DCONFIG_PPP_SUPPORT=1\" \"-DCONFIG_LWIP_PPP_SUPPORT=1\" \"-DLWIP_PPP_API=1\" \"-DPAP_SUPPORT=1\"" \

Now the Arduino project should build! If yes, give me five!

The clean and scalable way

It is possible to use the same distribution system arduino-esp32 uses for delivering of their official build: upload online a ready to use zip with your custom build!

In my sketch.yaml I have the following:

# sketch.yaml
# ...
    platforms:
      - platform: 'esp32:esp32 (2.0.12)'
        platform_index_url: 'https://espressif.github.io/arduino-esp32/package_esp32_index.json'
# ...

So I need a index file to list my builds, a build name a zip file because the records of the index files contains something like:

# package_esp32_index.json from https://espressif.github.io/arduino-esp32/package_esp32_index.json
# ...
          "name": "esp32",
          "architecture": "esp32",
          "version": "2.0.12",
          "category": "ESP32",
          "url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.12/esp32-2.0.12.zip",
          "archiveFileName": "esp32-2.0.12.zip",
          "checksum": "SHA-256:9a4f844ca67812c547a9635cdb0dd2c347cae7a3e855f95f9d490b2f8d340dbe",
          "size": "250664387",
# ...

Lets start with the zip file.

Create the zip

Inspired by the others scripts in the tools folder and on the script at components/arduino/.github/scripts/on_release.sh, I wrote something like this:

#!/bin/bash
source ./tools/config.sh

set -ex

DIR=$(pwd)

# define some stuff
BUILD_VERSION=2.0.12
ARCHIVE_FILE_NAME="esp32-$BUILD_VERSION.zip"
ARCHIVE_FILE_PATH="$DIR/dist/$ARCHIVE_FILE_NAME" 

ARCHIVE_CONTENT_DIR_NAME=esp32
ARCHIVE_CONTENT_DIR_PATH="$DIR/dist/$ARCHIVE_CONTENT_DIR_NAME"
rm -rf "$ARCHIVE_CONTENT_DIR_PATH"
mkdir -p "$ARCHIVE_CONTENT_DIR_PATH"
rm -rf "$ARCHIVE_FILE_PATH"

# copy arduino stuff
ESP32_ARDUINO="$AR_COMPS/arduino"
cp -r "$ESP32_ARDUINO"/* "$ARCHIVE_CONTENT_DIR_PATH/"
rm -rf "$ARCHIVE_CONTENT_DIR_PATH/.git"

# copy compiled stuff to dest folder
ESP32_ARDUINO="$ARCHIVE_CONTENT_DIR_PATH"
echo "Copy files to Arduino folder ${ESP32_ARDUINO}"
ESP32_ARDUINO=$ESP32_ARDUINO ./tools/copy-to-arduino.sh

# Replace tools locations in platform.txt
PKG_DIR="$ARCHIVE_CONTENT_DIR_PATH"
echo "Generating platform.txt..."
cat "$DIR/out/platform.txt" | \
sed 's/tools.xtensa-esp32-elf-gcc.path={runtime.platform.path}\/tools\/xtensa-esp32-elf/tools.xtensa-esp32-elf-gcc.path=\{runtime.tools.xtensa-esp32-elf-gcc.path\}/g' | \
sed 's/tools.xtensa-esp32s2-elf-gcc.path={runtime.platform.path}\/tools\/xtensa-esp32s2-elf/tools.xtensa-esp32s2-elf-gcc.path=\{runtime.tools.xtensa-esp32s2-elf-gcc.path\}/g' | \
sed 's/tools.xtensa-esp32s3-elf-gcc.path={runtime.platform.path}\/tools\/xtensa-esp32s3-elf/tools.xtensa-esp32s3-elf-gcc.path=\{runtime.tools.xtensa-esp32s3-elf-gcc.path\}/g' | \
sed 's/tools.xtensa-esp-elf-gdb.path={runtime.platform.path}\/tools\/xtensa-esp-elf-gdb/tools.xtensa-esp-elf-gdb.path=\{runtime.tools.xtensa-esp-elf-gdb.path\}/g' | \
sed 's/tools.riscv32-esp-elf-gcc.path={runtime.platform.path}\/tools\/riscv32-esp-elf/tools.riscv32-esp-elf-gcc.path=\{runtime.tools.riscv32-esp-elf-gcc.path\}/g' | \
sed 's/tools.riscv32-esp-elf-gdb.path={runtime.platform.path}\/tools\/riscv32-esp-elf-gdb/tools.riscv32-esp-elf-gdb.path=\{runtime.tools.riscv32-esp-elf-gdb.path\}/g' | \
sed 's/tools.esptool_py.path={runtime.platform.path}\/tools\/esptool/tools.esptool_py.path=\{runtime.tools.esptool_py.path\}/g' | \
sed 's/debug.server.openocd.path={runtime.platform.path}\/tools\/openocd-esp32\/bin\/openocd/debug.server.openocd.path=\{runtime.tools.openocd-esp32.path\}\/bin\/openocd/g' | \
sed 's/debug.server.openocd.scripts_dir={runtime.platform.path}\/tools\/openocd-esp32\/share\/openocd\/scripts\//debug.server.openocd.scripts_dir=\{runtime.tools.openocd-esp32.path\}\/share\/openocd\/scripts\//g' | \
sed 's/debug.server.openocd.scripts_dir.windows={runtime.platform.path}\\tools\\openocd-esp32\\share\\openocd\\scripts\\/debug.server.openocd.scripts_dir.windows=\{runtime.tools.openocd-esp32.path\}\\share\\openocd\\scripts\\/g' \
 > "$PKG_DIR/platform.txt"

# create the zip
cd "$ARCHIVE_CONTENT_DIR_PATH/.."
zip -r "$ARCHIVE_FILE_PATH" \
  "$ARCHIVE_CONTENT_DIR_NAME/platform.txt" \
  "$ARCHIVE_CONTENT_DIR_NAME/boards.txt" \
  "$ARCHIVE_CONTENT_DIR_NAME/programmers.txt" \
  "$ARCHIVE_CONTENT_DIR_NAME/cores" \
  "$ARCHIVE_CONTENT_DIR_NAME/libraries" \
  "$ARCHIVE_CONTENT_DIR_NAME/tools" \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/espota.exe \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/espota.py \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/gen_esp32part.py \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/gen_esp32part.exe \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/gen_insights_package.py \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/gen_insights_package.exe \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/partitions \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/ide-debug \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/sdk \
  "$ARCHIVE_CONTENT_DIR_NAME/tools/platformio-build*.py \
  "$ARCHIVE_CONTENT_DIR_NAME/variants"

cd "$DIR"
# ...

Yeah it’s very long! But it just create a directory with all the stuff, replace some lines (mainly to get the Arduino IDE or CLI use the right path to dependencies) and create the zip with the needed stuff (custom compiled sdk, headers and source files).

Note: here I got an error about the arch command. I installed arch from coreutils (via paru -S coreutils-arch (I use Arch BTW)).

The output zip is around 60MB, more less than the official one (around 260MB) because here I built the esp32 target only. I miss esp32s1, esp32s2 etc.

Create the index json file

The above script continues for creating the index file starting from the generated one:

# ...
# create platform_index file
INDEX_FILE_PATH="./dist/package_esp32_index.json"
cp "./out/package_esp32_index.template.json" "$INDEX_FILE_PATH"

BASE_URL=https://buro.fra1.digitaloceanspaces.com/arduino
ZIP_URL="$BASE_URL/$ARCHIVE_FILE_NAME"
ZIP_CHECKSUM=$(sha256sum "$ARCHIVE_FILE_PATH" | awk '{print $1}')
ARCHIVE_FILE_SIZE=$(stat -c%s "$ARCHIVE_FILE_PATH")

sed -i "s|\(.*\"url\"\) *: *\"\".*|\1: \"$ZIP_URL\",|" "$INDEX_FILE_PATH"
sed -i "s/\(.*\"archiveFileName\"\) *: *\"\".*/\1: \"$ARCHIVE_FILE_NAME\",/" "$INDEX_FILE_PATH"
sed -i "s/\(.*\"checksum\"\) *: *\"\".*/\1: \"SHA-256:$ZIP_CHECKSUM\",/" "$INDEX_FILE_PATH"
sed -i "s/\(.*\"size\"\) *: *\"\".*/\1: \"$ARCHIVE_FILE_SIZE\",/" "$INDEX_FILE_PATH"
sed -i "s/\(.*\"version\"\) *: *\"\".*/\1: \"$BUILD_VERSION\",/" "$INDEX_FILE_PATH"

I set my S3 bucket as url for the zip.

I tried to make a filesystem URL like ‘file:///home/buro/esp32-build/ … no, it doesn’t work. (I get unsupported protocol scheme "file"). So get your space in the web, change urls and upload everything somewhere!
OR you can try to serve the zip from your localhost, with python -m http.server for example.

Setup your Arduino project

With the zip hosted by my S3 bucket, I set up my project sketch like the following:

# sketch.yaml
# ...
    platforms:
      - platform: 'esp32:esp32 (2.0.12)'
        platform_index_url: 'https://buro.fra1.digitaloceanspaces.com/arduino/package_esp32_index.json'
# ...

It works!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.