Zephyr [1] includes a shell [2] subsystem, which is a pretty cool feature.
Unfortunately, the default device tree for the NUCLEO-G0B1RE board configures
the shell to use usart2 on pins PA2 and PA3 of the STM32G0B1
microcontroller. The issue with this setup is that these pins are also
connected to a USART interface of the ST-LINK controller on the board. As a
result, this configuration effectively makes it impossible to use Zephyr’s
shell.
There are (at least) two solutions to solve the problem:
I wanted to try something out using Zephyr’s shell, and since I had a NUCLEO-G0B1RE lying around, I decided to use that board for my tests. What was supposed to be a quick task turned into a bit of a lengthy debugging session.
Here’s what happened:
I have a Zephyr workspace directory on my computer that I use for quick tests like this. It looks like this:
zephyr_workspace/
├── bootloader
├── build
├── .cache
├── modules
├── tools
├── .venv
├── .west
├── zephyr # Zephyr source code (v4.1.0 commit 21b20de1e)
├── .env
└── README.md
Within that workspace directory, I built Zephyr’s shell_module example:
cd zephyr_workspace
source .venv/bin/activate
west build -p always -b nucleo_g0b1re/stm32g0b1xx zephyr/samples/subsys/shell/shell_module/
A quick look into the file build/zephyr/zephyr.dts revealed, that the default
device tree for the STM32G0B1 controller on the NUCLEO-G0B1RE board uses
usart2 for the shell, and that pin PA2 is used for TX and pin PA3 is used
for RX.
# zephyr.dts
(...)
chosen {
zephyr,flash-controller = &flash;
zephyr,console = &usart2;
zephyr,shell-uart = &usart2;
zephyr,uart-mcumgr = &usart2;
zephyr,sram = &sram0;
zephyr,flash = &flash0;
zephyr,code-partition = &slot0_partition;
zephyr,canbus = &fdcan1;
};
(...)
usart2: serial@40004400 {
compatible = "st,stm32-usart", "st,stm32-uart";
reg = < 0x40004400 0x400 >;
clocks = < &rcc 0x3c 0x20000 >;
resets = < &rctl 0x591 >;
interrupts = < 0x1c 0x0 >;
status = "okay";
pinctrl-0 = < &usart2_tx_pa2 &usart2_rx_pa3 >;
pinctrl-names = "default";
current-speed = < 0x1c200 >;
};
(...)
A quick look at the NUCLEO-G0B1RE schematics showed that pin PA2 of the
microcontroller is connected to pin 6 of connector CN10, and pin PA3 to pin
34 of the same connector. So, I hooked up a small UART-to-USB transceiver board
to these pins (plus GND), connected that board to my computer, and then
connected the NUCLEO-G0B1RE via its onboard USB port.
Once everything was wired up, I launched minicom (a serial communication
program on Linux) and configured it to use the USB interface of the UART-to-USB
board. Then I flashed the project onto the NUCLEO-G0B1RE using west:
west flash --runner openocd
At first the output of the board in minicom looked promissing:
uart:~$
I did get the shell prompt. But when I tried to interact with it, absolutely nothing happened. I couldn’t type any commands. And that’s where the debugging began:
build directory, rebuilt the project, and flashed it again.
Just to rule out any build issues.The first time I tried this, I wasn’t at home and didn’t have access to an
oscilloscope to probe the communication lines, so I decided to put the testing
on hold. A few days later, when I finally had the chance to examine the
signals, I noticed something odd: when sending commands to the NUCLEO-G0B1RE,
the voltage at pin PA3 didn’t toggle between 3.3V and 0V as expected during
UART communication. It only dropped slightly, just a few millivolts below 3.3V.
And that’s when it hit me: the NUCLEO-G0B1RE has an onboard ST-LINK module used to flash the STM32G0B1 microcontroller. Maybe the to ST-LINK controller and the STM32G0B1 are not only connected via JTAG, but also via UART. A closer look at the schematics confirmed my suspicion. The ST-LINK can communicate with the STM32G0B1 via the usart2 interface. As long as the ST-LINK is powered and active, its TX line holds the RX pin of usart2 on the STM32G0B1 high. That’s why the commands sent from my computer didn’t get to the microcontroller.
There are (at least) two solutions to this problem.
To work around the issue, you can use an overlay file to configure a different
USART interface on the STM32G0B1. Create a file named nucleo_g0b1re.overlay
in the zephyr/samples/subsys/shell/shell_module/boards/ directory with the
following content:
/ {
chosen {
zephyr,console = &usart1;
zephyr,shell-uart = &usart1;
zephyr,uart-mcumgr = &usart1;
};
};
Then, rebuild the project using the same command as before. If you check the
file build/zephyr/zephyr.dts again, you’ll see the following:
(...)
zephyr,console = &usart1;
zephyr,shell-uart = &usart1;
zephyr,uart-mcumgr = &usart1;
(...)
pinctrl-0 = < &usart1_tx_pc4 &usart1_rx_pc5 >;
(...)
So, you’ll need to change the pins your UART-to-USB board is connected to. Pin
PC4 connects to pin 35 of connector CN10, and pin PC5 connects to pin 37
of the same connector.
After changing the connections, flash the project again using the command from
above. You’ll see the same output in minicom as before, but this time, you
can actually interact with the shell:
uart:~$ help
Please press the <Tab> button to see all available commands.
You can also use the <Tab> button to prompt or auto-complete all commands or its subcommands.
You can try to call commands with <-h> or <--help> parameter for more information.
Shell supports following meta-keys:
Ctrl + (a key from: abcdefklnpuw)
Alt + (a key from: bf)
Please refer to shell documentation for more details.
Available commands:
bypass : Bypass shell
clear : Clear screen.
date : Date commands
demo : Demo commands
device : Device commands
devmem : Read/write physical memory
Usage:
Read memory at address with optional width:
devmem <address> [<width>]
Write memory at address with mandatory width and value:
devmem <address> <width> <value>
dynamic : Demonstrate dynamic command usage.
help : Prints the help message.
history : Command history.
kernel : Kernel commands
log : Commands for controlling logger
log_test : Log test
rem : Ignore lines beginning with 'rem '
resize : Console gets terminal screen size or assumes default in
case the readout fails. It must be executed after each
terminal width change to ensure correct text display.
retval : Print return value of most recent command
section_cmd : Demo command using section for subcommand registration
shell : Useful, not Unix-like shell commands.
shell_uart_release : Uninitialize shell instance and release uart, start
loopback on uart. Shell instance is reinitialized when
'x' is pressed
stats : Stats commands
version : Show kernel version
uart:~$
Alternatively, it should be possible to configure usart2 to use different pins, but I haven’t checked whether the NUCLEO-G0B1RE board has any available pins that won’t interfere with other connections.
There’s another way to work around the problem without changing any software.
You can put the ST-LINK controller into reset after flashing the STM32G0B1.
This is what the pin header JP1 next to the USB port is for. Simply place
a jumper across the two pins to short them.
However, this also disables the power supply to the STM32G0B1, which of corse
isn’t ideal. To avoid that, you need to move the jumper on pin header JP2. By
default (at least on my boards), the jumper on JP2 shorts the two pins
labeled “STLK”. Move this jumper all the way to the other side, onto the pins
labeled “CHG”.
It’s probably best to do this while the board is powered off. So the sequence would be:
JP2 to the new position.PA2) and 34 (PA3) of
connector CN10.JP1.Now you should see the shell prompt in minicom and be able to interact with
it.
I’m not entirely sure why the default device tree setup for the NUCLEO-G0B1RE board in Zephyr results in a configuration that makes using the shell impossible. I’ll try to investigate the reason behind this setup and see if it can be changed. But until then, you’ll need to make some adjustments to either the software or hardware to get Zephyr’s shell_module sample running.
Well, I hope this was helpful!
\
2025-05-27:
Take care,
Andreas
This is the seventh post in a series of posts (in a tutorial) about building an embedded Linux system for a Raspberry Pi 3 Model B+. A summary post for the tutorial can be found here. The summary post also talks about …
<project/root/dir>,Now it’s finally time to put it all together and run a entire embedded Linux system on a Raspberry Pi 3 Model B+. Let’s get started.
After Building a Root Filesystem for a Raspberry Pi 3 Model B+ with busybox, we have to make some minor adjustments to it.
First of all let’s add a root user to the system. This user won’t have a
password right now. You should not use such a setup in production, but for now
it’s ok. In order to do so, we have to create an etc directory to the root
filesystem and add two files to it.
cd <project/root/dir>/rootfs
mkdir etc
echo "root:x:0:0:/root:/bin/sh" > etc/passwd
echo "root::::::::" > etc/shadow
chmod 0600 etc/shadow
Noto: For more information about these files and its entries run the
commands man passwd and man 5 shadow
Linux will try to run a program called init at startup. This usually starts
up and monitors other programs (like a shell for example). The init program
that busybox provides reads a configuration file etc/inittab. So let’s create
that file:
cat > etc/inittab << EOF
::sysinit:/etc/init.d/rcS
::askfirst:-/bin/ash
EOF
This file will cause init to run the script /etc/init.d/rcS and afterwards
ask the user to press enter before it runs the /bin/ash shell. We can use
/etc/init.d/rcS to mound to pseudo file systems (/proc and /sys). So
let’s create the necessary directories, the script, and make the script
executable:
mkdir proc sys
mkdir etc/init.d
cat > etc/init.d/rcS << EOF
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
EOF
chmod u+x etc/init.d/rcS
On your host system all files in the root filesystem belong to your user, but on the Raspberry Pi they have to belong to the root user. So the ownership of all files in the created root filesystem have to be changed:
cd <project/root/dir>
sudo chown -R root:root rootfs
Note: For the next step make sure where to find your root filesystem
partion of your SD card when inserted into your host system. In my case it is
under /dev/sdb2.
Now it’s time to bring the root filesystem on the SD card that has been used earlier when running U-Boot and when running the kernel. So insert it into your host system, mount the root filesystem partition and copy the root filesystem onto the SD card.
cd <project/root/dir>
sudo mkdir /mnt/rootfs
sudo mount /dev/sdb2 /mnt/rootfs
sudo rsync -aHAXP --stats rootfs/ /mnt/rootfs/
Now you can unmount the SD card from your host system:
$> sudo umount /mnt/boot
and place the card into the card holder of the Raspberry Pi. Once the SD card
is in the Raspberry Pi and you have a connection to the UART interface, you can
power up the board. You should see the same output as shown in the post Run
U-Boot on a Raspberry Pi. Once you
have the u-boot command prompt, the following commands make the kernel boot:
U-Boot> setenv bootargs 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rw rootwait
U-Boot> load mmc 0:1 $kernel_addr_r Image
15034880 bytes read in 1362 ms (10.5 MiB/s)
U-Boot> load mmc 0:1 $fdt_addr_r bcm2837-rpi-3-b-plus.dtb
21027 bytes read in 5 ms (4 MiB/s)
U-Boot> booti $kernel_addr_r - $fdt_addr_r
## Flattened Device Tree blob at 02600000
Booting using the fdt blob at 0x2600000
Working FDT set to 2600000
Using Device Tree in place at 0000000002600000, end 0000000002608222
Working FDT set to 2600000
Starting kernel ...
After Starting kernel ... more output should appear. It looks something like
this:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 4.19.127-v8-schuam+ (andreas@penguin) (gcc version 13.2.0 (crosstool-NG 1.26.0)) #1 SMP PREEMPT Fri Jan 31 10:24:51 CET 2025
[ 0.000000] Machine model: Raspberry Pi 3 Model B Plus Rev 1.3
... output shortened ...
[ 1.349537] mmcblk0: mmc0:0002 N/A 1.88 GiB
[ 1.349980] of_cfs_init: OK
[ 1.358888] mmcblk0: p1 p2
[ 1.382711] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 1.391031] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 1.398479] devtmpfs: error mounting -2
[ 1.411324] Freeing unused kernel memory: 2880K
[ 1.416150] Run /sbin/init as init process
[ 1.552705] usb 1-1: new high-speed USB device number 2 using dwc2
Please press Enter to activate this console. [ 1.737027] usb 1-1: New USB device found, idVendor=0424, idProduct=2514, bcdDevice= b.b3
... output shortened ...
[ 3.640081] random: crng init done
Since the kernel and the init program write their output to the same
terminal, it’s a little hard to see, but the init program actually printed
Please press Enter to activate this console.. So if you press enter, you will
get a console with the following command prompt:
~ #
And now you can use your new embedded Linux on the Raspberry Pi 3 Model B+ and
run command like ps, ls or what ever:
~ # ls
bin linuxrc proc sys
etc lost+found sbin usr
Well, that’s it for this series of post about embedded Linux on the Raspberry Pi 3 Model B+. Have fun with it.
Take care,
Andreas
According to the book “Mastering Embedded Linux Programming” [1] there are four things you need to build an embedded Linux system. They are:
Back in 2023 I started a series of post on how to build such an embeeded Linux system for and run it on a Raspberry Pi 3 Model B+. This was supposed to become a little tutorial. Unfortunately, I somehow didn’t find the time to finish it until now. But now it finally consist of seven post (excluding this one) that explain all necessary steps:
In case you are wondering why that specific Raspberry Pi model? Well, I just happen to have one of them lying around 😊.
Ideally development system is a computer that runs Linux. I used an Arch Linux
system. I haven’t tried to go through the process on any other computer. I have
to admit I don’t exactly know which packages have to be installed on the
system, as my installation had pretty much all that was needed already
installed. Maybe I’ll find the time to check that in the future. One thing that
is essential though, you need to be able to run commands using sudo.
A long explanation on how to communicate with the UART of the Raspberry Pi is out of the scope of this tutorial. In short:
You need to connect three of the GPIO pins (GND, UART_TXD0, and UART_RXD0) of the Raspberry Pi to some UART to USB cable/board and connect that cable/board to one of the USB ports of your development computer. On your computer you need a program to display the serial communication. To find out which pins of the Raspberry Pi to use, have a look in the article Raspberry Pi UART Communication using Python and C [2]. One such UART to USB cable is the USB-RS232-WE-5000-BT_3.3 - FDTI [3] by FDTI. For possible programs to display the serial communication, I recommand to have a look into the article Working with the serial console [4] in the Arch Wiki.
Within the tutorial, I will refere to the host system a couple of times. This
is the development computer. I also refere to the <project/root/dir>
directory multiple times. This is some directory on your host system that
should be empty at the beginning. It could be some directory in your home
directory (e.g. /home/<your-username>/embedded_linux_tutorial/). So whenever
you see <project/root/dir> somewhere in a command, make sure you replace it
with the correct path to the project root directory on your machine.
Before we actually get started, we need to prepare the SD card. An in depth explanation on how to prepare the SD card can be found in the article “Boot a Raspberry Pi 4 using u-boot and Initramfs” [5] by Hechao. Please follow the instruction in that article. Afterwards you should have an SD card with two partions on it. One with a FAT32 and one with an ext4 file system on it.
Well, let’s get started with the actual tutorial and build a toolchain.
Take care,
Andreas
This is the sixth post in a series of posts (in a tutorial) about building an embedded Linux system for a Raspberry Pi 3 Model B+. A summary post for the tutorial can be found here. The summary post also talks about …
<project/root/dir>,The fourth and final piece of the puzzle of building an embedded Linux system is the root filesystem. Let’s build that part now.
There are many way to build a Root Filesystem for Linux. I decided to use Busybox [1]. The following steps show how to do it.
First of all let’s create a directory that will hold a copy of the root filesystem on the host machine:
cd <project/root/dir>
mkdir rootfs
Now we need the source code of busybox. In the <project/root/dir>, run:
git clone git://busybox.net/busybox.git
cd busybox
git checkout 1_36_1
After checking out the tag 1_36_1, I was on commit 1a64f6a2 of the busybox
repository.
To build the root filesystem, we need the toolchain that was built before.
Therefore we have to prepend our PATH. In case you followed my description
in Building a (Cross) Toolchain,
you just have to adjust <project/root/dir> in the following command.
Otherwise you might have to adjust more. To prepend the PATH, run:
PATH=<project/root/dir>/x-tools/aarch64-rpi3-linux-gnu/bin/:$PATH
Additionally we need to set two environment variable:
export CROSS_COMPILE=aarch64-rpi3-linux-gnu-
export ARCH=arm64
Now we can start building the root filesystem.
Note: The following command will delete a file called .config in your current working directory. In case you have been working on a configuration for a root filesystem before, you might want to save this .config file for later use. For good measure, you should run:
make distclean
Then set the default configuration for Busybox:
make defconfig
make menuconfig didn’t work on my machine, due to some package dependency
problems. At the time of writing this post, I didn’t want to bother with it,
and instead of using make menuconfig to adjust some parts of the
configuration, I opened the file .config in an editor and chanaged the
following lines manually:
In the line starting with CONFIG_CROSS_COMPILER_PREFIX, I added the content
of the CROSS_COMPILE environment variable:
CONFIG_CROSS_COMPILER_PREFIX="aarch64-rpi3-linux-gnu-"
In the line starting with CONFIG_PREFIX I changed the path to the path of the
newly created rootfs directory:
CONFIG_PREFIX="../rootfs"
This changes the directory into which the root filesystem will be installed.
To link the executable of busybox statically, I changed the line that read like this:
# CONFIG_STATIC is not set
to
CONFIG_STATIC=y
To actually build and install the root filesystem into the ´rootfs` directory, run:
make
make install
If you followed along this tutorial (including the previous posts) and run
tree -L 2 in your <project/root/dir>, you should see the following:
$> cd <project/root/dir>
$> tree -L 2
.
├── busybox
│ ├── ... (output shorened)
├── crosstool-ng
│ ├── bin
│ ├── ... (output shortened)
├── linux
│ ├── ... (output shorened)
├── rootfs
│ ├── bin
│ ├── sbin
│ ├── usr
│ └── linuxrc -> bin/busybox
├── rpi
│ └── firmware-master
├── u-boot
│ ├── ... (output shortened)
│ ├── u-boot.bin
│ ├── ... (output shortened)
└── x-tools
└── aarch64-rpi3-linux-gnu
For how to use the root filesystem, see the post Run a Root Filesystem for a Raspberry Pi 3 Model B+.
Take care,
Andreas
Ever wondered where files go when you package software with bitbake?
In my last post I discussed where
files go when you build software with bitbake. I did that in an example
project and used openssl as an example software package. If you haven’t read
that post, I encourage you to do so before reading this one. I know that the
last post is kind of long. If you don’t want to read all of it, go at least
through the
Introduction to see
how the project is set up in order to follow along with the current post. I use
the exact same set up here.
In the last post, it would have been possible to run bitbake openssl to run
through the whole process from downloading the source code all the way to a
finished openssl package. I didn’t do that. Instead, I split the process up
and ran individual tasks manually. This allowed me to explain everything in
more detail. It also allowed me to stop after the compile task. Missing was
what happens afterwards. And that is what this post is all about. But instead
of keep using openssl as an example, I decided to use tzdata for this post.
Why? Because with tzdata it is a little easier to explain bitbake’s FILES
variable.
I will start by building tzdata the same way I build openssl in the last
post (without going into details about that process) and then I will explain
the steps (tasks) necessary to wrap everything up into packages.
tzdata with BitbakeOK, let’s build the tzdata software the same way I built openssl in my last
post. The very first step is to
cd into the project directory and source the oe-init-build-env script:
cd <path/to/example/project>
source poky/oe-init-build-env build/
After that, the current working directory is <path/to/example/project>/build.
All the following command have to be run in this build directory. Also all
given paths are relative to the build directory.
By the way, the recipe for tzdata can be found under
../poky/meta/recipes-extended/timezone/ and is called tzdata.bb. To build
tzdata, we need to run the following commands:
bitbake -c fetch tzdata
bitbake -c unpack tzdata
bitbake -c patch tzdata
bitbake -c configure tzdata
bitbake -c compile tzdata
Note: We could have just invoked bitbake -c compile tzdata and bitbake
would have taken care of running the tasks that the compile task depends on.
But I wanted to show the five steps I ran in the last
post once more.
Note: Just like in the last
post, it’s a good idea to
create an environment file for tzdata to analyze some bitbake variables.
This can be done by running: bitbake -e tzdata > environment.txt. I will
refer to that file as the “environment file”.
bitbake’s WORKDIR variable is a different for tzdata compared to
openssl. I won’t analyze how it is set, as I have done that in detail in the
last post. Just this
much, tzdata sets the PACKAGE_ARCH variable explicitly to all. To do so,
tzdata.bb inherits allarch. allarch.bbclass can be found under
../poky/meta/classes/ and sets the PACKAGE_ARCH variable. The whole
WORKDIR variable for tzdata is set to the
tmp/work/all-poky-linux/tzdata/2023c-r0 directory within the build
directory.
After the compile task you end up with a WORKDIR that looks like this:
$ tree -L 1 tmp/work/all-poky-linux/tzdata/2023c-r0/
tmp/work/all-poky-linux/tzdata/2023c-r0/
├── build
├── deploy-source-date-epoch
├── recipe-sysroot
├── recipe-sysroot-native
├── source-date-epoch
├── sstate-install-deploy_source_date_epoch
├── temp
├── tz
└── configure.sstate
Except for configure.sstate all other entries are directories with more or
less sub-directories and files in them.
The compiled files ended up in the build directory inside the WORKDIR. That
is because the recipe for tzdata sets the B variable to ${WORKDIR}/build
and the recipe defines the whole compile task and used the B variable
within it.
Note: tzdata is not a “normal” software where the output of the
compilation task is one or multiple executables. Instead the compilation yields
files in the timezone information format (TZif). This does not matter for the
purpose of this article.
tzdata with BitbakeNow let’s look into the steps necessary to package tzdata and where files end
up during that process.
do_installThe first step is to run the install task:
bitbake -c install tzdata
This task places the compiled files into a “holding” area that bitbake’s D
variable points to. D is set to ${WORKDIR}/image. The install task is
entirely defined in tzdata.bb and it actually uses the D variable 😀.
That means that after the install task finishes you can find a new directory
called image in the WORKDIR:
$ tree -L 4 tmp/work/all-poky-linux/tzdata/2023c-r0/image/
tmp/work/all-poky-linux/tzdata/2023c-r0/image/
├── etc
│ ├── localtime -> /usr/share/zoneinfo/Universal
│ └── timezone
└── usr
└── share
└── zoneinfo
├── Africa
├── (...) // output truncated
The image directory contains two sub-directories (etc and usr). These are
the sub-directories that would contain files from the tzdata package on a
Linux machine.
do_packageOnce the holding area is filled. It’s time for the next step which is the
package task:
bitbake -c package tzdata
The package task creates three new directories in the WORKDIR that are of
interest here:
package (pointed to by bitbake’s PKGD variable)
It contains the whole tzdata package before it is split up into individual
packages. In this case it’s basically a copy of the image directory.
packages-split (pointed to by bitbake’s PKGDEST variable)
We will look into this directory in just a bit.
pkgdata (pointed to by bitbake’s PKGDESTWORKDIR variable)
This is a temporary work directory used by the package task to keep some
metadata about the individual packages.
If you look into the packages-split directory, you will find the following:
$ tree -L 1 tmp/work/all-poky-linux/tzdata/2023c-r0/packages-split/
tmp/work/all-poky-linux/tzdata/2023c-r0/packages-split/
├── tzdata
├── tzdata-africa
├── tzdata-americas
├── tzdata-antarctica
├── tzdata-arctic
├── tzdata-asia
├── tzdata-atlantic
├── tzdata-australia
├── tzdata-core
├── tzdata-europe
├── tzdata-misc
├── tzdata-pacific
├── tzdata-posix
├── tzdata-right
└── tzdata-src
Where do these directories come from?
If you look into the environment file and search for the PACKAGES variable,
you will find that this variable is a list of packages with the same names as
the directories. These PACKAGES are defined in the recipe file tzdata.bb.
Also in tzdata.bb, you can find multiple lines that look like these examples:
FILES:tzdata-antarctica += "${datadir}/zoneinfo/Antarctica"
FILES:tzdata-arctic += "${datadir}/zoneinfo/Arctic"
These definitions define which files from tzdata go into which of the
individual packages and therefore different directories in
${WORKDIR}/packages-split.
do_packagedataThis step creates metadata for the subsequent step of generating the actual
packages. The data can be found in ${WORKDIR}/pkgdata-pdata-input.
do_package_write_rpmNow it is finally time to generate the packages that can then be installed into
a distribution image. In this case we need to run the package_write_rpm task.
Why to we create rpm packages?
If you have a look into the set up of the example project that we work in all
the time, you find the PACKAGE_CLASSES variable set to package_rpm in the
conf/local.conf file (see the
Introduction of my
last post). This variable determines which type of packages will be build. An
alternative to rpm packages would, for example, be deb packages.
All right let’s run the package_write_rpm task:
bitbake -c package_write_rpm tzdata
This will create all the individual packages in a sub-directory of the
${DEPLOY_DIR_RPM} directory. The name of this sub-directory depends on the
PACKAGE_ARCH variable. Usually it is just exactly the value of that variable,
but if PACKAGE_ARCH is set to all (see above), the sub-directory is called
noarch. The reason for why that is the case can be found in the documentation
file ../poky/documentation/migration-guides/migration-2.3.rst.
All right, looking into tmp/deploy/rpm/noarch (${DEPLOY_DIR_RPM}/noarch)
reveals the following:
$ tree -L 1 tmp/deploy/rpm/noarch/
tmp/deploy/rpm/noarch/
├── tzdata-2023c-r0.noarch.rpm
├── tzdata-africa-2023c-r0.noarch.rpm
├── tzdata-americas-2023c-r0.noarch.rpm
├── tzdata-antarctica-2023c-r0.noarch.rpm
├── tzdata-arctic-2023c-r0.noarch.rpm
├── tzdata-asia-2023c-r0.noarch.rpm
├── tzdata-atlantic-2023c-r0.noarch.rpm
├── tzdata-australia-2023c-r0.noarch.rpm
├── tzdata-core-2023c-r0.noarch.rpm
├── tzdata-europe-2023c-r0.noarch.rpm
├── tzdata-misc-2023c-r0.noarch.rpm
├── tzdata-pacific-2023c-r0.noarch.rpm
├── tzdata-posix-2023c-r0.noarch.rpm
└── tzdata-right-2023c-r0.noarch.rpm
There you have it, readily build tzdata packages that can be installed into a
distribution image, but this would be a topic for another post.
Take care,
Andreas
Ever wondered where files go when you build software with bitbake?
Well, I have. The build directory seemed to be a mess at the beginning. While
the OpenEmbedded Build System Concepts [1] section of
the Yocto Project Overview and Concepts Manual [2] does a
really good job of explaining the general concept, I still found it difficult
to really understand all the paths in the build directory. That’s why I
thought, I would do the following:
bitbake variables.I hope this will shine a light on the apparent mess in the build directory.
To do so, I started out by setting up an example project directory. This was
done in accordance to the directory structure I suggested in a previous
post. I wanted to keep the
example relatively simple, yet it should include a little more then just the
“default” layers that poky supplies. Since I’m working on a project that is
supposed to run on a BeagleBone Black board, I include the meta-ti and the
meta-arm layers from the OpenEmbedded Layer
Index [3]. The latter one was include, because
meta-ti depends on it.
So I started out with an example project directory that looked like this:
$ cd <path/to/example/project>
$ tree -L 2
.
├── bitbake
│ ├── downloads
│ └── sstate-cache
├── build
│ └── conf
├── layers
│ ├── meta-arm
│ └── meta-ti
├── poky
│ ├── (...) // output truncated
│ └── SECURITY.md
└── README.md
Note: For those interested in the exact setup: meta-arm, meta-ti, and
poky were added as git submodules to the project. I used the kirkstone
branch for all these submodules. The specific commits are:
meta-arm: 98cf1495dbca946b8c6a82efdfce0afb63ff41e2
meta-ti: 155218f03ee8222eeb02f11ea9bc41135cf28e38
poky: d8d6d921fad14b82167d9f031d4fca06b5e01883
Besides some other stuff that is not relevant for this post, the file
./build/conf/local.conf contains the following variables:
MACHINE ?= "beaglebone"
DL_DIR ?= "${TOPDIR}/../bitbake/downloads"
SSTATE_DIR ?= "${TOPDIR}/../bitbake/sstate-cache"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
The file ./build/conf/bblayers.conf looks like this:
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
/home/andreas/projects/active/beagle/yocto/poky/meta \
/home/andreas/projects/active/beagle/yocto/poky/meta-poky \
/home/andreas/projects/active/beagle/yocto/poky/meta-yocto-bsp \
/home/andreas/projects/active/beagle/yocto/layers/meta-arm/meta-arm-toolchain \
/home/andreas/projects/active/beagle/yocto/layers/meta-arm/meta-arm \
/home/andreas/projects/active/beagle/yocto/layers/meta-ti/meta-ti-bsp \
"
All right, now let’s look at an example software package and build it with
bitbake. For whatever reason I decided to look at the package openssl for
this post.
openssl with BitbakeAfter setting up the project, you could just do the following to build and
package openssl:
cd <path/to/example/project>
source poky/oe-init-build-env build/
bitbake openssl
But to demonstrate what’s going on during the build process, I skipped the last
command. In the following, I will go through the build process in small steps
by running the individual tasks that are needed to build openssl. You can
run bitbake -c listtasks openssl to see all available tasks related to the
openssl recipe. Not all of them are needed to build openssl. I will just go
through the ones relevant to the build process.
Note: You can find the recipe that build openssl in
<path/to/example/project>/poky/meta/recipes-connectivity/openssl/. The recipe
file is called openssl_3.0.12.bb.
OK, let’s finally get started.
Note: All the following commands are run from within the build directory
inside the example project directory. If you want to follow along make sure
that you have run the first two commands from the previous command listing. The
current working directory should be <path/to/example/project/build.
do_fetchThe first command you need to run is:
bitbake -c fetch openssl
This command fetches all the source files necessary to build the software. If
you actually start from scratch, it will actually do a little more than just
fetching the source code for the particular software (openssl in this case).
I will ignore that “little more” for now. The really interesting result of this
command is a new file called: openssl-3.0.12.tar.gz:
$ tree ../bitbake/downloads
../bitbake/downloads/
├── (...) // output truncated
├── openssl-3.0.12.tar.gz
└── openssl-3.0.12.tar.gz.done
Note: Remember we are in the build directory. That’s the reason why we
had to use ../bitbake/downloads.
So, how did bitbake know what to download?
Well, if you take a look into the recipe file (openssl_3.0.12.bb), you will
find the following lines:
SRC_URI = "http://www.openssl.org/source/openssl-${PV}.tar.gz \
file://run-ptest \
file://0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch \
file://afalg.patch \
file://0001-Configure-do-not-tweak-mips-cflags.patch \
"
The first line tells bitbake what to download. The ${PV} variable in the
path is not explicitly set in the recipe, but taken from the recipe file name
(“3.0.12”).
And why does the downloaded file end up in ../bitbake/downloads/?
This is determined by the $DL_DIR variable set in conf/local.conf
(see Introduction).
do_unpackNext, the source files have to be unpacked. This can be done with:
bitbake -c unpack openssl
Where do the unpacked files end up?
If you look into the Yocto Project Overview and Concepts Manual [2] (specifically section “4.3.5.1 Source
Fetching”), you will see that two directories become important here. They are
set in bitbake’s variables WORKDIR and S.
Before explaining were these two variables are set, let’s look at the result of
the previous command. For the openssl recipe, the WORKDIR variable is set
to tmp/work/armv7at2hf-neon-poky-linux-gnueabi/openssl/2.0.12-r0 within the
build directory. It is actually set as an absolute path, I just skipped the
beginning to safe some space. If you look into that directory, you will find
the following:
$ ls tmp/work/armv7at2hf-neon-poky-linux-gnueabi/openssl/2.0.12-r0
openssl-3.0.12 0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch
recipe-sysroot 0001-Configure-do-not-tweak-mips-cflags.patch
recipe-sysroot-native afalg.patch
source-date-epoch run-ptest
temp
On the left hand side you can see five directories which I will not discuss for
now. On the right hand side you can see the four files. These are the four
files that were additionally set in the SRC_URI variable in the recipe file
(see do_fetch). The do_unpack task didn’t download them. It
found them under ../poky/meta/recipes-connectivity/openssl/openssl/ and
copied them into the WORKDIR.
The S variable is set to the openssl-3.0.12 directory in the WORKDIR.
Again, this is actually done as an absolute path, but I shortened it to safe
some space. If you look into the S directory, you will find the decompressed
source code that was compressed into the
../bitbake/downloads/openssl-3.0.12.tar.gz archive.
All right now let’s look into how WORKDIR and S are set. It’s a little
complicated (at least for WORKDIR).
WORKDIRLooking into the Variable Glossary [4]
reveals that the WORKDIR for a recipe is defined as:
${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
In our case, this unfolds to be the
tmp/work/armv7at2hf-neon-poky-linux-gnueabi/openssl/3.0.12-r0 within the
build directory.
But how do we get there?
Well, that’s a lot to unpack. Let’s do it! If not otherwise noted, the
following information comes from the Variable
Glossary [4]. Additionally, when trying to analyze
something like such bitbake variables, it is helpful to get bitbake’s
environment for a recipe. I temporarily saved it into a file called
environment.txt by running:
bitbake -e openssl > environment.txt
Note: In the following, I will refer to the environment.txt file as the
“environment file”.
Let’s dissect the WORKDIR variable piece by piece:
TMPDIR
If not specifically set in conf/local.conf, the default is
${TOPDIR}/tmp. TOPDIR is the build directory. Since I didn’t explicitly
set TMPDIR it is the tmp directory inside the build directory.
MULTIMACH_TARGET_SYS
This variable is set to ${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}. Well,
another variable to take apart:
PACKAGE_ARCH
Some recipes set this variable explicitly (e.g. to MACHINE_ARCH). This
is not the case here. If it is not set explicitly in the recipe,
PACKAGE_ARCH is set to TUNE_PKGARCH in
../poky/meta/conf/bitbake.conf. All right, let’s find out where
TUNE_PKGARCH is defined. Bear with me 💪:
TUNE_PKGARCH
This is were it gets a little complicated. If you look into the environment file, you will find the following line:
TUNE_PKGARCH="armv7at2hf-neon"
Immediately above that line you can find a “pre-expansion value” that looks like this:
${ARMPKGARCH}${ARMPKGSFX_THUMB}${ARMPKGSFX_DSP}${ARMPKGSFX_EABI}${ARMPKGSFX_ENDIAN}${ARMPKGSFX_FPU}
And above that, some options where that value might come from.
Analyzing this part of the environment file reveals it was set in
line 12 of ../poky/meta/conf/machine/include/arm/arch-arm.inc. I
trust that you can verify that for yourself.
I won’t go into further detail on where to find these new variables.
I hope, after reading this post, you will have a good idea how to
analyze that yourself. But I will explain why arch-arm.inc plays a
role in setting TUNE_PKGARCH.
So, why is arch-arm.inc relevant?
If you remember, I set the MACHINE variable to beaglebone in
conf/local.conf (see Introduction). This causes
bitbake to look for a machine configuration file called
beaglebone.conf and it finds it in
../layers/meta-ti/meta-ti-bsp/conf/machine/. This file has the
following line in it:
require conf/machine/include/ti33x.inc
The file ti33x.inc can be found in
../layers/meta-ti/meta-ti-bsp/conf/machine/include/ and in turn
requires conf/machine/include/arm/armv7a/tune-cortexa8.inc. Since
bitbake searches in all layer directories that are set in the
BBLAYERS variable in conf/bblayers.conf (see
Introduction), it finds the required file under
../poky/meta/conf/machine/include/arm/armv7a/.
What follows now is a trace of required files. Starting point is
tune-cortexa8.inc. It requires the next file in the list and so
forth until the last file will finally be arch-arm.inc´. I will just
list the names of the required files. All these file can be found
under ../poky/meta/conf/machine/include/arm/`.
Please verify this trace for yourself if you want. Have fun 😉!
TARGET_VENDOR
Since I set the DISTRO variable to poky in conf/local.conf (see
Introduction), the TARGET_VENDOR variable is set to
-poky in ../poky/meta-poky/conf/distro/poky.conf. Looking into the
environment file gave a first clue on where to look 😀.
TARGET_OS
This variable is set in ../poky/meta/conf/bibake.conf to the value:
linux${LIBCEXTENSION}${ABIEXTENSION}. Again, I started by looking into
the environment file. As with TUNE_PKGARCH, I won’t go any deeper in
explaining where the contained variables come from. By now, you should
have a good idea on how to find them yourself.
PN
This variable is usually extracted from the recipe file name. In our case it
is set to openssl. More details can be found in the Variable
Glossary [4].
EXTENDPE
This variable has to do with the epoch of a recipe. In our case it is just set to an empty string. More details can be found in the Variable Glossary [4].
PV
This is the package/recipe version and is normally extracted from the file
name of the recipe. In our case it is set to 3.0.12. More details can be
found in the Variable
Glossary [4].
PR
This is the revision of the recipe. The default value is r0 and that is the
value that is used here. The Variable
Glossary [4] explains how it is set.
SCompared to the WORKDIR variable, finding out what S is set to is quite
easy. Some package recipes set this variable explicitly. In such a case you
would find it in the recipe file. That is not the case with openssl. As it is
not set in the recipe itself, the definition from
../poky/meta/conf/bitbake.conf is used. There it is set to
${WORKDIR}/${BP}. BP on the other hand is set to ${BPN}-${PV}. And these
two variables are also set in bitbake.conf by means of so called “executable
meta data”. BPN is basically the package name and PV the package version
(see the Variable
Glossary [4]).
do_patchAfter unpacking the source code, the next step is to patch the sources (if
necessary). The recipe for openssl does define some patches as seen earlier.
Running bitbake -c patch openssl may take longer than you might think. This
is because bitbake first needs to build some tools that are needed to do the
actual patching. But once the command finishes, the patches we saw earlier have
been applied. More details about the do_patch task can be found in the Yocto
Project Overview and Concepts
Manual [2] (specifically in the section “4.3.5.2
Patching”).
do_configureThe do_configure task configures the software. This is pretty specific to the
software being build. In the case of openssl, the recipe defines the whole
task. One of the main steps it takes is running a perl script that can be found
here: ${S}/Configure. The do_configure task is the first time the B
variable becomes relevant. It contains the path to where the software is build.
If the recipe does not set it, it is set to ${S}, but the openssl recipe
sets is explicitely to ${WORKDIR}/build.
After running bitbake -c configure openssl, you can check that the B
directory exists and mainly contains a bunch of empty directories (and a few
files). Again, this task takes a little longer than you might expect. Also
because bitbake has to build some auxiliary software, before it can configure
openssl.
More details about the do_patch task can be found in the Yocto Project
Overview and Concepts
Manual [2] (specifically in the section “4.3.5.3
Configuration, Compilation, and Staging”).
do_compileFinally lets actually build/compile openssl:
bitbake -c compile openssl
Once this task has finished, you can find all compiled object files and the
linked software in the B directory. I won’t show the whole content of that
directory, because this would be way to much. But two files that can be found
in the B directory are:
And corresponding links called libcrypto.so and libssl.so.
More details about the do_patch task can be found in the Yocto Project
Overview and Concepts
Manual [2] (specifically in the section “4.3.5.3
Configuration, Compilation, and Staging”).
Originally I thought I would also write about the packaging of software as
well. But this post has become quite long already. I will probably do that in a
following post. Stay tuned.
Take care,
Andreas
Just as yesterday’s post, I started writing this post about a year ago, but never finished it. Let’s change that now.
So, I had the following problem: I wanted to turn the data in a
pandas.DataFrame [1] with a MultiIndex into a list of nested
dictionaries. Wait what? Let me clarify with an example:
The following short script, sets up an example DataFrame:
import pandas as pd
import numpy as np
one = 6 * ["Alice"] + 6 * ["Bob"]
two = 2 * (2 * ["foo"] + 2 * ["bar"] + 2 * ["baz"])
three = 6 * ["one", "two"]
tuples = list(zip(*[one, two, three]))
index = pd.MultiIndex.from_tuples(tuples, names=["first", "second", "third"])
np.random.seed(42)
df = pd.DataFrame(
np.random.randint(low=1, high=11, size=(3, 12)),
index=["A", "B", "c"],
columns=index
)
If you print it (print(df)), you get the following output:
first Alice Bob
second foo bar baz foo bar baz
third one two one two one two one two one two one two
A 7 4 8 5 7 10 3 7 8 5 4 8
B 8 3 6 5 2 8 6 2 5 1 10 6
c 9 1 10 3 7 4 9 3 5 3 7 5
At this point, I needed a way to convert every row (A, B, and C) into a
nested dictionary. Let’s say the dictionary for row A is stored in dict_A.
Printing it (print(dict_A)) should lead to the following output (I
reformatted it a little bit for better readability):
{
'Alice': {
'bar': {'one': 8, 'two': 5},
'baz': {'one': 7, 'two': 10},
'foo': {'one': 7, 'two': 4}
},
'Bob': {
'bar': {'one': 8, 'two': 5},
'baz': {'one': 4, 'two': 8},
'foo': {'one': 3, 'two': 7}
}
}
If you think to yourself: Why would you want to do that? Well, I had a
Jinja [2] template and the data in the DataFrame was
supposed to be used in that template. A nested dictionary seemed to be the best
data format for passing the data into the Jinja’s render function.
Ok, let’s look at how I accomplished that.
First I thought that this should be pretty easy to do. Every row in a
DataFrame is of type <class 'pandas.core.series.Series'>. You can verify
that with:
print(type(df.iloc[0]))
Series objects have a method called
to_dict
[3] that I was hoping would to the job:
print(df.iloc[0].to_dict())
But the output of that command looks like this (again a little reformatted):
{
('Alice', 'foo', 'one'): 7,
('Alice', 'foo', 'two'): 4,
('Alice', 'bar', 'one'): 8,
('Alice', 'bar', 'two'): 5,
('Alice', 'baz', 'one'): 7,
('Alice', 'baz', 'two'): 10,
('Bob', 'foo', 'one'): 3,
('Bob', 'foo', 'two'): 7,
('Bob', 'bar', 'one'): 8,
('Bob', 'bar', 'two'): 5,
('Bob', 'baz', 'one'): 4,
('Bob', 'baz', 'two'): 8
}
Well, not exactly what I was hoping for.
After some research and some tinkering around, I wrote the following function:
def convert_series_to_dict(series):
if (series.index.nlevels == 1):
return series.to_dict()
else:
output_dict = {}
for header in series.index.levels[0]:
if (header in series.index):
output_dict[header] = convert_series_to_dict(series.xs(header))
return output_dict
It’s a recursive function that checks the number of levels in the index of the
passed in Series. It that number is one, it just calls the to_dict method
on it. Otherwise it traverses through the levels and builds up a nested
dictionary. If you call this function on the first row of the DataFrame:
print(convert_series_to_dict(df.iloc[0]))
you get exactly, what I was trying to get (see above in the Background/Problem section).
As a final touch, to create the list of nested dictionaries that I wanted, the following was added to the script:
list_of_dicts = []
for index, row in df.iterrows():
list_of_dicts.append(convert_series_to_dict(row))
I won’t add the output you would get by printing list_of_dicts. I guess you
can imagine what is looks like.
2024-10-23:
Take care,
Andreas
Ok, I think I had this problem about two years ago (not quite sure). Then a little over a year ago, I saw that someone had pretty much the exact same problem on stackoverflow and I answered it there. Well, I was kind of late and someone else had answered before, but I felt that my solution got to the point of the question just a little better. Then I started to write a post for my website about it, but somehow never quite finished and published it. Today I finally got to do so 😃.
Enough bla bla bla, let’s get to the problem.
Imagen you have an excel file with measurement data. The first header row tells
you what was measured, the second one what the measurement unit is. Every row
beneath that represents a set of measurements taken at a specific time. Now you
need to read in the data from the excel file into a pandas.DataFrame. While
doing so, you also want to convert the measurement data into SI units.
pandas.read_excel function has an optional argument converters. To use it,
you have to pass in a dictionary. Each key in that dictionary is a column name
and the corresponding value a lambda function that will be applied to that
column. The problem is, the excel file at hand has two header rows represented
as a MultiIndex in pandas. So what do you use for the keys in the
converters dictionary?
To better demonstrate the problem and to have an example to work with, imagine you had an excel file with the following table:
| width | height |
|---|---|
| nano_meter | milli_meter |
| 1 | 4 |
| 2 | 5 |
| 3 | 6 |
Back when I had the problem, it took me quite some time to figure out how to
set up the converters dictionary. When I look at it now, it’s actually pretty
easy. But I guess that’s just how it always is. Once you know how to do
something, it seems obvious and easy 🤓. Anyways, all you have to do is, use
tuples as keys in the dictionary. Here we go:
import pandas as pd
converters = {
("width", "nano_meter"): lambda nm: nm / 1_000_000_000,
("height", "milli_meter"): lambda mm: mm / 1_000,
}
data = pd.read_excel("<PATH/TO/EXCEL/FILE>", header=[0, 1], converters=converters)
print(data)
This short script produces the following output:
width height
nano_meter milli_meter
0 1.000000e-09 0.004
1 2.000000e-09 0.005
2 3.000000e-09 0.006
Of course after the conversion the units in the DataFrame are not correct
anymore. So you should rename them or remove them. Even though this was not
part of the exercise, here is how you can remove the units header:
data.columns = data.columns.droplevel(1)
Printing the data now yields the following:
width height
0 1.000000e-09 0.004
1 2.000000e-09 0.005
2 3.000000e-09 0.006
Take care,
Andreas
When I started out using Yocto, I was never sure about the best way to set up a directory structure for a project that uses Yocto. By now I do have a way to set up project directories that works for me. I thought, I’d share it. On one hand, as inspiration for others, and on the other hand, to possibly receive feedback from experts who can tell me if this makes sense or suggest improvements.
There are certain things that I like the project directory structure to support. They are:
.gitignore is fairly easy.
Ideally there are only a few directories that have to be added to
.gitignore and not a long list of direcoties and files that are scattered
all over the place.One thing I did to figure out how to set up a project directory structure was reading the Yocto Project Reference Manual [1]. Well, at least parts of it. In that manual you can find a chapter about the Source Directory Structure [2]. This chapter describes the internal directory structure of the poky directory that you can obtain by cloning the poky Git repository:
git clone git://git.yoctoproject.org/poky
But this directory structure is not what this article is about. The poky
directory that the previous command generates is just one part of the project
directory. I just mention the Source Directory
Structure [2] chapter of the Yocto
Project Reference Manual
[1], because it talks about the
build directory within the poky directory. Although I’m a big fan of
Yocto’s documentation (it is really good), I don’t like the idea of building
anything inside source directories. This is a general thing for me and not only
related to Yocto. Luckily the reference manual also mentions that you can run
“out of tree” builds which I much prefer.
All right, enough of the introduction. Here comes my current take on how to set up a project using Yocto. This might change over time and I would love to hear your thoughts about it. So don’t hesitate to contact me about it. Whether you do or don’t like my approach, any feedback is welcome.
Assume your current working directory contains a directory for your (embedded)
Linux project that uses Yocto as a build system and is called <project_dir>.
I like the project directory to be set up in such a way that running tree on
it, reveals the following:
$ tree -a -L 2 <project_dir>/
<project_dir>/
├── bitbake // ignored by Git
│ ├── downloads
│ └── sstate-cache
├── build // ignored by Git
│ └── conf
├── .git
│ └── (...) // output truncated
├── layers
│ ├── meta-bootstrap
│ ├── <meta-layer-1> // possibly added as git submodule
│ ├── <meta-layer-2> // possibly added as git submodule
│ ├── (...)
│ └── <meta-layer-n> // possibly added as git submodule
├── poky // added as git submodule
│ └── (...) // output truncated
├── .gitignore
├── .gitmodules
└── README.md
(...)
The following sections will describe each of the directories within the
<project_dir> in more detail.
The <project_dir> is the top level directory of the project and should be
named meaningfully. It is also the top-level directory for version control (the
.git directory resides in it). Besides some sub-directories which will be
discussed next, the project_dir contains three files:
.gitignore
The .gitignore file is used to ignore the bitbake and the build
directory. It might look like this:
# .gitignore
bitbake
build
.gitmodules
This file is generated by Git, when adding submodules.
README.md
Well, any decent project should have a README.md file giving an
introduction and possible further useful information about the project.
The bitbake directory is used for files downloaded or generated by bitbake.
Sources that bitbake downloads during a build process will go into the
downloads sub-directory. In order for that to work, bitbakes’s DL_DIR
variable has to be set appropriately in the build/conf/local.conf file. The
shared state cache files that bitbakes generates during a build process will
go into the sstate-cache sub-directory. In order for that to work,
bitbakes’s SSTATE_DIR variable has to be set appropriately in the
build/conf/local.conf file.
The bitbake directory is just an additional hierarchy level to keep the
<project_dir> clean and I feel that the two containing sub-directories kind
of belong into the same context.
Since everything in this directory is either downloaded or generated, the
bitbake directory is not under version control (ignored by git, see above:
.gitignore).
There may be cases when I want the downloads or sstate-cache direcetory to
be physically stored somewhere else, but not within the project itself. In such
case, I still like to have them in the bitbake directory, but I just use
symbolic links.
The build directory is were bitbakes builds software, packages that software,
and generates images. Everything in this directory is either generated by the
poke/os-init-build-env script (see section about the
meta-bootstrap directory) or during
bitbake build runs. Therefore this directory is not under version control
(ignored by git, see above: .gitignore).
Note: Even though everything in the build directory is generated, be
careful when deleting it. It should be possible to delete this directory and
recreate it fairly easily, but the conf sub-directory is a little bit
special. It is perfectly fine to edit the contents of the conf sub-directory,
mainly the local.conf and the bblayers.conf files during development. But
once you reach a state that you feel builds your project nicely, you should
make sure the configuration of your build environment is saved in your project
repository. This is what I like to use the meta-bootstrap layer for (see
corresponding section below). Before wiping out the build directory, make
sure you can automatically recreate the configuration files and you don’t lose
any local configuration that you might still need later on.
The directory that Git does its thing in. No need to worry about it too much. You just should not mess with it.
I’m not sure if this is just me, but to me it seems as if most people place
additional layers that are not part of the poky directory directly into the
<project_dir>. I like to introduce an additional level (layers) into the
directory hierarchy. To me this feels better: it keeps the top-level
<project_dir> cleaner and all layers (except the ones that belong to poke)
are within one directory.
The names <meta-layer-1> through <meta-layer-n> are just placeholders for
the real layer names. These layers may come from different sources:
The directory structure and the name of any layer within the layers directory
should follow the recommendations from the Yocto project. You can read about
how to create layers in the Yocto Project Development Tasks
Manual [4], specifically in the chapter about
Understanding and Creating
Layers [5].
The meta-bootstrap directory within layers is kind of special and will be
explained separately.
To set up a build environment or to start working with an existing build
environment (within the build directory), you usually go into the
<project_dir> and run:
source poky/oe-init-build-env build
If you don’t have a build environment yet and you have not set the
TEMPLATECONF environment variable, this sets up a more or less empty build
directory based on templates in the poky directory. All it contains are some
configuration files, namely:
These files are used by bitbake to build your project.
This is all fine, if you just start out with a project. But once you have set
up a working build environment, you what to have some means for new developers
or yourself to recreate the build environment automatically. This is where the
meta-bootstrap directory comes into play. It allows you to integrate a
template configuration into your project. There is a Yocto Project Development
Tasks Manual [4] that explains how this is done. It is
called Creating a Custom Template Configuration
Directory
[6].
I don’t want to go into what’s inside the poky directory, as it is very well
described here [2]. One way to get the
poky directory is to download a release tar ball from the Yocto Project
Website [7], but the What I wish I’d known about
Yocto Project [8] article recommends using Git instead. Compared
to the Introduction, I actually prefer adding it as a submodule to my projects.
The outlined project directory structure with its four “main” directories
(bitbake, build, layers, poky) fulfills all my requirements:
build/conf during the development. But everything that
is needed to generate a new clean build environment should be merged into the
meta-bootstrap layer anyways. Any
configuration in build/conf/local.conf that is
actually “local” to the current build host should not be part of the project
repository. An example for such a “local” configuration might be the setting
of Bitbake’s
BB_NUMBER_THREADS
[9] variable.bitbake or build.
This allows me to ignore these directories easily in
.gitignore and I can get rid of these directories without
the fear of loosing work (with the mentioned exception of some configuration
files in build/conf/.layers/meta-bootstrap
layer, I am able to quickly set up a new build environment and trigger a
build process within it.
Well, this is how I like to setup a project that uses Yocto. As mentioned
before, I would love to hear your feedback about it (either by email or as a
comment at the very bottom of this post).
2024-01-18:
downloads and the sstate-cache
directory outside the project directory.2024-01-19:
2024-01-22:
Take care,
Andreas
This is the fifth post in a series of posts (in a tutorial) about building an embedded Linux system for a Raspberry Pi 3 Model B+. A summary post for the tutorial can be found here. The summary post also talks about …
<project/root/dir>,After Building the Linux Kernel for a Raspberry Pi, it can be run on a Raspberry Pi. I used the same SD card that I had used to Run U-Boot on a Raspberry Pi, install the kernel on it, and than used U-Boot to boot the kernel.
It took me quite some time to make that work. Somehow most explanations about how to make that work, didn’t work for me (for some reason or another). Anyway, at the end and after some research I made it happen. The rest of the post explains how.
If you followed my description on Running U-Boot on a Raspberry Pi, you should be able to insert the SD card into your host system, run the following commands, and see four files on the card:
$> sudo mkdir -p /mnt/boot
$> sudo mount /dev/sdb1 /mnt/boot
$> ls /mnt/boot
bootcode.bin config.txt start.elf u-boot.bin
Now, we will copy two more files onto the SD card. One is the kernel itself, the other one is a device tree that the kernel needs to configure the hardware of the Raspberry Pi.
$> sudo cp <project/root/dir>/linux/arch/arm64/boot/Image /mnt/boot/
$> sudo cp <project/root/dir>/linux/arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b-plus.dtb /mnt/boot/
Now you can unmount the SD card from your host system:
$> sudo umount /mnt/boot
and place the card into the card holder of the Raspberry Pi. Once the SD card
is in the Raspberry Pi and you have a connection to the UART interface, you can
power up the board. You should see the same output as shown in the post Run
U-Boot on a Raspberry Pi. Once you
have the u-boot command prompt, the following commands make the kernel boot:
U-Boot> setenv bootargs 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rw rootwait
U-Boot> load mmc 0:1 $kernel_addr_r Image
15034880 bytes read in 1186 ms (12.1 MiB/s)
U-Boot> load mmc 0:1 $fdt_addr_r bcm2837-rpi-3-b-plus.dtb
21027 bytes read in 3 ms (6.7 MiB/s)
U-Boot> booti $kernel_addr_r - $fdt_addr_r
## Flattened Device Tree blob at 02600000
Booting using the fdt blob at 0x2600000
Working FDT set to 2600000
Using Device Tree in place at 0000000002600000, end 0000000002608222
Working FDT set to 2600000
Starting kernel ...
After Starting kernel ... more output should appear. It looks something like
this:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 4.19.127-v8-schuam+ (andreas@penguin) (gcc version 13.2.0 (crosstool-NG 1.26.0)) #1 5
[ 0.000000] Machine model: Raspberry Pi 3 Model B Plus Rev 1.3
... output shortened ...
[ 0.000000] Kernel command line: 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rw rt
... output shortened ...
[ 1.376872] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 1.385220] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 1.392808] devtmpfs: error mounting -2
[ 1.405762] Freeing unused kernel memory: 2880K
[ 1.410599] Run /sbin/init as init process
[ 1.414876] Run /etc/init as init process
[ 1.419093] Run /bin/init as init process
[ 1.423271] Run /bin/sh as init process
[ 1.427268] Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux .
... output shortened ...
[ 1.495295] ---[ end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. S-
The output stops with a Kernel panic message. Don’t panic yourself! This was
expected, as there is not Root filesystem on the SD card yet. Let’s Build a
Root Filesystem for a Raspberry Pi 3 Model
B+ next.
2025-02-01:
2025-01-20:
2023-11-13:
Take care,
Andreas