Kubernetes cluster for manufacturing engineering: tale of an epic commissioning

- Introduction A quite big company that produces parts for some of the most important automotive industry companies was interested in a cloud-based system to monitor the overall efficiency of their production machines, analyze some key parameters and optimize the production activities scheduling. Goal of the project was to connect eleven industrial manufacturing machines to the cloud, extract specific machine data and develop a web application for the visualization and management of such data, while guaranteeing information confidentiality and security. Furthermore, one fundamental requirement of such system has been the bidirectional integration with the customer’s ERP system, in order to synchronize the production JOBs and manage their execution on the corresponding machines. To fulfill such requirements, the project team has engineered and then implemented an hybrid edge-cloud solution in which the software has been packed into various containers that are orchestrated and managed, at the edge level, by a Kubernetes cluster. This technology ensures an optimal load balancing between the available resources as well as a high availability in case of hardware or software failures. While IT enterprises do not question the value of containerized applications anymore, the use of such kind of technologies within a manufacturing environment hasn’t been completely explored yet. In the following paragraphs we will go into details on how we engineered and built the system despite all the difficulties we had to overcome. …

Posted on

FreeBSD, Nginx and htpasswd file generation

On most Linux distros it is possible to generate the htpasswd file entries simply using the htpasswd command line util. On FreeBSD the easiest way to accomplish the same task is using OpenSSL itself: openssl passwd -apr1 Enter the password twice to get a nice hash, copy that in the htpasswd file with the usual user:password_hash syntax.

Posted on

Disable head parking Western Digital drives

Most Western Digital hard drives’ firmware let the heads park themselves after a certain amount of seconds in case the disk is not actively performing any operation. This might be useful to keep power consumption under control but is actually harmful for disks that run 24/7 (WD Red for example). Luckily there is a way to disable head parking, this can be done directly from Linux using a tool called idle3ctl. …

Posted on

FreeBSD, NGINX and TLSv1.3

After a six months hiatus here is a new blogpost. This saturday I finally found the time to upgrade the configuration of the server that hosts this very website. Software stack is pretty simple: FreeBSD (version 12.0-p6),nginx (version 1.15.10) and OpenSSL (version 1.1.1a-freebsd). Install the required software: $ pkg install nginx-devel py36-certbot Get a SSL certificate from letsencrypt: $ certbot-3.6 certonly --standalone -d domain.tld -d www.domain.tld Certfiles location is /usr/local/etc/letsencrypt/live/<domain.tld>, you might, or might not, want to move them to another directory. …

Posted on

qemu/KVM PCI passthrough

PCI passthrough is the process of attaching a PCI-E device directly to a VM; CPU support (namely VT-D for Intel and AMD-V for AMD) and motherboard support (IOMMU) are required for PCI passthrough to work properly. Hardware configuration used: AMD Ryzen 1700x Gigabyte X370 K7 Nvidia Geforce GTX260 32 GiB of RAM and a few HDDs Fedora 29 as host OS The system only has a single graphic card because it is normally used as headless compute server for which a GPU is not really required; the graphic card is also very very old Nvidia Geforce GTX260 with a standard non UEFI BIOS. If using an UEFI enabled graphic card it is probably required to install the OS in UEFI mode using a virtual UEFI BIOS. This guide assumes you already have a working Windows virtual machine and are familiar with libvirt. First of all, edit GRUB to enable IOMMU and blacklist nouveau kernel module so that the graphic card is not picked up anymore by the host: …

Posted on

Remotely unlock a full disk encrypted Fedora/CentOS server

The idea here is to be able to power on and unlock a remote Full Disk Encrypted (FDE from now on) server. I will leave the how “remotely power on” to the reader to figure out and focus on the other part. The easiest way to accomplish it is by using a program called: dracut-crypt-ssh. $ yum install dropbear dracut dracut-network openssh libblkid-devel gcc $ git clone https://github.com/dracut-crypt-ssh/dracut-crypt-ssh.git $ cd dracut-crypt-ssh $ ./configure $ make $ sudo make install After compiling and installing dracut-crypt-ssh configure grub to instruct dracut to add networking to initramfs: …

Posted on

Resize QCOW2 disk image

QCOW2 disk images can be easily grown using libvirt command line utils. Unfortunately it isn’t possible to grow QCOW2 images in-place or online. First of all, power off the virtual machine, grow the file and make a copy of it: $ qemu-img resize image.qcow2 +200G $ cp image.qcow2 image-new.qcow2 Identify the specific partion you intend to grow: $ virt-filesystems -a image.qcow2 -l Name Type VFS Label Size Parent /dev/sda1 filesystem ext4 - 536870912 - /dev/sda3 filesystem xfs - 45885612000 - Expand the actual partition: …

Posted on

Intel CPU, Hyper-Threading and Spectre STIBP mitigation

Yesterday I was reading phoronix 0 and phoronix 1 articles on STIBP mitigation impact on CPU performance, since I run a pretty old laptop equiped with a Sandy Bridge CPU I figured that I should do my own tests to see how bad things really are or aren’t. CPU: Intel Core i3-2310M - 2 cores / 4 threads Motherboard: Lenovo Thinkpad RAM: 2x4 GB DDR3 @1333 MHz HDD: Plextor M5pro OS: Fedora 29 x86_64 with stock kernels My benchmark of choice is compiling the Linux kernel (version 4.19.2). What I do is download the kernel version to /dev/shm ramdisk and compile it using the defconfig configuration while checking how many seconds it takes to complete the task. …

Posted on

Xorg present flip failed

In the last couple of months Xorg has been crashing more or less on a daily basis. What happens is that while you are there browsing the internet, or certain times even doing literally nothing, Xorg crashes and after a second of black screen the user is sent back to the login page. Hardware configuration of my machine is: CPU: AMD Ryzen 7 1700x Motherboard: Gigabyte X370 K7 - BIOS F23d RAM: 2x16 GB DDR4 HDD: Samsung 850 Pro GPU: Nvidia GTX260 Monitor 0: Dell U2412M connected via DVI-D Monitor 1: Dell U2412M connected via HDMI-DVI cable OS: Fedora 28 and Fedora 29 x86_64 GPU driver: nouveau, various versions others: varius versions of Linux, Xorg, mesa, etc I don’t think having two monitors is the culprit nor using a very old Nvidia graphic card is because a friend of mine runs a completely different system (AMD Radeon RX480, single monitor connected using Display Port) and still suffer from the very same problem. Upgrading from Fedora 28 to Fedora 29 nor installing updates in a timely manner solved the issue for me; searching on the interwebz also did not yeld any result. After yet another crash, today I finally decided it was time to investigate the issue. First thing I noticed is that Xorg log file is literally spammed with the following error: …

Posted on

FreeBSD network performance on KVM/Qemu

Today I red an article that was comparing Fedora 29 and FreeBSD 11.2 network performance in a KVM/Qemu environment. Since I use KVM/Qemu and also Fedora and Freebsd I powered on a couple of vm and did my own tests. Results are quite interesting, I expected FreeBSD to be faster but it turns out Fedora 29 actually is. Host system configuration: CPU: Ryzen 7 1700x @4 GHz Motherboard: Gigabyte X370 K7 - BIOS F23d RAM: 2x16 GB DDR4 @3133 MHz CAS 16 HDD: Some Samsung SSD Operating systems Host: Fedora 29 x86_64 Fedora VM0: Fedora 29 X86_64 Fedora VM1: Fedora 29 X86_64 FreeBSD VM0: FreeBSD 11.2 x86_64 FreeBSD VM1: FreeBSD 11.2 x86_64 Virtualization techonology: Qemu+KVM Linux kernel version: 4.18.17-300.fc29.x86_64 What I did was setup 2 hidentical Fedora 29 virtual machines and 2 hidentical FreeBSD 11.2 virtual machines, every one of them had iperf3 installed on it. …

Posted on