User:Vazhnov Alexey/Record GNSS traces

From OpenStreetMap Wiki
Jump to navigation Jump to search

Forum discussion: Wrocław: high precision GNSS tracks.

Intro

TBD.

Another projects

Workflow

  1. Charge the battery
  2. Put all the hardware to the vehicle
  3. Go outside of building
  4. Power on the SBC
  5. Wait till green LED is on
  6. Drive, have fun
  7. Return to the building
  8. Connect SBC to ethernet
  9. SSH into SBC
  10. Stop all scripts
  11. rsync all tracks
  12. Run Post processing
  13. Check resulting file in JOSM
  14. Upload to OSM

If Ethernet cable is connected when OS starts, then "Home mode" activated:

  • no gpspipe to be running.

Hardware

  • Receiver SparkFun GPS-RTK-SMA ZED-F9P
  • IP67 active antenna: U-Blox ANN-MB-00-00, with 5 meters cable
  • Single board computer: Freescale (NXP) i.MX53 Quick Start board (1 GB RAM), it uses 5V x 0.3-0.5A when running
  • Single board computer: Olimex A20-OLinuXino-LIME2, SoC Allwinner A20
  • Rechargable LI-PO battery 3.7V 6600mAh with JST connector (3x18650)
  • Noname powerbank with 3 x 18650 and TPOWER TP4366 controller (1A maximum)
  • Bluetooth USB-BT4: 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (CSR8510 A10)
  • Wi-Fi USB dongle TP-Link TL-WN725N v1 (chip RTL8188EUS);
  • Case: Obudowa plastikowa Kradex wentylowana 159x139x59mm;
  • Power switch with LED: Przełącznik ON-OFF monostabilny - PBW-12BPW;
  • Rainproof bag: https://www.sportarsenal.cz/brasny/art--310/

Power consumption

See also: https://linux-sunxi.org/AXP209/PMIC_control_Linux.

It is possible to see information from current and voltage sensors (each multiplied by 1'000'000): 

 $ cat /sys/power/axp_pmu/ac/amperage
354375
 $ cat /sys/power/axp_pmu/ac/voltage
5006500

So you can calculate power (A*V) by: 

 $ echo "$(echo scale=3\; $(cat /sys/power/axp_pmu/ac/voltage) \* $(cat /sys/power/axp_pmu/ac/amperage)) / 10^12" | bc -l
1.968

CPU frequency

To decrease power consumption, lower CPU frequencies by editing /etc/default/cpufrequtils: 

ENABLE=true
MIN_SPEED=144000  # was: 480000
MAX_SPEED=528000  # was: 1010000
GOVERNOR=ondemand

To check, run after reboot: cpufreq-info --human --stats

If read information from sensors, power consumption decreasing from 2.00-2.67W to 1.92-2.28W.

DRAM speed

Default Armbian settings are not bad: 

 $ grep '^CONFIG_DRAM' /usr/lib/u-boot/A20-OLinuXino-Lime2-eMMC_defconfig | sort
CONFIG_DRAM_CLK=384
CONFIG_DRAM_DQS_GATING_DELAY=0
CONFIG_DRAM_EMR1=4
CONFIG_DRAM_MBUS_CLK=300
CONFIG_DRAM_SUN4I=y
CONFIG_DRAM_TIMINGS_VENDOR_MAGIC=y
CONFIG_DRAM_TPR3=0
CONFIG_DRAM_ZQ=127

Disable HDMI and SATA

TODO.

Software

Linux distribution: Armbian 21.02.3 Buster.

Ansible installer

It is possible to do almost everything described in this page, by running an Ansible playbook.

See https://gitlab.com/vazhnov/gnss-web-control-panel-ansible

gpsd package

sudo apt-get -V --no-install-recommends -t buster-backports install gpsd gpsd-clients
# For ubxtool:
sudo apt-get -V --no-install-recommends install python3-gps python3-serial

udev rules

Package GPSD already has udev rules for u-blox receiver, /lib/udev/rules.d/60-gpsd.rules: 

ATTRS{idVendor}=="1546", ATTRS{idProduct}=="01a9", SYMLINK+="gps%n", TAG+="systemd", ENV{SYSTEMD_WANTS}="gpsdctl@%k.service"

Just check these UDEV rules works: 

 $ sudo udevadm trigger --subsystem-match=tty --action=change
 $ ls -l /dev/gps*
lrwxrwxrwx 1 root root  7 Aug  2  2021 /dev/gps0 -> ttyACM0

gpsd

/etc/default/gpsd: 

# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
DEVICES="/dev/gps0"
GPSD_OPTIONS="-n -r"
# -n			    = don't wait for client connects to poll GPS
# -r               	    = use GPS time even if no fix
USBAUTO="true"

gpsd autostart

In Debian, by default, GPSD doesn't start automatically on OS boot. The daemon starts when any client tries to connect to GPSD socket: 

 $ systemctl list-sockets --all 'gpsd*'
LISTEN             UNIT        ACTIVATES
/var/run/gpsd.sock gpsd.socket gpsd.service
127.0.0.1:2947     gpsd.socket gpsd.service
[::1]:2947         gpsd.socket gpsd.service

3 sockets listed.

This is OK for our purposes.

gpsd test

It is better to do a test here. Run cgps. After a few minutes, you should see a coordinates.

chrony

Documentation for both NTP and chrony: GPSD time service HOWTO.

For system time synchronization with GPSD, add to /etc/chrony/chrony.conf: 

allow
# set larger delay to allow the NMEA source to overlap with
# the other sources and avoid the falseticker status
refclock SHM 0 refid GPS precision 1e-1 offset 0.9999 delay 0.2
refclock SHM 1 refid PPS precision 1e-7

Check with ntpshmmon (script from GPSD package), that GPSD sends information: 

 $ sudo ntpshmmon -o -n5
ntpshmmon: version 3.20
#      Name     Offset           Clock                Real                 L Prc
sample NTP0          0.040272238  1623360368.040470350  1623360368.000198112 0 -20
sample NTP0          0.039400083  1623360369.039597848  1623360369.000197765 0 -20
sample NTP0          0.041342620  1623360370.041540039  1623360370.000197419 0 -20
sample NTP0          0.042419833  1623360371.042616905  1623360371.000197072 0 -20
sample NTP0          0.043458045  1623360372.043654770  1623360372.000196725 0 -20

Then check with chronyc sources (add -v for a legend/help), in my case remote server was selected as the most precision time source: 

 $ chronyc sources
210 Number of sources = 6
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#x GPS                           0   4   377    20   -958ms[ -958ms] +/-  200ms
#? PPS                           0   4     0     -     +0ns[   +0ns] +/-    0ns
^* svl1.ntp.netnod.se            1   8   377    97  +2127us[+2167us] +/-   23ms
…

/etc/default/gnss_scripts

Put some settings for a scripts and systemd unit into /etc/default/gnss_scripts: 

# If GNSS_DEVICE changed, update also `ConditionPathExists=` in systemd unit file with
# sudo systemctl edit gpspipe-nmea.service

# When to start shutdown process. Used by `cron_check_battery_shutdown.sh`.
BATTERY_PERCENT_MIN='10'

# `NET_DEV` is using in `start_services.sh` to stop `gpspipe` if Ethernet cable is connected - home mode
NET_DEV='eth0'

# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
GNSS_DEVICE='/dev/gps0'
NMEA_LOG_DIR='/tmp/gpspipe'
NMEA_ARCHIVE_DIR='/var/log/gpspipe'
GPSPIPE_EXTRA_ARGS='--nmea -n 10000'

# webgps: https://gitlab.com/gpsd/gpsd/-/blob/master/contrib/webgps.py.in
WEBGPS_WORKDIR='/tmp/webgps'
# Use `--no-html-head` if you want to include the result somewhere else
WEBGPS_ARGS='--no-strict-version-check c'

Create user account for scripts

sudo useradd --system --no-create-home --groups dialout --comment 'GPSD GNSS scripts' gnss_scripts
sudo mkdir -pv /var/log/gpspipe
sudo chown gnss_scripts:dialout /var/log/gpspipe

Scripts autostart

Both classic /etc/rc.local and Systemd are in use:

Systemd

Create unit: 

sudo systemctl edit --force --full gpspipe-nmea.service

[Unit]
Description=Write NMEA logs from GPSD
Documentation=https://wiki.openstreetmap.org/wiki/User:Vazhnov_Alexey/Record_GNSS_traces
# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
ConditionPathExists=/dev/gps0
Requires=gpsd.service

[Service]
Type=simple
# Restart=always
# Restart=on-failure
Restart=on-success
MemoryMax=100M
User=gnss_scripts
Group=gnss_scripts
EnvironmentFile=/etc/default/gnss_scripts
ExecStartPre=mkdir -pv "$NMEA_LOG_DIR"
ExecStart=sh -c 'gpspipe $GPSPIPE_EXTRA_ARGS -o "$NMEA_LOG_DIR/gpspipe_$(date --iso-8601=seconds).nmea"'

[Install]
WantedBy=multi-user.target

sudo systemctl edit --force --full gpspipe-raw.service

[Unit]
Description=Write RAW logs from GPSD
Documentation=https://wiki.openstreetmap.org/wiki/User:Vazhnov_Alexey/Record_GNSS_traces
# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
ConditionPathExists=/dev/gps0
Requires=gpsd.service

[Service]
Type=simple
Restart=on-success
MemoryMax=100M
User=gnss_scripts
Group=gnss_scripts
EnvironmentFile=/etc/default/gnss_scripts
Environment="UBXOPTS=-P 27.12"
ExecStartPre=mkdir -pv "$NMEA_LOG_DIR"
ExecStartPre=ubxtool -e RAWX
ExecStart=sh -c 'gpspipe --raw -o "$NMEA_LOG_DIR/gpspipe_$(date --iso-8601=seconds).ubx"'

[Install]
WantedBy=multi-user.target

This is up to you, enable this service to start on system boot up or not. If yes: 

sudo systemctl enable gpspipe-nmea.service
sudo systemctl enable gpspipe-raw.service
gzip and save files on shutdown
sudo systemctl edit --force --full gpspipe_archive_on_shutdown.service

[Unit]
Description=Archive NMEA logs on shutdown
Documentation=https://wiki.openstreetmap.org/wiki/User:Vazhnov_Alexey/Record_GNSS_traces
After=network.target

[Service]
Type=oneshot
RemainAfterExit=true
EnvironmentFile=/etc/default/gnss_scripts
ExecStart=/bin/true
ExecStop=sh -c 'if [ "$(cat $NMEA_LOG_DIR/*.nmea | wc -l)" -gt 200 ]; then gzip --keep $NMEA_LOG_DIR/*.nmea; mkdir -pv $NMEA_ARCHIVE_DIR; mv $NMEA_LOG_DIR/*.nmea.gz $NMEA_ARCHIVE_DIR/; fi'
TimeoutStopSec=2min

[Install]
WantedBy=multi-user.target
# As I understand, you cannot use 'WantedBy=shutdown.target' and 'Before=umount.target' together.

sudo systemctl daemon-reload
sudo systemctl reenable gpspipe_archive_on_shutdown.service

Home mode (Ethernet cable is connected)

/etc/rc.local: 

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

bash /var/lib/gnss_web_control/code-repository/scripts/start_services.sh &

exit 0

Armbian automatic shutdown on low battery power

(merged into mainline Armbian: https://github.com/armbian/build/pull/3084, wait for next release)

Script cron_check_battery_shutdown.sh should be executed every 5 minutes from a cron. It initiates a system shutdown if battery is discharging and battery power is less than 10% (can be changed in variable BATTERY_PERCENT_MIN from /etc/default/gnss_scripts).

It uses function batteryinfo() from Armbian file 30-armbian-sysinfo.

Create a file /etc/cron.d/armbian-check-battery like this: 

*/5 * * * * root bash /usr/lib/armbian/armbian-check-battery-shutdown

Post processing

nmea_rename_by_first_timestamp.sh

After system boot, time often is not synced. When first NMEA file is creating, it can receive wrong filename. The order of NMEA files can be wrong because of this.

This script renames files accordingly to information inside first GPZDA record: https://gitlab.com/vazhnov/gnss-web-control/-/blob/main/scripts/nmea_rename_by_first_timestamp.sh

gpx_filter.sh

See also: GPSBabel/Using filters, https://help.openstreetmap.org/questions/79521/filter-out-gnss-trackpoints-by-radius

This script does:

  • concatenate all NMEA files with filenames starting like "gpspipe_2021-04-04",
  • sort points by timestamp (but -x sort,time doesn't work for me in GPSBabel 1.7.0),
  • filter out track points with HDOP < 2 or satellites number < 12,
  • filter out track points near my house, because there are some noise when I start/finish recording,
  • save file in GPX format.

gpx_filter.sh: 

#!/usr/bin/env bash
set -o nounset
set -o errexit
set -o pipefail
shopt -s dotglob

PREFIX="$1"

IN_FILES=""
for IN_FILE in ./"$PREFIX"*.nmea; do
  IN_FILES="$IN_FILES -f $IN_FILE"
done

# shellcheck disable=SC2086 # Double quote to prevent globbing and word splitting.
gpsbabel -t -i nmea $IN_FILES \
    -x sort,time \
    -x discard,hdop=2 \
    -x discard,sat=12 \
    -x transform,wpt=trk,del \
    -x radius,distance=0.1K,lat=51.0754,lon=17.0398,exclude,nosort \
    -x transform,trk=wpt,del \
    -o gpx -F "${PREFIX}_hdop2_sat12_radius.gpx"
sed -i -E -- '/\<name\>|\<cmt\>|\<desc\>/d' "${PREFIX}_hdop2_sat12_radius.gpx"

Filesystem

It is better to have separate disk partitions for /var/logs. I tried f2fs: mkfs.f2fs works fine, mount works fine. But when I do ls, I got an error: 

ls: cannot access '/var/log/': Invalid argument

So I use ext4 for /var/logs. 

sudo mkfs.ext4 -L "MicroSD_32GB_var_log" -M "/var/log" /dev/mmcblk0p2

/etc/fstab: 

LABEL="MicroSD_32GB_var"			/var/log	ext4	nosuid,noatime,nodiratime,errors=remount-ro,commit=180	0 0

Bluetooth and Python/Flask web-interface

See User:Vazhnov Alexey/Record GNSS traces/Bluetooth and web panel.

RTK / RAW / NMEA / GPX

I didn't find a free source of correction data ("RTK"?) in Wrocław. After creating first NMEA tracks, I satisfied with their quality and decided to not record RAW binary information from receiver for post-processing.

Result: GPX tracks

Uploaded GPS tracks: https://www.openstreetmap.org/user/Alexey%20Vazhnov/traces/tag/u-blox%20zed-f9p

TODO

  • Better metal ground plane
  • Current setup spends whole bag, I need to make it more compact
  • Better power supply: my SBC can eat more than 1A when initializing, and my powerbank switches off sometimes on start
  • Replace SBC with a simple logger
  • Add Bluetooth to have connection with Android smartphone
  • Better USB connector(s): sometimes logging stops and I see this in /var/log/syslog:
May  4 13:34:50 lime2 kernel: [    4.741656] usb 4-1: new full-speed USB device number 3 using ohci-platform
May  4 13:34:50 lime2 kernel: [    4.968626] usb 4-1: New USB device found, idVendor=1546, idProduct=01a9, bcdDevice= 1.00
May  4 13:34:50 lime2 kernel: [    4.968648] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
May  4 13:34:50 lime2 kernel: [    4.968658] usb 4-1: Product: u-blox GNSS receiver
May  4 13:34:50 lime2 kernel: [    4.968665] usb 4-1: Manufacturer: u-blox AG - www.u-blox.com
…
May  4 14:47:02 lime2 kernel: [ 4369.671459] usb 4-1: USB disconnect, device number 3
May  4 14:47:02 lime2 kernel: [ 4369.672705] pps pps0: removed
May  4 14:47:02 lime2 kernel: [ 4369.673165] cdc_acm 4-1:1.0: failed to set dtr/rts
May  4 14:49:09 lime2 chronyd[1119]: Can't synchronise: no selectable sources
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=User:Vazhnov_Alexey/Record_GNSS_traces&oldid=2547903"