User:Vazhnov Alexey/Record GNSS traces

From OpenStreetMap Wiki
Jump to navigation Jump to search

Forum discussion: Wrocław: high precision GNSS tracks.

Workflow

  1. Charge the battery
  2. Put all the hardware to the vehicle
  3. Go outside of building
  4. Power on the SBC
  5. Wait till green LED is on
  6. Drive, have fun
  7. Return to the building
  8. Connect SBC to ethernet
  9. SSH into SBC
  10. Stop all scripts
  11. rsync all tracks
  12. Run Post processing
  13. Check resulting file in JOSM
  14. Upload to OSM

Hardware

Power consumption

It is possible to see information from current and voltage sensors (each multiplied by 1'000'000):

 $ cat /sys/power/axp_pmu/ac/amperage
354375
 $ cat /sys/power/axp_pmu/ac/voltage
5006500

So you can calculate power (A*V) by:

 $ echo "$(echo scale=3\; $(cat /sys/power/axp_pmu/ac/voltage) \* $(cat /sys/power/axp_pmu/ac/amperage)) / 10^12" | bc -l
1.968

CPU frequency

To decrease power consumption, lower CPU frequencies by editing /etc/default/cpufrequtils:

ENABLE=true
MIN_SPEED=144000  # was: 480000
MAX_SPEED=528000  # was: 1010000
GOVERNOR=ondemand

To check, run after reboot: cpufreq-info --human --stats

If read information from sensors, power consumption decreasing from 2.00-2.67W to 1.92-2.28W.

DRAM speed

Default Armbian settings are not bad:

 $ grep '^CONFIG_DRAM' /usr/lib/u-boot/A20-OLinuXino-Lime2-eMMC_defconfig | sort
CONFIG_DRAM_CLK=384
CONFIG_DRAM_DQS_GATING_DELAY=0
CONFIG_DRAM_EMR1=4
CONFIG_DRAM_MBUS_CLK=300
CONFIG_DRAM_SUN4I=y
CONFIG_DRAM_TIMINGS_VENDOR_MAGIC=y
CONFIG_DRAM_TPR3=0
CONFIG_DRAM_ZQ=127

Disable HDMI and SATA

TODO.

Software

Linux distribution: Armbian 21.02.3 Buster.

sudo apt-get -V --no-install-recommends -t buster-backports install gpsd gpsd-clients
# For ubxtool:
sudo apt-get -V --no-install-recommends install python3-gps python3-serial

udev rules

Package GPSD already has udev rules for u-blox receiver, /lib/udev/rules.d/60-gpsd.rules:

ATTRS{idVendor}=="1546", ATTRS{idProduct}=="01a9", SYMLINK+="gps%n", TAG+="systemd", ENV{SYSTEMD_WANTS}="gpsdctl@%k.service"

Nothing to do here.

gpsd

/etc/default/gpsd:

# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
DEVICES="/dev/gps0"
GPSD_OPTIONS="-n -r"
# -n			    = don't wait for client connects to poll GPS
# -r               	    = use GPS time even if no fix
USBAUTO="true"

gpsd autostart

In Debian, by default, GPSD doesn't start automatically on OS boot. The daemon starts when any client tries to connect to GPSD socket:

 $ systemctl list-sockets --all 'gpsd*'
LISTEN             UNIT        ACTIVATES
/var/run/gpsd.sock gpsd.socket gpsd.service
127.0.0.1:2947     gpsd.socket gpsd.service
[::1]:2947         gpsd.socket gpsd.service

3 sockets listed.

This is OK for our purposes.

gpsd test

It is better to do a test here. Run cgps. After a few minutes, you should see a coordinates.

chrony

Documentation for both NTP and chrony: GPSD time service HOWTO.

For system time synchronization with GPSD, add to /etc/chrony/chrony.conf:

allow
# set larger delay to allow the NMEA source to overlap with
# the other sources and avoid the falseticker status
refclock SHM 0 refid GPS precision 1e-1 offset 0.9999 delay 0.2
refclock SHM 1 refid PPS precision 1e-7

Check with ntpshmmon (script from GPSD package), that GPSD sends information:

 $ sudo ntpshmmon -o -n5
ntpshmmon: version 3.20
#      Name     Offset           Clock                Real                 L Prc
sample NTP0          0.040272238  1623360368.040470350  1623360368.000198112 0 -20
sample NTP0          0.039400083  1623360369.039597848  1623360369.000197765 0 -20
sample NTP0          0.041342620  1623360370.041540039  1623360370.000197419 0 -20
sample NTP0          0.042419833  1623360371.042616905  1623360371.000197072 0 -20
sample NTP0          0.043458045  1623360372.043654770  1623360372.000196725 0 -20

Then check with chronyc sources (add -v for a legend/help), in my case remote server was selected as the most precision time source:

 $ chronyc sources
210 Number of sources = 6
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#x GPS                           0   4   377    20   -958ms[ -958ms] +/-  200ms
#? PPS                           0   4     0     -     +0ns[   +0ns] +/-    0ns
^* svl1.ntp.netnod.se            1   8   377    97  +2127us[+2167us] +/-   23ms
…

/etc/default/gnss_scripts

Put some settings for a scripts and systemd unit into /etc/default/gnss_scripts:

# If GNSS_DEVICE changed, update also `ConditionPathExists=` in systemd unit file with
# sudo systemctl edit gpspipe.service

# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
GNSS_DEVICE='/dev/gps0'
NMEA_LOG_DIR='/tmp/gpspipe'
NMEA_ARCHIVE_DIR='/var/log/gpspipe'
WEBGPS_WORKDIR='/tmp/webgps'
WEBGPS_ARGS='--no-strict-version-check c'

Scripts user

sudo useradd --system --no-create-home --groups dialout --comment 'GPSD GNSS scripts' gnss_scripts
sudo mkdir -pv /var/log/gpspipe
sudo chown gnss_scripts:dialout /var/log/gpspipe

Scripts autostart

Here you have two options, classic /etc/rc.local or Systemd:

Systemd

Create unit:

sudo systemctl edit --force --full gpspipe.service
[Unit]
Description=Write NMEA logs from GPSD
Documentation=https://wiki.openstreetmap.org/wiki/User:Vazhnov_Alexey/Record_GNSS_traces
# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
ConditionPathExists=/dev/gps0
Requires=gpsd.service

[Service]
Type=simple
# Restart=always
# Restart=on-failure
Restart=on-success
MemoryMax=100M
User=gnss_scripts
Group=gnss_scripts
EnvironmentFile=/etc/default/gnss_scripts
ExecStartPre=mkdir -pv "$NMEA_LOG_DIR"
ExecStart=sh -c 'gpspipe -r -n 10000 -o "$NMEA_LOG_DIR/gpspipe_$(date --iso-8601=seconds).nmea"'

[Install]
WantedBy=multi-user.target

This is up to you, enable this service to start on system boot up or not. If yes:

sudo systemctl enable gpspipe.service
gzip and save files on shutdown
sudo systemctl edit --force --full gpspipe_archive_on_shutdown.service
[Unit]
Description=Archive NMEA logs on shutdown
Documentation=https://wiki.openstreetmap.org/wiki/User:Vazhnov_Alexey/Record_GNSS_traces
After=network.target

[Service]
Type=oneshot
RemainAfterExit=true
EnvironmentFile=/etc/default/gnss_scripts
ExecStart=/bin/true
ExecStop=sh -c 'if [ "$(cat $NMEA_LOG_DIR/*.nmea | wc -l)" -gt 200 ]; then gzip --keep $NMEA_LOG_DIR/*.nmea; mkdir -pv $NMEA_ARCHIVE_DIR; mv $NMEA_LOG_DIR/*.nmea.gz $NMEA_ARCHIVE_DIR/; fi'
TimeoutStopSec=2min

[Install]
WantedBy=multi-user.target
# As I understand, you cannot use 'WantedBy=shutdown.target' and 'Before=umount.target' together.
sudo systemctl daemon-reload
sudo systemctl reenable gpspipe_archive_on_shutdown.service

Classic /etc/rc.local

Sometimes in the code below, echo is using to print something to the hardware RS232 port, which is useful for debugging.

/etc/rc.local:

#!/usr/bin/env bash
echo "Current date (code from /etc/rc.local): $(date --rfc-email)"
sudo -u gnss_scripts bash /path/to/gpspipe_start.sh &
sudo -u gnss_scripts bash /path/to/gpsrinex_start.sh &
exit 0

gpspipe_start.sh

Script gpspipe_start.sh periodically creates a new files, to not lose whole track in case of hard poweroff:

#!/usr/bin/env bash

set -o nounset
# set -o errexit
set -o pipefail
shopt -s dotglob

# Name `gps0` is set in `/lib/udev/rules.d/60-gpsd.rules` and usually points to `/dev/ttyACM0`.
GNSS_DEVICE="/dev/gps0"
NMEA_LOG_DIR="/tmp/gpspipe"
source /etc/default/gnss_scripts

sleep 10
mkdir -pv "$NMEA_LOG_DIR"

# If file exists and is a character special file:
if [ -c "$GNSS_DEVICE" ]; then
  while true; do
    echo "Current date (code from gpspipe_start.sh): $(date --rfc-email)"
    echo "Output of chronyc sources -v:"
    chronyc sources -v
    gpspipe -r -n 5000 -o "$NMEA_LOG_DIR/gpspipe_$(date --iso-8601=seconds).nmea"
    EXIT_CODE="$?"
    echo "gpspipe_start.sh: exit code was: $EXIT_CODE"
    if [ "$EXIT_CODE" -ne 0 ]; then
      sleep 5
    fi
  done
else
  echo "gpspipe_start.sh: no $GNSS_DEVICE file found, exiting."
fi

Post processing

nmea_rename_by_first_timestamp.sh

After system boot, time often is not synced. When first NMEA file is creating, it can receive wrong filename. The order of NMEA files can be wrong because of this.

This script renames files accordingly to information inside first GPZDA record: https://gitlab.com/vazhnov/gnss-web-control/-/blob/main/scripts/nmea_rename_by_first_timestamp.sh

gpx_filter.sh

See also: GPSBabel/Using filters, https://help.openstreetmap.org/questions/79521/filter-out-gnss-trackpoints-by-radius

This script does:

  • concatenate all NMEA files with filenames starting like "gpspipe_2021-04-04",
  • sort points by timestamp (but -x sort,time doesn't work for me in GPSBabel 1.7.0),
  • filter out track points with HDOP < 2 or satellites number < 12,
  • filter out track points near my house, because there are some noise when I start/finish recording,
  • save file in GPX format.

gpx_filter.sh:

#!/usr/bin/env bash
set -o nounset
set -o errexit
set -o pipefail
shopt -s dotglob

PREFIX="$1"

IN_FILES=""
for IN_FILE in ./"$PREFIX"*.nmea; do
  IN_FILES="$IN_FILES -f $IN_FILE"
done

# shellcheck disable=SC2086 # Double quote to prevent globbing and word splitting.
gpsbabel -t -i nmea $IN_FILES \
    -x sort,time \
    -x discard,hdop=2 \
    -x discard,sat=12 \
    -x transform,wpt=trk,del \
    -x radius,distance=0.1K,lat=51.0754,lon=17.0398,exclude,nosort \
    -x transform,trk=wpt,del \
    -o gpx -F "${PREFIX}_hdop2_sat12_radius.gpx"
sed -i -E -- '/\<name\>|\<cmt\>|\<desc\>/d' "${PREFIX}_hdop2_sat12_radius.gpx"

Filesystem

It is better to have separate disk partitions for /var/logs. I tried f2fs: mkfs.f2fs works fine, mount works fine. But when I do ls, I got an error:

ls: cannot access '/var/log/': Invalid argument

So I use ext4 for /var/logs.

sudo mkfs.ext4 -L "MicroSD_32GB_var_log" -M "/var/log" /dev/mmcblk0p2

/etc/fstab:

LABEL="MicroSD_32GB_var"			/var/log	ext4	nosuid,noatime,nodiratime,errors=remount-ro,commit=180	0 0

Bluetooth and Python/Flask web-interface

See User:Vazhnov Alexey/Record GNSS traces/Bluetooth and web panel.

RTK / RAW / NMEA / GPX

I didn't find a free source of correction data ("RTK"?) in Wrocław. After creating first NMEA tracks, I satisfied with their quality and decided to not record RAW binary information from receiver for post-processing.

Result: GPX tracks

Uploaded GPS tracks: https://www.openstreetmap.org/user/Alexey%20Vazhnov/traces/tag/u-blox%20zed-f9p

TODO

  • Better metal ground plane
  • Current setup spends whole bag, I need to make it more compact
  • Better power supply: my SBC can eat more than 1A when initializing, and my powerbank switches off sometimes on start
  • Replace SBC with a simple logger
  • Add Bluetooth to have connection with Android smartphone
  • Better USB connector(s): sometimes logging stops and I see this in /var/log/syslog:
May  4 13:34:50 lime2 kernel: [    4.741656] usb 4-1: new full-speed USB device number 3 using ohci-platform
May  4 13:34:50 lime2 kernel: [    4.968626] usb 4-1: New USB device found, idVendor=1546, idProduct=01a9, bcdDevice= 1.00
May  4 13:34:50 lime2 kernel: [    4.968648] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
May  4 13:34:50 lime2 kernel: [    4.968658] usb 4-1: Product: u-blox GNSS receiver
May  4 13:34:50 lime2 kernel: [    4.968665] usb 4-1: Manufacturer: u-blox AG - www.u-blox.com
…
May  4 14:47:02 lime2 kernel: [ 4369.671459] usb 4-1: USB disconnect, device number 3
May  4 14:47:02 lime2 kernel: [ 4369.672705] pps pps0: removed
May  4 14:47:02 lime2 kernel: [ 4369.673165] cdc_acm 4-1:1.0: failed to set dtr/rts
May  4 14:49:09 lime2 chronyd[1119]: Can't synchronise: no selectable sources