General Purpose to Special Purpose

I've been working on DMS, my driving management system. It's a "basic app" with a primary goal, tell me how far each of the cities are along the route when I'm taking a road trip. 

I started with using Google Distance Matrix API, however after a heavy usage month and having to pay, I thought, I have computers, maybe I can find a way to do that myself. Enter Open Street Maps. Cool, but well.....I am gonna need ram.

Historically, I used a raspberry pi for the whole operation. Actually scratch that, in the beginning I was using Python everywhere and single page apps, which meant I didn't need my own host, but I also had to go keep checking in on that app, and decided I'd go local.

Ok, back on topic, the title of this post is General Purpose to Special Purpose, and well this is to talk about my journey of using more general purpose compute than was necessary, and slowly but surely shrinking it to my special purpose needs.

If you look up the requirements to run the Open Street Maps Routing Machine (OSRM) , for the USA, it's something less than 64 GB but over 50 GB which, if I am being honest...is a bit bigger than my raspberry pi has in ram (Raspberry Pi 4 with 4 GB of ram). 

As a rule of thumb, the RAM needed to run OSRM is about 5x the file size of the map you are using. So for a map of the USA us-latest.osm.pbf that is 9.3 GB in size, you'll need about 50 GB of memory*

 My first iteration was to use my laptop to host both OSRM, and Nominatim, a lat/lon to city and vice versa lookup tool. This worked for my needs, but means I gotta lug that monster around and waste energy and all that stuff....so what to do, what to do.

Well first up. Was trying to figure out if I can use different databases. Like what if I sorted out that I'm only going to be in one state the entire time so I would only need to load one state worth of data. Sure, that'll save me some memory, but what if I'm crossing multiple States now? We're starting to get into territory where we're taking up a lot of space again. It's still significantly smaller than the entire United States but still problematic.

This is where things get really interesting to me. You see you can carve polygons out of this data. In fact, that's all that's being done when we get a subset of the entire world. It's just a polygon cut out of it.

Then, I got to thinking what if I took the route that I'm planning on going and cut that route out. That would be significantly smaller so I tried it. It didn't work. Just to be clear, I'm not saying it couldn't work. I'm just saying that it didn't. So what I did was add a 50 mile border around my entire route. This still saved me tons of memory and enabled me to work within the bounds of my Raspberry Pi's memory.

 

Region	Disk Space Size**	Estimated Ram
United States of America	10.2 GB	51 GB
US Midwest	2.0 GB	10 GB
US Northeast	1.5 GB	7.5 GB
US Pacific	152 MB	760 GB
US South	3.5 GB	17.5 GB
US West	2.8 GB	14 GB
Colorado	304 MB	1.5 GB
Utah	134 MB	670 MB
My route crossing 5 states	648 MB	3.2 GB
My route crossing 5 states (using route polygon)	439 MB	2.1 GB

What is also interesting is if I pre-plan all my routes that I'm planning on using ahead of time I can trade hard drive memory for RAM memory. When I'm ready to take a particular trip I just swap out the database and I'm off and running. And let me tell you I can throw a big hard drive on a Raspberry Pi.

If you look in the table above you can see that for me to cross 5 states, I was able to do it in 439 MB of hdd space as opposed to the unoptimized version taking up 648 MB. This doesn’t seem like a lot but when you multiply that delta by 5 it’s a lot.

 

I should also note, While crossing 5 states and just loading those states up, would fit in the Pi’s memory, we also need to be able to run nominatim, which requires around 1GB of RAM. I did try to use the route polygon carved version for Nominatim but I was having lots of issues finding lat/lon < -- > city mappings which my tool really needs.

By shrinking the data we went from using a general purpose solution to a special purpose solution and it was able to work for our needs.

I hope you enjoyed hearing about my journey!

* https://blog.afi.io/blog/hosting-the-osrm-api-on-amazon-ec2-running-osrm-backend-as-a-web-service/ 

** Source: https://download.geofabrik.de/north-america/us.html

Portable Car Media Server

***

This is a work in progress post, it has most of the content of what I did, but it's after the fact, and I haven't replicated it to ensure it works exactly as described, but you should have enough info to accomplish what I did!

***

 

(Note I have Amazon Affiliate links below FYI)

I decided to build a portable Jellyfin Server. I wanted my kids to be able to login and watch shows as needed while we were driving and thus started going down the path.

I bought a travel router from amazon. I came across this on HackerNews a while back, and liked the idea, which is where this all started.

We used it on our last road trip and it worked quite well.

Parts required (I used stuff I had around, so the links below are largely placeholders for you to try, with the exception of the travel router and power bank, I did buy those).

1 Usb-A to Usb-micro cable

1 Usb-A to Usb-c cable

1 Power Bank

1 Raspberry Pi (I selected 3b+ as it's what I had lying around)

1 SD Card (If you already have a pi up and running, you may not need this)

1 GL-MT3000 (Beryl AX) Pocket-Sized Wi-Fi 6 Wireless Travel Gigabit Router

1 Storage usb drive

1 Shutdown Key (Or just find a spare usb device you can bring along)

1 Storage Case (Totally optional)

Key Stages:

1. Get Pi Setup and running (Assumed you have set this up)
2. Get Jellyfin installed on Pi
3. Setup USB Drive for media
   
4. Get nginx installed on Pi
5. Static IP on Travel Router
   
6. Get Adblocker running on Travel Router with Redirect
7. Get USB Shutdown Key Created

Installing Jellyfin

https://pimylifeup.com/raspberry-pi-jellyfin/

Ultimately I just searched install jellyfin on raspberry pi and it just worked.

Setup USB Drive for media

TODO: More details on how I setup with the USB drive.

Installing nginx

https://engineerworkshop.com/blog/setup-an-nginx-reverse-proxy-on-a-raspberry-pi-or-any-other-debian-os/

Ultimately we will follow the above however there are some details that are a bit fuzzy for me. It has you edit this

sudo nano example.com.conf

file, however my system wasn't setup like that,  so instead of that I used default (likely this command):

ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default

From there we need to update:

sudo nano /etc/nginx/sites-available/default

to contain this:

server {
	listen 80;
	server_name jellyfin.car.com;
	location / {
	proxy_pass http://localhost:8096;
	}
}

This means that we will redirect a request coming in for jellyfin.car.com to the jellyfin server. You can replace jellyfin.car.com with whatever you want, but that's what I used.

I specifically used localhost, because when I was doing setup, I was on my main wifi, so when I swapped between wifi's as long as I had the request coming in via jellyfin.car.com (on my main or the travel wifi) it handled correctly.

Static IP

https://docs.gl-inet.com/router/en/3/setup/gl-e750/more_settings/#static-ip-address-binding

This should generally be what you need to do. Figure out your mac address, and then put in a static IP there. I think the default IP range is 192.168.8.x so I set my pi to something like 192.168.8.31.

Get Adblocker running on Travel Router with Redirect

https://medium.com/@life-is-short-so-enjoy-it/homelab-adding-local-dns-entry-into-adguard-home-arpa-and-pushing-to-clients-from-udm-se-8493253830e5

Next we are going to turn on the adguard (A sweet feature of the travel router), and point the redirect to our pi!

Specifically we are going to do the section called add local DNS entries into Adguard.

I put in 

192.168.8.31 jellyfin.car.com

(instead of their arpa example)

Now when I visit jellyfin.car.com on the wifi that router is on, the adguard will look that URL up, and then return the pi's ip address, which in turn, means that it will connect over NGINX on port 8096

 

Get USB Shutdown Key Created

http://blog.onaclovtech.com/2024/03/usb-shutdown-key.html

Finally, we will setup a USB shutdown key, this makes it so when we are ready to shutdown for the day we can kill our pi, and unplug power after maybe 30 seconds.

You can use any old usb device I have an old 32 mb usb key (my first perhaps?) that I am using, I labelled it Shutdown Key.

Results

 

Once you have this setup, you can kick it off, the system should last on the battery bank more or less 5-7 hours (maybe more, maybe less, depending on use).

Alternatively you can buy two banks, and then just run the router on one, and the pi on another. 

Finally you "could" possibly just run your router off the USB power in the car, so you don't need the power bank at all, however I didn't test that out, I think the router can handle power interruptions, the pi generally can get finnicky on that, so I wanted a way to be in control of that which is why it's on a power bank, so if I shut the car off I don't accidentally forget to gracefully power the pi off, and kill it in the middle of my trip.

***

This is a work in progress post, it has most of the content of what I did, but it's after the fact, and I haven't replicated it to ensure it works exactly as described, but you should have enough info to accomplish what I did!

***

USB Shutdown Key

While building up a raspberry pi system recently I needed an easy way to shutdown the device without having to ssh into it.

I had an idea pop into my head, what if I plug in a usb drive and maybe it automatically runs a script on the usb key, or something and powers the device down.

Well after a bit of googling it turns out.... you can use udev rules to do what I want, and no need to put anything on my drive.

This is the process.

First, you figure out the vendor ID and the product ID of the usb key. 

The way I did this was by running lsusb and noting the devices, since this is a raspberry pi, the list was short.

I then plugged the usb drive in, and ran lsusb again. I noted which device it was.

Then I ran lsusb -v to get the values for idVendor and idProduct. 

I then copied them into this line (remember not to include the 0x), in the applicable spots. 

ACTION=="add" , ATTRS{idProduct}=="2168" , ATTRS{idVendor}=="0ea0" , RUN+="/usr/local/bin/my_shutdown_script.sh"

I then placed that line into this file:

sudo nano /etc/udev/rules.d/99-usb-shutdown.rules

Next, I edited /usr/local/bin/my_shutdown_script.sh to contain the following:

#!/bin/sh

sudo shutdown -h now 

Finally I ran 

sudo chmod +x /usr/local/bin/my_shutdown_script.sh

on the file and rebooted, just to make sure everything was stable.

Now when I plug in the usb drive, bam, it just shuts down.

Thanks to this post for the details they used, which was exactly what I did with the minor note that I used lsusb -v (they just mentioned using lsusb ;))! 

https://www.reddit.com/r/linuxquestions/comments/yiw13c/trying_to_create_a_udev_event_to_safely_shutdown/

Building your own DVR Part II

This is a continuation from Raspberry Pi + HD Homerun Dual = OTA Dvr

The original post here was outdated and I decided to remove it.

There will be future posts but it's slow going.

This is going to be a very step by step tutorial as I go through things :)

Raspberry Pi + HDHomeRun Dual = OTA DVR

My latest project has involved working towards a cord cutting world.

Lets get started.

The Parts

(prices as of current posting)
35.09 Raspberry Pi Model B+ (I initially used, Raspberry Pi Model B but don't expect any difference)
About $10 Research your own microsd card (for Model B I used this, I will post what I use when I get my B+)
18.99 Powered USB Hub (Any version should work, I just picked something cheap)
110.77 HDHomeRun Dual (There is an upgrade the HDHomeRun Connect-2, I haven't tried it but if you're feeling luck here's the link HD Homerun Extend-2, looks like it'll encode to other formats which is nice).
45.74 HD Antenna (This one picks quite a few channels up and has worked really well thus far, Antennaweb.com recommended a different one, but after adding a mount it was more than this one).
47.99 Router (I'm using the WRT54G because it was a spare one I had on hand).No Longer Needed.
64.99 1 TB Harddrive (I had an existing one, so some of these costs I'm not accounting for myself so my break even is sooner)
This comes out to a grand total of about $300. In my case I pay around $28 per month for my Dish Network Subscription. This means it'll take to about 12 months and I'll be ahead for costs (assuming I buy everything at once, and don't have spare parts I can just use).

Note: I didn't include powering the Raspberry pi in this calculation of cost, I plan to see if I can plug directly into the powered usb hub, and plug in the HDD from there, but I don't know for sure on that, however it uses the same plug as MOST new phones out there these days, so you probably already have a cable. (one can be found here though)

Setup

The HD Home Run Dual is a REALLY Easy piece of equipment to use. All I did was plug it into the wall, plug into the router, and plug into the antenna.

You ABSOLUTELY need to be connected over ethernet with the raspberry pi (So NO raspberry pi Model A since it doesn't have an ethernet port).

I installed Rasbian (at some point in the past).

For the setup I booted into LXDE, (login and type startx).

I found this useful to remove a few unnecessary programs, but since we have a powered usb hub we are going to be storing the data on an external drive as the size of the saved file is a bit large.

http://www.sbprojects.com/projects/raspberrypi/tweaks.php

The Raspberry Pi will also need the following:

Download libhdhomerun, HDHomeRun Config GTK from here:
http://www.silicondust.com/support/downloads/linux/

Here is a GIST of the commands I used after I downloaded and extracted the above zip files, and I entered into the hdhomerun_config_gui folder.



Finally I ran the following to test that the tuner was running.

hdhomerun_config_gui

I then updated the firmware on the tuner (Disclaimer, do so at your own risk, not sure if it's required, but I did it).

Clicking on the Update tab, you can select the firmware also downloaded from Silicondust website above,

If you're using the DUAL it should be this driver: HDHR3-US (hdhomerun3_atsc).

If  you're using the Extend you'll have to research which version of the firmware is right.

Recording

At this point I believe I exited the gui and re-opened it once I got the firmware updated (I got worried for a split second cause it couldn't find my tuner anymore, but it was something as simple as that to get going again). I was able to scan and see some channels and click on View to see them playing.

If you note on the left side the name of the tuner is listed (I believe it's id - 0,1), you'll need this number, also if you type hdhomerun_config discover via command line you should be able to get it right.

My general process for recording TV is this: (may be improved in the future)

1. Set Channel
2. Save Stream for some period of time.
3. Open Handbrake and transcode to another format.

I decided to just save the default format of the stream (TS?).

Looking at the developer gui You will see there is the ability to pipe into another program.

As of right now I believe that the "save" command will basically just write bytes to a file. So there is very little processing overhead happening, adding transcoding into the mix might be too much for the pi (but may not be, I really haven't verified).

I haven't tried it but you MAY be able to record BOTH tuners from one raspberry pi at a time.

Here is the single line command I used to record something last night, I'll explain what each means in a minute.

date; sleep 3600; date; hdhomerun_config set /tuner0/channel auto:27; timeout 3700 hdhomerun_config save /tuner0 test_show.ts;

Ok so I'll break down each section.

date prints the date/time
sleep will sleep the number of seconds allocated, I put in the dates around to verify it is waiting the right amount of time, and it appears to be.
hdhomerun_config ID set /tuner0/channel auto:27; This will set the first tuner to channel 27 auto selecting the modulation
timeout 3700 run a command for just over an hour (When I ran the .ts file, it was usually a little shorter than the timeout time, so I just expand it over a bit, you can edit it back if you need to using video editing software).
hdhomerun_config save /tuner0 test_show.ts; This is what actually saves your OTA channel.

Note: Remember to navigate to your external hard drive when you run this (you may be able to point the .ts file to the path, but I haven't tried it), otherwise you'll run out of memory pretty fast. When I recorded about 4.5 hours it came out to 26.1 GB of data. Transcoding down should come out to 5-7 GB so it'll go quite a bit down (using Normal on Handbrake and MKV, other settings may come out smaller).

Good luck and let me know what you come up with!

Keep an eye out for a future post, I'm planning on building a NODEJS server that will allow me to schedule recordings from the internet this will be similar to my post about controlling my roku/chromecast from the internet, I'll be using Firebase as an intermediary. (I'd love to go full on DVR solution but just worry MythTV or some other equivalent would be too heavy).

Latest Projects

I received a Raspberry Pi for fathers day, so I've played around a little with it. So far I've built two projects, and they're located at github.

This is a wrapper to download a mRSS feed and then play it in a random order on your Raspberry Pi
https://github.com/onaclov2000/omxrssplayer

This is a Text to Speech wrapper, that uses google to do the translating, it could really do some improvement, by counting the letters, then splitting on or before the 100th letter (space).
https://github.com/onaclov2000/pytts

I'm working on using the code from this link:
https://github.com/raspberrypi/firmware/tree/master/opt/vc/src/hello_pi/hello_font
to try to overlay a news feed of sorts over the OMXrssplayer (wrapper) app I wrote.

I would like to keep working on the pytts, and get a pystt (speech to text), and then have it listen to my voice and do things (kinda like Jarvis from Iron Man :) )

I thought it would be fun to setup a chatbot and then connect all three, so I say something, it relays it to the chatbot, which responds, then it converts that to speech, so it can talk to you!

Anywho if you like it, and want to start building up components that would be cool, let me know, I would love to build it to work with Wolfram Alpha too, but even more I'd like for it to have a learning algorithm so it can be taught, and/or learn. (I like AI, but don't know much about it yet).