Making my own box office sign with Scrapy and an Inky Frame

The algorithm got me again
webscraping
python
raspberry pi
automation
Author

Lindsay Lee

Published

February 19, 2024

I’m a flawed human being and therefore susceptible to ads I receive on the internet. One day I was scrolling and stumbled upon this little gadget by Pimoroni: the Inky Frame 7.3”, a fun little e-ink screen that you can program with micropython. I bought it without much a plan for what I would use it for, but thought it would be a fun excuse to practice my python skills. I soon got the idea to combine my loves: I spend all my free time and money at the Belcourt Theatre in Nashville, and thought it would be cool to use the screen as a little display that shows the showtimes for the day.

Accomplishing this means several things have to happen:

  1. Web scrape the showtimes from the theater’s website
  2. Format the showtimes into an image that works well on the Inky Frame display
  3. Push the image output to the Inky Frame
  4. Schedule all of the above to run daily automatically

Here I go through what I did for each of these steps. All the code is available on GitHub.

Web scrape the showtimes

To web scrape the showtimes, I used the popular python package Scrapy. There is a really useful and fun course on LinkedIn Learning by Ryan Mitchell called “Web Scraping with Python”, which is where I first learned how to do this. She also has a book “Web Scraping with Python” that is super informative. I’ll leave most of the detail to her, but here is the gist:

First install Scrapy:

pip install Scrapy

Then you initialize a new project like so:

scrapy startproject belcourt

Navigate to the new project folder and initialize a spider (a “spider” is the program that will do all the scraping that you specify). Here I’m creating a spider called showtimes that will scrape the website belcourt.org:

cd belcourt
scrapy genspider showtimes belcourt.org

In the items.py script that is generated, I defined the structure of the output that I wanted by defining a new class called BelcourtItem.

import scrapy

class BelcourtItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    currenttime = scrapy.Field()
    date = scrapy.Field()
    shows = scrapy.Field()

In the showtimes.py script in the spiders folder, specify which content from the website should be pulled into your output object of class BelcourtItem. This involves inspecting the HTML of the website and finding the relevant elements. See Ryan Mitchell’s course/book for tips on how to do this. I also suggest learning more about searching for HTML elements with XPATH–this YouTube video was the best resource I’ve come across.

To run the spider, navigate the console to the spiders folder and run spider like below. This will print the output to a json file called output_showtimes.json:

cd belcourt/spiders
scrapy runspider showtimes.py -O output_showtimes.json

You can see what this json file looks like in my GitHub repository.

Voila! Easy! Now we need to take the data from this json file and turn it into a fairly attractive jpg.

Format the output into an image

Converting this json into a jpg mostly involved getting familiar with the Pillow python package. It can be installed like so:

pip install Pillow

The script I wrote is called belcourt_generate_image.py, and you can find it in the GitHub repository.

First you read in the json output produced by the spider. Then specify some basics like the dimensions, title and subtitle and fonts. The dimensions match what is specified by Pimoroni in their “Getting Started with Inky Frame” resource. I read the fonts directly from ttf files which are also loaded into the repository. I also loaded the Belcourt logo so I could use it in my jpg output.

The rest of the script is essentially telling python where exactly to place the different elements of the image. Finally, the output is saved to a jpg. One important note is that it is important that the resulting jpg not be “progressive.” I don’t really know what that means, but the frame can’t handle it otherwise for some reason:

background.save('result.jpg', progressive = False)

Now that we’ve got a jpg, time to get it on the screen!

Load image into Inky Frame

Here is where we start getting into slightly uncharted territory. Pimoroni provides some nice guides and templates for displaying content on the screen. I first followed this guide to display a static image on the screen that was loaded directly into memory. That’s not good enough though, because we need the screen to load the jpg via the web. Luckily they have another example that does something similar. I adapted this script into my own version showtimes_from_web.py, available in the GitHub repository. This script requires wifi credentials saved in a secrets.py file, as described in the Pimoroni guide above. It also requires a network_manager.py file, available from Pimoroni’s example GitHub. It also requires a micro SD card be installed, which luckily comes with the Inky Frame.

Most importantly, the script needs a URL to access the image from. First I tried to use the link to the jpg in the repository: https://github.com/lindsayevanslee/inky-frame/blob/main/belcourt/result.jpg?raw=true. However, this link leads to a redirect, which the urllib micropython package used by the Inky Frame cannot handle. Therefore we need a stable, direct link to the jpg. Luckily GitHub offers the ability to create a webpage for your repository using GitHub Pages. This can be configured by going to the settings for the repository, then going to “Pages”. I chose to use “GitHub Actions” as the source and the “main” branch as the branch to publish to. By default README.md will be used as the main page for the resulting site. I linked to the jpg in the README like so:

# inky-frame
Code powering my shiny new Inky Frame

Here is the latest output:
![Image with latest showtimes from the Belcourt Theatre](belcourt/result.jpg)

This will load the jpg to the GitHub Page, and then you can copy the link to the jpg from there for use in the showtimes_from_web.py script.

Automate to run daily

There are two aspects of automation that need to be implemented: automatically updating the jpg every day with the day’s showtimes, and automating the refresh of the screen to display the new jpg.

Automate the jpg update

GitHub Actions can be used to execute scripts on a schedule and make commits to the repository. I used this guide to set up a GitHub Action that would run the spider and generate the jpg every day in the early morning. The syntax for defining the schedule is called cron, and this website is super helpful for figuring out how to configure the cron syntax to do what you want.

This GitHub Action requires giving the workflows read/write permissions by enabling: Settings > Actions > General > Workflow Permissions > Read and write permissions. The workflow is defined by a YAML file that I’ve called .github/workflows/actions.yml in the GitHub repository. The script runs the spider, runs the belcourt_generate_image.py script, and then commits the resulting jpg to the repository.

A second GitHub Action is needed to update the GitHub Pages deployment every time the jpg is updated. By going to Settings > Pages, you can configure the deployment of the webpage. GitHub automatically provides an Action template for you, and there is additional detail in the documentation. One change I made to the YAML file was to schedule the deployment to run after the first action that updates the jpg. I wanted to do this by triggering the deployment after the new changes were committed to the repository. I found this blog that was trying to do the same thing, but I couldn’t get it to work. It’s got something to do with setting SSH keys, which I’ll figure out at a later date. Instead I again used cron to schedule the deployment to run half an hour after the first action, which should be enough time for the jpg generation to finish.

Automate the Inky Frame refresh

The final step in this process is to automate the Inky Frame to refresh the image every day at a certain time. This is where I ran into some trouble.

The Inky Frame will run anything saved as main.py when it starts up. If main.py is set to:

#run showtimes_from_web.py on start-up
with open("showtimes_from_web.py") as f:
    exec(f.read())

then my script showtimes_from_web.py will run on start-up, which pulls down the jpg from the GitHub page and displays it on the screen. The screen will start up when it is plugged in to USB power or when the battery pack is turned on. However, I don’t want to have to start up the screen manually each day, and I also don’t want to leave it plugged in all the time, so I need to find a way to get it to refresh itself while on battery power.

I did a bunch of searching, and there are some helper functions out there that seemed promising.

I tried a bunch of things first. I tried adding inky_frame.sleep_for(1) to main.py which should theoretically cause it to go to sleep for a minute and then start back up, but that didn’t trigger another run of main.py. I also tried adding this to the showtimes_from_web.py script itself, but that also didn’t work. I tried running another gc.collect() at the end of showtimes_from_web.py in order to ensure that as much of the RAM was available as possible, but that also didn’t work. I tried to find a way to “close” the jpg (the showtimes_from_web.py script has an open_file command, so I figured, maybe it needs to be closed again in order for the script to truly terminate), but it doesn’t seem to be needed because it looks like the decode function called directly after open_file contains a closing function within it.

In order to see if the sleep_for() function does wake the frame back up after it sleeps, I tried putting it at the beginning of the main.py function instead of at the end. It does indeed wake back up and continue running main.py. I thought there might be some issue with the showtimes_from_web.py fully finishing and perhaps the sleep_for() script never actually gets executed when it is at the end of main.py. One indicator that the script isn’t fully finishing is that the busy symbol on the screen doesn’t go away after the screen is refreshed.

I tried following this blog using timers but this also didn’t work. I could see the timer was working but it didn’t spark a refresh, which was more evidence that there is an issue with the showtimes script fully finishing.

Then I noticed that there was a file on my Inky Frame called state.json that seemed to indicate the script was continuing to run. I tried deleting this file by using inky_helper.clear_state(). the console. After doing this and including sleep_for() at the end of main.py, I was able to trigger a refresh when it was plugged in to USB power! But on battery power, it still didn’t work.

After months of googling and trying new things, I saw that others have posted about having the same sort of issue on both the Pimoroni forums and in their GitHub issues. In that thread someone posts a patch for micropython that they say solves the issue. I tried installing this patch, and it does seem like it prevents the showtimes script from getting stuck and not fully finishing. However, after running sleep_for(), the script just stops and doesn’t start main.py from the beginning as I want.

I reread the thread on the forum a few times and noticed that the original poster actually executes their graphics.update() and sleep_for() commands within a while loop. When I add this loop to my own script, like this:

#updated main.py
import inky_frame

while True:
  
    with open("showtimes_from_web.py") as f:
        exec(f.read())
        
    inky_frame.sleep_for(1)

…it does successfully cause a refresh after the screen sleeps! We’re getting closer! However after the first refresh, I got a strange EPERM error. After further copying the techniques of the original poster and adding some error logging to my scripts, I saw the error was occurring at the uos.mount() command in showtimes_from_web.py at the second run-through. I found the documentation for uos, and saw that uos.mount() throws this EPERM error when the file system is already mounted. I tried adding a uos.umount() command at the very end of the showtimes_from_web.py script, and this seems to have solved the issue! The screen refreshes continuously!

Finally, the moment of truth…does it refresh continuously when on battery power? I unplug the frame, turn on the battery pack, perform a reset (by holding down the A, E, and reset buttons), and wait…and it works! The screen refreshes on its own!

Gif of baby pumping his fists screaming YES!

It me

All of the scripts that I used on the Inky Frame (these are the only ones loaded–I deleted all other default examples and libraries that it comes with) are in the inky_frame_scripts folder of the GitHub repository. As a last step, I increased the sleep time to 60 minutes with sleep_for(60). Hopefully this doesn’t drain the battery too quickly. I may further increase it later.

Now I’ll finally be able to sleep at night. Until I come up with another silly idea.

Update 3/21/2024

Read my follow-up to this post for some fun battery chat.