Real-time Dashboard in Python: Streaming and Refreshing
6 min READ
By: Sophia Yang
By: Sophia Yang
Data scientists use data visualization to communicate data and generate insights. It’s essential for data scientists to know how to create meaningful visualization dashboards, especially real-time dashboards. This article talks about two ways to get your real-time dashboard in Python:
First, we use streaming data and create an auto-updated streaming dashboard.
Second, we use a “Refresh” button to refresh the dashboard whenever we need the dashboard to be refreshed.
For demonstration purposes, the plots and dashboards are very basic, but you will get the idea of how we do a real-time dashboard.
To show how we create this dashboard with a real-world API, we use weather data from the OpenWeather API as an example in this article.
If you would like to try out the code mentioned in this article, you will need to get access to the OpenWeather weather API. You can sign up at the Open Weather website and then get the API keys.
Set up
Before going into the code, let’s install the needed packages:
Here we import the packages used for this article:
import operator as op
import numpy as npimport pandas as pd
import requests
import param
import panel as pn
import hvplot.pandas
import hvplot.streamz
import holoviews as hv
from holoviews.element.tiles import EsriImagery
from holoviews.selection import link_selections
from datashader.utils import lnglat_to_meters
from streamz.dataframe import PeriodicDataFrame
Streaming data and dashboard
First, we make a function ‘weather_data’ to get weather data for a list of cities using the OpenWeather API. The output is a pandas dataframe with each row representing each city.
def weather_data(cities, openweathermap_api_key=openweathermap_api_key): """ Get weather data for a list of cities using the openweathermap API """ L = [] for c in cities: res = requests.get(f'http://api.openweathermap.org/data/2.5/weather?q={c}&appid={openweathermap_api_key}&units=imperial') L.append(res.json()) df = pd.DataFrame(L) df['lon'] = df['coord'].map(op.itemgetter('lon')) df['lat'] = df['coord'].map(op.itemgetter('lat')) df['Temprature'] = df['main'].map(op.itemgetter('temp')) df['Humidity'] = df['main'].map(op.itemgetter('humidity')) df['Wind Speed'] = df['wind'].map(op.itemgetter('speed')) return df[['name','lon', 'lat','Temprature','Humidity','Wind Speed']]
Second, we use ‘streamz’ to create a streaming dataframe based on San Francisco’s weather data. The function ‘streaming_weather_data’ is used as a callback function by the PeriodicDataFrame function to create a ‘streamz’ streaming dataframe ‘df’. ‘streamz’ docs documented how ‘PeriodicDataFrame’ works:
streamz provides a high-level convenience class for this purpose, called a PeriodicDataFrame. A PeriodicDataFrame uses Python’s asyncio event loop (used as part of Tornado in Jupyter and other interactive frameworks) to call a user-provided function at a regular interval, collecting the results and making them available for later processing.
def streaming_weather_data(**kwargs):
"""
callback function
get San Francisco weather data
"""
df = weather_data(['San Francisco'])
df['time'] = [pd.Timestamp.now()]
return df.set_index('time')
Make a streaming dataframe
df = PeriodicDataFrame(streaming_weather_data, interval='30s')
The ‘streamz’ streaming dataframe ‘df’ looks like this, with values updated every 30s (since we set interval=’30′).
Third, we make some very basic plots using ‘hvPlot’ to plot the ‘streamz’ dataframe, and then we use ‘panel’ to organize the plots and put everything in a dashboard. If you would like to know more about how to use ‘hvPlot’ to plot ‘streamz’ dataframe, please see hvPlot docs.
# panel dashboard for streaming data pn_realtime = pn.Column( pn.Row( df[['Temprature']].hvplot.line(title='Temprature', backlog=1000), df[['Humidity']].hvplot.line(title='Humidity', backlog=1000)), df[['Wind Speed']].hvplot.line(title='Wind Speed', backlog=1000))
Here is what the dashboard looks like. Since the streaming dataframe updates every 30s, this dashboard will automatically update every 30s as well. Here we see that Temperature changed, while humidity and wind speed did not.
Great, now you know how to make a streaming dataframe and a streaming dashboard.
Refresh dashboard
Sometimes, we don’t really need a streaming dashboard. Instead, we might just like to refresh the dashboard whenever we see it. In this section, I am going to show you how to make a “Refresh” button in your dashboard and refresh the dashboard with new data whenever you click “Refresh.”
First, let’s make a simple function ‘weather_plot’ to plot weather data on a map given city names and return both the plot and the data.
With the plotting function ready, we can start making the refreshable dashboard. I use the ‘param.Action’ function to trigger an update whenever we click on the button ‘Refresh’. In addition, I use the ‘param.ObjectSelector’ function to create a dropdown menu of the dataframe columns we are interested in plotting and ‘param.ObjectSelector’ trigger an update whenever we select a different option in the dropdown menu. Then the ‘@param.depends(‘action’, ‘select_column’)’ decorator tells the ‘get_plot’ function to rerun whenever we click the Refresh button or select another column.
Our final dashboard ‘pane’ is a ‘panel’ object, which can be served by running:
panel serve realtime_dashboard.py
or
panel serve realtime_dashboard.ipynb
For more information on ‘panel’ deployment, please refer to the ‘panel’ docs.
Now you know how to make a real-time streaming dashboard and a refreshable dashboard in python using ‘hvplot’ , ‘panel’ and ‘streamz’. Hope you find this article helpful!
Sophia Yang is a Senior Data Scientist at Anaconda, Inc., where she uses data science to facilitate decision making for various departments across the company. She volunteers as a Project Incubator at NumFOCUS to help Open Source Scientific projects grow. She is also the author of multiple Python open-source libraries such as condastats, cranlogs, PyPowerUp, intake-stripe, and intake-salesforce. She holds an MS in Statistics and Ph.D. in Educational Psychology from The University of Texas at Austin.
