library(httr)
library(jsonlite)
library(tidyverse)
library(rvest)
weather <- function(x, API_key){
url <- paste0("https://api.darksky.net/forecast/",API_key,
"/", x[1], ",", x[2],
"?units=ca&exclude=hourly,alerts,flags")
rep <- GET(url)
table <- fromJSON(content(rep, "text"))
current.weather.info <- with(table,
data.frame(Air_temp = currently$temperature,
Weather = currently$summary,
Atm_pres = currently$pressure,
Wind_str = currently$windSpeed,
Wind_dir = currently$windBearing,
Cloud_cover = currently$cloudCover,
PrecipProb = currently$precipProbability,
PrecipInt = currently$precipIntensity,
Moon = daily$data$moonPhase[1]))
return(current.weather.info)
}My favorite hobby, in addition to R coding of course, is fishing. Most of the time, I fish European sea bass (Dicentrarchus labrax) in estuaries. The sea bass is a predatory fish that has a broad range of preys: crabs, sand eels, prawns, shrimps and other fish. To catch these predators, I don’t use live baits, I prefer to use artificial lures that imitate a specific prey.
In theory, it is quite easy to catch a fish:
Use a lure that imitate the current prey of the sea bass.
Animate the lure in a spot where the fish are active.
Catch a really big fish !
In practice, it is an other story ! Indeed, the feeding activity, the position of the European sea bass in the estuary and their preys will vary depending on different parameters:
- the characteristics of the riverbed, which will depend where I fish
- the time of the day: the sea bass is more active during dawn and dusk
- the current and water level associated with the tide. The water level in estuaries is constantly varying to greater or lesser degree due to the tide influence. It is also influenced by the river flow which can be higher in case of heavy rains.
As you understand, there are many parameters potentially influencing the results of my fishing session. This is why I decided to create a shiny application to augment the number and the length of the fish caught during my sessions. To reach this objective, I need to better understand the activity, the position and the prey of the sea bass depending on the parameters described above.
Requirements of my application
- It must store data about my fishing session:
| Information needed | Description of the variables | Where do I get the data ? |
|---|---|---|
| Time | Time when a fish is caught, time since the beginning of the session | R |
| Catch | Species and length of the fish caught | Geolocation from smartphone? |
| Lures | Type, length, color of lure used | Weather API |
- It must record data about my catch and the artificial lures used:
| Information needed | Description of the variables | Where do I get the data ? |
|---|---|---|
| Time | Time when a fish is caught, time since the beginning of the session | R |
| Catch | Species and length of the fish caught | User input |
| Lures | Type, length, color of lure used | User input |
It must be adapted to small screens because I will always use the application on my phone.
It must remain free.
Collecting the data
Getting my gps location
My gps location is collected by using a bit of Javascript in the header of the shiny application. This code has been developed by AugusT and is available on his github repository.
Weather API
For the weather data, I found a free API called Dark Sky. I made a function that takes as input the coordinates of a place and the API user key and returns the current weather conditions in a dataframe:
Web scrapping for Tide data
I created a function to scrap information about the tide on a french website. The following function takes no argument and return the current water level, the tide status (going up or down) and time since the tide peak for the location I fish.
tide <- function(){
# Set the current time and time zone
Sys.setenv(TZ="Europe/Paris")
time <- as.POSIXct(Sys.time())
url <- "https://services.data.shom.fr/hdm/vignette/grande/BOUCAU-BAYONNE?locale=en"
# Read the web page that contains the tide data
text <- url %>%
read_html() %>%
html_text()
# Clean the html data to get a dataframe with two cols Time and water level:
text <- as.character(sub(".*var data = *(.*?) *\\;.*", "\\1", text))
text <- unlist(str_split( substr(text, 1, nchar(text)-2), "\\],"))
tidy_df <- data.frame(hour=NA,Water=NA)
for(i in 1:length(text)){
text_dat <- unlist(str_split(text[i], '"'))[c(2,3)]
text_dat[1] <- substr(text_dat[1], 1, nchar(text_dat[1])-1)
text_dat[2] <- as.numeric(substr(text_dat[2], 2, nchar(text_dat[2])))
tidy_df[i,] <- text_dat
}
tidy_df$hour <- as.POSIXct(paste(format(Sys.time(),"%Y-%m-%d"), tidy_df$hour))
# Some lines to get the tide status (going down or up) :
n_closest <- which(abs(tidy_df$hour - time) == min(abs(tidy_df$hour - time)))
water_level <- as.numeric(tidy_df[n_closest, 2])
all_decrea <- all(tidy_df$Water[(n_closest-6):(n_closest+6)] ==
cummin(tidy_df$Water[(n_closest-6):(n_closest+6)] ))
all_increa <- all(tidy_df$Water[(n_closest-6):(n_closest+6)] ==
cummax(tidy_df$Water[(n_closest-6):(n_closest+6)] ))
maree <- ifelse(all_decrea, "Down", ifelse(all_increa, "Up", "Dead"))
# Compute time since the last peak :
last_peak <- max(cumsum(rle(diff(as.numeric(tidy_df$Water), lag = 2) > 0)$lengths)
[cumsum(rle(diff(as.numeric(tidy_df$Water), lag = 2) >0)$lengths) < n_closest])
time_after <- as.numeric(difftime(tidy_df$hour[n_closest], tidy_df$hour[last_peak], units = "mins"))
# Return the list with the results :
return(list(Water_level = water_level,
Maree = maree,
Time_peak = time_after))
}The shiny application
The main problem I encountered while developing this application was data storage. Shinyapps.io host freely your shiny application but there were some problems when I used the shiny application to modify the csv files. The solution I found was to store the data in my dropbox account, you can find here more details on the subject and alternatives solutions. I used the package rdrop2 to access and modify the data with the shiny application.
Here are the main steps of this application :
When the application is started, it reads a csv file stored on my dropbox to see if a fishing session is running or not. If not the user can start a fishing session.
When starting a new session, a line with coordinates, weather conditions, and tide condition is added to the csv file previously mentioned.
If a fish is caught, the user can fill out a form to store the data in a second csv file. This file contains : the time, the species and length of the fish and information about the fishing lure used (type, color, length).
The user can end the fishing session by pushing a button. This will register the ending time, weather conditions, and tide condition in the first csv file.
A simplified graph is showed below:
UI side
The user interface of the application is built using the miniUI package. This package allows R user to develop shiny application adapted to small screens.
# Load libraries
library(shiny)
library(shinyWidgets)
library(googlesheets)
library(miniUI)
library(leaflet)
library(rdrop2)
Sys.setenv(TZ="Europe/Paris")
#Import the functions for weather API and webscrapping
suppressMessages(source("api_functions.R"))
# Load the dropbox token :
token <<- readRDS("token.rds")
# Minipage for small screens
ui <- miniPage(
# Javascript that give user location (input$lat,input$long)
tags$script('$(document).ready(function () {
navigator.geolocation.getCurrentPosition(onSuccess, onError);
function onError (err) {
Shiny.onInputChange("geolocation", false);
}
function onSuccess (position) {
setTimeout(function () {
var coords = position.coords;
console.log(coords.latitude + ", " + coords.longitude);
Shiny.onInputChange("geolocation", true);
Shiny.onInputChange("lat", coords.latitude);
Shiny.onInputChange("long", coords.longitude);
}, 1100)
}
});'),
gadgetTitleBar("Catch them all", left = NULL, right = NULL),
miniTabstripPanel(
#First panel depends if a fishing session is started or not
miniTabPanel("Session", icon = icon("sliders"),
miniContentPanel(uiOutput("UI_sess", align = "center"),
uiOutput("UI", align = "center"))
),
# Second panel displays the location of the previous fishing session with the number of fish caught
miniTabPanel("Map", icon = icon("map-o"),
miniContentPanel(scrollable = FALSE,padding = 0,
div(style="text-align:center",
prettyRadioButtons("radio", inline = TRUE, label = "",
choices = list("3 dernières sessions" = 1,
"3 Meilleures Sessions" = 2,
"Tout afficher" = 3),
selected = 1)),
leafletOutput("map", height = "93%")
))
)
)Server side
The server side is mainly composed by observeEvent functions. The utility of each observeEvent is provided in the script as commentary.
server <- function(input, output, session){
source("api_functions.R")
# Read the csv file containing information about fishing session. If a session is running,
# display the UI that allows the user to input data about the fish caught. If a session is not started,
# display a button to start the session.
observeEvent(input$go ,{
dat <<- drop_read_csv("/app_peche/session.csv", header = T, stringsAsFactors = F, dtoken = token)
output$UI<- renderUI({
tagList(
if(rev(dat$Status)[1] == "end"){
actionButton("go","Start session")}
else{
actionButton("go","End session")
}
)
})
output$UI_sess<- renderUI({
if(rev(dat$Status)[1] == "end"){
tagList(textInput("comments", label = h3("Commentaires"), value = "NA"))
}else{
input$catch
tagList(
selectInput("species", label = h3("Espèces"),
choices = list("Bar" = "bar",
"Bar moucheté" = "bar_m",
"Alose" = "alose",
"Alose Feinte" = "alose_f",
"Maquereau" = "maquereau",
"Chinchard" = "chinchard"), selected = "bar"),
sliderInput("length",label = h3("Taille du poisson"),value=25,min=0,max=80, step=1),
selectInput("lure", label = h3("Type de leurre"),
choices = list("Shad" = "shad",
"Slug" = "slug",
"Jerkbait" = "jerkbait",
"Casting jig" = "jig",
"Topwater" = "topwater"), selectize = FALSE),
selectInput("color_lure", label = h3("Couleur du leurre"),
choices = list("Naturel" = "naturel",
"Sombre" = "sombre",
"Clair" = "clair",
"Flashy" = "flashy" ), selectize = FALSE),
selectInput("length_lure", label = h3("Taille du leurre"),
choices = list("Petit" = "petit",
"Moyen" = "moyen",
"Grand" = "grand"), selectize = FALSE),
actionButton("catch","Rajoutez cette capture aux stats!"),
textInput("comments1", label = h3("Commentaire avant la fin ?"), value = "NA")
)
}
})
}, ignoreNULL = F)
#If the button is pushed, create the line to be added in the csv file.
observeEvent(input$go,{
#Tide + geoloc + Weather
c_tide <- unlist(tide())
geoloc <- c(input$lat,input$long)
current.weather.info <- weather(geoloc)
# Two outcomes depending if the session starts or ends. This gives the possibility
# to the user to add a comment before starting the session or after ending the session
if(rev(dat$Status)[1] == "end"){
n_ses <- c(rev(dat$Session)[1]+1)
stat_ses <- c("beg")
time_beg <- as.character(as.POSIXct(Sys.time()))
comment <- input$comments
dat.f <- data.frame(n_ses, stat_ses, time_beg ,geoloc[2], geoloc[1], current.weather.info, c_tide[1], c_tide[2], c_tide[3], comment)
names(dat.f)<-names(dat)
a <- rbind(dat,dat.f)
}else{
n_ses <- c(rev(dat$Session)[1])
stat_ses <- c("end")
time_beg <- as.character(as.POSIXct(Sys.time()))
comment1 <- input$comments1
dat.f<- data.frame(n_ses, stat_ses, time_beg ,geoloc[2], geoloc[1], current.weather.info, c_tide[1], c_tide[2], c_tide[3], comment1)
names(dat.f)<-names(dat)
a <- rbind(dat,dat.f)
}
# Write csv in temporary files of shiny server
write_csv(as.data.frame(a), "session.csv")
# Upload it to dropbox account
drop_upload("session.csv", path = "App_peche", mode = "overwrite", dtoken = token)
})
# Add a line to the catch csv file whenever a fish is caught
observeEvent(input$catch,{
caugth <- drop_read_csv("/app_peche/catch.csv", header = T, stringsAsFactors = F, dtoken = token)
n_ses <- c(rev(dat$Session)[1])
time <- as.POSIXct(Sys.time())
time_after_beg <- round(as.numeric(difftime(time, rev(dat$Time)[1], units = "mins")), digits = 0)
catch <- data.frame(n_ses,
time = as.character(time),
min_fishing = as.character(time_after_beg),
species = input$species,
length = input$length,
lure = input$lure,
colour = input$color_lure,
length_lure = input$length_lure)
b <- rbind(caugth,catch)
# Write csv in temporary files of shiny server
write_csv(as.data.frame(b), "catch.csv")
# Upload it to dropbox account
drop_upload("catch.csv", path = "App_peche", mode = "overwrite", dtoken = token)
})
# Create the map with the results of previous session depending on the choice of the user :
observeEvent(input$radio,{
output$map <- renderLeaflet({
map_data <- map_choice(input$radio)
leaflet(map_data) %>% addTiles() %>%
addPopups(lng = ~Long,
lat = ~Lat,
with(map_data,
sprintf("<b>Session %.0f : %.1f h</b> <br/> %s <br/> %.0f poissons <br/> hauteur d'eau: %.0f m, %s, %.0f min après l'étal",
n_ses,
duration,
Time,
nb,
Water_level,
Tide_status,
Tide_time)),
options = popupOptions(maxWidth = 100, minWidth = 50))
})
})
}Conclusion and future improvments
You can find a dummy example of this application (not linked to the dropbox account) here. I have been using this application for 1 year without any problems! The data I collected will be presented in the next post.
In the coming months, I must find a new free API to replace the actual one. Indeed, the weather API I am using has been bought by Apple and the free requests will be stopped in the following year.