revised [Version2, 2020-7-5] [Version1, 2019-12-19]
Abstract
The purpose of this project is to create an application that uses AI (artificial intelligence) to determine that a mountain is now in a beautiful landscape, and automatically notify it to the smartphone. You can enjoy the scenery with a live camera even in a distant mountain. However, the mountains are often covered with clouds, and the weather changes over a short period of time. Therefore, there will not be many opportunities to see the beautiful scenery. The smartphone app described below will automatically notify you when a photo that is so beautiful that you can see it up to the top of the mountain is obtained. Wouldn't it be fun if you received such a notification while drinking coffee?
Outline of the project
The weather in the mountains changes in a short time. Therefore, the scenery is not always beautiful so that you can see the summit. Here, we have created an app that automatically stores only the photos in which the mountain summit is clearly visible. As an example, this app recognizes the scenery of Rishiri mountain that is towering in northern Hokkaido of Japan in real-time and saves good photos. This app incorporates a trained model of this mountain images, but of course, it is easy to replace them with your favorite mountain images.
Preparing mountain images
For image recognition, we used MIT's Personal Image Classifier [1]. First, we should classify many photos of this mountain into four categories, as shown in Fig1, depending on how they look. For this purpose, we quote images of a live camera [2] installed at the foot of the mountain. Get the image at appropriate intervals and give it a category label, as shown in Fig.1 and Fig.2. For each label, it is good to have more than 10 images. Enter them into MIT's Personal Image Classifier via your PC's web browser.
Fig.1 Classification into four categories
Fig.2 Preparing images for classification training
Training of the recognition model
When the images are ready, perform training as shown in Fig.3. Training takes place on the Neural Network. Adjust the network structure and some hyperparameter values if necessary. At the scale of this example, training usually ends in tens of seconds. Then download the trained model to your PC and use it in the App Inventor program as shown below.
Fig.3 Train our own model using MIT Personal Image Classifier
Using the trained model in App Inventor
The extension (named personalimageclassifier.aix) for using the downloaded model in App Inventor can be obtained from the reference [1]. The usage is very simple as shown in Fig.4. First, set the downloaded model to the Classifier properties, and then give the image you want to recognize to the block for classifying. As a result of recognition, the degree to which the image should be classified into each label is shown. That is, you get a list of label/confidence pairs. Finally, the image is classified into the label with the highest confidence value.
Fig.4 Blocks of the Personal Image Classifier
Getting the weather forecast
Before using this app, it is better to check the weather forecast for this mountain. If the weather is too bad, you can't expect a good picture. Press the button shown in Fig.5 to automatically display mountain weather up to 6 hours ahead. The weather forecast requires parsing a JSON file from OpenWeatherMap [3]. So we use our own JSON Decoding Extension (named FoYoJSON.aix) [4]. Fig.6 provides an overview of it. In order to use OpenWeatherMap, you need to get your own API key.
Fig.5 Checking weather forecast
Fig.6 Getting and decoding weather forecast JSON file
Running the app
An example of running the app is shown in Fig.7. In this case, it was recognized as "Cloudy but looks good" because there are many clouds but the mountainside looks good. Such a case is normal and it is rare to see it well up to the top.
Fig.7 Get an image and then recognize
Automatic notification of good scenery
If you are lucky, you will encounter a scene where you can see the summit as shown in Fig.8. In such a case, the recognition results in "Clear enough to see the summit". In that case, You can know that in two ways. One way is to feel the vibration of your smartphone, another way is to receive an e-mail automatically. If you choose the latter, the recognized image is automatically sent to the registered mail ID. Whichever you choose, if you have a PC nearby you will be able to enjoy the larger, clearer images immediately on your PC screen rather than your smartphone.
Fig.8 Automatic notification of beautiful scenery via email
Probably the easiest way to send emails automatically without human intervention is to use Taifun's extension [5] shown in Fig. 9 but note that it is a paid extension.
Fig.9 Using an extension to send emails via SMTP without user interaction
Notice
(1) If the live camera image of your choice has a high resolution, you may need to reduce the resolution and pass it to the image classifier.
(2) To get and display the weather forecast, you need your own API Key for OpenWeatherMap.
(3) If Error_908 is displayed when running the app, please allow access to External_Storage on your Android settings.
References
[1] MIT's Personal Image Classifier
https://appinventor.mit.edu/explore/resources/ai/personal-image-classifier
[2] Live camera at Rishiri Town Hall (in Japanese)
http://www.town.rishiri.hokkaido.jp/rishiri/
[3] OpenWeatherMap
https://openweathermap.org
[4] JSON Decoding Extension by FoYo
http://sparse-dense.blogspot.com/2019/01/a-json-decoding-extension-for-app.html?m=0
[5] Taifun's mail extension (paid)
https://puravidaapps.com/mail.php
