My DevOps Journey

Building a weather data collection system using AWS S3 and OpenWeather API

Blessing Eseosa Osarumwense's photo
Blessing Eseosa Osarumwense
·

3 min read

Hello! Welcome to my first blog where I will be documenting my experience with the #DevOpsAllStarsChallenge. For the first project, it entails developing a system that collects realtime weather data from the OpenWeather API and stores the information in an AWS S3 bucket in JSON format. This solution demonstrates key skills like API integration, cloud storage management, version control with Git, and Python programming.

This project was particularly exciting for me, as it required extensive research and seeking clarification from colleagues to resolve the errors and challenges I encountered along the way. I was able to learn in the process. The below architecture visualizes the working process of this project.

Prerequisites

To get started, I made sure to have the following resources:

  1. Resources:

    • Basic knowledge of AWS services - S3, IAM

    • AWS account - Access key and Secret access key

    • OpenWeather API key (sign up on OpenWeather)

    • Integrated Development Environment

    • Bash scripting fundamentals

    • Github account

  2. Directories and files:

    My first step was to create a source folder for three directories named src, data and test. Added an init file to the src directory to make directory a python package, also created a README file.

     mkdir weather-dashboard
 cd weather-dashboard/
 mkdir src tests data
 touch src/__init__.py/weather_dashboard.py
 touch requirements.txt README.md .env

    Then went ahead to create .gitignore files, to specify what does not need tracking.

     echo ".env" >> .gitignore
 echo ".zip" >> .gitignore
 echo ".__pycache__/" >> .gitignore

    The next steps were to install the necessary requirements files by running

     echo "boto3==1.26.137" >> requirements.txt
 echo "python-dotenv==1.0.0" >> requirements.txt
 echo "requests==2.28.2" >> requirements.txt
 pip install -r requirements.txt

    The requirements.txt file includes:

    • python-dotenv: manages environment variable.

    • boto3: AWS SDK for python, which allows interaction with AWS.

    • requests: For making HTTP requests to fetch weather data from OpenWeather API.

  3. Environment setup:

    To configure AWS credentials and OpenWeather API. AWS configure will need Access key, Secret access key, Region and output format (JSON). All these can be gotten from your AWS account.

    Note: It is suggested to generate keys for an IAM user, instead of root user.

    The bucket name has to be globally unique and must follow the S3 bucket naming rules.

     aws configure
 echo "OPENWEATHER_API_KEY=<Your-Weather-Key>" >> .env
 echo "AWS_BUCKET_NAME=<your-bucket-name>" >> .env

    • Run Python Script:

      The python code should be run in the weather_dashboard .py file created earlier. You should have your S3 bucket created successfully after running the script. Remember to delete the bucket after the project.

         python3 src/weather_dashboard.py

You can feel free to try your hands on this project and share your thoughts. Check out the my Github repository. Thank you for reading!

Cloud ComputingDevopsbeginnerprojectsprogress