Photo by Lukas from Pexels

This post will take you through how to take a python app built using the streamlit package, dockerize it and deploy it to Google App Engine.

What you Need:

gitTo clone the GitHub repository that contains a streamlit app which is ready to be deployed.
Docker — To bundle application code, requirements and configuration together into a Docker image.
Google Cloud SDKTo deploy the app onto GCP (You don’t need this if you are already working on a virtual machine on Google Cloud.) (Follow directions from the install page — after downloading and unzipping the files, run the install file using…


Photo by M. B. M. on Unsplash

In this post we will write a python script that fetches stock market data using the yfinance package, processes the data and uploads the data into a Google BigQuery table which can be used for further analysis or visualization.

Fetch Stock Market Data

To fetch the data of a particular stock, we need the stock’s ticker symbol. Once you have the ticker symbol, you can either use yf.Ticker() to fetch historical stock market data for a single stock or yf.download() to fetch data for multiple tickers at the same time. Let’s fetch stock market data for stocks in the S&P 500. …


BigQuery (Source:cloud.google.com/bigquery)

What is BigQuery?
-Serverless
-Highly Scalable
-Highly Efficient
-Cloud-based Data Warehouse
-with a Distributed SQL Query Engine that can process terabytes of data in seconds.

The traditional way to work with a data warehouse is to start with an ETL job i.e. Extract the raw data from the source, Transform the data and Load it into a data warehouse. The ETL pipeline to load data into BigQuery is typically written in Apache Beam or Apache Spark which extracts raw data (either streaming data or batch files) ,transforms this data (performs cleanup and/or aggregations) and then loads it into BigQuery.


Image by slightly_different from Pixabay

We all know how frustrating job hunting can be. The numerous hours spent filling out applications, countless number of rejections and worse yet, recruiters who don’t even acknowledge your existence. It can be pretty overwhelming and at times you just feel like giving up everything and moving to the wilderness of Montana or maybe that’s just me !

So, what can you do to stand out from other candidates and have an edge over them, well, having an exceptional skill set and stellar experience definitely helps but otherwise creating a personal website to showcase the best of your abilities might…


Learn more about Google’s serverless, highly scalable and lightning fast Data Warehouse here : BigQuery.

In this article we will be looking at how to access and query data stored in BigQuery using R and Python.

Using R

We will be using the bigrquery package created by Hadley Wickham which provides a very simple and easy to use interface to Google’s BigQuery API. Go ahead and install the package if you haven’t already.

#Install and load packages
install.packages("bigrquery")
library(bigrquery)

Before you can start querying, you need to authorize bigrquery so that it can access the BigQuery projects. The easiest way to do…


  1. Order of Operations
    From
    — choose Where the data comes from and what tables to be joined
    Join is the first thing that happens when a query is executed. If you are joining tables and then using a filter on the joined data, it is a good practice to filter the data first before joining as it is quicker and less memory intensive. Pre-filtering and Pre-aggregation can be used to speed up queries by almost 50 times.
    Where — Filter the base data
    Yo can’t use a where clause to filter aggregated data as the Where is executed first before the aggregation, so…


Setup:

  1. Create an RDS instance on AWS. you can follow the steps here to create one.
  2. Install the python library that connects to an Oracle database.
    pip install cx_Oracle
  3. The cx_Oracle package requires Oracle Instant Client to connect to the database. So download the Oracle Instant Client for windows from here.
    Ref: https://oracle.github.io/odpi/doc/installation.html#windows
  4. Install Oracle Client libraries, make sure it matches your python installation (64bit/32bit).
  5. Unzip Instant Client into a directory.
  6. Add Environment Variable Path pointing to this directory.
  7. Restart Your System for the path variable changes to take affect.

Now that you have all the tools required. Open up python…


I recently completed a course on Coursera by Amazon Web Services called AWS Fundamentals: Going Cloud-Native, though i have worked with EC2 instances before, it was a very good introduction to all the other services available on AWS including their Database, Storage and Networking services. I now also have a better understanding of how the AWS platform works as a whole and the basic building blocks behind it.

The notes below covers the entire course and i tried to capture everything that is essential as succinctly as possible. …


In this post we will look at how to install ChromeDriver on your aws EC2 instance and get selenium up and running.

Check out my previous posts.

1. https://medium.com/@praneeth.jm/launching-and-connecting-to-an-aws-ec2-instance-6678f660bbe6

2. https://medium.com/@praneeth.jm/running-python-scripts-on-an-aws-ec2-instance-8c01f9ee7b2f

Installing ChromeDriver

  1. Change directory to temp folder.
  2. Get ChromeDriver using wget
  3. Unzip ChromeDriver
  4. Move ChromeDriver to the bin applications folder.
cd/tmp/
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
chromedriver --version

Installing Google Chrome

For the ChromeDriver to work we also need to install Google Chrome. After you run the first the curl command in the code block below, it might take a little while to download and install the necessary files.

curl https://intoli.com/install-google-chrome.sh…


So you have an EC2 instance up and running on AWS (If you don’t have it already, take a look at this post: https://medium.com/@praneeth.jm/launching-and-connecting-to-an-aws-ec2-instance-6678f660bbe6).

Now let’s see how we can setup a python environment, transfer python scripts from your local machine to the remote instance and run them.

AWS EC2 Instance on Putty

To see everything that’s installed in your instance, type in the following commands:

cd /usr/bin/
ls
  1. You will probably see python 2.7 already installed which is the older version, so let’s go ahead and install python 3.6
sudo yum install python36

Even after installing python 3.6, running python --version in the Putty…

Praneeth Kandula

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store