Trending February 2024 # How Machine Learning Is Enhancing Mobile Gadgets And Applications # Suggested March 2024 # Top 10 Popular

You are reading the article How Machine Learning Is Enhancing Mobile Gadgets And Applications updated in February 2024 on the website Kientrucdochoi.com. We hope that the information we have shared is helpful to you. If you find the content interesting and meaningful, please share it with your friends and continue to follow and support us for the latest updates. Suggested March 2024 How Machine Learning Is Enhancing Mobile Gadgets And Applications

Mobile developers have a great deal to pick up from progressive changes that on-device ML can offer. This is a result of the innovation’s capacity to reinforce mobile applications—in particular, taking into consideration smoother customer experiences equipped for utilizing incredible highlights, for example, giving exact area-based recommendations or promptly detecting plant illnesses. This fast improvement of mobile machine learning has occurred as a response to various basic issues that traditional machine learning has toiled with. In truth, the composing is on the divider. Future mobile applications will require faster preparing paces and lower latency.  

Better Voice Services and Reduction in Churn Rate

Voice services are telecoms’ characteristic specialized field. A few organizations are banding together with the pioneers in speech and voice services, joining, for instance, Alexa environment. Others build up their own solutions or secure smaller new businesses. South Korean organizations are standing out. As of late, SK Telecom has presented its artificial intelligence-based voice assistant service for the home, which was a response to the move by its local rival – KT deployed its artificial intelligence collaborator to a hotel in South Korea with English language support. Machine Learning is additionally helpful in decreasing churn rates, which can average every year from 10 to as much as 67%. Telecoms can prepare algorithms to anticipate when a customer is probably going to go to another organization, and what offer could keep them from doing it.  

Smart Home Services Improved Security

Since information shouldn’t be sent to a server or the cloud for processing, cybercriminals have fewer chances to misuse any vulnerabilities in this data transference, along these lines safeguarding the sanctity of the information. This enables mobile developers to meet GDPR guidelines on data security all the more effectively. On-device ML solutions likewise offer decentralization, much similarly that blockchain does. As it were, it’s harder for hackers to bring down a connected system of concealed devices through a DDoS assault when compared with a similar assault focusing on a centralised server. This innovation could likewise demonstrate to be valuable for drones and law authorization pushing ahead. The Apple cell phone chips are likewise improving client security and privacy. For example, they fill in as the foundation of Face ID. This iPhone feature depends on an on-device neural net that gathers data on all the various ways its user’s face may look, filling in as a more precise and secure identification strategy. This and future classes of AI-empowered hardware will make ready for more secure cell phone experiences for clients, offering mobile engineers an extra layer of encryption to protect clients’ information.  

Predictive Maintenance

Mobile towers are the ideal product for ML predictive maintenance solutions. They are hard to access and require tedious on-site investigations of muddled modules, for example, power generators or air conditioners. Also, towers are powerless against interruptions, as they contain a ton of significant hardware. There are different potential uses of ML in the maintenance of mobile towers, for example, empowered surveillance, where video and picture analysis can help recognize peculiarities. The media communications infrastructure is as of now outfitted with different sensors. The information those sensors gather can be utilized for preparing ML models, which will anticipate potential disappointments. This would diminish downtime and fix expenses, and furthermore improve the coverage. Nokia utilizes ML algorithms are to change the best setup for 5G antennas and indoor positioning of articles or designing uplink and downlink channels. An active system configuration data can be cross-checked with asset management frameworks to amplify network use and improve coverage.  

Lower Costs

On-device MLI is likewise intended to spare you a fortune, as you won’t need to pay external suppliers to actualize or keep up these solutions. As recently referenced, you won’t require the cloud or the Internet for such solutions. GPUs and AI-explicit chips are the costliest cloud services you can purchase. Running models on-device implies you don’t have to pay for these clusters, because of the inexorably complex Neural Processing Units (NPUs) cell phones have nowadays. Keeping away from the overwhelming, data-processing bad dream among mobile and the cloud is an enormous cost-saver for organizations that pick on-device ML solutions. Having this on-device derivation likewise brings down data bandwidth demands, at last sparing a heavy sum in expenses.

You're reading How Machine Learning Is Enhancing Mobile Gadgets And Applications

Is Reinforcement (Machine) Learning Overhyped?

Imagine you are about to sit down to play a game with a friend. But this isn’t just any friend – it’s a computer program that doesn’t know the rules of the game. It does, however, understand that it has a goal, and that goal is to win.

Because this friend doesn’t know the rules, it starts by making random moves. Some of them make absolutely no sense, and winning for you is easy. But let’s just say you enjoy playing with this friend so much that you decide to devote the rest of your life (and future lives if you believe in that idea) to exclusively playing this game.

The digital friend will eventually win because it gradually learns the winning moves required to beat you. This scenario may seem far-fetched, but it should give you a basic idea of how reinforcement learning (RL) – an area of machine learning (ML) – roughly works.

Just how intelligent is reinforcement learning?

Human intelligence encompasses many characteristics, including the attainment of knowledge, a desire to expand intellectual capacity, and intuitive thinking. Our capacity for intelligence, however, was largely questioned when Garry Kasparov, a champion chess player, lost to an IBM computer named Deep Blue. Besides capturing the attention of the public, doomsday scenarios depicting a world where robots rule humans took hold of mainstream consciousness.

Deep Blue, however, was not an average opponent. Playing with this program is analogous to a match with a thousand-year-old human that devoted their entire life to continuously playing chess. Accordingly, Deep Blue was skilled in playing a specific game – not in other intellectual pursuits like playing an instrument, writing a book, conducting a scientific experiment, raising a child, or fixing a car.

In no way am I attempting to downplay the achievement of the creation of Deep Blue. Instead, I am simply suggesting that the idea that computers can surpass us in intellectual capability requires careful examination, starting with a breakdown of RL mechanics.

How Reinforcement Learning Works

As mentioned previously, RL is a subset of ML concerned with how intelligent agents should act in an environment to maximize the notion of cumulative reward.

In plain terms, RL robot agents are trained on a reward and punishment mechanism where they are rewarded for correct moves and punished for the wrong ones. RL Robots don’t “think” about the best actions to make – they just make all the moves possible in order to maximize chances of success.

Drawbacks of Reinforcement Learning

The main drawback of reinforcement learning is the exorbitant amount of resources it requires to achieve its goal. This is illustrated by the success of RL in another game called GO – a popular 2-player game where the goal is to use playing pieces (called stones) to maximize territory on a board while avoiding the loss of stones.

AlphaGo Master, a computer program that defeated human players in Go, required a massive investment that included many engineers, thousands of years worth of game-playing experience, and an astonishing 256 GPUs and 128,000 CPU cores.

That’s a lot of energy to use in learning to win a game. This then begs the question of whether it is rational to design AI that cannot think intuitively. Shouldn’t AI research attempt to mimic human intelligence?

One argument favoring RL is that we should not expect AI agents to behave like humans, and its use to solve complex problems warrants further development. On the other hand, an argument against RL is that AI research should focus on enabling machines to do things that only humans and animals are presently capable of doing. When viewed in that light, AI’s comparison to human intelligence is appropriate.

Quantum Reinforcement Learning

There’s an emerging field of reinforcement learning that purportedly solves some of the problems outlined above. Quantum reinforcement learning (QRL) has been studied as a way to speed up calculations.

Primarily, QRL should speed up learning by optimizing the exploration (finding strategies) and exploitation (picking the best strategy) phases. Some of the current applications and proposed quantum calculations improve database search, factoring large numbers into primes, and much more.

While QRL still hasn’t arrived in a groundbreaking fashion, there’s an expectation that it may resolve some of the great challenges for regular reinforcement learning.

Business Cases for RL

As I mentioned before, in no way do I want to undermine the importance of RL research and development. In fact, at Oxylabs, we have been working on RL models that will optimize web scraping resource allocation.

With that said, here is just a sample of some real-life uses for RL derived from a McKinsey report highlighting current use cases across a wide range of industries:

Optimizing silicon and chip design, optimizing manufacturing processes, and improving yields for the semiconductor industry

Increasing yields, optimizing logistics to reduce waste and costs, and improving margins in agriculture

Reducing time to market for new systems in the aerospace and defense industries

Optimizing design processes and increasing manufacturing yields for the automotive industries

Optimizing mine design, managing power generation and applying holistic logistics scheduling to optimize operations, reduce costs and increase yields in mining

Increasing yields through real-time monitoring and precision drilling, optimizing tanker routing and enabling predictive maintenance to prevent equipment failure and outages in the oil and gas industry

Facilitating drug discovery, optimizing research processes, automating production and optimizing biologic methods for the pharmaceutical industry

Optimizing and managing networks and applying customer personalization in the telecom industry

Optimizing routing, network planning, warehouse operations in transport and logistics

Extracting data from websites with the use of next-generation proxies

  Rethinking Reinforcement Learning

Reinforcement learning may be limited, but it’s hardly overrated. Moreover, as research and development into RL increases, so do potential use cases across almost every sector of the economy.

Wide-scale adoption depends on several factors, including optimizing the design of algorithms, configuring learning environments, and the availability of computing power.

Author:

Enhancing Flipped Learning With Microlectures

Flipped learning turns the traditional model of instruction on its head. Robert Talbert, author of Flipped Learning, explains that in a flipped environment, students’ “first contact with new concepts moves from the group learning space to the individual learning space.” In other words, students first learn about new material through individual work so that the time that they are together with their instructor and peers can be devoted to interactive, higher-order activities where they can benefit from the support of others.

In their book In-Class Flip, Martha A. Ramírez and Carolina R. Buitrago note that a variety of content can be used to facilitate learning in the individual space, including videos, readings, sketchnotes, infographics, podcasts, and HyperDocs. Talbert explains that individual-space tasks should be well-structured, guiding students in processing and making sense of the new content. The three authors also discuss the importance of incorporating accountability into individual-space tasks by having students complete notes, quizzes, practice exercises, or other activities.

When utilizing video content for flipped learning, one helpful structure that educators can use is microlectures, which are instructional videos that focus on one particular concept or skill. (For a concrete example of a microlecture, check out this video I made for preservice teachers.) Instructional designer Hua Zheng explains that microlectures have three key characteristics:

They are shorter than 10 minutes. (In fact, research indicates that six minutes is the sweet spot for maintaining student engagement.)

They foster connection between learners and the instructor.

They prompt learners to actively engage with the content rather than passively taking it all in.

Educators can utilize the microlecture approach to create enriching flipped content for students to engage with during the individual space.

Video Design Principles and Accessibility

When creating microlectures, it’s important to keep research-based video design principles in mind. Vanderbilt University professor Cynthia J. Brame highlights four principles that can be particularly helpful for managing cognitive load:

Signaling. Point out key ideas in the video. This can be done through verbal emphasis on certain concepts or through visual cues (such as arrows or highlighting).

Segmenting. Chunk the content into manageable pieces. This makes it easier for students to follow along and see how the concepts are building upon each other.

Weeding. Omit any unnecessary information that does not align with the learning objective(s) and any visuals that may be distracting to learners. This includes audio (like background music) along with visuals (such as unnecessary graphics or busy slides).

Matching modality. Provide information in multiple formats. For example, convey information both aurally and through related visuals on the screen.

Additionally, it’s important to ensure that your microlectures are accurately captioned for accessibility and other purposes. The Clips app for iPad and iPhone, for example, makes it easy to record videos with accurate, embedded captions (via the Live Titles feature). With Flip, you can edit captions via the web, and Loom allows you to edit the video transcript, which will automatically update the captions as well.

Keep It Active

There are numerous ways to build interactivity, structure, and accountability into your microlectures. For example, platforms like Edpuzzle and PlayPosit allow you to embed interactive questions into your videos. Zheng notes that you can also simply tell students to pause the video and complete an activity or use a timer. For example, you can easily embed a YouTube timer in Google Slides, PowerPoint, or Keynote and play it whenever you want students to complete an activity.

Below are a few examples of activities that students can pause and complete while watching microlectures. Please note that there are no rules regarding how many pauses to include or when to include them; do what you feel is best, based on your content, context, and knowledge of your students.

Beginning of the microlecture: Activate students’ prior knowledge and spark their interest in the new content by having them complete a retrieval practice activity, quick write, entrance ticket, anticipation guide, Know-Wonder-Learn chart, brainstorm, poll, or thinking routine (such as See, Think, Wonder or 3-2-1 Bridge).

Middle of the microlecture: Have students complete a graphic organizer or guided notes sheet as they learn about the concept or skill, and provide them with opportunities to engage with the content. For example, students can pause the video to create sketchnotes, complete sample problems, respond to a scenario, participate in a virtual gallery walk, do a writing exercise, complete a mini choice board, practice a skill, and more!

End of the microlecture: Have students complete a check for understanding and to reflect on their learning. For example, students can complete an exit ticket, a geometric forms reflection, or a thinking routine (such as “I used to think… now I think…” or Connect, Extend, Challenge).

Alternatively, you can add the microlecture (along with descriptive hyperlinks to any digital tasks) to a Google Slide or Doc; an Apple Keynote, Numbers, or Pages document; a Wakelet collection; a Padlet board; or a webpage so that students have a one-stop shop for everything they need. You can even add the microlecture to a Google Form so that students can watch the video and respond to questions in the same place!

Next Steps

Ready to start creating microlectures for your learners? Check out this article I wrote that includes a helpful planning template, suggested tools for creating microlectures, and other helpful tips. Additionally, this microlecture peer review rubric can help you evaluate and revise your microlectures in order to ensure that they employ video-design best practices.

Google Colab For Machine Learning And Deep Learning

“Memory Error” – that all too familiar dreaded message in Jupyter notebooks when we try to execute a machine learning or deep learning algorithm on a large dataset. Most of us do not have access to unlimited computational power on our machines. And let’s face it, it costs an arm and a leg to get a decent GPU from existing cloud providers. So how do we build large deep learning models without burning a hole in our pockets? Step up – Google Colab!

It’s an incredible online browser-based platform that allows us to train our models on machines for free! Sounds too good to be true, but thanks to Google, we can now work with large datasets, build complex models, and even share our work seamlessly with others. That’s the power of Google Colab.

What is Google Colab?

Google Colaboratory is a free online cloud-based Jupyter notebook environment that allows us to train our machine learning and deep learning models on CPUs, GPUs, and TPUs.

Here’s what I truly love about Colab. It does not matter which computer you have, what it’s configuration is, and how ancient it might be. You can still use Google Colab! All you need is a Google account and a web browser. And here’s the cherry on top – you get access to GPUs like Tesla K80 and even a TPU, for free!

TPUs are much more expensive than a GPU, and you can use it for free on Colab. It’s worth repeating again and again – it’s an offering like no other.

Are you are still using that same old Jupyter notebook on your system for training models? Trust me, you’re going to love Google Colab.

What is a Notebook in Google Colab? Google Colab Features

Colab provides users free access to GPUs and TPUs, which can significantly speed up the training and inference of machine learning and deep learning models.

Colab’s interface is web-based, so installing any software on your local machine is unnecessary. The interface is also intuitive and user-friendly, making it easy to get started with coding.

Colab allows multiple users to work on the same notebook simultaneously, making collaborating with team members easy. Colab also integrates with other Google services, such as Google Drive and GitHub, making it easy to share your work.

Colab notebooks support markdown, which allows you to include formatted text, equations, and images alongside your code. This makes it easier to document your work and communicate your ideas.

Colab comes pre-installed with many popular libraries and tools for machine learning and deep learning, such as TensorFlow and PyTorch. This saves time and eliminates the need to manually install and configure these tools.

GPUs and TPUs on Google Colab

Ask anyone who uses Colab why they love it. The answer is unanimous – the availability of free GPUs and TPUs. Training models, especially deep learning ones, takes numerous hours on a CPU. We’ve all faced this issue on our local machines. GPUs and TPUs, on the other hand, can train these models in a matter of minutes or seconds.

If you still need a reason to work with GPUs, check out this excellent explanation by Faizan Shaikh.

It gives you a decent GPU for free, which you can continuously run for 12 hours. For most data science folks, this is sufficient to meet their computation needs. Especially if you are a beginner, then I would highly recommend you start using Google Colab.

Google Colab gives us three types of runtime for our notebooks:

CPUs,

GPUs, and

TPUs

As I mentioned, Colab gives us 12 hours of continuous execution time. After that, the whole virtual machine is cleared and we have to start again. We can run multiple CPU, GPU, and TPU instances simultaneously, but our resources are shared between these instances.

Let’s take a look at the specifications of different runtimes offered by Google Colab:

It will cost you A LOT to buy a GPU or TPU from the market. Why not save that money and use Google Colab from the comfort of your own machine?

How to Use Google Colab?

You can go to Google Colab using this link. This is the screen you’ll get when you open Colab:

You can also import your notebook from Google Drive or GitHub, but they require an authentication process.

Google Colab Runtimes – Choosing the GPU or TPU Option

The ability to choose different types of runtimes is what makes Colab so popular and powerful. Here are the steps to change the runtime of your notebook:

Step 2: Here you can change the runtime according to your need:

A wise man once said, “With great power comes great responsibility.” I implore you to shut down your notebook after you have completed your work so that others can use these resources because various users share them. You can terminate your notebook like this:

Using Terminal Commands on Google Colab

You can use the Colab cell for running terminal commands. Most of the popular libraries come installed by default on Google Colab. Yes, Python libraries like Pandas, NumPy, scikit-learn are all pre-installed.

If you want to run a different Python library, you can always install it inside your Colab notebook like this:

!pip install 

library_name

Pretty easy, right? Everything is similar to how it works in a regular terminal. We just you have to put an exclamation(!) before writing each command like:

!ls

or:

!pwd

Cloning Repositories in Google Colab

You can also clone a Git repo inside Google Colaboratory. Just go to your GitHub repository and copy the clone link of the repository:

Then, simply run:

And there you go!

Uploading Files and Datasets

Here’s a must-know aspect for any data scientist. The ability to import your dataset into Colab is the first step in your data analysis journey.

The most basic approach is to upload your dataset to Colab directly:

You can also upload your dataset to any other platform and access it using its link. I tend to go with the second approach more often than not (when feasible).

Saving Your Notebook

All the notebooks on Colab are stored on your Google Drive. The best thing about Colab is that your notebook is automatically saved after a certain time period and you don’t lose your progress.

If you want, you can export and save your notebook in both *.py and *.ipynb formats:

Not just that, you can also save a copy of your notebook directly on GitHub, or you can create a GitHub Gist:

I love the variety of options we get.

Exporting Data/Files from Google Colab

You can export your files directly to Google Drive, or you can export it to the VM instance and download it by yourself:

Exporting directly to the Drive is a better option when you have bigger files or more than one file. You’ll pick up these nuances as you work on bigger projects in Colab.

Sharing Your Notebook

Google Colab also gives us an easy way of sharing our work with others. This is one of the best things about Colab:

What’s Next?

Google Colab now also provides a paid platform called Google Colab Pro, priced at $9.99 a month. In this plan, you can get the Tesla T4 or Tesla P100 GPU, and an option of selecting an instance with a high RAM of around 27 GB. Also, your maximum computation time is doubled from 12 hours to 24 hours. How cool is that?

You can consider this plan if you need high computation power because it is still quite cheap when compared to other cloud GPU providers like AWS, Azure, and even GCP.

Recommendations

If you’re new to the world of Deep Learning, I have some excellent resources to help you get started in a comprehensive and structured manner:

Related

How Machine Learning Improves Cybersecurity?

Here is how machine learning improves cybersecurity

Today, deploying robust cybersecurity solutions is unfeasible without significantly depending on machine learning. Simultaneously, without a thorough, rich, and full approach to the data set, it is difficult to properly use machine learning. MI can be used by cybersecurity systems to recognise patterns and learn from them in order to detect and prevent repeated attacks and adjust to different behaviour. It can assist cybersecurity teams in being more proactive in preventing dangers and responding to live attacks. It can help businesses use their assets more strategically by reducing the amount of time invested in mundane tasks.  

Machine Learning in Cyber Security

ML may be used in different areas within Cyber Security to improve security procedures and make it simpler for security analysts to swiftly discover, prioritise, cope with, and remediate new threats in order to better comprehend previous cyber-attacks and build appropriate defence measures.  

Automating Tasks

The potential of machine learning in cyber security to simplify repetitive and time-consuming processes like triaging intelligence, malware detection, network log analysis, and vulnerability analysis is a significant benefit. By adding machine learning into the security workflow, businesses may complete activities quicker and respond to and remediate risks at a rate that would be impossible to do with only manual human capabilities. By automating repetitive operations, customers may simply scale up or down without changing the number of people required, lowering expenses. AutoML is a term used to describe the process of using machine learning to automate activities. When repetitive processes in development are automated to help analysts, data scientists, and developers be more productive, this is referred to as AutoML.  

Threat Detection and Classification

In order to identify and respond to threats, machine learning techniques are employed in applications. This may be accomplished by analysing large data sets of security events and finding harmful behaviour patterns. When comparable occurrences are recognised, ML works to autonomously deal with them using the trained ML model. For example, utilising Indicators of Compromise, a database to feed a machine learning model may be constructed (IOCs). These can aid in real-time monitoring, identification, and response to threats. Malware activity may be classified using ML classification algorithms and IOC data sets. A study by Darktrace, a Machine Learning based Enterprise Immune Solution, alleges to have stopped assaults during the WannaCry ransomware outbreak as an example of such an application.  

Phishing

Traditional phishing detection algorithms aren’t fast enough or accurate enough to identify and distinguish between innocent and malicious URLs. Predictive URL categorization methods based on the latest machine learning algorithms can detect trends that signal fraudulent emails. To accomplish so, the models are trained on characteristics such as email headers, body data, punctuation patterns, and more in order to categorise and distinguish the harmful from the benign.  

WebShell

WebShell is a malicious block of software that is put into a website and allows users to make changes to the server’s web root folder. As a result, attackers have access to the database. As a result, the bad actor is able to acquire personal details. A regular shopping cart behaviour may be recognised using machine learning, and the system can be programmed to distinguish between normal and malicious behaviour.  

Network Risk Scoring

Quantitative methods for assigning risk rankings to network segments aid organisations in prioritising resources. ML may be used to examine prior cyber-attack datasets and discover which network regions were more frequently targeted in certain assaults. With regard to a specific network region, this score can assist assess the chance and effect of an attack. As a result, organisations are less likely to be targets of future assaults. When doing company profiling, you must determine which areas, if compromised, can ruin your company. It might be a CRM system, accounting software, or a sales system. It’s all about determining which areas of your business are the most vulnerable. If, for example, HR suffers a setback, your firm may have a low-risk rating. However, if your oil trading system goes down, your entire industry may go down with it. Every business has its own approach to security. And once you grasp the intricacies of a company, you’ll know what to safeguard. And if a hack occurs, you’ll know what to prioritise.  

Human Interaction

Computers, as we all know, are excellent at solving complex problems and automating things that people might accomplish, but which PCs excel at. Although AI is primarily concerned with computers, people are required to make educated judgments and receive orders. As a result, we may conclude that people cannot be replaced by machines. Machine learning algorithms are excellent at interpreting spoken language and recognising faces, but they still require people in the end.  

Conclusion

Machine learning is a powerful technology. However, it is not a magic bullet. It’s crucial to remember that, while technology is improving and AI and machine learning are progressing at a rapid pace, technology is only as powerful as the brains of the analysts who manage and use it.

Stock Prices Prediction Using Machine Learning And Deep Learning

17 minutes

⭐⭐⭐⭐⭐

Rating: 5 out of 5.

Introduction

Predicting how the stock market will perform is one of the most difficult things to do. There are so many factors involved in the prediction – physical factors vs. psychological, rational and irrational behavior, etc. All these aspects combine to make share prices volatile and very difficult to predict with a high degree of accuracy.

Can we use machine learning as a game-changer in this domain? Using features like the latest announcements about an organization, their quarterly revenue results, etc., machine learning techniques have the potential to unearth patterns and insights we didn’t see before, and these can be used to make unerringly accurate predictions.

The core idea behind this article is to showcase how these algorithms are implemented. I will briefly describe the technique and provide relevant links to brush up on the concepts as and when necessary. In case you’re a newcomer to the world of time series, I suggest going through the following articles first:

Are you a beginner looking for a place to start your data science journey? Presenting a comprehensive course, full of knowledge and data science learning, curated just for you! This course covers everything from basics of Machine Learning to Advanced concepts of ML, Deep Learning and Time series.

Understanding the Problem Statement

We’ll dive into the implementation part of this article soon, but first it’s important to establish what we’re aiming to solve. Broadly, stock market analysis is divided into two parts – Fundamental Analysis and Technical Analysis.

Fundamental Analysis involves analyzing the company’s future profitability on the basis of its current business environment and financial performance.

Technical Analysis, on the other hand, includes reading the charts and using statistical figures to identify the trends in the stock market.

As you might have guessed, our focus will be on the technical analysis part. We’ll be using a dataset from Quandl (you can find historical data for various stocks here) and for this particular project, I have used the data for ‘Tata Global Beverages’. Time to dive in!

Note: Here is the dataset I used for the code: Download

We will first load the dataset and define the target variable for the problem:

Python Code:



There are multiple variables in the dataset – date, open, high, low, last, close, total_trade_quantity, and turnover.

The columns Open and Close represent the starting and final price at which the stock is traded on a particular day.

High, Low and Last represent the maximum, minimum, and last price of the share for the day.

Total Trade Quantity is the number of shares bought or sold in the day and Turnover (Lacs) is the turnover of the particular company on a given date.

Another important thing to note is that the market is closed on weekends and public holidays. Notice the above table again, some date values are missing – 2/10/2024, 6/10/2024, 7/10/2024. Of these dates, 2nd is a national holiday while 6th and 7th fall on a weekend.

The profit or loss calculation is usually determined by the closing price of a stock for the day, hence we will consider the closing price as the target variable. Let’s plot the target variable to understand how it’s shaping up in our data:

#setting index as date df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d') df.index = df['Date'] #plot plt.figure(figsize=(16,8)) plt.plot(df['Close'], label='Close Price history')

In the upcoming sections, we will explore these variables and use different techniques to predict the daily closing price of the stock.

Moving Average Introduction

‘Average’ is easily one of the most common things we use in our day-to-day lives. For instance, calculating the average marks to determine overall performance, or finding the average temperature of the past few days to get an idea about today’s temperature – these all are routine tasks we do on a regular basis. So this is a good starting point to use on our dataset for making predictions.

The predicted closing price for each day will be the average of a set of previously observed values. Instead of using the simple average, we will be using the moving average technique which uses the latest set of values for each prediction. In other words, for each subsequent step, the predicted values are taken into consideration while removing the oldest observed value from the set. Here is a simple figure that will help you understand this with more clarity.

We will implement this technique on our dataset. The first step is to create a dataframe that contains only the Date and Close price columns, then split it into train and validation sets to verify our predictions.

Implementation



Just checking the RMSE does not help us in understanding how the model performed. Let’s visualize this to get a more intuitive understanding. So here is a plot of the predicted values along with the actual values.

#plot valid['Predictions'] = 0 valid['Predictions'] = preds plt.plot(train['Close']) plt.plot(valid[['Close', 'Predictions']]) Inference

The RMSE value is close to 105 but the results are not very promising (as you can gather from the plot). The predicted values are of the same range as the observed values in the train set (there is an increasing trend initially and then a slow decrease).

In the next section, we will look at two commonly used machine learning techniques – Linear Regression and kNN, and see how they perform on our stock market data.

Linear Regression Introduction

The most basic machine learning algorithm that can be implemented on this data is linear regression. The linear regression model returns an equation that determines the relationship between the independent variables and the dependent variable.

The equation for linear regression can be written as:

Here, x1, x2,….xn represent the independent variables while the coefficients θ1, θ2, …. θn  represent the weights. You can refer to the following article to study linear regression in more detail:

For our problem statement, we do not have a set of independent variables. We have only the dates instead. Let us use the date column to extract features like – day, month, year,  mon/fri etc. and then fit a linear regression model.

Implementation

We will first sort the dataset in ascending order and then create a separate dataset so that any new feature created does not affect the original data.

#setting index as date values df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d') df.index = df['Date'] #sorting data = df.sort_index(ascending=True, axis=0) #creating a separate dataset new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close']) for i in range(0,len(data)): new_data['Date'][i] = data['Date'][i] new_data['Close'][i] = data['Close'][i] #create features from fastai.structured import add_datepart add_datepart(new_data, 'Date') new_data.drop('Elapsed', axis=1, inplace=True)  #elapsed will be the time stamp

This creates features such as:

‘Year’, ‘Month’, ‘Week’, ‘Day’, ‘Dayofweek’, ‘Dayofyear’, ‘Is_month_end’, ‘Is_month_start’, ‘Is_quarter_end’, ‘Is_quarter_start’,  ‘Is_year_end’, and  ‘Is_year_start’.

Note: I have used add_datepart from fastai library. If you do not have it installed, you can simply use the command pip install fastai. Otherwise, you can create these feature using simple for loops in python. I have shown an example below.

Apart from this, we can add our own set of features that we believe would be relevant for the predictions. For instance, my hypothesis is that the first and last days of the week could potentially affect the closing price of the stock far more than the other days. So I have created a feature that identifies whether a given day is Monday/Friday or Tuesday/Wednesday/Thursday. This can be done using the following lines of code:

new_data['mon_fri'] = 0 for i in range(0,len(new_data)): if (new_data['Dayofweek'][i] == 0 or new_data['Dayofweek'][i] == 4):     new_data['mon_fri'][i] = 1 else:     new_data['mon_fri'][i] = 0

We will now split the data into train and validation sets to check the performance of the model.

#split into train and validation train = new_data[:987] valid = new_data[987:] x_train = train.drop('Close', axis=1) y_train = train['Close'] x_valid = valid.drop('Close', axis=1) y_valid = valid['Close'] #implement linear regression from sklearn.linear_model import LinearRegression model = LinearRegression() model.fit(x_train,y_train) Results #make predictions and find the rmse preds = model.predict(x_valid) rms=np.sqrt(np.mean(np.power((np.array(y_valid)-np.array(preds)),2))) rms 121.16291596523156

The RMSE value is higher than the previous technique, which clearly shows that linear regression has performed poorly. Let’s look at the plot and understand why linear regression has not done well:

#plot valid['Predictions'] = 0 valid['Predictions'] = preds valid.index = new_data[987:].index train.index = new_data[:987].index plt.plot(train['Close']) plt.plot(valid[['Close', 'Predictions']]) Inference

As seen from the plot above, for January 2024 and January 2023, there was a drop in the stock price. The model has predicted the same for January 2023. A linear regression technique can perform well for problems such as Big Mart sales where the independent features are useful for determining the target value.

k-Nearest Neighbours Introduction

Another interesting ML algorithm that one can use here is kNN (k nearest neighbours). Based on the independent variables, kNN finds the similarity between new data points and old data points. Let me explain this with a simple example.

Consider the height and age for 11 people. On the basis of given features (‘Age’ and ‘Height’), the table can be represented in a graphical format as shown below:

To determine the weight for ID #11, kNN considers the weight of the nearest neighbors of this ID. The weight of ID #11 is predicted to be the average of it’s neighbors. If we consider three neighbours (k=3) for now, the weight for ID#11 would be = (77+72+60)/3 = 69.66 kg.

For a detailed understanding of kNN, you can refer to the following articles:

Implementation #importing libraries from sklearn import neighbors from sklearn.model_selection import GridSearchCV from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler(feature_range=(0, 1))

Using the same train and validation set from the last section:

#scaling data x_train_scaled = scaler.fit_transform(x_train) x_train = pd.DataFrame(x_train_scaled) x_valid_scaled = scaler.fit_transform(x_valid) x_valid = pd.DataFrame(x_valid_scaled) #using gridsearch to find the best parameter params = {'n_neighbors':[2,3,4,5,6,7,8,9]} knn = neighbors.KNeighborsRegressor() model = GridSearchCV(knn, params, cv=5) #fit the model and make predictions model.fit(x_train,y_train) preds = model.predict(x_valid) Results #rmse rms=np.sqrt(np.mean(np.power((np.array(y_valid)-np.array(preds)),2))) rms 115.17086550026721

There is not a huge difference in the RMSE value, but a plot for the predicted and actual values should provide a more clear understanding.

#plot valid['Predictions'] = 0 valid['Predictions'] = preds plt.plot(valid[['Close', 'Predictions']]) plt.plot(train['Close']) Inference

The RMSE value is almost similar to the linear regression model and the plot shows the same pattern. Like linear regression, kNN also identified a drop in January 2023 since that has been the pattern for the past years. We can safely say that regression algorithms have not performed well on this dataset.

Let’s go ahead and look at some time series forecasting techniques to find out how they perform when faced with this stock prices prediction challenge.

Auto ARIMA Introduction

ARIMA is a very popular statistical method for time series forecasting. ARIMA models take into account the past values to predict the future values. There are three important parameters in ARIMA:

p (past values used for forecasting the next value)

q (past forecast errors used to predict the future values)

d (order of differencing)

Parameter tuning for ARIMA consumes a lot of time. So we will use auto ARIMA which automatically selects the best combination of (p,q,d) that provides the least error. To read more about how auto ARIMA works, refer to this article:

Implementation from pyramid.arima import auto_arima data = df.sort_index(ascending=True, axis=0) train = data[:987] valid = data[987:] training = train['Close'] validation = valid['Close'] model = auto_arima(training, start_p=1, start_q=1,max_p=3, max_q=3, m=12,start_P=0, seasonal=True,d=1, D=1, trace=True,error_action='ignore',suppress_warnings=True) model.fit(training) forecast = model.predict(n_periods=248) forecast = pd.DataFrame(forecast,index = valid.index,columns=['Prediction']) Results rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-np.array(forecast['Prediction'])),2))) rms 44.954584993246954 #plot plt.plot(train['Close']) plt.plot(valid['Close']) plt.plot(forecast['Prediction']) Inference

As we saw earlier, an auto ARIMA model uses past data to understand the pattern in the time series. Using these values, the model captured an increasing trend in the series. Although the predictions using this technique are far better than that of the previously implemented machine learning models, these predictions are still not close to the real values.

As its evident from the plot, the model has captured a trend in the series, but does not focus on the seasonal part. In the next section, we will implement a time series model that takes both trend and seasonality of a series into account.

Prophet Introduction

There are a number of time series techniques that can be implemented on the stock prediction dataset, but most of these techniques require a lot of data preprocessing before fitting the model. Prophet, designed and pioneered by Facebook, is a time series forecasting library that requires no data preprocessing and is extremely simple to implement. The input for Prophet is a dataframe with two columns: date and target (ds and y).

Prophet tries to capture the seasonality in the past data and works well when the dataset is large. Here is an interesting article that explains Prophet in a simple and intuitive manner:

Implementation #importing prophet from fbprophet import Prophet #creating dataframe new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close']) for i in range(0,len(data)): new_data['Date'][i] = data['Date'][i] new_data['Close'][i] = data['Close'][i] new_data['Date'] = pd.to_datetime(new_data.Date,format='%Y-%m-%d') new_data.index = new_data['Date'] #preparing data new_data.rename(columns={'Close': 'y', 'Date': 'ds'}, inplace=True) #train and validation train = new_data[:987] valid = new_data[987:] #fit the model model = Prophet() model.fit(train) #predictions close_prices = model.make_future_dataframe(periods=len(valid)) forecast = model.predict(close_prices) Results #rmse forecast_valid = forecast['yhat'][987:] rms=np.sqrt(np.mean(np.power((np.array(valid['y'])-np.array(forecast_valid)),2))) rms 57.494461930575149 #plot valid['Predictions'] = 0 valid['Predictions'] = forecast_valid.values plt.plot(train['y']) plt.plot(valid[['y', 'Predictions']]) Inference

Prophet (like most time series forecasting techniques) tries to capture the trend and seasonality from past data. This model usually performs well on time series datasets, but fails to live up to it’s reputation in this case.

As it turns out, stock prices do not have a particular trend or seasonality. It highly depends on what is currently going on in the market and thus the prices rise and fall. Hence forecasting techniques like ARIMA, SARIMA and Prophet would not show good results for this particular problem.

Long Short Term Memory (LSTM) Introduction

LSTMs are widely used for sequence prediction problems and have proven to be extremely effective. The reason they work so well is because LSTM is able to store past information that is important, and forget the information that is not. LSTM has three gates:

The input gate: The input gate adds information to the cell state

The forget gate: It removes the information that is no longer required by the model

The output gate: Output Gate at LSTM selects the information to be shown as output

For a more detailed understanding of LSTM and its architecture, you can go through the below article:

For now, let us implement LSTM as a black box and check it’s performance on our particular data.

Implementation #importing required libraries from sklearn.preprocessing import MinMaxScaler from keras.models import Sequential from keras.layers import Dense, Dropout, LSTM #creating dataframe data = df.sort_index(ascending=True, axis=0) new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close']) for i in range(0,len(data)): new_data['Date'][i] = data['Date'][i] new_data['Close'][i] = data['Close'][i] #setting index new_data.index = new_data.Date new_data.drop('Date', axis=1, inplace=True) #creating train and test sets dataset = new_data.values train = dataset[0:987,:] valid = dataset[987:,:] #converting dataset into x_train and y_train scaler = MinMaxScaler(feature_range=(0, 1)) scaled_data = scaler.fit_transform(dataset) x_train, y_train = [], [] for i in range(60,len(train)): x_train.append(scaled_data[i-60:i,0]) y_train.append(scaled_data[i,0]) x_train, y_train = np.array(x_train), np.array(y_train) x_train = np.reshape(x_train, (x_train.shape[0],x_train.shape[1],1)) # create and fit the LSTM network model = Sequential() model.add(LSTM(units=50, return_sequences=True, input_shape=(x_train.shape[1],1))) model.add(LSTM(units=50)) model.add(Dense(1)) model.fit(x_train, y_train, epochs=1, batch_size=1, verbose=2) #predicting 246 values, using past 60 from the train data inputs = new_data[len(new_data) - len(valid) - 60:].values inputs = inputs.reshape(-1,1) inputs = scaler.transform(inputs) X_test = [] for i in range(60,inputs.shape[0]): X_test.append(inputs[i-60:i,0]) X_test = np.array(X_test) X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1)) closing_price = model.predict(X_test) closing_price = scaler.inverse_transform(closing_price) Results rms=np.sqrt(np.mean(np.power((valid-closing_price),2))) rms 11.772259608962642 #for plotting train = new_data[:987] valid = new_data[987:] valid['Predictions'] = closing_price plt.plot(train['Close']) plt.plot(valid[['Close','Predictions']]) Inference

Wow! The LSTM model can be tuned for various parameters such as changing the number of LSTM layers, adding dropout value or increasing the number of epochs. But are the predictions from LSTM enough to identify whether the stock price will increase or decrease? Certainly not!

As I mentioned at the start of the article, stock price is affected by the news about the company and other factors like demonetization or merger/demerger of the companies. There are certain intangible factors as well which can often be impossible to predict beforehand.

Conclusion

Time series forecasting is a very intriguing field to work with, as I have realized during my time writing these articles. There is a perception in the community that it’s a complex field, and while there is a grain of truth in there, it’s not so difficult once you get the hang of the basic techniques.

Frequently Asked Questions

Q1. Is it possible to predict the stock market with Deep Learning?

A. Yes, it is possible to predict the stock market with Deep Learning algorithms such as moving average, linear regression, Auto ARIMA, LSTM, and more.

Q2. What can you use to predict stock prices in Deep Learning?

A. Moving average, linear regression, KNN (k-nearest neighbor), Auto ARIMA, and LSTM (Long Short Term Memory) are some of the most common Deep Learning algorithms used to predict stock prices.

Q3. What are the two methods to predict stock price?

A. Fundamental Analysis and Technical Analysis are the two ways of analyzing and predicting stock prices.

Related

Update the detailed information about How Machine Learning Is Enhancing Mobile Gadgets And Applications on the Kientrucdochoi.com website. We hope the article's content will meet your needs, and we will regularly update the information to provide you with the fastest and most accurate information. Have a great day!